Data acquisition method and device of rich energy sensor, equipment and medium

Abstract

The application provides a kind of data acquisition method, device, electronic equipment and medium of rich energy sensor, the data acquisition method, comprising: based on the first feature information of three-dimensional model, the image collected is matched, the corresponding area of image in three-dimensional model is determined, realizes pixel registration;Wherein, the first feature information is in the unit of pixel of three-dimensional model, including first point information, first line information, first surface information;According to the three-dimensional information of corresponding area in three-dimensional model, the pixel point of image is valued;The pixel point of image after valuation contains three-dimensional information;Determine the moving target in image;The attribute information of moving target is acquired, and the corresponding data measurement mode is selected based on attribute information;Wherein, data measurement mode is used to realize the corresponding function of rich energy sensor, including: specific target identification and measurement, meteorological element measurement, foreign matter detection, vibration wave extraction and micro-deformation measurement;Data measurement mode is executed, and measurement data is obtained.

Description

Technical Field

[0001] This invention relates to the field of multifunctional sensors, and mainly to a data acquisition method, device, equipment and medium for a high-energy sensor. Background Technology

[0002] There is an urgent need for real-time multi-element measurement in many application fields such as smart industry, smart agriculture, transportation, healthcare, and meteorology. Currently, multi-element measurement is mainly achieved through two methods: multi-element measurement through the coordinated measurement of multiple single sensors, and multi-element measurement through multi-source heterogeneous sensors. Among these, single sensors can only measure information in one dimension, and the cost of coordinated measurement is high, the operation is complex, the limitations are large, and it is difficult to apply on a large scale. Multi-source heterogeneous sensor measurements are difficult to correlate, fuse, and analyze, and situation generation is difficult. They require supporting communication equipment and specific network environments, resulting in high costs. Summary of the Invention

[0003] (a) Technical problems to be solved

[0004] This invention provides a data acquisition method, device, electronic device, and storage medium for a rich energy sensor, to solve the aforementioned technical problems.

[0005] (II) Technical Solution

[0006] The first aspect of this invention provides a data acquisition method for a rich-energy sensor, comprising: matching an acquired image with first feature information of a three-dimensional model to determine the corresponding region of the image in the three-dimensional model; wherein the first feature information is in pixels of the three-dimensional model and includes first point information, first line information, and first surface information; the first point information is the feature information of a pixel in the model, the first line information is the feature information of a line connecting different pixels in the model, and the first surface information is the feature information of a single target in the model; determining a moving target in the image; acquiring attribute information of the moving target, and determining a data measurement method based on the attribute information; wherein the attribute information includes the position information of the moving target in the three-dimensional model, and the position information is composed of the three-dimensional information of the pixels constituting the moving target; wherein the data measurement method is used to realize the corresponding functions of the rich-energy sensor, the functions including: specific target identification and measurement, meteorological element measurement, foreign object detection, vibration wave extraction, and micro-deformation measurement; executing the data measurement method to obtain measurement data.

[0007] Optionally, based on the first feature information of the 3D model, the acquired image is matched to determine the corresponding region of the image in the 3D model, including: extracting the second feature information of the acquired image; wherein the second feature information is in pixels of the image and includes second point information, second line information, and second surface information; the second point information is the feature information of pixels in the image, the second line information is the feature information of the line connecting adjacent pixels in the image, and the second surface information is the feature information of a single target in the image; matching the image with the 3D model is performed according to the first feature information and the second feature information to obtain a matching result; wherein the matching result is the corresponding region in the 3D model with the highest matching value to the image.

[0008] Optionally, matching the image and the 3D model based on the first feature information and the second feature information to obtain a matching result includes: sequentially matching the second point information with the first point information, the second line information with the first line information, and the second surface information with the first surface information to obtain point matching values, line matching values, and surface matching values; calculating the matching value between each region in the 3D model and the image; wherein the matching value is obtained by averaging the point matching value, line matching value, and surface matching value; and determining the region with the highest matching value as the matching result.

[0009] Optionally, before determining the moving target in the image, the method further includes: classifying the image content based on semantic segmentation to obtain multiple classification targets; wherein, determining the image content classification based on semantic segmentation includes: extracting features from image pixels to obtain a first feature map; each pixel in the feature map has a corresponding ground truth label, which is used to record the semantic information of the pixel; normalizing the first feature map based on the ground truth label to obtain a second feature map; determining the semantic category to which the pixel belongs based on the ground truth label of each pixel in the second feature map; grouping the feature map channels according to the semantic category to determine a feature group; and obtaining multiple targets based on the feature group; wherein the pixels of each target include semantic information.

[0010] Optionally, classifying image content based on semantic segmentation to obtain multiple targets also includes: assigning a value to each pixel of the target based on a 3D model, wherein the assigned pixel contains 3D information; wherein the 3D information is consistent with the 3D spatial vector information in the 3D model.

[0011] Optionally, determining a moving target in an image includes: performing a subtraction operation on the corresponding pixel values ​​in continuously acquired images to obtain a subtracted image; extracting pixels from the subtracted image based on a set threshold to obtain the moving target; and / or performing a difference operation between the currently acquired image and the background image to obtain a grayscale image of the moving target region; extracting the moving region in the grayscale image based on a set threshold to obtain the moving target; wherein the background image is updated according to the currently acquired image.

[0012] Optionally, the attribute information of the moving target is obtained, and the data measurement method is determined based on the moving target information, including: establishing a coupling relationship between the moving target and the fixed target; obtaining the attribute information of the moving target based on the semantic information and three-dimensional information of the moving target; wherein the attribute information of the moving target includes the shape, position and motion parameters of the moving target.

[0013] A second aspect of the present invention provides a data acquisition device, comprising: a matching module, used to match acquired images based on first feature information of a three-dimensional model to determine the corresponding region of the image in the three-dimensional model; wherein the first feature information is in units of pixels of the three-dimensional model and includes first point information, first line information, and first surface information; the first point information is the feature information of a pixel in the model, the first line information is the feature information of a line connecting different pixels in the model, and the first surface information is the feature information of a single target in the model; an assignment module, used to assign values ​​to the pixels of the image according to the three-dimensional information of the corresponding region in the three-dimensional model; the pixels of the image after assignment contain three-dimensional information; a determination module, used to determine a moving target in the image; a function selection module, used to acquire attribute information of the moving target and select a corresponding data measurement method based on the attribute information; wherein the attribute information includes the position information of the moving target in the three-dimensional model, and the position information is composed of the three-dimensional information of the pixels constituting the moving target; the data measurement method is used to realize the corresponding functions of a rich energy sensor, the functions including: specific target identification and measurement, meteorological element measurement, foreign object detection, vibration wave extraction, and micro-deformation measurement; and a measurement module, used to execute the data measurement method to obtain measurement data.

[0014] A third aspect of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements each step of the data acquisition method according to any one of claims 1 to 8.

[0015] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements each step of the data acquisition method according to any one of claims 1 to 8.

[0016] (III) Beneficial Effects

[0017] The data acquisition method, apparatus, device, and medium for the energy-rich sensor provided by this invention have at least the following beneficial effects:

[0018] By matching the acquired images using three dimensions—point information, line information, and surface information—the accuracy of feature matching is effectively improved, enabling precise pairing of the image with the corresponding region of the 3D model, thereby improving the precise pairing of subsequent measurement data with the 3D model region.

[0019] By identifying moving targets, the data measurement method is selected. Based on the selected data measurement method, the corresponding detector in the Funeng sensor is activated, enabling the Funeng sensor to measure multiple elements in real time, thereby improving the breadth and scalability of Funeng sensor applications. Attached Figure Description

[0020] Figure 1 The flowchart illustrating the data acquisition method of the energy-rich sensor in an embodiment of the present invention is shown in the schematic diagram.

[0021] Figure 2 This illustration shows a flowchart of matching acquired images in an embodiment of the present invention;

[0022] Figure 3 This illustration schematically shows an embodiment of the present invention. Figure 3 The flowchart illustrating the process of obtaining the matching result in an embodiment of the present invention is shown schematically.

[0023] Figure 4 This illustration shows a flowchart of image content classification based on semantic segmentation in an embodiment of the present invention;

[0024] Figure 5 This schematically illustrates a flowchart for determining a moving target in an image according to an embodiment of the present invention;

[0025] Figure 6 A block diagram of an energy-rich sensor in an embodiment of the present invention is shown schematically;

[0026] Figure 7 A block diagram of a data acquisition device in an embodiment of the present invention is shown schematically.

[0027] Figure 8 The diagram illustrates the hardware structure of an electronic device in an embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0029] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0030] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection, an electrical connection, or a connection that allows communication between them; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0031] In the description of this invention, it should be understood that the terms "longitudinal", "length", "circumferential", "front", "rear", "left", "right", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the subsystem or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0032] Throughout the accompanying drawings, identical elements are represented by the same or similar reference numerals. Conventional structures or configurations may be omitted where they might cause confusion in understanding the invention. Furthermore, the shapes, sizes, and positional relationships of the components in the drawings do not reflect actual size, scale, or actual positional relationships. Additionally, any reference numerals placed between parentheses in the claims should not be construed as limiting the claims.

[0033] Similarly, to simplify the invention and aid in understanding one or more of the various disclosed aspects, in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together into a single embodiment, figure, or description thereof. The use of terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicates that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0034] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0035] Figure 1 The flowchart illustrating the data acquisition method of the energy-rich sensor in an embodiment of the present invention is shown.

[0036] like Figure 1 As shown, the data acquisition method includes operations S110 to S150, and this data acquisition method is applied to the energy sensor.

[0037] In operation S110, based on the first feature information of the three-dimensional model, the acquired image is matched to determine the corresponding region of the image in the three-dimensional model; wherein, the first feature information is in units of pixels of the three-dimensional model, including first point information, first line information, and first surface information; the first point information is the feature information of the pixel in the model, the first line information is the feature information of the line connecting different pixel points in the model, and the first surface information is the feature information of a single target in the model;

[0038] In operation S120, the pixel points of the image are assigned values ​​based on the three-dimensional information of the corresponding region in the three-dimensional model; the pixel points of the image after assignment contain three-dimensional information.

[0039] In operation S130, the moving target in the image is determined;

[0040] In operation S140, the attribute information of the moving target is acquired, and the data measurement method is determined based on the attribute information; wherein, the attribute information includes the position information of the moving target in the three-dimensional model, and the position information is composed of the three-dimensional information of the pixels constituting the moving target; the data measurement method is used to realize the corresponding functions of the energy sensor, the functions including: specific target identification and measurement, meteorological element measurement, foreign object detection, vibration wave extraction and micro-deformation measurement;

[0041] In operation S150, the data measurement mode is executed to obtain measurement data.

[0042] This invention matches acquired images using three dimensions: point information, line information, and surface information, achieving pixel registration and effectively improving the accuracy of feature matching. This enables precise pairing of the image with the corresponding region of the 3D model, improving the accuracy of subsequent measurement data pairing with the 3D model region. Furthermore, based on the identification of moving targets, the corresponding data measurement method is selected, achieving precise matching between the data measurement method and the target being measured, effectively improving measurement accuracy. Based on the selected data measurement method, the corresponding detector in the Funeng sensor is activated, enabling real-time measurement of multiple elements by the Funeng sensor, thus improving the breadth and scalability of the Funeng sensor's application scenarios.

[0043] Figure 2 The flowchart illustrating the matching of acquired images in an embodiment of the present invention is shown in the illustration.

[0044] like Figure 2 As shown, matching the acquired images includes operations S210 to S220.

[0045] In operation S210, the second feature information of the acquired image is extracted.

[0046] The second feature information is in units of pixels in the image and includes second point information, second line information, and second surface information. The second point information is the feature information of pixels in the image, the second line information is the feature information of the line connecting adjacent pixels in the image, and the second surface information is the feature information of a single target in the image.

[0047] In this embodiment of the invention, a compressed neural network is used for feature information extraction. The compression method of the neural network includes: obtaining redundant parameters in the neural network layer by layer; modifying the weights of the redundant parameters to 0 to obtain a first neural network; and decomposing the convolution kernel matrix in the first neural network into a row convolution kernel matrix and a column convolution kernel matrix to obtain the compressed neural network. Furthermore, a hardware predictor can be used to analyze and predict the acceleration of the compressed neural network deployed in actual hardware, and feed this analysis back to the guidance gating, effectively improving the speed and accuracy of feature extraction from the compressed neural network. In this embodiment of the invention, an adaptive mapping algorithm for absolute geographic coordinates and feature labels is used to achieve one-to-one registration of geographic coordinates and fixed target features.

[0048] In operation S220, the image and the 3D model are matched based on the first feature information and the second feature information to obtain the matching result.

[0049] The matching result is the region in the 3D model that has the highest matching value with the image.

[0050] Figure 3 The flowchart illustrating the matching results in an embodiment of the present invention is shown schematically.

[0051] like Figure 3 As shown, the matching results obtained include operations S310 to S330.

[0052] In operation S310, the second point information is matched with the first point information, the second line information with the first line information, and the second surface information with the first surface information in sequence to obtain the point matching value, the line matching value, and the surface matching value.

[0053] In operation S320, the matching value between each region in the 3D model and the image is calculated; the matching value is obtained by averaging the point matching value, line matching value, and surface matching value.

[0054] In operation S330, the region with the highest matching value is determined as the matching result.

[0055] In this embodiment of the invention, feature extraction and image matching are performed through a multi-feature combination approach. Image matching is conducted from three dimensions: points, lines, and surfaces. This effectively solves the problems of rapid composite of geographic coordinates in different environments with the angle grid points of the energy-rich sensor, and the difficulty of ensuring high similarity between feature points in images and 3D point clouds using traditional feature point and corner point-based methods, thus failing to achieve robust feature point matching results. Based on the relatively invariant geometric structure of different environmental scenes, line features and surface (region) features, in addition to point features, are used for image feature matching, effectively improving the accuracy of feature matching and achieving precise pairing between the image and the corresponding region of the 3D model. This, in turn, improves the precise pairing of subsequent measurement data with the 3D model region.

[0056] In this embodiment of the invention, pixels of the image are assigned values ​​based on the three-dimensional information of the corresponding region in the three-dimensional model. The assigned pixels contain three-dimensional information. This three-dimensional information is determined based on the spatial coordinates and temporal information of the three-dimensional model. Each assigned pixel contains dynamic vector information (x, y, z, t). That is, in this embodiment of the invention, each pixel not only contains two-dimensional planar information (x, y), but also three-dimensional spatial information (z) and temporal dimension information (t).

[0057] In this embodiment of the invention, a 3D model serves as a data storage carrier compatible with geographic information, used for storing, managing, updating, and optimizing multi-source heterogeneous urban data. This 3D model has a unified multi-scale framework, capable of compatiblely and uniformly managing 3D urban models and geographic information of different sizes. The method for constructing the 3D model includes: uniformly rasterizing the urban 3D model to obtain a 3D urban voxel model; this 3D urban voxel model consists of multiple 3D voxels, each containing 3D spatial location information. A 3D urban model database of different data types is constructed using 3D voxels as units. Attribute information of the voxels is obtained through interpolation to obtain multiple knowledge elements, wherein each knowledge element contains 3D voxels storing different urban information. A 3D model database is constructed based on a multi-attribute fusion mechanism of knowledge elements. Users select the appropriate 3D model from the 3D model database for data collection based on their needs.

[0058] Furthermore, when constructing 3D model features, this embodiment of the invention proposes to utilize a massive 3D scene scheduling scheme, a multi-level, hybrid spatial index structure, and a complex 3D scene data organization method based on scene graphs and a visual perception-driven complex 3D scene data scheduling method to achieve real-time calculation of view frustum scene culling, occlusion culling, and screen space error, constraining the adaptive scheduling of the model at different levels of detail. This effectively solves the problems of low local scene data scheduling and loading efficiency and relative rendering latency in complex urban 3D scenes. Based on this, virtual texture technology is applied to large-scale texture data management, the mapping method between virtual textures and physical texture blocks, and the determination of texture page levels based on the size of objects in the scene space, to complete the virtual texture-based rendering pipeline design. This allows for rapid loading of texture blocks required by the current local model, reducing bandwidth consumption and rendering batches, thereby effectively improving data scheduling efficiency and accelerating the construction of local 3D scenes.

[0059] In this embodiment of the invention, before operating S120 to determine the moving target in the image, the method further includes: classifying the image content based on semantic segmentation to obtain multiple classification targets.

[0060] Figure 4 The flowchart illustrating the classification of image content based on semantic segmentation in an embodiment of the present invention is shown.

[0061] like Figure 4 As shown, the classification of image content based on semantic segmentation includes operations S410 to S450.

[0062] During operation of S410, feature extraction is performed on the image pixels to obtain the first feature map.

[0063] In this feature map, each pixel has a corresponding truth label, which is used to record the semantic information of the pixel.

[0064] In operation S420, the first feature map is normalized based on the truth label to obtain the second feature map.

[0065] In operation S430, the semantic category to which the pixel belongs is determined based on the truth label of each pixel in the second feature map.

[0066] In operation S440, feature map channels are grouped according to their semantic categories to determine feature groups.

[0067] In operation S450, multiple classification targets are obtained based on feature groups; wherein the pixels of each classification target include semantic information.

[0068] In this embodiment of the invention, semantic segmentation divides an image into multiple targets, each composed of multiple pixels, facilitating subsequent identification of moving targets. Furthermore, in the semantic segmentation stage, this embodiment introduces feature map normalization and feature channel grouping to process image pixels. Feature map normalization standardizes the category feature regions in the image, mitigating feature mismatch issues caused by style variations and effectively increasing the generalization ability of the semantic segmentation method. Channel feature grouping calculates the weight of each channel feature and groups channels with similar weights within each category, aligning channel distribution without loss of semantic information, further enhancing deassociation between channels and improving the accuracy of semantic segmentation.

[0069] Furthermore, the process of classifying image content based on semantic segmentation to obtain multiple targets also includes: assigning a value to each pixel of the target based on a 3D model, with each pixel containing 3D information. This 3D information is determined based on the spatial coordinates and temporal information of the 3D model, and each pixel contains dynamic vector information (x, y, z, t). That is, in this embodiment of the invention, each pixel not only contains two-dimensional planar information (x, y), but also three-dimensional spatial information (z) and temporal dimension information (t).

[0070] Figure 5 The flowchart illustrating the determination of moving targets in an image is shown in an embodiment of the present invention.

[0071] like Figure 5 As shown, determining the moving target in the image includes operations S410 to S440.

[0072] In operation S510, a subtraction operation is performed on the corresponding pixel values ​​in the continuously acquired images to obtain the subtracted image;

[0073] In operation S520, pixel extraction is performed on the subtracted image based on a set threshold to obtain the moving target;

[0074] When operating the S530, the currently acquired image is compared with the background image to obtain a grayscale image of the moving target area.

[0075] In operation S540, the motion region in the grayscale image is extracted based on a set threshold to obtain the moving target; wherein, the background image is updated according to the currently acquired image.

[0076] In this embodiment of the invention, determining the data measurement method based on the moving target information includes steps S610 to S620.

[0077] In operating S610, the coupling relationship between moving targets and stationary targets is established;

[0078] In operation S620, the attribute information of the moving target is obtained based on the semantic and three-dimensional information of the moving target; the attribute information of the moving target includes the shape, position and motion parameters of the moving target.

[0079] In this embodiment of the invention, for example, the type of moving target can be determined based on the attribute information of the moving target, and the corresponding data measurement method can be determined. By executing the measurement method, measurement data is obtained and uploaded to the three-dimensional model.

[0080] For example, measurement methods may include specific target identification and measurement, meteorological element measurement, foreign object detection, and vibration wave extraction and micro-deformation measurement. Specific target identification and measurement includes: when the moving target is perceived as a person, sensing information such as the person's movement and position; when the moving target is a car, sensing information such as the car's movement and position. Meteorological element measurement includes: when the moving target is a leaf, sensing wind speed; when the moving target is a raindrop, sensing rainfall; when the moving target is puddles, sensing water depth; when the moving target is the ocean, sensing wave speed; when the moving target is snow, sensing snowfall; when the moving target is hail, sensing hail level / intensity. Foreign object detection includes: when the moving target is metal, sensing the metal type and shape. Vibration wave extraction and micro-deformation measurement includes: when the moving target is a person or a vehicle, sensing non-line-of-sight targets. It should be noted that the above measurement methods are merely illustrative examples for the convenience of those skilled in the art and do not represent that the technical solution of this application is limited to implementing the above measurement methods.

[0081] In this embodiment of the invention, both image acquisition and data measurement are performed using a rich-energy sensor. This rich-energy sensor consists of multiple individual detectors. Once the data measurement method is determined, the corresponding individual detector performs the measurement, obtains the measurement data, and uploads it to the 3D model.

[0082] Figure 6 A block diagram of an energy-rich sensor in an embodiment of the present invention is shown schematically.

[0083] according to Figure 6It is known that the basic components of this FunEnergy sensor include: a visible light image sensing module, a satellite positioning chip, an inertial navigation module, a computing chip embedded with environmental feature clouds (such as urban feature clouds), a data bus, and a power supply module. Furthermore, depending on actual needs, auxiliary modules such as infrared image sensing modules, lasers, and wireless communication can be optionally installed on the FunEnergy sensor.

[0084] The visible light image acquisition module is used to acquire two-dimensional time-dependent image data. It employs a human-eye-like design with wide and narrow field-of-view fusion, featuring high resolution in the center field of view and lower resolution at the edges. This allows the camera module to balance high resolution and wide field-of-view perception capabilities. Utilizing the complementarity and redundancy of scene information descriptions from different perspectives, it meets the diverse data requirements of various functions within the energy-rich sensor. This results in video streams with rich spatiotemporal resolution, which, after registration with urban feature clouds, provides abundant foundational data support for the energy-rich sensor.

[0085] The infrared image sensing module is used to acquire two-dimensional time-dependent image data under specific conditions. For example, when the visible light image acquisition module is affected by adverse conditions such as low light or fog, activating the infrared sensing module can resist these interferences. Therefore, in special applications, the complementarity of infrared and visible light cameras can be utilized to generate images with richer information dimensions, providing richer basic data support for the various functions of the sensor.

[0086] The city 3D feature cloud module provides 3D coordinate information for video or image data. It is a lightweight 3D model of the city, typically built upon a fixed group of urban targets, and carries rich information on urban feature points and absolute geographical locations. In target and environment perception, it assigns information such as location, size, and target trajectory to sensor data.

[0087] The navigation and positioning module is an embedded chip used to provide coarse positioning for the Fu-Neng sensor. It enables the computing chip to extract local feature maps from a massive urban feature map, facilitating efficient registration between real-time sensor images and the urban feature map. Furthermore, the navigation and positioning module can also directly provide vector and non-vector positioning services to the sensor or platform.

[0088] The inertial navigation module is used to obtain the azimuth information of the energy-rich sensor after deployment, calculate the sensor's observation angle, and use image matching algorithms to quickly obtain the spatial coordinate information of each pixel in the image, thus providing basic support for the final target and environmental perception. High-precision laser or fiber optic gyroscopes are typically used.

[0089] The computing and storage chip is used to store and retrieve the city's 3D feature cloud and process the input sensor video streams, realizing various functions of the energy-rich sensor by combining multiple algorithms. These algorithms include video registration, target detection, vibration wave extraction, and meteorological element perception.

[0090] Laser and other auxiliary modules are used to provide a reference for image data and improve data acquisition accuracy. Funsens sensors can be equipped with optional modules such as laser ranging, lidar, and microwave detection to obtain more application functions, such as precise modeling, image localization, micro-deformation measurement, and target perception. Battery and wireless communication modules can also be added for flexible use.

[0091] Among them, the satellite positioning chip can adopt Beidou positioning technology, or radar positioning technology, high-precision IMU positioning technology, GPS positioning technology, hybrid positioning technology, etc.

[0092] The energy-rich sensor provided in this invention effectively overcomes the drawbacks of multi-source heterogeneous sensors, such as the high difficulty in data correlation, fusion, and analysis, and the difficulty in simultaneously achieving high measurement accuracy and scale with a single sensor. The energy-rich sensor provided in this invention achieves real-time measurement of multiple elements through pixel registration based on time-dependent image data and 3D environmental feature point clouds, combined with environmental pixel information related to target occlusion, and using a single detector corresponding to the data measurement method. This energy-rich sensor has the advantages of low cost, high measurement accuracy, and wide measurement range.

[0093] Figure 7 A block diagram of a data acquisition device 700 in an embodiment of the present invention is shown schematically.

[0094] This device can be built into an electronic device to perform data acquisition methods applied to energy-rich sensors. Figure 7 It can be seen that the device mainly includes:

[0095] The matching module 710 is used to match the acquired image based on the feature information of the 3D model to determine the corresponding region of the image in the 3D model; wherein, the 3D feature information includes point information, line information, and surface information. The first point information is the feature information of the pixels in the model, the first line information is the feature information of the line connecting different pixels in the model, and the first surface information is the feature information of a single target in the model.

[0096] The assignment module 720 is used to assign values ​​to the pixels of the image based on the three-dimensional information of the corresponding region in the three-dimensional model; the pixels of the image after assignment contain three-dimensional information.

[0097] The module 730 determines the moving targets in the image.

[0098] The function selection module 740 is used to acquire the attribute information of the moving target and select the corresponding data measurement method based on the attribute information. The attribute information includes the position information of the moving target in the three-dimensional model, and the position information is composed of the three-dimensional information of the pixels constituting the moving target. The data measurement method is used to realize the corresponding functions of the energy sensor, including: specific target identification and measurement, meteorological element measurement, foreign object detection, vibration wave extraction and micro-deformation measurement.

[0099] The measurement module 750 is used to perform data measurement and obtain measurement data.

[0100] Figure 8 The diagram illustrates the hardware structure of an electronic device in an embodiment of the present invention.

[0101] The electronic device described in this embodiment of the invention includes: a memory 81, a processor 82, and a computer program stored in the memory 81 and executable on the processor. When the processor executes the program, it implements the aforementioned... Figure 1 The data acquisition method described in the illustrated embodiment.

[0102] Furthermore, the electronic device also includes: at least one input device 83; and at least one output device 84. The memory 81, processor 82, input device 83, and output device 84 are connected via a bus 85. Specifically, the input device 83 may be a camera, touch panel, physical button, or mouse, etc. The output device 84 may be a display screen. The memory 81 may be a high-speed random access memory (RAM) or non-volatile memory, such as a disk drive. The memory 81 stores a set of executable program code, and the processor 82 is coupled to the memory 81.

[0103] Furthermore, embodiments of the present invention also provide a computer-readable storage medium, which may be disposed in the electronic device described in the above embodiments, and the computer-readable storage medium may be as described above. Figure 8 The electronic device in the illustrated embodiment. A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the aforementioned... Figure 1 The data acquisition method described in the illustrated embodiment. Further, the computer storage medium can also be a USB flash drive, external hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, or any other medium capable of storing program code.

[0104] It should be noted that the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product.

[0105] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, as some steps can be performed in other orders or simultaneously according to the present invention. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention. In the above embodiments, the descriptions of each embodiment have their own emphasis; for parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0106] The above is a description of a data acquisition method, device, electronic device, and storage medium for a rich energy sensor provided by the present invention. For those skilled in the art, based on the ideas of the embodiments of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A data acquisition method for an energy-rich sensor, characterized in that, include: Based on the first feature information of the 3D model, the acquired image is matched to determine the corresponding region of the image in the 3D model; wherein, the first feature information is in units of pixels of the 3D model, including first point information, first line information, and first surface information; the first point information is the feature information of a pixel in the model, the first line information is the feature information of a line connecting different pixels in the model, and the first surface information is the feature information of a single target in the model; The pixels of the image are assigned values ​​based on the three-dimensional coordinates of the corresponding region in the three-dimensional model; the pixels of the image after assignment contain three-dimensional information. Identify moving targets in the image after assignment; The attribute information of the moving target is acquired, and a corresponding data measurement method is selected based on the attribute information; wherein, the attribute information includes the position information of the moving target in the three-dimensional model, and the position information is composed of the three-dimensional information of the pixels constituting the moving target; the data measurement method is used to realize the corresponding functions of the energy sensor, and the functions include: specific target identification and measurement, meteorological element measurement, foreign object detection, vibration wave extraction and micro-deformation measurement; Perform the data measurement method described above to obtain measurement data; Before determining the moving targets in the image, the image content is classified based on semantic segmentation to obtain multiple classification targets; The semantic segmentation-based image content classification includes: extracting features from image pixels to obtain a first feature map; each pixel in the feature map has a corresponding ground truth label, which records the semantic information of the pixel; normalizing the first feature map based on the ground truth label to obtain a second feature map; determining the semantic category to which the pixel belongs based on the ground truth label of each pixel in the second feature map; grouping the feature map channels according to the semantic category to determine a feature group; and obtaining multiple classification targets based on the feature group; wherein the pixels of each classification target include semantic information.

2. The data acquisition method according to claim 1, characterized in that, The method of matching the acquired image with the first feature information based on the 3D model to determine the corresponding region of the image in the 3D model includes: Extract the second feature information of the acquired image; wherein, the second feature information is in pixels of the image and includes second point information, second line information, and second surface information; the second point information is the feature information of a pixel in the image, the second line information is the feature information of a line connecting adjacent pixels in the image, and the second surface information is the feature information of a single target in the image; The image and the 3D model are matched based on the first feature information and the second feature information to obtain the matching result; wherein, the matching result is the corresponding region in the 3D model with the highest matching value with the image.

3. The data acquisition method according to claim 2, characterized in that, The matching between the image and the 3D model based on the first feature information and the second feature information to obtain the matching result includes: The second point information is matched with the first point information, the second line information with the first line information, and the second surface information with the first surface information in sequence to obtain the point matching value, the line matching value, and the surface matching value. Calculate the matching value between each region in the 3D model and the image; wherein the matching value is obtained by averaging the point matching value, line matching value, and area matching value; The region with the highest matching value is selected as the matching result.

4. The data acquisition method according to claim 2, characterized in that, Feature information is extracted from the compressed neural network, wherein the compression method of the neural network includes: Obtain redundant parameters in the neural network layer by layer; The weights of the redundant parameters are modified to 0 to obtain the first neural network; The convolution kernel matrix in the first neural network is decomposed into a row convolution kernel matrix and a column convolution kernel matrix to obtain the compressed neural network.

5. The data acquisition method according to claim 1, characterized in that, Determining the moving target in the image includes: Perform a subtraction operation on the corresponding pixel values ​​in the continuously acquired images to obtain the subtracted image; Pixels are extracted from the subtracted images based on a set threshold to obtain the moving target; The grayscale image of the moving target region is obtained by performing a difference operation between the currently acquired image and the background image. The motion region in the grayscale image is extracted based on a set threshold to obtain the moving target; wherein the background image is updated according to the currently acquired image.

6. The data acquisition method according to claim 1, characterized in that, The step of acquiring the attribute information of the moving target and determining the data measurement method based on the moving target information includes: Establish the coupling relationship between the moving target and the stationary target; Based on the semantic and three-dimensional information of the moving target, the attribute information of the moving target is obtained; wherein, the attribute information of the moving target includes the shape, position, and motion parameters of the moving target.

7. A data acquisition device for implementing the data acquisition method according to any one of claims 1 to 6, characterized in that, include: The matching module is used to match the acquired image based on the first feature information of the three-dimensional model to determine the corresponding region of the image in the three-dimensional model; wherein, the first feature information is in pixels of the three-dimensional model and includes first point information, first line information, and first surface information; the first point information is the feature information of a pixel in the model, the first line information is the feature information of a line connecting different pixels in the model, and the first surface information is the feature information of a single target in the model; The assignment module is used to assign values ​​to the pixels of the image based on the three-dimensional information of the corresponding region in the three-dimensional model; the pixels of the image after assignment contain three-dimensional information. The determination module is used to determine moving targets in the image; The function selection module is used to acquire the attribute information of the moving target and select the corresponding data measurement method based on the attribute information; wherein, the attribute information includes the position information of the moving target in the three-dimensional model, and the position information is composed of the three-dimensional information of the pixels constituting the moving target; the data measurement method is used to realize the corresponding functions of the energy sensor, including: specific target identification and measurement, meteorological element measurement, foreign object detection, vibration wave extraction and micro-deformation measurement; The measurement module is used to perform the data measurement method and obtain measurement data.

8. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements each step of the data acquisition method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements each step of the data acquisition method according to any one of claims 1 to 6.

Images