High-precision three-dimensional map construction method and system based on multi-modal feature fusion unmanned aerial vehicle inspection

Abstract

The application belongs to the technical field of unmanned aerial vehicle intelligent inspection, and more particularly relates to a high-precision three-dimensional map construction method and system based on multi-modal feature fusion unmanned aerial vehicle inspection. The method comprises the following steps: acquiring and synchronizing point cloud data and image data, then converting the laser radar point cloud data to the camera coordinate system, and then performing noise reduction processing and distortion correction; dynamically distributing radar and camera sensor weights according to the environmental light intensity and point cloud density; extracting radar point cloud features and camera visual features, then distributing the corresponding features according to the radar and camera sensor weights, and finally performing multi-modal fusion to obtain fusion features; and performing joint SLAM three-dimensional mapping according to the fusion features to generate a three-dimensional map. The application solves the problems of insufficient sensor complementarity, poor environmental adaptability and low map precision in traditional inspection.

Description

Technical Field

[0001] This invention belongs to the technical field of intelligent inspection of unmanned aerial vehicles (UAVs), and more specifically, relates to a method and system for constructing high-precision three-dimensional maps for UAV inspection based on multimodal feature fusion. Background Technology

[0002] With the maturity of drone technology, the workload of map and line inspection is increasing rapidly. Traditional manual inspection is far from meeting the needs of today's smart grid construction. Using drones for intelligent inspection can reduce workload, lower operation and maintenance costs, and improve inspection quality and scientific management technology. It has important research significance and broad application prospects in the construction of high-precision 3D maps.

[0003] Chinese patent document CN117008150A discloses a high-precision map construction method and system based on lidar and camera fusion, including: acquiring multi-view images, camera intrinsic and extrinsic parameters, and point cloud data at the same timestamp from multiple view cameras, and preprocessing the acquired multi-view and point cloud data to obtain preprocessed multi-view images and point cloud data, and inputting all preprocessed point cloud data, multi-view images, and projection matrices corresponding to the multi-view images into a pre-trained multimodal high-precision map construction model to obtain prediction results corresponding to different road categories. The prediction results include the coordinate point set of lane lines, curbs, and sidewalks.

[0004] Currently, drone inspections mainly rely on single sensors. Existing fusion solutions often employ static weight allocation or simple data overlay, failing to consider sensor complementarity in dynamic environments, resulting in insufficient robustness in complex scenarios. LiDAR SLAM for map building is prone to loop closure false detections in repetitive structural scenes, such as dense power transmission tower clusters, leading to excessive cumulative map errors. Visual SLAM suffers from severe pose estimation drift under dynamic occlusion conditions, such as moving vehicle scenes. Summary of the Invention

[0005] The present invention aims to overcome at least one of the defects of the prior art and provide a high-precision 3D map construction method and system based on multimodal feature fusion for UAV inspection, so as to solve the problems of target detection accuracy and map construction robustness of human and machine in complex environments, as well as the problem that multi-sensor fusion schemes do not consider the complementarity of sensors in dynamic environments.

[0006] The detailed technical solution of this invention is as follows: A high-precision 3D map construction method based on multimodal feature fusion for UAV inspection, the method comprising: S1. Acquire point cloud data and image data and synchronize them in time. Then, convert the LiDAR point cloud data to the camera coordinate system and perform noise reduction and distortion correction. S2. Dynamically allocate the weights of radar and camera sensors based on ambient light intensity and point cloud density; S3. Extract radar point cloud features and camera visual features, then assign them to the corresponding features according to the radar and camera sensor weights, and finally perform multimodal fusion to obtain fused features. S4. Perform joint SLAM 3D mapping based on fusion features.

[0007] Furthermore, S1 specifically includes: S11. The lidar and camera sensors mounted on the UAV acquire point cloud data and image data, respectively. S12. The acquisition timestamps of the LiDAR and camera sensors are synchronized by clock. The LiDAR sampling frequency is 10Hz and the camera frequency is 30Hz. The spatiotemporal information is aligned by cubic spline interpolation. S13. Transform the lidar point cloud to the camera coordinate system: (1); In formula (1), q is the radar point cloud transformation matrix in the transformed camera coordinate system, where R and t are calibration parameters, and q is the point cloud coordinate in the radar coordinate system. S14. Perform noise reduction processing on the LiDAR point cloud transformed to the camera coordinate system: The DBSCAN algorithm is used to remove outliers in the point cloud. The density of data points is defined by setting two parameters: neighborhood distance ε and minimum number of points minPts in the neighborhood of the core point. By identifying high-density and low-density regions, the points are divided into core points and non-core points, and the points in the low-density regions are treated as noise for noise reduction. S15. Perform ICP registration on the continuous lidar point cloud frames, and at the same time, use bilinear interpolation of the image to correct the distortion of the image data.

[0008] Furthermore, the dynamic allocation of radar and camera sensor weights based on ambient light intensity and point cloud density specifically involves: (2); In formula (2), For radar weights, Where I is the camera weight, I is the ambient light intensity, and D is the point cloud density; In addition, a light intensity threshold and a point cloud density threshold are set: When the light intensity I is less than the set light intensity threshold, the visual weight, i.e. the image data weight, is forcibly set to 0, and only LiDAR point cloud data is used. When the point cloud density D is less than the set point cloud density threshold, the weight of the LiDAR point cloud data is forcibly set to 0, and only image data is used.

[0009] Furthermore, the point cloud density threshold is set as a dynamic threshold, and the threshold range of the point cloud density threshold is dynamically adjusted according to actual needs.

[0010] Furthermore, the extraction of radar point cloud features and camera visual features is as follows: A multimodal extraction module is constructed, including a radar point cloud data feature extraction network and a camera image data feature extraction network; The radar point cloud data feature extraction network extracts the x, y, z, and i features of each spatial point in the radar data through a first feature extraction module and a second feature extraction module, respectively. Multiple extraction modules are used to enhance the extracted features. x, y, and z represent the coordinates of each point in the point cloud in three-dimensional space, and i represents the signal strength value returned by the radar for each point. The extracted features are then upsampled to the point cloud features. eigenvector space; A camera image data feature extraction network extracts color and spatial features from camera image data. This network comprises three convolutional layers, three batch normalization layers, and three activation function layers, extracting visual features. In the eigenvector space; The camera image data feature extraction network is stacked in a 3-layer convolutional layer, which consists of a convolutional layer, a normalization layer, and an activation function layer.

[0011] Furthermore, based on the weights assigned to the corresponding features according to the radar and camera sensor weights, multimodal fusion is finally performed to obtain the fused features, specifically as follows: Construct a multimodal fusion module, including: Input layer: Radar point cloud features and camera visual features , where n and m are the feature dimensions of the point cloud and the image, respectively; Attention layer: Enhances cross-modal feature association through self-attention mechanism, outputting fused features. Radar point cloud features are used as the query, and camera visual features are used as the key and value; attention weights are calculated using the standard scaled dot product method. (3); Calculate attention weights: , The meaning is the dimension of the key tensor, which yields the fused features. : (4); In formula (4), σ is the ReLU activation function; Output layer: Input the fused features into the convolutional network to generate the multimodal fused feature extraction result, i.e., the multimodal fused feature map.

[0012] Furthermore, the joint SLAM 3D mapping based on the fusion features specifically involves: Dense depth completion: The size of the fused feature map is restored by deconvolution upsampling, generating a dense depth map with 8 channels; Point cloud coordinate mapping: Mapping dense depth maps back to spatial point clouds, and the resulting fused feature map is restored to the spatial point cloud representation through spatial coordinate transformation; Finally, SLAM 3D mapping was combined with Open3D to generate a 3D map.

[0013] The present invention also includes a system for constructing a high-precision 3D map based on multimodal feature fusion UAV inspection, the system comprising: a UAV, a hardware module, and a software module; The hardware modules include: a LiDAR sensor, a vision module (camera), and a computing unit; LiDAR sensor: used to acquire radar point cloud data; Vision module: used to acquire image data; Computational unit: used for feature extraction, feature fusion, and map construction of data generated by the two sensors; The software modules include: a data synchronization module, a dynamic weight allocation module, and a 3D map construction module. Data synchronization module: Enables sensor clock synchronization and data distribution via ROS; Dynamic weight allocation module: calculates radar weights and camera weights in real time and outputs them to the calculation unit; 3D map building module: It performs point cloud processing based on PCL and then generates a 3D map through Open3D.

[0014] On the other hand, the present invention also includes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a method for constructing a high-precision 3D map for UAV inspection based on multimodal feature fusion. Compared with the prior art, the beneficial effects of the present invention are as follows: (1) The high-precision three-dimensional map construction method and system based on multi-modal feature fusion for UAV inspection provided by the present invention adopts a multi-modal attention fusion network, enhances feature correlation through an attention mechanism, solves the problem of the failure of a single sensor under occlusion or low light, and automatically switches to high-weight sensor data when a certain sensor fails, such as when the camera is interfered by fog, ensuring data continuity.

[0015] (2) The high-precision three-dimensional map construction method and system based on multi-modal feature fusion for UAV inspection provided by the present invention adaptively adjusts the weights of radar and vision according to environmental light and point cloud density, improving sensor complementarity; at the same time, fuses the point cloud geometric structure and visual semantic information to generate a high-precision three-dimensional map with an accuracy ≤ 5 cm. Brief Description of the Drawings

[0016] Figure 1 is a flowchart of the map construction method described in the present invention.

[0017] Figure 2 is a schematic diagram of the feature extraction module and cross-modal fusion module described in Embodiment 1 of the present invention.

[0018] Figure 3 is a schematic diagram of three-dimensional map construction described in Embodiment 1 of the present invention. Detailed Embodiments

[0019] The present invention will be further described below in conjunction with the drawings and embodiments.

[0020] It should be noted that the following detailed descriptions are all exemplary and intended to provide further explanations of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.

[0021] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0022] Without conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0023] Embodiment 1 Refer Figure 1 , this embodiment provides a high-precision three-dimensional map construction method based on multi-modal feature fusion for UAV inspection, and the method includes: S1. Sensor data synchronization and preprocessing: acquire point cloud data and image data and synchronize them in time, then convert the LiDAR point cloud data to the camera coordinate system, and then perform noise reduction and distortion correction.

[0024] In this embodiment, after the UAV takes off, the onboard LiDAR and camera sensors acquire point cloud data and image data respectively. Then, the acquisition timestamps of the LiDAR and camera sensors are synchronized by a clock. The LiDAR sampling frequency is 10Hz and the camera is 30Hz. Cubic spline interpolation is used to align the spatiotemporal information. LiDAR coordinate transformation: This involves transforming LiDAR point cloud data from the radar coordinate system to the camera coordinate system. The formula is as follows: (1); In formula (1), q is the radar point cloud transformation matrix in the transformed camera coordinate system, where R and t are calibration parameters, and q is the point cloud coordinate in the radar coordinate system. Then, noise reduction processing is performed on the LiDAR point cloud transformed into the camera coordinate system: the DBSCAN algorithm is used to remove outliers in the point cloud. The density of data points is defined by setting two parameters: neighborhood distance ε and minimum number of points minPts in the core point neighborhood. By identifying high-density and low-density regions, the points are divided into core points and non-core points, and the points in the low-density regions are treated as noise for noise reduction processing.

[0025] ICP registration was performed on continuous lidar point cloud frames (50 iterations, error threshold 0.1m). Simultaneously, bilinear interpolation of the image was used to correct image distortion.

[0026] S2, Dynamic Weight Allocation Mechanism: Calculate the weights of the radar and camera sensors based on the ambient light intensity I (unit: lux) and the point cloud density D (unit: points / m²): (2); In formula (2), For radar weights, Camera weights; Dynamically adjusting the threshold: By setting the weights of the two modalities, the effect of feature fusion is improved, specifically: When the light intensity I is less than the set light intensity threshold, the visual weight is forcibly set to 0, and only the LiDAR point cloud data is used. When the point cloud density D is less than the set point cloud density threshold, the weight of the LiDAR point cloud data is forcibly set to 0, and only image data is used.

[0027] Furthermore, the point cloud density also includes a dynamic threshold: according to the actual task requirements, in this embodiment, the threshold range of the set point cloud density threshold is dynamically adjusted for high resolution required for power transmission line inspection. For example, the transmission tower area D ≥ 1000 points / m².

[0028] Preferably, in this embodiment, the illumination intensity I = 50 lux and the point cloud density D = 1000 points / At that time, the weight allocation was: radar weight: 0.67, camera weight: 0.33. The input was fed into the attention network, and the output fused features were used for target detection.

[0029] S3: Extract radar point cloud features and camera visual features and perform feature fusion: Extract radar point cloud features and camera visual features, then assign them to the corresponding features according to the radar and camera sensor weights, and finally perform multimodal fusion to obtain fused features.

[0030] Construct a multimodal extraction module, such as Figure 2 As shown, it includes: a radar point cloud data feature extraction network and a camera image data feature extraction network; Radar point cloud data feature extraction networks, such as VGG, ResNet, and PointNet++, extract local geometric features (normal vectors, curvature, point cloud density gradients, etc.): The x, y, z, and i features of each spatial point in the radar data are extracted using a first feature extraction module and a second feature extraction module. Multiple extraction modules enhance the extracted features, where x, y, and z are the coordinates of each point in the point cloud in three-dimensional space, and i is the intensity value of the signal returned by the radar for each point. These features are then upsampled to the point cloud features. The feature vector space, both feature extraction modules are used to extract the three-dimensional coordinates of point clouds in the space, and can further extract more fine-grained features; A camera image data feature extraction network extracts texture features (color histogram, edge direction, texture pattern) using a ResNet-50 network: The network extracts color and spatial features from camera image data. This network comprises three convolutional layers, three batch normalization layers, and three activation function layers, extracting features to a visual feature set. In the feature vector space; preferably, the stacking method of the camera image data feature extraction network is: cascaded (convolutional layer + normalization layer + activation function layer) * 3.

[0031] Construct a multimodal fusion module, such as Figure 2 As shown, it includes: Input layer: Radar point cloud features and camera visual features , where n and m are the feature dimensions of the point cloud and the image, respectively; Attention layer: Enhances cross-modal feature association through self-attention mechanism, outputting fused features. Radar point cloud features are used as the query, and camera visual features are used as the key and value; attention weights are calculated using the standard scaled dot product method. Output layer: Inputs the fused features into the convolutional network to generate multimodal fused features; The multimodal fusion process is as follows: By constructing query, key, and value tensors: (3); Calculate attention weights: , The meaning is the dimension of the key tensor, which yields the fused features. : (4); In formula (4), σ is the ReLU activation function.

[0032] S4: Perform joint SLAM 3D mapping based on the fusion features to generate a 3D map, such as... Figure 3 As shown: Dense depth completion: The size of the fused feature map is restored by deconvolution upsampling, generating a dense depth map with 8 channels; Point cloud coordinate mapping: Mapping dense depth maps back to spatial point clouds, and the resulting fused feature map is restored to the spatial point cloud representation through spatial coordinate transformation; Finally, SLAM 3D mapping was combined with Open3D to generate a 3D map.

[0033] Example 2 This embodiment provides a system for constructing a high-precision 3D map based on multimodal feature fusion UAV inspection, the system comprising: a UAV, a hardware module, and a software module; The hardware module includes a LiDAR sensor, a vision module, and a computing unit; the LiDAR sensor is used to acquire radar point cloud data; the vision module is used to acquire image data; and the computing unit is used to perform feature extraction, feature fusion, and map construction on the data generated by the two sensors. The software modules include a data synchronization module, a dynamic weight allocation module, and a 3D map construction module: The data synchronization module realizes sensor clock synchronization and data distribution through ROS (Robot Operating System); The dynamic weight allocation module calculates radar weights and camera weights in real time and outputs them to the computing unit; The 3D map construction module realizes point cloud processing based on PCL (Point Cloud Library) and then generates a 3D map through Open3D.

[0034] Example 3 This embodiment also provides a computer-readable storage medium storing executable instructions, which, when executed, cause the machine to perform the above-described method for intelligent UAV inspection and high-precision 3D map construction based on multimodal feature fusion.

[0035] Specifically, a system or apparatus equipped with a readable storage medium may be provided, on which software program code implementing the functions of any of the embodiments described above is stored, and the computer or processor of the system or apparatus can read and execute the instructions stored in the readable storage medium.

[0036] In this case, the program code itself, which can be read from the readable medium, can perform the functions of any of the above embodiments, and therefore the computer-readable code and the readable storage medium storing the computer-readable code constitute a part of this specification.

[0037] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer or the cloud via a communication network.

[0038] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0039] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solutions of the present invention, and are not intended to limit the specific implementation of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention should be included within the protection scope of the claims of the present invention.

Claims

1. A method for constructing high-precision 3D maps for UAV inspection based on multimodal feature fusion, characterized in that: The method includes: S1. Acquire and synchronize point cloud data and image data, then convert the LiDAR point cloud data to the camera coordinate system, and then perform noise reduction and distortion correction. S2. Dynamically allocate the weights of radar and camera sensors based on ambient light intensity and point cloud density; S3. Extract radar point cloud features and camera visual features, then assign them to the corresponding features according to the radar and camera sensor weights, and finally perform multimodal fusion to obtain fused features. S4. Perform joint SLAM 3D mapping based on the fusion features to generate a 3D map.

2. The high-precision 3D map construction method for UAV inspection based on multimodal feature fusion according to claim 1, characterized in that, S1 specifically includes: S11. The lidar and camera sensors mounted on the UAV acquire point cloud data and image data, respectively. S12. Synchronize the data acquisition timestamps of the LiDAR and camera by clock synchronization. The LiDAR sampling frequency is 10Hz and the camera frequency is 30Hz. Cubic spline interpolation is used to align the spatiotemporal information. S13. Convert the LiDAR point cloud data to the camera coordinate system: （1）； In formula (1), q is the radar point cloud transformation matrix in the transformed camera coordinate system, where R and t are calibration parameters, and q is the point cloud coordinate in the radar coordinate system. S14. Perform noise reduction processing on LiDAR point clouds: The DBSCAN algorithm is used to remove outliers in the point cloud. The density of data points is defined by setting two parameters: neighborhood distance ε and minimum number of points minPts in the neighborhood of the core point. By identifying high-density and low-density regions, the points are divided into core points and non-core points, and the points in the low-density regions are treated as noise for noise reduction. S15. Perform ICP registration on consecutive LiDAR point cloud frames, and at the same time, use bilinear interpolation of the image to correct distortion of the image data.

3. The high-precision 3D map construction method for UAV inspection based on multimodal feature fusion according to claim 2, characterized in that, The specific steps for dynamically allocating radar and camera sensor weights based on ambient light intensity and point cloud density are as follows: （2）； In formula (2), For radar weights, Where I is the camera weight, I is the ambient light intensity, and D is the point cloud density; Secondly, it also includes: When the light intensity I is less than the set light intensity threshold, the visual weight is forcibly set to 0, and only the LiDAR point cloud data is used. When the point cloud density D is less than the set point cloud density threshold, the weight of the LiDAR point cloud data is forcibly set to 0, and only image data is used.

4. The high-precision 3D map construction method for UAV inspection based on multimodal feature fusion according to claim 1 or 3, characterized in that, The extraction of radar point cloud features and camera visual features includes: A multimodal extraction module is constructed, including a radar point cloud data feature extraction network and a camera image data feature extraction network; The radar point cloud data feature extraction network extracts the x, y, z, and i features of each spatial point in the radar data through a first feature extraction module and a second feature extraction module. Multiple extraction modules are used to enhance the extracted features. x, y, and z represent the coordinates of each point in the point cloud in three-dimensional space, and i represents the signal intensity value returned by the radar for each point. The extracted features are then upsampled to the point cloud features. eigenvector space; A camera image data feature extraction network extracts color and spatial features from camera image data. This network comprises three convolutional layers, three batch normalization layers, and three activation function layers, extracting visual features. In the eigenvector space; The camera image data feature extraction network is stacked as follows: three convolutional layers, normalization layers, and activation function layers cascaded one after the other.

5. The high-precision 3D map construction method for UAV inspection based on multimodal feature fusion according to claim 4, characterized in that, Based on the weights assigned to the corresponding features from the radar and camera sensors, multimodal fusion is finally performed to obtain the fused features, specifically: Construct a multimodal fusion module, including: Input layer: Radar point cloud features and camera visual features , where n and m are the feature dimensions of the point cloud and the image, respectively; Attention layer: Enhances cross-modal feature association through self-attention mechanism, outputting fused features. Radar point cloud features are used as the query, and camera visual features are used as the key and value; attention weights are calculated using the standard scaled dot product method. （3）； Calculate attention weights: , The meaning is the dimension of the key tensor, which yields the fused features. : （4）； In formula (4), σ is the ReLU activation function; Output layer: Input the fused features into the convolutional network to generate the multimodal fused feature extraction result, i.e., the multimodal fused feature map.

6. The high-precision 3D map construction method for UAV inspection based on multimodal feature fusion according to claim 5, characterized in that, The joint SLAM 3D mapping based on fusion features specifically involves: Dense depth completion: The feature map size is restored by deconvolution upsampling, generating a dense depth map with 8 channels; Point cloud coordinate mapping: Mapping the dense depth map back to the spatial point cloud, and then restoring the final fused feature map to the spatial point cloud representation through spatial coordinate transformation; Combine SLAM 3D mapping and generate 3D maps using Open3D.

7. The high-precision 3D map construction method for UAV inspection based on multimodal feature fusion according to claim 3, characterized in that, The point cloud density threshold range of D is also dynamically adjusted according to actual needs.

8. A system for constructing a high-precision 3D map based on multimodal feature fusion for UAV inspection, characterized in that, The system includes: a drone, hardware modules, and software modules. The hardware modules include: a LiDAR sensor, a vision module (camera), and a computing unit; LiDAR sensor: used to acquire radar point cloud data; Vision module: used to acquire image data; Computational unit: used for feature extraction, feature fusion, and map construction of data generated by the two sensors; The software modules include: a data synchronization module, a dynamic weight allocation module, and a 3D map construction module. Data synchronization module: Enables sensor clock synchronization and data distribution via ROS; Dynamic weight allocation module: calculates radar weights and camera weights in real time and outputs them to the calculation unit; 3D map building module: It performs point cloud processing based on PCL and then generates a 3D map through Open3D.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images