Target detection method and device based on image and laser radar data feature fusion

Abstract

The invention relates to a target detection method and device based on image and laser radar data feature fusion, and the method comprises the steps: collecting and preprocessing RGB images and laser radar point cloud data, inputting the preprocessed data into a pre-trained FP-RCNN model, and obtaining a target detection result; wherein the process of obtaining the target detection result by the FP-RCNN model comprises the steps of extracting a multi-layer feature map of an RGB image, and obtaining a weighted feature map through FFPA layer processing; obtaining a segmentation mask by adopting a segmentation network; identifying a 2D bounding box, and correcting the 2D projection range by using the segmentation mask; mapping the corrected 2D projection range and the segmentation mask to a 3D space to generate a 3D candidate area; extracting a point cloud feature from the laser radar point cloud data, and fusing the point cloud feature with the segmentation mask to obtain a multi-modal feature; and generating a target detection result based on the multi-modal features. Compared with the prior art, the accuracy of model prediction is improved.

Description

Technical Field

[0001] This invention relates to the fields of autonomous driving technology and deep learning technology, and in particular to a target detection method and device based on the fusion of image and lidar data features. Background Technology

[0002] The development of autonomous driving technology, especially in the perception systems of autonomous vehicles, has made 3D object detection a key technology. With the increasing prevalence of autonomous vehicles, accurate environmental perception is crucial for safe and efficient autonomous driving systems. In autonomous driving, single-modal sensors often struggle to meet the demands for high precision and robustness; therefore, multimodal sensor fusion has become a research hotspot. Fusion of data from LiDAR and cameras can fully leverage their respective advantages, improving the accuracy and robustness of object detection. The precise depth information provided by LiDAR, combined with the high-resolution image information from cameras, helps in a better understanding of objects and the environment within the scene.

[0003] In recent years, scholars have conducted extensive research on multimodal fusion, primarily focusing on spatial transformation-based methods and deep learning-based methods. While the former is simple and intuitive, it heavily relies on global alignment and cannot fully capture details of local regions, especially when there are significant differences between data from different modalities, often leading to the loss of important information. Deep learning-based methods, particularly convolutional neural networks (CNNs), have made significant progress in the multimodal field, solving many problems that spatial transformation methods cannot overcome. However, deep learning-based methods also face some challenges, particularly in local feature matching and sensor scale inconsistency. The difficulty in local feature matching stems from the fact that while the design of local receptive fields in CNNs helps capture local features when processing image data, when processing cross-modal data, the local receptive fields of CNNs often fail to cover the global relationships and semantic information across modalities, resulting in the loss of local information. The problem of sensor scale inconsistency arises because data from different modalities not only originate from different sensors but also exhibit significant differences in spatial distribution and scale.

[0004] Chinese patent application CN118172535A discloses a 3D target detection method based on a self-attention mechanism that fuses images and point clouds. It combines image and point cloud data and integrates the self-attention mechanism into the PointNet network, which directly processes point cloud data, achieving efficient 3D target detection in autonomous driving scenarios through deep learning. However, this patent has weak adaptability to targets of different scales, only splicing global features and semantic vectors, resulting in a coarse fusion and feature redundancy. Furthermore, this patent only employs a single multi-head self-attention mechanism, which may still suffer from local information loss and insufficient focus on key regions. Generating frustum point clouds by mapping the corner points of 2D detection boxes may lead to candidate regions containing too many background point clouds.

[0005] To address the aforementioned issues of weak adaptability to targets of different scales, loss of local information, and feature mismatch, it is necessary to investigate an efficient optimization method to improve the accuracy of model detection. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a target detection method and device based on the fusion of image and lidar data features.

[0007] The objective of this invention can be achieved through the following technical solutions: A target detection method based on the fusion of image and lidar data features, the method comprising: RGB images and LiDAR point cloud data are acquired and preprocessed. The preprocessed data is then input into a pre-trained FP-RCNN model to obtain target detection results. The process of obtaining target detection results using the FP-RCNN model includes: extracting multi-layer feature maps from RGB images using a convolutional neural network; processing the multi-layer feature maps using an FFPA layer to obtain weighted feature maps; processing the weighted feature maps using a segmentation network to obtain a segmentation mask; identifying 2D bounding boxes using the Faster RCNN framework based on the weighted feature maps, and correcting the 2D projection range of the 2D bounding boxes using the segmentation mask; mapping the corrected 2D projection range and the segmentation mask to 3D space using a 3D projection algorithm to generate 3D candidate regions; cropping a subset of point clouds corresponding to the 3D candidate regions from the LiDAR point cloud data, extracting point cloud features, and fusing the point cloud features with the segmentation mask to obtain multimodal features; and predicting the target's category, 3D position, size, and orientation using a regression network based on the multimodal features to generate 3D bounding boxes, which are then output as target detection results.

[0008] Furthermore, the process of obtaining a weighted feature map by processing the multi-layer feature map through the FFPA layer includes: Feature fusion is performed using a bidirectional feature fusion method to generate a feature map pyramid; channel attention and spatial attention weighting are applied to the feature map pyramid, and adaptive weighted fusion is performed to obtain a weighted feature map.

[0009] Furthermore, the process of generating a feature map pyramid through feature fusion using the aforementioned bidirectional feature fusion method includes: Based on the convolutional neural network, the preprocessed RGB image is processed layer by layer to transfer detailed information from low scale to high scale and obtain feature maps with feature scale from low to high. Starting with the highest-scale feature map, the strong semantic information of the high-scale features is transferred to the low-scale feature map through upsampling and convolution operations, thus obtaining feature maps with feature scales from low to high. Align the feature maps with the same feature scale from low to high with each other, and add them element by element to obtain a fused feature map. Perform a convolution operation to eliminate the checkerboard effect caused by upsampling and obtain a feature map pyramid.

[0010] Furthermore, the process of weighting channel attention and spatial attention includes: Channel attention weighting processing: Global average pooling is performed on the feature maps in the feature map pyramid to obtain statistical information for each channel; based on the statistical information for each channel, channel attention weights are generated through a fully connected layer or convolutional layer and a sigmoid function; the channel attention weights are applied to the preprocessed original RGB image to weight the features of each channel of the original RGB image and obtain a channel-weighted feature map; Spatial attention weighting processing: The channel weighted feature map is generated by average pooling along the channel dimension and compressed to obtain a two-dimensional spatial feature map; based on the spatial feature map, spatial attention weights are generated by a small neural network with fully connected / convolutional layers and a sigmoid function; the weights are applied to each channel weighted feature map to obtain a spatial weighted feature map.

[0011] Furthermore, the process of identifying 2D bounding boxes using the Faster R-CNN framework and correcting the 2D projection range of the 2D bounding boxes using the segmentation mask includes: Based on the Faster RCNN framework, 2D bounding boxes of detected targets are obtained from the weighted feature map through region proposal generation, bounding box classification and regression, and filtering. Each pixel in the segmentation mask is mapped one-to-one with each pixel in the 2D bounding box, and the corresponding pixel is marked as the target pixel; The pixels in the 2D bounding box are filtered, and only the pixels corresponding to the target pixel are retained to generate the corrected 2D projection range.

[0012] Furthermore, the FP-RCNN model is trained based on a multimodal fusion loss function, which includes classification loss, 3D bounding box regression loss, and image segmentation loss.

[0013] Furthermore, the expression for the classification loss is: in, For classifying losses, It is the number of categories. It is a real category label, if the first Each sample belongs to category ,but ,otherwise , It is the probability predicted by the model, representing the probability of the first... Each sample belongs to category The probability of; The expression for the 3D bounding box regression loss is: in, It is the 3D bounding box regression loss. The coordinates of the center point of the 3D bounding box. This refers to the size of the 3D bounding box. and These are the actual value and the predicted value, respectively. dimensional error, It is smooth function; The expression for the image segmentation loss is: in, It is the image segmentation loss; The expression for the multimodal fusion loss function is: in, and It is the weighted coefficient of regression loss and segmentation loss.

[0014] Furthermore, the point cloud features are extracted using a PointNet network, which includes an MLP layer, a T-Net layer, and a NetVLAD pooling layer. The process of extracting the point cloud features and fusing them with the segmentation mask to obtain multimodal features includes: Spatial alignment of the point cloud subsets corresponding to the 3D candidate regions is performed using T-Net feature transformation; By using MLP layer-by-layer mapping, the aligned point cloud subset is mapped from 3D to higher dimensions in turn, capturing the local structural features of the point cloud. The local structural features of the point cloud are globally integrated to generate a fixed-dimensional global feature vector of the point cloud, which has the same dimension as the 3D candidate region. The global feature vector of the point cloud is fused with the semantic features transformed from the segmentation mask to obtain multimodal features.

[0015] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the target detection method based on the fusion of image and lidar data features as described above.

[0016] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the target detection method based on the fusion of image and lidar data features as described above.

[0017] Compared with the prior art, the beneficial effects of the present invention include: 1. In this invention, the FP-RCNN model is used to achieve accurate target detection. The FP-RCNN model processes RGB images by adding an FFPA layer to the original three-layer neural network model to improve the accuracy of RGB image detection. The model can distinguish image information at multiple feature scales and simultaneously fuse key image features with their corresponding point cloud features. The image segmentation network classifies each pixel and can accurately mark the outline or region of the object. This model can accurately identify objects in complex backgrounds and effectively estimate their position information. It shows high accuracy in classifying objects at intersections of highways and densely populated streets.

[0018] 2. This invention constructs a feature pyramid and adaptively weights multi-scale features, which solves the problem of poor performance of traditional CNNs when processing targets of different sizes. It is more adaptable to the detection of small and large targets. The fusion process combines the enhanced features of RGB images with the features of LiDAR point clouds, rather than simple projection or stitching, thus preserving the core value of the two modalities of data.

[0019] 3. To address the limitations of single attention mechanisms, this invention employs a dual hybrid mechanism combining channel attention and spatial attention. This mechanism not only emphasizes the feature differences between different channels but also highlights key regions in the image space, avoiding information omissions caused by single attention mechanisms. It can adaptively allocate attention weights, ignore irrelevant background information, focus on the target region, improve detection robustness in complex scenes, and achieve better performance.

[0020] 4. This invention combines 2D detection boxes with image segmentation masks to generate 3D candidate regions (Frustum), correcting the region bias caused by relying solely on 2D bounding boxes, making the target contour mapping in 3D space more accurate. It also improves the PointNet network by replacing max pooling with NetVLAD, optimizing the aggregation of local and global features of point clouds, and improving the effectiveness of feature extraction.

[0021] 5. This invention adds a segmentation loss to the classification loss and 3D bounding box regression loss. By calculating the classification error of each pixel, the target contour and semantic information are accurately labeled, further reducing the detection error. The multi-task loss is weighted and balanced to avoid the single task loss from dominating the training, so that the model can be optimized simultaneously in classification, localization and segmentation tasks. Attached Figure Description

[0022] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a flowchart illustrating the algorithm implementation of the FP-RCNN model of this invention. Figure 3 This is a schematic diagram of the FFPA layer of the present invention; Figure 4 This is a schematic diagram of the Pointnet network framework of the present invention; Figure 5 This is a comparison chart of Loss and RMSE before and after optimization of the Faster RCNN model in an embodiment of the present invention; Figure 6 This is a comparison image of the 2D detection bounding boxes of the RGB image before and after optimization in an embodiment of the present invention; Figure 7 This is an example diagram showing the 3D detection results of pedestrians and vehicles in an urban environment according to the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0024] Example 1 This embodiment discloses a target detection method based on the fusion of image and lidar data features. The method is as follows: Figure 1 As shown, steps S1-S3 are included, as detailed below: Step S1: Acquire RGB images and LiDAR point cloud data; Step S2: Preprocess the collected data; Step S3: Input the preprocessed data into the pre-trained FP-RCNN model to obtain the object detection results.

[0025] In step S3, the pre-trained FP-RCNN model detects the target (a vehicle in this embodiment) by using a 3D bounding box in the RGB image. The algorithm implementation flow of the FP-RCNN model is as follows: Figure 2 As shown, the process comprises three stages: 3D region proposal, 3D instance segmentation, and 3D bounding box generation. Initially, RGB images are processed through an FFPA (Feature Fusion Pyramid Attention) layer, which synergistically combines a pyramid feature network and a hybrid attention mechanism to extract enhanced feature representations. These processed images are then optimized using the Faster R-CNN framework to facilitate the derivation of 2D region proposals and subsequent frustum generation. This dual output is then integrated with point cloud data improved through foreground masking techniques to generate comprehensive 3D region proposals. Feature extraction is performed using PointNet, which aggregates point-wise features into a global representation through MLP and pooling operations. Based on these extracted features, 3D objects are detected. A regression network predicts the class, location, and orientation of each object, outputting a 3D bounding box containing its size, location, and orientation. Finally, post-processing techniques such as NMS are applied to eliminate duplicate detections and further improve accuracy.

[0026] The process of obtaining object detection results using the FP-RCNN model is as follows: Step S201: Use a convolutional neural network to extract multi-layer feature maps of the RGB image.

[0027] Step S202: Process the multi-layer feature map through the FFPA layer to obtain a weighted feature map.

[0028] FFPA layer details are as follows Figure 3 As shown, a multi-scale feature discrimination network and a dual self-attention model are combined to improve the expressiveness and prediction efficiency of multi-scale features.

[0029] Specifically, in step S202, the process of processing multi-layer feature maps through the FFPA layer to obtain a weighted feature map includes: Feature fusion is performed using a bidirectional feature fusion method to generate a feature map pyramid; channel attention and spatial attention weighting are applied to the feature map pyramid, and adaptive weighted fusion is performed to obtain a weighted feature map.

[0030] Basic convolutional neural networks perform poorly when dealing with objects of different scales, while FFPA layers can significantly improve the detection and understanding of objects of different sizes through adaptive weighted multi-scale features. FFPA layers aim to enhance the expressiveness of multi-scale features and improve prediction efficiency.

[0031] The process of generating a feature map pyramid by performing feature fusion using the bidirectional feature fusion method includes: Based on the convolutional neural network, the preprocessed RGB image is processed layer by layer to transfer detailed information from low scale to high scale and obtain feature maps with feature scale from low to high. Starting with the highest-scale feature map, the strong semantic information of the high-scale features is transferred to the low-scale feature map through upsampling and convolution operations, thus obtaining feature maps with feature scales from low to high. Align the feature maps with the same feature scale from low to high with each other, and add them element by element to obtain a fused feature map. Perform a convolution operation to eliminate the checkerboard effect caused by upsampling and obtain a feature map pyramid.

[0032] The formula for the feature pyramid is expressed as: Among them, Indicates the first i Feature map of the layer.

[0033] The process of weighted processing of channel attention and spatial attention includes: Channel attention weighting: Global average pooling is performed on the feature maps in the feature map pyramid to obtain statistical information for each channel; based on the statistical information for each channel, channel attention weights are generated through fully connected layers or convolutional layers and the Sigmoid function; the channel attention weights are applied to the preprocessed original RGB image to weight the features of each channel of the original RGB image and obtain a channel-weighted feature map; Spatial attention weighted processing: Average pooling is performed along the channel dimension to generate a two-dimensional spatial feature map; based on the spatial feature map, spatial attention weights are generated by a small neural network with fully connected / convolutional layers and a sigmoid function; the weights are applied to each channel weighted feature map to obtain a spatial weighted feature map.

[0034] The aforementioned combined attention mechanism allows the model to focus on important regions while ignoring irrelevant information. Global pooling, by performing global average pooling on the input feature map, yields statistical information for each channel.

[0035] Assume the input feature map is Its number of channels is The height and width are respectively and The expression for global average pooling of feature maps in the feature map pyramid is: in, Indicates the first The passage is in the location eigenvalues.

[0036] The expression for calculating channel attention weights is: .

[0037] in, This represents the activation function, typically the sigmoid function. and These are the weights and bias parameters.

[0038] The weighted expression for the features of each channel is: .

[0039] The expression for generating a two-dimensional spatial feature map by average pooling along the channel dimension of the channel-weighted feature map is as follows: .

[0040] Spatial attention weights are also calculated using convolutional or fully connected layers to obtain a weight map with the same spatial size as the input feature map. The expression is: .

[0041] Applying the weights to the weighted feature maps of each channel, the expression for obtaining the spatially weighted feature map is: .

[0042] Feature map weighted by attention mechanism These will be passed to subsequent network layers for processing. Ultimately, the feature map output by the FFPA layer is a weighted fusion of feature maps from all scales. Assuming there are feature maps at multiple scales... F 1 ,F 2 ,...F n The fusion process can be represented as: in, It is the first The attention weights of each feature map are applied to the final output. It is a weighted sum of all feature maps.

[0043] Step S203: The weighted feature map is processed using a segmentation network to obtain the segmentation mask.

[0044] The feature map obtained after the feature pyramid and attention mechanism is denoted as follows: The mask output by the image segmentation network is denoted as... The two dimensions are consistent, that is M∈R H×W ,in and M represents the height and width of the feature map, respectively. i,j For pixels ( i,j The corresponding category label. Image segmentation output formula: in, This indicates the selected image segmentation network for the feature map. Process and output the segmentation mask. .

[0045] Step S204: Based on the weighted feature map, identify 2D bounding boxes using the Faster RCNN framework, and use segmentation masks to correct the 2D projection range of the 2D bounding boxes.

[0046] Specifically, in step S204, the process of identifying 2D bounding boxes using the Faster R-CNN framework and correcting the 2D projection range of the 2D bounding boxes using the segmentation mask includes: Based on the Faster RCNN framework, 2D bounding boxes of detected targets are obtained from the weighted feature map through region proposal generation, bounding box classification and regression, and filtering. Each pixel in the segmentation mask is mapped one-to-one with each pixel in the 2D bounding box, and the corresponding pixel is marked as the target pixel; The pixels in the 2D bounding box are filtered, and only the pixels corresponding to the target pixel are retained to generate the corrected 2D projection range.

[0047] 2D bounding box B =( x 1 ,y 1 ,x 2 ,y 2 ),lie in I superior,( x 1 ,y 1 )and( x 2 ,y2 ) are the coordinates of the upper left corner and the lower right corner respectively. The image segmentation mask will provide more detailed object region information. Assume that the 2D projection range is , then the range can be corrected according to the mask : where is the pixel in the segmentation mask with coordinates (x, y), is the 2D bounding box, is the background pixel.

[0048] The specific process of obtaining the 2D bounding box of the detection target in the weighted feature map is as follows: Region proposal generation: For driverless targets, 3 scales and 3 aspect ratios are preset, a total of 9 anchor points, covering the target morphology; Anchor matching calculates the IoU (Intersection over Union) between the anchor point and the true target bounding box (GroundTruth): IoU≥0.7 is marked as a positive sample, corresponding to the true target; IoU≤0.3 is marked as a negative sample, corresponding to the background; 0.3 < IoU < 0.7 is marked as an ignored sample to avoid blurred interference; Bounding box classification and regression: Classification branch: Output the target / background probability p through a 1×1 convolution, retain the anchor points with p≥0.5, and determine them as target candidates; Regression branch: Output the coordinate offset to correct the initial coordinates of the anchor points; Filtering: Sort in descending order according to the target probability, delete the overlapping boxes with IoU>0.7, and obtain K initial candidate boxes; To correct the coordinate deviation of the initial boxes, generate the final 2D bounding box through RoI mapping, fixed pooling, and quadratic optimization.

[0049] Step S205, map the corrected 2D projection range and the segmentation mask to the 3D space through a 3D projection algorithm to generate a 3D candidate region.

[0050] Step S206, crop the point cloud subset corresponding to the 3D candidate region from the lidar point cloud data, extract the point cloud features, and fuse the point cloud features with the segmentation mask to obtain multi-modal features.

[0051] In this embodiment, Frustum PointNet is used to generate a 3D candidate region (Frustum) to obtain multi-modal features. The basic framework of PointNet is as Figure 4 shown. The PointNet network mainly includes an input layer (Input), an MLP layer, a T-Net layer, and a NetVLAD pooling layer.

[0052] Before the process of generating the 3D candidate region, combine the 2D detection box and the image segmentation mask through the above step S204 to obtain the corrected 2D projection range This allows for the generation of more accurate 3D candidate regions. This is because it can be assumed that the spatial projection range of the 3D candidate region is the same as the 2D projection range. .

[0053] The generation of 3D candidate regions relies not only on 2D bounding boxes, but also on segmentation masks. This allows the 3D candidate region to be accurately mapped to the object's outline in 3D space. Assume the coordinates of the 3D candidate region projected into 3D space are... The calculation expression is: .

[0054] For point cloud data in 3D candidate regions, PointNet extracts local and global features. Assuming... For point cloud data, For a subset of point clouds in a 3D candidate region, the features extracted by PointNet are: The calculation expression is: .

[0055] Image segmentation output Point cloud features The components will be fused to form a multimodal output: .

[0056] Point cloud features are also extracted using the PointNet network. The process of extracting point cloud features in step S206 and fusing them with the segmentation mask to obtain multimodal features specifically includes: Spatial alignment of the point cloud subsets corresponding to the 3D candidate regions is performed using T-Net feature transformation; By using MLP layer-by-layer mapping, the aligned point cloud subset is mapped from 3D to higher dimensions in turn, capturing the local structural features of the point cloud. The local structural features of the point cloud are globally integrated to generate a fixed-dimensional global feature vector of the point cloud, which has the same dimension as the 3D candidate region. Multimodal features are obtained by fusing the global feature vector of the point cloud with the semantic features transformed from the segmentation mask.

[0057] Step S207: Based on multimodal features, a regression network is used to predict the target's category, 3D location, size, and orientation, generating a 3D bounding box as the target detection result output.

[0058] The FP-RCNN model is trained based on a multimodal fusion loss function, which includes classification loss, 3D bounding box regression loss, and image segmentation loss.

[0059] Classification loss is used to optimize the classification task of the target, ensuring that each 3D target is correctly classified. This part of the loss function typically uses cross-entropy loss. For the... For each sample, the classification loss is... It can be represented as: in, For classifying losses, It is the number of categories (e.g., vehicles, pedestrians, etc.). It is a real category label, if the first Each sample belongs to category ,but ,otherwise , It is the probability predicted by the model, representing the probability of the first... Each sample belongs to category The probability of.

[0060] 3D bounding box regression loss is used to optimize the position, size, and orientation of an object in 3D space, and is suitable for smoothing. The loss is used to calculate the difference between the predicted bounding box and the true bounding box. For the first... 3D bounding boxes of each sample, smoothed loss It can be represented as: in, It is the 3D bounding box regression loss. The coordinates of the center point of the 3D bounding box. This refers to the size of the 3D bounding box. and These are the actual value and the predicted value, respectively. Errors in dimensions (including position, size, orientation, etc.). It is smooth The function is defined as: The goal of image segmentation networks is to classify each pixel to accurately label the contours or regions of objects. Segmentation loss typically uses cross-entropy loss to calculate the classification error for each pixel. For image segmentation tasks, the... Loss of one pixel It can be represented as: in, It is the image segmentation loss.

[0061] The total loss function is a weighted sum of the individual loss functions. Taking into account classification loss, bounding box regression loss, and segmentation loss, the final total loss is... This can be represented as: in, and It is a weighted coefficient of regression loss and segmentation loss, used to balance the weights of different tasks.

[0062] This invention optimizes the prediction accuracy of the 2D model in RGB image processing through the FFPA strategy in the feature recognition stage, providing more accurate data input for 3D object detection using the FP-RCNN 3D model. Regarding the attention mechanism, this invention differs from the single attention mechanism in the comparison files by superimposing a dual self-attention mechanism to improve the model's generalization ability and accuracy. Simultaneously, in the point cloud data processing stage, max pooling in the PointNet framework is replaced with NetVLAD, further improving the model's prediction accuracy.

[0063] Example 2 This embodiment is based on a practical application example of a target detection method based on the fusion of image and lidar data features disclosed in Embodiment 1 above. To demonstrate the improved effect of this method, it is compared with the prior art.

[0064] This method achieves the detection and understanding of objects of different sizes by adding an FFPA layer, which enhances the expressiveness of multi-scale features and improves prediction efficiency and accuracy. In this embodiment, a model without an FFPA layer and a model using the RMSPROP algorithm are selected as comparison models.

[0065] Figure 5 The figure shows the curves of loss and root mean square error (RMSE) changing with the number of iterations when using different strategies during the training of a neural network. In the figure, the curve without FFPA represents the model without an FFPA layer, the curve with FFPA represents the model with an FFPA layer, and the curve with promotion represents the model using the RMSPROP algorithm. During training, the maximum learning rate, weight decay parameter, and momentum factor were set to 0.001, 0.01, and 0.9, respectively. The network was trained for 20 epochs, with 177 iterations per epoch. Experiments show that the FFPA layer effectively accelerates the loss reduction and error reduction during training, especially in the initial stages. In contrast, the model without an FFPA layer, although eventually converging, converges more slowly and exhibits greater fluctuations during training. Therefore, FFPA can significantly improve efficiency and reduce error.

[0066] Figure 6To compare the use of models without FFPA layers and models with FFPA layers for detecting the same target, among which... Figure 6 (a) Scores for detection boxes without FFPA. Figure 6 (b) shows the score of the detection box with FFPA added. It can be seen that after adding the FFPA layer, the score of the detection box is improved, the correct label is displayed, and it is more stable and accurate.

[0067] The model with the FFPA layer of this method is applied to urban environments. During the detection process, as described in Example 1, the enhanced features of the RGB image are fused with the LiDAR point cloud features to obtain, as shown in Example 1. Figure 7 The images show the detection results of pedestrians and vehicles in urban environments, revealing different city streets and commercial areas with complex environments. By fusing point cloud data, the system can more accurately perceive surrounding buildings, streets, and other obstacles.

[0068] Example 3 Based on Embodiment 1, this embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the target detection method based on the fusion of image and lidar data features as described above.

[0069] At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the target detection method based on the fusion of image and LiDAR data features described above. Of course, in addition to software implementation, this invention does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0070] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0071] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0072] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A target detection method based on the fusion of image and lidar data features, characterized in that, The method includes: RGB images and LiDAR point cloud data are acquired and preprocessed. The preprocessed data is then input into a pre-trained FP-RCNN model to obtain target detection results. The process of obtaining target detection results using the FP-RCNN model includes: extracting multi-layer feature maps from RGB images using a convolutional neural network; processing the multi-layer feature maps using an FFPA layer to obtain weighted feature maps; processing the weighted feature maps using a segmentation network to obtain a segmentation mask; identifying 2D bounding boxes using the Faster RCNN framework based on the weighted feature maps, and correcting the 2D projection range of the 2D bounding boxes using the segmentation mask; mapping the corrected 2D projection range and the segmentation mask to 3D space using a 3D projection algorithm to generate 3D candidate regions; cropping a subset of point clouds corresponding to the 3D candidate regions from the LiDAR point cloud data, extracting point cloud features, and fusing the point cloud features with the segmentation mask to obtain multimodal features; and predicting the target's category, 3D position, size, and orientation using a regression network based on the multimodal features to generate 3D bounding boxes, which are then output as target detection results.

2. The target detection method based on image and lidar data feature fusion according to claim 1, characterized in that, The process of processing the multi-layer feature map through the FFPA layer to obtain the weighted feature map includes: Feature fusion is performed using a bidirectional feature fusion method to generate a feature map pyramid; channel attention and spatial attention weighting are applied to the feature map pyramid, and adaptive weighted fusion is performed to obtain a weighted feature map.

3. The target detection method based on image and lidar data feature fusion according to claim 2, characterized in that, The process of generating a feature map pyramid by performing feature fusion using the bidirectional feature fusion method includes: Based on the convolutional neural network, the preprocessed RGB image is processed layer by layer to transfer detailed information from low scale to high scale and obtain feature maps with feature scale from low to high. Starting with the highest-scale feature map, the strong semantic information of the high-scale features is transferred to the low-scale feature map through upsampling and convolution operations, thus obtaining feature maps with feature scales from low to high. Align the feature maps with the same feature scale from low to high with each other, and add them element by element to obtain a fused feature map. Perform a convolution operation to eliminate the checkerboard effect caused by upsampling and obtain a feature map pyramid.

4. The target detection method based on image and lidar data feature fusion according to claim 2, characterized in that, The process of weighting channel attention and spatial attention includes: Channel attention weighting processing: Global average pooling is performed on the feature maps in the feature map pyramid to obtain statistical information for each channel; based on the statistical information for each channel, channel attention weights are generated through a fully connected layer or convolutional layer and a sigmoid function; the channel attention weights are applied to the preprocessed original RGB image to weight the features of each channel of the original RGB image and obtain a channel-weighted feature map; Spatial attention weighting processing: The channel weighted feature map is generated by average pooling along the channel dimension and compressed to obtain a two-dimensional spatial feature map; based on the spatial feature map, spatial attention weights are generated by a small neural network with fully connected / convolutional layers and a sigmoid function; the weights are applied to each channel weighted feature map to obtain a spatial weighted feature map.

5. The target detection method based on image and lidar data feature fusion according to claim 1, characterized in that, The process of identifying 2D bounding boxes using the Faster R-CNN framework and correcting the 2D projection range of the 2D bounding boxes using the segmentation mask includes: Based on the Faster RCNN framework, 2D bounding boxes of detected targets are obtained from the weighted feature map through region proposal generation, bounding box classification and regression, and filtering. Each pixel in the segmentation mask is mapped one-to-one with each pixel in the 2D bounding box, and the corresponding pixel is marked as the target pixel; The pixels in the 2D bounding box are filtered, and only the pixels corresponding to the target pixel are retained to generate the corrected 2D projection range.

6. The target detection method based on image and lidar data feature fusion according to claim 1, characterized in that, The FP-RCNN model is trained based on a multimodal fusion loss function, which includes classification loss, 3D bounding box regression loss, and image segmentation loss.

7. The target detection method based on image and lidar data feature fusion according to claim 6, characterized in that, The expression for the classification loss is: in, For classifying losses, It is the number of categories. It is a real category label, if the first Each sample belongs to category ,but ,otherwise , It is the probability predicted by the model, representing the probability of the first... Each sample belongs to category The probability of; The expression for the 3D bounding box regression loss is: in, It is the 3D bounding box regression loss. The coordinates of the center point of the 3D bounding box. This refers to the size of the 3D bounding box. and These are the actual value and the predicted value, respectively. dimensional error, It is smooth function; The expression for the image segmentation loss is: in, It is the image segmentation loss; The expression for the multimodal fusion loss function is: in, and It is the weighted coefficient of regression loss and segmentation loss.

8. The target detection method based on image and lidar data feature fusion according to claim 1, characterized in that, The point cloud features are extracted using a PointNet network, which includes an MLP layer, a T-Net layer, and a NetVLAD pooling layer. The process of extracting the point cloud features and fusing them with the segmentation mask to obtain multimodal features includes: Spatial alignment of the point cloud subsets corresponding to the 3D candidate regions is performed using T-Net feature transformation; By using MLP layer-by-layer mapping, the aligned point cloud subset is mapped from 3D to higher dimensions in turn, capturing the local structural features of the point cloud. The local structural features of the point cloud are globally integrated to generate a fixed-dimensional global feature vector of the point cloud, which has the same dimension as the 3D candidate region. The global feature vector of the point cloud is fused with the semantic features transformed from the segmentation mask to obtain multimodal features.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the target detection method based on the fusion of image and lidar data features as described in any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the target detection method based on the fusion of image and lidar data features as described in any one of claims 1-8.

Images