Three-dimensional target detection method and system based on multimodal fusion

Abstract

The present invention discloses a three-dimensional target detection method and system based on multimodal fusion, wherein the method includes the following steps: S1, obtaining image data collected by a camera; S2, obtaining original point cloud data collected by a laser radar and performing noise reduction preprocessing on the point cloud; S3, inputting the image and point cloud data into the trained corresponding networks respectively to obtain corresponding image features and point cloud features; S4, mapping the image features of the two-dimensional image to a three-dimensional point cloud spatial coordinate system; S5, cascading the point cloud features with the mapped image features, inputting them into an adaptive expressiveness enhancement fusion module for recalibration, and obtaining weighted fusion features; S6, inputting the weighted fusion features into a detection output network to generate a bounding box for the target, thereby completing three-dimensional target detection. The present invention effectively suppresses unimportant feature representations and can improve the accuracy of 3D target detection in subsequent detection output modules.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving environment perception technology, specifically a three-dimensional target detection method and system based on multimodal fusion. Background Art

[0002] With the rapid development of deep learning, autonomous driving technology has received a significant boost. As a crucial component of autonomous driving perception technology, three-dimensional object detection provides the foundation for autonomous driving, providing data support for vehicle path planning and behavioral decision-making. Autonomous driving requires accurate location and detection of obstacles in space to avoid accidents in complex traffic scenarios. Therefore, achieving high-precision and efficient object detection through various approaches is attracting increasing attention from researchers. Multimodal fusion object detection performs well in complex traffic scenarios, bridging the gaps in information and enabling information complementarity between various sensors, making it a hot topic for researchers. However, existing three-dimensional object detection algorithms are significantly affected by lighting variations and object occlusion in autonomous driving scenarios, resulting in missed detections, false detections, and low detection accuracy. Addressing these issues presents significant challenges. Summary of the Invention

[0003] The purpose of the present invention is to provide a three-dimensional target detection method based on multimodal fusion, which is used to solve the technical problems in the prior art that the three-dimensional target detection algorithm is greatly disturbed due to problems such as lighting changes and target occlusion in autonomous driving scenarios, resulting in missed detections, false detections and low detection accuracy.

[0004] The three-dimensional target detection method based on multimodal fusion includes the following steps.

[0005] S1. Obtain image data collected by the camera.

[0006] S2. Obtain the original point cloud data collected by the lidar and perform noise reduction preprocessing on the point cloud.

[0007] S3. Input the image and point cloud data into the trained corresponding networks respectively to obtain the deep semantic feature information of the targets in their respective data, i.e., the corresponding image features and point cloud features.

[0008] S4. Map the image features of the two-dimensional image to a three-dimensional point cloud space coordinate system.

[0009] S5. Cascading the point cloud features with the mapped image features, inputting the cascaded image and point cloud features into an adaptive expressiveness enhancement fusion module, recalibrating the fused features, and multiplying the weights with the original corresponding channels to obtain weighted fused features.

[0010] S6. Input the weighted fusion features into the detection output network to generate a bounding box for the target, thereby completing three-dimensional target detection.

[0011] Preferably, step S3 includes:

[0012] S3.1. Processing the data configured as point cloud: Use the PointNet network as the backbone network for point cloud feature extraction to perform feature extraction on the input point cloud data.

[0013] S3.2. Process the data configured as images: Use the MobileNetv2 deep neural network as the basic convolutional layer to extract features from the input image data.

[0014] Preferably, the step S3.1 specifically includes: constructing the input feature layer of the point cloud according to the direct coordinate convolution method of PointNet, calculating the original data coordinate information to obtain the input feature map, and then using the convolutional neural network to complete the extraction of point cloud features.

[0015] The step S3.2 specifically includes: the model uses the first few layers of parameters of the trained MobileNetv2 as the initial convolution kernel values ​​of the image feature extractor to obtain image features.

[0016] Preferably, the step S4 specifically includes: first, given a target pixel point i on the bird's-eye view of the point cloud, and extracting K (generally ten points) laser radar point clouds j closest to the target pixel point; then we project the laser radar point cloud j onto the front view of the camera to obtain the corresponding adjacent pixel point x j ; Retrieve the image feature point x corresponding to the target pixel i i ; From each adjacent pixel x j To the camera front view and the image feature point x i Do the difference to get the continuous geometric offset x j -x i ; Finally, we will adjacent pixel point x j The corresponding input image feature f j and continuous geometric offset x j -x i Input the BP neural network to generate corresponding features mapped to the three-dimensional point cloud space coordinate system.

[0017] For each target pixel i, the BP neural network calculates all its adjacent pixel points x. j The BP neural network output is summed to output its image feature point x i Image feature h mapped to the three-dimensional point cloud space coordinate system i , the formula is as follows.

[0018]

[0019] Among them, f j is the input image feature corresponding to the lidar point cloud j, x j -x i is the geometric offset from the adjacent pixel point j to the target i projected on the camera front view, and concat(·) is the concatenation of multiple vectors.

[0020] Preferably, in step S5, an adaptive expressiveness enhancement fusion module is constructed with the aid of an attention mechanism. The adaptive expressiveness enhancement fusion module is used to enhance the expressiveness of different features, focusing on important features while suppressing non-important features. Specifically, the following steps are performed: first, given an input image and a cascade feature of a point cloud, the size of the feature layer is H×W×C. After a maximum pooling operation, the candidate size is changed to 1×1; then, the first fully connected layer is used to reduce the number of channels to C / r, where r is a scaling factor used to reduce the number of channels and thus reduce the amount of computation; then, a ReLU activation function is used to keep the dimension of the feature unchanged; the second fully connected layer is used to increase the dimension, and the number of channels is returned to C; a Sigmoid function is used to output the weight of each channel; finally, the features are recalibrated, and the weights are multiplied by the original corresponding channels to obtain the weighted fusion features of the point cloud and image.

[0021] Preferably, step S6 includes: inputting the weighted fusion features into the detection output network, and generating a corresponding bounding box and a corresponding score of the bounding box for the corresponding target through the region proposal network RPN processing; for the generated bounding box, we use the adaptive non-maximum prediction algorithm A-NMS to process it, remove the bounding box with a lower score, and retain the bounding box with the largest score as the final target detection result.

[0022] Preferably, in step S6, the detailed steps of A-NMS include: sorting all bounding boxes according to scores, selecting the bounding box with the highest score, and suppressing all other bounding boxes that have significant overlap with the bounding box with the highest score using a predefined threshold; this process is applied to the remaining bounding boxes in turn, and the intersection over union (IoU) of the bounding box with the highest score and the bounding boxes of other candidate regions is calculated, where the intersection over union (IoU) is the ratio of the intersection and union between the predicted box and the true bounding box; the calculation formula of the A-NMS algorithm is as follows.

[0023]

[0024] Among them, S i is the bounding box score after comparison and calculation, s i is the initial score of the bounding box, M is the bounding box with the highest score, and b i is the current bounding box, IoU(M,bi ) are M and b i The intersection-over-union ratio, N i 、N t Both are thresholds.

[0025] If the current bounding box b i The score IoU(M,b i )<N t , then remove the bounding box b i ; if N i <IoU(M,b i )<N t , then the bounding box b i Will be given IoU(M,b i ) penalty ratio is combined with the original score as a penalty factor; if IoU(M, b i )<N i , then b i If the overlap with M is small, then the bounding box b is not i Remove and keep the original score.

[0026] The present invention also provides a three-dimensional target detection system based on multimodal fusion, including a monocular camera, a laser radar, a camera and a data processing unit. The data processing unit is connected to the monocular camera and the laser radar to obtain data. The monocular camera collects images in front of the viewing angle in real time and sends them to the data processing unit; at the same time, the laser radar scans a point cloud map in the construction environment and sends it to the data processing unit; the data processing unit executes a computer program stored in itself to implement the steps of the three-dimensional target detection method based on multimodal fusion as described above to extract the target in front.

[0027] Preferably, the data processing unit includes.

[0028] The point cloud feature extraction module integrates the trained PointNet network model and is used to process data configured as point clouds to obtain point cloud features.

[0029] The image feature extraction module integrates a trained MobileNetv2 network model and is used to process data configured as images to obtain image features.

[0030] The spatial transformation module is used to map two-dimensional image features to the feature space of three-dimensional point cloud.

[0031] The adaptive expressiveness enhancement fusion module is used to cascade point cloud features with image features, recalibrate the cascaded image and point cloud features, enhance the expressiveness of different features, and then multiply the weights with the original corresponding channels to obtain weighted fusion features.

[0032] The detection output module is used to input the fusion features of the image and point cloud into the region proposal network (RPN), generate a bounding box for the target, use the non-maximum suppression algorithm (A-NMS) to remove redundant bounding boxes, obtain the final bounding box for 3D target detection, and then complete 3D target detection.

[0033] The present invention has the following advantages:

[0034] 1. This paper uses the attention mechanism to construct an adaptive expressiveness-enhanced fusion module to process the cascaded features of point clouds and images, thereby achieving the purpose of fusing cascaded features. Since the attention mechanism can enhance the importance of different features, some features in the fused features are not important, so we can assign smaller weights, while important features are assigned larger weights. Therefore, after adaptive expressiveness-enhanced fusion, fused features with different weights can be output, effectively suppressing the expression of unimportant features, which can improve the accuracy of 3D object detection in the subsequent detection output module.

[0035] 2. Therefore, the present invention can effectively perceive the environment using data obtained from cameras and lidar. Even in complex traffic environments, this method can still detect targets in the environment, ensuring the safety and stability of vehicle driving. The present invention fully utilizes the complementary advantages of sensors, and by fusing features of images lacking depth information and point clouds lacking color and texture information, it avoids the shortcomings of single-sensor target detection.

[0036] 3. This invention uses a spatial transformation projection algorithm to map 2D image features to 3D point cloud bird's-eye view features. These features are then concatenated with the point cloud bird's-eye view features and fed into an adaptive expressiveness enhancement fusion module, increasing the weight of important features and reducing the importance of less prominent ones. For the detection output network, the A-NMS algorithm effectively reduces missed and false detections of 3D objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a basic flow chart of the three-dimensional target detection method based on multimodal fusion in the present invention.

[0038] Figure 2 This is a schematic diagram of the spatial transformation projection algorithm provided by the spatial transformation module in the present invention.

[0039] Figure 3 Schematic diagram of the adaptive expressiveness enhancement fusion module provided by the present invention.

[0040] Figure 4 Schematic diagram of the system structure of the three-dimensional target detection system based on multimodal fusion in the present invention. DETAILED DESCRIPTION

[0041] The specific implementation methods of the present invention will be further explained in detail below through the description of embodiments with reference to the accompanying drawings, so as to help those skilled in the art to have a more complete, accurate and in-depth understanding of the inventive concept and technical solution of the present invention.

[0042] Example 1:

[0043] like Figure 1-4 As shown, the present invention provides a three-dimensional target detection method based on multimodal fusion, which includes the following steps.

[0044] S1. Obtain image data collected by the camera.

[0045] S2. Obtain the original point cloud data collected by the lidar and perform noise reduction preprocessing on the point cloud.

[0046] S3. Input the image and point cloud data into the trained corresponding networks respectively to obtain the deep semantic feature information of the targets in their respective data, i.e., the corresponding image features and point cloud features.

[0047] This step includes two parts: processing the data configured as a point cloud and processing the data configured as an image.

[0048] S3.1. Processing data configured as point clouds: Use the PointNet network as the backbone network for point cloud feature extraction to extract features from the input point cloud data; construct the input feature layer of the point cloud based on PointNet's direct coordinate convolution method, calculate the original data coordinate information, obtain the input feature map, and then use the convolutional neural network to complete the extraction of point cloud features.

[0049] S3.2. Process the data configured as images: Use the MobileNetv2 deep neural network as the basic convolutional layer to extract features from the input image data. To prevent the network parameters of the image feature extractor from being trained at the beginning, the model uses the trained parameters of the first few layers of MobileNetv2 as the initial convolution kernel values ​​of the image feature extractor to save training time and extract image features more quickly.

[0050] S4. Map the image features of the two-dimensional image to a three-dimensional point cloud space coordinate system.

[0051] Since point cloud features and image features are in different dimensions, we need to use a spatial transformation module to map the two-dimensional image features to the feature space of the three-dimensional point cloud, in preparation for the subsequent cascading of image features and point cloud features in the three-dimensional space.

[0052] This step specifically includes: first, given the target pixel i on the bird's-eye view of the point cloud, we extract K (generally ten points) lidar point clouds j closest to the target pixel; then we project the lidar point cloud j onto the front view of the camera to obtain the corresponding adjacent pixel x j ; Retrieve the image feature point x corresponding to the target pixel i i ; From each adjacent pixel x j To the camera front view and the image feature point x i Do the difference to get the continuous geometric offset x j -x i ; Finally, we will adjacent pixel point x j The corresponding input image feature f j and continuous geometric offset x j -x i Input the BP neural network to generate corresponding features mapped to the three-dimensional point cloud space coordinate system.

[0053] The image features are encoded based on the neighboring offsets between the LiDAR point cloud and the target pixel i to simulate the dependency between the point clouds. Each target pixel i provides input to the BP neural network. For each target pixel i, the BP neural network performs a j The BP neural network output is summed to output its image feature point x i Image feature h mapped to the three-dimensional point cloud space coordinate system i , the formula is as follows.

[0054]

[0055] Among them, f j is the input image feature corresponding to the lidar point cloud j, x j -x i is the geometric offset from the adjacent pixel point j to the target i projected on the camera front view, and concat(·) is the concatenation of multiple vectors.

[0056] S5. cascade the point cloud features with the mapped image features, recalibrate the cascaded image and point cloud features through an adaptive expressiveness enhancement fusion module, and then multiply the weights with the original corresponding channels to obtain weighted fusion features.

[0057] In this step, we primarily leverage the attention mechanism to construct an adaptive expressiveness enhancement fusion module to process the concatenated features of the point cloud and image, achieving the goal of fusing the cascaded features. This module enhances the expressiveness of different features, focusing on important features while suppressing less important ones. After weighted operations in the module, it outputs fused features of the image and point cloud with different weights.

[0058] Specifically: First, given an input image and a concatenated feature of a point cloud, the size of the feature layer is H×W×C. After the maximum pooling operation, the candidate size is changed to 1×1. Then, the first fully connected layer is used to reduce the number of channels to C / r, where r is a scaling factor used to reduce the number of channels and thus reduce the amount of calculation. Then, the ReLU activation function is used, and the dimension of the feature remains unchanged. The second fully connected layer is used to increase the dimension, and the number of channels returns to C; the Sigmoid function is used to output the weight of each channel; finally, the feature is recalibrated, and the weight is multiplied with the original corresponding channel to obtain the weighted fusion feature of the point cloud and image.

[0059] S6. Input the weighted fusion features into the detection output network to generate a bounding box for the target, thereby completing three-dimensional target detection.

[0060] This step specifically involves inputting the weighted fusion features into the detection output network and processing them through the region proposal network (RPN) to generate a corresponding bounding box and score for the corresponding target. Since each target generates a large number of bounding boxes in the candidate region, we use the adaptive non-maximum prediction algorithm (A-NMS) to process the generated bounding boxes, removing those with lower scores while retaining the bounding box with the highest score as the final target detection result. Detected targets include cars, pedestrians, and cyclists.

[0061] The detailed steps of A-NMS are: it sorts all bounding boxes according to their scores, selects the bounding box with the highest score, and uses a predefined threshold to suppress all other bounding boxes that have significant overlap with the bounding box with the highest score. This process is applied to the remaining bounding boxes in turn, and the intersection over union (IoU) of the highest-scoring bounding box with the bounding boxes of other candidate regions is calculated. The intersection over union (IoU) is the ratio of the intersection and union between the predicted box and the true bounding box. The calculation formula of the A-NMS algorithm is as follows:

[0062]

[0063] Among them, S i is the bounding box score after comparison and calculation, s i is the initial score of the bounding box, M is the bounding box with the highest score, and b iis the current bounding box, IoU(M,b i ) are M and b i The intersection-over-union ratio, N i 、N t Both are thresholds.

[0064] If the current bounding box b i The score IoU(M,b i )<N t , then remove the bounding box b i ; if N i <IoU(M,b i )<N t , then the bounding box b i Will be given IoU(M,b i ) penalty ratio is combined with the original score as a penalty factor; if IoU(M, b i )<N i , then b i If the overlap with M is small, then the bounding box b is not i Remove and keep the original score. This method can reduce IoU(M,b i ) is less than N i The probability of missed detection and false detection of objects in the bounding box.

[0065] Example 2:

[0066] like Figure 1-4 As shown, an embodiment of the present invention further provides a three-dimensional target detection system based on multimodal fusion, including a monocular camera, a laser radar, a camera and a data processing unit. The data processing unit is connected to the monocular camera and the laser radar to obtain data. The monocular camera collects images in front of the viewing angle in real time and sends them to the data processing unit; at the same time, the laser radar scans a point cloud map in the construction environment and sends it to the data processing unit; the data processing unit executes a computer program stored in itself to implement the steps of the three-dimensional target detection method based on multimodal fusion described in Example 1 to extract the target in front, and the steps include.

[0067] S1. Obtain image data collected by the camera.

[0068] S2. Obtain the original point cloud data collected by the lidar and perform noise reduction preprocessing on the point cloud.

[0069] S3. Input the image and point cloud data into the trained corresponding networks respectively to obtain the deep semantic feature information of the targets in their respective data, i.e., the corresponding image features and point cloud features.

[0070] S4. Map the image features of the two-dimensional image to a three-dimensional point cloud space coordinate system.

[0071] S5. cascade the point cloud features with the mapped image features, recalibrate the cascaded image and point cloud features through an adaptive expressiveness enhancement fusion module, and then multiply the weights with the original corresponding channels to obtain weighted fusion features.

[0072] S6. Input the weighted fusion features into the detection output network to generate a bounding box for the target, thereby completing three-dimensional target detection.

[0073] The specific definition of the above steps can be found in Example 1 and will not be described in detail here.

[0074] The data processing unit can be integrated into the vehicle control system and includes the following modules: a point cloud feature extraction module, an image feature extraction module, a space transformation projection module, an adaptive expressiveness enhancement fusion module and a detection output module.

[0075] The point cloud feature extraction module integrates a trained PointNet network model to process data configured as point clouds. It uses the PointNet network as the backbone for point cloud feature extraction, constructs the input feature layer of the point cloud based on PointNet's direct coordinate convolution method, calculates the raw data coordinate information, and generates an input feature map. A convolutional neural network is then used to complete feature extraction of the point cloud.

[0076] The image feature extraction module integrates a pre-trained MobileNetv2 network model to process image-configured data, using the MobileNetv2 deep neural network as the base convolutional layer to extract features from the input image data. To prevent the network parameters of the image feature extractor from being trained at the outset, the model uses the pre-trained parameters of the first few layers of MobileNetv2 as the initial convolution kernel values ​​for the image feature extractor, saving training time and enabling faster image feature extraction.

[0077] The spatial transformation module: Since point cloud features and image features are in different dimensions, our spatial transformation module can map two-dimensional image features to the feature space of three-dimensional point clouds, preparing for the subsequent cascading of image features and point cloud features in three-dimensional space.

[0078] The adaptive expressiveness enhancement fusion module concatenates the point cloud features extracted by the point cloud feature extraction network with the image features mapped by the spatial transformation module to produce concatenated image and point cloud features. This module enhances the expressiveness of different features, focusing on important features while suppressing unimportant ones. After weighted operations in the adaptive expressiveness enhancement fusion module, it outputs fused feature maps of the image and point cloud with different weights.

[0079] The detection output module inputs the fused features of the concatenated image and point cloud into the Region Proposal Network (RPN) to generate bounding boxes for the objects. Because different objects generate numerous bounding boxes, the A-NMS algorithm is used to remove redundant bounding boxes to obtain the final bounding box for 3D object detection, thereby completing 3D object detection.

[0080] The present invention is described above by way of example in conjunction with the accompanying drawings. It is obvious that the specific implementation of the present invention is not limited to the above-mentioned method. As long as various non-substantial improvements are made using the inventive concept and technical solution of the present invention, or the inventive concept and technical solution are directly applied to other occasions without improvement, they are all within the scope of protection of the present invention.

Claims

1. A three-dimensional object detection method based on multimodal fusion, characterized by: The following steps are involved: S1. Obtain image data collected by the camera; S2. Obtain the original point cloud data collected by the lidar and perform noise reduction preprocessing on the point cloud; S3. Input the image and point cloud data into the trained corresponding networks respectively to obtain the deep semantic feature information of the target in each data, that is, the corresponding image features and point cloud features; S4, mapping the image features of the two-dimensional image to a three-dimensional point cloud space coordinate system; S5. Concatenate the point cloud features with the mapped image features, input the concatenated image and point cloud features into an adaptive expressiveness enhancement fusion module, recalibrate the fused features, and multiply the weights by the original corresponding channels to obtain weighted fused features; S6. Input the weighted fusion features into the detection output network to generate a bounding box for the target, thereby completing 3D target detection. Step S4 specifically includes: first, a target pixel point on a given point cloud bird's-eye view , and extract the K lidar point clouds closest to the target pixel ; Then the lidar point cloud Project to the camera front view to get the corresponding adjacent pixel points ; Retrieve and target pixel Corresponding image feature points ; From each adjacent pixel To the camera front view and image feature points Subtraction to obtain continuous geometric offset ; Finally, the adjacent pixels Corresponding input image features and continuous geometry offset Input BP neural network to generate corresponding features mapped to the three-dimensional point cloud space coordinate system; For each target pixel , BP neural network through all its adjacent pixels The BP neural network output is summed to output its image feature points Image features mapped to a three-dimensional point cloud space coordinate system , the formula is as follows, in, Corresponding to the lidar point cloud Input image features, From adjacent pixels To the target The geometric offset produced by projection on the camera front view, is the concatenation of multiple vectors; In step S5, an adaptive expressiveness enhancement fusion module is constructed with the help of the attention mechanism. The adaptive expressiveness enhancement fusion module is used to enhance the expressiveness of different features, focusing on important features while suppressing non-important features. Specifically, the following steps are performed: first, given an input image and a cascade feature of a point cloud, the size of the feature layer is , after the maximum pooling operation, the spatial size becomes 1×1; then, the first fully connected layer is used to reduce the number of channels to , It is a scaling factor used to reduce the number of channels and thus reduce the amount of computation; then the ReLU activation function is used, and the dimension of the feature remains unchanged; the second fully connected layer is used to increase the dimension, and the number of channels changes back to ; The Sigmoid function is used to output the weight of each channel; finally, the concatenated features of the point cloud and image are recalibrated, and the weights are multiplied with the original corresponding channels to obtain the weighted fusion features of the point cloud and image.

2. The three-dimensional object detection method based on multimodal fusion according to claim 1, characterized in that: Step S3 includes: S3.

1. Processing the data configured as point cloud: Using the PointNet network as the backbone network for point cloud feature extraction to extract features from the input point cloud data; S3.

2. Process the data configured as images: Use the MobileNetv2 deep neural network as the basic convolutional layer to extract features from the input image data.

3. The three-dimensional object detection method based on multimodal fusion according to claim 2, characterized in that: Step S3.1 specifically includes: constructing the input feature layer of the point cloud according to the direct coordinate convolution method of PointNet, calculating the original data coordinate information to obtain the input feature map, and then using the convolutional neural network to complete the extraction of point cloud features; Step S3.2 specifically includes: the model uses the trained first few layer parameters of MobileNetv2 as the initial convolution kernel values ​​of the image feature extractor to obtain image features.

4. The three-dimensional target detection method based on multimodal fusion according to claim 1, characterized in that: Step S6 includes: inputting the weighted fusion features into the detection output network, and generating a corresponding bounding box and a corresponding score for the corresponding target through the region proposal network RPN processing; using the adaptive non-maximum prediction algorithm A-NMS to process the generated bounding box, removing the bounding box with a lower score, and retaining the bounding box with the largest score as the final target detection result.

5. The three-dimensional target detection method based on multimodal fusion according to claim 4, characterized in that: In step S6, the detailed steps of A-NMS include: sorting all bounding boxes according to their scores, selecting the bounding box with the highest score, and suppressing all other bounding boxes that have significant overlap with the bounding box with the highest score using a predefined threshold; this process is applied to the remaining bounding boxes in turn, and the intersection over union (IoU) of the bounding box with the highest score and the bounding boxes of other candidate regions is calculated. The IoU is the ratio of the intersection and union between the predicted box and the true value bounding box; the calculation formula of the A-NMS algorithm is as follows: in, is the bounding box score after comparison and calculation, is the initial score of the bounding box, is the bounding box with the highest score, is the current bounding box, for and The intersection-over-union ratio between 、 All are threshold values; If the current bounding box The bounding box with the highest score The score between , it means and The overlap is small and does not fit the bounding box Remove, keep the original score; if , then the bounding box will be given The weight of the penalty ratio is combined with the original score as a penalty factor; if , then remove the bounding box .

6. A three-dimensional object detection system based on multimodal fusion, characterized by: The method comprises a monocular camera, a laser radar, a camera and a data processing unit. The data processing unit is connected to the monocular camera and the laser radar to obtain data. The monocular camera collects images in front of the viewing angle in real time and sends them to the data processing unit. At the same time, the laser radar scans a point cloud map in the construction environment and sends it to the data processing unit. The data processing unit executes a computer program stored in itself to implement the steps of the three-dimensional target detection method based on multimodal fusion according to any one of claims 1 to 5 to extract the target in front.

7. The three-dimensional object detection system based on multimodal fusion according to claim 6, characterized in that: The data processing unit includes: The point cloud feature extraction module integrates the trained PointNet network model and is used to process data configured as point clouds to obtain point cloud features; The image feature extraction module integrates the trained MobileNetv2 network model and is used to process data configured as images to obtain image features; The spatial transformation module is used to map the two-dimensional image features to the feature space of the three-dimensional point cloud; Adaptive expressiveness enhancement fusion module, which is used to concatenate point cloud features with image features, recalibrate the concatenated image and point cloud features to enhance the expressiveness of different features, and then multiply the weights with the original corresponding channels to obtain weighted fusion features; The detection output module is used to input the fusion features of the image and point cloud into the region proposal network (RPN), generate a bounding box for the target, use the non-maximum suppression algorithm (A-NMS) to remove redundant bounding boxes, obtain the final bounding box for 3D target detection, and then complete 3D target detection.

Images