Multi-modal target detection system and detection method based on foreground perception enhancement

Abstract

The invention discloses a multi-modal target detection system and detection method based on foreground perception enhancement, fully combines the advantages of a two-dimensional image and a three-dimensional point cloud, and solves the problem of low attention of foreground points. Through a foreground and background mask mechanism and a focus convolution module, foreground information is respectively enhanced in two-dimensional and three-dimensional features, and feature expression is further optimized through a channel and a space attention module. And finally, the enhanced fusion features subjected to channel cascade pass through a region suggestion network and a confidence feed-forward network, and a detection result with relatively high confidence is screened out. Different initial weights are determined according to target categories, the number difference and the pose difference of foreground and background points in a prediction bounding box and a real bounding box are calculated to perform loss aggregation, back propagation is performed to further optimize the position precision of the prediction bounding box, an aggregation loss function of foreground perception provides effective supervision in the training process, and the prediction accuracy of the prediction bounding box is improved. And the robustness and the accuracy of the model are further improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection, and in particular relates to a multimodal target detection system and a detection method based on foreground perception enhancement. Background Art

[0002] Multimodal 3D object detection technology significantly improves the perception of the environment of unmanned system inspection vehicles by fusing data from different sensors (such as cameras and LiDAR). In complex environments, it helps unmanned system inspection vehicles more accurately identify obstacles, detect facility status, and respond to emergencies.

[0003] In the existing technology, target detection technology based on two-dimensional images is relatively mature. However, in a complex three-dimensional environment, relying solely on two-dimensional images to represent three-dimensional objects will cause the loss of three-dimensional structural information, which affects the accuracy and robustness of target detection. In addition, factors such as the viewing angle limitations of sensors such as LiDAR and object occlusion often lead to incomplete three-dimensional point cloud data, which in turn affects the detection effect. In the point cloud data obtained by sensors such as LiDAR, the foreground points account for a very small proportion. A large number of background points increases the complexity of processing and increases the training time. It may also introduce redundant features, affecting the training efficiency and accuracy of the model. Summary of the invention

[0004] The technical problem to be solved by the present invention is to overcome the defects of the multimodal target detection system in the prior art that the multimodal fusion effect is poor and the processing of background information limits the efficiency and accuracy of target detection, thereby providing a multimodal target detection system and detection method based on foreground perception enhancement.

[0005] The present invention discloses a multimodal target detection method based on foreground perception enhancement, comprising the following steps: Step S1: using a camera to collect two-dimensional image data of the road, using a laser radar to collect three-dimensional point cloud data of the road, and saving the calibration file and true value annotation file of the camera; Step S2: inputting the two-dimensional image data and the three-dimensional point cloud data into a multimodal fusion detection model, extracting a two-dimensional feature map of the two-dimensional image data, and extracting a three-dimensional feature map of the three-dimensional point cloud data; Step S3: assigning a weight to each pixel of the two-dimensional feature map through a two-dimensional feature enhancement module based on a foreground-background mask mechanism to obtain a foreground-background weight mask; and calculating a two-dimensional focus feature map based on the two-dimensional feature map and the foreground-background weight mask; Step S4: Processing the three-dimensional feature map through a focal convolution module to obtain an output position, where the output position is a combination of an important position of the three-dimensional feature map and its extended area and other unimportant positions; and calculating a three-dimensional focal feature map based on the three-dimensional feature map and the output position; Step S5: The two-dimensional focus feature map and the three-dimensional focus feature map are channel-cascaded to obtain fused features; global average pooling and global maximum pooling are performed on each channel of the fused features to generate a channel description vector; average pooling and maximum pooling are performed on the fused features along the channel dimension to generate a spatial attention map; enhanced fused features are calculated based on the channel description vector and the spatial attention map; Step S6: the enhanced fusion feature is input into the region proposal network to generate a candidate bounding box, the prediction confidence of the candidate bounding box is calculated through the confidence feedforward network, and the candidate bounding box whose prediction confidence exceeds a preset threshold is defined as the predicted bounding box; Step S7: Obtain the positions of the predicted bounding box and the true bounding box, and assign initial weights to each bounding box based on different target categories; calculate the pose difference and the difference in the number of foreground and background points between the predicted bounding box and the true bounding box; combine the initial weight with the foreground and background point difference and the pose difference to construct a foreground-aware aggregation loss function, and optimize the multimodal fusion detection model based on the back propagation of the loss function.

[0006] Furthermore, the step S2 includes the following steps: Step S2.1: extracting features of the two-dimensional image data through an image feature extraction network, wherein the image feature extraction network includes a ResNet-50 backbone network and an application feature pyramid network; the ResNet-50 backbone network extracts a multi-scale feature map of the two-dimensional image data layer by layer, and the application feature pyramid network fuses multiple layers of the multi-scale feature map from top to bottom to generate a two-dimensional feature map; Step S2.2: extracting features of the three-dimensional point cloud data through a voxel three-dimensional backbone network; the voxel three-dimensional backbone network divides the three-dimensional point cloud data into spatial voxel grids of different resolutions to form a voxelized space, and captures three-dimensional structural information in the voxelized space to generate a three-dimensional feature map.

[0007] Furthermore, the step S3 includes the following steps: Step S3.1: Obtain a two-dimensional feature map: ; Wherein, H and W represent the height and width of the two-dimensional feature map respectively, and C represents the number of channels of the two-dimensional feature map; Step S3.2: Define foreground-background weight mask: ; ; The value of the foreground-background weight mask at each pixel position is , represents the horizontal coordinate of the pixel, represents the vertical coordinate of the pixel, represents the foreground pixel weight, represents a background pixel weight, wherein the foreground pixel weight is greater than the background pixel weight; Step S3.3: Calculate the two-dimensional focus feature map: ; in, Represents the two-dimensional focus feature map at the pixel position At , the eigenvalue of channel k; is the eigenvalue of the two-dimensional feature map.

[0008] Further, the step S4 includes the following steps: Step S4.1: Obtaining a three-dimensional feature map: ; Wherein, H and W represent the height and width of the three-dimensional feature map respectively, and C represents the number of channels of the three-dimensional feature map; Step S4.2: Determine the importance values ​​of the points in the three-dimensional feature map to form a cube importance map: ; in, Represents the set of all points with features in the three-dimensional feature map, express Any point in ; is the center of the cube importance map, indicating point The importance value of is a predefined threshold, and finally a set of important points is obtained ; Step S4.3: Calculate the dynamic output convolution kernel of the cube importance map: ; in, represents the elements of the convolution kernel, Represents the elements within the convolution kernel; Step S4.4: Calculate the output position based on the dynamic output convolution kernel: .

[0009] in, This part merges the important points with their indices. The input point set that does not belong to The points are added to the result to ensure that the information of all input point sets is not lost. Finally, the formula The filtered important points and their neighborhood point sets are integrated with the unfiltered original point set to generate an output point set containing all key points and global information.

[0010] Furthermore, the step S5 includes the following steps: Step S5.1: Perform global average pooling and global maximum pooling on each channel of the fused feature to generate a channel description vector: ; in, represents the activation function, and represents the preset hyperparameters, The weight of the average feature of the attention channel, The weight used to focus on the largest feature of the channel; represents global average pooling, represents global maximum pooling; Step S5.2: Perform average pooling and maximum pooling on the fused features along the channel dimension to generate a spatial attention map: ; in, represents the activation function, Represents the convolution operation; Step S5.3: Calculate enhanced fusion features based on the channel description vector and the spatial attention map: .

[0011] in, represents the original 3D feature map, z represents the 3D feature channel attention map, Represents a 3D feature space attention map.

[0012] Further, the step S6 includes the following steps: Step S6.1: input the enhanced fusion feature into the region proposal network, generate a candidate bounding box, and output the position and size of the candidate bounding box; Step S6.2: Calculating the prediction confidence of the candidate bounding box through a confidence feedforward network; Step S6.3: Using a preset threshold to filter the candidate bounding boxes, and retaining the candidate bounding boxes with a prediction confidence greater than the preset threshold; Step S6.4: performing non-maximum suppression on the selected bounding boxes to remove candidate bounding boxes whose overlap amount exceeds a preset overlap threshold; Step S6.5: performing fine regression on the candidate bounding box, optimizing the coordinates and size of the candidate bounding box, so that the accuracy of the candidate bounding box is higher; Step S6.6: adjusting the centroid position of the candidate bounding box during the fine regression process so that the centroid of the candidate bounding box is more consistent with the centroid of the real target; Step S6.7: The candidate bounding box obtained by fine regression is used as the predicted bounding box, including position, size and centroid information.

[0013] Further, the step S7 includes the following steps: Step S7.1: Obtain the positions of the predicted bounding box and the true bounding box, and assign initial weights to each bounding box based on different target categories; wherein the centroid coordinates of the predicted bounding box are , the centroid coordinates of the true bounding box are The centroid coordinates of the real bounding box are obtained from the true value annotation file, and the centroid coordinates of the predicted bounding box are calculated by its eight corner points: ; Step S7.2: Calculate the Euclidean distance between the centroid coordinates of the real bounding box and the predicted bounding box: ; Step S7.3: Calculate the normal vector and the angle between the real bounding box and the predicted bounding box: ; ; ; in and are the front upper left corner and the back lower right corner of the predicted bounding box, respectively. and are the upper front left corner and lower back right corner of the ground truth bounding box, and are the normal vectors of the predicted bounding box and the true bounding box, respectively.

[0014] Step S7.4: Get the number of foreground and background points in the real bounding box and the predicted bounding box, and calculate their differences respectively: and ; Step S7.5: Combine the initial weights with the foreground-background point differences and the pose differences to construct a foreground-aware aggregation loss function, and optimize the multimodal fusion detection model based on back propagation of the loss function: ; Among them, a, b, and c represent the initial weights of different target categories. represents the distance difference hyperparameter, represents the angle difference hyperparameter, represents the foreground point hyperparameter, Represents the background point hyperparameter, which is used to control the intensity of supervision of different differences in the model.

[0015] A multimodal target detection system based on foreground perception enhancement, the system is executed to implement the above-mentioned multimodal target detection method.

[0016] Beneficial effects: The present invention discloses a multimodal target detection system and detection method based on foreground perception enhancement, which fully combines the advantages of two-dimensional images and three-dimensional point clouds, and solves the problem of low attention of foreground points. The present invention strengthens the foreground information in two-dimensional and three-dimensional features respectively through the foreground and background mask mechanism and the focal convolution module, and further optimizes the feature expression through the channel and spatial attention modules. Finally, the enhanced fusion features after channel cascading pass through the region proposal network and the confidence feedforward network to screen out the detection results with higher confidence. Different initial weights are determined according to the target category, and the difference in the number and posture of the foreground and background points in the predicted bounding box and the true bounding box is calculated to perform loss aggregation, and back propagation is performed to further optimize the position accuracy of the predicted bounding box. The foreground-aware aggregation loss function provides effective supervision during the training process, further improving the robustness and accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flowchart of the main method steps of the present invention; Figure 2 It is a schematic block diagram of the main structure of the system of the present invention; Figure 3 It is a schematic block diagram of the foreground point perception module structure of the present invention. DETAILED DESCRIPTION

[0019] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present application, so the present application is not limited by the specific embodiments disclosed below.

[0020] Embodiment 1: Reference Figure 1 , Figure 2 and Figure 3 As shown, this embodiment provides a multimodal target detection method based on foreground perception enhancement, comprising the following steps: Step S1: Use a camera to collect two-dimensional image data of the road, use a laser radar to collect three-dimensional point cloud data of the road, and save the calibration file and true value annotation file of the camera. In this embodiment, the two-dimensional image data is saved in .bin format, including the spatial coordinates (X, Y, Z) and reflection intensity information of each laser radar point; the three-dimensional point cloud data is saved in .png or .jpg format, including the left and right camera images at the corresponding timestamp; the calibration file is saved in .txt format, recording the coordinate transformation relationship between the camera and the laser radar. As a preferred embodiment of this embodiment, the coordinate transformation relationship is the transformation matrix from the laser radar coordinate system to the camera coordinate system; the true value annotation file is saved in .txt format, containing the three-dimensional bounding box information of each target object, including category (such as car, pedestrian, etc.), position, size and orientation, etc.

[0021] Step S2: inputting the two-dimensional image data and the three-dimensional point cloud data into a multimodal fusion detection model, extracting a two-dimensional feature map of the two-dimensional image data, and extracting a three-dimensional feature map of the three-dimensional point cloud data; Step S2.1: extracting features of the two-dimensional image data through an image feature extraction network, wherein the image feature extraction network includes a ResNet-50 backbone network and an applied feature pyramid network (FPN); the ResNet-50 backbone network extracts a multi-scale feature map of the two-dimensional image data layer by layer, and the applied feature pyramid network fuses multiple layers of the multi-scale feature map from top to bottom to generate a two-dimensional feature map; through the ResNet-50 backbone network and the applied feature pyramid network, feature information at different levels can be effectively integrated, thereby improving the detection capability of the target object.

[0022] Step S2.2: Extract features of the three-dimensional point cloud data through a voxel three-dimensional backbone network; the voxel three-dimensional backbone network divides the three-dimensional point cloud data into spatial voxel grids of different resolutions to form a voxelized space, and captures three-dimensional structural information in the voxelized space to generate a three-dimensional feature map. In the voxel three-dimensional backbone network, the three-dimensional point cloud data is divided into spatial voxel grids of different resolutions, so that the network can capture three-dimensional structural information in the voxelized space to enhance the spatial understanding of the scene and the detection accuracy.

[0023] Step S3: assigning a weight to each pixel of the two-dimensional feature map through a two-dimensional feature enhancement module based on a foreground-background mask mechanism to obtain a foreground-background weight mask; and calculating a two-dimensional focus feature map based on the two-dimensional feature map and the foreground-background weight mask; Step S3.1: Obtain a two-dimensional feature map: ; Wherein, H and W represent the height and width of the two-dimensional feature map respectively, and C represents the number of channels of the two-dimensional feature map; Step S3.2: Define foreground-background weight mask: ; ; The value of the foreground-background weight mask at each pixel position is , represents the horizontal coordinate of the pixel, represents the vertical coordinate of the pixel, represents the foreground pixel weight, represents a background pixel weight, wherein the foreground pixel weight is greater than the background pixel weight; Step S3.3: Calculate the two-dimensional focus feature map: ; in, Represents the two-dimensional focus feature map at the pixel position At , the eigenvalue of channel k; is the eigenvalue of the two-dimensional feature map.

[0024] Step S4: Processing the three-dimensional feature map through a focal convolution module to obtain an output position, where the output position is a combination of an important position of the three-dimensional feature map and its extended area and other unimportant positions; and calculating a three-dimensional focal feature map based on the three-dimensional feature map and the output position; Step S4.1: Obtaining a three-dimensional feature map: ; Wherein, H and W represent the height and width of the three-dimensional feature map respectively, and C represents the number of channels of the three-dimensional feature map; Step S4.2: Determine the importance values ​​of the points in the three-dimensional feature map to form a cube importance map: ; in, represents a three-dimensional feature map, Represents a point in a three-dimensional feature map; is the center of the cube importance map, indicating point The importance value of It is a predefined threshold, which is preset according to the actual scenario and data set; In this embodiment, the importance value It is obtained by a sigmoid function, which can map the input feature value to a range. Then, it is obtained by a predefined threshold , filter out important points; Step S4.3: Calculate the dynamic output convolution kernel of the cube importance map: ; in, represents the elements of the convolution kernel, represents the elements in the convolution kernel; in this embodiment, Represents the elements within the 3x3x3 convolution kernel.

[0025] Step S4.4: Calculate the output position based on the dynamic output convolution kernel: .

[0026] in, This part merges the important points with their indices. The input point set that does not belong to The points are added to the result to ensure that the information of all input point sets is not lost. Finally, the formula The filtered important points and their neighborhood point sets are integrated with the unfiltered original point set to generate an output point set containing all key points and global information.

[0027] Step S5: The two-dimensional focus feature map and the three-dimensional focus feature map are channel-cascaded to obtain fused features; global average pooling and global maximum pooling are performed on each channel of the fused features to generate a channel description vector; average pooling and maximum pooling are performed on the fused features along the channel dimension to generate a spatial attention map; enhanced fused features are calculated based on the channel description vector and the spatial attention map; Step S5.1: Perform global average pooling and global maximum pooling on each channel of the fused feature to generate a channel description vector: ; in, represents the activation function, and represents the preset hyperparameters, The weight of the average feature of the attention channel, The weight used to focus on the largest feature of the channel; represents global average pooling, represents global maximum pooling; Step S5.2: Perform average pooling and maximum pooling on the fused features along the channel dimension to generate a spatial attention map: ; in, represents the activation function, Represents the convolution operation; Step S5.3: Calculate enhanced fusion features based on the channel description vector and the spatial attention map: .

[0028] in, represents the original 3D feature map, z represents the 3D feature channel attention map, Represents a 3D feature space attention map.

[0029] Step S6: the enhanced fusion feature is input into the region proposal network to generate a candidate bounding box, the prediction confidence of the candidate bounding box is calculated through the confidence feedforward network, and the candidate bounding box whose prediction confidence exceeds a preset threshold is defined as the predicted bounding box; Step S6.1: input the enhanced fusion feature into the region proposal network, generate a candidate bounding box, and output the position and size of the candidate bounding box; Step S6.2: Calculating the prediction confidence of the candidate bounding box through a confidence feedforward network; Step S6.3: Using a preset threshold to filter the candidate bounding boxes, and retaining the candidate bounding boxes with a prediction confidence greater than the preset threshold; Step S6.4: performing non-maximum suppression on the selected bounding boxes to remove candidate bounding boxes whose overlap amount exceeds a preset overlap threshold; Step S6.5: performing fine regression on the candidate bounding box, optimizing the coordinates and size of the candidate bounding box, so that the accuracy of the candidate bounding box is higher; Step S6.6: adjusting the centroid position of the candidate bounding box during the fine regression process so that the centroid of the candidate bounding box is more consistent with the centroid of the real target; Step S6.7: The candidate bounding box obtained by fine regression is used as the predicted bounding box, including position, size and centroid information.

[0030] Step S7: Obtain the positions of the predicted bounding box and the true bounding box, assign initial weights to each bounding box based on different target categories; calculate the pose difference and the difference in the number of foreground and background points between the predicted bounding box and the true bounding box; combine the initial weight with the foreground and background point difference and the pose difference to construct a foreground-aware aggregation loss function, and optimize the multimodal fusion detection model based on the back propagation of the loss function, specifically optimizing the ResNet-50 backbone network of the image feature extraction network and the voxel three-dimensional backbone network. The improved predicted bounding box is finally obtained through the optimized multimodal fusion detection model. In this embodiment, the method steps described in step S7 are implemented by a foreground point perception module.

[0031] Step S7.1: Obtain the positions of the predicted bounding box and the true bounding box, and assign initial weights to each bounding box based on different target categories; wherein the centroid coordinates of the predicted bounding box are , the centroid coordinates of the true bounding box are The centroid coordinates of the real bounding box are obtained from the true value annotation file, and the centroid coordinates of the predicted bounding box are calculated by its eight corner points: ; Step S7.2: Calculate the Euclidean distance between the centroid coordinates of the real bounding box and the predicted bounding box: ; Step S7.3: Calculate the normal vector and the angle between the real bounding box and the predicted bounding box: ; ; ; Step S7.4: Get the number of foreground and background points in the real bounding box and the predicted bounding box, and calculate their differences respectively: and ; Step S7.5: Combine the initial weights with the foreground-background point differences and the pose differences to construct a foreground-aware aggregation loss function, and optimize the multimodal fusion detection model based on back propagation of the loss function: ; Wherein, a, b, and c represent the initial weights of different target categories. In this embodiment, a is for cars, b is for pedestrians, and c is for bicycles. represents the distance difference hyperparameter, represents the angle difference hyperparameter, represents the foreground point hyperparameter, Represents the background point hyperparameter, which is used to control the intensity of supervision of different differences in the model.

[0032] Specifically, starting from the loss function, the partial derivative (gradient) of the loss with respect to each model parameter is calculated layer by layer through the chain rule (chain derivation). The gradient calculated at each layer is cached during the back-propagation process so that it can be efficiently propagated to the previous layer. An optimization algorithm (Adam) is used to adjust each parameter according to its gradient. Through multiple iterations, the model parameters are gradually updated, so that the loss function value gradually decreases. The process of forward propagation, back-propagation, and parameter update is repeated until the set stopping condition is reached, such as the loss function converges or the specified number of training rounds is reached. This process of updating parameters through back-propagation and gradient descent enables the model to gradually approach the optimal solution, thereby performing better on new data.

[0033] This embodiment discloses a multimodal target detection method based on foreground perception enhancement, which fully combines the advantages of two-dimensional images and three-dimensional point clouds, and solves the problem of low attention of foreground points. The present invention strengthens the foreground information in two-dimensional and three-dimensional features respectively through the foreground and background mask mechanism and the focal convolution module, and further optimizes the feature expression through the channel and spatial attention modules. Finally, the enhanced fusion features after channel cascade pass through the region proposal network and the confidence feedforward network to screen out the detection results with higher confidence. Different initial weights are determined according to the target category, and the difference in the number and posture of the foreground and background points in the predicted bounding box and the real bounding box is calculated to perform loss aggregation, and back propagation is performed to further optimize the position accuracy of the predicted bounding box. The foreground-aware aggregation loss function provides effective supervision during the training process, further improving the robustness and accuracy of the model.

[0034] Embodiment 2: This embodiment provides a multimodal target detection system based on foreground perception enhancement, and the system is executed to implement the multimodal target detection method as described in the first embodiment.

[0035] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0036] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent application shall be subject to the attached claims.

Claims

1. A multimodal target detection method based on foreground perception enhancement, characterized in that: The following steps are involved: Step S1: using a camera to collect two-dimensional image data of the road, using a laser radar to collect three-dimensional point cloud data of the road, and saving the calibration file and true value annotation file of the camera; Step S2: inputting the two-dimensional image data and the three-dimensional point cloud data into a multimodal fusion detection model, extracting a two-dimensional feature map of the two-dimensional image data, and extracting a three-dimensional feature map of the three-dimensional point cloud data; Step S3: assigning a weight to each pixel of the two-dimensional feature map through a two-dimensional feature enhancement module based on a foreground-background mask mechanism to obtain a foreground-background weight mask; Calculating a two-dimensional focus feature map according to the two-dimensional feature map and the foreground and background weight mask; Step S4: Processing the three-dimensional feature map through a focal convolution module to obtain an output position, where the output position is a combination of an important position of the three-dimensional feature map, its extended area, and other unimportant positions; Calculate a three-dimensional focus feature map according to the three-dimensional feature map and the output position; Step S5: The two-dimensional focus feature map and the three-dimensional focus feature map are cascaded to obtain fusion features; Performing global average pooling and global maximum pooling on each channel of the fused feature to generate a channel description vector; performing average pooling and maximum pooling on the fused feature along the channel dimension to generate a spatial attention map; and calculating enhanced fused features based on the channel description vector and the spatial attention map; Step S6: The enhanced fusion feature is input into the region proposal network to generate a candidate bounding box, the prediction confidence of the candidate bounding box is calculated through the confidence feedforward network, and the candidate bounding box whose prediction confidence exceeds a preset threshold is defined as the predicted bounding box.

2. The multimodal target detection method based on foreground perception enhancement according to claim 1 is characterized in that: Also included are the following method steps: Step S7: obtaining the positions of the predicted bounding box and the true bounding box, and assigning an initial weight to each bounding box based on different target categories; Calculating the pose difference and the difference in the number of foreground and background points between the predicted bounding box and the true bounding box; The initial weights are combined with the foreground-background point differences and the pose differences to construct a foreground-aware aggregation loss function, and the multimodal fusion detection model is optimized based on back propagation of the loss function.

3. The multimodal target detection method based on foreground perception enhancement according to claim 1 is characterized in that: The step S2 comprises the following steps: Step S2.1: extracting features of the two-dimensional image data through an image feature extraction network, wherein the image feature extraction network includes a ResNet-50 backbone network and an application feature pyramid network; the ResNet-50 backbone network extracts a multi-scale feature map of the two-dimensional image data layer by layer, and the application feature pyramid network fuses multiple layers of the multi-scale feature map from top to bottom to generate a two-dimensional feature map; Step S2.2: extracting features of the three-dimensional point cloud data through a voxel three-dimensional backbone network; the voxel three-dimensional backbone network divides the three-dimensional point cloud data into spatial voxel grids of different resolutions to form a voxelized space, and captures three-dimensional structural information in the voxelized space to generate a three-dimensional feature map.

4. The multimodal target detection method based on foreground perception enhancement according to claim 1 is characterized in that: The step S3 comprises the following steps: Step S3.1: Obtain a two-dimensional feature map: ； Wherein, H and W represent the height and width of the two-dimensional feature map respectively, and C represents the number of channels of the two-dimensional feature map; Step S3.2: Define foreground-background weight mask: ； ； The value of the foreground-background weight mask at each pixel position is , represents the horizontal coordinate of the pixel, represents the vertical coordinate of the pixel, represents the foreground pixel weight, represents a background pixel weight, wherein the foreground pixel weight is greater than the background pixel weight; Step S3.3: Calculate the two-dimensional focus feature map: ； in, Represents the two-dimensional focus feature map at the pixel position At , the eigenvalue of channel k; is the eigenvalue of the two-dimensional feature map.

5. The multimodal target detection method based on foreground perception enhancement according to claim 1 is characterized in that: The step S4 comprises the following steps: Step S4.1: Obtaining a three-dimensional feature map: ； Wherein, H and W represent the height and width of the three-dimensional feature map respectively, and C represents the number of channels of the three-dimensional feature map; Step S4.2: Determine the importance values ​​of the points in the three-dimensional feature map to form a cube importance map: ； in, represents a three-dimensional feature map, Represents a point in a three-dimensional feature map; is the center of the cube importance map, indicating point The importance value of is a predefined threshold; Step S4.3: Calculate the dynamic output convolution kernel of the cube importance map: ； in, represents the elements of the convolution kernel, Represents the elements within the convolution kernel; Step S4.4: Calculate the output position based on the dynamic output convolution kernel: ； in, This part merges the important points with their indices. The input point set that does not belong to The points are added to the results; The filtered important points and their neighborhood point sets are integrated with the unfiltered original point set to generate an output point set containing all key points and global information.

6. The multimodal target detection method based on foreground perception enhancement according to claim 1 is characterized in that: The step S5 comprises the following steps: Step S5.1: Perform global average pooling and global maximum pooling on each channel of the fused feature to generate a channel description vector: ； in, represents the activation function, and represents the preset hyperparameters, The weight of the average feature of the attention channel, The weight used to focus on the largest feature of the channel; represents global average pooling, represents global maximum pooling; Step S5.2: Perform average pooling and maximum pooling on the fused features along the channel dimension to generate a spatial attention map: ； in, represents the activation function, Represents the convolution operation; Step S5.3: Calculate enhanced fusion features based on the channel description vector and the spatial attention map: ； in, represents the original 3D feature map, z represents the 3D feature channel attention map, Represents a 3D feature space attention map.

7. The multimodal target detection method based on foreground perception enhancement according to claim 1 is characterized in that: The step S6 comprises the following steps: Step S6.1: input the enhanced fusion feature into the region proposal network, generate a candidate bounding box, and output the position and size of the candidate bounding box; Step S6.2: Calculating the prediction confidence of the candidate bounding box through a confidence feedforward network; Step S6.3: Using a preset threshold to filter the candidate bounding boxes, and retaining the candidate bounding boxes with a prediction confidence greater than the preset threshold; Step S6.4: performing non-maximum suppression on the selected bounding boxes to remove candidate bounding boxes whose overlap amount exceeds a preset overlap threshold; Step S6.5: performing fine regression on the candidate bounding box, optimizing the coordinates and size of the candidate bounding box, so that the accuracy of the candidate bounding box is higher; Step S6.6: adjusting the centroid position of the candidate bounding box during the fine regression process so that the centroid of the candidate bounding box is more consistent with the centroid of the real target; Step S6.7: The candidate bounding box obtained by fine regression is used as the predicted bounding box, including position, size and centroid information.

8. The multimodal target detection method based on foreground perception enhancement according to claim 2 is characterized in that: The step S7 comprises the following steps: Step S7.1: Obtain the positions of the predicted bounding box and the true bounding box, and assign initial weights to each bounding box based on different target categories; wherein the centroid coordinates of the predicted bounding box are , the centroid coordinates of the true bounding box are The centroid coordinates of the real bounding box are obtained from the true value annotation file, and the centroid coordinates of the predicted bounding box are calculated by its eight corner points: ； Step S7.2: Calculate the Euclidean distance between the centroid coordinates of the real bounding box and the predicted bounding box: ； Step S7.3: Calculate the normal vector and the angle between the real bounding box and the predicted bounding box: ； ； ； in and are the front upper left corner and the back lower right corner of the predicted bounding box, respectively. and are the upper front left corner and lower back right corner of the ground truth bounding box, and are the normal vectors of the predicted bounding box and the true bounding box respectively; Step S7.4: Get the number of foreground and background points in the real bounding box and the predicted bounding box, and calculate their differences respectively: and ; Step S7.5: Combine the initial weights with the foreground-background point differences and the pose differences to construct a foreground-aware aggregation loss function, and optimize the multimodal fusion detection model based on back propagation of the loss function: ； Among them, a, b, and c represent the initial weights of different target categories. represents the distance difference hyperparameter, represents the angle difference hyperparameter, represents the foreground point hyperparameter, Represents the background point hyperparameter, which is used to control the intensity of supervision of different differences in the model.

9. A multimodal target detection system based on foreground perception enhancement, characterized in that: The system is executed to implement the multimodal target detection method as described in any one of claims 1 to 8.

Images