Target detection method and device based on multi-modal fusion, equipment and storage medium

Abstract

The invention relates to the technical field of automatic driving environment perception, and discloses a target detection method, device and equipment based on multi-modal fusion, and a storage medium, which are used for improving the accuracy of three-dimensional target detection. The multi-modal fusion-based target detection method comprises the steps of obtaining a two-dimensional detection frame and a three-dimensional detection frame of a current frame; performing data association on the two-dimensional detection frame and the three-dimensional detection frame through a dynamic adaptive bidirectional matching strategy to obtain a matching result, and calculating a matching score based on target geometric consistency, category confidence and spatial distribution similarity through the matching strategy; performing multi-stage optimization processing based on the matching result to obtain an optimized three-dimensional detection result; and performing track consistency verification on the optimized three-dimensional detection result in combination with the motion prediction information of the historical frame to obtain a fused three-dimensional target detection result.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving environment perception technology, and in particular to a target detection method, device, equipment and storage medium based on multimodal fusion. Background Art

[0002] Existing technologies usually first obtain the 2D detection frame and 3D detection frame of the current frame separately, and then use a simple method based on intersection over union (IoU) or fixed threshold to associate the 2D detection frame and 3D detection frame data. The match is determined only based on the degree of spatial overlap of the detection frames. The associated results are then subjected to conventional optimization processing, such as non-maximum suppression, to reduce duplicate detections, thereby obtaining a 3D target detection result.

[0003] The data association method based on intersection-over-union or fixed threshold is too simple and does not consider multidimensional factors. As a result, in complex scenarios, such as when the target is occluded or the shape changes greatly, the data association accuracy is low and it is easy to produce mismatches. Conventional optimization processing methods are relatively simple and cannot effectively deal with the various interference factors in complex scenarios. The optimization effect is limited, which affects the accuracy of the final three-dimensional target detection results. Summary of the Invention

[0004] The present invention provides a target detection method, apparatus, device and storage medium based on multimodal fusion to solve the problem in the prior art that three-dimensional target detection results are not accurate enough due to a single detection scheme.

[0005] A first aspect of the present invention provides a target detection method based on multimodal fusion, comprising: obtaining a two-dimensional detection frame and a three-dimensional detection frame of a current frame; performing data association on the two-dimensional detection frame and the three-dimensional detection frame through a dynamic adaptive bidirectional matching strategy to obtain a matching result, wherein the matching strategy calculates a matching score based on target geometric consistency, category confidence, and spatial distribution similarity; performing multi-stage optimization processing based on the matching result to obtain an optimized three-dimensional detection result; and performing trajectory consistency verification on the optimized three-dimensional detection result in combination with motion prediction information of historical frames to obtain a fused three-dimensional target detection result.

[0006] In an implementable embodiment, the data association of the two-dimensional detection box and the three-dimensional detection box through the dynamic adaptive bidirectional matching strategy to obtain a matching result comprises: extracting geometric features of the two-dimensional detection box and the three-dimensional detection box, and calculating a geometric consistency score; calculating a category confidence score according to category prediction results of the two-dimensional detection box and the three-dimensional detection box; analyzing the distribution of the two-dimensional detection box and the three-dimensional detection box in the image space, and calculating a spatial distribution similarity score; calculating a matching score based on the geometric consistency score, the category confidence score and the spatial distribution similarity score, and determining a matching result according to the matching score.

[0007] In an implementable embodiment, the multi-stage optimization processing based on the matching result comprises: for a matched detection pair, state optimization is performed by minimizing an IoU residual, a shape residual and a ground constraint residual; for a three-dimensional detection box that fails to match, elimination or retention is performed based on a preset confidence threshold; for a two-dimensional detection box that fails to match, a new three-dimensional detection box is generated through monocular reconstruction and ground height adjustment.

[0008] In an implementable embodiment, the state optimization of the matched detection pair by minimizing the IoU residual, the shape residual and the ground constraint residual comprises: calculating an intersection over union residual between the matched two-dimensional detection box and the three-dimensional detection box, and minimizing the intersection over union residual to adjust the position and size of the three-dimensional detection box; analyzing the shape features of the three-dimensional detection box, calculating a shape residual, and minimizing the shape residual to optimize the shape of the three-dimensional detection box; calculating and optimizing a ground constraint residual through the spatial constraint relationship between the three-dimensional detection box and the ground to ensure that the three-dimensional detection box meets the geometric conditions of ground fitting.

[0009] In an implementable embodiment, the elimination or retention of the three-dimensional detection box that fails to match based on the preset confidence threshold comprises: for each three-dimensional detection box that fails to match, comparing its confidence with the preset confidence threshold; if the confidence of the three-dimensional detection box is lower than the confidence threshold, eliminating it; if the confidence of the three-dimensional detection box is not lower than the confidence threshold, retaining it.

[0010] In an implementable embodiment, the generation of a new three-dimensional detection box through monocular reconstruction and ground height adjustment for the two-dimensional detection box that fails to match comprises: estimating the three-dimensional spatial position and size of the two-dimensional detection box that fails to match by using a preset camera intrinsic parameter and a monocular vision method to obtain an initial three-dimensional detection box; obtaining ground height information, and determining the ground height at the position of the initial three-dimensional detection box based on the ground height information; and taking the adjusted initial three-dimensional detection box as a new three-dimensional detection result to supplement the three-dimensional information corresponding to the two-dimensional detection box that fails to match.

[0011] In an implementation, the motion prediction information of the historical frames is combined to perform trajectory consistency verification on the optimized three-dimensional detection result, to obtain a fused three-dimensional target detection result, including: using the three-dimensional position and motion state information of the tracked targets in the historical frames, the expected position and motion trajectory of each target in the current frame are predicted through a motion model; the optimized three-dimensional detection result is compared with the predicted expected position and motion trajectory, and the deviation between the actual detection result and the predicted result is calculated; the optimized three-dimensional detection result is adjusted according to the calculated deviation, to obtain a three-dimensional target detection result fused with current detection and motion prediction.

[0012] In an implementation, the motion prediction information of the historical frames is combined to perform trajectory consistency verification on the optimized three-dimensional detection result, to obtain a fused three-dimensional target detection result, including: using the three-dimensional position and motion state information of the tracked targets in the historical frames, the expected position and motion trajectory of each target in the current frame are predicted through a motion model; the optimized three-dimensional detection result is compared with the predicted expected position and motion trajectory, and the deviation between the actual detection result and the predicted result is calculated; the optimized three-dimensional detection result is adjusted according to the calculated deviation, to obtain a three-dimensional target detection result fused with current detection and motion prediction.

[0013] In an implementation, the motion prediction information of the historical frames is combined to perform trajectory consistency verification on the optimized three-dimensional detection result, to obtain a fused three-dimensional target detection result, including: using the three-dimensional position and motion state information of the tracked targets in the historical frames, the expected position and motion trajectory of each target in the current frame are predicted through a motion model; the optimized three-dimensional detection result is compared with the predicted expected position and motion trajectory, and the deviation between the actual detection result and the predicted result is calculated; the optimized three-dimensional detection result is adjusted according to the calculated deviation, to obtain a three-dimensional target detection result fused with current detection and motion prediction.

[0014] In an implementation, the motion prediction information of the historical frames is combined to perform trajectory consistency verification on the optimized three-dimensional detection result, to obtain a fused three-dimensional target detection result, including: using the three-dimensional position and motion state information of the tracked targets in the historical frames, the expected position and motion trajectory of each target in the current frame are predicted through a motion model; the optimized three-dimensional detection result is compared with the predicted expected position and motion trajectory, and the deviation between the actual detection result and the predicted result is calculated; the optimized three-dimensional detection result is adjusted according to the calculated deviation, to obtain a three-dimensional target detection result fused with current detection and motion prediction.

[0015] In a second aspect, the present application provides a target detection device based on multi-modal fusion, comprising: an acquisition module configured to acquire a two-dimensional detection box and a three-dimensional detection box of a current frame; an association module configured to perform data association on the two-dimensional detection box and the three-dimensional detection box through a dynamic adaptive bidirectional matching strategy to obtain a matching result, wherein the matching strategy is based on target geometric consistency, category confidence and spatial distribution similarity to calculate a matching score; an optimization module configured to perform multi-stage optimization processing based on the matching result to obtain an optimized three-dimensional detection result; and a verification module configured to perform trajectory consistency verification on the optimized three-dimensional detection result in combination with motion prediction information of a historical frame to obtain a fused three-dimensional target detection result.

[0016] In an implementable embodiment, the association module is specifically configured to: extract geometric features of the two-dimensional detection box and the three-dimensional detection box to calculate a geometric consistency score; calculate a category confidence score according to category prediction results of the two-dimensional detection box and the three-dimensional detection box; analyze the distribution of the two-dimensional detection box and the three-dimensional detection box in image space to calculate a spatial distribution similarity score; calculate a matching score based on the geometric consistency score, the category confidence score and the spatial distribution similarity score, and determine a matching result according to the matching score.

[0017] In an implementable embodiment, the optimization module comprises: an optimization unit configured to perform state optimization on a matched detection pair by minimizing an IoU residual, a shape residual and a ground constraint residual; a processing unit configured to perform elimination or reservation on a three-dimensional detection box that fails to match based on a preset confidence threshold; and a generation unit configured to generate a new three-dimensional detection box for a two-dimensional detection box that fails to match through monocular reconstruction and ground height adjustment.

[0018] In an implementable embodiment, the optimization unit is specifically configured to: calculate an intersection-over-union residual between a matched two-dimensional detection box and a three-dimensional detection box, and minimize the intersection-over-union residual to adjust the position and size of the three-dimensional detection box; analyze shape features of the three-dimensional detection box, calculate a shape residual, and minimize the shape residual to optimize the shape of the three-dimensional detection box; calculate and optimize a ground constraint residual through a spatial constraint relationship between the three-dimensional detection box and the ground to ensure that the three-dimensional detection box meets the geometric conditions of ground fitting.

[0019] In an implementable embodiment, the processing unit is specifically configured to: compare the confidence of each three-dimensional detection box that fails to match with a preset confidence threshold; if the confidence of the three-dimensional detection box is lower than the confidence threshold, eliminate the three-dimensional detection box; and if the confidence of the three-dimensional detection box is not lower than the confidence threshold, retain the three-dimensional detection box.

[0020] In an implementable embodiment, the generating unit is specifically configured to: estimate the three-dimensional spatial position and size of the two-dimensional detection frame that fails to be matched successfully by using preset camera internal parameters and a monocular vision method, to obtain an initial three-dimensional detection frame; obtain ground height information, determine the ground height of the position where the initial three-dimensional detection frame is located based on the ground height information; and take the adjusted initial three-dimensional detection frame as a new three-dimensional detection result, to supplement the three-dimensional information corresponding to the two-dimensional detection frame that fails to be matched successfully.

[0021] In an implementable embodiment, the verifying module comprises: a predicting unit configured to predict the expected position and motion trajectory of each target in the current frame by a motion model, by using the three-dimensional position and motion state information of the tracked targets in the historical frames; a calculating unit configured to compare the optimized three-dimensional detection result with the predicted expected position and motion trajectory, and calculate the deviation between the actual detection result and the predicted result; and an adjusting unit configured to adjust the optimized three-dimensional detection result according to the calculated deviation, to obtain the three-dimensional target detection result that fuses the current detection and motion prediction.

[0022] In an implementable embodiment, the predicting unit is specifically configured to: extract the continuous three-dimensional position coordinates and motion velocity information of the tracked targets from the historical frame data; select a linear motion model or a nonlinear motion model according to the motion characteristics of the targets; and recursively calculate the expected three-dimensional position and motion trajectory of each target in the current frame by the selected motion model.

[0023] In an implementable embodiment, the calculating unit is specifically configured to: calculate the Euclidean distance between the center point of the optimized three-dimensional detection frame and the expected position at the corresponding time point on the predicted trajectory; calculate the included angle deviation between the motion direction of the optimized three-dimensional detection frame and the motion direction of the predicted trajectory; and generate a trajectory consistency score based on the Euclidean distance and the included angle deviation, as the deviation result.

[0024] In an implementable embodiment, the adjusting unit is specifically configured to: if the calculated deviation result is less than a preset threshold, adopt the optimized three-dimensional detection result as the final output; and if the calculated deviation result is not less than the preset threshold, correct the position and motion state parameters of the optimized three-dimensional detection frame according to the predicted trajectory, to obtain the fused three-dimensional target detection result.

[0025] The third aspect of the present application provides an electronic device, comprising: a memory and at least one processor, the memory having instructions stored therein; the at least one processor invoking the instructions in the memory to cause the electronic device to perform the target detection method based on multi-modal fusion described above.

[0026] The fourth aspect of the present application provides a computer readable storage medium, wherein instructions are stored in the computer readable storage medium, and when the instructions are executed on a computer, the computer is caused to execute the target detection method based on multi-modal fusion.

[0027] In the technical solution provided by the present application, the two-dimensional detection box and the three-dimensional detection box of the current frame are obtained; the data correlation of the two-dimensional detection box and the three-dimensional detection box is performed through a dynamic adaptive bidirectional matching strategy to obtain a matching result, and the matching strategy calculates a matching score based on target geometric consistency, category confidence and spatial distribution similarity; multi-stage optimization processing is performed based on the matching result to obtain an optimized three-dimensional detection result; and trajectory consistency verification is performed on the optimized three-dimensional detection result in combination with motion prediction information of a historical frame to obtain a fused three-dimensional target detection result. In the embodiment of the present application, the matching score is calculated based on target geometric consistency, category confidence and spatial distribution similarity, and multi-dimensional comprehensive consideration is performed for data correlation, so that the two-dimensional detection box and the three-dimensional detection box can be more accurately matched in a complex scene, the accuracy of data correlation is improved, multi-stage optimization processing is performed based on the matching result, compared with a conventional single optimization mode, interference factors in a complex scene can be more comprehensively processed, the optimization effect is improved, and at the same time, the trajectory consistency verification is performed on the optimized three-dimensional detection result in combination with the motion prediction information of the historical frame, the historical information is fully utilized, the target trajectory can be more accurately judged to be consistent or not, the target motion state mutation or abnormal motion condition is effectively coped with, and therefore a more accurate and reliable fused three-dimensional target detection result is obtained. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 An embodiment schematic diagram of the target detection method based on multi-modal fusion in the embodiment of the present application is shown in FIG. 1. Figure 2 Another embodiment schematic diagram of the target detection method based on multi-modal fusion in the embodiment of the present application is shown in FIG. 2. Figure 3 An embodiment schematic diagram of the point cloud detector false detection in the embodiment of the present application is shown in FIG. 3. Figure 4 Another embodiment schematic diagram of the target detection method based on multi-modal fusion in the embodiment of the present application is shown in FIG. 4. Figure 5 A nuScenes dataset sensor hardware configuration diagram in the experimental embodiment of the present application is shown in FIG. 5. Figure 6 A KITTI dataset sensor hardware configuration diagram in the experimental embodiment of the present application is shown in FIG. 6. Figure 7 A comparison between the original 3D detection and the refined 3D detection result in the experimental embodiment of the present application is shown in FIG. 7. Figure 8A schematic diagram of an algorithm for reconstructing the depth of a detection frame and adjusting the z-axis coordinates in an experimental embodiment of the present invention; Figure 9 Schematic diagram of an embodiment of a target detection device based on multimodal fusion in an embodiment of the present invention; Figure 10 2 is a schematic diagram of another embodiment of an object detection device based on multimodal fusion according to an embodiment of the present invention; Figure 11 FIG. 1 is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The embodiments of the present invention provide a target detection method, apparatus, device and storage medium based on multimodal fusion, which improve the accuracy of three-dimensional target detection.

[0030] The terms "first," "second," "third," "fourth," and so on (if any) in the description and claims of the present invention and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that shown or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, product, or apparatus.

[0031] It is understandable that the execution subject of the present invention can be a target detection device based on multimodal fusion, or a terminal or a server, which is not limited here. The embodiment of the present invention is described by taking the server as the execution subject as an example.

[0032] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 In one embodiment of the present invention, a method for target detection based on multimodal fusion includes: 101. Obtain the 2D detection frame and 3D detection frame of the current frame; The image of the current frame is processed through the 2D detection network to generate a two-dimensional detection box in the image plane, including the pixel coordinates and detection score of each detection box; the laser point cloud of the current frame is processed using the 3D detection network to generate a three-dimensional detection box in the lidar coordinate system, including the three-dimensional position, size, yaw angle and confidence score of each detection box. The 2D detection network can be a YOLOv5 network, etc., and the 3D detection network can be PointPillars, SECOND or CenterPoint.

[0033] 102、by dynamic adaptive bidirectional matching strategy for two-dimensional detection frame and three-dimensional detection frame data association, get matching results, matching strategy based on target geometric consistency, class confidence and spatial distribution similarity calculation matching score; Geometric consistency is evaluated by comparing the similarity of two-dimensional detection frame and three-dimensional detection frame in shape and size; the class confidence is based on the prediction results of two-dimensional detector and three-dimensional detector and its confidence score; the spatial distribution similarity analyzes the relative position relationship of detection frame in image space and point cloud space; by weighted sum of the score of the three factors, the comprehensive matching score between each pair of two-dimensional detection frame and three-dimensional detection frame is obtained; according to the height of matching score, the matching successful detection pair and the three-dimensional detection frame and two-dimensional detection frame which are not matched successfully are divided.

[0034] 103、based on the matching results of multi-stage optimization processing, get the optimized three-dimensional detection results; For the matched two-dimensional and three-dimensional detection frame pair, the optimization process is realized by minimizing the three-dimensional IoU residual and the two-dimensional IoU residual at the same time: the three-dimensional IoU residual measures the spatial overlap difference between the three-dimensional detection frame and the three-dimensional frame restored by inverse projection of the two-dimensional frame projected in the image plane, and the two-dimensional IoU residual measures the plane overlap difference between the original two-dimensional detection frame and the two-dimensional frame projected by the three-dimensional frame; At the same time, shape constraint residual is introduced, which adjusts the size parameters of three-dimensional frame by using the aspect ratio information of target category prior, to ensure that its geometric shape conforms to the actual object characteristics; Ground constraint residual fits the ground point cloud by RANSAC algorithm, forces the bottom of three-dimensional frame to conform to the ground, and eliminates the positioning deviation in the vertical direction. The residual term constitutes a nonlinear optimization objective function, which is solved iteratively by using Ceres solver to correct the center position, size and yaw angle of three-dimensional frame.

[0035] For the three-dimensional detection frame which is not matched successfully, if its confidence score is lower than the preset confidence threshold, it is directly eliminated, otherwise it is retained; the two-dimensional detection frame which is not matched successfully generates an initial three-dimensional frame after monocular reconstruction, and optimizes its depth and z-axis coordinates by using the ground height adjustment algorithm to ensure that the frame body bottom conforms to the ground.

[0036] By integrating the optimization processing results of matched detection pairs, three-dimensional detection frames and two-dimensional detection frames which are not matched successfully, the three-dimensional detection results after multi-stage optimization are obtained.

[0037] 104、combined with the motion prediction information of historical frames, the optimized three-dimensional detection results are verified for trajectory consistency, and the fused three-dimensional target detection results are obtained.

[0038] The three-dimensional position, speed and acceleration information of the tracked target in the historical frame is used to construct a state transition model, the expected position and motion trajectory of each target in the current frame is predicted through Kalman filtering to generate a predicted state including a position covariance matrix; the optimized three-dimensional detection result is spatially associated with the predicted trajectory, data association is realized through the Hungarian algorithm, the Mahalanobis distance between the detection result and the predicted position is calculated as a matching measure, and the detection pairs with short distance and small covariance are preferentially associated; for the unassociated detection result, secondary matching is performed according to the similarity of the motion state and the historical trajectory, and temporary occlusion or detection loss is tolerated; if the deviation between the detection result and the predicted trajectory exceeds a threshold, the current detection position is adjusted through weighted fusion, and the weight is determined by the detection confidence and the prediction covariance; finally, the corrected three-dimensional detection box is subjected to trajectory smoothing processing, the historical position information in the sliding window is used to fit a cubic spline curve, the instantaneous detection noise is eliminated, and a three-dimensional target detection result that is continuous in space and time and has consistent motion state is generated.

[0039] In the embodiment of the application, the matching score is calculated based on target geometric consistency, category confidence and spatial distribution similarity, multi-dimensional comprehensive consideration is performed for data association, the two-dimensional detection box and the three-dimensional detection box can be more accurately matched in a complex scene, the accuracy of data association is improved, multi-stage optimization processing is performed based on the matching result, compared with the conventional single optimization mode, the interference factors in the complex scene can be more comprehensively processed, the optimization effect is improved, at the same time, the trajectory consistency of the optimized three-dimensional detection result is verified in combination with the motion prediction information of the historical frame, the historical information is fully utilized, the target trajectory can be more accurately judged, the target motion state mutation or abnormal motion condition can be effectively coped with, and therefore more accurate and reliable fused three-dimensional target detection result is obtained.

[0040] Please refer to Figure 2 Another embodiment of the target detection method based on multi-modal fusion in the embodiment of the application includes: 201, acquiring a two-dimensional detection box and a three-dimensional detection box of a current frame; The step 201 is similar to the execution process of the step 101 described above, and will not be described here.

[0041] 202, performing data association on the two-dimensional detection box and the three-dimensional detection box through a dynamic adaptive bidirectional matching strategy to obtain a matching result, and the matching strategy calculates a matching score based on target geometric consistency, category confidence and spatial distribution similarity; The geometric features of the two-dimensional detection frame and the three-dimensional detection frame are extracted, and a geometric consistency score is calculated; according to the category prediction results of the two-dimensional detection frame and the three-dimensional detection frame, a category confidence score is calculated; the distribution of the two-dimensional detection frame and the three-dimensional detection frame in the image space is analyzed, and a spatial distribution similarity score is calculated; a matching score is calculated based on the geometric consistency score, the category confidence score and the spatial distribution similarity score, and a matching result is determined according to the matching score.

[0042] The eight vertices of the three-dimensional detection frame are mapped to the two-dimensional image plane through the camera projection matrix calibrated in advance to obtain the projected two-dimensional polygon region; the intersection over union between the projected two-dimensional polygon region and the original two-dimensional detection frame is calculated as the basic geometric consistency measure; at the same time, the projection coordinates of the center point of the three-dimensional detection frame in the image plane and the Euclidean distance of the center point of the two-dimensional detection frame are extracted respectively and normalized; further, a plurality of points are uniformly sampled along the four edges of the two-dimensional detection frame, and the average value of the shortest distance from each sampling point to the edge of the projected three-dimensional detection frame is calculated; based on the three geometric features of the calculated intersection over union, Euclidean distance and average value of the shortest distance, a weighted fusion method is used to calculate the final geometric consistency score, wherein the intersection over union feature is given the maximum weight, the center point distance is given the second place, and the edge alignment degree is given appropriate weight; further, to enhance robustness, the distance feature can be nonlinearly mapped using a Gaussian kernel function, and an adaptive normalization method is used to eliminate the influence of detection frames of different scales, and the geometric consistency score can comprehensively reflect the matching degree of the two-dimensional detection frame and the three-dimensional detection frame in terms of position, shape and edge alignment. The prior art usually only relies on a single intersection over union (IoU) to evaluate the geometric matching degree of two-dimensional and three-dimensional detection frames, which has the problems of single evaluation dimension, insensitivity to edge alignment and shape difference, and is easily affected by local occlusion and scale change in complex scenes or small target detection. The present scheme realizes multi-dimensional collaborative evaluation of the position, shape and edge alignment of the detection frame by fusing the IoU, normalized center point distance and average value of the shortest edge distance, and using weighted fusion and nonlinear mapping to enhance robustness, which significantly improves the comprehensiveness and accuracy of two-dimensional-three-dimensional detection frame matching in complex scenes.

[0043] The prediction probability values of two detection boxes for the same target category are obtained, the class prediction probability of the two-dimensional detector is multiplied element by element with the class prediction probability of the three-dimensional detector to obtain an initial joint probability distribution; considering the importance difference of different categories in the autonomous driving scene, a category weight coefficient is introduced to give higher weight to key categories such as vehicles and pedestrians; a cross-modal confidence calibration model is established, the mapping relationship between the two-dimensional confidence and the three-dimensional confidence is fitted through the kernel density estimation method, and the deviation caused by the difference between the confidence distributions of different detectors is eliminated; the calibrated confidence difference absolute value is calculated, and the exponential function is used to convert it into a similarity measure, and finally the weighted joint category probability and the calibrated confidence similarity are linearly weighted and fused, wherein the category probability part occupies the dominant weight, and the comprehensive category confidence score is formed, which considers the semantic consistency of the cross-modal detection results and solves the problem of inconsistent output scales of different detectors through confidence calibration, and can effectively distinguish the correct and incorrect matching of the detection box pair. The prior art generally directly fuses the original category probability of the two-dimensional and three-dimensional detection boxes, such as simple average or maximum fusion, or only through fixed weight weighting, without considering the importance difference of different categories in the autonomous driving scene and the scale inconsistency problem of the confidence distribution of different detectors, resulting in insufficient discrimination ability for key categories (such as vehicles and pedestrians) and being easily affected by the detector bias. The present scheme combines the joint probability distribution by element by element multiplication, introduces the category weight coefficient to strengthen the key categories, uses the kernel density estimation to calibrate the cross-modal confidence and convert it into a similarity measure, and finally weights and fuses the joint probability and the calibrated similarity, which not only guarantees the semantic consistency, but also eliminates the output difference of the detector, significantly improving the accuracy and robustness of the cross-modal detection box matching in complex scenes.

[0044] A local neighborhood space is constructed in the image plane, a circular region with a fixed radius is defined centered on the target detection box, the number distribution characteristics of all two-dimensional detection boxes in the region are counted, and the number distribution characteristics of the corresponding three-dimensional detection boxes after projection are calculated to obtain the local density distribution histogram in two modalities; the relative position relationship between the target detection box and other detection boxes in its neighborhood is extracted to construct a spatial relationship description matrix, which records the azimuth angle and distance ratio in two-dimensional space and projected three-dimensional space respectively; the distribution similarity is measured by calculating the Bhattacharyya coefficient of the two density distribution histograms, and the spatial relationship matrix difference is compared using the Frobenius norm; the distribution similarity measure and the spatial relationship difference are weighted and fused, with the density distribution characteristics being given a higher weight to highlight the subject structure similarity; finally, the fused distance measure is converted into a similarity score using a Gaussian radial basis function, and an adaptive normalization method is used to eliminate the scale difference caused by scene complexity, which can effectively represent the topological structure consistency between detection boxes in a multi-target scene. The prior art generally only judges the spatial relationship of detection boxes in a multi-target scene through simple overlap or a single distance threshold, which leads to false matching in dense scenes due to local occlusion or scale difference. The present scheme constructs the density distribution histogram and spatial relationship description matrix of the local neighborhood, combines the Bhattacharyya coefficient and the Frobenius norm to quantify the distribution similarity and structure difference, and uses weighted fusion and Gaussian radial basis function conversion to achieve fine evaluation of the multi-target topological structure consistency, significantly improving the robustness and accuracy of cross-modal detection box matching in complex scenes.

[0045] The three scores are normalized and assigned different weights, with the geometric feature weight being the highest, the category feature weight being the second, and the spatial distribution feature weight being the lowest. After obtaining the comprehensive matching score by weighted summation, a dynamic threshold strategy is used to select candidate matching pairs, an improved Hungarian algorithm is used for optimal bipartite graph matching to ensure that each detection box is matched with at most one corresponding box, and after the matching is completed, the matching result is output, including the matched detection pairs, the three-dimensional detection boxes that are not matched successfully, and the two-dimensional detection boxes that are not matched successfully.

[0046] For example, when normalizing the geometric consistency score, the category confidence score and the spatial distribution similarity score, the Min-Max normalization method can be used to linearly map each dimension score to the interval [0, 1] to eliminate the dimensional difference; according to the multi-modal detection task characteristics, the weight can be assigned, wherein the geometric feature weight can be set to 0.6 to highlight the core position of the spatial position matching, the category feature weight can be set to 0.3 to reflect the semantic consistency, and the spatial distribution feature weight can be set to 0.1 as an auxiliary constraint; the comprehensive matching score is calculated by a weighted sum formula St = 0.6 * Sg + 0.3 * Sc + 0.1 * Ss, wherein St, Sg, Sc and Ss are the comprehensive matching score, the normalized consistency score, the category confidence score and the spatial distribution similarity score respectively; a dynamic threshold strategy is adopted to automatically adjust the screening threshold according to the current frame detection box density, and only the candidate matching pairs with a score higher than the threshold are retained; finally, an improved Hungarian algorithm is used to construct a cost matrix to solve the global optimal bipartite graph matching, so that each two-dimensional / three-dimensional detection box is matched with at most one corresponding box, and after the matching is completed, three types of results are output: matched detection pairs, unmatched three-dimensional detection boxes and unmatched two-dimensional detection boxes.

[0047] 203、for the matched detection pairs, state optimization is performed by minimizing the IoU residual, shape residual and ground constraint residual; The intersection over union residual between the matched two-dimensional detection box and the three-dimensional detection box is calculated, and the intersection over union residual is minimized to adjust the position and size of the three-dimensional detection box; the shape feature of the three-dimensional detection box is analyzed, the shape residual is calculated, and the shape residual is minimized to optimize the shape of the three-dimensional detection box; the ground constraint residual is calculated and optimized through the spatial constraint relationship between the three-dimensional detection box and the ground, to ensure that the three-dimensional detection box meets the geometric conditions of ground fitting. The prior art fails to consider the influence of the detection box position, size, shape and ground constraint on the detection result, resulting in limited detection accuracy in complex scenes. However, by calculating and minimizing the intersection over union residual, the position and size of the three-dimensional detection box can be accurately adjusted to make it more matched with the two-dimensional detection box, the shape residual can be calculated and optimized to effectively optimize the shape of the three-dimensional detection box to fit the actual shape of the target, and by calculating and optimizing the ground constraint residual, the three-dimensional detection box can meet the geometric conditions of ground fitting, significantly improving the accuracy and reliability of three-dimensional target detection in complex scenes.

[0048] The intersection over union (IoU) residual optimization target is to minimize the spatial overlap difference (3D IoU) between the three-dimensional detection box and the three-dimensional box generated by inverse projection of the two-dimensional box projected on the image plane of the two-dimensional detection box, and the planar overlap difference (2D IoU) between the two-dimensional detection box and the two-dimensional box projected by the three-dimensional box.

[0049] 3D IoU residual: The residual between the three-dimensional detection box B3D and the optimized three-dimensional box Bopt is defined as:

[0050] in, It is the 3D intersection-union ratio, which is calculated as the ratio of the intersection to the union of the volumes of two boxes.

[0051] 3D IoU residual: The optimized three-dimensional box Bopt is projected onto the image plane to obtain a two-dimensional box π(Bopt). The residual with the original two-dimensional detection box B2D is:

[0052] Shape constraint residuals:

[0053] in, and are the length and width of the kth 3D semantic box respectively, μ is the prior value of the aspect ratio, and μ is different for objects of different categories.

[0054] Ground constraint residuals :

[0055] in, is the height of the ground, is the z coordinate of the kth 3D semantic box, is the height of the kth 3D semantic box.

[0056] 204. Eliminate or retain unmatched 3D detection frames based on a preset confidence threshold; For each 3D detection frame that is not successfully matched, its confidence is compared with the preset confidence threshold; if the confidence of the 3D detection frame is lower than the confidence threshold, it is eliminated; if the confidence of the 3D detection frame is not lower than the confidence threshold, it is retained.

[0057] If the confidence s1 of the 3D detection frame is lower than the confidence threshold s2, it is considered as a misjudgment and is removed; otherwise, the detection frame is retained, that is, the retention condition is s1 ≥ s2.

[0058] like Figure 3 As shown, Figure 3 (a) indicates that the electric poles beside the road may have certain appearance features similar to pedestrians in visual presentation (marked by red ellipse). Figure 3In the point cloud detection information in (b), the point cloud-based detector indeed misdetects a utility pole as a pedestrian (marked by a red box). Because the semantic characteristics of a utility pole are fundamentally different from those of a pedestrian, and the utility pole does not match any 2D detection bounding boxes in the image, the confidence level of such misdetected objects in point cloud detection is typically lower than the preset confidence threshold. Filtering based on the confidence threshold can effectively reduce such misdetections of utility poles as pedestrians, improving detection accuracy.

[0059] 205. For the unmatched 2D detection frame, a new 3D detection frame is generated through monocular reconstruction and ground height adjustment; Using preset camera intrinsic parameters and monocular vision methods, the 3D spatial position and size of the unmatched 2D detection frame are estimated to obtain an initial 3D detection frame. The ground height information is obtained and the ground height of the initial 3D detection frame is determined based on the ground height information. The adjusted initial 3D detection frame is used as the new 3D detection result to supplement the 3D information corresponding to the unmatched 2D detection frame.

[0060] Due to the sparsity of LiDAR point clouds, missed 3D object detections are common, resulting in some 2D detection frames failing to find matching 3D detection frames. For these unmatched 2D detection frames, this step uses a monocular 3D reconstruction method to generate an initial 3D detection frame. This initial frame is then optimized using ground elevation information to ensure that the reconstructed 3D frame is consistent with the actual scene.

[0061] For monocular 3D reconstruction: For each unmatched 2D detection box, assume that it corresponds to a 3D target. Given the camera intrinsic parameter matrix K, the rotation matrix R of the target in the camera coordinate system and the positions of the three vanishing points (VP) satisfy the following relationship: The specific steps are to calculate the rotation matrix (R) for the point (VP):

[0062] in, is the pixel coordinate of the i-th vanishing point, is the i-th column vector of the rotation matrix R. By sampling a point on the upper edge of the two-dimensional detection frame as a vertex projection of the three-dimensional detection frame, the projection positions of the remaining vertices are deduced, and then the initial three-dimensional detection frame is generated.

[0063] Solve the PnP problem: Sample four vertices to solve the PnP (Perspective-n-Point) problem, obtain the rotation matrix R and translation vector t of the reconstructed 3D detection box to the camera, and optimize the pose of the initial 3D box by minimizing the reconstruction projection error:

[0064] wherein, is the i-th vertex pixel coordinate of the two-dimensional bounding box, is the corresponding three-dimensional space point, is the projection function.

[0065] Optimization of depth and z-axis coordinate: The depth estimation of monocular observation of a single frame is not accurate, which leads to possible deviation of the z-coordinate (height direction) of the three-dimensional bounding box. To solve this problem, the depth and z-coordinate are adjusted using the ground segmentation information.

[0066] The RANSAC algorithm is used to segment the ground from the original laser point cloud. The steps of the RANSAC algorithm are as follows: first, randomly sample K points from the original point cloud; then, fit a plane model to the K points; calculate the distance d of all points in the original point cloud to the plane, and set a threshold dthres, if d < dthres, mark the point as an inlier, otherwise mark it as an outlier; repeat the above steps until a certain number of iterations or a stop condition is met, select the model with the maximum number of inliers from all iterations; then estimate the model parameters using all inliers, and output the final ground model and the corresponding inliers, i.e. the ground points. Assuming that the number of iterations is M times, the probability of a ground point being selected from the original point cloud is p, and the probability of the fitted ground model not containing non-ground points is z, then:

[0067] Depth and height adjustment: Adjust the depth d of the reconstructed cuboid along the camera optical axis direction to make it enclose as many non-ground point clouds as possible.

[0068] According to the local ground height (the mean value of the ground point cloud in the neighborhood with a radius r), the height of the bottom of the cuboid is corrected :

[0069] where h is the height of the cuboid.

[0070] 206、Integrate the matching successful detection pair, the unmatching successful three-dimensional bounding box and the unmatching successful two-dimensional bounding box, and obtain the optimized three-dimensional detection result; 207、Combine the motion prediction information of the historical frame to perform trajectory consistency verification on the optimized three-dimensional detection result, and obtain the fused three-dimensional target detection result.

[0071] In the embodiment of the present application, the dynamic adaptive bidirectional matching strategy is used to comprehensively score match the target geometric consistency, category confidence and spatial distribution similarity, effectively solving the mis-matching problem caused by relying on single spatial overlap in the prior art, and the multi-residual optimization is used to improve the state estimation accuracy for the matched detection targets, while the confidence screening or monocular reconstruction is used for the unmatched targets, overcoming the performance degradation of the traditional method under the conditions of occlusion and missed detection, further combining the historical motion prediction for trajectory verification, enhancing the time sequence stability, thereby significantly improving the matching accuracy and detection robustness in complex scenes, reducing the missed detection and false detection caused by factors such as occlusion and deformation, and finally realizing more reliable three-dimensional target detection.

[0072] Please refer to Figure 4 Another embodiment of the target detection method based on multi-modal fusion in the embodiment of the present application includes: 401, obtaining a two-dimensional detection box and a three-dimensional detection box of a current frame; 402, data association is performed on the two-dimensional detection box and the three-dimensional detection box through a dynamic adaptive bidirectional matching strategy to obtain a matching result, and the matching strategy is based on target geometric consistency, category confidence and spatial distribution similarity to calculate a matching score; 403, multi-stage optimization processing is performed based on the matching result to obtain an optimized three-dimensional detection result; 404, the three-dimensional position and motion state information of the tracked targets in the historical frames are used to predict the expected position and motion trajectory of each target in the current frame through a motion model; The continuous three-dimensional position coordinates and motion speed information of the tracked targets are extracted from the historical frame data; a linear motion model or a nonlinear motion model is selected according to the motion characteristics of the target; the expected three-dimensional position and motion trajectory of each target in the current frame are recursively calculated through the selected motion model.

[0073] For each tracked target in the historical frame data, the three-dimensional position coordinate sequence of continuous multiple frames is extracted in turn and arranged in time sequence, the instantaneous speed between each frame is calculated through the position difference and time interval between adjacent frames, and then the motion speed sequence synchronized with the position sequence is generated; a model is selected based on the motion characteristics of the target, if the target speed fluctuation range in the historical frame is less than a preset threshold and the motion direction changes smoothly, it is determined as uniform linear motion and a linear motion model is selected, the state vector of this model includes three-dimensional position and speed components, the average speed is solved as the state transition parameter through the least square method fitting the historical position and speed data; if the target speed fluctuation is significant or the motion direction frequently changes, it is determined as non-uniform or curve motion and a constant turning rate and acceleration model is selected, the extended state vector of this model includes yaw angle, angular velocity and acceleration components, the angular velocity and acceleration parameters are estimated through the curvature change and speed trend fitting of the historical trajectory; When recursively calculating the expected three-dimensional position and motion trajectory of each target in the current frame for the selected motion model, if a linear motion model is used, the uniform propagation formula is used based on the three-dimensional position and velocity components contained in its state vector. Directly extrapolate the current frame position, where and are the position and velocity of the previous frame respectively, is the time interval between frames, and the velocity parameters are updated by the least square fitting result of historical data to ensure the smoothness of the trajectory; if the constant turning rate and acceleration model is adopted, the yaw angle, angular velocity and acceleration components in its extended state vector are used to calculate the velocity parameters through the discrete time state transfer equation Iteratively update the state, where is the state vector of the previous frame, F is the nonlinear transfer matrix constructed based on angular velocity and acceleration, The three-dimensional position component is extracted from the updated state vector as the expected position of the current frame, and a complete motion trajectory containing information such as position, velocity, and yaw angle is generated through the recursive results of multiple consecutive frames. At the same time, algorithms such as Kalman filtering or particle filtering are used to dynamically correct the uncertainty in the recursive process to improve the accuracy of trajectory prediction.

[0074] Existing technologies generally rely on a single fixed model, such as the uniform velocity CV model or the uniform acceleration CA model, or simple multi-model switching, such as the IMM algorithm, to predict target trajectories. These models are limited in their ability to dynamically adapt to the target's full range of motion patterns, from uniform velocity to complex maneuvers. Furthermore, model parameters often rely on prior assumptions or offline statistics, resulting in insufficient adaptability to dynamic scenarios such as sudden acceleration and steering. While some solutions incorporate deep learning to extract features, they suffer from issues such as decoupling features from motion modeling and poor real-time performance. This solution intelligently selects linear or nonlinear motion models by analyzing target velocity fluctuations and directional change rates, dynamically optimizing model parameters using historical data, and simultaneously integrating multi-dimensional information such as three-dimensional position, velocity, and yaw angle to generate a complete trajectory. This approach effectively overcomes the limitations of existing technologies, such as rigid models, static parameters, and single information, significantly improving trajectory prediction accuracy and robustness in complex dynamic scenarios.

[0075] 405. Compare the optimized three-dimensional detection result with the predicted expected position and motion trajectory, and calculate the deviation between the actual detection result and the predicted result; Calculate the Euclidean distance between the center point of the optimized 3D detection box and the expected position of the corresponding time point on the predicted trajectory; calculate the angular deviation between the movement direction of the optimized 3D detection box and the movement direction of the predicted trajectory; and generate a trajectory consistency score based on the Euclidean distance and the angular deviation as the deviation result.

[0076] The three-dimensional detection frame center point coordinates of the current frame target are extracted from the optimized three-dimensional detection result, and the expected position coordinates at the corresponding time point are obtained from the prediction trajectory. The position deviation quantitative value is obtained by calculating the Euclidean distance in the three-dimensional space of the two, that is, the sum of the squares of the differences in the three coordinate axis directions is calculated and then the square root is taken. The actual motion direction vector of the optimized three-dimensional detection frame is calculated according to the position change of the three-dimensional detection frame between continuous frames, and the expected motion direction vector of the corresponding time period is extracted from the continuous state of the prediction trajectory. The angle deviation of the two direction vectors is calculated by the vector dot product formula, that is, the ratio of the dot product of the two vectors to the product of their lengths is calculated, and then the inverse cosine function is taken to obtain the angle value. Finally, the Euclidean distance of the position deviation and the direction angle deviation are normalized, and different weight coefficients are assigned to reflect the influence degree of the trajectory consistency, the weighted position deviation and direction deviation are added to obtain the comprehensive deviation value, and then the comprehensive deviation value is converted into the trajectory consistency score through the preset score mapping function. The score range is usually set to 0 to 1, and the closer the score is to 1, the higher the consistency of the detection result and the prediction trajectory, which is used as the final deviation evaluation result.

[0077] The prior art generally only measures the deviation of the detection result and the prediction trajectory by a single position Euclidean distance, ignoring the key dynamic feature of the motion direction, resulting in an incomplete evaluation result, especially when the target makes a turning motion, which is prone to misjudgment. Although some schemes introduce direction information, they mostly use simple threshold comparison or non-normalized weighted sum, lacking coordinated quantization and unified score mapping of position and direction deviation. The present scheme synchronously calculates the Euclidean distance of the three-dimensional detection frame center point and the predicted position, and the angle deviation of the actual and expected motion direction, and then maps it to the trajectory consistency score of 0 to 1 after normalization and weighting, which not only overcomes the one-sidedness of single index evaluation, but also realizes the intuitive comparability of deviation results in different scenes through standardized scoring, significantly improving the accuracy and robustness of trajectory evaluation in complex dynamic scenes.

[0078] 406、According to the calculated deviation, the optimized three-dimensional detection result is adjusted to obtain a fused three-dimensional target detection result.

[0079] If the calculated deviation result is less than the preset threshold, the optimized three-dimensional detection result is used as the final output; if the calculated deviation result is not less than the preset threshold, the position and motion state parameters of the optimized three-dimensional detection frame are corrected according to the prediction trajectory to obtain a fused three-dimensional target detection result.

[0080] If the deviation result is less than the preset threshold, it indicates that the optimized three-dimensional detection result is highly consistent with the predicted trajectory, and at this time the center point coordinates, size parameters and motion state of the detection box are directly used as the final output; if the deviation result is not less than the preset threshold, a trajectory fusion correction process is started: the center point of the detection box is corrected by weighted translation according to the expected position of the predicted trajectory in the current frame, the weight is dynamically determined by the ratio of the deviation result and the threshold, at the same time the orientation angle of the detection box is adjusted in combination with the motion direction of the predicted trajectory, so that the deviation of the orientation angle with the trajectory direction is not more than the preset angle tolerance, and then the motion state of the detection box is corrected by smooth transition using the speed and acceleration parameters of the predicted trajectory, the detection speed and the predicted speed are fused by Kalman filtering algorithm or the like, and finally the fused three-dimensional target detection result is generated, so that the output result is better than the single detection or prediction result in terms of position accuracy and motion continuity.

[0081] In the embodiment of the application, in order to solve the problem of low matching accuracy caused by the prior art relying only on the intersection-over-union or fixed threshold for two-dimensional and three-dimensional detection box association, the dynamic adaptive bidirectional matching strategy is adopted, the matching score is calculated by comprehensively considering the target geometric consistency, class confidence and spatial distribution similarity and other multi-dimensional factors, the data association robustness in complex scenes is significantly improved, at the same time, the motion model based on historical trajectory is constructed to realize accurate trajectory prediction, and the joint scoring mechanism of Euclidean distance and direction angle deviation is designed to dynamically fuse the detection result and the prediction information, and finally the fusion result can still maintain high-precision positioning and motion continuity under complex interference, thereby effectively solving the technical problems of the traditional method that the optimization effect is limited in complex scenes and error matching is easy to occur.

[0082] The following is an experimental embodiment, the experiment proposes a target detection framework (GOOD) based on multi-modal fusion, the experiment verifies the proposed target detection method based on multi-modal fusion based on two automatic driving public data sets of nuScenes and KITTI. The experiment strictly follows the technical solution of the application, and focuses on verifying the effectiveness of the dynamic adaptive bidirectional matching strategy, the multi-stage optimization processing and the trajectory consistency verification three core modules.

[0083] In terms of experimental configuration, it includes: (1) The sensor configuration is as shown in Figure 5 and Figure 6 Figure 5 The nuScenes adopts a ring system of 1 laser radar + 6 cameras, Figure 6 The KITTI adopts 1 laser radar + 4 front-view cameras.

[0084] (2) The two-dimensional detection network adopts YOLOv5, and the three-dimensional detection network selects three mainstream architectures of PointPillars, SECOND and CenterPoint.​

[0085] (3) Evaluation metrics include: core metrics: mAP (mean Average Precision), NDS (nuScenes Detection Score); error metrics: mATE (mean Average Translation Error), mASE (mean Average Size Error), mAOE (mean Average Orientation Error), etc. For KITTI dataset, the evaluation is performed according to the difficulty level (easy / medium / difficult) respectively.

[0086] nuScenes: nuScenes provides a large laser point cloud sequence dataset, each frame is accompanied by semantic labels. The dataset contains 1000 sequences, of which 700 can be used for training, 150 for validation, and 150 for testing. It also annotates 10 classes that can be used for detection evaluation. Each sequence is about 20 seconds long, with a data sampling frequency of 10 Hz. In addition to mAP (mean Average Precision), nuScenes also uses NDS (nuScenes Detection Score) as the official metric for evaluating detector performance. The definition of NDS is as follows:

[0087] where, is the error subset including translation, size, orientation, attribute, and velocity. It is worth noting that the mAP in this dataset is calculated based on the overhead center distance of different thresholds {0.5 meters, 1 meter, 2 meters, 4 meters}, rather than the standard 3D box IoU.

[0088] KITTI: KITTI dataset contains 7481 frames of training samples and 7518 frames of test samples, each frame of sample is annotated with 3D and 2D information of cars, pedestrians and cyclists on the street. Each class is divided into three levels according to the size of the object, whether it is truncated or occluded: easy, medium and difficult. Different methods can be evaluated in this dataset using 3D IoU calculated AP (Average Precision). In this experiment, the original 7481 frames of training samples are divided into 3740 frames for training samples of learning-based methods, and 3741 frames for validation samples.

[0089] For 2D detection networks, we select YOLOv5l pre-trained model and train it on nuScenes train data set for 100 epochs. All 3D detection models are trained using 8 GTX 1080Ti GPUs. We use the laser point cloud based 3D detection network as the baseline for comparison, as shown in Table 1, which is a summary of the performance of the proposed method on the nuScenes trainval data set compared with each corresponding baseline method. We record the NDS, mAP, mATE (mean average location error), mASE (mean average size error), mAOE (mean average orientation error), mAVE (mean average velocity error), and mAAE (mean average attribute error), where↑indicates that the higher the better and↓indicates that the lower the better. As expected, GOOD outperforms PointPillars, SECOND, and CenterPoint by 6.2%, 5.0%, and 4.2% in NDS score, and by 9.1%, 6.4%, and 3.9% in mAP score, respectively. As for the error indicators, although GOOD performs slightly worse than SECOND in mAOE score (about 1.4%), this is because the 3D bounding boxes reconstructed by the add module may have incorrect orientations under single-frame constraints, but in comparison with all other baseline methods in error indicators, GOOD still performs better.

[0090] Table 1 Performance comparison of different laser 3D object detection baseline methods and GOOD on nuScenes trainval data set

[0091] Compared with the laser-camera fusion-based 3D object detection method, this paper compares the GOOD with the pre-fusion PointPainting and post-fusion CLOCs based on learning. Considering that both of them need a 3D detector to generate or provide a 3D detection box, PointPillars is selected as the 3D detector, and PointPainting with PointPillar is named as Painted PointPillars. In addition, YOLOv5 is selected as the 2D detector of CLOCs and GOOD. Given that the open-source CLOCs algorithm only supports the KITTI dataset, experiments are conducted on KITTI. In addition, considering that the KITTI data is not time-sequential, that is, the data before and after are not continuous, which means that the GOOD cannot apply the 3D Multiple Object Tracking (3D MOT) part. The IoU threshold of Car, Pedestrian and Cyclist is set to 0.7, 0.5 and 0.5 respectively. The results are summarized in Table 2, which shows that although the pre-fusion method PointPainting performs well in Pedestrian detection, because the point cloud details of small objects may be more supplemented by images, the accuracy of Car detection is lower compared with PointPillars based on pure laser point cloud due to the error caused by sensor calibration and synchronization. Compared with the post-fusion method CLOCs, since the occlusion and truncation are not serious, in this case, the optimization constraint brought by 2D detection is feasible and effective, so the GOOD performs better on Car in the easy part of the KITTI, with an improvement of nearly 1.5% in 3D AP. Since the shape of different Cyclist may change frequently on the moderate or hard part data, and the number of cyclists in the validation set is relatively small in general, the improvement is relatively small compared with CLOCs due to the introduction of shape residuals or several error cases. However, it is worth mentioning that the performance of the GOOD can be further improved with the help of 3DMOT, which can be evaluated in the following ablation study experiments.

[0092] Table 2 Performance comparison results of different 3D object detection methods on KITTI validation dataset

[0093] Ablation experiments: Considering that the nuScenes dataset is sampled in order, this paper can conduct a complete validation on each part of the GOOD on this basis.

[0094] Optimization module and 3D MOT module: First, evaluate the effectiveness of each module in the optimization and 3D MOT pipeline. The optimization part includes the refine module, keep or delete (KoD) module, and add module. The results are shown in Table 3, from which it can be concluded that the refine module contributes the most to reducing errors, as constraints from 2D detection can help correct the state of 3D detection, as shown in (a), (b), and (c) of Figure 7 Figure 7 As can be seen from (a), (b), and (c), in the original point cloud-based 3D detection, the red 3D box representing the truck has a certain angle between the red arrow indicating its orientation and the green arrow indicating the actual orientation of the truck point cloud, i.e., the detection box is slightly skewed compared to the actual situation, which can be adjusted by the constraints of the 2D detection box. The KoD module can help improve the mAP score by deleting mismatched 3D instances with low detection scores, as they may be false or redundant detections. In addition, with the help of the add module, a significant improvement in the mAP score can be observed. Figure 10 An example is provided to vividly illustrate the depth and z coordinate of the reconstruction cuboid adjustment algorithm proposed in this paper: the child and his mother in the figure are both detected by the 2D detector, as shown in (a); but the child is missed by the 3D detector, as shown in (b). In (c), single-frame image 3D target detection is performed on the unmatched 2D candidate box, but its estimated depth is not accurate, as shown in (d). In this paper, the depth is adjusted by using the depth and z coordinate adjustment algorithm, and the final result is shown in (e). In addition, the introduction of 3D MOT is also beneficial to improve the mAP score and reduce the error index. Figure 8 Figure 8 Figure 8 Figure 8 Figure 8

[0095] Table 3 Ablation experiment statistics on nuScenes dataset

[0096] Matching module: In addition, a control experiment is designed to prove the effectiveness of the matching method in this paper, as shown in Table 4. The 3D detector used is pointpillars, and the 2D detector is YOLOv5. The results show that, compared with single-side nearest neighbor matching, double-side nearest neighbor matching performs better in terms of NDS and mAP score.

[0097] Table 4 Performance comparison results of different 3D-2D matching methods

[0098] ​​​​​​The target detection method based on multi-modal fusion in the embodiments of the application is described above, and the target detection device based on multi-modal fusion in the embodiments of the application is described below. Please refer to Figure 9 One embodiment of the target detection device based on multi-modal fusion in the embodiments of the application includes: The acquisition module 901 is configured to acquire a two-dimensional detection box and a three-dimensional detection box of a current frame. The association module 902 is configured to perform data association on the two-dimensional detection box and the three-dimensional detection box by using a dynamic adaptive bidirectional matching strategy to obtain a matching result, and the matching strategy is based on target geometric consistency, category confidence, and spatial distribution similarity to calculate a matching score. The optimization module 903 is configured to perform multi-stage optimization processing based on the matching result to obtain an optimized three-dimensional detection result. The verification module 904 is configured to perform trajectory consistency verification on the optimized three-dimensional detection result in combination with motion prediction information of a historical frame to obtain a fused three-dimensional target detection result.

[0099] In the embodiments of the application, the matching score is calculated based on target geometric consistency, category confidence, and spatial distribution similarity, and multi-dimensional comprehensive consideration is performed for data association, so that the two-dimensional detection box and the three-dimensional detection box can be more accurately matched in a complex scene, the accuracy of data association is improved, multi-stage optimization processing is performed based on the matching result, compared with a conventional single optimization mode, interference factors in a complex scene can be more comprehensively processed, the optimization effect is improved, and at the same time, the trajectory consistency verification is performed on the optimized three-dimensional detection result in combination with the motion prediction information of the historical frame, the historical information is fully utilized, the target trajectory can be more accurately determined to be consistent or not, target motion state mutation or abnormal motion conditions are effectively coped with, and therefore a more accurate and reliable fused three-dimensional target detection result is obtained.

[0100] Please refer to Figure 10 Another embodiment of the target detection device based on multi-modal fusion in the embodiments of the application includes: The acquisition module 901 is configured to acquire a two-dimensional detection box and a three-dimensional detection box of a current frame. The association module 902 is configured to perform data association on the two-dimensional detection box and the three-dimensional detection box by using a dynamic adaptive bidirectional matching strategy to obtain a matching result, and the matching strategy is based on target geometric consistency, category confidence, and spatial distribution similarity to calculate a matching score. The optimization module 903 is configured to perform multi-stage optimization processing based on the matching result to obtain an optimized three-dimensional detection result. The verification module 904 is configured to perform trajectory consistency verification on the optimized three-dimensional detection result in combination with motion prediction information of a historical frame to obtain a fused three-dimensional target detection result.

[0101] Optionally, the association module 902 can be specifically used for: extracting geometric features of the two-dimensional detection box and the three-dimensional detection box, calculating a geometric consistency score; calculating a category confidence score according to the category prediction results of the two-dimensional detection box and the three-dimensional detection box; analyzing the distribution of the two-dimensional detection box and the three-dimensional detection box in the image space, calculating a spatial distribution similarity score; calculating a matching score based on the geometric consistency score, the category confidence score and the spatial distribution similarity score, and determining a matching result according to the matching score.

[0102] Optionally, the optimization module 903 includes: an optimization unit 9031, configured to perform state optimization on the matched detection pair by minimizing the IoU residual, the shape residual and the ground constraint residual; a processing unit 9032, configured to perform elimination or reservation on the three-dimensional detection box that fails to match based on a preset confidence threshold; a generation unit 9033, configured to generate a new three-dimensional detection box for the two-dimensional detection box that fails to match by monocular reconstruction and ground height adjustment.

[0103] Optionally, the optimization unit 9031 can be specifically used for: calculating an intersection-over-union residual between the matched two-dimensional detection box and the three-dimensional detection box, and minimizing the intersection-over-union residual to adjust the position and size of the three-dimensional detection box; analyzing the shape features of the three-dimensional detection box, calculating a shape residual, and minimizing the shape residual to optimize the shape of the three-dimensional detection box; calculating and optimizing a ground constraint residual through the spatial constraint relationship between the three-dimensional detection box and the ground, to ensure that the three-dimensional detection box meets the geometric conditions of ground fitting.

[0104] Optionally, the processing unit 9032 can be specifically used for: for each three-dimensional detection box that fails to match, comparing its confidence with a preset confidence threshold; if the confidence of the three-dimensional detection box is lower than the confidence threshold, eliminating it; if the confidence of the three-dimensional detection box is not lower than the confidence threshold, reserving it.

[0105] Optionally, the generation unit 9033 can be specifically used for: using a preset camera intrinsic parameter and a monocular vision method to estimate the three-dimensional spatial position and size of the two-dimensional detection box that fails to match, to obtain an initial three-dimensional detection box; obtaining ground height information, determining the ground height of the position where the initial three-dimensional detection box is located based on the ground height information; taking the adjusted initial three-dimensional detection box as a new three-dimensional detection result, to supplement the three-dimensional information corresponding to the two-dimensional detection box that fails to match.

[0106] Optionally, the verification module 904 can be specifically used for: The prediction unit 9041 is configured to predict the expected position and motion trajectory of each target in the current frame by using the three-dimensional position and motion state information of the tracked target in the historical frame and a motion model. The calculation unit 9042 is configured to compare the optimized three-dimensional detection result with the predicted expected position and motion trajectory, and calculate the deviation between the actual detection result and the predicted result. The adjustment unit 9043 is configured to adjust the optimized three-dimensional detection result according to the calculated deviation, and obtain a three-dimensional target detection result fused with the current detection and motion prediction.

[0107] Optionally, the prediction unit 9041 can be specifically configured to: extract the continuous three-dimensional position coordinates and motion velocity information of the tracked target from the historical frame data, select a linear motion model or a nonlinear motion model according to the motion characteristics of the target, and recursively calculate the expected three-dimensional position and motion trajectory of each target in the current frame by using the selected motion model.

[0108] Optionally, the calculation unit 9042 can be specifically configured to: calculate the Euclidean distance between the center point of the optimized three-dimensional detection box and the expected position at the corresponding time point on the predicted trajectory, calculate the angle deviation between the motion direction of the optimized three-dimensional detection box and the motion direction of the predicted trajectory, and generate a trajectory consistency score based on the Euclidean distance and the angle deviation as the deviation result.

[0109] Optionally, the adjustment unit 9043 can be specifically configured to: If the calculated deviation result is less than a preset threshold, the optimized three-dimensional detection result is used as the final output; if the calculated deviation result is not less than the preset threshold, the position and motion state parameters of the optimized three-dimensional detection box are corrected according to the predicted trajectory, and a fused three-dimensional target detection result is obtained.

[0110] In the embodiment of the application, the two-dimensional detection box and the three-dimensional detection box of the current frame are obtained by a dynamic adaptive bidirectional matching strategy, the data of the two-dimensional detection box and the three-dimensional detection box are associated, a matching result is obtained, the matching strategy calculates a matching score based on target geometric consistency, category confidence and spatial distribution similarity, for the matched detection pairs, state optimization is performed by minimizing the IoU residual, shape residual and ground constraint residual, for the three-dimensional detection boxes that are not matched successfully, elimination or reservation is performed based on a preset confidence threshold, for the two-dimensional detection boxes that are not matched successfully, new three-dimensional detection boxes are generated by monocular reconstruction and ground height adjustment, the optimized three-dimensional detection result is verified for trajectory consistency in combination with the motion prediction information of the historical frame, and a fused three-dimensional target detection result is obtained, which effectively solves the problems of occlusion, false detection and motion mutation in a complex scene, and thus improves the accuracy of the three-dimensional target detection result.

[0111] The above Figure 9 and Figure 10 The multi-modal fusion based target detection device in the embodiment of the application is described in detail from the perspective of a modular functional entity, and the electronic device in the embodiment of the application is described in detail from the perspective of hardware processing.

[0112] Referring to Figure 11 The electronic device includes a processor 1100 and a memory 1101, and the memory 1101 stores machine executable instructions that can be executed by the processor 1100, and the processor 1100 executes the machine executable instructions to implement the multi-modal fusion based target detection method described above.

[0113] Further, Figure 11 The electronic device shown in the figure further includes a bus 1102 and a communication interface 1103, and the processor 1100, the communication interface 1103 and the memory 1101 are connected through the bus 1102.

[0114] The memory 1101 can include a high-speed random access memory (RAM), and can also include a non-volatile memory, for example, at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 1103 (which can be wired or wireless), and the Internet, a wide area network, a local area network, a metropolitan area network, etc. can be used. The bus 1102 can be an ISA bus, a PCI bus or an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 11 In the figure, only one bidirectional arrow is used to represent the bus, but it does not mean that there is only one bus or only one type of bus.

[0115] The processor 1100 can be an integrated circuit chip having a processing capability of signals. In the implementation process, each step of the above method can be completed by the integrated logic circuit of hardware in the processor 1100 or the instruction in the form of software. The processor 1100 described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. Each method, step and logic block diagram disclosed in the embodiments of the present disclosure can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present disclosure can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read only memory, a programmable read only memory or an electrically erasable programmable memory, a register, etc. The storage medium in the art. The storage medium is located in the memory 1101, and the processor 1100 reads the information in the memory 1101, and combines the hardware to complete the method steps of the above embodiments.

[0116] The present application also provides an electronic device, the computer device includes a memory and a processor, the memory has computer readable instructions stored therein, the computer readable instructions are executed by the processor, so that the processor executes the steps of the target detection method based on multi-modal fusion in each of the above embodiments. The present application also provides a computer readable storage medium, which can be a non-volatile computer readable storage medium, and can also be a volatile computer readable storage medium, the computer readable storage medium has instructions stored therein, when the instructions run on the computer, the computer executes the steps of the target detection method based on multi-modal fusion.

[0117] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above described system, device and unit can refer to the corresponding process in the foregoing method embodiment, which will not be described here.

[0118] The integrated unit, if in the form of a software function unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on such an understanding, the technical solutions of the present application, essentially or in the part that contributes to the prior art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and various other media that can store program codes.

Claims

1. A target detection method based on multimodal fusion, characterized in that: The target detection method based on multimodal fusion includes: Get the 2D and 3D detection frames of the current frame; Performing data association on the two-dimensional detection frame and the three-dimensional detection frame using a dynamic adaptive bidirectional matching strategy to obtain a matching result, wherein the matching strategy calculates a matching score based on target geometric consistency, category confidence, and spatial distribution similarity; Perform multi-stage optimization processing based on the matching results to obtain optimized three-dimensional detection results; Combined with the motion prediction information of the historical frames, the trajectory consistency check is performed on the optimized three-dimensional detection result to obtain a fused three-dimensional target detection result.

2. The target detection method based on multimodal fusion according to claim 1, characterized in that: The step of performing data association on the two-dimensional detection frame and the three-dimensional detection frame by using a dynamic adaptive bidirectional matching strategy to obtain a matching result includes: Extracting geometric features of the two-dimensional detection frame and the three-dimensional detection frame, and calculating a geometric consistency score; Calculating a category confidence score based on the category prediction results of the two-dimensional detection frame and the three-dimensional detection frame; Analyzing the distribution of the two-dimensional detection frame and the three-dimensional detection frame in the image space, and calculating a spatial distribution similarity score; A matching score is calculated based on the geometric consistency score, the category confidence score, and the spatial distribution similarity, and a matching result is determined according to the matching score.

3. The target detection method based on multimodal fusion according to claim 1, characterized in that: The multi-stage optimization process based on the matching result includes: For successfully matched detection pairs, the state is optimized by minimizing the IoU residual, shape residual, and ground constraint residual; For the 3D detection frames that are not successfully matched, they are removed or retained based on the preset confidence threshold; For unmatched 2D detection frames, new 3D detection frames are generated through monocular reconstruction and ground height adjustment.

4. The target detection method based on multimodal fusion according to claim 3, characterized in that: The successfully matched detection pairs are optimized by minimizing the IoU residual, shape residual, and ground constraint residual, including: Calculating the IoU residual between the successfully matched 2D detection frame and the 3D detection frame, and minimizing the IoU residual to adjust the position and size of the 3D detection frame; Analyzing shape features of the three-dimensional detection frame, calculating shape residuals, and minimizing the shape residuals to optimize the shape of the three-dimensional detection frame; Through the spatial constraint relationship between the 3D detection frame and the ground, the ground constraint residual is calculated and optimized to ensure that the 3D detection frame meets the geometric conditions of ground fit.

5. The target detection method based on multimodal fusion according to claim 3, characterized in that: The unmatched 3D detection frames are removed or retained based on a preset confidence threshold, including: For each 3D detection box that is not successfully matched, its confidence is compared with the preset confidence threshold; If the confidence level of the 3D detection frame is lower than the confidence threshold, it will be removed; If the confidence of the 3D detection frame is not lower than the confidence threshold, it is retained.

6. The target detection method based on multimodal fusion according to claim 3, characterized in that: The method of generating a new 3D detection frame for the unmatched 2D detection frame through monocular reconstruction and ground height adjustment includes: Using the preset camera intrinsic parameters and monocular vision method, the 3D spatial position and size of the unmatched 2D detection frame are estimated to obtain the initial 3D detection frame; Acquiring ground height information, and determining the ground height of the location of the initial three-dimensional detection frame based on the ground height information; The adjusted initial 3D detection frame is used as the new 3D detection result to supplement the 3D information corresponding to the unmatched 2D detection frame.

7. The target detection method based on multimodal fusion according to claim 1, characterized in that: The process of combining the motion prediction information of the historical frames and performing trajectory consistency check on the optimized three-dimensional detection results to obtain the fused three-dimensional target detection results includes: Using the three-dimensional position and motion state information of the tracked targets in the historical frames, the expected position and motion trajectory of each target in the current frame are predicted through the motion model; Comparing the optimized three-dimensional detection results with the predicted expected position and motion trajectory, and calculating the deviation between the actual detection results and the predicted results; The optimized three-dimensional detection result is adjusted according to the calculated deviation to obtain a fused three-dimensional target detection result.

8. The target detection method based on multimodal fusion according to claim 7, characterized in that: The method of using the three-dimensional position and motion state information of the tracked targets in the historical frames to predict the expected position and motion trajectory of each target in the current frame through a motion model includes: Extract the continuous three-dimensional position coordinates and motion speed information of the tracked target from the historical frame data; Select a linear motion model or a nonlinear motion model according to the target motion characteristics; The expected three-dimensional position and motion trajectory of each target in the current frame are recursively calculated using the selected motion model.

9. The target detection method based on multimodal fusion according to claim 7, characterized in that: The step of comparing the optimized three-dimensional detection result with the predicted expected position and motion trajectory and calculating the deviation between the actual detection result and the predicted result includes: Calculate the Euclidean distance between the center point of the optimized 3D detection box and the expected position of the corresponding time point on the predicted trajectory; Calculate the angle deviation between the movement direction of the optimized 3D detection frame and the movement direction of the predicted trajectory; A trajectory consistency score is generated based on the Euclidean distance and the angle deviation as a deviation result.

10. The target detection method based on multimodal fusion according to claim 7, characterized in that: The step of adjusting the optimized three-dimensional detection result according to the calculated deviation to obtain a three-dimensional target detection result that integrates the current detection and motion prediction includes: If the calculated deviation result is less than the preset threshold, the optimized 3D detection result is used as the final output; If the calculated deviation result is not less than the preset threshold, the position and motion state parameters of the optimized 3D detection frame are corrected according to the predicted trajectory to obtain the fused 3D target detection result.

11. A target detection device based on multimodal fusion, characterized in that: The target detection device based on multimodal fusion includes: The acquisition module is used to obtain the two-dimensional detection frame and the three-dimensional detection frame of the current frame; an association module, configured to associate data between the two-dimensional detection frame and the three-dimensional detection frame using a dynamic adaptive bidirectional matching strategy to obtain a matching result, wherein the matching strategy calculates a matching score based on target geometric consistency, category confidence, and spatial distribution similarity; An optimization module, configured to perform multi-stage optimization processing based on the matching results to obtain optimized three-dimensional detection results; The verification module is used to perform trajectory consistency verification on the optimized three-dimensional detection result in combination with the motion prediction information of the historical frame to obtain a fused three-dimensional target detection result.

12. An electronic device, characterized in that: The electronic device comprises: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instructions in the memory so that the electronic device executes the target detection method based on multimodal fusion as described in any one of claims 1 to 10.

13. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the target detection method based on multimodal fusion according to any one of claims 1 to 10 is implemented.

Images