Target detection method and device based on multi-modal fusion, equipment and storage medium

Abstract

The present application relates to the technical field of automatic driving environment perception, and discloses a target detection method and device based on multi-modal fusion, equipment and storage medium, which are used for improving the accuracy of three-dimensional target detection. The target detection method based on multi-modal fusion comprises the following steps: obtaining a two-dimensional detection box and a three-dimensional detection box of a current frame; performing data association on the two-dimensional detection box and the three-dimensional detection box through a dynamic adaptive bidirectional matching strategy to obtain a matching result, wherein the matching strategy is based on target geometric consistency, category confidence and spatial distribution similarity to calculate a matching score; performing multi-stage optimization processing based on the matching result to obtain an optimized three-dimensional detection result; and performing trajectory consistency verification on the optimized three-dimensional detection result in combination with motion prediction information of a historical frame to obtain a fused three-dimensional target detection result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving environment perception, and particularly relates to a target detection method and device based on multi-modal fusion, equipment and a storage medium. BACKGROUND

[0002] The prior art usually first obtains a two-dimensional detection box and a three-dimensional detection box of a current frame, and then adopts a simple data association method based on an intersection over union (IoU) or a fixed threshold to associate the two-dimensional detection box and the three-dimensional detection box, and only determines whether the two are matched according to the overlapping degree of the detection boxes in space, and then performs a conventional optimization process, such as a non-maximum suppression operation, to reduce repeated detection, so as to obtain a three-dimensional target detection result.

[0003] The data association method based on the intersection over union or the fixed threshold is too simple and does not consider multi-dimensional factors, resulting in low accuracy of data association in complex scenes, such as target occlusion and large shape changes, and easy generation of false matches. The conventional optimization processing method is relatively single, and it is difficult to effectively handle various interference factors in complex scenes, and the optimization effect is limited, thereby affecting the accuracy of the final three-dimensional target detection result. SUMMARY

[0004] The present application provides a target detection method and device based on multi-modal fusion, equipment and a storage medium to solve the problem that the three-dimensional target detection result is not accurate enough due to the single detection scheme in the prior art.

[0005] The first aspect of the present application provides a target detection method based on multi-modal fusion, comprising: obtaining a two-dimensional detection box and a three-dimensional detection box of a current frame; performing data association on the two-dimensional detection box and the three-dimensional detection box through a dynamic self-adaptive bidirectional matching strategy to obtain a matching result, wherein the matching strategy calculates a matching score based on target geometric consistency, class confidence and spatial distribution similarity; performing multi-stage optimization processing based on the matching result to obtain an optimized three-dimensional detection result; and performing trajectory consistency verification on the optimized three-dimensional detection result in combination with motion prediction information of a historical frame to obtain a fused three-dimensional target detection result.

[0006] In an implementable embodiment, the data association of the two-dimensional detection box and the three-dimensional detection box through the dynamic adaptive bidirectional matching strategy to obtain a matching result comprises: extracting geometric features of the two-dimensional detection box and the three-dimensional detection box, and calculating a geometric consistency score; calculating a category confidence score according to category prediction results of the two-dimensional detection box and the three-dimensional detection box; analyzing the distribution of the two-dimensional detection box and the three-dimensional detection box in the image space, and calculating a spatial distribution similarity score; calculating a matching score based on the geometric consistency score, the category confidence score and the spatial distribution similarity score, and determining a matching result according to the matching score.

[0007] In an implementable embodiment, the multi-stage optimization processing based on the matching result comprises: for a matched detection pair, state optimization is performed by minimizing an IoU residual, a shape residual and a ground constraint residual; for a three-dimensional detection box that fails to match, elimination or retention is performed based on a preset confidence threshold; for a two-dimensional detection box that fails to match, a new three-dimensional detection box is generated through monocular reconstruction and ground height adjustment.

[0008] In an implementable embodiment, the state optimization of the matched detection pair by minimizing the IoU residual, the shape residual and the ground constraint residual comprises: calculating an intersection over union residual between the matched two-dimensional detection box and the three-dimensional detection box, and minimizing the intersection over union residual to adjust the position and size of the three-dimensional detection box; analyzing the shape features of the three-dimensional detection box, calculating a shape residual, and minimizing the shape residual to optimize the shape of the three-dimensional detection box; calculating and optimizing a ground constraint residual through the spatial constraint relationship between the three-dimensional detection box and the ground to ensure that the three-dimensional detection box meets the geometric conditions of ground fitting.

[0009] In an implementable embodiment, the elimination or retention of the three-dimensional detection box that fails to match based on the preset confidence threshold comprises: for each three-dimensional detection box that fails to match, comparing its confidence with the preset confidence threshold; if the confidence of the three-dimensional detection box is lower than the confidence threshold, eliminating it; if the confidence of the three-dimensional detection box is not lower than the confidence threshold, retaining it.

[0010] In an implementable embodiment, the generation of a new three-dimensional detection box through monocular reconstruction and ground height adjustment for the two-dimensional detection box that fails to match comprises: estimating the three-dimensional spatial position and size of the two-dimensional detection box that fails to match by using a preset camera intrinsic parameter and a monocular vision method to obtain an initial three-dimensional detection box; obtaining ground height information, and determining the ground height at the position of the initial three-dimensional detection box based on the ground height information; and taking the adjusted initial three-dimensional detection box as a new three-dimensional detection result to supplement the three-dimensional information corresponding to the two-dimensional detection box that fails to match.

[0011] In an implementation, the motion prediction information of the historical frames is combined to perform trajectory consistency verification on the optimized three-dimensional detection result, to obtain a fused three-dimensional target detection result, including: using the three-dimensional position and motion state information of the tracked targets in the historical frames, an expected position and motion trajectory of each target in the current frame is predicted through a motion model; the optimized three-dimensional detection result is compared with the predicted expected position and motion trajectory, a deviation between the actual detection result and the predicted result is calculated; the optimized three-dimensional detection result is adjusted according to the calculated deviation, to obtain a three-dimensional target detection result fused with current detection and motion prediction.

[0012] In an implementation, the motion prediction information of the historical frames is combined to perform trajectory consistency verification on the optimized three-dimensional detection result, to obtain a fused three-dimensional target detection result, including: using the three-dimensional position and motion state information of the tracked targets in the historical frames, an expected position and motion trajectory of each target in the current frame is predicted through a motion model; the optimized three-dimensional detection result is compared with the predicted expected position and motion trajectory, a deviation between the actual detection result and the predicted result is calculated; the optimized three-dimensional detection result is adjusted according to the calculated deviation, to obtain a three-dimensional target detection result fused with current detection and motion prediction.

[0013] In an implementation, the motion prediction information of the historical frames is combined to perform trajectory consistency verification on the optimized three-dimensional detection result, to obtain a fused three-dimensional target detection result, including: using the three-dimensional position and motion state information of the tracked targets in the historical frames, an expected position and motion trajectory of each target in the current frame is predicted through a motion model; the optimized three-dimensional detection result is compared with the predicted expected position and motion trajectory, a deviation between the actual detection result and the predicted result is calculated; the optimized three-dimensional detection result is adjusted according to the calculated deviation, to obtain a three-dimensional target detection result fused with current detection and motion prediction.

[0014] In an implementation, the motion prediction information of the historical frames is combined to perform trajectory consistency verification on the optimized three-dimensional detection result, to obtain a fused three-dimensional target detection result, including: using the three-dimensional position and motion state information of the tracked targets in the historical frames, an expected position and motion trajectory of each target in the current frame is predicted through a motion model; the optimized three-dimensional detection result is compared with the predicted expected position and motion trajectory, a deviation between the actual detection result and the predicted result is calculated; the optimized three-dimensional detection result is adjusted according to the calculated deviation, to obtain a three-dimensional target detection result fused with current detection and motion prediction.

[0015] In a second aspect, the present application provides a target detection device based on multi-modal fusion, comprising: an acquisition module configured to acquire a two-dimensional detection box and a three-dimensional detection box of a current frame; an association module configured to perform data association on the two-dimensional detection box and the three-dimensional detection box through a dynamic adaptive bidirectional matching strategy to obtain a matching result, wherein the matching strategy is based on target geometric consistency, category confidence and spatial distribution similarity to calculate a matching score; an optimization module configured to perform multi-stage optimization processing based on the matching result to obtain an optimized three-dimensional detection result; and a verification module configured to perform trajectory consistency verification on the optimized three-dimensional detection result in combination with motion prediction information of a historical frame to obtain a fused three-dimensional target detection result.

[0016] In an implementable embodiment, the association module is specifically configured to: extract geometric features of the two-dimensional detection box and the three-dimensional detection box to calculate a geometric consistency score; calculate a category confidence score according to category prediction results of the two-dimensional detection box and the three-dimensional detection box; analyze the distribution of the two-dimensional detection box and the three-dimensional detection box in image space to calculate a spatial distribution similarity score; calculate a matching score based on the geometric consistency score, the category confidence score and the spatial distribution similarity score, and determine a matching result according to the matching score.

[0017] In an implementable embodiment, the optimization module comprises: an optimization unit configured to perform state optimization on a matched detection pair by minimizing an IoU residual, a shape residual and a ground constraint residual; a processing unit configured to perform elimination or reservation on a three-dimensional detection box that fails to match based on a preset confidence threshold; and a generation unit configured to generate a new three-dimensional detection box through monocular reconstruction and ground height adjustment for a two-dimensional detection box that fails to match.

[0018] In an implementable embodiment, the optimization unit is specifically configured to: calculate an intersection-over-union residual between a matched two-dimensional detection box and a three-dimensional detection box, and minimize the intersection-over-union residual to adjust the position and size of the three-dimensional detection box; analyze shape features of the three-dimensional detection box, calculate a shape residual, and minimize the shape residual to optimize the shape of the three-dimensional detection box; calculate and optimize a ground constraint residual through a spatial constraint relationship between the three-dimensional detection box and the ground to ensure that the three-dimensional detection box meets the geometric conditions of ground fitting.

[0019] In an implementable embodiment, the processing unit is specifically configured to: compare the confidence of each three-dimensional detection box that fails to match with a preset confidence threshold; if the confidence of the three-dimensional detection box is lower than the confidence threshold, eliminate the three-dimensional detection box; and if the confidence of the three-dimensional detection box is not lower than the confidence threshold, retain the three-dimensional detection box.

[0020] In an implementable embodiment, the generating unit is specifically configured to: estimate the three-dimensional spatial position and size of the two-dimensional detection frame that fails to be matched successfully by using preset camera internal parameters and a monocular vision method, to obtain an initial three-dimensional detection frame; obtain ground height information, determine the ground height of the position where the initial three-dimensional detection frame is located based on the ground height information; and take the adjusted initial three-dimensional detection frame as a new three-dimensional detection result, to supplement the three-dimensional information corresponding to the two-dimensional detection frame that fails to be matched successfully.

[0021] In an implementable embodiment, the verifying module comprises: a predicting unit configured to predict the expected position and motion trajectory of each target in the current frame by a motion model, by using the three-dimensional position and motion state information of the tracked targets in the historical frames; a calculating unit configured to compare the optimized three-dimensional detection result with the predicted expected position and motion trajectory, and calculate the deviation between the actual detection result and the predicted result; and an adjusting unit configured to adjust the optimized three-dimensional detection result according to the calculated deviation, to obtain the three-dimensional target detection result that fuses the current detection and motion prediction.

[0022] In an implementable embodiment, the predicting unit is specifically configured to: extract the continuous three-dimensional position coordinates and motion velocity information of the tracked targets from the historical frame data; select a linear motion model or a nonlinear motion model according to the motion characteristics of the targets; and recursively calculate the expected three-dimensional position and motion trajectory of each target in the current frame by the selected motion model.

[0023] In an implementable embodiment, the calculating unit is specifically configured to: calculate the Euclidean distance between the center point of the optimized three-dimensional detection frame and the expected position at the corresponding time point on the predicted trajectory; calculate the included angle deviation between the motion direction of the optimized three-dimensional detection frame and the motion direction of the predicted trajectory; and generate a trajectory consistency score based on the Euclidean distance and the included angle deviation, as the deviation result.

[0024] In an implementable embodiment, the adjusting unit is specifically configured to: if the calculated deviation result is less than a preset threshold, adopt the optimized three-dimensional detection result as the final output; and if the calculated deviation result is not less than the preset threshold, correct the position and motion state parameters of the optimized three-dimensional detection frame according to the predicted trajectory, to obtain the fused three-dimensional target detection result.

[0025] The third aspect of the present application provides an electronic device, comprising: a memory and at least one processor, the memory having instructions stored therein; the at least one processor invoking the instructions in the memory to cause the electronic device to perform the target detection method based on multi-modal fusion described above.

[0026] The fourth aspect of the present application provides a computer readable storage medium, wherein instructions are stored in the computer readable storage medium, and when the instructions are executed on a computer, the computer is caused to execute the target detection method based on multi-modal fusion.

[0027] In the technical solution provided by the present application, the two-dimensional detection box and the three-dimensional detection box of the current frame are obtained; the data correlation of the two-dimensional detection box and the three-dimensional detection box is performed through a dynamic adaptive bidirectional matching strategy to obtain a matching result, and the matching strategy calculates a matching score based on target geometric consistency, category confidence and spatial distribution similarity; multi-stage optimization processing is performed based on the matching result to obtain an optimized three-dimensional detection result; and trajectory consistency verification is performed on the optimized three-dimensional detection result in combination with motion prediction information of a historical frame to obtain a fused three-dimensional target detection result. In the embodiment of the present application, the matching score is calculated based on target geometric consistency, category confidence and spatial distribution similarity, and multi-dimensional comprehensive consideration is performed for data correlation, so that the two-dimensional detection box and the three-dimensional detection box can be more accurately matched in a complex scene, the accuracy of data correlation is improved, multi-stage optimization processing is performed based on the matching result, compared with a conventional single optimization mode, interference factors in a complex scene can be more comprehensively processed, the optimization effect is improved, and at the same time, the trajectory consistency verification is performed on the optimized three-dimensional detection result in combination with the motion prediction information of the historical frame, the historical information is fully utilized, the target trajectory can be more accurately judged to be consistent or not, the target motion state mutation or abnormal motion condition is effectively coped with, and therefore a more accurate and reliable fused three-dimensional target detection result is obtained. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 An embodiment schematic diagram of the target detection method based on multi-modal fusion in the embodiment of the present application;

[0029] Figure 2 Another embodiment schematic diagram of the target detection method based on multi-modal fusion in the embodiment of the present application;

[0030] Figure 3 An embodiment schematic diagram of point cloud detector false detection in the embodiment of the present application;

[0031] Figure 4 Another embodiment schematic diagram of the target detection method based on multi-modal fusion in the embodiment of the present application;

[0032] Figure 5 A nuScenes data set sensor hardware configuration diagram in the experimental embodiment of the present application;

[0033] Figure 6 A KITTI data set sensor hardware configuration diagram in the experimental embodiment of the present application;

[0034] Figure 7 For the original 3D detection and the refined 3D detection result in the experimental example of the present application;

[0035] Figure 8 For a schematic diagram of the depth and z-axis coordinate adjustment algorithm of the reconstructed detection frame in the experimental example of the present application;

[0036] Figure 9 For a schematic diagram of one embodiment of the target detection device based on multi-modal fusion in the embodiment of the present application;

[0037] Figure 10 For another embodiment of the target detection device based on multi-modal fusion in the embodiment of the present application;

[0038] Figure 11 For a schematic diagram of one embodiment of the electronic device in the embodiment of the present application. DETAILED DESCRIPTION

[0039] The embodiment of the present application provides a target detection method, device and equipment based on multi-modal fusion, and improves the accuracy of three-dimensional target detection.

[0040] The terms "first", "second", "third", "fourth" and the like (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the term "comprising" or "having" and any variation thereof is intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0041] It can be understood that the execution subject of the present application can be a target detection device based on multi-modal fusion, and can also be a terminal or a server, and the specific embodiments are not limited herein. The embodiment of the present application takes the server as the execution subject for example.

[0042] For the convenience of understanding, the specific process of the embodiment of the present application is described below, please refer to Figure 1 The embodiment of the target detection method based on multi-modal fusion in the embodiment of the present application includes:

[0043] 101、obtain the two-dimensional detection frame and the three-dimensional detection frame of the current frame;

[0044] The current frame image is processed by a 2D detection network to generate a two-dimensional detection box in the image plane, containing the pixel coordinates and detection score of each detection box; the current frame laser point cloud is processed by a 3D detection network to generate a three-dimensional detection box in the laser radar coordinate system, containing the three-dimensional position, size, yaw angle and confidence score of each detection box, the 2D detection network can be YOLOv5 network, etc., and the 3D detection network can be PointPillars, SECOND or CenterPoint.

[0045] 102, data association is performed on the two-dimensional detection box and the three-dimensional detection box by a dynamic adaptive bidirectional matching strategy to obtain a matching result, and the matching strategy calculates a matching score based on target geometric consistency, class confidence and spatial distribution similarity;

[0046] The geometric consistency is evaluated by comparing the similarity of the shape and size of the two-dimensional detection box and the three-dimensional detection box; the class confidence is based on the prediction results and confidence scores of the target class by the two-dimensional detector and the three-dimensional detector; and the spatial distribution similarity analyzes the relative position relationship of the detection box in the image space and the point cloud space; the scores of the three factors are summed by weighting to obtain the comprehensive matching score between each pair of two-dimensional detection box and three-dimensional detection box; according to the matching score, the matched detection pairs, the three-dimensional detection boxes and the two-dimensional detection boxes that are not matched are divided.

[0047] 103, based on the matching result, a multi-stage optimization process is performed to obtain an optimized three-dimensional detection result;

[0048] For the matched two-dimensional and three-dimensional detection box pairs, the optimization process is realized by simultaneously minimizing the three-dimensional IoU residual and the two-dimensional IoU residual: the three-dimensional IoU residual measures the spatial overlap difference between the three-dimensional detection box and the three-dimensional box restored by inverse projection of the two-dimensional box projected by the three-dimensional box in the image plane, and the two-dimensional IoU residual measures the plane overlap difference between the original two-dimensional detection box and the two-dimensional box projected by the three-dimensional box; a shape constraint residual is also introduced to adjust the size parameters of the three-dimensional box using the aspect ratio information of the target class prior to ensure that its geometric shape conforms to the actual object characteristics; a ground constraint residual is obtained by fitting the ground point cloud by RANSAC algorithm to force the bottom of the three-dimensional box to conform to the ground and eliminate the vertical positioning deviation. The residual terms constitute a nonlinear optimization objective function, which is solved iteratively by using a Ceres solver to correct the center position, size and yaw angle of the three-dimensional box.

[0049] For the three-dimensional detection boxes that are not matched, if the confidence score is lower than a preset confidence threshold, they are directly removed, otherwise they are retained; for the two-dimensional detection boxes that are not matched, after generating an initial three-dimensional box by monocular reconstruction, a ground height adjustment algorithm is used to optimize the depth and z-axis coordinates to ensure that the bottom of the box conforms to the ground.

[0050] The three-dimensional detection result optimized through multiple stages is obtained by integrating the matching success detection pair, the optimized processing result of the three-dimensional detection frame which is not matched successfully and the two-dimensional detection frame.

[0051] 104. The three-dimensional target detection result after fusion is obtained by performing trajectory consistency verification on the optimized three-dimensional detection result combined with the motion prediction information of the historical frame.

[0052] The state transition model is constructed by using the three-dimensional position, speed and acceleration information of the tracked target in the historical frame, the expected position and motion trajectory of each target in the current frame are predicted through Kalman filtering to generate the predicted state including the position covariance matrix, the optimized three-dimensional detection result is spatially associated with the predicted trajectory, the Hungarian algorithm is used to realize data association, the Mahalanobis distance between the detection result and the predicted position is calculated as the matching measure, and the detection pairs with short distance and small covariance are preferentially associated; for the unassociated detection result, secondary matching is performed according to the similarity of the motion state and the historical trajectory, and temporary occlusion or detection loss is tolerated; if the deviation between the detection result and the predicted trajectory exceeds the threshold, the current detection position is adjusted through weighted fusion, and the weight is determined by the detection confidence and the prediction covariance; finally, the trajectory smoothing processing is performed on the corrected three-dimensional detection frame, the historical position information in the sliding window is used to fit a cubic spline curve, the instantaneous detection noise is eliminated, and the three-dimensional target detection result which is continuous in space and time and consistent in motion state is generated.

[0053] In the embodiment of the application, the matching score is calculated based on the target geometric consistency, the category confidence and the spatial distribution similarity, the data association is comprehensively considered in multiple dimensions, the two-dimensional detection frame and the three-dimensional detection frame can be more accurately matched in a complex scene, the accuracy of data association is improved, the multi-stage optimization processing is performed based on the matching result, compared with the conventional single optimization mode, the interference factors in the complex scene can be more comprehensively processed, the optimization effect is improved, and the trajectory consistency verification is performed on the optimized three-dimensional detection result combined with the motion prediction information of the historical frame, the historical information is fully utilized, the target trajectory can be more accurately judged, the target motion state mutation or abnormal motion condition can be effectively coped with, and therefore the more accurate and reliable three-dimensional target detection result after fusion is obtained.

[0054] Please refer to Figure 2 Another embodiment of the target detection method based on multi-modal fusion in the embodiment of the application includes:

[0055] 201. Obtain the two-dimensional detection frame and the three-dimensional detection frame of the current frame.

[0056] The step 201 is similar to the execution process of the step 101 described above, and will not be described here.

[0057] 202、corresponding to the 2D bounding box and the 3D bounding box by a dynamic adaptive bidirectional matching strategy, to obtain a matching result, the matching strategy being based on target geometric consistency, category confidence and spatial distribution similarity to calculate a matching score;

[0058] extracting geometric features of the 2D bounding box and the 3D bounding box, calculating a geometric consistency score; calculating a category confidence score according to category prediction results of the 2D bounding box and the 3D bounding box; analyzing distribution of the 2D bounding box and the 3D bounding box in an image space, calculating a spatial distribution similarity score; calculating a matching score based on the geometric consistency score, the category confidence score and the spatial distribution similarity score, and determining the matching result according to the matching score.

[0059] mapping eight vertices of the 3D bounding box to a 2D image plane through a camera projection matrix calibrated in advance to obtain a projected 2D polygon region; calculating an intersection over union between the projected 2D polygon region and the original 2D bounding box as a basic geometric consistency measure; at the same time, extracting a projection coordinate of a center point of the 3D bounding box in the image plane and a Euclidean distance of the center point of the 2D bounding box, and performing normalization processing; further, uniformly sampling a plurality of points along four edges of the 2D bounding box, calculating an average value of shortest distances from each sampling point to edges of the projected 3D bounding box; based on the three geometric features of the calculated intersection over union, Euclidean distance and average value of shortest distances, a final geometric consistency score is calculated in a weighted fusion manner, wherein the intersection over union feature is given the largest weight, the center point distance is given the second largest weight, and the edge alignment degree is given a proper weight; further, to enhance robustness, a Gaussian kernel function can be used for nonlinear mapping of the distance feature, and an adaptive normalization method is used to eliminate the influence of detection boxes of different scales, and the geometric consistency score can comprehensively reflect the matching degree of the 2D bounding box and the 3D bounding box in terms of position, shape and edge alignment. The prior art usually only relies on a single intersection over union (IoU) to evaluate the geometric matching degree of the 2D and 3D bounding boxes, and there are problems of single evaluation dimension, insensitivity to edge alignment and shape difference, especially in complex scenes or small target detection, which is easily affected by local occlusion and scale change. The present scheme realizes multi-dimensional collaborative evaluation of the position, shape and edge alignment of the detection box by fusing the three geometric features of IoU, normalized center point distance and average value of edge shortest distances, and using weighted fusion and nonlinear mapping to enhance robustness, which significantly improves the comprehensiveness and accuracy of 2D-3D bounding box matching in complex scenes.

[0060] The prediction probability values of two detection boxes for the same target category are obtained, the class prediction probability of the two-dimensional detector is multiplied element by element with the class prediction probability of the three-dimensional detector to obtain an initial joint probability distribution; considering the importance difference of different categories in the autonomous driving scene, a category weight coefficient is introduced to give higher weight to key categories such as vehicles and pedestrians; a cross-modal confidence calibration model is established, the mapping relationship between the two-dimensional confidence and the three-dimensional confidence is fitted through the kernel density estimation method, and the deviation caused by the difference between the confidence distributions of different detectors is eliminated; the calibrated confidence difference absolute value is calculated, and the exponential function is used to convert it into a similarity measure, and finally the weighted joint category probability and the calibrated confidence similarity are linearly weighted and fused, wherein the category probability part occupies the dominant weight, and the comprehensive category confidence score is formed, which not only considers the semantic consistency of the cross-modal detection results, but also solves the problem of inconsistent output scales of different detectors through confidence calibration, and can effectively distinguish the correct and incorrect matching of the detection box pair. The prior art generally directly fuses the original category probability of the two-dimensional and three-dimensional detection boxes, such as simple average or maximum fusion, or only through fixed weight weighting, without considering the importance difference of different categories in the autonomous driving scene and the scale inconsistency problem of the confidence distribution of different detectors, resulting in insufficient discrimination ability for key categories (such as vehicles and pedestrians) and being easily affected by the detector bias. The present scheme combines the joint probability distribution by element by element multiplication, introduces the category weight coefficient to strengthen the key categories, uses the kernel density estimation to calibrate the cross-modal confidence and convert it into a similarity measure, and finally weights and fuses the joint probability and the calibrated similarity, which not only ensures the semantic consistency, but also eliminates the output difference of the detector, significantly improving the accuracy and robustness of the cross-modal detection box matching in complex scenes.

[0061] A local neighborhood space is constructed in the image plane, a circular region with a fixed radius is defined centered on the target detection box, the number distribution characteristics of all two-dimensional detection boxes in the region are counted, and the number distribution characteristics of the corresponding three-dimensional detection boxes after projection are calculated to obtain the local density distribution histogram in two modalities; the relative position relationship between the target detection box and other detection boxes in its neighborhood is extracted to construct a spatial relationship description matrix, which records the azimuth angle and distance ratio in two-dimensional space and projected three-dimensional space respectively; the distribution similarity is measured by calculating the Bhattacharyya coefficient of the two density distribution histograms, and the spatial relationship matrix difference is compared using the Frobenius norm; the distribution similarity measure and the spatial relationship difference are weighted and fused, with the density distribution characteristics being given a higher weight to highlight the subject structure similarity; finally, the fused distance measure is converted into a similarity score using a Gaussian radial basis function, and an adaptive normalization method is used to eliminate the scale difference caused by scene complexity, which can effectively represent the topological structure consistency between detection boxes in a multi-target scene. The prior art generally only judges the spatial relationship of detection boxes in a multi-target scene through simple overlap or a single distance threshold, which leads to false matching in dense scenes due to local occlusion or scale difference. The present scheme constructs the density distribution histogram and spatial relationship description matrix of the local neighborhood, combines the Bhattacharyya coefficient and the Frobenius norm to quantify the distribution similarity and structure difference, and uses weighted fusion and Gaussian radial basis function conversion to achieve fine evaluation of the multi-target topological structure consistency, significantly improving the robustness and accuracy of cross-modal detection box matching in complex scenes.

[0062] The three scores are normalized and assigned different weights, with the geometric feature weight being the highest, the category feature weight being the second, and the spatial distribution feature weight being the lowest. After obtaining the comprehensive matching score by weighted summation, a dynamic threshold strategy is used to select candidate matching pairs, an improved Hungarian algorithm is used for optimal bipartite graph matching to ensure that each detection box is matched with at most one corresponding box, and after matching is completed, the matching result is output, including matched detection pairs, three-dimensional detection boxes that are not matched successfully, and two-dimensional detection boxes that are not matched successfully.

[0063] For example, when normalizing the geometric consistency score, the category confidence score and the spatial distribution similarity score, the Min-Max normalization method can be used to linearly map each dimension score to the interval [0, 1] to eliminate the dimensional difference; according to the characteristics of the multi-modal detection task, the weight can be assigned, wherein the geometric feature weight can be set to 0.6 to highlight the core position of the spatial position matching, the category feature weight can be set to 0.3 to reflect the semantic consistency, and the spatial distribution feature weight can be set to 0.1 as an auxiliary constraint; the comprehensive matching score is calculated by a weighted sum formula St = 0.6 * Sg + 0.3 * Sc + 0.1 * Ss, wherein St, Sg, Sc and Ss are the comprehensive matching score, the normalized consistency score, the category confidence score and the spatial distribution similarity score respectively; a dynamic threshold strategy is adopted to automatically adjust the screening threshold according to the current frame detection box density, and only the candidate matching pairs with a score higher than the threshold are retained; finally, an improved Hungarian algorithm is used to construct a cost matrix to solve the global optimal bipartite graph matching, so that each two-dimensional / three-dimensional detection box is matched with at most one corresponding box, and after the matching is completed, three types of results are output: matched detection pairs, unmatched three-dimensional detection boxes and unmatched two-dimensional detection boxes.

[0064] 203、for the matched detection pairs, state optimization is performed by minimizing the IoU residual, the shape residual and the ground constraint residual;

[0065] The intersection over union residual between the matched two-dimensional detection box and the three-dimensional detection box is calculated, and the intersection over union residual is minimized to adjust the position and size of the three-dimensional detection box; the shape features of the three-dimensional detection box are analyzed, the shape residual is calculated, and the shape residual is minimized to optimize the shape of the three-dimensional detection box; the ground constraint residual is calculated and optimized through the spatial constraint relationship between the three-dimensional detection box and the ground, to ensure that the three-dimensional detection box meets the geometric conditions of ground fitting. The prior art fails to consider the influence of the detection box position, size, shape and ground constraint on the detection result, resulting in limited detection accuracy in complex scenes. However, by calculating and minimizing the intersection over union residual, the position and size of the three-dimensional detection box can be accurately adjusted to make it more matched with the two-dimensional detection box, the shape residual can be calculated and optimized to effectively optimize the shape of the three-dimensional detection box to fit the actual shape of the target, and by calculating and optimizing the ground constraint residual, the three-dimensional detection box can meet the geometric conditions of ground fitting, significantly improving the accuracy and reliability of three-dimensional target detection in complex scenes.

[0066] The intersection over union (IoU) residual optimization target is to minimize the spatial overlap difference (3D IoU) between the three-dimensional detection box and the three-dimensional box generated by inverse projection of the two-dimensional box projected from the two-dimensional detection box in the image plane, and the planar overlap difference (2D IoU) between the two-dimensional detection box and the two-dimensional box projected from the three-dimensional box.

[0067] 3D IoU Residual: The residual between the 3D bounding box B3D and the optimized 3D bounding box Bopt is defined as follows:

[0068]

[0069] in, It is a three-dimensional intersection-union ratio, calculated as the ratio of the intersection to the union of the volumes of the two frames.

[0070] 3D IoU Residual: Projecting the optimized 3D bounding box Bopt onto the image plane yields a 2D bounding box π(Bopt). The residual between this π and the original 2D detection bounding box B2D is:

[0071]

[0072] Shape constraint residuals:

[0073]

[0074] in, and and represent the length and width of the k-th 3D semantic box, respectively, and μ is the prior value of the aspect ratio. The μ values ​​for different object categories are not the same.

[0075] Ground constraint residuals :

[0076]

[0077] in, It is the height of the ground. Let z be the z-coordinate of the k-th 3D semantic box. The height of the k-th 3D semantic box.

[0078] 204. For unmatched 3D detection boxes, remove or retain them based on a preset confidence threshold;

[0079] For each 3D detection box that fails to match, its confidence level is compared with a preset confidence threshold. If the confidence level of the 3D detection box is lower than the confidence threshold, it is removed. If the confidence level of the 3D detection box is not lower than the confidence threshold, it is retained.

[0080] If the confidence level s1 of the 3D detection box is lower than the confidence threshold s2, it is considered a misjudgment and is removed; otherwise, the detection box is retained, i.e., the retention condition is s1≥s2.

[0081] like Figure 3 As shown, Figure 3 (a) indicates that the utility poles along the roadside may visually resemble pedestrians (marked with red ovals). And... Figure 3In point cloud detection information (b), the point cloud-based detector did indeed misdetect a utility pole as a pedestrian (marked by the red box). Since the semantic features of a utility pole are fundamentally different from those of a pedestrian, and the utility pole does not match any two-dimensional detection box in the image information, the confidence level of such misdetected targets in point cloud detection is usually lower than the preset confidence threshold. Filtering based on the confidence threshold can effectively reduce such errors of misdetecting utility poles as pedestrians and improve detection accuracy.

[0082] 205. For unmatched 2D detection boxes, generate new 3D detection boxes through monocular reconstruction and ground height adjustment;

[0083] Using preset camera intrinsics and monocular vision methods, the three-dimensional spatial position and size of the unmatched two-dimensional detection boxes are estimated to obtain the initial three-dimensional detection boxes; ground height information is obtained, and the ground height of the initial three-dimensional detection boxes is determined based on the ground height information; the adjusted initial three-dimensional detection boxes are used as new three-dimensional detection results to supplement the three-dimensional information corresponding to the unmatched two-dimensional detection boxes.

[0084] Due to the sparsity of LiDAR point clouds, missed detections of 3D objects are common, resulting in some 2D bounding boxes failing to find matching 3D bounding boxes. To address these unmatched 2D bounding boxes, this step generates initial 3D bounding boxes using a monocular 3D reconstruction method and optimizes their depth and z-axis coordinates using ground height information to ensure the reconstructed 3D bounding boxes match the actual scene.

[0085] For monocular 3D reconstruction:

[0086] For each unmatched 2D bounding box, assume it corresponds to a 3D target. Given the camera intrinsic matrix K, the rotation matrix R of the target in the camera coordinate system and the positions of the three vanishing points (VPs) satisfy the following relationship: The specific steps are as follows: calculate the rotation matrix (R) for each point (VP):

[0087]

[0088] in, Let be the pixel coordinates of the i-th vanishing point. Let be the i-th column vector of the rotation matrix R. By sampling a point at the top edge of the 2D detection box as a vertex projection of the 3D detection box, the projection positions of the remaining vertices are derived, thus generating the initial 3D detection box.

[0089] Solving the Perspective-n-Point (PnP) problem: Sampling four vertices to solve the PnP problem yields the rotation matrix R and translation vector t from the reconstructed 3D bounding box to the camera. The pose of the initial 3D bounding box is then optimized by minimizing the reconstruction projection error.

[0090]

[0091] wherein, is the i-th vertex pixel coordinate of the two-dimensional bounding box, is the corresponding three-dimensional space point, is the projection function.

[0092] Optimizing the depth and z-axis coordinate:

[0093] The depth estimation of single-frame monocular observation is not accurate, which may cause deviation of the z-coordinate (height direction) of the three-dimensional bounding box. To solve this problem, the depth and z-coordinate are adjusted using the ground segmentation information.

[0094] The RANSAC algorithm is used to segment the ground from the original laser point cloud. The steps of the RANSAC algorithm are as follows: first, randomly sample K points from the original point cloud; then, fit a plane model to the K points; calculate the distance d of all points in the original point cloud to the plane, and set a threshold dthres, if d < dthres, mark the point as an inlier, otherwise mark it as an outlier; repeat the above steps until a certain number of iterations or a stop condition is met, select the model with the maximum number of inliers from all iterations; then estimate the model parameters using all inliers, and output the final ground model and the corresponding inliers, i.e. the ground points. Assuming that the number of iterations is M times, the probability of a ground point being selected from the original point cloud is p, and the probability of the fitted ground model not containing non-ground points is z, then:

[0095]

[0096] Depth and height adjustment:

[0097] Adjust the depth d of the reconstructed cuboid along the camera optical axis direction, so that it encloses as many non-ground point clouds as possible.

[0098] According to the local ground height ( calculated by the mean value of the ground point cloud within the radius r neighborhood), the height of the bottom of the cuboid is corrected :

[0099]

[0100] wherein, h is the height of the cuboid.

[0101] 206、Integrate the matching successful detection pair, the un-matching successful three-dimensional bounding box and the un-matching successful two-dimensional bounding box, and obtain the optimized three-dimensional detection result;

[0102] 207. The optimized three-dimensional detection result is subjected to trajectory consistency verification in combination with motion prediction information of historical frames, so as to obtain a fused three-dimensional target detection result.

[0103] In the embodiment of the present application, multi-dimensional scoring matching is performed by dynamically adaptive bidirectional matching strategy in combination with target geometric consistency, category confidence and spatial distribution similarity, effectively solving the mis-matching problem caused by the dependence on single spatial overlap in the prior art, and the state estimation precision is improved by using multi-residual optimization for the matched detection pairs, and the un-matched targets are screened by confidence or supplemented by monocular reconstruction, overcoming the performance degradation of the traditional method under the conditions of occlusion and missed detection, further combining with historical motion prediction for trajectory verification, enhancing the time sequence stability, thereby significantly improving the matching accuracy and detection robustness in complex scenes, reducing the false detection and missed detection caused by factors such as occlusion and deformation, and finally realizing more reliable three-dimensional target detection.

[0104] Please refer to Figure 4 Another embodiment of the target detection method based on multi-modal fusion in the embodiment of the present application includes:

[0105] 401. Obtain the two-dimensional detection frame and the three-dimensional detection frame of the current frame;

[0106] 402. Data association is performed on the two-dimensional detection frame and the three-dimensional detection frame by a dynamic adaptive bidirectional matching strategy, and a matching result is obtained, and the matching strategy calculates a matching score based on target geometric consistency, category confidence and spatial distribution similarity;

[0107] 403. Multi-stage optimization processing is performed based on the matching result, and an optimized three-dimensional detection result is obtained;

[0108] 404. The three-dimensional position and motion state information of the tracked targets in the historical frames are used to predict the expected position and motion trajectory of each target in the current frame by a motion model;

[0109] The continuous three-dimensional position coordinates and motion velocity information of the tracked targets are extracted from the historical frame data; a linear motion model or a nonlinear motion model is selected according to the motion characteristics of the targets; the expected three-dimensional position and motion trajectory of each target in the current frame are recursively calculated by the selected motion model.

[0110] From the historical frame data, for each tracked target, the three-dimensional position coordinate sequence of its continuous multiple frames is extracted in turn and arranged in time sequence, the instantaneous speed between each frame is calculated through the position difference and time interval of adjacent frames, and then the motion speed sequence synchronized with the position sequence is generated; based on the motion characteristics of the target, the model is selected, if the fluctuation range of the target speed in the historical frame is less than the preset threshold and the motion direction changes gently, it is determined as uniform linear motion and the linear motion model is selected, the state vector of this model contains three-dimensional position and speed components, the average speed is solved as the state transition parameter by fitting the historical position and speed data through the least square method; if the target speed fluctuation is significant or the motion direction changes frequently, it is determined as non-uniform or curve motion and the constant steering rate and acceleration model is selected, the extended state vector of this model contains yaw angle, angular velocity and acceleration components, the angular velocity and acceleration parameters are estimated by fitting the curvature change and speed trend of the historical trajectory;

[0111] When the expected three-dimensional position and motion trajectory of each target in the current frame are recursively calculated based on the selected motion model, if the linear motion model is used, the three-dimensional position and speed components contained in the state vector thereof are used to directly extrapolate the current frame position by using the uniform speed propagation formula , wherein and are the position and speed of the last frame respectively, is the time interval between frames, and the speed parameter is updated through the least square fitting result of the historical data to ensure the smoothness of the trajectory; if the constant steering rate and acceleration model is used, based on the yaw angle, angular velocity and acceleration components in the extended state vector thereof, the state quantity is iteratively updated through the discrete time state transition equation , wherein is the state vector of the last frame, F is a nonlinear transition matrix constructed according to the angular velocity and acceleration, is the process noise, and then the three-dimensional position component is extracted from the updated state vector as the expected position of the current frame, and the complete motion trajectory containing position, speed, yaw angle and other information is generated through the recursive results of continuous multiple frames, and the uncertainty in the recursive process is dynamically corrected by using Kalman filtering or particle filtering algorithm, so as to improve the accuracy of the trajectory prediction.

[0112] Existing technologies generally rely on a single fixed model, such as a uniform velocity CV model or a uniform acceleration CA model, or simple multi-model switching, such as the IMM algorithm, for target trajectory prediction. Their shortcomings lie in their inability to dynamically adapt to the target's motion patterns across all scenarios, from uniform velocity to complex maneuvers. Furthermore, model parameters typically depend on prior assumptions or offline statistics, resulting in insufficient adaptability to dynamic scenarios such as sudden acceleration and turning. While some solutions incorporate deep learning to extract features, they suffer from decoupling of features from motion modeling and poor real-time performance. This proposed solution analyzes target velocity fluctuations and the rate of change of direction, intelligently selects linear or nonlinear motion models, and dynamically optimizes model parameters using historical data. Simultaneously, it integrates multi-dimensional information such as 3D position, velocity, and yaw angle to generate a complete trajectory. This effectively overcomes the limitations of existing technologies, such as rigid models, static parameters, and limited information, significantly improving the accuracy and robustness of trajectory prediction in complex dynamic scenarios.

[0113] 405. Compare the optimized 3D detection results with the predicted expected position and trajectory, and calculate the deviation between the actual detection results and the predicted results;

[0114] Calculate the Euclidean distance between the center point of the optimized 3D detection box and the expected position at the corresponding time point on the predicted trajectory; calculate the angular deviation between the motion direction of the optimized 3D detection box and the motion direction of the predicted trajectory; generate a trajectory consistency score based on the Euclidean distance and the angular deviation as the deviation result.

[0115] The coordinates of the center point of the 3D detection box of the target in the current frame are extracted from the optimized 3D detection results. At the same time, the expected position coordinates at the corresponding time point are obtained from the predicted trajectory. The Euclidean distance between the two in 3D space is calculated by squaring the differences in the three coordinate axes, summing them, and then taking the square root to obtain the quantitative value of the position deviation. The actual motion direction vector of the optimized 3D detection box is calculated based on the position changes of the box between consecutive frames. At the same time, the expected motion direction vector for the corresponding time period is extracted from the continuous state of the predicted trajectory. The angle deviation between the two direction vectors is calculated using the vector dot product formula. That is, the ratio of the dot product of the two vectors to the product of their respective magnitudes is calculated, and then the inverse cosine function of the ratio is taken to obtain the angle value. Finally, the Euclidean distance of the position deviation and the angle deviation are normalized and assigned different weight coefficients to reflect their influence on the trajectory consistency. The weighted position deviation and the direction deviation are added together to obtain the comprehensive deviation value. The comprehensive deviation value is then converted into a trajectory consistency score through a preset scoring mapping function. The score range is usually set between 0 and 1. The closer the score is to 1, the higher the consistency between the detection result and the predicted trajectory. This score is used as the final deviation evaluation result.

[0116] Existing technologies typically measure the deviation between the detection result and the predicted trajectory using only the Euclidean distance of a single position, ignoring the crucial dynamic feature of motion direction. This leads to incomplete evaluation results, especially prone to misjudgment when the target undergoes maneuvering movements such as turning. While some solutions incorporate directional information, they often employ simple threshold comparisons or unnormalized weighted sums, lacking coordinated quantification and unified scoring mapping for position and direction deviations. This proposed solution simultaneously calculates the Euclidean distance between the center point of the 3D detection box and the predicted position, as well as the angular deviation between the actual and expected motion directions. After normalizing and weighting both, it maps them to a trajectory consistency score ranging from 0 to 1. This overcomes the limitations of single-index evaluation and achieves intuitive comparability of deviation results across different scenarios through standardized scoring, significantly improving the accuracy and robustness of trajectory evaluation in complex dynamic scenes.

[0117] 406. Adjust and optimize the 3D detection results based on the calculated deviations to obtain the fused 3D target detection results.

[0118] If the calculated deviation is less than the preset threshold, the optimized 3D detection result is used as the final output; if the calculated deviation is not less than the preset threshold, the position and motion state parameters of the optimized 3D detection box are corrected according to the predicted trajectory to obtain the fused 3D target detection result.

[0119] If the deviation result is less than the preset threshold, it indicates that the optimized 3D detection result is highly consistent with the predicted trajectory. In this case, the center point coordinates, size parameters, and motion state of the detection box are directly used as the final output. If the deviation result is not less than the preset threshold, the trajectory fusion correction process is started: according to the expected position of the predicted trajectory in the current frame, the center point of the detection box is weighted and translated, and the weight is dynamically determined by the ratio of the deviation result to the threshold. At the same time, the orientation angle of the detection box is adjusted in combination with the motion direction of the predicted trajectory so that its deviation from the trajectory direction does not exceed the preset angle tolerance. Then, the motion state of the detection box is smoothly corrected by using the velocity and acceleration parameters of the predicted trajectory. The detection velocity and the prediction velocity are fused by algorithms such as Kalman filtering to finally generate the fused 3D target detection result, ensuring that the output result is better than the single detection or prediction result in terms of position accuracy and motion continuity.

[0120] In the embodiment of the present application, in view of the problem of low matching accuracy caused by the existing technology relying only on the intersection-over-union or fixed threshold for two-dimensional and three-dimensional detection box association, the present scheme adopts a dynamic adaptive bidirectional matching strategy, and calculates the matching score by comprehensively considering multi-dimensional factors such as target geometric consistency, category confidence and spatial distribution similarity, thereby significantly improving the robustness of data association in complex scenes. At the same time, by constructing a motion model based on historical trajectories, accurate trajectory prediction is realized, and a joint scoring mechanism of Euclidean distance and direction angle deviation is designed to dynamically fuse the detection results and prediction information. The final fusion result can still maintain high-precision positioning and motion continuity under complex interference, effectively solving the technical problems of limited optimization effect and easy to produce false matching of traditional methods in complex scenes.

[0121] The following is an experimental embodiment. The present experiment proposes a target detection framework (GOOD) based on multi-modal fusion. The present experiment verifies the proposed target detection method based on multi-modal fusion based on two automatic driving public data sets, nuScenes and KITTI. The experiment strictly follows the technical solution of the present application, and focuses on verifying the effectiveness of the three core modules of the dynamic adaptive bidirectional matching strategy, multi-stage optimization processing and trajectory consistency verification.

[0122] In terms of experimental configuration, it includes:

[0123] (1) The sensor configuration is shown in Figure 5 and Figure 6 . Figure 5 The nuScenes adopts a ring system of 1 laser radar + 6 cameras, Figure 6 The KITTI adopts 1 laser radar + 4 front-view cameras.

[0124] (2) The two-dimensional detection network adopts YOLOv5, and the three-dimensional detection network selects three mainstream architectures, PointPillars, SECOND and CenterPoint.

[0125] (3) The evaluation indexes include: core indexes: mAP (mean average precision), NDS (nuScenes detection score); error indexes: mATE (mean translation error), mASE (mean size error), mAOE (mean orientation error), etc.; especially for the KITTI data set, the evaluation is carried out according to the difficulty level (easy / medium / difficult).

[0126] nuScenes: nuScenes provides a large laser point cloud sequence dataset with semantic labels attached to each frame. The dataset contains 1000 sequences, with 700 sequences available for training, 150 sequences for validation, and 150 sequences for testing. Ten classes are annotated for detection evaluation. Each sequence is about 20 seconds long, with a data sampling frequency of 10 Hz. In addition to mAP (mean Average Precision), nuScenes also uses NDS (nuScenes Detection Score) as an official metric to evaluate detector performance. The definition of NDS is as follows:

[0127]

[0128] where, is the error subset including translation, size, orientation, attribute, and velocity. It is worth noting that the mAP in this dataset is calculated based on the overhead view center distance with different thresholds {0.5 meters, 1 meter, 2 meters, 4 meters}, rather than the standard 3D box IoU.

[0129] KITTI: The KITTI dataset contains 7481 training samples and 7518 test samples, each of which is annotated with 3D and 2D information for cars, pedestrians, and cyclists on the street. Each class is divided into three levels according to the size of the object, whether it is truncated or occluded: simple, medium, and difficult. Different methods can be evaluated using 3D IoU calculated AP (Average Precision) in this dataset. In this experiment, the original 7481 training samples are divided into 3740 samples for training based on learning methods and 3741 samples for validation.

[0130] For 2D detection networks, the experiment selects YOLOv5l pre-trained model and trains on nuScenes training dataset for 100 epochs. All 3D detection models are trained using 8 GTX 1080Ti GPUs. The experiment takes the laser point cloud-based 3D detection network as the benchmark for comparison, as shown in Table 1, which is a summary of the performance of the proposed method on the nuScenes trainval dataset compared with each corresponding benchmark method. The experiment records NDS, mAP, mATE (mean average location error), mASE (mean average size error), mAOE (mean average orientation error), mAVE (mean average velocity error), and mAAE (mean average attribute error), where↑represents the higher the better and↓represents the lower the better. As expected, GOOD performs better than PointPillars, SECOND, and CenterPoint, respectively, with a relative increase of 6.2%, 5.0%, and 4.2% in NDS score and an increase of 9.1%, 6.4%, and 3.9% in mAP score. As for each error index, although GOOD performs slightly worse than SECOND in mAOE score (about 1.4%), this is because the 3D detection box reconstructed by the add module may have the wrong orientation under the single-frame constraint, but in comparison with all other benchmark methods in error index, GOOD still performs better.

[0131] Table 1 Performance comparison results of different laser 3D target detection benchmark methods and GOOD on nuScenes trainval dataset

[0132]

[0133] Compared with the laser-camera fusion-based 3D object detection method, this paper compares the GOOD with the pre-fusion PointPainting and post-fusion CLOCs based on learning. Considering that both of them need a 3D detector to generate or provide a 3D detection box, PointPillars is selected as the 3D detector, and PointPainting with PointPillar is named as Painted PointPillars. In addition, YOLOv5 is selected as the 2D detector of CLOCs and GOOD. Given that the open-source CLOCs algorithm only supports the KITTI dataset, experiments are conducted on KITTI. In addition, considering that the KITTI data is not time-sequential, that is, the data before and after are not continuous, which means that the GOOD cannot apply the 3D Multiple Object Tracking (3D MOT) part. The IoU threshold of Car, Pedestrian and Cyclist is set to 0.7, 0.5 and 0.5 respectively. The results are summarized in Table 2, which shows that although the pre-fusion method PointPainting performs well in Pedestrian detection, because the point cloud details of small objects may be more supplemented by images, the accuracy of Car detection is lower compared with PointPillars based on pure laser point cloud due to the error caused by sensor calibration and synchronization. Compared with the post-fusion method CLOCs, because the occlusion and truncation are not serious, in this case, the optimization constraint brought by 2D detection is feasible and effective, so the GOOD performs better on Car in the easy part of the KITTI, with an improvement of nearly 1.5% in 3D AP. Because the shape of different Cyclist may change frequently on the moderate or hard part data, and the number of cyclists in the validation set is relatively small in general, the improvement is relatively small compared with CLOCs due to the introduction of shape residuals or several error cases. However, it is worth mentioning that the performance of the GOOD can be further improved with the help of 3DMOT, which can be evaluated in the following ablation study experiments.

[0134] Table 2 Performance comparison results of different 3D object detection methods on KITTI validation dataset

[0135] Ablation experiments:

[0136] Considering that the nuScenes dataset is sampled in order, this paper can conduct a complete validation on each part of the GOOD on this basis.

[0137] Optimization module and 3D MOT module: First, evaluate the effectiveness of each module in the optimization and 3D MOT pipeline. The optimization part includes the refine module, keep or delete (KoD) module, and add module. The results are shown in Table 3, where it can be concluded that the refine module contributes the most to reducing errors, as constraints from 2D detection can help correct the state of 3D detection, as shown in Figure 7 Fig. 3 (a), (b), and (c). As can be seen from Figure 7 Fig. 3 (a), (b), and (c), in the original point cloud-based 3D detection, the red 3D box representing the truck has a certain angle between the red arrow indicating its orientation and the green arrow indicating the actual orientation of the truck point cloud, i.e., the detection box is slightly skewed compared to the actual situation, which can be adjusted by the constraints of the 2D detection box. The KoD module can help improve the mAP score by deleting mismatched 3D instances with low detection scores, as they may be false or redundant detections. In addition, with the help of the add module, a significant improvement in the mAP score can be observed. Figure 10 An example is provided to vividly illustrate the depth and z coordinate of the reconstruction cuboid adjustment algorithm proposed in this paper: the child and his mother in the figure are both detected by the 2D detector, as shown in Figure 8 Fig. 4 (a); but the child is missed by the 3D detector, as shown in Figure 8 Fig. 4 (b). In Figure 8 Fig. 4 (c), single-frame image 3D target detection is performed on the unmatched 2D candidate box, but its estimated depth is not accurate, as shown in Figure 8 Fig. 4 (d). In this paper, the depth is adjusted by using the depth and z coordinate adjustment algorithm, and the final result is shown in Figure 8 Fig. 4 (e). In addition, the introduction of 3D MOT is also beneficial to improve the mAP score and reduce the error index.

[0138] Table 3 Ablation experiment statistics on nuScenes dataset

[0139]

[0140] Matching module: In addition, a control experiment is designed to prove the effectiveness of the matching method in this paper, as shown in Table 4. The 3D detector used is pointpillars, and the 2D detector is YOLOv5. The results show that compared with single-side nearest neighbor matching, double-side nearest neighbor matching performs better in terms of NDS and mAP score.

[0141] Table 4 Performance comparison results of different 3D-2D matching methods

[0142]

[0143] The target detection method based on multi-modal fusion in the embodiments of the present application is described above, and the target detection device based on multi-modal fusion in the embodiments of the present application is described below. Please refer to Figure 9 An embodiment of the target detection device based on multi-modal fusion in the embodiments of the present application includes:

[0144] The acquisition module 901 is configured to acquire a two-dimensional detection box and a three-dimensional detection box of a current frame.

[0145] The association module 902 is configured to perform data association on the two-dimensional detection box and the three-dimensional detection box through a dynamic adaptive bidirectional matching strategy to obtain a matching result, and the matching strategy is based on target geometric consistency, category confidence and spatial distribution similarity to calculate a matching score.

[0146] The optimization module 903 is configured to perform multi-stage optimization processing based on the matching result to obtain an optimized three-dimensional detection result.

[0147] The verification module 904 is configured to perform trajectory consistency verification on the optimized three-dimensional detection result in combination with motion prediction information of a historical frame to obtain a fused three-dimensional target detection result.

[0148] In the embodiments of the present application, the matching score is calculated based on target geometric consistency, category confidence and spatial distribution similarity, and multi-dimensional comprehensive consideration is performed for data association, so that the two-dimensional detection box and the three-dimensional detection box can be more accurately matched in a complex scene, the accuracy of data association is improved, multi-stage optimization processing is performed based on the matching result, compared with a conventional single optimization mode, interference factors in a complex scene can be more comprehensively handled, the optimization effect is improved, and at the same time, the trajectory consistency verification is performed on the optimized three-dimensional detection result in combination with the motion prediction information of the historical frame, the historical information is fully utilized, the target trajectory can be more accurately judged to be consistent or not, target motion state mutation or abnormal motion condition can be effectively coped with, and thus a more accurate and reliable fused three-dimensional target detection result is obtained.

[0149] Please refer to Figure 10 Another embodiment of the target detection device based on multi-modal fusion in the embodiments of the present application includes:

[0150] The acquisition module 901 is configured to acquire a two-dimensional detection box and a three-dimensional detection box of a current frame.

[0151] The association module 902 is configured to perform data association on the two-dimensional detection box and the three-dimensional detection box through a dynamic adaptive bidirectional matching strategy to obtain a matching result, and the matching strategy is based on target geometric consistency, category confidence and spatial distribution similarity to calculate a matching score.

[0152] The optimization module 903 is configured to perform multi-stage optimization processing based on the matching result, to obtain an optimized three-dimensional detection result.

[0153] The verification module 904 is configured to perform trajectory consistency verification on the optimized three-dimensional detection result in combination with motion prediction information of a historical frame, to obtain a fused three-dimensional target detection result.

[0154] Optionally, the association module 902 can be specifically configured to:

[0155] extract geometric features of the two-dimensional detection box and the three-dimensional detection box, calculate a geometric consistency score; calculate a category confidence score according to category prediction results of the two-dimensional detection box and the three-dimensional detection box; analyze distribution of the two-dimensional detection box and the three-dimensional detection box in an image space, calculate a spatial distribution similarity score; calculate a matching score based on the geometric consistency score, the category confidence score and the spatial distribution similarity score, and determine the matching result according to the matching score.

[0156] Optionally, the optimization module 903 includes:

[0157] The optimization unit 9031 is configured to perform state optimization on a matched detection pair by minimizing an IoU residual, a shape residual and a ground constraint residual.

[0158] The processing unit 9032 is configured to perform elimination or reservation on a three-dimensional detection box that fails to match, based on a preset confidence threshold.

[0159] The generation unit 9033 is configured to generate a new three-dimensional detection box for a two-dimensional detection box that fails to match, by monocular reconstruction and ground height adjustment.

[0160] Optionally, the optimization unit 9031 can be specifically configured to:

[0161] calculate an intersection-over-union residual between the matched two-dimensional detection box and the three-dimensional detection box, and minimize the intersection-over-union residual to adjust a position and a size of the three-dimensional detection box; analyze shape features of the three-dimensional detection box, calculate a shape residual, and minimize the shape residual to optimize a shape of the three-dimensional detection box; calculate and optimize a ground constraint residual through a spatial constraint relationship between the three-dimensional detection box and the ground, to ensure that the three-dimensional detection box meets a geometric condition of ground adhesion.

[0162] Optionally, the processing unit 9032 can be specifically configured to:

[0163] For each three-dimensional detection box that fails to match, compare a confidence of the three-dimensional detection box with a preset confidence threshold; if the confidence of the three-dimensional detection box is lower than the confidence threshold, eliminate the three-dimensional detection box; if the confidence of the three-dimensional detection box is not lower than the confidence threshold, reserve the three-dimensional detection box.

[0164] Optionally, the generating unit 9033 can be specifically configured to:

[0165] The three-dimensional spatial position and size of the two-dimensional detection frame that fails to match are estimated by using the preset camera internal parameter and the monocular vision method to obtain an initial three-dimensional detection frame; ground height information is obtained, and the ground height of the position where the initial three-dimensional detection frame is located is determined based on the ground height information; and the adjusted initial three-dimensional detection frame is taken as a new three-dimensional detection result to supplement the three-dimensional information corresponding to the two-dimensional detection frame that fails to match.

[0166] Optionally, the checking module 904 can be specifically configured to:

[0167] The prediction unit 9041 is configured to predict the expected position and motion trajectory of each target in the current frame by using the three-dimensional position and motion state information of the tracked targets in the historical frames and a motion model.

[0168] The calculation unit 9042 is configured to compare the optimized three-dimensional detection result with the predicted expected position and motion trajectory, and calculate the deviation between the actual detection result and the predicted result.

[0169] The adjustment unit 9043 is configured to adjust the optimized three-dimensional detection result according to the calculated deviation to obtain a three-dimensional target detection result that fuses the current detection and motion prediction.

[0170] Optionally, the prediction unit 9041 can be specifically configured to:

[0171] The continuous three-dimensional position coordinates and motion velocity information of the tracked targets are extracted from the historical frame data; a linear motion model or a nonlinear motion model is selected according to the motion characteristics of the targets; and the expected three-dimensional position and motion trajectory of each target in the current frame are recursively calculated by using the selected motion model.

[0172] Optionally, the calculation unit 9042 can be specifically configured to:

[0173] The Euclidean distance between the center point of the optimized three-dimensional detection frame and the expected position at the corresponding time point on the predicted trajectory is calculated; the included angle deviation between the motion direction of the optimized three-dimensional detection frame and the motion direction of the predicted trajectory is calculated; and the trajectory consistency score is generated based on the Euclidean distance and the included angle deviation as the deviation result.

[0174] Optionally, the adjustment unit 9043 can be specifically configured to:

[0175] If the calculated deviation result is less than a preset threshold, the optimized three-dimensional detection result is taken as the final output; if the calculated deviation result is not less than the preset threshold, the position and motion state parameters of the optimized three-dimensional detection frame are corrected according to the predicted trajectory to obtain the fused three-dimensional target detection result.

[0176] In the embodiment of the present application, the two-dimensional detection frame and the three-dimensional detection frame of the current frame are obtained, and the data correlation of the two-dimensional detection frame and the three-dimensional detection frame is performed through a dynamic adaptive bidirectional matching strategy to obtain a matching result. The matching strategy calculates a matching score based on target geometric consistency, category confidence, and spatial distribution similarity. For the matched detection pairs, state optimization is performed by minimizing the IoU residual, shape residual, and ground constraint residual. For the three-dimensional detection frames that are not matched successfully, elimination or reservation is performed based on a preset confidence threshold. For the two-dimensional detection frames that are not matched successfully, new three-dimensional detection frames are generated through monocular reconstruction and ground height adjustment. The optimized three-dimensional detection result is verified for trajectory consistency in combination with the motion prediction information of the historical frames to obtain a fused three-dimensional target detection result, effectively solving the problems of occlusion, false detection, and motion mutation in complex scenes, thereby improving the accuracy of the three-dimensional target detection result.

[0177] The above Figure 9 and Figure 10 The target detection device based on multi-modal fusion in the embodiment of the present application is described in detail from the perspective of modular functional entities. The electronic device in the embodiment of the present application is described in detail from the perspective of hardware processing.

[0178] Referring to Figure 11 The electronic device includes a processor 1100 and a memory 1101 storing machine executable instructions executable by the processor 1100, and the processor 1100 executes the machine executable instructions to implement the above-mentioned target detection method based on multi-modal fusion.

[0179] Further, Figure 11 The electronic device shown in the figure further includes a bus 1102 and a communication interface 1103, and the processor 1100, the communication interface 1103, and the memory 1101 are connected through the bus 1102.

[0180] The memory 1101 can include a high-speed random access memory (RAM) and can also include a non-volatile memory such as at least one disk memory. The communication between the system network element and at least one other network element is realized through at least one communication interface 1103 (which can be wired or wireless), and the Internet, a wide area network, a local area network, a metropolitan area network, etc. can be used. The bus 1102 can be an ISA bus, a PCI bus, or an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 11 In the figure, only one bidirectional arrow is used to represent the bus, but it does not mean that there is only one bus or only one type of bus.

[0181] The processor 1100 can be an integrated circuit chip having a processing capability of signals. In the implementation process, each step of the above method can be completed by the integrated logic circuit of hardware in the processor 1100 or the instruction in the form of software. The processor 1100 described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. Each method, step and logic block diagram disclosed in the embodiments of the present disclosure can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present disclosure can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read only memory, a programmable read only memory or an electrically erasable programmable memory, a register, etc. The storage medium in the art. The storage medium is located in the memory 1101, and the processor 1100 reads the information in the memory 1101, and combines the hardware to complete the method steps of the above embodiments.

[0182] The present application also provides an electronic device, the computer device includes a memory and a processor, the memory has computer readable instructions stored therein, the computer readable instructions are executed by the processor, so that the processor executes the steps of the target detection method based on multi-modal fusion in each of the above embodiments. The present application also provides a computer readable storage medium, which can be a non-volatile computer readable storage medium, and can also be a volatile computer readable storage medium, the computer readable storage medium has instructions stored therein, when the instructions run on the computer, the computer executes the steps of the target detection method based on multi-modal fusion.

[0183] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above described system, device and unit can refer to the corresponding process in the foregoing method embodiment, which will not be described here.

[0184] The integrated unit, if in the form of a software function unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on such an understanding, the technical solutions of the present application, essentially or in the part that contributes to the prior art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and various other media that can store program codes.

Claims

1. A target detection method based on multimodal fusion, characterized in that, The target detection method based on multimodal fusion includes: Obtain the 2D and 3D bounding boxes of the current frame; The two-dimensional detection box and the three-dimensional detection box are associated by a dynamic adaptive bidirectional matching strategy to obtain a matching result. The matching strategy calculates the matching score based on the geometric consistency of the target, the category confidence and the spatial distribution similarity. Based on the matching results, a multi-stage optimization process is performed to obtain the optimized 3D detection results. By combining the motion prediction information of historical frames, the optimized 3D detection results are subjected to trajectory consistency verification to obtain the fused 3D target detection results; The multi-stage optimization process based on the matching results includes: for successfully matched detection pairs, state optimization is performed by minimizing the IoU residual, shape residual, and ground constraint residual; for unmatched 3D detection boxes, they are eliminated or retained based on a preset confidence threshold; for unmatched 2D detection boxes, new 3D detection boxes are generated through monocular reconstruction and ground height adjustment. The IoU residual refers to: for successfully matched 2D and 3D detection box pairs, the 3D intersection-union ratio (IoU) residual and the 2D IoU residual are calculated respectively. The 3D IoU residual is defined as 1 minus the 3D IoU value between the original 3D detection box and the optimized 3D box, while the 2D IoU residual is defined as 1 minus the 2D IoU value between the original 2D detection box and the 2D box projected onto the image plane.

2. The target detection method based on multimodal fusion according to claim 1, characterized in that, The step of associating the two-dimensional and three-dimensional detection boxes using a dynamic adaptive bidirectional matching strategy to obtain matching results includes: Extract the geometric features of the two-dimensional detection box and the three-dimensional detection box, and calculate the geometric consistency score; Based on the category prediction results of the two-dimensional and three-dimensional detection boxes, calculate the category confidence score; Analyze the distribution of the two-dimensional and three-dimensional detection boxes in the image space and calculate the spatial distribution similarity score; A matching score is calculated based on the geometric consistency score, the category confidence score, and the spatial distribution similarity, and the matching result is determined based on the matching score.

3. The target detection method based on multimodal fusion according to claim 1, characterized in that, The state optimization of the successfully matched detection pairs is performed by minimizing the IoU residual, shape residual, and ground constraint residual, including: Calculate the cross-union ratio (CUI) residual between the successfully matched 2D and 3D detection boxes, and minimize the CUI residual to adjust the position and size of the 3D detection box; Analyze the shape features of the 3D detection box, calculate the shape residual, and minimize the shape residual to optimize the shape of the 3D detection box; By calculating and optimizing the ground constraint residuals based on the spatial constraint relationship between the 3D detection frame and the ground, the 3D detection frame can meet the geometric conditions for ground adhesion.

4. The target detection method based on multimodal fusion as described in claim 1, characterized in that, The step of discarding or retaining unmatched 3D detection boxes based on a preset confidence threshold includes: For each 3D detection box that fails to match, its confidence level is compared with a preset confidence threshold. If the confidence level of a 3D detection box is lower than the confidence level threshold, it will be removed. If the confidence level of the 3D detection box is not lower than the confidence level threshold, then it is retained.

5. The target detection method based on multimodal fusion according to claim 1, characterized in that, The process of generating new 3D detection boxes for unmatched 2D detection boxes through monocular reconstruction and ground height adjustment includes: Using preset camera intrinsics and monocular vision methods, the three-dimensional spatial position and size of the unmatched two-dimensional detection boxes are estimated to obtain the initial three-dimensional detection boxes; Obtain ground height information, and determine the ground height of the initial three-dimensional detection box based on the ground height information; The adjusted initial 3D detection box is used as the new 3D detection result to supplement the 3D information corresponding to the unmatched 2D detection boxes.

6. The target detection method based on multimodal fusion according to claim 1, characterized in that, The process of combining motion prediction information from historical frames to perform trajectory consistency verification on the optimized 3D detection results, resulting in fused 3D target detection results, includes: Using the three-dimensional position and motion state information of the tracked targets in historical frames, the expected position and motion trajectory of each target in the current frame are predicted by a motion model. The optimized 3D detection results are compared with the predicted expected position and motion trajectory, and the deviation between the actual detection results and the predicted results is calculated. The optimized 3D detection results are adjusted based on the calculated deviations to obtain the fused 3D target detection results.

7. The target detection method based on multimodal fusion according to claim 6, characterized in that, The step of using the 3D position and motion state information of tracked targets in historical frames to predict the expected position and motion trajectory of each target in the current frame through a motion model includes: Extract continuous three-dimensional position coordinates and motion velocity information of tracked targets from historical frame data; Choose either a linear motion model or a nonlinear motion model based on the target's motion characteristics; The expected three-dimensional position and motion trajectory of each target in the current frame are recursively calculated using the selected motion model.

8. The target detection method based on multimodal fusion according to claim 6, characterized in that, The step of comparing the optimized 3D detection results with the predicted expected position and trajectory, and calculating the deviation between the actual detection results and the predicted results, includes: Calculate the Euclidean distance between the center point of the optimized 3D detection box and the expected position on the predicted trajectory at the corresponding time point; Calculate the angular deviation between the motion direction of the optimized 3D detection box and the motion direction of the predicted trajectory; A trajectory consistency score is generated based on the Euclidean distance and included angle deviation as the deviation result.

9. The target detection method based on multimodal fusion according to claim 6, characterized in that, The step of adjusting the optimized 3D detection result based on the calculated deviation to obtain a 3D target detection result that integrates the current detection and motion prediction includes: If the calculated deviation is less than the preset threshold, the optimized 3D detection result is used as the final output. If the calculated deviation is not less than the preset threshold, the position and motion state parameters of the optimized 3D detection box are corrected according to the predicted trajectory to obtain the fused 3D target detection result.

10. A target detection device based on multimodal fusion, characterized in that, The target detection device based on multimodal fusion includes: The acquisition module is used to acquire the 2D and 3D detection bounding boxes of the current frame; The association module is used to associate the two-dimensional detection box and the three-dimensional detection box with a dynamic adaptive bidirectional matching strategy to obtain a matching result. The matching strategy calculates the matching score based on the geometric consistency of the target, the category confidence and the spatial distribution similarity. The optimization module is used to perform multi-stage optimization processing based on the matching results to obtain optimized 3D detection results; The verification module is used to combine the motion prediction information of historical frames to perform trajectory consistency verification on the optimized three-dimensional detection results, so as to obtain the fused three-dimensional target detection results. The multi-stage optimization process based on the matching results includes: for successfully matched detection pairs, state optimization is performed by minimizing the IoU residual, shape residual, and ground constraint residual; for unmatched 3D detection boxes, they are eliminated or retained based on a preset confidence threshold; for unmatched 2D detection boxes, new 3D detection boxes are generated through monocular reconstruction and ground height adjustment. The IoU residual refers to: for successfully matched 2D and 3D detection box pairs, the 3D intersection-union ratio (IoU) residual and the 2D IoU residual are calculated respectively. The 3D IoU residual is defined as 1 minus the 3D IoU value between the original 3D detection box and the optimized 3D box, while the 2D IoU residual is defined as 1 minus the 2D IoU value between the original 2D detection box and the 2D box projected onto the image plane.

11. An electronic device, characterized in that, The electronic device includes: a memory and at least one processor, wherein the memory stores instructions; The at least one processor invokes the instructions in the memory to cause the electronic device to execute the multimodal fusion-based target detection method as described in any one of claims 1-9.

12. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions are executed by the processor, they implement the target detection method based on multimodal fusion as described in any one of claims 1-9.

Images