Target detection task truth value generation method and device based on multi-sensor fusion

Through multi-sensor fusion and timing tracking algorithms, combined with manual annotation correction, the problems of high cost and low effect of target detection truth value generation in the prior art are solved, and higher quality and low cost truth value generation are achieved.

CN119024329APending Publication Date: 2024-11-26SHENZHEN DEEPROUTE AI CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411032091.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The truth value generation scheme of existing target detection tasks is costly and has low effect, especially when multi-sensor fusion, the millimeter-wave radar and ultrasonic data are ignored, resulting in poor truth value effect.

Method used

Using the target detection task truth generation method with multi-sensor fusion, the target detection is performed using offline or online models by acquiring multiple sensor data (camera images, lidar point clouds, millimeter wave radar and ultrasonic radar data), combined with timing tracking algorithms Track the target trajectory and introduce manual annotation for correction through multi-sensor cross-validation and scene complexity judgment.

Benefits of technology

It realizes lower cost and higher quality truth value generation, makes full use of historical and future information, and improves detection accuracy and resource utilization efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119024329A_ABST
    Figure CN119024329A_ABST
Patent Text Reader

Abstract

The invention discloses a target detection task truth value generation method and device based on multi-sensor fusion. The method comprises the following steps: acquiring various sensor data; performing target detection on the various sensor data through an offline model or an online model to obtain a corresponding target detection result; based on the obtained target detection result, tracking the target by using a time sequence tracking algorithm to obtain a target trajectory containing historical detection information and prediction information; according to the tracked target trajectory, performing cross validation by using multiple sensors to obtain a fusion detection result after cross validation; and performing scene complexity judgment according to the fusion detection result, and introducing manual annotation based on a scene complexity judgment result to correct the fusion detection result to obtain a corrected true value result. According to the method, historical information and future information are fully utilized by adding a tracking algorithm, so that truth value generation with lower cost and higher quality is realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method and device for generating true values ​​for target detection tasks based on multi-sensor fusion. Background Art

[0002] Environmental perception is the core module of the autonomous driving system. Generally, deep learning technology is used to convert the unstructured data collected by sensors into structured semantic information (for example, image information collected by cameras, point cloud information collected by Lidar, and speed information collected by millimeter-wave radar, etc.), and then converted into structured description information of the surrounding environment (for example, the spatial position of a pedestrian, or which lane a vehicle is traveling on) through a deep learning model. Models that can perceive obstacles such as pedestrians or vehicles that can participate in traffic are generally called target detection models, and this data conversion process is called a target detection task.

[0003] The most important point of deep learning technology is that it requires massive, high-quality manually labeled information (also called true value), such as the real location of surrounding obstacles in the collected data, the current lane line information, etc. In order to reduce the cost of manually labeling these data, these data are generally processed in some offline processing method to generate some automatic labeling results. This offline processing step is called the generation of true value.

[0004] Existing solutions for the true value generation of target detection tasks mainly include the following:

[0005] (1) Use more powerful deep learning models for detection. Since the autonomous driving system needs to run on automotive hardware and has high real-time requirements, the performance of the on-board hardware and software is much worse than that of offline or cloud-based systems. The same data can be processed offline using a more powerful model, and the accuracy of the target detection results can be much better. This can be used as the true value data for training the on-board model without having to worry too much about processing time and hardware performance.

[0006] (2) Use tracking algorithms to optimize the trajectory of obstacles. When running on the vehicle side, some models only detect single-frame data, without time series information, or with short time series information, and cannot obtain future detection information. Therefore, offline, long-term tracking or some other prior information can be used to improve detection performance and generate true value data for training the vehicle side model.

[0007] (3) Use multi-sensor information fusion to obtain more accurate results. Each sensor has its advantages and disadvantages. Sometimes, the vehicle-side model only uses one or several sensors, but not all sensors. In offline processing, the results of all sensors can be used according to the characteristics of each sensor to improve the effect of template detection and generate true value data for training the vehicle-side model.

[0008] In the existing truth value generation schemes for target detection tasks, the method using large models also requires training offline models, which requires more data, and the resources consumed by large model reasoning will also increase a lot. When the amount of training data processed is too large, more resources will be consumed. Fusion of multiple sensors and tracking algorithms to improve the effect of the truth value are often more economical means, and require less computing resources and time costs. However, these methods may require some clever designs, as well as strong a priori assumptions and rich experience. Moreover, multi-sensor fusion schemes often only use image data and lidar point cloud data, and often ignore millimeter-wave radar or ultrasonic data, resulting in low truth value effects.

[0009] Therefore, the prior art needs to be improved. Summary of the invention

[0010] The technical problem to be solved by the present invention is that, in view of the defects of the prior art, the present invention provides a method and device for generating true values ​​for target detection tasks based on multi-sensor fusion, so as to solve the problems of high cost and low true value effect of traditional truth value generation schemes for target detection tasks.

[0011] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0012] In a first aspect, the present invention provides a method for generating true values ​​of target detection tasks based on multi-sensor fusion, comprising:

[0013] Acquire various sensor data;

[0014] Performing target detection on the multiple sensor data through an offline model or an online model to obtain corresponding target detection results;

[0015] Based on the target detection results, the target is tracked using a time series tracking algorithm to obtain a target trajectory containing historical detection information and prediction information;

[0016] According to the target trajectory being tracked, multiple sensors are used for cross-validation to obtain the cross-validated fusion detection results;

[0017] The scene complexity is judged according to the fusion detection result, and manual annotation is introduced based on the scene complexity judgment result to correct the fusion detection result to obtain a corrected true value result.

[0018] In one implementation, the acquiring of multiple sensor data includes:

[0019] The image data captured by the camera, the point cloud data collected by the laser radar, the millimeter wave radar data collected by the millimeter wave radar, and the ultrasonic radar data collected by the ultrasonic radar are acquired to obtain the multiple sensor data.

[0020] In one implementation, performing target detection on the multiple sensor data by using an offline model or an online model to obtain corresponding target detection results includes:

[0021] Performing target detection on the multiple sensor data by using the online model to obtain target detection results corresponding to the online model;

[0022] Or based on the vehicle configuration information, the online model version and the target detection scheme, judging whether the target detection result corresponding to the online model meets the detection task requirements;

[0023] If the detection task requirement is not met, target detection is performed on the multiple sensor data through the offline model to obtain a target detection result corresponding to the offline model.

[0024] In one implementation, based on the obtained target detection result, the target is tracked using a time series tracking algorithm to obtain a target trajectory including historical detection information and prediction information, including:

[0025] The detection results of each sensor or each model are tracked separately, and the timing information is used to delete the misdetected targets or make up for the missed targets, and the prior knowledge of the target position, speed and target size is corrected to obtain the target trajectory containing historical detection information and prediction information.

[0026] In one implementation, the detection results of each sensor or each model are individually tracked, and the misdetected targets are deleted or missed targets are supplemented by using the timing information, and the position, speed and target size prior knowledge of the target are corrected, including:

[0027] For the detection results of the LiDAR point cloud, the global coordinate system of the obstacles between different frames is matched, and the nearest point and the center point are tracked. The initial velocity is obtained by filtering with the Kalman filter, and a more accurate velocity is obtained by the iterative nearest point method.

[0028] For the detection result of the camera image, matching is performed from the two-dimensional detection result of the camera image, and two-dimensional tracking is performed on the camera image by means of an optical flow method, and three-dimensional tracking is performed in a three-dimensional space;

[0029] For millimeter wave radar and ultrasonic radar, correct the speed of the target or the position of objects at close range;

[0030] For the tracking of all sensors, they are tracked in the forward and reverse time order respectively to obtain the target trajectory containing historical detection information and prediction information.

[0031] In one implementation, the method of performing cross-validation using multiple sensors according to the tracked target trajectory to obtain a cross-validated fusion detection result includes:

[0032] Perform timestamp matching based on the tracked target trajectory, find the closest moment between two or more sensors, and perform target trajectory matching at the single-frame level;

[0033] Count the number of successful single-frame matches, and determine whether the target trajectory match is successful based on the counted number;

[0034] If the target trajectory is matched successfully, the successfully matched target trajectory is smoothed by filtering technology, and attribute information of heading angle, target category and target size is added according to the target fusion result to obtain the cross-validated fusion detection result.

[0035] In one implementation, the judging of scene complexity according to the fusion detection result, and introducing manual annotation based on the scene complexity judgment result to correct the fusion detection result to obtain a corrected true value result, include:

[0036] Determine whether there is a difference in the detection results of the same target between different sensors according to the fusion detection result;

[0037] If there is a disagreement, the corresponding scene is determined to be a complex scene, and the detection results of all sensors are re-detected using the offline model, the target is tracked by the time series tracking algorithm, and cross-validation is performed using multiple sensors;

[0038] It is determined again whether there is a disagreement in the detection results of the same target between different sensors. If there is a disagreement again, the manual annotation is introduced to correct the fusion detection result to obtain the corrected true value result.

[0039] In a second aspect, the present invention provides a device for generating a true value of a target detection task based on multi-sensor fusion, comprising:

[0040] Data acquisition module, used to acquire various sensor data;

[0041] A target detection module is used to perform target detection on the multiple sensor data through an offline model or an online model to obtain corresponding target detection results;

[0042] A time series tracking module is used to track the target using a time series tracking algorithm based on the target detection results, and obtain a target trajectory containing historical detection information and prediction information;

[0043] A cross-validation module is used to perform cross-validation using multiple sensors according to the tracked target trajectory to obtain a cross-validated fusion detection result;

[0044] The detection correction module is used to make a scene complexity judgment based on the fusion detection result, and introduce manual annotation based on the scene complexity judgment result to correct the fusion detection result to obtain a corrected true value result.

[0045] In a third aspect, the present invention provides a terminal, comprising: a processor and a memory, wherein the memory stores a target detection task truth value generation program based on multi-sensor fusion, and the target detection task truth value generation program based on multi-sensor fusion is used to implement the operation of the target detection task truth value generation method based on multi-sensor fusion as described in the first aspect when executed by the processor.

[0046] In a fourth aspect, the present invention further provides a medium, which is a computer-readable storage medium, and which stores a truth value generation program for a target detection task based on multi-sensor fusion. When the truth value generation program for a target detection task based on multi-sensor fusion is executed by a processor, it is used to implement the operation of the truth value generation method for a target detection task based on multi-sensor fusion as described in the first aspect.

[0047] The present invention adopts the above technical solution to achieve the following effects:

[0048] The present invention detects targets on multiple sensor data through an offline model or an online model, and can track the target using a time series tracking algorithm based on the target detection result to obtain a target trajectory containing historical detection information and prediction information; and can use multiple sensors to perform cross-validation based on the tracked target trajectory, thereby making a scene complexity judgment based on the fusion detection result, and introducing manual annotation based on the scene complexity judgment result to correct the fusion detection result to obtain a corrected true value result. The present invention fully utilizes historical information and future information by adding a tracking algorithm to achieve lower cost and higher quality true value generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0050] Figure 1 It is a flow chart of a method for generating true values ​​of target detection tasks based on multi-sensor fusion in one implementation of the present invention.

[0051] Figure 2 It is a functional principle diagram of a terminal in one implementation of the present invention.

[0052] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0054] Exemplary Methods

[0055] In the existing truth value generation schemes for target detection tasks, the method using large models also requires training offline models, which requires more data, and the resources consumed by large model reasoning will also increase a lot. When the amount of training data processed is too large, more resources will be consumed. Fusion of multiple sensors and tracking algorithms to improve the effect of the truth value are often more economical means, and require less computing resources and time costs. However, these methods may require some clever designs, as well as strong a priori assumptions and rich experience. Moreover, multi-sensor fusion schemes often only use image data and lidar point cloud data, and often ignore millimeter-wave radar or ultrasonic data, resulting in low truth value effects.

[0056] In view of the above technical problems, an embodiment of the present invention provides a method for generating true values ​​of target detection tasks based on multi-sensor fusion. The method performs target detection on multiple sensor data through an offline model or an online model, and can track the target based on the obtained target detection result using a time series tracking algorithm to obtain a target trajectory containing historical detection information and prediction information; and can use multiple sensors to perform cross-validation based on the tracked target trajectory, so as to make a scene complexity judgment based on the fusion detection result, and introduce manual annotation based on the scene complexity judgment result to correct the fusion detection result to obtain a corrected true value result. The present invention fully utilizes historical information and future information by adding a tracking algorithm to achieve lower cost and higher quality true value generation.

[0057] like Figure 1 As shown, an embodiment of the present invention provides a method for generating true value of target detection task based on multi-sensor fusion, comprising the following steps:

[0058] Step S100, acquiring various sensor data.

[0059] In this embodiment, the results of vehicle-side perception can be directly used as one of the input sources to significantly reduce costs. Only when the scene is very complex and there are many differences between various fusion sensors in the subsequent algorithm, a more powerful offline large model is used to improve the detection accuracy. In very extreme or long-tail scenarios, manual annotation is introduced to correct the automatically generated true value results. Based on the detection results of the vehicle or offline large model, time series tracking is introduced to use historical and future information to solve false detections and missed detections and correct the location of obstacles. At the same time, targeted strategies are specified according to the characteristics of different sensors to optimize the generated true value effect.

[0060] Specifically, in an implementation of this embodiment, step S100 includes the following steps:

[0061] Step S101, acquiring image data captured by a camera, point cloud data collected by a laser radar, millimeter-wave radar data collected by a millimeter-wave radar, and ultrasonic radar data collected by an ultrasonic radar, to obtain the multiple sensor data.

[0062] In this embodiment, the method for generating true value of target detection task based on multi-sensor fusion is mainly divided into the following stages:

[0063] S1, obtains various sensor data and uses them as input for offline models or online models;

[0064] S2, obtain the detection results through the offline model or the online model;

[0065] S3, tracking by time series tracking algorithm;

[0066] S4, cross-validation using multiple sensors;

[0067] S5: Determine whether manual labeling is needed.

[0068] For the above S1 stage, it is necessary to obtain multiple sensor data; among them, the multiple sensors include but are not limited to: cameras (surround view cameras, periscopic cameras), laser radars (Lidar), millimeter wave radars, ultrasonic radars and other sensors; the multiple sensor data obtained include but are not limited to: image data taken by cameras, point cloud data collected by lidars, millimeter wave radar data collected by millimeter wave radars, and ultrasonic radar data collected by ultrasonic radars, etc.

[0069] like Figure 1 As shown, in one implementation of the embodiment of the present invention, the method for generating a true value of a target detection task based on multi-sensor fusion further includes the following steps:

[0070] Step S200: performing target detection on the multiple sensor data through an offline model or an online model to obtain corresponding target detection results.

[0071] In this embodiment, for the various sensor data obtained in the above S1 stage, these sensor data can be used as the input of the perception model in the S2 stage to perform target detection through the perception model; wherein, the perception model includes: an offline model or an online model; that is, after obtaining the data of each sensor, the vehicle-side model (that is, the online model) can be directly used to perform target detection to obtain the target detection result of the vehicle-side model; or target detection can be performed through an offline large model to obtain the target detection result of the offline large model.

[0072] Specifically, in an implementation of this embodiment, step S200 includes the following steps:

[0073] Step S201, performing target detection on the multiple sensor data through the online model to obtain a target detection result corresponding to the online model;

[0074] Step S202, based on the vehicle configuration information, the online model version and the target detection scheme, determining whether the target detection result corresponding to the online model meets the detection task requirements;

[0075] Step S203: If the detection task requirement is not met, target detection is performed on the multiple sensor data using the offline model to obtain a target detection result corresponding to the offline model.

[0076] In this embodiment, after the data of each sensor is obtained, the S2 stage is entered. In this stage, the detection results of the vehicle-side model can be directly reused as the basis for tracking in the subsequent S3 stage. Generally, the collected data will include the detection results of the vehicle-side model, but considering different vehicle configurations, different versions, different solutions and other issues, some sensors may be missing on certain models. For example, some low-end vehicles are not equipped with lidar, or some models are not integrated with driving and parking, and the driving detection results will not run in the parking state. At this time, the offline large model can be used for target detection, and the perception results (target detection results) of the offline large model can be used as the input of the subsequent S3 stage. At the same time, if in the S5 stage, it is judged that the current scene is more complicated, or the differences between the sensors are too large, the offline large model will be used for detection again, and the complete process will be repeated.

[0077] In this embodiment, both the vehicle-side online detection results and the offline large model detection are used as the basis of the true value, which reduces the cost on the one hand, and uses a more powerful offline large model to improve the detection accuracy on the other hand, so that the trajectory of the target can be accurately tracked later.

[0078] like Figure 1 As shown, an embodiment of the present invention provides a method for generating true value of target detection task based on multi-sensor fusion, comprising the following steps:

[0079] Step S300, based on the obtained target detection result, the target is tracked using a time series tracking algorithm to obtain a target trajectory including historical detection information and prediction information.

[0080] In this embodiment, for the target detection results corresponding to the online model and the offline model, in the S3 stage, a time series tracking algorithm can be used to track the target to obtain long-term historical information and future information, and the long-term historical information and future information can be used to track the detection results, thereby improving the quality of the true value.

[0081] Specifically, in an implementation of this embodiment, step S300 includes the following steps:

[0082] Step S301, individually track the detection results of each sensor or each model, use the timing information to delete the misdetected targets or make up for the missed targets, and correct the target position, speed and target size prior knowledge to obtain the target trajectory containing historical detection information and prediction information.

[0083] In this embodiment, the target detection results of each sensor or each model will be tracked separately in the S3 stage, and the timing tracking algorithm will use the timing information to delete the falsely detected targets or make up for the missed targets. Especially for some single-frame detection models, it is easy to have falsely detected targets or missed targets. Even for multi-frame detection models, only a small number of historical results will be used, and long-term historical results will not be used, let alone future detection results. Similarly, multi-frame detection models will also have falsely detected targets or missed targets. In addition to using timing information to delete falsely detected targets or make up for missed targets, the timing tracking algorithm will also correct the position and speed of the object during the tracking process, as well as relatively strong prior knowledge of the object size.

[0084] Specifically, in an implementation of this embodiment, step S301 includes the following steps:

[0085] Step S301a, for the detection results of the laser radar point cloud, match them according to the global coordinate system of the obstacle between different frames, track them according to the closest point and the center point, use Kalman filtering to obtain the initial speed, and use the iterative closest point method to obtain a more accurate speed;

[0086] Step S301b, matching the detection result of the camera image from the two-dimensional detection result of the camera image, and performing two-dimensional tracking on the camera image by using an optical flow method, and performing three-dimensional tracking in a three-dimensional space;

[0087] Step S301c, for millimeter wave radar and ultrasonic radar, correct the speed of the target or the position of the object at a close distance;

[0088] Step S301d: All sensors are tracked in chronological order and reverse chronological order respectively to obtain a target trajectory including historical detection information and prediction information.

[0089] In this embodiment, when the target is tracked using a time-series tracking algorithm, in the part of tracking the lidar point cloud perception results, target matching can be performed according to the global coordinate system of the obstacle between different frames, the center can be corrected according to the visible edge of the point cloud, and the target can be tracked at the nearest point and the center point at the same time. The initial velocity can be obtained by filtering using a Kalman filter, and a more accurate velocity can be obtained using the iterative closest point (ICP) method.

[0090] For the tracking of camera image perception results, target matching can be performed from the two-dimensional detection results on the image, and optical flow (for example, optical flow method) or other similar methods can be used to perform two-dimensional target tracking on the image, and three-dimensional target tracking can also be performed in three-dimensional space; wherein, the optical flow method includes but is not limited to: positioning algorithms including those based on visual SLAM algorithms.

[0091] For millimeter-wave radar and ultrasonic radar, only simple tracking is performed, because the main function of these two sensors is to correct the speed of objects or the position of objects at close range.

[0092] It is worth mentioning that in this embodiment, all the above tracking will be tracked in the forward and reverse order of time to ensure that both historical and future information are used, so as to obtain the target trajectory containing historical detection information and prediction information.

[0093] like Figure 1 As shown, an embodiment of the present invention provides a method for generating true value of target detection task based on multi-sensor fusion, comprising the following steps:

[0094] Step S400 , cross-validation is performed using multiple sensors according to the tracked target trajectory to obtain a cross-validated fusion detection result.

[0095] In this embodiment, for the tracking results in stage S3, in stage S4, a perception result is obtained by integrating the characteristics of different sensors. For example, the telephoto lens is responsible for very far targets, the fisheye lens is responsible for very close targets, the laser radar is responsible for correcting the position of the object, the millimeter wave radar is responsible for giving a more accurate target speed, and the ultrasonic wave gives a more accurate position of nearby objects, etc. The cross-validation of each sensor includes trajectory tracking, which can greatly improve the correctness, position accuracy, speed accuracy, etc. of the detected objects.

[0096] Specifically, in an implementation of this embodiment, step S400 includes the following steps:

[0097] Step S401, performing timestamp matching according to the tracked target trajectory, finding the closest time between two or more sensors, and performing target trajectory matching at a single frame level;

[0098] Step S402, counting the number of successful single-frame matches, and judging whether the target trajectory match is successful according to the counted number;

[0099] Step S403, if the target trajectory is matched successfully, the successfully matched target trajectory is smoothed by filtering technology, and attribute information of heading angle, target category and target size is added according to the target fusion result to obtain the cross-validated fusion detection result.

[0100] In this embodiment, for the trajectories of each object generated in the S3 stage, matching can be done from the granularity of the trajectory, so that many mismatching situations can be avoided through the timing information itself. The process mainly includes: first doing timestamp matching, finding the closest moment between two or more sensors, and then doing single-frame level matching, counting the number of single-frame matches and judging whether the trajectory matching is successful. If it is a one-to-many match, then the proportion of matches with a certain trajectory is greater than a certain threshold and is considered successful. Different tracking trajectories can be further filtered, for example, using filtering techniques such as RTS Smoother to further improve the smoothness of the trajectory. At the same time, more robust attributes are given according to the fusion results of the target, for example, attribute information such as heading angle, target category, and target size is added.

[0101] In some scenarios, for targets with sufficiently long trajectories, it is also necessary to determine whether the target is stationary. If it is stationary, the position accuracy of the target can be further optimized by using the invariance characteristics of the global coordinates.

[0102] In this embodiment, long-term historical and future information is used to track the detection results to improve the quality of the true value, and the complementary characteristics between different sensors are used to improve the sensor quality.

[0103] like Figure 1 As shown, an embodiment of the present invention provides a method for generating true value of target detection task based on multi-sensor fusion, comprising the following steps:

[0104] Step S500, performing scene complexity judgment according to the fusion detection result, and introducing manual annotation based on the scene complexity judgment result to correct the fusion detection result to obtain a corrected true value result.

[0105] In this embodiment, by detecting the complexity of the scene, manual annotation is introduced when necessary to improve the quality of automatic annotation. For example, in very extreme or long-tail scenes, manual annotation is introduced to correct the automatically generated true value results.

[0106] Specifically, in an implementation of this embodiment, step S500 includes the following steps:

[0107] Step S501, judging whether there is a difference in the detection results of the same target between different sensors according to the fusion detection result;

[0108] Step S502: if there is a disagreement, the corresponding scene is determined to be a complex scene, and the detection results of all sensors are re-detected using the offline model, the target is tracked by the time series tracking algorithm, and cross-validation is performed using multiple sensors;

[0109] Step S503, again determining whether there is a disagreement in the detection results of the same target between different sensors. If there is a disagreement again, the manual annotation is introduced to correct the fusion detection result to obtain the corrected true value result.

[0110] In this embodiment, in stage S5, based on the results of stage S4, it is determined whether the data scene is very complex, resulting in differences in the detection of many objects between different sensors, as well as very long-tail and extreme scenes, such as a scene with a large number of pedestrians crossing the road. If the scene is judged to be more complex, it is possible to return to stage S2 and use the offline large model to re-detect the results of all sensors, and repeat the above steps. When judging again, if the differences between sensors are still large, manual annotation can be introduced to correct the true value results.

[0111] This embodiment achieves the following technical effects through the above technical solution:

[0112] This embodiment uses an offline model or an online model to detect targets on multiple sensor data, and can track the target using a time series tracking algorithm based on the target detection results to obtain a target trajectory containing historical detection information and prediction information; and can use multiple sensors to perform cross-validation based on the tracked target trajectory, so as to make a scene complexity judgment based on the fusion detection results, and introduce manual annotation based on the scene complexity judgment results to correct the fusion detection results to obtain the corrected true value results. This embodiment fully utilizes historical information and future information by adding a tracking algorithm to achieve lower cost and higher quality true value generation.

[0113] Exemplary Devices

[0114] Based on the above embodiments, the present invention further provides a device for generating true value of target detection task based on multi-sensor fusion, comprising:

[0115] Data acquisition module, used to acquire various sensor data;

[0116] A target detection module is used to perform target detection on the multiple sensor data through an offline model or an online model to obtain corresponding target detection results;

[0117] A time series tracking module is used to track the target using a time series tracking algorithm based on the target detection results, and obtain a target trajectory containing historical detection information and prediction information;

[0118] A cross-validation module is used to perform cross-validation using multiple sensors according to the tracked target trajectory to obtain a cross-validated fusion detection result;

[0119] The detection correction module is used to make a scene complexity judgment based on the fusion detection result, and introduce manual annotation based on the scene complexity judgment result to correct the fusion detection result to obtain a corrected true value result.

[0120] This embodiment achieves the following technical effects through the above technical solution:

[0121] This embodiment uses an offline model or an online model to detect targets on multiple sensor data, and can track the target using a time series tracking algorithm based on the target detection results to obtain a target trajectory containing historical detection information and prediction information; and can use multiple sensors to perform cross-validation based on the tracked target trajectory, so as to make a scene complexity judgment based on the fusion detection results, and introduce manual annotation based on the scene complexity judgment results to correct the fusion detection results to obtain the corrected true value results. This embodiment fully utilizes historical information and future information by adding a tracking algorithm to achieve lower cost and higher quality true value generation.

[0122] Based on the above embodiment, the present invention further provides a terminal, whose principle block diagram can be as follows: Figure 2 shown.

[0123] The terminal includes: a processor, a memory, an interface, a display screen and a communication module connected via a system bus; wherein the processor of the terminal is used to provide computing and control capabilities; the memory of the terminal includes a storage medium and an internal memory; the storage medium stores an operating system and a computer program; the internal memory provides an environment for the operation of the operating system and the computer program in the storage medium; the interface is used to connect to external devices; the display screen is used to display corresponding information; and the communication module is used to communicate with a cloud server or other devices.

[0124] When the computer program is executed by a processor, it is used to implement the operation of a method for generating true values ​​for target detection tasks based on multi-sensor fusion.

[0125] It can be understood by those skilled in the art that Figure 2 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal to which the scheme of the present invention is applied. The specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0126] In one embodiment, a terminal is provided, which includes: a processor and a memory, the memory storing a target detection task truth value generation program based on multi-sensor fusion, and the target detection task truth value generation program based on multi-sensor fusion is used to implement the operation of the above-mentioned target detection task truth value generation method based on multi-sensor fusion when executed by the processor.

[0127] In one embodiment, a storage medium is provided, wherein the storage medium stores a target detection task truth value generation program based on multi-sensor fusion, and when the target detection task truth value generation program based on multi-sensor fusion is executed by a processor, it is used to implement the operation of the above-mentioned target detection task truth value generation method based on multi-sensor fusion.

[0128] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and volatile memory.

[0129] In summary, the present invention provides a method and device for generating true values ​​of target detection tasks based on multi-sensor fusion, the method comprising: acquiring multiple sensor data; performing target detection on the multiple sensor data through an offline model or an online model to obtain corresponding target detection results; based on the obtained target detection results, tracking the target using a time series tracking algorithm to obtain a target trajectory containing historical detection information and prediction information; cross-validating using multiple sensors according to the tracked target trajectory to obtain a cross-validated fusion detection result; judging the scene complexity according to the fusion detection result, and introducing manual annotation based on the scene complexity judgment result to correct the fusion detection result to obtain a corrected true value result. The present invention fully utilizes historical information and future information by adding a tracking algorithm to achieve lower cost and higher quality true value generation.

[0130] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for generating true value of target detection task based on multi-sensor fusion, characterized in that: include: Acquire various sensor data; Performing target detection on the multiple sensor data through an offline model or an online model to obtain corresponding target detection results; Based on the target detection results, the target is tracked using a time series tracking algorithm to obtain a target trajectory containing historical detection information and prediction information; According to the target trajectory being tracked, multiple sensors are used for cross-validation to obtain the cross-validated fusion detection results; The scene complexity is judged according to the fusion detection result, and manual annotation is introduced based on the scene complexity judgment result to correct the fusion detection result to obtain a corrected true value result.

2. The method for generating true value of target detection task based on multi-sensor fusion according to claim 1 is characterized in that: The acquisition of multiple sensor data includes: The image data captured by the camera, the point cloud data collected by the laser radar, the millimeter wave radar data collected by the millimeter wave radar, and the ultrasonic radar data collected by the ultrasonic radar are acquired to obtain the multiple sensor data.

3. The method for generating true value of target detection task based on multi-sensor fusion according to claim 1 is characterized in that: The performing target detection on the multiple sensor data by using an offline model or an online model to obtain corresponding target detection results includes: Performing target detection on the multiple sensor data by using the online model to obtain target detection results corresponding to the online model; Or based on the vehicle configuration information, the online model version and the target detection scheme, judging whether the target detection result corresponding to the online model meets the detection task requirements; If the detection task requirement is not met, target detection is performed on the multiple sensor data through the offline model to obtain a target detection result corresponding to the offline model.

4. The method for generating true value of target detection task based on multi-sensor fusion according to claim 1, characterized in that: Based on the obtained target detection result, the target is tracked using a time series tracking algorithm to obtain a target trajectory containing historical detection information and prediction information, including: The detection results of each sensor or each model are tracked separately, and the timing information is used to delete the misdetected targets or make up for the missed targets, and the prior knowledge of the target position, speed and target size is corrected to obtain the target trajectory that contains historical detection information and prediction information.

5. The method for generating true value of target detection task based on multi-sensor fusion according to claim 1, characterized in that: The detection results of each sensor or each model are tracked separately, and the misdetected targets are deleted or missed targets are supplemented by using the time series information, and the position, speed and target size prior knowledge of the target are corrected, including: For the detection results of the LiDAR point cloud, the global coordinate system of the obstacles between different frames is matched, and the nearest point and the center point are tracked. The initial velocity is obtained by filtering with the Kalman filter, and a more accurate velocity is obtained by the iterative nearest point method. For the detection result of the camera image, matching is performed from the two-dimensional detection result of the camera image, and two-dimensional tracking is performed on the camera image by means of an optical flow method, and three-dimensional tracking is performed in a three-dimensional space; For millimeter wave radar and ultrasonic radar, correct the speed of the target or the position of objects at close range; For the tracking of all sensors, they are tracked in the forward and reverse time order respectively to obtain the target trajectory containing historical detection information and prediction information.

6. The method for generating true value of target detection task based on multi-sensor fusion according to claim 1, characterized in that: The method uses a variety of sensors to perform cross-validation based on the target trajectory being tracked to obtain a cross-validated fusion detection result, including: Perform timestamp matching based on the tracked target trajectory, find the closest moment between two or more sensors, and perform target trajectory matching at the single-frame level; Count the number of successful single-frame matches, and determine whether the target trajectory match is successful based on the counted number; If the target trajectory is matched successfully, the successfully matched target trajectory is smoothed by filtering technology, and attribute information of heading angle, target category and target size is added according to the target fusion result to obtain the cross-validated fusion detection result.

7. The method for generating true value of target detection task based on multi-sensor fusion according to claim 1, characterized in that: The judging the scene complexity according to the fusion detection result, and introducing manual annotation based on the scene complexity judgment result to correct the fusion detection result to obtain a corrected true value result, including: Determine whether there is a difference in the detection results of the same target between different sensors according to the fusion detection result; If there is a disagreement, the corresponding scene is determined to be a complex scene, and the detection results of all sensors are re-detected using the offline model, the target is tracked by the time series tracking algorithm, and cross-validation is performed using multiple sensors; It is determined again whether there is a disagreement in the detection results of the same target between different sensors. If there is a disagreement again, the manual annotation is introduced to correct the fusion detection result to obtain the corrected true value result.

8. A device for generating true value of target detection task based on multi-sensor fusion, characterized in that: include: Data acquisition module, used to acquire various sensor data; A target detection module is used to perform target detection on the multiple sensor data through an offline model or an online model to obtain corresponding target detection results; A time series tracking module is used to track the target using a time series tracking algorithm based on the target detection results, and obtain a target trajectory containing historical detection information and prediction information; A cross-validation module is used to perform cross-validation using multiple sensors according to the tracked target trajectory to obtain a cross-validated fusion detection result; The detection correction module is used to make a scene complexity judgment based on the fusion detection result, and introduce manual annotation based on the scene complexity judgment result to correct the fusion detection result to obtain a corrected true value result.

9. A terminal, characterized in that: include: A processor and a memory, wherein the memory stores a target detection task truth generation program based on multi-sensor fusion, and when the target detection task truth generation program based on multi-sensor fusion is executed by the processor, it is used to implement the operation of the target detection task truth generation method based on multi-sensor fusion as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a target detection task truth value generation program based on multi-sensor fusion, and when the target detection task truth value generation program based on multi-sensor fusion is executed by a processor, it is used to implement the operation of the target detection task truth value generation method based on multi-sensor fusion as described in any one of claims 1-7.