Multi-sensor data fusion method, multi-source fusion perception system and computer device

Abstract

The application discloses a multi-sensing data fusion method, a multi-source fusion perception system, computer equipment and a storage medium. Point cloud data and video data under the same time and the same scene are acquired. Target detection is performed on the point cloud data to obtain a first detection result, and target detection is performed on the video data to obtain a second detection result. The first detection result and the second detection result are fused to obtain an initial fusion result. It is judged whether the initial fusion result needs to be corrected. If the judgment result is that the initial fusion result needs to be corrected, the initial fusion result is corrected according to a correction type of the judgment result, and a data fusion result is output. The correction type includes multi-target misjudgment correction, confidence correction or trajectory correction. Through the above method, more accurate target information is obtained, and the accuracy of target detection is improved.

Description

Technical Field

[0001] This disclosure relates to the field of road detection technology, and in particular to a multi-sensor data fusion method, a multi-source fusion sensing system, a computer device, and a storage medium. Background Technology

[0002] As the number of vehicles in cities increases, it becomes more and more difficult to monitor the driving process. In practical applications, it is often necessary to monitor objects on the road in order to obtain target information about those objects.

[0003] In existing technologies, the methods for monitoring targets on roads generally involve installing cameras on the road, continuously capturing video data of the area to be monitored, and then analyzing and processing the video data to obtain detection information of the targets in the video data.

[0004] However, when using sensing data collected by a single sensor for target detection, false detections or false alarms are prone to occur, and these problems cannot be corrected, resulting in low accuracy of the detection results. Summary of the Invention

[0005] Therefore, it is necessary to provide a multi-sensor data fusion method, a multi-source fusion sensing system, a computer device, and a storage medium to address the aforementioned problem of missed detection.

[0006] A multi-sensor data fusion method includes:

[0007] Acquire point cloud data and video data from the same moment and the same scene;

[0008] The first detection result is obtained by performing object detection on point cloud data, and the second detection result is obtained by performing object detection on video data.

[0009] The first and second detection results are fused to obtain an initial fusion result;

[0010] Determine whether the initial fusion result needs to be corrected. If the result indicates that correction is needed, correct the initial fusion result according to the correction type of the result and output the data fusion result. The correction type includes: multi-target misjudgment correction or trajectory correction.

[0011] In one embodiment, the first detection result and the second detection result are fused to obtain an initial fusion result, including:

[0012] Based on the extrinsic parameter matrix, the point cloud targets in the first detection result are transformed from the point cloud coordinate system to the three-dimensional coordinate system, and based on the intrinsic parameter matrix, the point cloud targets are transformed from the three-dimensional coordinate system to the two-dimensional coordinate system where the video targets in the second detection result are located;

[0013] The target objects in the initial fusion result are determined based on the point cloud targets and video targets in the two-dimensional coordinate system; among them, point cloud targets and video targets with an information matching probability greater than the matching threshold are considered to be the same target object, and the information matching probability is used to characterize the positional overlap between point cloud targets and video targets.

[0014] In one embodiment, the correction type is multi-target misjudgment correction, and the initial fusion result is corrected according to the correction type of the judgment result, including:

[0015] If in the initial fusion result there is a situation where the information matching probability of a point cloud target corresponding to a target object is greater than the first matching threshold with multiple video targets, then the multiple video targets in the initial fusion result will be marked as different target objects respectively;

[0016] If in the initial fusion result there is a situation where the information matching probability of one video target corresponding to a target object is greater than the second matching threshold with multiple point cloud targets, then the multiple point cloud targets in the initial fusion result will be marked as different target objects.

[0017] In one embodiment, the correction type is confidence correction, and the initial fusion result is corrected according to the correction type of the judgment result, including:

[0018] The initial fusion result includes the first confidence level and the second confidence level of the target object. The first confidence level is the confidence level of the target object category determined based on the point cloud data, and the second confidence level is the confidence level of the target object category determined based on the video data.

[0019] The first and second confidence levels of the target object are transformed, and the category corresponding to the larger confidence level of the two transformed values ​​is taken as the category of the target object.

[0020] In one embodiment, the correction type is target color correction, and the initial fusion result is corrected according to the correction type of the judgment result, including:

[0021] The detection method checks whether the target objects included in the initial fusion result have corresponding video targets in the second detection result.

[0022] If the target object included in the initial fusion result has a corresponding video target in the second detection result, then the color information of the corresponding video target is obtained based on the video data, and the color information is used as the color information of the target object.

[0023] In one embodiment, the correction type is a missed detection correction, and the initial fusion result is corrected according to the correction type of the judgment result, including:

[0024] If the target object included in the initial fusion result has a corresponding point cloud target in the first detection result but no corresponding video target in the second detection result, then the first detection result of the point cloud target is mapped to the two-dimensional coordinate system where the video target is located in the second detection result, and used as the second detection result of the target object;

[0025] If the target object included in the initial fusion result has a corresponding video target in the second detection result but no corresponding point cloud target in the first detection result, then the second detection result of the video target is transformed into the point cloud coordinate system where the point cloud target in the first detection result is located through the mutual learning model, and is used as the first detection result of the target object.

[0026] In one embodiment, the correction type is false detection correction, and the initial fusion result is corrected according to the correction type of the judgment result, including:

[0027] The first confidence level of the target objects included in the initial fusion result is obtained. The first confidence level is the confidence level of the target object category determined based on the point cloud data.

[0028] If the first confidence level is less than the confidence threshold, obtain the area of ​​the detection box corresponding to the target object in the first detection result included in the initial fusion result;

[0029] If the area of ​​the detection box is less than the area threshold, the target object is removed from the initial fusion result.

[0030] In one embodiment, the correction type is trajectory correction, and the initial fusion result is corrected according to the correction type of the judgment result, including:

[0031] The initial fusion results include the trajectory direction of the point cloud target corresponding to the target object and the trajectory direction of the corresponding video target.

[0032] If the trajectory direction of the point cloud target changes abruptly, while the trajectory direction of the video target does not change abruptly, the trajectory direction of the target included in the initial fusion result is determined based on the trajectory direction of the video target.

[0033] In one embodiment, the point cloud data is obtained by detecting the area to be measured using roadside lidar.

[0034] In one embodiment, the correction type also includes one or more of the following: confidence correction, target color correction, missed detection correction, and false detection correction.

[0035] A multi-source fusion sensing system, comprising:

[0036] The data acquisition module is used to acquire point cloud data and video data from the same time and scene.

[0037] The target detection module is used to perform target detection on point cloud data to obtain a first detection result, and to perform target detection on video data to obtain a second detection result;

[0038] The target fusion module is used to fuse the first detection result and the second detection result to obtain an initial fusion result;

[0039] The result correction module is used to determine whether the initial fusion result needs to be corrected. If the result indicates that correction is needed, the initial fusion result is corrected according to the correction type of the result, and the data fusion result is output. The correction types include: multi-target misjudgment correction or trajectory correction.

[0040] A computer device includes a memory and a processor, the memory storing a computer program that, when executed by the processor, performs the following steps:

[0041] Acquire point cloud data and video data from the same moment and the same scene;

[0042] The first detection result is obtained by performing object detection on point cloud data, and the second detection result is obtained by performing object detection on video data.

[0043] The first and second detection results are fused to obtain an initial fusion result;

[0044] Determine whether the initial fusion result needs to be corrected. If the result indicates that correction is needed, correct the initial fusion result according to the correction type of the result and output the data fusion result. The correction type includes: multi-target misjudgment correction or trajectory correction.

[0045] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0046] Acquire point cloud data and video data from the same moment and the same scene;

[0047] The first detection result is obtained by performing object detection on point cloud data, and the second detection result is obtained by performing object detection on video data.

[0048] The first and second detection results are fused to obtain an initial fusion result;

[0049] Determine whether the initial fusion result needs to be corrected. If the result indicates that correction is needed, correct the initial fusion result according to the correction type of the result and output the data fusion result. The correction type includes: multi-target misjudgment correction or trajectory correction.

[0050] The aforementioned multi-sensor data fusion method, multi-source fusion sensing system, computer equipment, and storage medium acquire point cloud data and video data from the same time and scene, perform target detection separately, and then fuse the first detection result obtained from the point cloud data and the second detection result obtained from the video data. The initial fusion result is corrected using the data characteristics of the detection results obtained from sensing data collected by different sensors, resulting in a more accurate target information output, thus improving the accuracy of target detection. Simultaneously, the initial fusion result combines the first detection result obtained from the point cloud data and the second detection result obtained from the video data. Thus, if a target appears in the first detection result but not in the second detection result, or vice versa, the process of correcting the initial fusion result can complement the two detection results, effectively avoiding missed detections. Attached Figure Description

[0051] Figure 1 This is a schematic diagram illustrating an application scenario of a multi-sensor data fusion method in one embodiment;

[0052] Figure 2 This is a flowchart illustrating a multi-sensor data fusion method in one embodiment;

[0053] Figure 3 This is a schematic diagram of the process of fusing the first detection result and the second detection result in one embodiment;

[0054] Figure 4 This is a flowchart illustrating the multi-target misjudgment correction process in one embodiment;

[0055] Figure 5 This is a schematic diagram of the confidence level correction process in one embodiment;

[0056] Figure 6 This is a schematic diagram of the target color correction process in one embodiment;

[0057] Figure 7 This is a schematic diagram of the process for correcting missed detections in one embodiment;

[0058] Figure 8 This is a schematic diagram of the false detection correction process in one embodiment;

[0059] Figure 9 This is a schematic diagram of the trajectory correction process in one embodiment;

[0060] Figure 10 This is a block diagram of a multi-source fusion sensing system in one embodiment;

[0061] Figure 11This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0063] As the number of vehicles in cities increases, it becomes more and more difficult to monitor the driving process. In practical applications, it is often necessary to monitor objects on the road in order to obtain target information about those objects.

[0064] In existing technologies, the methods for monitoring targets on roads generally involve installing cameras on the road, continuously capturing video data of the area to be monitored, and then analyzing and processing the video data to obtain detection information of the targets in the video data, such as their category, location, and size.

[0065] However, using perception data acquired by a single sensor for target detection is prone to false detections or missed detections, and these issues cannot be corrected, resulting in low accuracy. For example, using video data acquired by a camera provides more complete information on the texture and color of the target object and has better accuracy in identifying the object category, but cameras cannot acquire depth positioning information and are greatly affected by lighting conditions. On the other hand, using point cloud data acquired by LiDAR provides more complete information on the target object's position, size, velocity, acceleration, and heading angle, and has high detection accuracy, but its ability to identify the object category is poor.

[0066] Therefore, it can be seen that when using sensing data collected by different types of sensors for target detection, some of the detection results have high accuracy, while others have low accuracy.

[0067] The multi-sensor data fusion method provided in this application can be applied to a multi-source fusion sensing system, also known as a "smart base station." This system can be installed on both sides of a road, collecting sensing data within its coverage area using different types of sensors. Target detection is then performed on the collected sensing data, yielding multiple detection results. These multiple detection results are then fused and corrected according to a preset fusion strategy to obtain the final target detection result. In this embodiment, point cloud data and video data collected by different sensors are used, and target detection is performed separately. The detection results from the point cloud data and the video data are fused. Furthermore, the accuracy characteristics of the detection results obtained from the sensing data collected by different sensors are utilized to correct the initial fusion result, outputting the data fusion result as the final target detection result. This provides more accurate target information, thereby improving the accuracy of target detection.

[0068] Furthermore, in this embodiment, target detection is performed on point cloud data and video data at the same time and in the same scene, respectively, resulting in point cloud detection results and video detection results. If a target object is not detected in the point cloud detection result but is detected in the video detection result, the video detection result for that target object is used as the final target detection result. Similarly, if a target object is not detected in the video detection result but is detected in the point cloud detection result, the point cloud detection result for that target object is used as the final target detection result. This complementary use of the two detection results effectively avoids the problem of missed detections.

[0069] The technical solutions involved in the embodiments of this application will be described below in conjunction with the application environment in which the embodiments of this application are applied.

[0070] The multi-sensor data fusion method provided in this application is applied to multi-source fusion sensing systems (smart base stations), such as... Figure 1 As shown, the multi-source fusion perception system includes a radar 101, a camera 102, and a server 103. The radar 101 and camera 102 communicate with the server 103 via wired or wireless means. The server 103 acquires point cloud data and video data from the radar 101 and camera 102 at the same time and in the same scene. It performs target detection on the point cloud data to obtain a first detection result, and performs target detection on the video data to obtain a second detection result. The first and second detection results are then fused to obtain an initial fusion result. The server then determines whether the initial fusion result needs correction. If correction is required, the initial fusion result is corrected according to the correction type, and the data fusion result is output. The correction types include one or more of the following: multi-target misjudgment correction, confidence correction, target color correction, missed detection correction, false detection correction, and trajectory correction.

[0071] The radar 101 and camera 102 are mounted on traffic poles or crossbars along the roadside. Optionally, the positional offset between the radar 101 and camera 102 is less than an offset threshold, making the radar 101 and camera 102 approximately installed in the same location. The server 103 can be a single server or a server cluster consisting of multiple servers.

[0072] In one embodiment, such as Figure 2 As shown, a multi-sensor data fusion method is provided, which can be applied to... Figure 1 Taking the server in the example, the following steps are included:

[0073] S210: Acquire point cloud data and video data at the same time and in the same scene.

[0074] The server acquires point cloud data via radar and video data via cameras.

[0075] Optionally, the radar can be an 8-line, 16-line, 24-line, 32-line, 64-line, or 128-line lidar, or a millimeter-wave radar; the camera can be a bullet camera, a dome camera, or a spherical camera.

[0076] Specifically, the point cloud data is obtained by the roadside lidar detecting the area to be measured. The server pre-processes the lidar and camera to synchronize the sampling frequency and time, so that the lidar and camera can respectively acquire point cloud data and video data at the same time and in the same scene.

[0077] Optionally, after determining the relative positions and angles of the LiDAR and camera on the roadside, the sampling frequencies of the LiDAR and camera are adjusted to make them the same or approximately the same. For example, the difference between the sampling frequencies of the LiDAR and camera is adjusted to be less than a frequency threshold. Further time synchronization processing is performed on the LiDAR and camera, acquiring a set of sampling timestamps t1 and t2 accurate to milliseconds in real time for time matching (if they cannot be directly acquired, they can be converted to timestamps on the same time axis through time axis matching, with the reference time axis being t, and the conversion time t' = t - Δt based on the time axis conversion difference Δt). Specifically, the absolute value of the difference between sampling timestamps t1 and t2, |tx - ty|, is calculated to see if it is less than a set fixed value δ, for example, δ = 10ms. If |tx - ty| < δ, the point cloud data and video data corresponding to sampling timestamps t1 and t2 are considered to have been acquired at the same time; if |tx - ty| > δ, the next set of sampling timestamps is searched for for time matching at a certain rate. It should be noted that other methods can also be used for time synchronization, and this application does not limit this approach.

[0078] S220. Perform target detection on the point cloud data to obtain the first detection result, and perform target detection on the video data to obtain the second detection result.

[0079] The first detection result includes detection information such as the position, size, velocity, acceleration, heading angle, and category of point cloud targets within the coverage area of ​​the lidar. Point cloud targets are target objects obtained by target detection of point cloud data. The second detection result includes detection information such as the category, color, texture, position, and velocity of video targets within the coverage area of ​​the camera. Video targets are target objects obtained by target detection of video data.

[0080] Specifically, the server can use pre-trained machine learning models to perform object detection on point cloud data and video data respectively, obtaining a first detection result for the point cloud data and a second detection result for the video data. For example, the SECOND deep learning model can be used for object detection on the point cloud data to obtain the first detection result, while the YOLOv3 deep learning model can be used for object detection on the video data to obtain the second detection result.

[0081] Optionally, the server can also use tracking algorithms such as the Kalman algorithm to process the point cloud data and / or video data to obtain the first detection result and / or the second detection result.

[0082] S230. The first detection result and the second detection result are fused to obtain the initial fusion result.

[0083] Both the first and second detection results include detection information for at least one target object. For example, the location of the target object.

[0084] Optionally, the server can fuse the first detection result and the second detection result based on the position of the point cloud target in the first detection result and the position of the video target in the second detection result to obtain an initial fusion result.

[0085] Specifically, the server obtains the distance between the position of each point cloud target in the first detection result and the position of each video target in the second detection result, and determines the point cloud targets in the first detection result and the video targets in the second detection result whose distances are less than a preset distance threshold as the same target. If there is a point cloud target in the first detection result whose distances to all video targets in the second detection result are greater than the preset distance threshold, then the point cloud target in the first detection result is treated as a separate target. Similarly, if there is a video target in the second detection result whose distances to all point cloud targets in the first detection result are greater than the preset distance threshold, then the video target in the second detection result is treated as a separate target, thereby obtaining the target in the initial fusion result. For example, the first detection result includes A1 to A5, 5 point cloud targets, and the second detection result includes B1 to B6, 6 video targets. The distances between point cloud target A2 and video target B2, between point cloud target A3 and video target B3, and between point cloud target A5 and video target B5 are all less than a preset distance threshold. The server then determines point cloud target A2 and video target B2 as the same target, point cloud target A3 and video target B3 as the same target, and A5 and video target B5 as the same target. Point cloud target A1, point cloud target A4, video target B1, video target B4, and video target B6 are determined as individual targets. The target targets in the final initial fusion result include A1, A2 / B2, A3 / B3, A4, B4, A5 / B5, and A6 / B6, corresponding to the first detection result and / or the second detection result for each target.

[0086] S240. Determine whether the initial fusion result needs to be corrected. If the result indicates that correction is needed, correct the initial fusion result according to the correction type of the result and output the data fusion result.

[0087] The correction types include one or more of the following: multi-target misjudgment correction, confidence correction, target color correction, missed detection correction, false detection correction, and trajectory correction.

[0088] Specifically, the server can determine whether the initial fusion result needs to be corrected and the corresponding correction type based on the first and second detection results of the target objects in the initial fusion result. Then, it can make targeted corrections to the initial fusion result based on the determined correction type and finally output the data fusion result.

[0089] Optionally, for a target that has both a first detection result and a second detection result in the initial fusion result, the server can determine whether it is necessary to perform multi-target misjudgment correction on the initial fusion result based on the quantitative relationship between the first detection result and the second detection result of the target.

[0090] Optionally, for a target object that has both a first detection result and a second detection result in the initial fusion result, the server can determine whether the confidence level of the target object is the same in the first detection result and the second detection result to determine whether the confidence level of the target object is the same.

[0091] Optionally, the server can determine whether target color correction is needed for the initial fusion result based on whether the target object in the initial fusion result has the second detection result and whether the second detection result has the color information of the target object.

[0092] Optionally, the server can determine whether to correct for missed detections in the initial fusion result based on whether the target in the initial fusion result has both the first detection result and the second detection result.

[0093] Optionally, for a target object that has a first detection result in the initial fusion result, the server can determine whether to perform false detection correction on the initial fusion result based on the size of the detection box obtained by the target object in the first detection result.

[0094] Optionally, for a target object that has a first detection result in the initial fusion result, the server can determine whether to perform false detection correction on the initial fusion result based on the size of the detection box obtained by the target object in the first detection result.

[0095] Optionally, for a target object that simultaneously possesses both the first and second detection results in the initial fusion result, the server can determine whether trajectory correction is needed for the initial fusion result based on the driving trajectory obtained from the first and second detection results of the target object.

[0096] In this embodiment, the method employs perception data collected by different types of sensors for target detection. Specifically, it utilizes point cloud data and video data collected by LiDAR and cameras for target detection. The method then fuses the first detection result obtained from the point cloud data and the second detection result obtained from the video data. The initial fusion result is corrected by leveraging the accuracy characteristics of the detection results obtained from perception data collected by different sensors, resulting in a more accurate target information output, thus improving the accuracy of target detection. Simultaneously, the initial fusion result combines the first detection result obtained from the point cloud data and the second detection result obtained from the video data. This means that if a target appears in the first detection result but not in the second detection result, or vice versa, the process of correcting the initial fusion result can complement the two detection results, effectively avoiding missed detections.

[0097] In one embodiment, to improve the accuracy of fusing the first and second detection results, such as Figure 3 As shown, the above S230 includes:

[0098] S310. Based on the extrinsic parameter matrix, transform the point cloud target in the first detection result from the point cloud coordinate system to the three-dimensional coordinate system, and based on the intrinsic parameter matrix, transform the point cloud target from the three-dimensional coordinate system to the two-dimensional coordinate system where the video target in the second detection result is located.

[0099] Specifically, after synchronizing the sampling frequency and time of the LiDAR and camera, the intrinsic and extrinsic parameter matrices between the LiDAR and camera are further obtained through calibration methods. The extrinsic parameter matrix is ​​used to achieve coordinate transformation between the LiDAR's point cloud coordinate system and the camera's 3D coordinate system, transforming the 3D coordinates of the point cloud target in the first detection result from the point cloud coordinate system to the camera's 3D coordinate system. The intrinsic parameter matrix is ​​used to achieve coordinate transformation between the camera's 3D coordinate system and the 2D coordinate system of the video target in the second detection result, further transforming the 3D coordinates of the point cloud target in the camera's 3D coordinate system to the 2D coordinate system of the video target, thus obtaining the 2D coordinates of the point cloud target in the 2D coordinate system.

[0100] Alternatively, the calibration method can be the checkerboard calibration method.

[0101] S320. Determine the target objects in the initial fusion result based on the point cloud targets and video targets in the two-dimensional coordinate system.

[0102] Among them, point cloud targets and video targets with an information matching probability greater than the matching threshold are considered to be the same target object. The information matching probability is used to characterize the positional overlap between point cloud targets and video targets.

[0103] Specifically, the server transforms the detection boxes of point cloud targets in the first detection result to the two-dimensional coordinate system of the video targets, and obtains the positional overlap between the detection boxes of point cloud targets and the detection boxes of video targets in the second detection result, such as the intersection over union (IOU), as the information matching probability between point cloud targets and video targets. Point cloud targets and video targets with an information matching probability greater than a matching threshold (e.g., 0.6) are identified as the same target object, while the remaining point cloud targets and video targets are identified as separate target objects, thus obtaining the target objects in the initial fusion result.

[0104] In this embodiment, the server transforms the point cloud targets in the first detection result into the two-dimensional coordinate system where the video targets in the second detection result are located based on the extrinsic and intrinsic parameter matrices. Then, in the two-dimensional coordinate system, the target objects in the initial fusion result are determined based on the information matching probability between the point cloud targets and the video targets. By transforming the point cloud targets and video targets to the same dimension through the extrinsic and intrinsic parameter matrices, and then fusing them, the accuracy of fusing the first and second detection results is improved.

[0105] In one embodiment, if the actual target is too close to the sensor (LiDAR / camera), target detection based on the sensor's collected perception data may result in multiple targets being mistakenly identified as a single target, necessitating multi-target misidentification correction. Therefore, if the correction type is multi-target misidentification correction, such as... Figure 4 As shown, the above S240 includes:

[0106] S410. If in the initial fusion result there is a situation where the information matching probability of one point cloud target corresponding to a target object is greater than the first matching threshold with multiple video targets, then the multiple video targets in the initial fusion result will be marked as different target objects respectively.

[0107] S420. If in the initial fusion result there is a situation where the information matching probability of one video target corresponding to a target object is greater than the second matching threshold with multiple point cloud targets, then the multiple point cloud targets in the initial fusion result will be marked as different target objects respectively.

[0108] Specifically, if the initial fusion result shows that the information matching probability between a point cloud target corresponding to a target object and multiple video targets is greater than the first matching threshold, the server determines that the first detection result contains a misjudgment of multiple point cloud targets being detected as a single point cloud target. Similarly, if the initial fusion result shows that the information matching probability between a video target corresponding to a target object and multiple point cloud targets is greater than the second matching threshold, the server determines that the second detection result contains a misjudgment of multiple video targets being detected as a single video target. For the aforementioned cases of multiple target misjudgments, the server marks the multiple video targets or multiple point cloud targets corresponding to the target object as different targets. For example, if the initial fusion result includes C1 (one point cloud target) and D1–D5 (five video targets), where the information matching probability between point cloud target C1 and video targets D1–D3 is greater than the first matching threshold, indicating that the first detection result contains a misjudgment of three point cloud targets as a single point cloud target, the server will mark the five video targets D1–D5 obtained from the second detection result as different targets and use them as targets in the output data fusion result.

[0109] In this embodiment, if the server determines that there is a multi-target misjudgment in the initial fusion result if the information matching probability between a point cloud target corresponding to a target object and multiple video targets is greater than the first matching threshold, or if the information matching probability between a video target corresponding to a target object and multiple point cloud targets is greater than the second matching threshold, then the server will mark the multiple video targets / point cloud targets in the initial fusion result as different target objects. This corrects the multi-target misjudgment in the initial fusion result, improves the accuracy of the target object information in the final output data fusion result, and thus improves the accuracy of target detection.

[0110] In one embodiment, since the sensors collect perception data of different types and dimensions of the target object, the corresponding detection results will also differ after target detection. For the same target object, if target detection is performed based on perception data collected by different sensors, the target object category will be different, i.e., there is a misclassification of category confidence, which requires confidence correction. Therefore, if the correction type is confidence correction, such as... Figure 5 As shown, the above S240 includes:

[0111] S510. Obtain the first confidence level and the second confidence level of the target object included in the initial fusion result. The first confidence level is the confidence level of the target object category determined based on the point cloud data, and the second confidence level is the confidence level of the target object category determined based on the video data.

[0112] Specifically, for a target object that has both a first detection result and a second detection result in the initial fusion result, if the first confidence level of the target object's category in the first detection result is different from the second confidence level of the target object's category in the second detection result, the server determines that there is a confidence misjudgment in the initial fusion result, that is, there is a misjudgment of the target object's category in either the first detection result or the second detection result, and confidence correction is required accordingly.

[0113] S520. The first confidence level and the second confidence level of the target object are transformed, and the category corresponding to the larger confidence level of the two obtained after transformation is taken as the category of the target object.

[0114] Specifically, the server transforms the first and second confidence scores of the target object in the initial fusion result, converting them to the same reference standard. The category corresponding to the larger of the two confidence scores is then used as the category of the target object in the data fusion result. For example, if the first confidence score of target object E1 is 60% (target object category is bus) and the second confidence score is 80% (target object category is van), the server will use the larger confidence score, i.e., the second confidence score, corresponding to the category of van, as the category of target object E1 in the output data fusion result, thereby correcting the confidence score of the initial fusion result.

[0115] Optionally, the correction type also includes category correction. For a target object that has both a first detection result and a second detection result in the initial fusion result, if the target object is classified as a human in the first detection result and as a non-motorized vehicle in the second detection result, the server will use non-motorized vehicle as the target object's category and output it as the data fusion result. This is because, in practical applications, when a person is driving a non-motorized vehicle (such as a bicycle), the distance between the person and the non-motorized vehicle is relatively close, which can easily lead to misjudgment of the non-motorized vehicle as a human. Therefore, in cases where one detection result identifies the target object as a human and another as a non-motorized vehicle for the same target object, the server will use non-motorized vehicle as the target object's category and output it as the data fusion result.

[0116] In this embodiment, for a target object that simultaneously possesses both a first detection result and a second detection result in the initial fusion result, the server converts the first confidence level and the second confidence level of the target object to the same reference standard. If the first confidence level and the second confidence level of the target object are different, the server determines that there is a confidence level misjudgment in the initial fusion result. The server then takes the category corresponding to the larger confidence level of the two confidence levels obtained after the conversion process as the category of the target object, thereby correcting the confidence level of the initial fusion result, improving the accuracy of the target object information in the final output data fusion result, and thus improving the accuracy of target detection.

[0117] In one embodiment, the video data acquired by the camera includes color information of the video target, while the point cloud data acquired by the LiDAR does not. If the final data fusion result requires color information of the target object, target color correction is necessary. Therefore, if the correction type is target color correction, such as... Figure 6 As shown, the above S240 includes:

[0118] S610. Detect whether the target object included in the initial fusion result has a corresponding video target in the second detection result.

[0119] S620. If the target object included in the initial fusion result has a corresponding video target in the second detection result, then the color information of the corresponding video target is obtained based on the video data, and the color information is used as the color information of the target object.

[0120] The first detection result obtained from point cloud data does not contain color information of the point cloud target, while the second detection result obtained from video data may contain color information of the video target.

[0121] Specifically, the server detects whether the target object included in the initial fusion result has a corresponding video target in the second detection result. If it does, and the second detection result obtained based on the video data includes the color information of the video target, then the color of the corresponding target object in the initial fusion result is supplemented. That is, the color information of the video target obtained in the second detection result is used as the color information of the corresponding target object and output as the data fusion result, thereby realizing the target color correction of the initial fusion result.

[0122] In this embodiment, if the target object included in the initial fusion result has a corresponding video target in the second detection result, the server obtains the color information of the corresponding video target based on the video data and uses the color information as the color information of the target object. This corrects the target color of the initial fusion result, improves the accuracy of the target object information in the final output data fusion result, and thus improves the accuracy of target detection.

[0123] In one embodiment, different sensors may experience limitations in data acquisition due to external factors such as blind spots, object obstruction, and ambient light, leading to missed detections in the detection results. Therefore, missed detection correction is required. Thus, if the correction type is missed detection correction, such as... Figure 7 As shown, the above S240 includes:

[0124] S710. If the target object included in the initial fusion result has a corresponding point cloud target in the first detection result but no corresponding video target in the second detection result, then the first detection result of the point cloud target is mapped to the two-dimensional coordinate system where the video target is located in the second detection result, and used as the second detection result of the target object.

[0125] Specifically, if the target object included in the initial fusion result has a corresponding point cloud target in the first detection result but no corresponding video target in the second detection result, the server determines that there is a missed detection in the second detection result. Then, the first detection result of the point cloud target corresponding to the target object is mapped to the two-dimensional coordinate system where the video target is located in the second detection result, and this is used as the second detection result of the target object.

[0126] S720. If the target object included in the initial fusion result has a corresponding video target in the second detection result, but no corresponding point cloud target in the first detection result, then the second detection result of the video target is transformed into the point cloud coordinate system of the point cloud data through the mutual learning model, and used as the first detection result of the target object.

[0127] Specifically, if the target object included in the initial fusion result has a corresponding video target in the second detection result, but no corresponding point cloud target in the first detection result, the server determines that there is a missed detection in the first detection result. Then, the second detection result of the video target corresponding to the target object is transformed into the point cloud coordinate system where the point cloud target in the first detection result is located through the mutual learning model, and used as the first detection result of the target object.

[0128] In this embodiment, if the target object included in the initial fusion result has a corresponding point cloud target in the first detection result but no corresponding video target in the second detection result, the server maps the first detection result of the point cloud target to the two-dimensional coordinate system where the video target is located in the second detection result, and uses it as the second detection result of the target object. If the target object included in the initial fusion result has a corresponding video target in the second detection result but no corresponding point cloud target in the first detection result, the server transforms the second detection result of the video target to the two-dimensional coordinate system where the point cloud target is located in the first detection result through a mutual learning model, and uses it as the first detection result of the target object. This achieves the correction of missed detections in the initial fusion result, and correspondingly improves the accuracy of the first detection result / second detection result.

[0129] In one embodiment, the confidence score of a category is used to characterize the probability that a target object belongs to that category. If the confidence score of a target object in the corresponding detection result is too low, a false detection may exist, and false detection correction is required. Therefore, if the correction type is false detection correction, such as... Figure 8 As shown, the above S240 includes:

[0130] S810. Obtain the first confidence level of the target objects included in the initial fusion result. The first confidence level is the confidence level of the category of the target objects determined based on the point cloud data.

[0131] Specifically, the server obtains the first confidence level of the target object in the initial fusion result, that is, the confidence level of the category of the point cloud target corresponding to the target object in the first detection result determined based on the point cloud data.

[0132] S820. If the first confidence level is less than the confidence threshold, obtain the area of ​​the detection box corresponding to the target object in the first detection result included in the initial fusion result.

[0133] S830. If the area of ​​the detection box is less than the area threshold, the target object is removed from the initial fusion result.

[0134] Specifically, the server compares the first confidence level of the target object with a preset confidence threshold. If the first confidence level is less than the confidence threshold, the server obtains the area of ​​the detection box of the point cloud target corresponding to the target object in the first detection result in the initial fusion result, and compares the area of ​​the detection box with a preset area threshold. If the area of ​​the detection box is less than the area threshold, the server determines that there is a false detection caused by the lidar being too far from the actual target in the initial fusion result, and then removes the target object from the initial fusion result. In practical applications, the larger the detection box obtained by target detection, the larger the target object is represented, or the closer the target object is to the sensor. Target detection based on sensing data of large targets, or target detection based on sensing data collected at close range, yields more accurate and reliable detection results. Conversely, target detection based on sensing data of small targets, or target detection based on sensing data collected at a distance, yields less reliable detection results.

[0135] In this embodiment, the server obtains the first confidence level of the point cloud target corresponding to the target object in the first detection result in the initial fusion result. If the first confidence level is less than a preset confidence level threshold, the server further obtains the area of ​​the detection box of the point cloud target corresponding to the target object in the first detection result. If the area of ​​the detection box is less than a preset area threshold, the target object is removed from the initial fusion result. This achieves false detection correction of the initial fusion result, improves the accuracy of the target object information in the final output data fusion result, and thus improves the accuracy of target detection.

[0136] In one embodiment, if the judgment on whether the driving trajectory has changed abruptly differs in different detection results for the same target object, then trajectory correction is required. Therefore, if the correction type is trajectory correction, such as... Figure 9 As shown, the above S240 includes:

[0137] S910. Obtain the initial fusion results, including the trajectory direction of the point cloud target corresponding to the target object and the trajectory direction of the corresponding video target.

[0138] Optionally, the server can obtain point cloud data from multiple points at different times, and obtain the coordinates of the point cloud target corresponding to the target object in the point cloud data at each time, thereby determining the trajectory and direction of the point cloud target corresponding to the target object. Similarly, the trajectory and direction of the video target corresponding to the target object can be determined.

[0139] S920. If the trajectory direction of the point cloud target changes abruptly, while the trajectory direction of the video target does not change abruptly, then the trajectory direction of the target included in the initial fusion result is determined based on the trajectory direction of the video target.

[0140] Among them, a change in the direction of travel of the trajectory refers to a 180-degree change in the direction of travel of the trajectory.

[0141] Specifically, the server can determine whether the trajectory direction of the point cloud target has changed based on its trajectory, and determine whether the trajectory direction of the video target has changed based on its trajectory. If the trajectory direction of the point cloud target and the trajectory direction of the video target are inconsistent, the server determines that there is a misjudgment of a trajectory direction change in the initial fusion result, and trajectory correction is required.

[0142] Because point cloud data only obtains the outline position of the target object, its ability to determine the head and tail of the target object is poor. Therefore, it is easy to misidentify the head and tail of the target object. In this case, the target object may not actually change its direction of travel, but the direction of travel may change due to a LiDAR detection error. In this case, the trajectory direction of the point cloud target determined based on the point cloud data will be inaccurate. Therefore, if the trajectory direction of the point cloud target corresponding to the same target object changes abruptly, but the trajectory direction of the corresponding video target does not change abruptly, the server determines the direction of travel of the target object included in the initial fusion result based on the trajectory direction of the corresponding video target.

[0143] In this embodiment, the server obtains the trajectory direction of the point cloud target corresponding to the target object and the trajectory direction of the corresponding video target in the initial fusion result. If the trajectory direction of the point cloud target changes but the trajectory direction of the video target does not change, the trajectory direction of the target object included in the initial fusion result is determined based on the trajectory direction of the video target. This achieves trajectory correction of the initial fusion result, improves the accuracy of the target object information in the final output data fusion result, and thus improves the accuracy of target detection.

[0144] It should be understood that, although Figures 2-9 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order in which these steps are executed, and they can be performed in other orders. Figures 2-9 At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0145] In one embodiment, such as Figure 10 As shown, a multi-source fusion sensing system is provided, including: a data acquisition module 1001, a target detection module 1002, a target fusion module 1103, and a result correction module 1004, wherein:

[0146] The data acquisition module 1001 is used to acquire point cloud data and video data at the same time and in the same scene;

[0147] The target detection module 1002 is used to perform target detection on point cloud data to obtain a first detection result, and to perform target detection on video data to obtain a second detection result;

[0148] The target fusion module 1003 is used to fuse the first detection result and the second detection result to obtain an initial fusion result;

[0149] The result correction module 1004 is used to determine whether the initial fusion result needs to be corrected. If the result indicates that correction is needed, the initial fusion result is corrected according to the correction type of the result, and the data fusion result is output. The correction type includes: multi-target misjudgment correction or trajectory correction.

[0150] In one embodiment, the target fusion module 1003 is specifically used for:

[0151] Based on the extrinsic parameter matrix, the point cloud targets in the first detection result are transformed from the point cloud coordinate system to the three-dimensional coordinate system, and based on the intrinsic parameter matrix, the point cloud targets are transformed from the three-dimensional coordinate system to the two-dimensional coordinate system where the video targets in the second detection result are located. Based on the point cloud targets and video targets in the two-dimensional coordinate system, the target objects in the initial fusion result are determined. In this process, point cloud targets and video targets with an information matching probability greater than the matching threshold are considered to be the same target object. The information matching probability is used to characterize the positional overlap between point cloud targets and video targets.

[0152] In one embodiment, the correction type is multi-target misjudgment correction, and the result correction module 1004 is specifically used for:

[0153] If in the initial fusion result there is a situation where the information matching probability of one point cloud target corresponding to a target object is greater than the first matching threshold with multiple video targets, then the multiple video targets in the initial fusion result will be marked as different target objects respectively; if in the initial fusion result there is a situation where the information matching probability of one video target corresponding to a target object is greater than the second matching threshold with multiple point cloud targets, then the multiple point cloud targets in the initial fusion result will be marked as different target objects respectively.

[0154] In one embodiment, the correction type is confidence correction, and the result correction module 1004 is specifically used for:

[0155] The initial fusion result includes the first confidence level and the second confidence level of the target object. The first confidence level is the confidence level of the target object category determined based on the point cloud data, and the second confidence level is the confidence level of the target object category determined based on the video data. The first confidence level and the second confidence level of the target object are transformed, and the category corresponding to the larger confidence level of the two obtained after transformation is taken as the category of the target object.

[0156] In one embodiment, the correction type is target color correction, and the result correction module 1004 is specifically used for:

[0157] The system detects whether the target objects included in the initial fusion result have corresponding video targets in the second detection result. If the target objects included in the initial fusion result have corresponding video targets in the second detection result, the system obtains the color information of the corresponding video targets based on the video data and uses the color information as the color information of the target objects.

[0158] In one embodiment, the correction type is a missed detection correction, and the result correction module 1004 is specifically used for:

[0159] If the target object included in the initial fusion result has a corresponding point cloud target in the first detection result but no corresponding video target in the second detection result, then the first detection result of the point cloud target is mapped to the two-dimensional coordinate system where the video target is located in the second detection result, and this is taken as the second detection result of the target object; if the target object included in the initial fusion result has a corresponding video target in the second detection result but no corresponding point cloud target in the first detection result, then the second detection result of the video target is transformed to the point cloud coordinate system where the point cloud target is located in the first detection result through a mutual learning model, and this is taken as the first detection result of the target object.

[0160] In one embodiment, the correction type is false detection correction, and the result correction module 1004 is specifically used for:

[0161] Obtain the first confidence level of the target objects included in the initial fusion result. The first confidence level is the confidence level of the category of the target object determined based on the point cloud data. If the first confidence level is less than the confidence level threshold, obtain the area of ​​the detection box corresponding to the target object included in the initial fusion result in the first detection result. If the area of ​​the detection box is less than the area threshold, remove the target object from the initial fusion result.

[0162] In one embodiment, the correction type is trajectory correction, and the result correction module 1004 is specifically used for:

[0163] The initial fusion result includes the trajectory direction of the point cloud target corresponding to the target object and the trajectory direction of the corresponding video target. If the trajectory direction of the point cloud target changes abruptly, but the trajectory direction of the video target does not change abruptly, the trajectory direction of the target object included in the initial fusion result is determined based on the trajectory direction of the video target.

[0164] In one embodiment, the point cloud data is obtained by detecting the area to be measured using roadside lidar.

[0165] In one embodiment, the correction type further includes one or more of the following: confidence correction, target color correction, missed detection correction, and false detection correction.

[0166] Specific limitations regarding the multi-source fusion sensing system can be found in the limitations of the multi-sensor data fusion method described above, and will not be repeated here. Each module in the aforementioned multi-source fusion sensing system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0167] In one embodiment, a computer device is provided, the internal structure of which can be shown as follows: Figure 11 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a multi-sensor data fusion method. The display screen can be an LCD screen or an e-ink display screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0168] Those skilled in the art will understand that Figure 11 The structure shown is merely a block diagram of a portion of the structure related to the solution of this application and does not constitute a limitation on the smart base station on which the solution of this application is applied. A specific smart base station may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0169] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0170] Acquire point cloud data and video data at the same time and in the same scene; perform object detection on the point cloud data to obtain a first detection result, and perform object detection on the video data to obtain a second detection result; fuse the first detection result and the second detection result to obtain an initial fusion result; determine whether the initial fusion result needs to be corrected; if the determination result is that it needs to be corrected, correct the initial fusion result according to the correction type of the determination result, and output the data fusion result; wherein, the correction type includes: multi-target misjudgment or trajectory correction.

[0171] In one embodiment, the processor further performs the following steps when executing the computer program:

[0172] Based on the extrinsic parameter matrix, the point cloud targets in the first detection result are transformed from the point cloud coordinate system to the three-dimensional coordinate system, and based on the intrinsic parameter matrix, the point cloud targets are transformed from the three-dimensional coordinate system to the two-dimensional coordinate system where the video targets in the second detection result are located. The target objects in the initial fusion result are determined based on the point cloud targets and video targets in the two-dimensional coordinate system. Among them, point cloud targets and video targets with an information matching probability greater than the matching threshold are the same target object, and the information matching probability is used to characterize the positional overlap between point cloud targets and video targets.

[0173] In one embodiment, the correction type is multi-target misjudgment correction, and the processor, when executing the computer program, further implements the following steps:

[0174] If in the initial fusion result there is a situation where the information matching probability of one point cloud target corresponding to a target object is greater than the first matching threshold with multiple video targets, then the multiple video targets in the initial fusion result will be marked as different target objects respectively; if in the initial fusion result there is a situation where the information matching probability of one video target corresponding to a target object is greater than the second matching threshold with multiple point cloud targets, then the multiple point cloud targets in the initial fusion result will be marked as different target objects respectively.

[0175] In one embodiment, the correction type is confidence correction, and the processor, when executing the computer program, further implements the following steps:

[0176] The initial fusion result includes the first confidence level and the second confidence level of the target object. The first confidence level is the confidence level of the target object category determined based on the point cloud data, and the second confidence level is the confidence level of the target object category determined based on the video data. The first confidence level and the second confidence level of the target object are transformed, and the category corresponding to the larger confidence level of the two obtained after transformation is taken as the category of the target object.

[0177] In one embodiment, the correction type is target color correction, and the processor, when executing the computer program, further implements the following steps:

[0178] The system detects whether the target objects included in the initial fusion result have corresponding video targets in the second detection result. If the target objects included in the initial fusion result have corresponding video targets in the second detection result, the system obtains the color information of the corresponding video targets based on the video data and uses the color information as the color information of the target objects.

[0179] In one embodiment, the correction type is a missed detection correction, and the processor, when executing the computer program, further implements the following steps:

[0180] If the target object included in the initial fusion result has a corresponding point cloud target in the first detection result but no corresponding video target in the second detection result, then the first detection result of the point cloud target is mapped to the two-dimensional coordinate system where the video target is located in the second detection result, and this is taken as the second detection result of the target object; if the target object included in the initial fusion result has a corresponding video target in the second detection result but no corresponding point cloud target in the first detection result, then the second detection result of the video target is transformed to the point cloud coordinate system where the point cloud target is located in the first detection result through a mutual learning model, and this is taken as the first detection result of the target object.

[0181] In one embodiment, the correction type is a false detection correction, and the processor, when executing the computer program, further implements the following steps:

[0182] Obtain the first confidence level of the target objects included in the initial fusion result. The first confidence level is the confidence level of the category of the target object determined based on the point cloud data. If the first confidence level is less than the confidence level threshold, obtain the area of ​​the detection box corresponding to the target object included in the initial fusion result in the first detection result. If the area of ​​the detection box is less than the area threshold, remove the target object from the initial fusion result.

[0183] In one embodiment, the correction type is trajectory correction, and the processor, when executing the computer program, also performs the following steps:

[0184] The initial fusion result includes the trajectory direction of the point cloud target corresponding to the target object and the trajectory direction of the corresponding video target. If the trajectory direction of the point cloud target changes abruptly, but the trajectory direction of the video target does not change abruptly, the trajectory direction of the target object included in the initial fusion result is determined based on the trajectory direction of the video target.

[0185] In one embodiment, the point cloud data is obtained based on detections of the area to be measured by roadside lidar. In one embodiment, the correction type also includes one or more of the following: confidence correction, target color correction, missed detection correction, and false detection correction.

[0186] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0187] Acquire point cloud data and video data at the same time and in the same scene; perform object detection on the point cloud data to obtain a first detection result, and perform object detection on the video data to obtain a second detection result; fuse the first detection result and the second detection result to obtain an initial fusion result; determine whether the initial fusion result needs to be corrected; if the determination result is that it needs to be corrected, correct the initial fusion result according to the correction type of the determination result, and output the data fusion result; wherein, the correction type includes: multi-target misjudgment correction or trajectory correction.

[0188] In one embodiment, when the computer program is executed by the processor, it further performs the following steps:

[0189] Based on the extrinsic parameter matrix, the point cloud targets in the first detection result are transformed from the point cloud coordinate system to the three-dimensional coordinate system, and based on the intrinsic parameter matrix, the point cloud targets are transformed from the three-dimensional coordinate system to the two-dimensional coordinate system where the video targets in the second detection result are located. The target objects in the initial fusion result are determined based on the point cloud targets and video targets in the two-dimensional coordinate system. Among them, point cloud targets and video targets with an information matching probability greater than the matching threshold are the same target object, and the information matching probability is used to characterize the positional overlap between point cloud targets and video targets.

[0190] In one embodiment, the correction type is multi-target misjudgment correction, and the computer program, when executed by a processor, further implements the following steps:

[0191] If in the initial fusion result there is a situation where the information matching probability of one point cloud target corresponding to a target object is greater than the first matching threshold with multiple video targets, then the multiple video targets in the initial fusion result will be marked as different target objects respectively; if in the initial fusion result there is a situation where the information matching probability of one video target corresponding to a target object is greater than the second matching threshold with multiple point cloud targets, then the multiple point cloud targets in the initial fusion result will be marked as different target objects respectively.

[0192] In one embodiment, the correction type is confidence correction, and the computer program, when executed by a processor, further implements the following steps:

[0193] The initial fusion result includes the first confidence level and the second confidence level of the target object. The first confidence level is the confidence level of the target object category determined based on the point cloud data, and the second confidence level is the confidence level of the target object category determined based on the video data. The first confidence level and the second confidence level of the target object are transformed, and the category corresponding to the larger confidence level of the two obtained after transformation is taken as the category of the target object.

[0194] In one embodiment, the correction type is target color correction, and the computer program, when executed by a processor, also performs the following steps:

[0195] The system detects whether the target objects included in the initial fusion result have corresponding video targets in the second detection result. If the target objects included in the initial fusion result have corresponding video targets in the second detection result, the system obtains the color information of the corresponding video targets based on the video data and uses the color information as the color information of the target objects.

[0196] In one embodiment, the correction type is a missed detection correction, and the computer program, when executed by a processor, further implements the following steps:

[0197] If the target object included in the initial fusion result has a corresponding point cloud target in the first detection result but no corresponding video target in the second detection result, then the first detection result of the point cloud target is mapped to the two-dimensional coordinate system where the video target is located in the second detection result, and this is taken as the second detection result of the target object; if the target object included in the initial fusion result has a corresponding video target in the second detection result but no corresponding point cloud target in the first detection result, then the second detection result of the video target is transformed to the point cloud coordinate system where the point cloud target is located in the first detection result through a mutual learning model, and this is taken as the first detection result of the target object.

[0198] In one embodiment, the correction type is a false detection correction, and the computer program, when executed by the processor, further implements the following steps:

[0199] Obtain the first confidence level of the target objects included in the initial fusion result. The first confidence level is the confidence level of the category of the target object determined based on the point cloud data. If the first confidence level is less than the confidence level threshold, obtain the area of ​​the detection box corresponding to the target object included in the initial fusion result in the first detection result. If the area of ​​the detection box is less than the area threshold, remove the target object from the initial fusion result.

[0200] In one embodiment, the correction type is trajectory correction, and the computer program, when executed by a processor, also performs the following steps:

[0201] The initial fusion result includes the trajectory direction of the point cloud target corresponding to the target object and the trajectory direction of the corresponding video target. If the trajectory direction of the point cloud target changes abruptly, but the trajectory direction of the video target does not change abruptly, the trajectory direction of the target object included in the initial fusion result is determined based on the trajectory direction of the video target.

[0202] In one embodiment, the point cloud data is obtained by detecting the area to be measured using roadside lidar.

[0203] In one embodiment, the correction type further includes one or more of the following: confidence correction, target color correction, missed detection correction, and false detection correction.

[0204] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0205] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0206] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A multi-sensor data fusion method, characterized by, Includes the following steps: Acquire point cloud data and video data from the same moment and the same scene; A first detection result is obtained by performing target detection on the point cloud data, and a second detection result is obtained by performing target detection on the video data; The first detection result and the second detection result are fused to obtain an initial fusion result; For a target object in the initial fusion result that simultaneously has a first detection result and a second detection result, the following steps are taken: First, based on the quantitative relationship between the first and second detection results, it is determined whether the initial fusion result needs multi-target misjudgment correction; second, based on whether the confidence levels of the target object in the first and second detection results are the same, it is determined whether the initial fusion result needs confidence correction; third, based on the travel trajectories of the target object in the first and second detection results, it is determined whether the initial fusion result needs trajectory correction. Based on the category of the target object in the first detection result and the second detection result, determine whether the initial fusion result needs to be corrected for category. For the target object with a first detection result in the initial fusion result, it is determined whether the initial fusion result needs to be corrected for false detection based on the detection box size obtained by the target object in the corresponding first detection result; For the target object with the second detection result in the initial fusion result, it is determined whether the initial fusion result needs to be color corrected based on whether the second detection result has the color relationship of the target object; If the judgment result indicates that correction is needed, the initial fusion result is corrected according to the correction type of the judgment result, and the data fusion result is output; wherein, the correction type includes one or more of the following: multi-target misjudgment correction, confidence correction, category correction, color correction, false detection correction, and trajectory correction; The false detection correction includes: obtaining a first confidence level of the target object based on the point cloud data; if the first confidence level is less than a confidence level threshold, obtaining the area of ​​the detection box of the target object in the first detection result; if the area of ​​the detection box is less than an area threshold, then removing the target object.

2. The method of claim 1, wherein, The process of fusing the first detection result and the second detection result to obtain an initial fusion result includes: Based on the extrinsic parameter matrix, the point cloud targets in the first detection result are transformed from the point cloud coordinate system to the three-dimensional coordinate system, and based on the intrinsic parameter matrix, the point cloud targets are transformed from the three-dimensional coordinate system to the two-dimensional coordinate system where the video targets in the second detection result are located; The target object in the initial fusion result is determined based on the point cloud target in the two-dimensional coordinate system and the video target; wherein, the point cloud target and the video target with an information matching probability greater than the matching threshold are the same target object, and the information matching probability is used to characterize the positional overlap between the point cloud target and the video target.

3. The method according to claim 2, characterized in that, The correction type is the multi-target misjudgment correction, and the correction of the initial fusion result according to the correction type of the judgment result includes: If in the initial fusion result there is a situation where the information matching probability of one point cloud target corresponding to a target object is greater than the first matching threshold with multiple video targets, then the multiple video targets in the initial fusion result will be marked as different target objects respectively; If in the initial fusion result there is a situation where the information matching probability of one video target corresponding to a target object is greater than the second matching threshold with multiple point cloud targets, then the multiple point cloud targets in the initial fusion result will be marked as different target objects.

4. The method of claim 1, wherein, The correction type is the confidence correction, and the correction of the initial fusion result according to the correction type of the judgment result includes: The initial fusion result includes a first confidence level and a second confidence level of the target object, wherein the first confidence level is the confidence level of the category of the target object determined based on the point cloud data, and the second confidence level is the confidence level of the category of the target object determined based on the video data; The first and second confidence levels of the target object are converted, and the category corresponding to the larger confidence level of the two obtained after conversion is taken as the category of the target object.

5. The method of claim 1, wherein, The correction type is the color correction, and the correction of the initial fusion result according to the correction type of the judgment result includes: The detection process checks whether the target objects included in the initial fusion result have corresponding video targets in the second detection result; If the target object included in the initial fusion result has a corresponding video target in the second detection result, then the color information of the corresponding video target is obtained based on the video data, and the color information is used as the color information of the target object.

6. The method of claim 1, wherein, The correction type is a missed detection correction, and the correction of the initial fusion result according to the correction type of the judgment result includes: If the target object included in the initial fusion result has a corresponding point cloud target in the first detection result but no corresponding video target in the second detection result, then the first detection result of the point cloud target is mapped to the two-dimensional coordinate system where the video target is located in the second detection result, and used as the second detection result of the target object; If the target object included in the initial fusion result has a corresponding video target in the second detection result but no corresponding point cloud target in the first detection result, then the second detection result of the video target is transformed into the point cloud coordinate system where the point cloud target in the first detection result is located through a mutual learning model, and is used as the first detection result of the target object.

7. The method of claim 1, wherein, The correction type is the trajectory correction, and the correction of the initial fusion result according to the correction type of the judgment result includes: The initial fusion result includes the trajectory direction of the point cloud target corresponding to the target object and the trajectory direction of the corresponding video target. If the trajectory direction of the point cloud target changes abruptly, while the trajectory direction of the video target does not change abruptly, then the trajectory direction of the target objects included in the initial fusion result is determined based on the trajectory direction of the video target.

8. The method according to claim 1, characterized in that, The point cloud data is obtained by detecting the area to be measured using roadside lidar. 9.A multi-source fusion perception system, characterized in that, The system includes: The data acquisition module is used to acquire point cloud data and video data from the same time and scene. The target detection module is used to perform target detection on the point cloud data to obtain a first detection result, and to perform target detection on the video data to obtain a second detection result; The target fusion module is used to fuse the first detection result and the second detection result to obtain an initial fusion result; The result correction module is used to, for targets in the initial fusion result that simultaneously have a first detection result and a second detection result, determine whether the initial fusion result needs multi-target misclassification correction based on the quantitative relationship between the first and second detection results for the target object; determine whether the initial fusion result needs confidence correction based on whether the confidence levels of the target object in the first and second detection results are the same; determine whether the initial fusion result needs trajectory correction based on the travel trajectory of the target object in the first and second detection results; determine whether the initial fusion result needs category correction based on the category of the target object in the first and second detection results; and, For the target object with a first detection result in the initial fusion result, based on the detection box size obtained by the target object in the corresponding first detection result, it is determined whether the initial fusion result needs to be corrected for false detections; and, For the target object in the initial fusion result that has a second detection result, determine whether the initial fusion result needs color correction based on whether the second detection result has a color relationship with the target object; and... If the judgment result indicates that correction is needed, the initial fusion result is corrected according to the correction type of the judgment result, and the data fusion result is output. The correction type includes one or more of the following: multi-target misjudgment correction, confidence correction, category correction, color correction, false detection correction, and trajectory correction. The false detection correction includes: obtaining the first confidence of the target object based on the point cloud data; if the first confidence is less than the confidence threshold, obtaining the area of ​​the detection box of the target object in the first detection result; if the area of ​​the detection box is less than the area threshold, then the target object is removed.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

11. A computer readable storage medium having stored thereon a computer program, characterized in that When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images