Sensor data fusion method and device, computer device and readable storage medium

Abstract

The application discloses a sensor data fusion method and device, computer equipment and a computer readable storage medium, relates to the unmanned driving technical field, and supplements through the millimeter wave detection result, enhances the sensing capability, expands the sensing range by using the cooperative sensing result and the roadside sensing result, and improves the sensing precision. The method comprises the following steps: when receiving sensing information, determining a target fusion mode in multiple fusion modes, and determining a target detection range by using a time-space synchronization operation; obtaining to-be-processed information in the target detection range, matching the to-be-processed information by using a matching rule, and obtaining a target matching result; performing compensation calculation on the to-be-processed information by using a dust detection algorithm, obtaining a millimeter wave compensation result, integrating the target matching result, the millimeter wave compensation result and scene sensing parameters of the sensing information by using an integration rule, and obtaining a target fusion result; and processing the target fusion result by using a Kalman algorithm, and obtaining obstacle-related information.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a sensor data fusion method, apparatus, computer equipment, and readable storage medium. Background Technology

[0002] With the rapid development of autonomous driving, autonomous driving in open-pit mines has also seen rapid progress. Due to the dusty and complex road conditions in mining areas, relying on a single sensor cannot solve all problems. Therefore, it is necessary to use multiple sensors to collect information about the vehicle's surrounding environment during autonomous driving. By leveraging the complementary characteristics of different sensors to acquire perceptual information, and then matching obstacle information extracted from the harsh mining environment with tracking target information, autonomous mining trucks can adapt to the complex mining environment during use, ensuring the safety and smoothness of the overall driving process.

[0003] In related technologies, sensor data fusion methods fuse the outputs of information from LiDAR, millimeter-wave radar, and images to obtain driving environment information. However, the applicant recognizes that vehicle operation in mining areas generates dust, causing LiDAR to miss detections and resulting in low perception accuracy of sensor data fusion. Furthermore, given the complex road conditions in mining areas, relying solely on sensors for environmental perception can lead to missed detections, resulting in low perception accuracy and a significant risk of vehicle collisions. Summary of the Invention

[0004] In view of this, this application provides a multi-sensor fusion method, apparatus, computer equipment, and readable storage medium. The main purpose is to solve the problems that dust generated by vehicles in mining areas causes LiDAR to miss detections, resulting in low perception accuracy of multi-sensor fusion. Furthermore, in the complex road conditions of mining areas, relying solely on sensors for environmental perception can lead to missed detections, resulting in low perception accuracy and a significant risk of vehicle collisions.

[0005] According to a first aspect of this application, a sensor data fusion method is provided, the method comprising:

[0006] When the input perception information is received, the target fusion mode corresponding to the perception information is determined among multiple fusion modes, and the target detection range corresponding to the target fusion mode is determined by using spatiotemporal synchronization operation;

[0007] Information to be processed that is within the target detection range is obtained from the perceived information, and the information to be processed is matched using the matching rules corresponding to the target fusion mode to obtain the target matching result;

[0008] The dust detection algorithm is used to perform compensation calculations on the information to be processed to obtain millimeter wave compensation results. The integration rules corresponding to the target fusion mode are used to integrate the target matching results, the millimeter wave compensation results, and the scene perception parameters of the perception information to obtain target fusion results.

[0009] The Kalman algorithm is obtained, and the target fusion result is processed using the Kalman algorithm to obtain obstacle-related information.

[0010] Optionally, determining the target detection range corresponding to the target fusion mode using spatiotemporal synchronization operations includes:

[0011] The system identifies multiple target sensors corresponding to the target fusion mode, acquires sensor status information from the perception information, and detects the sensor status information. The multiple target sensors include lidar, millimeter-wave radar, and camera sensors. The sensor status information indicates the working status of the multiple target sensors.

[0012] If the sensor status information indicates that the multiple target sensors are working normally, then the coordinate information of each target sensor is obtained;

[0013] The sample vehicle corresponding to the perceived information is determined, the rear axle center position information of the sample vehicle is obtained, and the coordinate information of each target sensor is adjusted using the rear axle center position information so that the coordinate origin of each target sensor is unified to the rear axle center of the sample vehicle.

[0014] Obtain the timestamp of the lidar, and sequentially determine the timestamps of the millimeter-wave radar and the camera sensor that are close to the timestamp of the lidar;

[0015] By using the timestamps of the lidar, the millimeter-wave radar, and the camera sensor, the lidar, the millimeter-wave radar, and the camera sensor are unified under the same timestamp.

[0016] The detection range of each target sensor is obtained to obtain multiple detection ranges, and the intersection of the multiple detection ranges is determined to obtain the target detection range.

[0017] Optionally, unifying the lidar, millimeter-wave radar, and camera sensor to the same timestamp using the timestamps of the lidar, the millimeter-wave radar, and the camera sensor includes:

[0018] Obtain the millimeter-wave detection result corresponding to the timestamp of the millimeter-wave radar and the camera detection result corresponding to the timestamp of the camera sensor;

[0019] Calculate the time difference between the timestamp of the millimeter-wave radar, the timestamp of the camera sensor, and the timestamp of the lidar to obtain the millimeter-wave time difference and the camera time difference.

[0020] Motion compensation is performed on the millimeter-wave detection results using the millimeter-wave time difference and on the camera detection results using the camera time difference, so that the lidar, the millimeter-wave radar, and the camera sensor are at the same timestamp.

[0021] Optionally, the step of acquiring the information to be processed within the target detection range from the perceived information, and matching the information to be processed using the matching rule corresponding to the target fusion pattern to obtain the target matching result includes:

[0022] The traditional sensing output results, intelligent sensing output results, millimeter-wave sensing output results, and scene sensing parameters of the lidar are obtained from the sensing information. The traditional sensing output results, intelligent sensing output results, millimeter-wave sensing output results, and scene sensing parameters of the lidar to be processed within the target detection range are obtained from the traditional sensing output results, intelligent sensing output results, millimeter-wave sensing output results, and scene sensing parameters of the lidar to be processed. The scene sensing parameters include communication cooperative sensing results and roadside sensing results.

[0023] The traditional sensing information of the lidar to be processed, the intelligent sensing information of the lidar to be processed, the millimeter-wave sensing information to be processed, and the scene sensing information to be processed are used as the information to be processed, and the matching rule corresponding to the target fusion mode is obtained.

[0024] The matching rules are used to determine a first content that matches the intelligent sensing information of the LiDAR to be processed in the traditional sensing information of the LiDAR to be processed. A second content that matches the first content is obtained in the intelligent sensing information of the LiDAR to be processed. The second content is used to replace the first content in the traditional sensing information of the LiDAR to be processed to obtain a first matching result. If there is a first unmatched content other than the second content in the intelligent sensing information of the LiDAR to be processed, the first unmatched content is filtered out.

[0025] Using the matching rules, a third content matching the millimeter-wave sensing information to be processed is determined in the first matching result; a fourth content matching the third content is obtained in the millimeter-wave sensing information to be processed; and the fourth content is used to replace the third content in the first matching result to obtain a second matching result.

[0026] Using the matching rules, a fifth content matching the scene perception information to be processed is determined in the second matching result; a sixth content matching the fifth content is obtained in the scene perception information to be processed; and the sixth content is used to replace the fifth content in the second matching result to obtain the target matching result.

[0027] Optionally, the step of using a dust detection algorithm to perform compensation calculations on the information to be processed to obtain a millimeter-wave compensation result, and using the integration rules corresponding to the target fusion mode to integrate the target matching result, the millimeter-wave compensation result, and the scene perception parameters of the perception information to obtain a target fusion result, includes:

[0028] Obtain millimeter-wave sensing information to be processed from the information to be processed, obtain the fourth content obtained from the millimeter-wave sensing information to be processed using the matching rule, and determine the second unmatched content in the millimeter-wave sensing information to be processed other than the fourth content.

[0029] The dust detection algorithm is obtained, and the dust detection algorithm is used to determine multiple location information of multiple dust particles. Multiple associated sensing information associated with the multiple location information is determined in the second unmatched content, and the multiple associated sensing information is used as the millimeter wave compensation result.

[0030] The scene perception parameters are obtained from the perception information, the integration rules corresponding to the target fusion mode are obtained, and the integration rules are used to integrate the target matching result, the millimeter wave compensation result, and the scene perception parameters to obtain the target fusion result.

[0031] Optionally, the method further includes:

[0032] When the sensing information includes traditional sensing output results of lidar, intelligent sensing output results of lidar, millimeter-wave sensing output results, and scene sensing parameters, the traditional sensing information of lidar to be processed, the intelligent sensing information of lidar to be processed, the millimeter-wave sensing information to be processed, and the scene sensing information to be processed are obtained from the information to be processed.

[0033] The following methods are used to obtain: 1) Obtain specified traditional LiDAR perception information excluding the traditional LiDAR perception information to be processed from the traditional LiDAR perception output results; 2) Obtain specified intelligent LiDAR perception information excluding the intelligent LiDAR perception information to be processed from the intelligent LiDAR perception output results; 3) Obtain specified millimeter-wave perception information excluding the millimeter-wave perception information to be processed from the millimeter-wave perception output results; 4) Obtain specified scene perception information excluding the scene perception information to be processed from the scene perception parameters.

[0034] Obtain the first content determined in the traditional sensing information of the lidar to be processed using the matching rules, and determine the third unmatched content in the traditional sensing information of the lidar to be processed, excluding the first content.

[0035] The target fusion result is generated based on the specified traditional sensing information of the LiDAR, the specified intelligent sensing information of the LiDAR, the specified millimeter-wave sensing information, the specified scene sensing information, the third unmatched content, the target matching result, the millimeter-wave compensation result, and the scene sensing parameters.

[0036] Optionally, processing the target fusion result using the Kalman algorithm to obtain obstacle-related information includes:

[0037] Obtain the previous fusion result of the target fusion result, determine the time information corresponding to the previous fusion result, and obtain the time information corresponding to the target fusion result. Calculate the target time difference using the time information corresponding to the previous fusion result and the time information corresponding to the target fusion result.

[0038] The time difference algorithm is obtained, and the time difference algorithm is used to calculate the target time difference to obtain the velocity information corresponding to the target fusion result;

[0039] The sample vehicle corresponding to the perceived information is obtained, and the preset motion model corresponding to the sample vehicle is obtained. The preset motion model is a preset virtual model of uniform motion.

[0040] The speed information is obtained, and the preset motion model is used to predict the speed information and the target time difference to obtain the predicted position information.

[0041] The velocity information is obtained from the target fusion result, multiple target sensors corresponding to the target fusion mode are obtained, and multiple sensor detection position information corresponding to the multiple target sensors is obtained.

[0042] The Kalman algorithm is used to update the velocity information, the target fusion result, the multiple sensor detection position information, and the predicted position information to obtain obstacle-related information.

[0043] According to a second aspect of this application, a sensor data fusion apparatus is provided, the apparatus comprising:

[0044] The determination module is used to determine the target fusion mode corresponding to the perception information among multiple fusion modes when receiving input perception information, and to determine the target detection range corresponding to the target fusion mode using spatiotemporal synchronization operation;

[0045] The matching module is used to obtain the information to be processed within the target detection range from the perceived information, and to match the information to be processed using the matching rules corresponding to the target fusion mode to obtain the target matching result;

[0046] The integration module is used to perform compensation calculations on the information to be processed using a dust detection algorithm to obtain millimeter-wave compensation results, and to integrate the target matching results, the millimeter-wave compensation results, and the scene perception parameters of the perception information using the integration rules corresponding to the target fusion mode to obtain target fusion results;

[0047] The processing module is used to acquire the Kalman algorithm and process the target fusion result using the Kalman algorithm to obtain obstacle-related information.

[0048] Optionally, the determining module is used to determine multiple target sensors corresponding to the target fusion mode, obtain sensor status information from the perception information, and detect the sensor status information. The multiple target sensors include lidar, millimeter-wave radar, and camera sensors. The sensor status information indicates the working status of the multiple target sensors. If the sensor status information indicates that the multiple target sensors are working normally, the coordinate information of each target sensor is obtained. The module determines the sample vehicle corresponding to the perception information, obtains the rear axle center position information of the sample vehicle, and adjusts the coordinate information of each target sensor using the rear axle center position information to unify the coordinate origin of each target sensor to the rear axle center of the sample vehicle. The module obtains the timestamp of the lidar and sequentially determines the timestamps of the millimeter-wave radar and camera sensors that are close to the timestamp of the lidar. Using the timestamps of the lidar, millimeter-wave radar, and camera sensors, the lidar, millimeter-wave radar, and camera sensors are unified to the same timestamp. The module obtains the detection range of each target sensor, obtains multiple detection ranges, and determines the intersection of the multiple detection ranges to obtain the target detection range.

[0049] Optionally, the determining module is used to obtain the millimeter-wave detection result corresponding to the timestamp of the millimeter-wave radar and the camera detection result corresponding to the timestamp of the camera sensor; calculate the time difference between the timestamp of the millimeter-wave radar, the timestamp of the camera sensor and the timestamp of the lidar respectively to obtain the millimeter-wave time difference and the camera time difference; perform motion compensation operation on the millimeter-wave detection result using the millimeter-wave time difference and perform motion compensation operation on the camera detection result using the camera time difference, so that the lidar, the millimeter-wave radar and the camera sensor are at the same timestamp.

[0050] Optionally, the matching module is configured to acquire, from the sensing information, the traditional sensing output result of the lidar, the intelligent sensing output result of the lidar, the millimeter-wave sensing output result, and the scene sensing parameters of the lidar; and to acquire, from the traditional sensing output result of the lidar, the intelligent sensing output result of the lidar, the millimeter-wave sensing output result, and the scene sensing parameters, the lidar traditional sensing information to be processed, the lidar intelligent sensing information to be processed, the millimeter-wave sensing information to be processed, and the scene sensing parameters to be processed, respectively, the scene sensing parameters including communication cooperative sensing results and roadside sensing results; to use the lidar traditional sensing information to be processed, the lidar intelligent sensing information to be processed, the millimeter-wave sensing information to be processed, and the scene sensing information to be processed as the information to be processed; and to acquire the matching rule corresponding to the target fusion mode; and to use the matching rule to determine, in the lidar traditional sensing information to be processed, a first content matching the lidar intelligent sensing information to be processed, and in the information to be processed... The process involves: processing the intelligent sensing information of the LiDAR to obtain a second content that matches the first content; replacing the first content in the traditional sensing information of the LiDAR to be processed with the second content to obtain a first matching result; wherein, if there is a first unmatched content other than the second content in the intelligent sensing information of the LiDAR to be processed, the first unmatched content is filtered out; determining a third content that matches the millimeter-wave sensing information to be processed in the first matching result using the matching rules; obtaining a fourth content that matches the third content in the millimeter-wave sensing information to be processed; replacing the third content in the first matching result with the fourth content to obtain a second matching result; determining a fifth content that matches the scene sensing information to be processed in the second matching result using the matching rules; obtaining a sixth content that matches the fifth content in the scene sensing information to be processed; replacing the fifth content in the second matching result with the sixth content to obtain the target matching result.

[0051] Optionally, the integration module is configured to: acquire millimeter-wave sensing information to be processed from the information to be processed; acquire a fourth content acquired from the millimeter-wave sensing information to be processed using the matching rule; and determine a second unmatched content in the millimeter-wave sensing information to be processed other than the fourth content; acquire the dust detection algorithm; use the dust detection algorithm to determine multiple location information of multiple dust particles; determine multiple associated sensing information related to the multiple location information in the second unmatched content; and use the multiple associated sensing information as the millimeter-wave compensation result; acquire the scene perception parameters from the sensing information; acquire the integration rule corresponding to the target fusion mode; and use the integration rule to perform an integration operation on the target matching result, the millimeter-wave compensation result, and the scene perception parameters to obtain the target fusion result.

[0052] Optionally, the apparatus further includes a generation module.

[0053] The generation module is used to, when the sensing information includes traditional LiDAR sensing output results, intelligent LiDAR sensing output results, millimeter-wave sensing output results, and scene sensing parameters, obtain the traditional LiDAR sensing information to be processed, the intelligent LiDAR sensing information to be processed, the millimeter-wave sensing information to be processed, and the scene sensing information to be processed from the information to be processed; respectively, obtain the specified traditional LiDAR sensing information excluding the traditional LiDAR sensing information to be processed from the traditional LiDAR sensing output results, obtain the specified intelligent LiDAR sensing information excluding the intelligent LiDAR sensing information to be processed from the intelligent LiDAR sensing output results, and obtain the specified intelligent LiDAR sensing information excluding the intelligent LiDAR sensing information to be processed from the millimeter-wave sensing output results. Obtain specified millimeter-wave sensing information excluding the millimeter-wave sensing information to be processed; obtain specified scene sensing information excluding the scene sensing information to be processed from the scene sensing parameters; obtain a first content determined by the matching rule in the traditional sensing information of the lidar to be processed, and determine a third unmatched content in the traditional sensing information of the lidar to be processed other than the first content; generate the target fusion result based on the specified traditional sensing information of the lidar, the specified intelligent sensing information of the lidar, the specified millimeter-wave sensing information, the specified scene sensing information, the third unmatched content, the target matching result, the millimeter-wave compensation result, and the scene sensing parameters.

[0054] Optionally, the processing module is configured to: acquire the previous fusion result of the target fusion result; determine the time information corresponding to the previous fusion result; acquire the time information corresponding to the target fusion result; calculate the target time difference using the time information corresponding to the previous fusion result and the time information corresponding to the target fusion result; acquire a time difference algorithm; calculate the speed information corresponding to the target fusion result using the time difference algorithm; acquire the sample vehicle corresponding to the perception information; acquire the preset motion model corresponding to the sample vehicle, wherein the preset motion model is a preset virtual model of uniform motion; acquire the speed information; perform a prediction operation on the speed information and the target time difference using the preset motion model to obtain predicted position information; acquire the speed information in the target fusion result; acquire multiple target sensors corresponding to the target fusion mode; acquire the multiple sensor detection position information corresponding to the multiple target sensors; and update the speed information, the target fusion result, the multiple sensor detection position information, and the predicted position information using the Kalman algorithm to obtain obstacle-related information.

[0055] According to a third aspect of this application, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in any of the first aspects above.

[0056] According to a fourth aspect of this application, a computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of the first aspects above.

[0057] By means of the above technical solutions, this application provides a sensor data fusion method, apparatus, computer equipment, and computer-readable storage medium. When receiving input sensing information, this application determines the target fusion mode corresponding to the sensing information among multiple fusion modes, determines the target detection range corresponding to the target fusion mode using spatiotemporal synchronization operation, acquires the information to be processed within the target detection range from the sensing information, matches the information to be processed using the matching rules corresponding to the target fusion mode to obtain the target matching result, performs compensation calculation on the information to be processed using a dust detection algorithm to obtain the millimeter-wave compensation result, and integrates the target matching result, millimeter-wave compensation result, and scene perception parameters of the sensing information using the integration rules corresponding to the target fusion mode to obtain the target fusion result, obtains the Kalman algorithm, processes the target fusion result using the Kalman algorithm to obtain obstacle-related information, supplements the fused sensing result with the detection results of millimeter-wave radar, enhances the vehicle's perception capability under dusty conditions, and expands the vehicle's perception range and improves perception accuracy by utilizing V2V collaborative perception results and roadside perception results, enabling the vehicle to better adapt to the mining environment.

[0058] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0059] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0060] Figure 1 This illustration shows a schematic flowchart of a sensor data fusion method provided in an embodiment of this application;

[0061] Figure 2A This illustration shows a schematic flowchart of a sensor data fusion method provided in an embodiment of this application;

[0062] Figure 2B This illustration shows a flowchart of a fusion tracking method provided in an embodiment of this application;

[0063] Figure 2C This illustration shows a flowchart of a multi-sensor perception fusion method provided in an embodiment of this application;

[0064] Figure 3A This illustration shows a schematic diagram of a sensor data fusion structure provided in an embodiment of this application;

[0065] Figure 3B This illustration shows a schematic diagram of a sensor data fusion structure provided in an embodiment of this application;

[0066] Figure 4 A schematic diagram of the device structure of a computer device provided in an embodiment of this application is shown. Detailed Implementation

[0067] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0068] This application provides a sensor data fusion method, such as... Figure 1 As shown, the method includes:

[0069] 101. When receiving input perception information, determine the target fusion mode corresponding to the perception information among multiple fusion modes, and use spatiotemporal synchronization operation to determine the target detection range corresponding to the target fusion mode.

[0070] Due to the dusty environment and complex road conditions in open-pit mines, relying on a single sensor cannot solve all problems. Therefore, data fusion using multiple sensors is crucial. However, even when fusing outputs from LiDAR, millimeter-wave radar, and images, related technologies still cannot accurately detect dusty conditions and may miss detections in complex road environments, resulting in low sensing accuracy.

[0071] To address this issue, this application proposes a sensor data fusion method. It utilizes traditional LiDAR perception output, LiDAR AI (Artificial Intelligence) perception output, millimeter-wave perception output, and sensor status information. The method employs algorithms such as multi-sensor spatiotemporal synchronization, FOV (angle of view) determination, spatial matching, millimeter-wave detection result compensation, and V2V (Vehicle-to-Vehicle) fusion to fuse the detection results. The fused results are then used for tracking, ultimately outputting the obstacle's position, size, orientation angle, category, speed, and tracking ID (Identity Document). The implementing entity of this application can be a multi-sensor fusion system. This system relies on the computing power of a server to provide services to users. The server can be a standalone server or a server providing basic cloud computing services such as cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. This allows the multi-sensor fusion system to analyze and obtain more accurate driving environment information, improving perception accuracy.

[0072] In this embodiment, when input sensing information is received, the multi-sensor fusion system determines the target fusion mode corresponding to the sensing information from multiple fusion modes. The sensing information is the sensing output of multiple sensors at a certain moment, and the fusion mode can be selected based on the actual operating effect. Then, the multi-sensor fusion system uses spatiotemporal synchronization to determine the target detection range corresponding to the target fusion mode. Spatiotemporal synchronization ensures that obstacle information detected by different sensors occurs at the same time point, and that the detected position information is unified under a single coordinate system. In this way, the multi-sensor fusion system can input sensing information and determine the corresponding target detection range according to the actual application scenario, improving the accuracy of fused sensing.

[0073] 102. Obtain the information to be processed within the target detection range from the perceived information, and use the matching rules corresponding to the target fusion mode to match the information to be processed to obtain the target matching result.

[0074] To ensure the safety of unmanned driving in mining areas, it is necessary to improve the accuracy and stability of fused perception. Therefore, a multi-sensor fusion system matches target perception information acquired from multiple sensors. In this embodiment, the multi-sensor fusion system acquires the information to be processed within the target detection range from the perception information, and uses the matching rules corresponding to the target fusion mode to match the information to be processed, thereby obtaining the target matching result. By matching multiple data in the information to be processed, the multi-sensor fusion system corrects and supplements the matching result, thereby improving the accuracy of fused perception.

[0075] 103. The dust detection algorithm is used to perform compensation calculations on the information to be processed to obtain the millimeter wave compensation result. The scene perception parameters of the target matching result, the millimeter wave compensation result, and the perception information are integrated using the integration rules corresponding to the target fusion mode to obtain the target fusion result.

[0076] In this embodiment, the multi-sensor fusion system utilizes a dust detection algorithm to perform compensation calculations on the information to be processed, obtaining millimeter-wave compensation results. Specifically, the compensation calculation uses the location information of dust detection by lidar to search for millimeter-wave radar detection results near the dust. If a millimeter-wave detection result is found, it is considered a missed detection caused by dust obstruction, and the millimeter-wave detection result is recalled, enhancing the vehicle's perception capabilities and improving perception accuracy. Next, the multi-sensor fusion system integrates the target matching results, millimeter-wave compensation results, and scene perception parameters of the perception information using the integration rules corresponding to the target fusion mode, obtaining the target fusion result. To avoid vehicle collision risks, this application employs V2V fusion technology, which fuses scene perception parameters including V2V collaborative perception results and roadside perception results. This addresses the risk of collisions caused by vehicle-side sensors failing to detect obstacles in complex road conditions.

[0077] 104. Obtain the Kalman algorithm, and use the Kalman algorithm to process the target fusion results to obtain obstacle-related information.

[0078] In this embodiment, the multi-sensor fusion system acquires Kalman algorithm data and processes the target fusion result using the Kalman algorithm to obtain obstacle-related information. Thus, the multi-sensor fusion system processes and tracks the obtained target fusion result, and by processing the target fusion result using the Kalman algorithm, it can obtain obstacle-related information such as the obstacle's position, size, orientation angle, category, speed, and tracking ID. This provides more accurate driving environment information, enabling the vehicle to better adapt to the mining environment.

[0079] The method provided in this application, when receiving input perception information, determines the target fusion mode corresponding to the perception information among multiple fusion modes, determines the target detection range corresponding to the target fusion mode using spatiotemporal synchronization operation, acquires the information to be processed within the target detection range from the perception information, matches the information to be processed using the matching rules corresponding to the target fusion mode to obtain the target matching result, performs compensation calculation on the information to be processed using a dust detection algorithm to obtain the millimeter-wave compensation result, integrates the target matching result, the millimeter-wave compensation result, and the scene perception parameters of the perception information using the integration rules corresponding to the target fusion mode to obtain the target fusion result, acquires the Kalman algorithm, processes the target fusion result using the Kalman algorithm to obtain obstacle-related information, supplements the fused perception result with the detection results of millimeter-wave radar, enhances the vehicle's perception capability under dusty conditions, and expands the vehicle's perception range and improves perception accuracy by utilizing V2V collaborative perception results and roadside perception results, enabling the vehicle to better adapt to the mining environment.

[0080] Furthermore, as a refinement and extension of the specific implementation methods of the above embodiments, and in order to fully illustrate the specific implementation process of this embodiment, this application provides another sensor data fusion method, such as... Figure 2A As shown, the method includes:

[0081] 201. When receiving input perception information, determine the target fusion mode corresponding to the perception information among multiple fusion modes.

[0082] Users manually input combined information into the multi-sensor fusion system. This information combination includes fusion of traditional and intelligent LiDAR perception outputs, millimeter-wave perception outputs, sensor status information, communication-coordinated perception results, and roadside perception results. Users can select the appropriate information combination based on the current driving environment. In this embodiment, when receiving input perception information, the multi-sensor fusion system determines the target fusion mode corresponding to the perception information from multiple fusion modes. Specifically, the multi-sensor fusion system primarily loads different fusion modes by reading a configuration file during program initialization. The system includes three fusion modes: Mode 1 uses fusion of traditional and intelligent LiDAR perception outputs; Mode 2 uses fusion of traditional LiDAR perception outputs, intelligent LiDAR perception outputs, and millimeter-wave perception outputs; and Mode 3 uses fusion of traditional LiDAR perception outputs, intelligent LiDAR perception outputs, millimeter-wave perception outputs, communication-coordinated perception results, and roadside perception results. Users can select the appropriate multi-sensor perception fusion input based on the vehicle's actual operating environment and performance. For example, millimeter-wave radar detection results may be poor in certain environments. Users can manually select to use only lidar detection results for perception fusion. This application describes mode three as an example, which supplements the fused perception results with millimeter-wave sensing output results, enhancing the vehicle's perception capability in dusty conditions. At the same time, it expands the vehicle's perception range and improves perception accuracy by utilizing communication-coordinated sensing results and roadside sensing results, enabling the vehicle to better adapt to the mining environment.

[0083] 202. Determine the sample vehicle corresponding to the sensing information, obtain the rear axle center position information of the sample vehicle, and use the rear axle center position information to adjust the coordinate information of each target sensor so that the coordinate origin of each target sensor is unified to the rear axle center of the sample vehicle.

[0084] To ensure the accuracy of multi-sensor data fusion, this application proposes spatial synchronization of multiple sensors, that is, unifying the coordinate axes of each sensor to the center of the vehicle's rear axle as the origin. In the embodiments of this application, the multi-sensor fusion system determines multiple target sensors corresponding to the target fusion mode, including LiDAR, millimeter-wave radar, and camera sensors. It should be noted that if the target fusion mode corresponding to the perception information is mode one, then the multiple target sensors include LiDAR and camera sensors; if the target fusion mode corresponding to the perception information is mode two, then the multiple sensors include LiDAR, millimeter-wave radar, and camera sensors. Next, the multi-sensor fusion system acquires sensor status information from the perception information and detects the sensor status information, whereby the sensor status information indicates the working status of multiple target sensors. If the sensor status information indicates that the multiple target sensors are working normally, then the coordinate information of each target sensor is acquired. In this way, the normal operation of multiple sensors can be monitored through sensor status information, and anomalies can be dealt with in a timely manner to avoid the data fusion of the multi-sensor fusion system being affected by abnormal sensor perception information. Subsequently, the multi-sensor fusion system identifies the sample vehicle corresponding to the perceived information, acquires the rear axle center position information of the sample vehicle, and uses this information to adjust the coordinate information of each target sensor, ensuring that the origin of each target sensor's coordinates is aligned with the rear axle center of the sample vehicle. By unifying the coordinate axes of each sensor to the rear axle center of the sample vehicle as the origin, the multi-sensor fusion system achieves spatial synchronization of multiple sensors, enabling more accurate determination of the target detection range and thus obtaining more accurate data for matching.

[0085] 203. Utilize the timestamps of LiDAR, millimeter-wave radar, and camera sensor to unify LiDAR, millimeter-wave radar, and camera sensor under the same timestamp.

[0086] Since multi-sensor data fusion relies on spatiotemporal synchronization, the positional information detected by different sensors needs to be unified under a single coordinate system, and the detected obstacle information must be at the same point in time. Therefore, time synchronization of multiple sensors is necessary. Using the lidar timestamp as a reference, millimeter-wave radar, V2V sensors, etc., find the closest detection frame to the lidar timestamp based on their own timestamps. Then, motion compensation is performed based on the time difference to synchronize them to the same timestamp as the lidar, thus completing the spatiotemporal synchronization operation of the multi-sensor system. In this embodiment, the multi-sensor fusion system acquires the lidar timestamp and sequentially determines the timestamps of the millimeter-wave radar and camera sensors that are close to the lidar timestamp. Next, the multi-sensor fusion system acquires the millimeter-wave detection result corresponding to the millimeter-wave radar timestamp and the camera detection result corresponding to the camera sensor timestamp, so that motion compensation can be performed on multiple detection results subsequently. Then, the multi-sensor fusion system calculates the time difference between the millimeter-wave radar timestamp, the camera sensor timestamp, and the lidar timestamp, respectively, to obtain the millimeter-wave time difference and the camera time difference. Finally, the multi-sensor fusion system uses millimeter-wave time difference to perform motion compensation on millimeter-wave detection results and camera time difference to perform motion compensation on camera detection results, so that the lidar, millimeter-wave radar, and camera sensors are at the same timestamp, thereby enabling the determination of the target detection range.

[0087] 204. Obtain the detection range of each target sensor, get multiple detection ranges, and determine the intersection of multiple detection ranges to obtain the target detection range.

[0088] Since each sensor has its own detection range, this application performs a Field of View (FOV) determination on the intersection of the detection ranges of multiple sensors to improve the accuracy of fused perception. Detection results within the FOV range are matched, and detection results outside the FOV range can be used as a supplement to the overall perception output, further improving perception accuracy. In this embodiment, the multi-sensor fusion system acquires the detection range of each target sensor, obtaining multiple detection ranges, and determines the intersection of these multiple detection ranges to obtain the target detection range. Each sensor has its own detection range, which is the actual perception range. The multi-sensor fusion system determines the target detection range based on the intersection of the detection ranges of the LiDAR, millimeter-wave radar, and camera sensors. It should be noted that if the target fusion mode is Mode 1, the target detection range is determined based on the intersection of the LiDAR and camera sensor detection ranges; if the target fusion mode is Mode 2, the target detection range is determined based on the intersection of the detection ranges of the LiDAR, millimeter-wave radar, and camera sensors. In this way, the multi-sensor fusion system determines the target detection range by performing FOV determination on multiple spatiotemporally synchronized sensors, enabling subsequent matching of detection results within the target detection range and improving the data fusion accuracy of the multi-sensor fusion system.

[0089] 205. Obtain the information to be processed within the target detection range from the perceived information.

[0090] In this embodiment, the multi-sensor fusion system acquires traditional LiDAR perception output results, intelligent LiDAR perception output results, millimeter-wave perception output results, and scene perception parameters from the perceived information. It then acquires, from these sources, the unprocessed traditional LiDAR perception information, unprocessed intelligent LiDAR perception information, unprocessed millimeter-wave perception information, and unprocessed scene perception information within the target detection range. The scene perception parameters include communication-coordinated perception results and roadside perception results. For the complex working conditions in mining areas, scene perception parameters can better expand the perception range. Furthermore, considering the problem that traditional LiDAR detection algorithms may regress a single target to multiple targets, this application employs a LiDAR AI detection algorithm to correct the detection results, utilizing intelligent LiDAR perception information to improve perception accuracy. Moreover, considering the inaccuracy of the bounding box size output by traditional LiDAR detection algorithms, this application also uses intelligent LiDAR perception information to correct the bounding box size. Then, the multi-sensor fusion system takes the traditional sensing information of the lidar to be processed, the intelligent sensing information of the lidar to be processed, the millimeter-wave sensing information to be processed, and the scene sensing information to be processed as the information to be processed, and can determine the sensing information for target matching, thereby improving the efficiency of data fusion in the multi-sensor fusion system.

[0091] It should be noted that if the target fusion mode is Mode 1, the multi-sensor fusion system acquires the traditional sensing output and intelligent sensing output of the LiDAR from the sensing information, and then acquires the LiDAR traditional sensing information and intelligent sensing information to be processed that are within the target detection range from the traditional sensing output and intelligent sensing output, respectively. The multi-sensor fusion system then uses the LiDAR traditional sensing information and intelligent sensing information to be processed as the information to be processed.

[0092] If the target fusion mode is Mode 2, the multi-sensor fusion system acquires the traditional LiDAR perception output, intelligent LiDAR perception output, and millimeter-wave perception output from the perceived information. It then acquires the LiDAR traditional perception information, intelligent LiDAR perception information, and millimeter-wave perception information within the target detection range from these three outputs. Finally, the multi-sensor fusion system uses these three information as the information to be processed.

[0093] 206. Use the matching rules corresponding to the target fusion mode to match the information to be processed and obtain the target matching result.

[0094] In this embodiment, the multi-sensor fusion system acquires the matching rules corresponding to the target fusion mode. Next, the multi-sensor fusion system uses the matching rules to determine a first content matching the intelligent sensing information of the LiDAR to be processed within the traditional sensing information of the LiDAR to be processed, and acquires a second content matching the first content within the intelligent sensing information of the LiDAR to be processed. That is, using the intelligent sensing information of the LiDAR to be processed as a benchmark, the system searches for traditional sensing information of the LiDAR to be processed within a certain range. Subsequently, the multi-sensor fusion system uses the second content to replace the first content in the traditional sensing information of the LiDAR to be processed, obtaining a first matching result. That is, the system replaces the corresponding traditional sensing information of the LiDAR to be processed with the intelligent sensing information of the LiDAR to be processed. If there is a first unmatched content in the intelligent sensing information of the LiDAR to be processed other than the second content, it indicates that the intelligent sensing information of the LiDAR to be processed is a false detection, and the first unmatched content is filtered out. It should be noted that after all the intelligent sensing information of the LiDAR to be processed has been traversed, if there is still unmatched traditional sensing information of the LiDAR to be processed, then the traditional sensing information of the LiDAR to be processed needs to be retained. In this way, the multi-sensor fusion system can match the traditional sensing output of lidar with the intelligent sensing output of lidar.

[0095] Then, the multi-sensor fusion system uses matching rules to determine the third content matching the millimeter-wave sensing information to be processed from the first matching result, and obtains the fourth content matching the third content from the millimeter-wave sensing information to be processed. In other words, it searches for the millimeter-wave sensing information to be processed within a certain range, using the first matching result as a benchmark. Next, the multi-sensor fusion system replaces the third content in the first matching result with the fourth content to obtain the second matching result, which is essentially replacing the corresponding first matching result with the millimeter-wave sensing information to be processed. The first matching results that do not match need to be retained, and the millimeter-wave sensing information that does not match needs to be compensated for later using a dust detection algorithm. In this way, the multi-sensor fusion system can match the traditional sensing output results of LiDAR, the intelligent sensing output results of LiDAR, and the millimeter-wave sensing output results. It should be noted that, to improve the accuracy of target matching, Kalman filtering can be used to fuse the first matching results and calculate the fusion coefficient of the detected target box. Since vehicle operation in mining environments generates dust, LiDAR may miss detections. Since millimeter waves have good penetration of dust, supplementing the detection results with millimeter-wave radar can enhance the sensing capability.

[0096] Subsequently, the multi-sensor fusion system uses matching rules to determine the fifth element matching the scene perception information to be processed from the second matching result, and then obtains the sixth element matching the fifth element from the scene perception information to be processed. In other words, it searches for scene perception information to be processed within a certain range, using the second matching result as a benchmark. It should be noted that the multi-sensor fusion system can also improve perception accuracy by fusing vehicle-side perception information with roadside perception information from the scene perception information to be processed, solving the problem of missed target detection due to dust obstruction and the complex working conditions in mining areas. Finally, the multi-sensor fusion system uses the sixth element to replace the fifth element in the second matching result. Second matching results that do not match are retained to obtain the target matching result, which is to replace the corresponding second matching result with the scene perception information to be processed. Unlike urban roads, mining areas have complex road conditions, and relying solely on sensors for environmental perception can lead to missed detections and collision risks. Therefore, combining V2V collaborative perception information sent from V2V, roadside perception information, and other vehicle positioning information can help avoid these risks.

[0097] It should be noted that if the target fusion mode is mode one, the multi-sensor fusion system will use the first matching result as the target matching result; if the target fusion mode is mode two, the multi-sensor fusion system will use the second matching result as the target matching result.

[0098] 207. The dust detection algorithm is used to perform compensation calculations on the information to be processed, and the millimeter wave compensation results are obtained.

[0099] Because lidar is prone to missing obstacles in areas with dust and moisture, a dust detection algorithm from traditional lidar algorithms can be used to retrieve dust location information from lidar point cloud reflection intensity and density information, supplementing the lidar output for missed detections. In this embodiment, the multi-sensor fusion system acquires millimeter-wave sensing information from the information to be processed, and obtains the fourth content obtained from the millimeter-wave sensing information using matching rules. Next, the multi-sensor fusion system determines the second unmatched content in the millimeter-wave sensing information, excluding the fourth content, which is the millimeter-wave sensing information that was not matched during the target matching process. Subsequently, the multi-sensor fusion system acquires a dust detection algorithm and uses it to determine multiple location information of multiple dust particles. Then, the multi-sensor fusion system determines multiple associated sensing information related to multiple location information in the second unmatched content, and uses these multiple associated sensing information as millimeter-wave compensation results. By iterating through the millimeter-wave output results in the second unmatched content to see if they are near the dust detection location, and retaining the results if they are, the perception capability of the sample vehicle can be enhanced.

[0100] 208. The target fusion result is obtained by integrating the target matching result, millimeter wave compensation result, and scene perception parameters of the perception information using the integration rules corresponding to the target fusion mode.

[0101] In this embodiment, the multi-sensor fusion system acquires scene perception parameters from the perceived information and obtains the integration rules corresponding to the target fusion mode. Then, the multi-sensor fusion system uses the integration rules to integrate the target matching results, millimeter-wave compensation results, and scene perception parameters to obtain the target fusion result. The multi-sensor fusion system utilizes LiDAR, millimeter-wave radar, V2V collaborative perception results, and roadside perception results for fusion, which can improve perception accuracy and enable the sample vehicle to better adapt to the mining environment.

[0102] In an optional implementation, to improve perception accuracy, the multi-sensor fusion system can integrate detection results outside the target detection range, matching results of traditional LiDAR perception outputs, intelligent LiDAR perception outputs, millimeter-wave perception outputs, communication-coordinated perception results, and roadside perception results within the target detection range, as well as unmatched traditional LiDAR perception outputs and millimeter-wave compensated detection results, communication-coordinated perception results, and roadside perception results. Specifically, when the perception information includes traditional LiDAR perception outputs, intelligent LiDAR perception outputs, millimeter-wave perception outputs, and scene perception parameters, the multi-sensor fusion system acquires the LiDAR traditional perception information, intelligent LiDAR perception information, millimeter-wave perception information, and scene perception information to be processed from the information to be processed. Next, the multi-sensor fusion system acquires specified LiDAR traditional perception information (excluding the LiDAR traditional perception information to be processed) from the LiDAR traditional perception output, specified LiDAR intelligent perception information (excluding the LiDAR intelligent perception information to be processed) from the LiDAR intelligent perception output, specified millimeter-wave perception information (excluding the millimeter-wave perception information to be processed) from the millimeter-wave perception output, and specified scene perception information (excluding the scene perception information to be processed) from the scene perception parameters. Subsequently, the multi-sensor fusion system acquires the first content determined using matching rules from the LiDAR traditional perception information to be processed, and the third unmatched content (excluding the first content) from the LiDAR traditional perception information to be processed. Finally, the multi-sensor fusion system generates a target fusion result based on the specified LiDAR traditional perception information, specified LiDAR intelligent perception information, specified millimeter-wave perception information, specified scene perception information, the third unmatched content, the target matching result, the millimeter-wave compensation result, and the scene perception parameters.

[0103] It should be noted that if the target fusion mode is Mode 1, the integration rule for Mode 1 is to integrate the matching results of the traditional sensing output of the matched LiDAR and the intelligent sensing output of the LiDAR, as well as the traditional sensing output of the unmatched LiDAR, to obtain the target fusion result. If the target fusion mode is Mode 2, the integration rule is to integrate the detection results outside the target detection range, the matching results of the traditional sensing output of the matched LiDAR, the intelligent sensing output of the LiDAR, and the millimeter-wave sensing output within the target detection range, the traditional sensing output of the unmatched LiDAR, and the millimeter-wave compensated detection results, to obtain the target fusion result. In this way, sensor data can be fused using different fusion modes according to different application scenarios. For example, the detection effect of millimeter-wave radar is poor in a certain environment, so it is possible to choose to use only the LiDAR detection results for perception fusion.

[0104] 209. Obtain the Kalman algorithm, and use the Kalman algorithm to process the target fusion results to obtain obstacle-related information.

[0105] After obtaining the target fusion result, the multi-sensor fusion system uses the target fusion result for tracking, providing the sample vehicle with relevant information such as the position, size, category, speed, and ID of obstacles. In this embodiment, the multi-sensor fusion system acquires the previous fusion result of the target fusion result and determines the time information corresponding to the previous fusion result. Next, the multi-sensor fusion system acquires the time information corresponding to the target fusion result and calculates the target time difference using the time information corresponding to the previous fusion result and the time information corresponding to the target fusion result. Subsequently, the multi-sensor fusion system acquires the time difference algorithm and calculates the target time difference using the time difference algorithm to obtain the speed information corresponding to the target fusion result. Further, the multi-sensor fusion system acquires the sample vehicle corresponding to the perception information and the preset motion model corresponding to the sample vehicle. The preset motion model is a preset virtual model of uniform motion. During the uniform motion of the sample vehicle, each perceived target has speed information. The multi-sensor fusion system then calculates the time interval based on the time information and predicts the target's location using the speed information and time interval through the preset motion model, thereby completing Kalman prediction. Then, the multi-sensor fusion system acquires speed information and uses a preset motion model to predict the speed information and target time difference to obtain predicted position information. Next, the multi-sensor fusion system extracts speed information from the target fusion result, acquires multiple target sensors corresponding to the target fusion pattern, and acquires the detection position information of multiple sensors corresponding to multiple target sensors. Finally, the multi-sensor fusion system uses the Kalman algorithm to update the speed information, target fusion result, multiple sensor detection position information, and predicted position information to obtain obstacle-related information, which improves perception accuracy and provides more precise driving environment information. In this way, the multi-sensor fusion system maintains a cache list using the obtained obstacle-related information to complete target tracking, enabling the vehicle to better adapt to the mining environment.

[0106] Based on the above process, the flowchart of a fusion tracking method proposed in this application is as follows:

[0107] like Figure 2BAs shown, the multi-sensor fusion system calculates the time difference between the currently detected target and the corresponding target in the previous frame, and determines whether velocity information exists in the current detection result. If no velocity information exists, it calculates the velocity information using the time difference. If velocity information exists, it performs Kalman prediction using a predefined motion model. Next, the multi-sensor fusion system updates information such as timestamps, obstacle heading angles, and obstacle heights, and then performs Kalman updates by inputting sensor detection positions and velocity information to obtain obstacle-related information.

[0108] In summary, the flowchart of a multi-sensor perception fusion proposed in this application is as follows:

[0109] like Figure 2C As shown, the input source information in the multi-sensor fusion system includes: 1. output results of traditional LiDAR algorithms and AI; 2. millimeter-wave output results; 3. V2V collaborative perception results and roadside perception results; and 4. vehicle positioning information. The user inputs a combination of information, which can be either 1, 2, 3, and 4, or 1, 3, and 4. When the user selects combination 1, 2, 3, and 4, the algorithm processing of the multi-sensor fusion system is as follows: first, multi-sensor spatiotemporal synchronization is performed; then, intersection FOV judgment and target matching are completed; then, fusion tracking is performed through millimeter-wave compensation and V2V information compensation, that is, the Hungarian matching algorithm is used to complete track matching; then, Kalman filtering is used to fuse the current frame result and the historical frame result, resulting in lower position noise in the output detection box; finally, the main controller provides the final output obstacle information, position, size, orientation angle, speed, and tracking ID. When the user selects combination 1, 3, or 4, the algorithm processing of the multi-sensor fusion system is as follows: first, multi-sensor spatiotemporal synchronization is performed; then, target matching is performed; next, fusion tracking is performed through V2V information compensation; and finally, the system outputs information about the obstacle, including its position, size, orientation angle, speed, and tracking ID.

[0110] The method provided in this application, when receiving input perception information, determines the target fusion mode corresponding to the perception information among multiple fusion modes, determines the target detection range corresponding to the target fusion mode using spatiotemporal synchronization operation, acquires the information to be processed within the target detection range from the perception information, matches the information to be processed using the matching rules corresponding to the target fusion mode to obtain the target matching result, performs compensation calculation on the information to be processed using a dust detection algorithm to obtain the millimeter-wave compensation result, integrates the target matching result, the millimeter-wave compensation result, and the scene perception parameters of the perception information using the integration rules corresponding to the target fusion mode to obtain the target fusion result, acquires the Kalman algorithm, processes the target fusion result using the Kalman algorithm to obtain obstacle-related information, supplements the fused perception result with the detection results of millimeter-wave radar, enhances the vehicle's perception capability under dusty conditions, and expands the vehicle's perception range and improves perception accuracy by utilizing V2V collaborative perception results and roadside perception results, enabling the vehicle to better adapt to the mining environment.

[0111] Furthermore, as Figure 1 To specifically implement the method, this application provides a sensor data fusion device, such as... Figure 3A As shown, the device includes: a determining module 301, a matching module 302, an integration module 303, and a processing module 304.

[0112] The determination module 301 is used to determine the target fusion mode corresponding to the perception information among multiple fusion modes when the input perception information is received, and to determine the target detection range corresponding to the target fusion mode by using spatiotemporal synchronization operation.

[0113] The matching module 302 is used to acquire the information to be processed within the target detection range from the perceived information, and to match the information to be processed using the matching rules corresponding to the target fusion mode to obtain the target matching result;

[0114] The integration module 303 is used to perform compensation calculations on the information to be processed using a dust detection algorithm to obtain millimeter wave compensation results, and to integrate the target matching results, millimeter wave compensation results, and scene perception parameters of the perception information using the integration rules corresponding to the target fusion mode to obtain target fusion results;

[0115] The processing module 304 is used to acquire the Kalman algorithm, process the target fusion result using the Kalman algorithm, and obtain obstacle-related information.

[0116] In specific application scenarios, the determining module 301 is used to determine multiple target sensors corresponding to the target fusion mode, acquire sensor status information from the perception information, and detect the sensor status information. The multiple target sensors include LiDAR, millimeter-wave radar, and camera sensors. The sensor status information indicates the working status of multiple target sensors. If the sensor status information indicates that the multiple target sensors are working normally, the coordinate information of each target sensor is acquired. The sample vehicle corresponding to the perception information is determined, the rear axle center position information of the sample vehicle is acquired, and the coordinate information of each target sensor is adjusted using the rear axle center position information to unify the coordinate origin of each target sensor to the rear axle center of the sample vehicle. The timestamp of the LiDAR is acquired, and the timestamps of the millimeter-wave radar and camera sensors that are close to the timestamp of the LiDAR are determined in sequence. The timestamps of the LiDAR, millimeter-wave radar, and camera sensors are unified to the same timestamp using the timestamps of the LiDAR, millimeter-wave radar, and camera sensors. The detection range of each target sensor is acquired, multiple detection ranges are obtained, and the intersection of multiple detection ranges is determined to obtain the target detection range.

[0117] In specific application scenarios, the determining module 301 is used to obtain the millimeter-wave detection results corresponding to the timestamp of the millimeter-wave radar and the camera detection results corresponding to the timestamp of the camera sensor; calculate the time difference between the timestamp of the millimeter-wave radar, the timestamp of the camera sensor and the timestamp of the lidar respectively to obtain the millimeter-wave time difference and the camera time difference; use the millimeter-wave time difference to perform motion compensation operation on the millimeter-wave detection results and use the camera time difference to perform motion compensation operation on the camera detection results, so that the lidar, millimeter-wave radar and camera sensor are under the same timestamp.

[0118] In specific application scenarios, the matching module 302 is used to acquire the traditional sensing output results, intelligent sensing output results, millimeter-wave sensing output results, and scene perception parameters of the LiDAR from the sensing information; and to acquire, from the traditional sensing output results, intelligent sensing output results, millimeter-wave sensing output results, and scene perception parameters of the LiDAR to be processed within the target detection range, the traditional sensing information, intelligent sensing information, millimeter-wave sensing information, and scene perception information to be processed, respectively. The scene perception parameters include communication cooperative sensing results and roadside sensing results; to use the traditional sensing information, intelligent sensing information, millimeter-wave sensing information, and scene perception information to be processed as the information to be processed; and to acquire the matching rules corresponding to the target fusion mode; and to use the matching rules to determine the matching between the traditional sensing information and the intelligent sensing information of the LiDAR to be processed. The process involves: firstly, obtaining a second content matching the first content from the intelligent sensing information of the LiDAR to be processed; and replacing the first content in the traditional sensing information of the LiDAR to be processed with the second content to obtain a first matching result. If there is a first unmatched content other than the second content in the intelligent sensing information of the LiDAR to be processed, the first unmatched content is filtered out. Then, using matching rules, a third content matching the millimeter-wave sensing information to be processed is determined from the first matching result. A fourth content matching the third content is obtained from the millimeter-wave sensing information to be processed, and the third content is replaced in the first matching result with the fourth content to obtain a second matching result. Finally, using matching rules, a fifth content matching the scene sensing information to be processed is determined from the second matching result. A sixth content matching the fifth content is obtained from the scene sensing information to be processed, and the fifth content is replaced in the second matching result with the sixth content to obtain a target matching result.

[0119] In specific application scenarios, the integration module 303 is used to acquire millimeter-wave sensing information to be processed from the information to be processed, acquire the fourth content acquired from the millimeter-wave sensing information to be processed using matching rules, and determine the second unmatched content in the millimeter-wave sensing information to be processed other than the fourth content; acquire a dust detection algorithm, use the dust detection algorithm to determine multiple location information of multiple dust particles, determine multiple associated sensing information related to multiple location information in the second unmatched content, and use the multiple associated sensing information as the millimeter-wave compensation result; acquire scene perception parameters from the sensing information, acquire the integration rules corresponding to the target fusion mode, and use the integration rules to integrate the target matching result, the millimeter-wave compensation result, and the scene perception parameters to obtain the target fusion result.

[0120] In specific application scenarios, such as Figure 3B As shown, the device also includes a generation module 305.

[0121] The generation module 305 is used to, when the perception information includes traditional LiDAR perception output results, intelligent LiDAR perception output results, millimeter-wave perception output results, and scene perception parameters, obtain the traditional LiDAR perception information to be processed, the intelligent LiDAR perception information to be processed, the millimeter-wave perception information to be processed, and the scene perception information to be processed from the information to be processed; respectively obtain the specified traditional LiDAR perception information excluding the traditional LiDAR perception information to be processed from the traditional LiDAR perception output results, the specified intelligent LiDAR perception information excluding the intelligent LiDAR perception information to be processed from the intelligent LiDAR perception output results, the specified millimeter-wave perception information excluding the millimeter-wave perception information to be processed from the millimeter-wave perception output results, and the specified scene perception information excluding the scene perception information to be processed from the scene perception parameters; obtain the first content determined by the matching rules in the traditional LiDAR perception information to be processed, and determine the third unmatched content in the traditional LiDAR perception information to be processed other than the first content; and generate a target fusion result based on the specified traditional LiDAR perception information, the specified intelligent LiDAR perception information, the specified millimeter-wave perception information, the specified scene perception information, the third unmatched content, the target matching result, the millimeter-wave compensation result, and the scene perception parameters.

[0122] In specific application scenarios, the processing module 304 is used to obtain the previous fusion result of the target fusion result, determine the time information corresponding to the previous fusion result, and obtain the time information corresponding to the target fusion result. It then calculates the target time difference using the time information corresponding to the previous fusion result and the time information corresponding to the target fusion result. Next, it obtains a time difference algorithm and calculates the target time difference using the time difference algorithm to obtain the speed information corresponding to the target fusion result. It also obtains the sample vehicle corresponding to the perception information and the preset motion model corresponding to the sample vehicle. The preset motion model is a preset virtual model of uniform motion. Finally, it obtains the speed information and performs prediction operations on the speed information and the target time difference using the preset motion model to obtain the predicted position information. It also obtains the speed information from the target fusion result, acquires multiple target sensors corresponding to the target fusion mode, and acquires the multiple sensor detection position information corresponding to the multiple target sensors. Finally, it uses the Kalman algorithm to update the speed information, the target fusion result, the multiple sensor detection position information, and the predicted position information to obtain obstacle-related information.

[0123] The apparatus provided in this application, when receiving input sensing information, determines the target fusion mode corresponding to the sensing information among multiple fusion modes, determines the target detection range corresponding to the target fusion mode using spatiotemporal synchronization operation, acquires the information to be processed within the target detection range from the sensing information, matches the information to be processed using the matching rules corresponding to the target fusion mode to obtain the target matching result, performs compensation calculation on the information to be processed using a dust detection algorithm to obtain the millimeter-wave compensation result, and integrates the target matching result, the millimeter-wave compensation result, and the scene perception parameters of the sensing information using the integration rules corresponding to the target fusion mode to obtain the target fusion result, acquires the Kalman algorithm, processes the target fusion result using the Kalman algorithm to obtain obstacle-related information, supplements the fused sensing result with the detection results of millimeter-wave radar, enhances the vehicle's perception capability under dusty conditions, and expands the vehicle's perception range and improves perception accuracy by utilizing V2V collaborative perception results and roadside perception results, enabling the vehicle to better adapt to the mining environment.

[0124] It should be noted that other corresponding descriptions of the functional units involved in the sensor data fusion device provided in this application embodiment can be found in the following references. Figure 1 and Figures 2A to 2C The corresponding descriptions in [the document] will not be repeated here.

[0125] In an exemplary embodiment, see Figure 4 The invention also provides a computer device including a bus, a processor, a memory, and a communication interface. It may also include an input / output interface and a display device, wherein the various functional units can communicate with each other via the bus. The memory stores a computer program, and the processor executes the program stored in the memory to perform the sensor data fusion method described in the above embodiments.

[0126] A computer-readable storage medium having a computer program stored thereon, the steps of a sensor data fusion method implemented when the computer program is executed by a processor.

[0127] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented in hardware or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) and includes several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0128] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing this application.

[0129] Those skilled in the art will understand that the modules in the apparatus of the implementation scenario can be distributed within the apparatus of the implementation scenario as described, or they can be located in one or more apparatuses different from this implementation scenario, with corresponding changes. The modules of the above-described implementation scenario can be combined into one module, or they can be further divided into multiple sub-modules.

[0130] The serial numbers in this application are for descriptive purposes only and do not represent the superiority or inferiority of the implementation scenario.

[0131] The above disclosures are only a few specific implementation scenarios of this application. However, this application is not limited to these. Any variations that can be conceived by those skilled in the art should fall within the protection scope of this application.

Claims

1. A sensor data fusion method, characterized in that, include: When input sensing information is received, the target fusion mode corresponding to the sensing information is determined among multiple fusion modes. The target detection range corresponding to the target fusion mode is determined by spatiotemporal synchronization operation. The multiple fusion modes include Mode 1, Mode 2, and Mode 3. Mode 1 uses the fusion of traditional sensing output results and intelligent sensing output results of LiDAR. Mode 2 uses the fusion of traditional sensing output results, intelligent sensing output results, and millimeter-wave sensing output results of LiDAR. Mode 3 uses the fusion of traditional sensing output results, intelligent sensing output results, millimeter-wave sensing output results, communication cooperative sensing results, and roadside sensing results. The target fusion mode is Mode 3. The process involves acquiring unprocessed information within the target detection range from the perceived information, matching the unprocessed information using the matching rules corresponding to the target fusion mode, and obtaining a target matching result. The matching rules for the target fusion mode include: acquiring traditional LiDAR perception output results, intelligent LiDAR perception output results, millimeter-wave perception output results, and scene perception parameters from the perceived information; and acquiring unprocessed LiDAR traditional perception information, unprocessed LiDAR intelligent perception information, unprocessed millimeter-wave perception information, and unprocessed scene perception information within the target detection range from the traditional LiDAR perception output results, the intelligent LiDAR perception output results, the millimeter-wave perception output results, and the scene perception parameters, respectively. The scene perception parameters include communication collaborative perception results and roadside perception results. The unprocessed LiDAR traditional perception information, the unprocessed LiDAR intelligent perception information, the unprocessed millimeter-wave perception information, and the unprocessed scene perception information are used as the unprocessed information. The matching rules are then used to determine the match between the unprocessed LiDAR traditional perception information and the target detection result. The process involves: processing first content matching LiDAR intelligent sensing information; obtaining second content matching the first content from the LiDAR intelligent sensing information to be processed; and replacing the first content in the traditional LiDAR sensing information to be processed with the second content to obtain a first matching result. If there is a first unmatched content other than the second content in the LiDAR intelligent sensing information to be processed, the first unmatched content is filtered out. Then, using the matching rules, a third content matching the millimeter-wave sensing information to be processed is determined in the first matching result; a fourth content matching the third content is obtained from the millimeter-wave sensing information to be processed; and the fourth content is used to replace the third content in the first matching result to obtain a second matching result. Finally, using the matching rules, a fifth content matching the scene sensing information to be processed is determined in the second matching result; a sixth content matching the fifth content is obtained from the scene sensing information to be processed; and the sixth content is used to replace the fifth content in the second matching result to obtain the target matching result. The dust detection algorithm is used to perform compensation calculations on the information to be processed to obtain a millimeter-wave compensation result. The integration rules corresponding to the target fusion mode are then used to integrate the target matching result, the millimeter-wave compensation result, and the scene perception parameters of the sensing information to obtain a target fusion result. The integration rules corresponding to the target fusion mode include: determining the second unmatched content in the millimeter-wave sensing information to be processed, excluding the fourth content; obtaining the dust detection algorithm, using the dust detection algorithm to determine multiple location information of multiple dust particles, determining multiple associated sensing information related to the multiple location information in the second unmatched content, and using the multiple associated sensing information as the millimeter-wave compensation result; and obtaining the traditional sensing information of the LiDAR (LiDAR radar) excluding the traditional sensing information of the LiDAR to be processed from the traditional sensing output result of the LiDAR. The system obtains the following: traditional sensing information from a specified LiDAR; intelligent sensing information from the LiDAR intelligent sensing output that excludes the LiDAR intelligent sensing information to be processed; millimeter-wave sensing information from the millimeter-wave sensing output that excludes the millimeter-wave sensing information to be processed; and scene sensing information from the scene sensing parameters that excludes the scene sensing information to be processed. It then determines a third unmatched content in the traditional sensing information of the LiDAR to be processed, excluding the first content. Based on the traditional sensing information of the specified LiDAR, the intelligent sensing information of the specified LiDAR, the specified millimeter-wave sensing information, the specified scene sensing information, the third unmatched content, the target matching result, the millimeter-wave compensation result, and the scene sensing parameters, it generates the target fusion result. The Kalman algorithm is obtained, and the target fusion result is processed using the Kalman algorithm to obtain obstacle-related information.

2. The method according to claim 1, characterized in that, The step of determining the target detection range corresponding to the target fusion mode using spatiotemporal synchronization operations includes: The system identifies multiple target sensors corresponding to the target fusion mode, acquires sensor status information from the perception information, and detects the sensor status information. The multiple target sensors include lidar, millimeter-wave radar, and camera sensors. The sensor status information indicates the working status of the multiple target sensors. If the sensor status information indicates that the multiple target sensors are working normally, then the coordinate information of each target sensor is obtained; The sample vehicle corresponding to the perceived information is determined, the rear axle center position information of the sample vehicle is obtained, and the coordinate information of each target sensor is adjusted using the rear axle center position information so that the coordinate origin of each target sensor is unified to the rear axle center of the sample vehicle. Obtain the timestamp of the lidar, and sequentially determine the timestamps of the millimeter-wave radar and the camera sensor that are close to the timestamp of the lidar; By using the timestamps of the lidar, the millimeter-wave radar, and the camera sensor, the lidar, the millimeter-wave radar, and the camera sensor are unified under the same timestamp. The detection range of each target sensor is obtained to obtain multiple detection ranges, and the intersection of the multiple detection ranges is determined to obtain the target detection range.

3. The method according to claim 2, characterized in that, The step of unifying the lidar, millimeter-wave radar, and camera sensor to the same timestamp using the timestamps of the lidar, millimeter-wave radar, and camera sensor includes: Obtain the millimeter-wave detection result corresponding to the timestamp of the millimeter-wave radar and the camera detection result corresponding to the timestamp of the camera sensor; Calculate the time difference between the timestamp of the millimeter-wave radar, the timestamp of the camera sensor, and the timestamp of the lidar to obtain the millimeter-wave time difference and the camera time difference. Motion compensation is performed on the millimeter-wave detection results using the millimeter-wave time difference and on the camera detection results using the camera time difference, so that the lidar, the millimeter-wave radar, and the camera sensor are at the same timestamp.

4. The method according to claim 1, characterized in that, The process of using the Kalman algorithm to process the target fusion result to obtain obstacle-related information includes: Obtain the previous fusion result of the target fusion result, determine the time information corresponding to the previous fusion result, and obtain the time information corresponding to the target fusion result. Calculate the target time difference using the time information corresponding to the previous fusion result and the time information corresponding to the target fusion result. The time difference algorithm is obtained, and the time difference algorithm is used to calculate the target time difference to obtain the velocity information corresponding to the target fusion result; The sample vehicle corresponding to the perceived information is obtained, and the preset motion model corresponding to the sample vehicle is obtained. The preset motion model is a preset virtual model of uniform motion. The speed information is obtained, and the preset motion model is used to predict the speed information and the target time difference to obtain the predicted position information. The velocity information is obtained from the target fusion result, multiple target sensors corresponding to the target fusion mode are obtained, and multiple sensor detection position information corresponding to the multiple target sensors is obtained. The Kalman algorithm is used to update the velocity information, the target fusion result, the multiple sensor detection position information, and the predicted position information to obtain obstacle-related information.

5. A sensor data fusion device, characterized in that, include: A determination module is used to determine the target fusion mode corresponding to the received sensing information among multiple fusion modes when the received sensing information is received. It then uses spatiotemporal synchronization to determine the target detection range corresponding to the target fusion mode. The multiple fusion modes include Mode 1, Mode 2, and Mode 3. Mode 1 uses the fusion of traditional sensing output results and intelligent sensing output results from a fusion LiDAR. Mode 2 uses the fusion of traditional sensing output results, intelligent sensing output results, and millimeter-wave sensing output results from a fusion LiDAR. Mode 3 uses the fusion of traditional sensing output results, intelligent sensing output results, millimeter-wave sensing output results, communication-coordinated sensing results, and roadside sensing results from a fusion LiDAR. The target fusion mode is Mode 3. The matching module is used to acquire unprocessed information within the target detection range from the perceived information, and to match the unprocessed information using the matching rules corresponding to the target fusion mode to obtain a target matching result. The matching rules corresponding to the target fusion mode include: acquiring the traditional LiDAR perception output result, the intelligent LiDAR perception output result, the millimeter-wave perception output result, and scene perception parameters from the perceived information; and acquiring unprocessed LiDAR traditional perception information, unprocessed LiDAR intelligent perception information, unprocessed millimeter-wave perception information, and unprocessed scene perception information within the target detection range from the traditional LiDAR perception output result, the intelligent LiDAR perception output result, the millimeter-wave perception output result, and the scene perception parameters, respectively. The scene perception parameters include communication cooperative perception results and roadside perception results; using the unprocessed LiDAR traditional perception information, the unprocessed LiDAR intelligent perception information, the unprocessed millimeter-wave perception information, and the unprocessed scene perception information as the unprocessed information; and determining the unprocessed LiDAR traditional perception information using the matching rules. The process involves: firstly matching the intelligent sensing information of the LiDAR to be processed with a first content; secondly obtaining the second content matching the first content from the intelligent sensing information of the LiDAR to be processed; and replacing the first content in the traditional sensing information of the LiDAR to be processed with the second content to obtain a first matching result. If there is a first unmatched content other than the second content in the intelligent sensing information of the LiDAR to be processed, the first unmatched content is filtered out. Then, using the matching rules, a third content matching the millimeter-wave sensing information to be processed is determined in the first matching result; a fourth content matching the third content is obtained from the millimeter-wave sensing information to be processed; and the fourth content is used to replace the third content in the first matching result to obtain a second matching result. Finally, using the matching rules, a fifth content matching the scene sensing information to be processed is determined in the second matching result; a sixth content matching the fifth content is obtained from the scene sensing information to be processed; and the sixth content is used to replace the fifth content in the second matching result to obtain the target matching result. The integration module is used to perform compensation calculations on the information to be processed using a dust detection algorithm to obtain a millimeter-wave compensation result, and to integrate the target matching result, the millimeter-wave compensation result, and the scene perception parameters of the perception information using the integration rules corresponding to the target fusion mode to obtain a target fusion result. The integration rules corresponding to the target fusion mode include: determining the second unmatched content in the millimeter-wave perception information to be processed, excluding the fourth content; obtaining the dust detection algorithm, using the dust detection algorithm to determine multiple location information of multiple dust particles, determining multiple associated perception information related to the multiple location information in the second unmatched content, and using the multiple associated perception information as the millimeter-wave compensation result; and obtaining the LiDAR information excluding the LiDAR to be processed from the traditional perception output results of the LiDAR. The process involves: obtaining traditional sensing information from a designated LiDAR; acquiring intelligent sensing information from the LiDAR intelligent sensing output that excludes the LiDAR intelligent sensing information to be processed; acquiring millimeter-wave sensing information from the millimeter-wave sensing output that excludes the millimeter-wave sensing information to be processed; and acquiring scene sensing information from the scene sensing parameters that excludes the scene sensing information to be processed. The process also involves determining a third unmatched content in the traditional sensing information of the LiDAR to be processed, excluding the first content; and generating the target fusion result based on the designated LiDAR traditional sensing information, the designated LiDAR intelligent sensing information, the designated millimeter-wave sensing information, the designated scene sensing information, the third unmatched content, the target matching result, the millimeter-wave compensation result, and the scene sensing parameters. The processing module is used to acquire the Kalman algorithm and process the target fusion result using the Kalman algorithm to obtain obstacle-related information.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

7. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images