Multi-sensor data fusion method, device, equipment and medium

Abstract

The invention relates to a multi-sensor data fusion method and device, equipment and a medium, and the method comprises the steps: obtaining data and GPS information of a plurality of sensors such as a binocular camera and a radar, taking the data and GPS information as input data, and carrying out the preprocessing, including one or more of image color space conversion, filtering and denoising, and data density reduction; further processing the preprocessed data, wherein the processing comprises one or more of target detection and tracking, coordinate system conversion, target UID distribution and aerial view angle projection; a precision time protocol (PTP) is adopted to synchronize processed data of a plurality of sensors, and the time is kept consistent; a central point of a road intersection is selected, GPS information is recorded, a fusion coordinate system is established, sensor and target position information is calculated according to the GPS information of a sensor, the information is mapped to the fusion coordinate system, and target, association speed, distance and other information and UID are screened and merged according to an overlap ratio threshold. According to the invention, modeling display of the whole intersection is realized, and a reliable basis is provided for traffic management decisions.

Description

Technical Field

[0001] The present invention belongs to the field of sensor data acquisition and multi-sensor data fusion, and specifically relates to a multi-sensor data fusion method, device, equipment and medium. Background Art

[0002] In the traffic management of modern road intersections, real-time and comprehensive information is crucial to improving traffic efficiency and ensuring driving safety. Traffic scenes are complex and changeable, involving multiple dynamic targets and emergencies, and the traffic target status information that a single sensor can provide is very limited. Relying on a single type of sensor alone, it is difficult to fully and accurately perceive and track the status of all traffic targets, which in turn affects the traffic management system to make timely and effective decisions. The complexity of traffic management requires comprehensive analysis and processing of the multi-dimensional status of different targets, such as the speed, direction, position of vehicles, and the dynamic changes of pedestrians. In order to improve the accuracy and reliability of decision-making, it is necessary to integrate multiple types of sensor data to achieve a more comprehensive and accurate perception effect. In practical applications, if multiple sensors, such as microwave radars, binocular cameras, etc., can be deployed at key locations such as road intersections, these devices will provide traffic data of different dimensions, thereby helping to identify the accurate location and dynamic changes of traffic subjects such as pedestrians and vehicles. Through advanced data fusion technology, heterogeneous data from different sensors can be effectively integrated, which can not only significantly improve the accuracy and real-time performance of data, but also generate a more intuitive and easy-to-understand display interface, helping traffic management personnel to make more reasonable and timely judgments and decisions on traffic conditions. This fused multi-dimensional traffic information can provide reliable support for traffic signal control, accident prevention, congestion management and other aspects. Summary of the invention

[0003] One or more embodiments of the present specification provide a multi-sensor data fusion method, apparatus, device, and medium for solving the technical problems raised by the background technology.

[0004] One or more embodiments of this specification adopt the following technical solutions: One or more embodiments of this specification provide a multi-sensor data fusion method, including: S1. Acquire sensor data and GPS information of multiple sensors, use the sensor data as input data, and preprocess the input data to obtain preprocessed data, wherein the preprocessing at least includes: one or more of image color space conversion, image filtering and denoising, and data decryption, the multiple sensors at least include a visual sensor and a distance sensor, the visual sensor is at least a binocular camera, and the distance sensor is at least a radar; S2. Further data processing is performed on the pre-processed data of different sensors to obtain processed data, wherein the data processing includes at least one or more of target detection and tracking, coordinate system conversion, target UID allocation, and bird's-eye view projection; S3, synchronizing the processed data of the multiple sensors using a precision time protocol (PTP) so that the time of the processed data is consistent; S4. Select a center point of a road intersection and record the GPS information of the center point, establish a fusion coordinate system based on the GPS information, calculate the sensor position information and the target position information according to the GPS information of the sensor, map the sensor position information and the target position information to the fusion coordinate system using a conversion formula, filter the target position information according to a preset overlap threshold to obtain overlapping targets, merge the overlapping targets, and associate the relevant information of the target collected by the sensor, including but not limited to speed and distance, with the UID of the target.

[0005] One or more embodiments of this specification provide a multi-sensor data fusion device, including: A data acquisition module, used to obtain sensor data and GPS information of multiple sensors, take the sensor data as input data, and preprocess the input data to obtain preprocessed data, wherein the preprocessing at least includes: one or more of image color space conversion, image filtering and denoising, and data de-encryption; A data processing module, for further processing the pre-processed data of different sensors to obtain processed data, wherein the data processing includes at least one or more of target detection and tracking, coordinate system conversion, target UID allocation, and bird's-eye view projection; A clock synchronization module, used for synchronizing the processed data of the multiple sensors using a precision time protocol (PTP) so that the time of the processed data is consistent; A data fusion module is used to select a center point of a road intersection and record the GPS information of the center point, establish a fusion coordinate system based on the GPS information, calculate the sensor position information and the target position information according to the GPS information of the sensor, map the sensor position information and the target position information to the fusion coordinate system using a conversion formula, filter the target position information according to a preset overlap threshold to obtain overlapping targets, merge the overlapping targets, and associate the relevant information of the target collected by the sensor, including but not limited to speed and distance, with the UID of the target.

[0006] One or more embodiments of this specification provide a multi-sensor data fusion device, including: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory has instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to: Acquire sensor data and GPS information of multiple sensors, use the sensor data as input data, and preprocess the input data to obtain preprocessed data, wherein the preprocessing at least includes: one or more of image color space conversion, image filtering and denoising, and data de-encryption, and the multiple sensors at least include a binocular camera and a radar; Further data processing is performed on the preprocessed data of different sensors to obtain processed data, wherein the data processing includes at least one or more of: target detection and tracking, coordinate system conversion, target UID allocation, and bird's-eye view projection; Synchronizing the processed data of the multiple sensors using a precision time protocol (PTP) so that the time of the processed data is consistent; Select a center point of a road intersection and record the GPS information of the center point, establish a fusion coordinate system based on the GPS information, calculate the sensor position information and the target position information according to the GPS information of the sensor, use a conversion formula to map the sensor position information and the target position information to the fusion coordinate system, filter the target position information according to a preset overlap threshold to obtain overlapping targets to be measured, merge the overlapping targets to be measured, and associate the relevant information of the target to be measured collected by the sensor, including but not limited to speed and distance, with the UID of the target to be measured.

[0007] One or more embodiments of this specification provide a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured to: Acquire sensor data and GPS information of multiple sensors, use the sensor data as input data, and preprocess the input data to obtain preprocessed data, wherein the preprocessing at least includes: one or more of image color space conversion, image filtering and denoising, and data de-encryption, and the multiple sensors at least include a binocular camera and a radar; Further data processing is performed on the preprocessed data of different sensors to obtain processed data, wherein the data processing includes at least one or more of: target detection and tracking, coordinate system conversion, target UID allocation, and bird's-eye view projection; Synchronizing the processed data of the multiple sensors using a precision time protocol (PTP) so that the time of the processed data is consistent; Select a center point of a road intersection and record the GPS information of the center point, establish a fusion coordinate system based on the GPS information, calculate the sensor position information and the target position information according to the GPS information of the sensor, use a conversion formula to map the sensor position information and the target position information to the fusion coordinate system, filter the target position information according to a preset overlap threshold to obtain overlapping targets to be measured, merge the overlapping targets to be measured, and associate the relevant information of the target to be measured collected by the sensor, including but not limited to speed and distance, with the UID of the target to be measured.

[0008] At least one of the above technical solutions adopted in the embodiments of this specification can achieve the following beneficial effects: The multi-sensor data fusion method provided by the present invention ensures the consistency of multi-sensor data through clock synchronization technology, solves the problem of data inconsistency caused by the asynchronous acquisition time of different sensors, and provides a solid foundation for data fusion. The position information and posture data of each sensor are obtained through a high-precision GPS device, and a unified fusion coordinate system is established. The detection data of each sensor is converted to the coordinate system for unified processing, which effectively solves the problem of data alignment between different sensors. This fusion coordinate system maps the target position information detected by multiple sensors to a unified spatial coordinate through a coordinate conversion formula, and determines whether it is the same target by calculating the target overlap, thereby improving the accuracy of target recognition. Finally, based on the target position information under the fusion coordinate system, the system can realize the holographic display of road intersections, providing traffic management personnel with a comprehensive and intuitive dynamic display of traffic scenes, greatly improving the efficiency of traffic management and decision-making support capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 It is a flowchart of a multi-sensor data fusion method provided by an embodiment of the present invention; Figure 2 is a logic flow chart of a multi-sensor data fusion method provided by an embodiment of the present invention; Figure 3 It is a schematic diagram showing the overlap of target detection results when projected into the local coordinate system; Figure 4 It is a structural schematic diagram of a multi-sensor data fusion device provided by an embodiment of the present invention; Figure 5 A schematic diagram of the structure of a multi-sensor data fusion device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0010] The following will be combined with the accompanying drawings of the specification to provide a detailed and complete description of the technical solutions of the embodiments of the present invention. It should be noted that the described embodiments are only examples of some of the present invention and do not represent all possible implementation methods of the present invention. Based on the embodiments of the present invention, any other implementation methods obtained by ordinary technicians in this field without creative work shall fall within the scope of protection of the present invention.

[0011] Figure 1 A schematic flow chart of a multi-sensor data fusion method provided in an embodiment of the present invention, which can be executed by a multi-sensor data fusion system. Figure 2 A logic flow chart of a multi-sensor data fusion method provided by an embodiment of the present invention. Certain input parameters or intermediate results in the process allow manual intervention and adjustment to help improve accuracy.

[0012] The method steps of the embodiment of this specification are as follows: S1. Acquire sensor data and GPS information of multiple sensors, use the sensor data as input data, and preprocess the input data to obtain preprocessed data, wherein the preprocessing at least includes: one or more of image color space conversion, image filtering and denoising, and data decryption, the multiple sensors at least include a visual sensor and a distance sensor, the visual sensor is at least a binocular camera, and the distance sensor is at least a radar; S2. Further data processing is performed on the pre-processed data of different sensors to obtain processed data, wherein the data processing includes at least one or more of target detection and tracking, coordinate system conversion, target UID allocation, and bird's-eye view projection; S3, synchronizing the processed data of the multiple sensors using a precision time protocol (PTP) so that the time of the processed data is consistent; S4. Select a center point of a road intersection and record the GPS information of the center point, establish a fusion coordinate system based on the GPS information, calculate the sensor position information and the target position information according to the GPS information of the sensor, map the sensor position information and the target position information to the fusion coordinate system using a conversion formula, filter the target position information according to a preset overlap threshold to obtain overlapping targets, merge the overlapping targets, and associate the relevant information of the target collected by the sensor, including but not limited to speed and distance, with the UID of the target.

[0013] The preprocessing in step S1 specifically includes the following steps: For the sensor data collected by the binocular camera, the sensor data at least includes visible light data: S101, connecting the binocular camera and the airborne edge computing platform through the internal Ethernet of the drone, and the edge computing platform obtains the real-time video stream information collected by the binocular camera through the RTSP video stream address; S102, decoding the original video into a single-frame picture in a unified RGB format; S103, preprocessing the single frame image, including color space conversion and image filtering and denoising, to improve image quality and provide optimized image information for subsequent road vehicle violation analysis.

[0014] For the sensor data collected by the radar, the sensor data at least includes point cloud data: S104, using height threshold and RANSAC plane fitting to remove ground points and highlight target objects above the ground; S105, using a statistical filtering method to remove outliers and noise points and retain valid data; S106. Align multiple frames of point cloud data using a registration algorithm (ICP, NDT) to generate a continuous and seamless point cloud map; S107, dividing the point cloud into voxel grids, and replacing all points in the grid with the centroid of each grid to reduce the amount of data and improve processing efficiency; S108. Randomly select part of the point cloud data to reduce data density and reduce calculation burden.

[0015] Step S2 specifically includes the following steps: For the sensor data collected by the binocular camera, the sensor data at least includes visible light data: S201, using parallax to calculate the depth information of the target to be detected from the binocular camera in the camera coordinate system, and mapping the 2D target detection result to the 3D space; S202, using Yolov5 algorithm to train a high-precision target detection model, taking the center point of the binocular camera's image as the origin of the pixel coordinate system, outputting the coordinate information of each target frame, i.e., the coordinate values ​​of the upper left corner (px1, py1) and the lower right corner (px2, py2), and according to the formula Calculate the coordinates of the center point of the target frame; S203, using the DeepSort tracking algorithm to track multiple targets to be measured, and assigning a UID to each target to be measured until the target to be measured leaves the field of view; S204: Calculate the external parameter matrix according to the position and posture state of the binocular camera , which contains the rotation matrix R and the translation vector T. Based on the external parameter matrix, the 3D coordinates (X, Y, Z) in the camera coordinate system are converted to the coordinates in the world coordinate system ( , , ), the conversion formula is: ; S205 : Projecting the visible light data to a bird's-eye view (BEV).

[0016] For the sensor data collected by the radar, the sensor data at least includes point cloud data: S211, based on the GPS information of the radar, using the UTM projection method, obtain the coordinate point of the radar center point in the radar reference system ( , , ), according to the position and posture state of the binocular camera, the external parameter matrix is ​​calculated , which contains the rotation matrix R and the translation vector T. Based on the external parameter matrix, the coordinates in the radar coordinate system ( , , ) is converted to the coordinates in the world coordinate system ( , , ), the expression is:

[0017] S212, projecting the point cloud data to a bird's-eye view (BEV); S213, discretizing the projected point cloud data onto a grid map of fixed resolution, and and , the continuous point cloud coordinates in the world coordinate system ( , ) is converted to grid coordinates i and j; The conversion formula is as follows:

[0018]

[0019] S214. At each grid position, the x, y coordinate values ​​and reflection intensity of the point cloud are used as three channels respectively to generate a BEV feature map, and a CNN network is used to perform target detection on the BEV feature map.

[0020] Step S201 specifically includes the following steps: S2011, aligning the camera calibration and stereo-corrected binocular camera images on the same horizontal line to eliminate parallax errors between images; S2012, calculating the parallax of the left and right images, and calculating the depth information of the target point based on the parallax and the camera baseline distance; The calculation formula is as follows:

[0021] Where f is the focal length of the camera, B is the baseline distance between cameras, Pixel The disparity value of the pixel point (x, y) is the depth value corresponding to the pixel point (x, y).

[0022] S2013: Combine the depth information with the image coordinates and convert them into three-dimensional coordinates.

[0023] The conversion formula is as follows:

[0024] in( , ) are the camera principal point coordinates.

[0025] Step S3 specifically includes the following steps: S301, select an edge computing node with a GPS time source as the PTP master clock; S302, multiple sensors installed at the same road intersection are configured as PTP slave clocks, and the multiple sensors receive time information from the master clock and adjust their own clocks to keep synchronization with the master clock; S303. Use a dedicated network at the same road intersection to minimize network delay and jitter and improve synchronization accuracy.

[0026] Step S4 specifically includes the following steps: S401, select a center point of a road intersection, record the GPS information of the center point, and use the UTM projection method to obtain the position of the center point in the world coordinate system based on the GPS information of the center point ( , , ), and establish a local BEV coordinate system with the center point as the origin, the north direction as the positive direction of the Y axis, and the east direction as the positive direction of the X axis; S402, through the conversion matrix , the binocular camera target detection results and the rasterization results of the radar point cloud in the world coordinate system are projected into the local coordinate system of the intersection. When the binocular camera target detection results and the rasterization results of the radar point cloud projected into the local coordinate system have target overlap, further fusion is performed.

[0027] The target detection results are projected into the local coordinate system and there is overlap. Figure 3 As shown, N is a reference direction, which is used to represent the absolute orientation of the scene coordinates. The target coincidence fusion method specifically includes the following steps: S501, performing reasoning on the visible light image data and the point cloud data respectively to obtain categories of the visible light reasoning result and the point cloud reasoning result, and determining whether the category of the visible light reasoning result is the same as the category of the point cloud reasoning result; S502: When the category of the visible light reasoning result is the same as the category of the point cloud reasoning result, calculate the overlap rate between the visible light target frame and the point cloud grid, and determine whether the overlap rate accounts for more than 50% of the visible light target frame; S503: When the overlap rate is greater than or equal to 50%, the point cloud target frame is hidden, and the data measured by the radar is bound to the UID of the visible light target frame.

[0028] After step S4, the method further includes a display method, which specifically includes the following steps: S601. Based on the merging result of the data fusion module, create a holographic display effect in the local coordinate system of the road intersection; S602, converting the local coordinate system and the world coordinate system through the conversion matrix, and using the GPS information of the road edge to generate a three-dimensional model of the intersection; S603, displaying traffic participating targets according to the data fusion result, the targets including one or more of cars, pedestrians and bicycles, marking the traffic participating targets with UIDs, and displaying their speeds and distances from intersections; S604: For the point cloud detection result, if it does not match the visible light detection result, the corresponding target information is displayed in a semi-transparent form.

[0029] Figure 4 A multi-sensor data fusion device is provided in an embodiment of the present invention. The device comprises: a data acquisition module, a data processing module, a clock synchronization module and a data fusion module.

[0030] A data acquisition module, used to obtain sensor data and GPS information of multiple sensors, take the sensor data as input data, and preprocess the input data to obtain preprocessed data, wherein the preprocessing at least includes: one or more of image color space conversion, image filtering and denoising, and data de-encryption; A data processing module, for further processing the pre-processed data of different sensors to obtain processed data, wherein the data processing includes at least one or more of target detection and tracking, coordinate system conversion, target UID allocation, and bird's-eye view projection; A clock synchronization module, used for synchronizing the processed data of the multiple sensors using a precision time protocol (PTP) so that the time of the processed data is consistent; A data fusion module is used to select a center point of a road intersection and record the GPS information of the center point, establish a fusion coordinate system based on the GPS information, calculate the sensor position information and the target position information according to the GPS information of the sensor, map the sensor position information and the target position information to the fusion coordinate system using a conversion formula, filter the target position information according to a preset overlap threshold to obtain overlapping targets, merge the overlapping targets, and associate the relevant information of the target collected by the sensor, including but not limited to speed and distance, with the UID of the target.

[0031] Figure 5 A schematic diagram of the structure of a multi-sensor data fusion device provided in an embodiment of the present invention includes: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory has instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to: Acquire sensor data and GPS information of multiple sensors, use the sensor data as input data, and preprocess the input data to obtain preprocessed data, wherein the preprocessing at least includes: one or more of image color space conversion, image filtering and denoising, and data de-encryption, and the multiple sensors at least include a binocular camera and a radar; Further data processing is performed on the preprocessed data of different sensors to obtain processed data, wherein the data processing includes at least one or more of: target detection and tracking, coordinate system conversion, target UID allocation, and bird's-eye view projection; Synchronizing the processed data of the multiple sensors using a precision time protocol (PTP) so that the time of the processed data is consistent; Select a center point of a road intersection and record the GPS information of the center point, establish a fusion coordinate system based on the GPS information, calculate the sensor position information and the target position information according to the GPS information of the sensor, use a conversion formula to map the sensor position information and the target position information to the fusion coordinate system, filter the target position information according to a preset overlap threshold to obtain overlapping targets to be measured, merge the overlapping targets to be measured, and associate the relevant information of the target to be measured collected by the sensor, including but not limited to speed and distance, with the UID of the target to be measured.

[0032] One or more embodiments of this specification provide a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured to: Acquire sensor data and GPS information of multiple sensors, use the sensor data as input data, and preprocess the input data to obtain preprocessed data, wherein the preprocessing at least includes: one or more of image color space conversion, image filtering and denoising, and data de-encryption, and the multiple sensors at least include a binocular camera and a radar; Further data processing is performed on the preprocessed data of different sensors to obtain processed data, wherein the data processing includes at least one or more of: target detection and tracking, coordinate system conversion, target UID allocation, and bird's-eye view projection; Synchronizing the processed data of the multiple sensors using a precision time protocol (PTP) so that the time of the processed data is consistent; Select a center point of a road intersection and record the GPS information of the center point, establish a fusion coordinate system based on the GPS information, calculate the sensor position information and the target position information according to the GPS information of the sensor, use a conversion formula to map the sensor position information and the target position information to the fusion coordinate system, filter the target position information according to a preset overlap threshold to obtain overlapping targets to be measured, merge the overlapping targets to be measured, and associate the relevant information of the target to be measured collected by the sensor, including but not limited to speed and distance, with the UID of the target to be measured.

[0033] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and non-volatile computer storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0034] The above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0035] The above description is only one or more embodiments of this specification and is not intended to limit this specification. For those skilled in the art, one or more embodiments of this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of this specification shall be included in the scope of the claims of this specification.

Claims

1. A multi-sensor data fusion method, characterized in that: The method comprises the following steps: S1. Acquire sensor data and GPS information of multiple sensors, use the sensor data as input data, and preprocess the input data to obtain preprocessed data, wherein the preprocessing at least includes: one or more of image color space conversion, image filtering and denoising, and data decryption, the multiple sensors at least include a visual sensor and a distance sensor, the visual sensor is at least a binocular camera, and the distance sensor is at least a radar; S2. Further data processing is performed on the pre-processed data of different sensors to obtain processed data, wherein the data processing includes at least one or more of: target detection and tracking, coordinate system conversion, target unique identifier UID allocation, and bird's-eye view projection; S3, synchronizing the processed data of the multiple sensors using the precision time protocol PTP to keep the time of the processed data consistent; S4. Select a center point of a road intersection and record the GPS information of the center point, establish a fusion coordinate system based on the GPS information, calculate the sensor position information and the target position information according to the GPS information of the sensor, map the sensor position information and the target position information to the fusion coordinate system using a conversion formula, filter the target position information according to a preset overlap threshold to obtain overlapping targets, merge the overlapping targets, and associate the relevant information of the target collected by the sensor, including but not limited to speed and distance, with the unique identifier UID of the target.

2. A multi-sensor data fusion method according to claim 1, characterized in that: The step S2 specifically includes the following steps: For the sensor data collected by the binocular camera, the sensor data at least includes visible light data: S201, using parallax to calculate the depth information of the target to be detected from the binocular camera in the camera coordinate system, and mapping the 2D target detection result to the 3D space; S202, using Yolov5 algorithm to train a high-precision target detection model, taking the center point of the binocular camera's image as the origin of the pixel coordinate system, outputting the coordinate information of each target frame, i.e., the coordinate values ​​of the upper left corner (px1, py1) and the lower right corner (px2, py2), and according to the formula Calculate the coordinates of the center point of the target frame; S203, using the DeepSort tracking algorithm to track multiple targets to be measured, and assigning a unique identifier UID to each target to be measured until the target to be measured leaves the field of view; S204: Calculate the external parameter matrix according to the position and posture state of the binocular camera , which contains the rotation matrix R and the translation vector T. Based on the external parameter matrix, the 3D coordinates (X, Y, Z) in the camera coordinate system are converted to the coordinates in the world coordinate system ( , , ), the conversion formula is: ； S205: Project the visible light data to a bird's-eye view BEV.

3. A multi-sensor data fusion method according to claim 1, characterized in that: The step S2 specifically includes the following steps: For the sensor data collected by the radar, the sensor data at least includes point cloud data: S211, based on the GPS information of the radar, using the UTM projection method, obtain the coordinate point of the radar center point in the radar reference system ( , , ), according to the position and posture state of the binocular camera, the external parameter matrix is ​​calculated , which contains the rotation matrix R and the translation vector T. Based on the external parameter matrix, the coordinates in the radar coordinate system ( , , ) is converted to the coordinates in the world coordinate system ( , , ), the expression is: S212, projecting the point cloud data to a bird's-eye view BEV; S213, discretizing the projected point cloud data onto a grid map of fixed resolution, and and , the continuous point cloud coordinates in the world coordinate system ( , ) is converted to grid coordinates i and j; The conversion formula is as follows: S214. At each grid position, the x, y coordinate values ​​and reflection intensity of the point cloud are used as three channels respectively to generate a BEV feature map, and a CNN network is used to perform target detection on the BEV feature map.

4. The multi-sensor data fusion method according to claim 1, characterized in that: The step S3 specifically comprises the following steps: S301, select an edge computing node with a GPS time source as the PTP master clock; S302, multiple sensors installed at the same road intersection are configured as PTP slave clocks, and the multiple sensors receive time information from the master clock and adjust their own clocks to keep synchronization with the master clock; S303. Use a dedicated network at the same road intersection to minimize network delay and jitter and improve synchronization accuracy.

5. The multi-sensor data fusion method according to claim 1, characterized in that: The step S4 specifically comprises the following steps: S401, select a center point of a road intersection, record the GPS information of the center point, and use the UTM projection method to obtain the position of the center point in the world coordinate system based on the GPS information of the center point ( , , ), and establish a local BEV coordinate system with the center point as the origin, the north direction as the positive direction of the Y axis, and the east direction as the positive direction of the X axis; S402, through the conversion matrix , the binocular camera target detection results and the rasterization results of the radar point cloud in the world coordinate system are projected into the local coordinate system of the intersection. When the binocular camera target detection results and the rasterization results of the radar point cloud projected into the local coordinate system have target overlap, further fusion is performed.

6. A multi-sensor data fusion method according to claim 5, characterized in that: When there is target overlap, the method for further fusion specifically includes the following steps: S501, performing reasoning on the visible light image data and the point cloud data respectively to obtain categories of the visible light reasoning result and the point cloud reasoning result, and determining whether the category of the visible light reasoning result is the same as the category of the point cloud reasoning result; S502: When the category of the visible light reasoning result is the same as the category of the point cloud reasoning result, calculate the overlap rate between the visible light target frame and the point cloud grid, and determine whether the overlap rate accounts for more than 50% of the visible light target frame; S503: When the overlap rate is greater than or equal to 50%, the point cloud target frame is hidden, and the data measured by the radar is bound to the unique identifier UID of the visible light target frame.

7. The multi-sensor data fusion method according to claim 1, characterized in that: After step S4, the method further includes a display method, specifically including: S601. Based on the merging result of the data fusion module, create a holographic display effect in the local coordinate system of the road intersection; S602, converting the local coordinate system and the world coordinate system through the conversion matrix, and using the GPS information of the road edge to generate a three-dimensional model of the intersection; S603: Display traffic participating targets according to the data fusion result, where the targets include one or more of cars, pedestrians and bicycles, mark the traffic participating targets with unique identifiers UID, and display their speeds and distances from intersections; S604: For the point cloud detection result, if it does not match the visible light detection result, the corresponding target information is displayed in a semi-transparent form.

8. A multi-sensor data fusion device, characterized in that: The device comprises: A data acquisition module, used to obtain sensor data and GPS information of multiple sensors, take the sensor data as input data, and preprocess the input data to obtain preprocessed data, wherein the preprocessing at least includes: one or more of image color space conversion, image filtering and denoising, and data de-encryption; A data processing module, for further processing the pre-processed data of different sensors to obtain processed data, wherein the data processing includes at least one or more of: target detection and tracking, coordinate system conversion, target unique identifier UID allocation, and bird's-eye view projection; A clock synchronization module, used for synchronizing the processed data of the multiple sensors using a precision time protocol (PTP) so as to keep the time of the processed data consistent; A data fusion module is used to select a center point of a road intersection and record the GPS information of the center point, establish a fusion coordinate system based on the GPS information, calculate the sensor position information and the target position information according to the GPS information of the sensor, map the sensor position information and the target position information to the fusion coordinate system using a conversion formula, filter the target position information according to a preset overlap threshold to obtain overlapping targets, merge the overlapping targets, and associate the relevant information of the target collected by the sensor, including but not limited to speed and distance, with the unique identifier UID of the target.

9. The multi-sensor data fusion device according to claim 8, characterized in that: The data processing module is used to execute the pre-processed data for different sensors, and further process the data to obtain processed data. The data processing includes at least one or more of: target detection and tracking, coordinate system conversion, target unique identifier UID allocation, and bird's-eye view projection, specifically including: For sensor data collected by a binocular camera, the sensor data at least includes visible light data; With respect to the sensor data collected by the radar, the sensor data at least includes point cloud data.

10. A multi-sensor data fusion device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory has instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to: Acquire sensor data and GPS information of multiple sensors, use the sensor data as input data, and preprocess the input data to obtain preprocessed data, wherein the preprocessing at least includes: one or more of image color space conversion, image filtering and denoising, and data de-encryption, and the multiple sensors at least include a binocular camera and a radar; Further data processing is performed on the pre-processed data of different sensors to obtain processed data, wherein the data processing includes at least one or more of: target detection and tracking, coordinate system conversion, target unique identifier UID allocation, and bird's-eye view projection; Synchronizing the processed data of the multiple sensors using the precision time protocol (PTP) so that the time of the processed data is consistent; A center point of a road intersection is selected, and the GPS information of the center point is recorded, a fusion coordinate system is established based on the GPS information, the sensor position information and the position information of the target to be measured are calculated according to the GPS information of the sensor, the sensor position information and the position information of the target to be measured are mapped to the fusion coordinate system using a conversion formula, the position information of the target to be measured is screened according to a preset overlap threshold value to obtain overlapping targets to be measured, the overlapping targets to be measured are merged, and the relevant information of the target to be measured collected by the sensor, including but not limited to speed and distance, is associated with the unique identifier UID of the target to be measured.

11. A non-volatile computer storage medium, characterized in that: A computer executable instruction is stored, wherein the computer executable instruction is configured to: Acquire sensor data and GPS information of multiple sensors, use the sensor data as input data, and preprocess the input data to obtain preprocessed data, wherein the preprocessing at least includes: one or more of image color space conversion, image filtering and denoising, and data de-encryption, and the multiple sensors at least include a binocular camera and a radar; Further data processing is performed on the pre-processed data of different sensors to obtain processed data, wherein the data processing includes at least one or more of: target detection and tracking, coordinate system conversion, target unique identifier UID allocation, and bird's-eye view projection; Synchronizing the processed data of the multiple sensors using the precision time protocol (PTP) so that the time of the processed data is consistent; A center point of a road intersection is selected, and the GPS information of the center point is recorded, a fusion coordinate system is established based on the GPS information, the sensor position information and the position information of the target to be measured are calculated according to the GPS information of the sensor, the sensor position information and the position information of the target to be measured are mapped to the fusion coordinate system using a conversion formula, the position information of the target to be measured is screened according to a preset overlap threshold value to obtain overlapping targets to be measured, the overlapping targets to be measured are merged, and the relevant information of the target to be measured collected by the sensor, including but not limited to speed and distance, is associated with the unique identifier UID of the target to be measured.

Images