A data fusion method and apparatus based on multiple sensors

Abstract

This application discloses a data fusion method and apparatus based on multiple sensors. The method includes: acquiring current image data; determining current weather conditions and current road conditions based on the current image data; identifying a target lidar among multiple lidars in a roadside sensing unit based on the current weather conditions; the multiple lidars emitting lasers of different wavelengths respectively; the roadside sensing unit also includes a millimeter-wave radar and a camera for acquiring current image data; acquiring first radar data of the target lidar and second radar data of the millimeter-wave radar; determining weight coefficients for the first radar data, second radar data, and current image data based on the current weather conditions and current road conditions; and fusing the first radar data, second radar data, and current image data based on the weight coefficients. By dynamically switching the laser wavelength and adjusting the weights, the method solves the problem of performance degradation of a single sensor under adverse weather conditions.

Description

Technical Field

[0001] This application relates to the field of intelligent transportation sensing technology, and in particular to a data fusion method and apparatus based on multiple sensors. Background Technology

[0002] With the rapid development of autonomous driving technology, roadside perception devices, as a key component of traffic environment perception, play a decisive role in the safety and intelligence level of road driving. Currently, mainstream roadside perception sensors mainly include LiDAR, millimeter-wave radar, ultrasonic radar, and cameras.

[0003] However, all types of sensors have inherent limitations: while lidar boasts high precision and high resolution, it is highly susceptible to changes in ambient light and weather conditions such as rain, snow, and fog, resulting in a significant performance drop in adverse weather conditions. Millimeter-wave radar offers all-weather operation, but its lower spatial resolution makes it difficult to accurately identify target details. Ultrasonic radar has a relatively short detection range and is generally suitable for short-range detection. Cameras are significantly affected by lighting and weather conditions, performing poorly at night or in inclement weather.

[0004] To overcome the limitations of single sensors, multi-sensor fusion technology has gradually developed. By fusing data from different types of sensors, the accuracy and stability of environmental perception can be improved. However, existing multi-sensor fusion systems still have many problems. Summary of the Invention

[0005] To address existing technical problems, this invention provides a data fusion method, apparatus, electronic device, and storage medium based on multiple sensors. First, current weather conditions and road conditions are obtained through analysis of current image data. Then, appropriate wavelengths of lidar and sensor weighting coefficients are selected based on the current weather and road conditions. Finally, data from lidar, millimeter-wave radar, and camera are fused. By dynamically switching the laser wavelength and adjusting the weights, the performance degradation problem of a single sensor under adverse weather conditions is solved.

[0006] In a first aspect, embodiments of this application provide a multi-sensor-based data fusion method applied to a roadside sensing unit, the method comprising:

[0007] Get the current image data;

[0008] The current weather conditions and current road conditions are determined based on the current image data; the current weather conditions include rainy / foggy weather conditions, snowy weather conditions, and non-special weather conditions.

[0009] Based on the current weather conditions, a target lidar is determined from among multiple lidars in the roadside sensing unit; the multiple lidars emit lasers of different wavelengths; the roadside sensing unit also includes a millimeter-wave radar and a camera that acquires the current image data; the multiple lidars include a first lidar, a second lidar, and a third lidar; the wavelength emitted by the first lidar is greater than the wavelength emitted by the second lidar, and the wavelength emitted by the second lidar is greater than the wavelength emitted by the third lidar;

[0010] Acquire the first radar data of the target lidar and the second radar data of the millimeter-wave radar;

[0011] The weighting coefficients of the first radar data, the second radar data, and the current image data are determined based on the current weather conditions and the current road conditions.

[0012] The first radar data, the second radar data, and the current image data are fused based on the weighting coefficients.

[0013] In one alternative embodiment, the current weather conditions include rainy / foggy weather conditions, snowy weather conditions, and non-special weather conditions;

[0014] Determining the current weather conditions and current road conditions based on the current image data includes:

[0015] Extract image features from the current image data; the image features include overall image contrast, random motion pixel ratio data, image gradient data, and local region grayscale standard deviation;

[0016] If the image features satisfy a first condition, the rainy / foggy weather condition is determined to be the current weather condition; or if the image features satisfy a second condition, the snowy weather condition is determined to be the current weather condition; or if the image features do not satisfy either the first or the second condition, the non-special weather condition is determined to be the current weather condition; the first condition is that the image gradient data is less than a gradient threshold, or the grayscale standard deviation of the local area is less than a standard deviation threshold; the second condition is that the proportion of randomly moving pixels is greater than a second noise threshold.

[0017] The current road conditions are determined based on the current image data.

[0018] In one optional embodiment, the plurality of lidars includes a first lidar, a second lidar, and a third lidar; the wavelength emitted by the first lidar is greater than the wavelength emitted by the second lidar, and the wavelength emitted by the second lidar is greater than the wavelength emitted by the third lidar.

[0019] The step of determining the target lidar among multiple lidars in the roadside sensing unit based on the current weather conditions includes:

[0020] If the current weather conditions are the non-special weather conditions, the second lidar is determined to be the target lidar;

[0021] Alternatively, if the current weather conditions are rainy or foggy, the first lidar is determined to be the target lidar.

[0022] Alternatively, if the current weather conditions are snowy, the third lidar is identified as the target lidar.

[0023] In one optional embodiment, the current road conditions include road type information;

[0024] Determining the current weather conditions and current road conditions based on the current image data includes:

[0025] Determine the current weather conditions based on the current image data;

[0026] Road features are extracted from the current image data based on a convolutional neural network.

[0027] The road type information is determined based on the road features.

[0028] In an optional embodiment, the image features further include current illumination intensity; determining the weighting coefficients of the first radar data, the second radar data, and the current image data based on the current weather conditions and the current road conditions includes:

[0029] If the current light intensity is greater than the first preset light intensity threshold, and the current weather conditions are the non-special weather conditions, the first image weight is determined as the weight coefficient of the current image data, the first laser weight is determined as the weight coefficient of the first radar data, and the first millimeter wave weight is determined as the weight coefficient of the second radar data.

[0030] Alternatively, if the current light intensity is greater than or equal to the second preset light intensity threshold, less than or equal to the first preset light intensity threshold, and the current weather condition is the non-special weather condition, then the second image weight is determined as the weight coefficient of the current image data, the second laser weight is determined as the weight coefficient of the first radar data, and the second millimeter wave weight is determined as the weight coefficient of the second radar data; the second image weight is less than the first image weight, the second laser weight is less than the first laser weight, and the second millimeter wave weight is greater than the first millimeter wave weight.

[0031] Alternatively, if the current light intensity is less than the second preset light intensity threshold, and the current weather conditions are rain / fog or snow, then the third image weight is determined as the weight coefficient of the current image data, the third laser weight is determined as the weight coefficient of the first radar data, and the third millimeter wave weight is determined as the weight coefficient of the second radar data; the third image weight is less than the second image weight, the third laser weight is less than the second laser weight, and the third millimeter wave weight is greater than the second millimeter wave weight.

[0032] The weighting coefficients of the first radar data, the second radar data, and the current image data are corrected based on the road type information.

[0033] In an optional embodiment, after correcting the weighting coefficients of the first radar data, the second radar data, and the current image data based on the road type information, the method further includes:

[0034] If the area corresponding to the point cloud of the first radar data in the current image data is a static area, the first radar data is marked as false detection data and the weight of the first radar data is reduced.

[0035] In one optional embodiment, the current road conditions include road material information and road curvature information;

[0036] After acquiring the first radar data of the target lidar and the second radar data of the millimeter-wave radar, the process further includes:

[0037] The road material information and the road curvature information are determined based on the first radar data and the current image data;

[0038] The point cloud processing threshold is determined based on the road material information;

[0039] The region of interest clipping range is determined based on the road curvature information.

[0040] The current image data is cropped based on the region of interest cropping range to obtain the cropped current image data;

[0041] The first radar data is subjected to point cloud sparsification based on the point cloud processing threshold to obtain the first radar data after sparsification.

[0042] In an optional embodiment, determining the road material information and the road curvature information based on the first radar data and the current image data includes:

[0043] Based on the first radar data, determine the laser echo intensity, laser incident intensity, laser incident angle, distance between adjacent points, and angle difference between adjacent points;

[0044] Determine road texture features based on the current image data;

[0045] The road reflection coefficient is determined based on the laser echo intensity, the laser incident intensity, and the cosine value of the laser incident angle.

[0046] The road material information is determined based on the road reflectance coefficient and the road texture features;

[0047] The azimuth rate of change is determined based on the distance between adjacent points and the angle difference between adjacent points.

[0048] The road curvature information is determined based on the azimuth angle change rate.

[0049] In an optional embodiment, the fusion of the first radar data, the second radar data, and the current image data based on the weighting coefficients includes:

[0050] The texture confidence of the camera is determined based on the current image data;

[0051] The shape confidence level of the target lidar is determined based on the first radar data;

[0052] The motion confidence level of the millimeter-wave radar is determined based on the second radar data;

[0053] Based on the weighted coefficients, the texture confidence, shape confidence, and motion confidence are fused together to output a comprehensive target classification result.

[0054] Secondly, embodiments of this application provide a multi-sensor data fusion device applied to a roadside sensing unit, the device comprising:

[0055] The first acquisition module is used to acquire the current image data;

[0056] The first determining module is used to determine the current weather conditions and current road conditions based on the current image data; the current weather conditions include rainy / foggy weather conditions, snowy weather conditions, and non-special weather conditions.

[0057] The second determining module is used to determine the target lidar among multiple lidars in the roadside sensing unit based on the current weather conditions; the multiple lidars emit lasers of different wavelengths respectively; the roadside sensing unit also includes a millimeter-wave radar and a camera that acquires the current image data; the multiple lidars include a first lidar, a second lidar, and a third lidar; the wavelength emitted by the first lidar is greater than the wavelength emitted by the second lidar, and the wavelength emitted by the second lidar is greater than the wavelength emitted by the third lidar;

[0058] The second acquisition module is used to acquire the first radar data of the target lidar and the second radar data of the millimeter-wave radar.

[0059] The third determining module is used to determine the weighting coefficients of the first radar data, the second radar data, and the current image data based on the current weather conditions and the current road conditions.

[0060] The data fusion module is used to fuse the first radar data, the second radar data, and the current image data based on the weighting coefficients.

[0061] Thirdly, embodiments of this application provide an electronic device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, code set, or instruction set. The processor loads and executes the at least one instruction, at least one program, code set, or instruction set to implement the multi-sensor-based data fusion method of the first aspect.

[0062] Fourthly, embodiments of this application provide a computer-readable storage medium storing at least one instruction or at least one program, wherein the at least one instruction or at least one program is loaded and executed by a processor to implement the multi-sensor-based data fusion method of the first aspect.

[0063] Fifthly, embodiments of this application provide a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the multi-sensor-based data fusion method of the first aspect.

[0064] The data fusion method, apparatus, electronic device, and storage medium based on multiple sensors provided in this application have the following technical effects:

[0065] The process involves acquiring current image data; determining current weather and road conditions based on the current image data; identifying a target lidar among multiple lidars in a roadside sensing unit based on the current weather conditions; each lidar emitting lasers of different wavelengths; the roadside sensing unit also includes a millimeter-wave radar and a camera that acquires the current image data; acquiring first radar data from the target lidar and second radar data from the millimeter-wave radar; determining weighting coefficients for the first radar data, the second radar data, and the current image data based on the current weather and road conditions; and fusing the first radar data, the second radar data, and the current image data based on the weighting coefficients. In this embodiment, the current weather and road conditions are first obtained through analysis of the current image data, and then appropriate wavelength lidars and sensor weighting coefficients are selected based on these conditions. Finally, the data from the lidar, millimeter-wave radar, and camera are fused. By dynamically switching the laser wavelength and adjusting the weights, the performance degradation problem of a single sensor under adverse weather conditions is solved. Attached Figure Description

[0066] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1 This is a schematic diagram of an application environment provided in an embodiment of this application;

[0068] Figure 2 This is a flowchart illustrating a multi-sensor-based data fusion method provided in an embodiment of this application. Figure 1 ;

[0069] Figure 3 This is a flowchart illustrating a multi-sensor-based data fusion method provided in an embodiment of this application. Figure 2 ;

[0070] Figure 4 This is a flowchart illustrating a method for determining weighting coefficients provided in an embodiment of this application;

[0071] Figure 5 This is a schematic diagram of the structure of a data fusion device based on multiple sensors provided in an embodiment of this application;

[0072] Figure 6 This is a block diagram of the overall hardware structure of a server based on a multi-sensor data fusion method provided in an embodiment of this application. Detailed Implementation

[0073] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0074] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0075] Please see Figure 1 , Figure 1 This is a schematic diagram of an application environment provided in an embodiment of this application. The roadside sensing unit includes a sensor assembly 101 and a server 102.

[0076] In one possible embodiment, the sensor assembly 101 includes a millimeter-wave radar, a camera, and multiple lidars, each emitting laser light at a different wavelength.

[0077] The lidar is used to collect first laser data and send it to server 102, the millimeter-wave radar is used to collect second laser data and send it to server 102, and the camera is used to collect current image data and send it to server 102.

[0078] Specifically, in this application, multiple lidars are respectively a first lidar emitting a 1550nm wavelength laser, a second lidar emitting a 1064nm wavelength laser, and a third lidar emitting a 905nm wavelength laser. They are installed in an equilateral triangle on a roadside bracket to form a triangular measurement array, covering a sensing area with a radius of 200 meters.

[0079] The millimeter-wave radar uses a 77GHz sensor, providing stable all-weather measurement capabilities with an accuracy of ±5cm. Unaffected by weather, it works in conjunction with lidar arrays of different wavelengths, providing supplementary data when lidar performance is limited, ensuring comprehensive and stable environmental perception. The millimeter-wave radar is installed at the center of the lidar array, operating synchronously with it, and its sensing range is matched to that of the lidar.

[0080] The camera is preferably an intelligent CMOS camera with a resolution of 8 million pixels, a frame rate of 30fps, a horizontal field of view of 22° and a vertical field of view of 17°. It is installed on the upper side of the lidar triangular array (30cm away from the center of the array), with the optical axis at a 15° angle downwards to the horizontal plane, ensuring that the overlap rate with the detection area of ​​the lidar and millimeter-wave radar is ≥80% and that there is no mutual obstruction.

[0081] The camera, lidar, and millimeter-wave radar use the same reference coordinate system. Spatial calibration is completed using Zhang's calibration method with an error of ≤3cm. Time synchronization is achieved through hardware trigger signals with a trigger interval of 33ms, matching the camera's frame rate.

[0082] In one possible embodiment, server 102 acquires current image data; determines current weather conditions and current road conditions based on the current image data; identifies a target lidar among multiple lidars in the roadside sensing unit based on the current weather conditions; the multiple lidars emit lasers of different wavelengths respectively; the roadside sensing unit further includes a millimeter-wave radar and a camera that acquires the current image data; acquires first radar data of the target lidar and second radar data of the millimeter-wave radar; determines weighting coefficients for the first radar data, the second radar data, and the current image data based on the current weather conditions and current road conditions; and fuses the first radar data, the second radar data, and the current image data based on the weighting coefficients.

[0083] In this embodiment, the current weather conditions and road conditions are first obtained by analyzing the current image data. Then, the appropriate wavelength of the lidar and the weight coefficient of the sensor are selected based on the current weather conditions and road conditions. Finally, the data of lidar, millimeter-wave radar and camera are fused. By dynamically switching the laser wavelength and adjusting the weight, the performance degradation problem of a single sensor under severe weather conditions is solved.

[0084] The following describes a specific embodiment of a multi-sensor-based data fusion method according to this application. Figure 2 This is a flowchart illustrating a multi-sensor-based data fusion method provided in an embodiment of this application. Figure 1This specification provides method operation steps as shown in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operation steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only execution order. In actual system or server products, the methods shown in the embodiments or drawings can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment). Specifically, as shown in the embodiments or drawings... Figure 2 As shown, this method, applied to the server of a roadside sensing unit, may include:

[0085] S201: Obtain the current image data.

[0086] S202: Determine the current weather conditions and current road conditions based on the current image data.

[0087] S203: Based on the current weather conditions, the target lidar is determined among the multiple lidars in the roadside sensing unit; the multiple lidars emit lasers of different wavelengths respectively; the roadside sensing unit also includes a millimeter-wave radar and a camera that acquires the current image data.

[0088] S204: Acquire the first radar data of the target lidar and the second radar data of the millimeter-wave radar.

[0089] S205: Determine the weighting coefficients of the first radar data, the second radar data, and the current image data based on the current weather conditions and the current road conditions.

[0090] S206: Based on the weighting coefficients, fuse the first radar data, the second radar data, and the current image data.

[0091] Figure 3 This is a flowchart illustrating a multi-sensor-based data fusion method provided in an embodiment of this application. Figure 2 The method may include:

[0092] S301: Obtain the current image data.

[0093] In this embodiment of the application, the current image data is the current image data captured by the camera.

[0094] S302: Determine the current weather conditions and current road conditions based on the current image data.

[0095] In this embodiment of the application, the current weather conditions include rainy / foggy weather conditions, snowy weather conditions, and non-special weather conditions.

[0096] In one possible embodiment, determining current weather conditions and current road conditions based on the current image data includes: first, extracting image features from the current image data. Image features include overall image contrast, random motion pixel percentage data, image gradient data, and local region grayscale standard deviation.

[0097] Optionally, if the image gradient data is less than the gradient threshold, or the grayscale standard deviation of the local area is less than the standard deviation threshold, the rainy / foggy weather condition is determined as the current weather condition.

[0098] Optionally, if the proportion of randomly moving pixels is greater than the second noise threshold, the snowy weather condition is determined to be the current weather condition.

[0099] Optionally, if none of the above conditions are met, the non-special weather conditions are determined as the current weather conditions. Alternatively, if the overall image contrast is greater than a contrast threshold, and the proportion of randomly moving pixels is less than a first noise threshold, the non-special weather conditions are determined as the current weather conditions.

[0100] In this embodiment, the Laplacian operator is used to calculate image gradient data. When the image gradient data is <50, it is determined to be foggy. Since rain and fog reduce image contrast, the standard deviation of grayscale in local areas is calculated. When the standard deviation of grayscale in local areas is <30, it is determined to be rainy. When the proportion of short-distance random moving pixels is >15% by optical flow detection, it is determined to be snowy. Alternatively, when the overall image contrast is >80 and there is no obvious dynamic noise, i.e., the proportion of random moving pixels is >2%, it can be determined to be sunny or cloudy, which are non-special weather conditions.

[0101] The aforementioned feature extraction is achieved through the preprocessing module within the camera's built-in AI processing chip, with a processing latency of <10ms, providing real-time weather judgment basis for wavelength switching and weight adjustment.

[0102] In this embodiment of the application, the current road conditions include road type information. A lightweight convolutional neural network (MobileNetV3) is used to extract road features from the current image data. The feature dimension is compressed to 64 dimensions, and the processing latency is <15ms. The road type information is determined based on the road features.

[0103] This application specifically divides roads into 8 typical types: highway scenarios, tunnel scenarios, urban road scenarios, rural road scenarios, provincial and national highway scenarios, two-way road scenarios, intersection scenarios, and fork-in-the-road scenarios.

[0104] S303: Based on the current weather conditions, determine the target lidar among the multiple lidars in the roadside sensing unit.

[0105] In this embodiment of the application, the plurality of lidars includes a first lidar, a second lidar, and a third lidar; the wavelength emitted by the first lidar is greater than the wavelength emitted by the second lidar, and the wavelength emitted by the second lidar is greater than the wavelength emitted by the third lidar.

[0106] In one possible embodiment, if the current weather conditions are the non-special weather conditions, the second lidar is determined to be the target lidar.

[0107] Under normal weather conditions, a second lidar with a wavelength of 1064nm is selected because it is less affected by ambient light interference and has high measurement accuracy.

[0108] In one possible embodiment, if the current weather conditions are rainy or foggy, the first lidar is determined to be the target lidar.

[0109] In rainy or foggy weather conditions, a first-generation lidar with a wavelength of 1550nm is selected to improve penetration.

[0110] In another possible embodiment, if the current weather conditions are snowy conditions, the third lidar is determined to be the target lidar.

[0111] In snowy conditions, a third lidar with a wavelength of 905nm is selected to improve reflectivity.

[0112] By using a multi-wavelength lidar array and a wavelength adaptive selection algorithm, the optimal wavelength is dynamically switched according to weather conditions. For example, in rainy or foggy weather, the 1550nm wavelength is used to increase the penetration rate by 40%. Combined with the all-weather measurement capability of the 77GHz millimeter-wave radar, the problem of limited performance of a single sensor in adverse weather conditions is effectively solved, ensuring high-precision perception in all weather conditions.

[0113] S304: Acquire the first radar data of the target lidar and the second radar data of the millimeter-wave radar.

[0114] S305: Determine the road material information and the road curvature information based on the first radar data and the current image data.

[0115] In one possible embodiment, determining the road material information and the road curvature information based on the first radar data and the current image data includes:

[0116] S3051: Determine the laser echo intensity, laser incident intensity, laser incident angle, distance between adjacent points, and angle difference between adjacent points based on the first radar data.

[0117] The laser incident intensity can be obtained through the laser radar transmitter, the laser echo intensity can be obtained through the laser radar receiver, and the laser incident angle can be obtained through the laser radar attitude sensor.

[0118] The distance and angle difference between adjacent points are calculated using point cloud data from lidar.

[0119] S3052: Determine road texture features based on the current image data.

[0120] Texture features are extracted from the current image data to obtain road texture features.

[0121] S3053: Determine the road reflection coefficient based on the laser echo intensity, the laser incident intensity, and the cosine value of the laser incident angle.

[0122] In this embodiment, the road reflection coefficient is determined based on the laser echo intensity, the laser incident intensity, and the cosine value of the laser incident angle θ. .

[0123] S3054: Determine the road material information based on the road reflectance coefficient and the road texture features.

[0124] The road reflectance coefficient R is corrected using road texture features. These features represent the confidence level of the road texture. For example, a cement texture indicates a high confidence level that the road material is cement, therefore, the road reflectance coefficient R is multiplied by a factor of 1.1. Similarly, an asphalt texture indicates a high confidence level that the road material is asphalt, therefore, the road reflectance coefficient R is multiplied by a factor of 0.9. Finally, the corrected road reflectance coefficient is used to find the corresponding road material information in a pre-defined table.

[0125] S3055: Determine the azimuth rate of change based on the distance between adjacent points and the angle difference between adjacent points.

[0126] In this embodiment of the application, based on the distance between adjacent points and the angle difference between adjacent points Determine the rate of change of azimuth angle .

[0127] S3056: Determine the road curvature information based on the azimuth angle change rate.

[0128] S306: Determine the point cloud processing threshold based on the road material information.

[0129] In this embodiment, the road reflectivity R, representing road material information, reflects the road surface's ability to reflect laser light and directly affects the point cloud density requirement. For low-reflectivity materials like asphalt pavement, more point cloud data needs to be retained to ensure penetration detection, while for high-reflectivity materials like cement pavement, the density needs to be reduced to suppress noise. Assuming a conventional point cloud processing threshold of 10cm, the point cloud processing threshold determined based on road material information can be 20cm (asphalt) or 5cm (cement).

[0130] S307: Determine the clipping range of the region of interest based on the road curvature information.

[0131] In this embodiment, roads can be classified into straight roads, ordinary curves, and sharp curves based on the azimuth change rate C, which represents road curvature information. For sharp curves with high curvature, the lateral detection range of the region of interest is expanded; for straight roads and ordinary curves with low curvature, the focus is on long-distance longitudinal detection. This configuration enhances the long-distance target recognition rate on straight roads while reducing the edge detection rate of curves.

[0132] S308: Crop the current image data based on the region of interest cropping range to obtain the cropped current image data.

[0133] S309: Perform point cloud sparsification processing on the first radar data based on the point cloud processing threshold to obtain the first radar data after sparsification processing.

[0134] In one possible embodiment, the current image data is cropped based on the region of interest (ROI) cropping range, and the first radar data is subjected to point cloud sparsification based on a point cloud processing threshold. Through point cloud sparsification and region-selective processing techniques, and by performing cluster analysis on the point cloud, points exceeding the point cloud processing threshold from the center point are marked as non-critical points and discarded, significantly reducing the computational load of non-critical data. By applying ROI cropping technology to the camera's current image data, only the ROI image region is retained, further reducing the large amount of image processing. This reduces system energy consumption, improves data processing efficiency, and ensures real-time performance.

[0135] In another possible embodiment, when the camera detects that the image sharpness drops by more than 20% for multiple consecutive frames (e.g., 3 frames) and detects an increase in rain and fog concentration, the switching of the first 1550nm lidar can be completed within 50ms, and the point cloud processing threshold can be adjusted simultaneously (e.g., from 10cm to 15cm) to retain more effective point clouds that penetrate the rain and fog.

[0136] S310: Determine the weighting coefficients of the first radar data, the second radar data, and the current image data based on the current weather conditions and the current road conditions.

[0137] S311: Based on the weighting coefficients, fuse the first radar data, the second radar data, and the current image data.

[0138] In one possible embodiment, fusing the first radar data, the second radar data, and the current image data based on the weighting coefficients includes:

[0139] S3111: Determine the texture confidence of the camera based on the current image data.

[0140] S3112: Determine the shape confidence level of the target lidar based on the first radar data.

[0141] S3113: Determine the motion confidence level of the millimeter-wave radar based on the second radar data.

[0142] S3114: Based on the weight coefficients, fuse the texture confidence, the shape confidence, and the motion confidence to output the target comprehensive classification result.

[0143] In the data fusion stage, the first step is sensor-level data processing (20ms): First, FPGA is used to realize distortion correction and adaptive exposure adjustment of camera images. Specifically, ISO100-800 is dynamically adjusted according to the light sensor data, and time synchronization is performed with the point cloud data of the target lidar and the echo data of the millimeter-wave radar to make the error ≤5ms.

[0144] Next is feature-level data processing (100ms): The NPU runs a lightweight CNN to extract image features, including target contours, color channel histograms, texture gradients, etc., and associates and matches them with the point cloud features of the LiDAR and the motion features of the millimeter-wave radar.

[0145] Finally, there is the decision-level data processing (50ms): based on DS evidence theory, the texture confidence of the camera is fused with the shape confidence of the target lidar and the motion confidence of the millimeter-wave radar to output the comprehensive target classification result.

[0146] Figure 4 This is a flowchart illustrating a method for determining weighting coefficients provided in an embodiment of this application. The method may include:

[0147] S401: Determine whether the current light intensity is greater than the first preset light intensity threshold. If yes, execute S402; otherwise, execute S405.

[0148] S402: Determine whether the current weather conditions are the non-special weather conditions. If yes, proceed to S403; otherwise, proceed to S404.

[0149] S403: Determine the first image weight as the weight coefficient of the current image data, determine the first laser weight as the weight coefficient of the first radar data, and determine the first millimeter wave weight as the weight coefficient of the second radar data.

[0150] In this embodiment, if the current light intensity is greater than 5000 Lux and the current weather conditions are not special weather conditions, it indicates that it is a sunny day. At this time, the weight of the current image data of the camera is 0.15, the weight of the first laser of the target lidar is 0.7, and the weight of the second radar data of the millimeter-wave radar is 0.15, which can achieve fine classification of the target.

[0151] S404: Maintain the initial weighting coefficients.

[0152] S405: Determine whether the current light intensity is greater than or equal to the second preset light intensity threshold. If yes, execute S406; otherwise, execute S408.

[0153] S406: Determine whether the current weather conditions are the non-special weather conditions. If yes, proceed to S407; otherwise, proceed to S404.

[0154] S407: Determine the second image weight as the weight coefficient of the current image data, determine the second laser weight as the weight coefficient of the first radar data, and determine the second millimeter wave weight as the weight coefficient of the second radar data.

[0155] In this embodiment of the application, the second image weight is less than the first image weight, the second laser weight is less than the first laser weight, and the second millimeter wave weight is greater than the first millimeter wave weight.

[0156] Specifically, the current light intensity is in the range of 1000-5000 Lux, and the current weather conditions are not special weather conditions, indicating that it is cloudy. At this time, the weight of the camera's current image data is 0.1, the weight of the first laser data of the target lidar is 0.6, and the weight of the second radar data of the millimeter-wave radar is 0.3, in order to balance the performance of lidar and millimeter-wave radar.

[0157] S408: Determine whether the current weather conditions are rainy / foggy or snowy. If yes, execute S409; otherwise, execute S404.

[0158] S409: Determine the third image weight as the weight coefficient of the current image data, determine the third laser weight as the weight coefficient of the first radar data, and determine the third millimeter wave weight as the weight coefficient of the second radar data.

[0159] In this embodiment of the application, the third image weight is less than the second image weight, the third laser weight is less than the second laser weight, and the third millimeter wave weight is greater than the second millimeter wave weight.

[0160] Specifically, if the current light intensity is less than 1000 Lux and the current weather conditions are rain, fog, or snow, it indicates that it is nighttime or severe weather. In this case, the weight of the camera's current image data is 0.05, the weight of the first laser data from the target lidar is 0.3, and the weight of the second radar data from the millimeter-wave radar is 0.65, with lidar and millimeter-wave radar as the main sensing sources.

[0161] S410: Correct the weighting coefficients of the first radar data, the second radar data, and the current image data based on the road type information.

[0162] It is also possible to dynamically adjust the weights of each sensor based on Kalman filtering and DS evidence theory, combined with factors such as ambient light intensity, weather conditions, and road type.

[0163] S411: If the area corresponding to the point cloud of the first radar data in the current image data is a static area, execute the procedure; otherwise, execute S404.

[0164] S412: Mark the first radar data as false detection data and reduce the weight of the first radar data.

[0165] The detection results of the target LiDAR can be verified by examining the texture continuity of the camera image. Specifically, if the corresponding area of ​​the LiDAR point cloud in the image is the sky or a static background, it is marked as a suspected false detection and its confidence level is reduced.

[0166] In another possible embodiment, a sensor reliability assessment model can be built based on a Bayesian network to calculate the signal-to-noise ratio, consistency, and integrity of each sensor in real time, forming a 12-dimensional sensor state vector. The Double-Q Learning (DQN) algorithm is used, taking road features and sensor state vectors as input, and outputting the optimal sensor weight configuration and fusion strategy. The spatiotemporal attention mechanism is used to achieve accurate alignment of multi-sensor data (time error <5ms), and a hierarchical progressive fusion strategy is adopted to improve performance in different scenarios.

[0167] An integrated sensor fault detection and recovery mechanism is implemented. When sensor performance degradation is detected, the strategy reconfiguration is completed within 200ms to ensure continuous and reliable system operation.

[0168] This application achieves a measurement accuracy of ±2cm in clear weather and ±3cm in rain and snow by fusing data from multiple sensors. Furthermore, through a multi-level reinforcement learning framework, it improves perception accuracy by 18% on urban roads and increases target detection distance on highways by 25%, providing accurate environmental data for autonomous driving. Combined with camera texture features, vehicle type recognition accuracy is increased from 85% to 96%, and traffic sign recognition accuracy reaches 98%.

[0169] This application also provides a data fusion device based on multiple sensors. Figure 5 This is a schematic diagram of the structure of a multi-sensor data fusion device provided in an embodiment of this application, as shown below. Figure 5 As shown, the device 500 includes:

[0170] The first acquisition module 501 is used to acquire the current image data;

[0171] The first determining module 502 is used to determine the current weather conditions and current road conditions based on the current image data;

[0172] The second determining module 503 is used to determine the target lidar among the multiple lidars of the roadside sensing unit based on the current weather conditions; the multiple lidars emit lasers of different wavelengths respectively; the roadside sensing unit also includes a millimeter-wave radar and a camera that acquires the current image data;

[0173] The second acquisition module 504 is used to acquire the first radar data of the target lidar and the second radar data of the millimeter-wave radar.

[0174] The third determining module 505 is used to determine the weighting coefficients of the first radar data, the second radar data and the current image data based on the current weather conditions and the current road conditions.

[0175] The data fusion module 506 is used to fuse the first radar data, the second radar data and the current image data based on the weighting coefficients.

[0176] In one optional embodiment, the current weather conditions include rainy / foggy weather conditions, snowy weather conditions, and non-special weather conditions; further comprising:

[0177] The first feature extraction module is used to extract image features from the current image data; the image features include overall image contrast, random motion pixel ratio data, image gradient data, and local region grayscale standard deviation.

[0178] The fourth determining module is used to determine the rainy / foggy weather condition as the current weather condition if the image features satisfy a first condition; or to determine the snowy weather condition as the current weather condition if the image features satisfy a second condition; or to determine the non-special weather condition as the current weather condition if the image features do not satisfy either the first or the second condition; the first condition is that the image gradient data is less than a gradient threshold, or the grayscale standard deviation of the local area is less than a standard deviation threshold; the second condition is that the proportion of randomly moving pixels is greater than a second noise threshold.

[0179] The fifth determining module is used to determine the current road conditions based on the current image data.

[0180] In one optional embodiment, the plurality of lidars includes a first lidar, a second lidar, and a third lidar; the wavelength emitted by the first lidar is greater than the wavelength emitted by the second lidar, and the wavelength emitted by the second lidar is greater than the wavelength emitted by the third lidar; further comprising:

[0181] The sixth determining module is used to determine the second lidar as the target lidar if the current weather conditions are the non-special weather conditions;

[0182] The seventh determining module is used to determine the first lidar as the target lidar, either if the current weather conditions are the rainy / foggy weather conditions.

[0183] The eighth determining module is used to determine the third lidar as the target lidar, either if the current weather conditions are the snowy conditions or if the current weather conditions are the snowy conditions.

[0184] In one optional embodiment, the current road conditions include road type information; and further include:

[0185] The ninth determining module is used to determine the current weather conditions based on the current image data;

[0186] The second feature extraction module is used to extract road features from the current image data based on a convolutional neural network.

[0187] The tenth determining module is used to determine the road type information based on the road features.

[0188] In an optional embodiment, the image features further include the current illumination intensity; and also include:

[0189] The eleventh determining module is used to determine the first image weight as the weight coefficient of the current image data, the first laser weight as the weight coefficient of the first radar data, and the first millimeter wave weight as the weight coefficient of the second radar data if the current light intensity is greater than the first preset light intensity threshold and the current weather conditions are the non-special weather conditions.

[0190] The twelfth determining module is used to determine, either if the current light intensity is greater than or equal to a second preset light intensity threshold, less than or equal to a first preset light intensity threshold, and the current weather conditions are the non-special weather conditions, a second image weight as the weight coefficient of the current image data, a second laser weight as the weight coefficient of the first radar data, and a second millimeter wave weight as the weight coefficient of the second radar data; wherein the second image weight is less than the first image weight, the second laser weight is less than the first laser weight, and the second millimeter wave weight is greater than the first millimeter wave weight;

[0191] The thirteenth determining module is used to determine, either if the current light intensity is less than the second preset light intensity threshold and the current weather conditions are rain / fog or snow, a third image weight as the weight coefficient of the current image data, a third laser weight as the weight coefficient of the first radar data, and a third millimeter wave weight as the weight coefficient of the second radar data; wherein the third image weight is less than the second image weight, the third laser weight is less than the second laser weight, and the third millimeter wave weight is greater than the second millimeter wave weight;

[0192] The weight correction module is used to correct the weight coefficients of the first radar data, the second radar data, and the current image data based on the road type information.

[0193] In an optional embodiment, it further includes:

[0194] The false detection marking module is used to mark the first radar data as false detection data and reduce the weight of the first radar data if the area corresponding to the point cloud of the first radar data in the current image data is a static area.

[0195] In one optional embodiment, the current road conditions include road material information and road curvature information; and further include:

[0196] The fourteenth determining module is used to determine the road material information and the road curvature information based on the first radar data and the current image data;

[0197] The fifteenth determining module is used to determine the point cloud processing threshold based on the road material information;

[0198] The sixteenth determining module is used to determine the clipping range of the region of interest based on the road curvature information;

[0199] The cropping module is used to crop the current image data based on the cropping range of the region of interest to obtain the cropped current image data;

[0200] The sparsity processing module is used to perform point cloud sparsity processing on the first radar data based on the point cloud processing threshold to obtain the first radar data after sparsity processing.

[0201] In an optional embodiment, it further includes:

[0202] The seventeenth determining module is used to determine the laser echo intensity, laser incident intensity, laser incident angle, distance between adjacent points, and angle difference between adjacent points based on the first radar data;

[0203] The eighteenth determining module is used to determine road texture features based on the current image data;

[0204] The nineteenth determining module is used to determine the road reflection coefficient based on the laser echo intensity, the laser incident intensity, and the cosine value of the laser incident angle;

[0205] The twentieth determining module is used to determine the road material information based on the road reflectance coefficient and the road texture features;

[0206] The twenty-first determining module is used to determine the azimuth rate of change based on the distance between adjacent points and the angle difference between adjacent points;

[0207] The twenty-second determining module is used to determine the road curvature information based on the azimuth angle change rate.

[0208] In an optional embodiment, it further includes:

[0209] The twenty-third determining module is used to determine the texture confidence of the camera based on the current image data;

[0210] The twenty-fourth determining module is used to determine the shape confidence level of the target lidar based on the first radar data;

[0211] The twenty-fifth determining module is used to determine the motion confidence of the millimeter-wave radar based on the second radar data;

[0212] The confidence fusion module is used to fuse the texture confidence, shape confidence, and motion confidence based on the weight coefficients, and output a comprehensive target classification result.

[0213] The apparatus and method embodiments in this application are based on the same application concept.

[0214] The methods and embodiments provided in this application can be executed on a computer terminal, server, or similar computing device. Taking running on a server as an example, Figure 6 This is a hardware structure block diagram of the main server for a data fusion method based on multiple sensors provided in an embodiment of this application. For example... Figure 6 As shown, the main server 600 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 610 (CPUs 610 may include, but are not limited to, microprocessors (MCUs) or programmable logic devices (FPGAs), a memory 630 for storing data, and one or more storage media 620 (e.g., one or more mass storage devices) for storing application programs 623 or data 622. The memory 630 and storage media 620 may be temporary or persistent storage. The program stored in the storage media 620 may include one or more modules, each module may include a series of instruction operations on the server. Furthermore, the CPU 610 may be configured to communicate with the storage media 620 and execute the series of instruction operations stored in the storage media 620 on the main server 600. The total server 600 may also include one or more power supplies 660, one or more wired or wireless network interfaces 650, one or more input / output interfaces 640, and / or one or more operating systems 621, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0215] The input / output interface 640 can be used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the main server 600. In one example, the input / output interface 640 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the input / output interface 640 may be a radio frequency (RF) module used for wireless communication with the Internet.

[0216] Those skilled in the art will understand that Figure 6 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, the main server 600 may also include more than Figure 6 The more or fewer components shown, or having the same Figure 6 The different configurations shown.

[0217] This application provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, code set, or instruction set. The processor loads and executes the at least one instruction, at least one program, code set, or instruction set to implement the above-described data processing method.

[0218] Embodiments of this application also provide a computer-readable storage medium, which can be disposed in a server to store at least one instruction, at least one program, code set, or instruction set related to implementing a multi-sensor-based data fusion method in the method embodiment. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the above-described multi-sensor-based data fusion method.

[0219] Optionally, in this embodiment, the storage medium may be located at at least one of the multiple network servers in a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0220] As can be seen from the embodiments of the multi-sensor data fusion method, apparatus, electronic device, or storage medium provided in this application, the present application acquires current image data; determines current weather conditions and current road conditions based on the current image data; determines a target lidar among multiple lidars in the roadside sensing unit based on the current weather conditions; the multiple lidars emit lasers of different wavelengths respectively; the roadside sensing unit also includes a millimeter-wave radar and a camera that acquires the current image data; acquires first radar data of the target lidar and second radar data of the millimeter-wave radar; determines weight coefficients for the first radar data, the second radar data, and the current image data based on the current weather conditions and the current road conditions; and fuses the first radar data, the second radar data, and the current image data based on the weight coefficients. In the embodiments of this application, the current weather conditions and current road conditions are first obtained through analysis of the current image data, and appropriate wavelength lidars and sensor weight coefficients are selected according to the current weather conditions and current road conditions. Finally, the data from lidars, millimeter-wave radars, and cameras are fused. By dynamically switching laser wavelengths and adjusting weights, the performance degradation problem of a single sensor under adverse weather conditions is solved.

[0221] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0222] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0223] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0224] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A data fusion method based on multiple sensors, characterized in that, The method, applied to a roadside sensing unit, includes: Get the current image data; The current weather conditions and current road conditions are determined based on the current image data; the current weather conditions include rainy / foggy weather conditions, snowy weather conditions, and non-special weather conditions; the current road conditions include road type information; Based on the current weather conditions, a target lidar is determined from among multiple lidars in the roadside sensing unit; the multiple lidars emit lasers of different wavelengths; the roadside sensing unit also includes a millimeter-wave radar and a camera that acquires the current image data; the multiple lidars include a first lidar, a second lidar, and a third lidar; the wavelength emitted by the first lidar is greater than the wavelength emitted by the second lidar, and the wavelength emitted by the second lidar is greater than the wavelength emitted by the third lidar; Acquire the first radar data of the target lidar and the second radar data of the millimeter-wave radar; The weighting coefficients of the first radar data, the second radar data, and the current image data are determined based on the current weather conditions and the current road conditions. The first radar data, the second radar data, and the current image data are fused based on the weighting coefficients. Determining the current weather conditions and current road conditions based on the current image data includes: Extract image features from the current image data; the image features include overall image contrast, random motion pixel ratio data, image gradient data, and local region grayscale standard deviation; If the image features satisfy a first condition, the rainy / foggy weather condition is determined to be the current weather condition; or if the image features satisfy a second condition, the snowy weather condition is determined to be the current weather condition; or if the image features do not satisfy either the first or the second condition, the non-special weather condition is determined to be the current weather condition; the first condition is that the image gradient data is less than a gradient threshold, or the grayscale standard deviation of the local area is less than a standard deviation threshold; the second condition is that the proportion of randomly moving pixels is greater than a second noise threshold. The current road conditions are determined based on the current image data; The step of determining the target lidar among multiple lidars in the roadside sensing unit based on the current weather conditions includes: If the current weather conditions are the non-special weather conditions, the second lidar is determined to be the target lidar; Alternatively, if the current weather conditions are rainy or foggy, the first lidar is determined to be the target lidar. Alternatively, if the current weather conditions are snowy, the third lidar is determined to be the target lidar; Determining the current weather conditions and current road conditions based on the current image data includes: Determine the current weather conditions based on the current image data; Road features are extracted from the current image data based on a convolutional neural network. The road type information is determined based on the road features; The image features also include the current light intensity; the determination of weighting coefficients for the first radar data, the second radar data, and the current image data based on the current weather conditions and the current road conditions includes: If the current light intensity is greater than the first preset light intensity threshold, and the current weather conditions are the non-special weather conditions, the first image weight is determined as the weight coefficient of the current image data, the first laser weight is determined as the weight coefficient of the first radar data, and the first millimeter wave weight is determined as the weight coefficient of the second radar data. Alternatively, if the current light intensity is greater than or equal to the second preset light intensity threshold, less than or equal to the first preset light intensity threshold, and the current weather condition is the non-special weather condition, then the second image weight is determined as the weight coefficient of the current image data, the second laser weight is determined as the weight coefficient of the first radar data, and the second millimeter wave weight is determined as the weight coefficient of the second radar data. Alternatively, if the current light intensity is less than the second preset light intensity threshold, and the current weather conditions are rain / fog or snow, then a third image weight is determined as the weight coefficient of the current image data, a third laser weight is determined as the weight coefficient of the first radar data, and a third millimeter-wave weight is determined as the weight coefficient of the second radar data; the second image weight is less than the first image weight, the second laser weight is less than the first laser weight, and the second millimeter-wave weight is greater than the first millimeter-wave weight; the third image weight is less than the second image weight, the third laser weight is less than the second laser weight, and the third millimeter-wave weight is greater than the second millimeter-wave weight. The weighting coefficients of the first radar data, the second radar data, and the current image data are corrected based on the road type information.

2. The data fusion method based on multiple sensors according to claim 1, characterized in that, After correcting the weighting coefficients of the first radar data, the second radar data, and the current image data based on the road type information, the method further includes: If the area corresponding to the point cloud of the first radar data in the current image data is a static area, the first radar data is marked as false detection data and the weight of the first radar data is reduced.

3. The data fusion method based on multiple sensors according to claim 1, characterized in that, The current road conditions include road material information and road curvature information; After acquiring the first radar data of the target lidar and the second radar data of the millimeter-wave radar, the process further includes: The road material information and the road curvature information are determined based on the first radar data and the current image data; The point cloud processing threshold is determined based on the road material information; The region of interest clipping range is determined based on the road curvature information. The current image data is cropped based on the region of interest cropping range to obtain the cropped current image data; The first radar data is subjected to point cloud sparsification based on the point cloud processing threshold to obtain the first radar data after sparsification.

4. The data fusion method based on multiple sensors according to claim 3, characterized in that, The step of determining the road material information and the road curvature information based on the first radar data and the current image data includes: Based on the first radar data, determine the laser echo intensity, laser incident intensity, laser incident angle, distance between adjacent points, and angle difference between adjacent points; Determine road texture features based on the current image data; The road reflection coefficient is determined based on the laser echo intensity, the laser incident intensity, and the cosine value of the laser incident angle. The road material information is determined based on the road reflectance coefficient and the road texture features; The azimuth rate of change is determined based on the distance between adjacent points and the angle difference between adjacent points. The road curvature information is determined based on the azimuth angle change rate.

5. The data fusion method based on multiple sensors according to claim 1, characterized in that, The fusion of the first radar data, the second radar data, and the current image data based on the weighting coefficients includes: The texture confidence of the camera is determined based on the current image data; The shape confidence level of the target lidar is determined based on the first radar data; The motion confidence level of the millimeter-wave radar is determined based on the second radar data; Based on the weighted coefficients, the texture confidence, shape confidence, and motion confidence are fused together to output a comprehensive target classification result.

6. A data fusion device based on multiple sensors, characterized in that, The device, applied to a roadside sensing unit, includes: The first acquisition module is used to acquire the current image data; The first determining module is used to determine the current weather conditions and current road conditions based on the current image data; the current weather conditions include rainy / foggy weather conditions, snowy weather conditions, and non-special weather conditions; the current road conditions include road type information; The second determining module is used to determine the target lidar among multiple lidars in the roadside sensing unit based on the current weather conditions; the multiple lidars emit lasers of different wavelengths respectively; the roadside sensing unit also includes a millimeter-wave radar and a camera that acquires the current image data; the multiple lidars include a first lidar, a second lidar, and a third lidar; the wavelength emitted by the first lidar is greater than the wavelength emitted by the second lidar, and the wavelength emitted by the second lidar is greater than the wavelength emitted by the third lidar; The second acquisition module is used to acquire the first radar data of the target lidar and the second radar data of the millimeter-wave radar. The third determining module is used to determine the weighting coefficients of the first radar data, the second radar data, and the current image data based on the current weather conditions and the current road conditions. The data fusion module is used to fuse the first radar data, the second radar data, and the current image data based on the weighting coefficients; Determining the current weather conditions and current road conditions based on the current image data includes: Extract image features from the current image data; the image features include overall image contrast, random motion pixel ratio data, image gradient data, and local region grayscale standard deviation; If the image features satisfy a first condition, the rainy / foggy weather condition is determined to be the current weather condition; or if the image features satisfy a second condition, the snowy weather condition is determined to be the current weather condition; or if the image features do not satisfy either the first or the second condition, the non-special weather condition is determined to be the current weather condition; the first condition is that the image gradient data is less than a gradient threshold, or the grayscale standard deviation of the local area is less than a standard deviation threshold; the second condition is that the proportion of randomly moving pixels is greater than a second noise threshold. The current road conditions are determined based on the current image data; The step of determining the target lidar among multiple lidars in the roadside sensing unit based on the current weather conditions includes: If the current weather conditions are the non-special weather conditions, the second lidar is determined to be the target lidar; Alternatively, if the current weather conditions are rainy or foggy, the first lidar is determined to be the target lidar. Alternatively, if the current weather conditions are snowy, the third lidar is determined to be the target lidar; Determining the current weather conditions and current road conditions based on the current image data includes: Determine the current weather conditions based on the current image data; Road features are extracted from the current image data based on a convolutional neural network. The road type information is determined based on the road features; The image features also include the current light intensity; the determination of weighting coefficients for the first radar data, the second radar data, and the current image data based on the current weather conditions and the current road conditions includes: If the current light intensity is greater than the first preset light intensity threshold, and the current weather conditions are the non-special weather conditions, the first image weight is determined as the weight coefficient of the current image data, the first laser weight is determined as the weight coefficient of the first radar data, and the first millimeter wave weight is determined as the weight coefficient of the second radar data. Alternatively, if the current light intensity is greater than or equal to the second preset light intensity threshold, less than or equal to the first preset light intensity threshold, and the current weather condition is the non-special weather condition, then the second image weight is determined as the weight coefficient of the current image data, the second laser weight is determined as the weight coefficient of the first radar data, and the second millimeter wave weight is determined as the weight coefficient of the second radar data. Alternatively, if the current light intensity is less than the second preset light intensity threshold, and the current weather conditions are rain / fog or snow, then a third image weight is determined as the weight coefficient of the current image data, a third laser weight is determined as the weight coefficient of the first radar data, and a third millimeter-wave weight is determined as the weight coefficient of the second radar data; the second image weight is less than the first image weight, the second laser weight is less than the first laser weight, and the second millimeter-wave weight is greater than the first millimeter-wave weight; the third image weight is less than the second image weight, the third laser weight is less than the second laser weight, and the third millimeter-wave weight is greater than the second millimeter-wave weight. The weighting coefficients of the first radar data, the second radar data, and the current image data are corrected based on the road type information.

Images