Methods, apparatuses, devices, and media for data fusion

Abstract

This disclosure relates to methods, apparatus, devices, and media for data fusion. The method includes determining the environmental state of a vehicle based on data collected by the vehicle. The method further includes adjusting the weights of a first sensor and a second sensor based on the influence of the environmental state on multiple sensors of the vehicle. The method also includes fusing data collected by the first sensor and the second sensor based on the weights of the first and second sensors to obtain a perception result. The method of embodiments of this disclosure can dynamically allocate sensor fusion weights based on the influence of the real-time environmental state on multiple sensors of the vehicle, thereby significantly improving the accuracy and confidence of the perception result.

Description

Technical Field

[0001] This disclosure relates to the field of vehicle technology, and more specifically, to methods, apparatus, devices, and media for data fusion. Background Technology

[0002] In fields such as autonomous driving and suspension anti-sighting, vehicles perceive their environment using sensors such as cameras, lidar, millimeter-wave radar, ultrasonic radar, and inertial measurement units (IMUs). Cameras can acquire high-resolution image information and achieve rich semantic recognition. LiDAR measures reflection time by emitting laser beams and can generate high-precision 3D point cloud data to achieve 3D modeling of the environment. Millimeter-wave radar can use electromagnetic waves in specific frequency bands to detect the distance, speed, and angle of targets and has long-range detection capabilities. Ultrasonic radar is mainly used for short-range obstacle detection. Inertial measurement units can sense the vehicle's own acceleration, angular velocity, and attitude angle to help compensate for the loss of Global Positioning System (GPS) signals. Summary of the Invention

[0003] Embodiments of this disclosure provide a method, apparatus, device, and medium for data fusion.

[0004] In a first aspect of this disclosure, a method for data fusion is provided. The method includes determining the environmental state of the vehicle based on data collected by the vehicle. The method further includes adjusting the weights of a first sensor and a second sensor based on the influence of the environmental state on multiple sensors of the vehicle. The method also includes fusing data collected by the first sensor and the second sensor based on the weights of the first sensor and the second sensor to obtain a perception result.

[0005] In a second aspect of this disclosure, an apparatus for data fusion is provided. The apparatus includes an environmental state acquisition module configured to determine the environmental state of the vehicle based on data collected by the vehicle. The apparatus also includes a weight adjustment module configured to adjust the weights of a first sensor and a second sensor based on the influence of the environmental state on multiple sensors of the vehicle. The apparatus further includes a perception result acquisition module configured to fuse data collected by the first sensor and the second sensor based on the weights of the first and second sensors to obtain a perception result.

[0006] In a third aspect of this disclosure, a controller is provided. The controller includes one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the method provided according to a first aspect of this disclosure.

[0007] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores computer-executable instructions, which are executed by a processor to implement the method provided according to a first aspect of this disclosure.

[0008] It should be understood that the description in the Summary of the Invention section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram of an example environment in which several embodiments of the present disclosure may be implemented is shown;

[0011] Figure 2 A flowchart of a method for data fusion according to some embodiments of the present disclosure is shown;

[0012] Figure 3 Example diagrams of a system for data fusion according to some embodiments of the present disclosure are shown;

[0013] Figure 4 A flowchart of a method for acquiring sensor data according to some embodiments of the present disclosure is shown;

[0014] Figure 5 A flowchart of a method for adjusting sensor weights according to some embodiments of the present disclosure is shown;

[0015] Figure 6 A block diagram of an apparatus for data fusion according to some embodiments of the present disclosure is shown; and

[0016] Figure 7 A schematic block diagram of a controller according to some embodiments of the present disclosure is shown. Detailed Implementation

[0017] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0018] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0019] It is understood that all user-related data involved in this technical solution should be obtained and used only after authorization from the user. This means that if it is necessary to use a user's personal information in this technical solution, the user's explicit consent and authorization are required before obtaining this data; otherwise, no related data collection and use will be carried out. It should also be understood that when implementing this technical solution, relevant laws and regulations should be strictly followed in the process of data collection, use, and storage, and necessary technical measures should be taken to protect user data security and ensure the secure use of data.

[0020] As mentioned above, vehicles perceive their environment using sensors such as cameras, lidar, millimeter-wave radar, ultrasonic radar, and inertial measurement units (IMUs). However, these sensors are affected (interferenceed) differently under different environmental conditions. For example, cameras have strong recognition capabilities in good lighting conditions, but their performance drops sharply at night or in rainy or foggy weather. LiDAR is relatively less affected by weather, but raindrops and fog can significantly increase point cloud noise. Millimeter-wave radar has accurate ranging and can work in all weather conditions, but it cannot provide rich semantic information. Ultrasonic radar is waterproof and dustproof and is not affected by lighting conditions, but it is greatly affected by temperature and has poor directionality. An IMU alone cannot achieve the perception of environmental targets. Some existing solutions use fixed fusion weights to fuse multiple sensors such as LiDAR, cameras, and millimeter-wave radar. However, using fixed fusion weights cannot cope with the reality that sensor performance fluctuates drastically with the environment. For example, in rainy weather, the noise of LiDAR point clouds increases dramatically. If a high weight is still assigned, it will lead to a large number of misjudgments. At night, the recognition rate of cameras drops sharply due to insufficient light. If the LiDAR weight is not increased accordingly, it will cause target loss. Especially in complex and harsh scenarios such as "heavy rain at night", the performance shortcomings of sensors will be amplified, and fixed fusion strategies cannot be dynamically adapted. This leads to a serious decrease in the reliability, accuracy, and robustness of sensor perception, which in turn affects the safety and comfort of vehicle control (such as suspension anti-aiming).

[0021] To address this, embodiments of this disclosure propose a data fusion scheme. In embodiments of this disclosure, the method includes determining the environmental state of the vehicle based on data collected by the vehicle. The method further includes adjusting the weights of a first sensor and a second sensor based on the influence of the environmental state on multiple vehicle sensors. The method also includes fusing the data collected by the first sensor and the second sensor based on their respective weights to obtain a perception result. Through the method of embodiments of this disclosure, sensor fusion weights can be dynamically allocated based on the influence of the real-time environmental state on multiple vehicle sensors, thereby significantly improving the accuracy and confidence of the perception result.

[0022] Figure 1 A schematic diagram of an example environment 100 in which various embodiments of this disclosure may be implemented is shown. For example... Figure 1 As shown, example environment 100 includes vehicle 102, which refers to any type of motorized or non-motorized vehicle capable of carrying people and / or goods and being mobile. Figure 1 As shown, vehicle 102 is illustrated as a car. It should be understood that although vehicle 102 is... Figure 1The vehicle is illustrated as a sedan, but this is merely exemplary and far from limited to this; examples may also include buses, trucks, motorcycles, and electric vehicles. In some embodiments of this disclosure, vehicle 102 may include a controller, for example, a domain controller for the vehicle or an Advanced Driver Assistance Systems (ADAS) controller.

[0023] In environment 100, the controller of vehicle 102 can determine the environmental state 106 of the vehicle based on data 104. Data 104 may include multi-source sensing data collected by multiple sensors in vehicle 102. For example, data 104 may include at least one of rainfall, illumination, image contrast, or point cloud reflectance. Environmental state 106 may include at least one of sunny day, rainy day, night, or nighttime downpour. In some embodiments of this disclosure, the controller of vehicle 102 can identify the collected multi-source sensing data through a multi-condition voting strategy.

[0024] In environment 100, vehicle 102 also includes sensor 112 (also referred to as the first sensor) and sensor 114 (also referred to as the second sensor). Sensor 112 and sensor 114 can be at least one of LiDAR, camera, or millimeter-wave radar. Sensor 112 can correspond to weight 108, and sensor 114 can correspond to weight 110. Weights 108 and 110 can be adjusted based on the influence of environmental state 106 on sensor 112 and sensor 114. For example, when environmental state 106 is determined to be "rainy," because LiDAR is interfered with by raindrops, while the camera provides a more stable view of contours, and millimeter-wave radar is less affected by raindrops, the controller can set the weight of the camera to a higher value, while setting the weights of the millimeter-wave radar and LiDAR to lower values. When environmental state 106 is determined to be "night," the performance of the camera is limited in low light, while LiDAR is not affected by light, and millimeter-wave radar can compensate for distance errors. The controller can set the weight of LiDAR to a higher value, while setting the weights of the camera and millimeter-wave radar to lower values.

[0025] In environment 100, sensor 112 can collect data 116, and sensor 114 can collect data 118. The controller of vehicle 102 can fuse data 116 and data 118 based on weights 108 and 110 to obtain perception result 120. For example, data 116 and data 118 can be a speed bump outline image output by a camera with a confidence level of 0.85; data 116 or data 118 can also be a speed bump point cloud output by a lidar with a confidence level of 0.88; data 116 or data 118 can also be a speed bump distance of 45m output by a millimeter-wave radar with a confidence level of 0.9. Perception result 120 can be "speed bump distance 45m, type is concrete speed bump, width 1.2m, confidence level 0.92".

[0026] In this way, vehicle 102 can dynamically allocate the weights 108 and 110 of sensors 112 and 114 according to the real-time environmental conditions 106, thereby maximizing the advantages of the sensors in complex and harsh environments such as "heavy rain at night" and significantly improving the accuracy and confidence of the perception results 120.

[0027] Figure 2 A flowchart of a method 200 for data fusion according to some embodiments of the present disclosure is shown. Method 200 can be performed by a vehicle. For example, method 200 can be performed by… Figure 1 Vehicle 102 is executing. (For example...) Figure 2 As shown in box 202, method 200 includes determining the environmental state of the vehicle based on data collected from the vehicle. For example, in... Figure 1 In the environment 100 shown, the vehicle 102 can collect data 104 to determine the environmental state 106. The data 104 may include at least one of rainfall, illumination, image contrast or point cloud reflectance, and the environmental state 106 may include at least one of sunny day, rainy day, night or night rainstorm.

[0028] In box 204, method 200 includes adjusting the weights of a first sensor and a second sensor based on the influence of multiple vehicle sensors on environmental conditions. For example, in... Figure 1In the environment 100 shown, vehicle 102 can adjust the weights 108 of sensor 112 and 110 of sensor 114 based on the influence of environmental state 106 on sensor 112 and sensor 114. In some embodiments of this disclosure, sensor 112 and sensor 114 can be at least one of lidar, camera, or millimeter-wave radar. For example, when environmental state 106 is determined to be "rainy," because lidar is interfered with by raindrops, while the camera provides a more stable view of contours, and millimeter-wave radar is less affected by raindrops, the controller can set the weight of the camera to a higher value, while setting the weights of millimeter-wave radar and lidar to lower values. When environmental state 106 is determined to be "night," the performance of the camera is limited in low light, lidar is not affected by light, and millimeter-wave radar can compensate for distance errors. The controller can set the weight of lidar to a higher value, while setting the weights of camera and millimeter-wave radar to lower values.

[0029] In box 206, method 200 includes fusing data acquired by the first sensor and data acquired by the second sensor based on the weights of the first sensor and the second sensor to obtain a perception result. For example, in... Figure 1 In the environment 100 shown, vehicle 102 can fuse data 116 collected by sensor 112 and data 118 collected by sensor 114 based on weights 108 and 110. For example, data 116 or data 118 can be a speed bump outline image output by a camera with a confidence level of 0.85, data 116 or data 118 can also be a speed bump point cloud output by a lidar with a confidence level of 0.88, data 116 or data 118 can also be a speed bump distance of 45m output by a millimeter-wave radar with a confidence level of 0.9, and the perception result 120 can be "speed bump distance 45m, type is concrete speed bump, width 1.2m, confidence level 0.92".

[0030] In this way, Method 200 can dynamically allocate the weights of the sensors according to the real-time environmental conditions, thereby maximizing the advantages of the sensors in complex and harsh environments such as "heavy rain at night" and significantly improving the accuracy and confidence of the perception results.

[0031] Figure 3 Example diagrams of a system 300 for data fusion according to some embodiments of the present disclosure are shown. Figure 3As shown, system 300 may include an environment perception layer 302, a hardware execution layer 308, a fusion decision layer 310, and a target output layer 312. The environment perception layer 302 may include a multi-source data acquisition module 304 and an environment state determination module 306. The multi-source data acquisition module 304 can acquire low-level sensor signals and high-level perception features from various vehicle sensors. For example, it can acquire low-level sensor signals such as rainfall and illumination from rain sensors and light sensors, and it can also acquire high-level perception features such as image contrast and point cloud reflectivity from cameras and lidar. Then, the multi-source data acquisition module 304 sends the acquired low-level sensor signals and high-level perception features to the environment state determination module 306. Then, the environment state determination module 306 determines the current environment state of the vehicle based on the acquired low-level sensor signals and high-level perception features using a preset method. For example, the environment state may include "sunny day", "rainy day", "night" or "nighttime rainstorm".

[0032] In one embodiment of this disclosure, the environmental state determination module 306 can determine the environmental state of the vehicle through a multi-condition voting method. For example, if the rain sensor outputs 250 drops / minute (heavy rain), the light sensor outputs 80 lux (night), the camera outputs a contrast ratio of 22% and a noise rate of 21% (matching the characteristics of a nighttime heavy rain), and the lidar outputs a low reflectivity point cloud ratio of 35% and an effective point cloud density decrease of 30% (calibration judgment result), then the environmental state is determined to be a nighttime heavy rain through the multi-condition voting method, with a recognition delay of 80ms and an accuracy of 100%.

[0033] like Figure 3 As shown, the environmental perception layer 302 can send the determined environmental state to the hardware execution layer 308 and the fusion decision layer 310. The hardware execution layer 308 can trigger vehicle execution based on the received environmental state. The vehicle actuators may include high beam headlights 314 and windshield wipers 316. For example, when the environmental state is "nighttime heavy rain", the hardware execution layer 308 can automatically turn on the high beam headlights and tilt the high beam headlights downward by 5 degrees. The hardware execution layer 308 can also activate the high-speed windshield wipers. In this way, the accuracy and confidence of the sensors can be improved.

[0034] In one embodiment of this disclosure, the hardware execution layer 308 can also adjust the parameters of the sensor 318 according to the environmental conditions. The parameters can be at least one of point cloud noise rate, raindrop filtering, shutter speed, or sensitivity. For example, the hardware execution layer 308 can enable a raindrop-specific filtering algorithm for the LiDAR, thereby reducing the point cloud noise rate of the LiDAR from 35% to 12%. The hardware execution layer 308 can also increase the sensitivity of the camera to 800, adjust the shutter speed to 1 / 50s, and simultaneously enable night enhancement and raindrop removal algorithms, thereby improving the clarity of the images captured by the camera.

[0035] In some embodiments of this disclosure, a raindrop-specific filtering algorithm can preprocess and construct a neighborhood of the original point cloud, quickly obtain the local point set of each point through spatial indexing, and then extract key local features, such as calculating point cloud density, analyzing spatial distribution anisotropy and reflection intensity characteristics, to identify discrete, isolated raindrop noise points with low reflectivity. Then, by combining multi-frame point cloud data and using motion compensation to analyze the motion trajectory of the points, the algorithm can further distinguish between short-falling raindrops and continuously moving real objects. Finally, the identified raindrop points are filtered out from the point cloud through threshold judgment or machine learning classification models.

[0036] like Figure 3 As shown, the fusion decision layer 310 may include a weight determination module 320, a data acquisition module 322, and a data fusion module 324. The weight determination module 320 can determine the weights of multiple sensors in the vehicle based on the received environmental conditions. In one embodiment of this disclosure, the weight determination module 320 can adjust the weights of the sensors based on the influence of environmental conditions on the multiple sensors of the vehicle and a preset weight strategy. For example, the preset weight strategy may include a preset dynamic fusion weight rule set, which can be calibrated based on a large amount of measured data and can be used to define the optimal weight ratio of each sensor under different environmental types. In one embodiment of this disclosure, the dynamic fusion weight rule set may be as shown in Table 1:

[0037] Table 1: Dynamic Fusion Weight Rule Set

[0038] In some embodiments of this disclosure, if the sensor 318 is affected by a first environmental state more than by a second environmental state, the weight of the sensor 318 in the first environmental state can be adjusted to be less than its weight in the second environmental state. For example, as shown in Table 1, the influence of the lidar on rainy days is greater than its influence on sunny days, so the weight of the lidar on rainy days is less than its weight on sunny days.

[0039] like Figure 3As shown, the data acquisition module 322, triggered by the vehicle actuator at the hardware execution layer 308, collects sensor data based on the sensor's adjustment parameters. Then, the data acquisition module 322 sends the collected sensor data to the data fusion module 324. The data fusion module 324 makes sensor data fusion decisions based on the weights determined by the weight determination module 320 for each sensor. For example, in a nighttime rainstorm, the weight determination module 320 can look up Table 1 to obtain a 40% weight for the lidar, a 30% weight for the camera, and a 30% weight for the millimeter-wave radar. Then, after raindrop filtering, the lidar outputs a speed bump point cloud (20% reflectivity, complete outline) with a confidence level of 0.88, weighted at 40%. After the camera's sensitivity is adjusted (800) and raindrops are removed, it outputs a speed bump outline image (with clear edges) with a confidence level of 0.85. It participates in the fusion with a weight of 30%. The millimeter-wave radar outputs a speed bump distance of 45m (with an error of ±0.5m) with a confidence level of 0.9. It participates in the fusion with a weight of 30%. Then, the data fusion module 324 outputs the fused perception result: "Speed ​​bump parameters: distance 45m, cement speed bump, width 1.2m, confidence level 0.92".

[0040] In some embodiments of this disclosure, the fusion decision layer 310 can fine-tune based on the target identifier confidence level at predetermined intervals. For example, a standard speed bump recognition confidence level greater than or equal to 0.8 is considered valid. If the confidence level of a sensor is less than a preset value multiple times in a row, the weight of the sensor is reduced by a certain percentage, and the reduced portion is allocated to sensors with higher confidence levels. For example, the fusion decision layer 310 can fine-tune based on the target identifier confidence level every 100ms. If the confidence level of a sensor is less than 0.6 three times in a row, its weight is reduced by 5%, and the reduced portion is allocated to sensors with confidence levels greater than 0.8. For example, if the confidence level of the rain camera is 0.9 and the confidence level of the LiDAR is 0.5, then 5% of the LiDAR weight is allocated to the camera.

[0041] like Figure 3As shown, the fusion decision layer 310 can send the fused perception results to the target output layer 312. The target output layer 312 may include the vehicle's suspension system. For example, the fusion decision layer 310 can send "a concrete speed bump 45 meters ahead, with a confidence level of 0.92" to the vehicle's suspension system. Then, the suspension system controller determines the suspension adjustment parameters, which may include damping, stiffness, etc. Damping can be used to control the speed and force of suspension compression and rebound, and stiffness can be used to control the suspension's resistance to deformation. For example, when the suspension system controller obtains "a concrete speed bump 45 meters ahead...", the system can determine the suspension adjustment parameters. After obtaining a perception result of "confidence level 0.92" for a concrete speed bump, the suspension controller can formulate a phased damping adjustment strategy. Just before the wheel contacts the speed bump (e.g., within the first 5 meters), the damping is increased instantaneously. As the wheel rolls over the top of the speed bump, a higher damping is maintained to control the vehicle's posture. After the wheel leaves the speed bump, the damping is rapidly reduced, allowing the suspension system to rebound and restore contact with the road surface. Furthermore, the suspension controller can slightly reduce the spring stiffness when passing over the speed bump, thereby actively absorbing more impact energy from the road surface and reducing the peak impact force of the speed bump on the vehicle body. In this way, the suspension system has sufficient time to complete its adjustment, allowing the vehicle to pass smoothly and achieving satisfactory vertical acceleration comfort.

[0042] In this way, the system 300 can achieve a complete closed loop from sensing the environment to optimizing the sensor's operating state. By controlling sensor parameters and vehicle actuators, it can reduce noise and interference from the data source and lower the target misjudgment rate under severe weather conditions. In addition, through dynamic weight allocation, the system 300 can maximize the advantages of the sensors in complex and severe environments such as "heavy rain at night" and significantly improve the accuracy and confidence of the sensing results.

[0043] Figure 4 A flowchart of a method 400 for acquiring sensor data according to some embodiments of the present disclosure is shown. Method 400 can be performed by a vehicle. For example, method 400 can be performed by… Figure 1 Vehicle 102 is executing. (For example...) Figure 4 As shown, in block 402, method 400 may include adjusting parameters of the first sensor based on environmental conditions. For example, in... Figure 1In the environment 100 shown, the vehicle 102 can adjust the parameters of the sensor 112 based on the environmental state 106. The parameters can be at least one of point cloud noise rate, raindrop filtering, shutter speed, or ISO. For example, the vehicle 102 can enable a raindrop-specific filtering algorithm for the LiDAR, thereby reducing the point cloud noise rate of the LiDAR from 35% to 12%. The vehicle 102 can also increase the ISO of the camera to 800, adjust the shutter speed to 1 / 50s, and simultaneously enable night enhancement and raindrop removal algorithms, thereby improving the clarity of the images captured by the camera.

[0044] In some embodiments of this disclosure, a raindrop-specific filtering algorithm can preprocess and construct a neighborhood of the original point cloud, quickly obtain the local point set of each point through spatial indexing, and then extract key local features, such as calculating point cloud density, analyzing spatial distribution anisotropy and reflection intensity characteristics, to identify discrete, isolated raindrop noise points with low reflectivity. Then, by combining multi-frame point cloud data and using motion compensation to analyze the motion trajectory of the points, the algorithm can further distinguish between short-falling raindrops and continuously moving real objects. Finally, the identified raindrop points are filtered out from the point cloud through threshold judgment or machine learning classification models.

[0045] In box 404, method 400 may include triggering one or more vehicle actuators in the vehicle based on environmental conditions. For example, in... Figure 1 In the illustrated environment 100, vehicle 102 can trigger one or more vehicle actuators based on environmental state 106. These actuators may include at least one of high beam headlights and windshield wipers. For example, in a nighttime rainstorm, vehicle 102 can automatically turn on its high beam headlights, tilting the beam downwards by 5 degrees. Vehicle 102 can also activate high-speed windshield wipers. In some embodiments of this disclosure, the controller of vehicle 102 can generate execution instructions and send these instructions to the controllers corresponding to the vehicle actuators.

[0046] In box 406, method 400 may include acquiring data from a first sensor when one or more vehicle actuators are activated, for example, in... Figure 1 In the environment 100 shown, vehicle 102 can collect data 116 from sensor 112 when high beams are on and windshield wipers are running at high speed. For example, after raindrop filtering, lidar can collect point clouds of speed bumps with 20% reflectivity, complete outline, and 0.88 confidence level. After sensitivity adjustment and raindrop removal, camera can collect outline images of speed bumps with clear edges and 0.85 confidence level.

[0047] In this way, method 400 can improve the accuracy and confidence of sensor data acquisition by dynamically optimizing the sensor's own operating parameters according to the environmental conditions, and then linking the vehicle actuator to improve the sensor's working environment, thereby enabling data acquisition under optimized conditions.

[0048] Figure 5 A flowchart of a method 500 for adjusting sensor weights according to some embodiments of the present disclosure is shown. Method 500 can be performed by a vehicle. For example, method 500 can be performed by… Figure 1 Vehicle 102 is executing. (For example...) Figure 5 As shown, in block 502, method 500 may include obtaining the confidence level of the first sensor. For example, in... Figure 1 In the environment 100 shown, vehicle 102 can acquire the confidence level of sensor 112. The confidence level can be used to indicate the reliability of the sensor data. In some embodiments of this disclosure, the sensor can determine the confidence level through internal self-testing algorithms, internal consistency of output data (e.g., point cloud density, image sharpness, etc.), or cross-validation with short-term historical data and other sensor data.

[0049] In box 504, method 500 may include determining whether the confidence level of the first sensor is less than a preset value. If the confidence level of the first sensor is less than the preset value, then box 506 is executed, for example, in... Figure 1 In the environment 100 shown, vehicle 102 can determine whether the confidence level of sensor 112 is less than 0.8. In block 506, method 500 may include reducing the weight of the first sensor, for example, in... Figure 1 In the environment 100 shown, if the confidence level of sensor 112 is less than 0.8, vehicle 102 can reduce the weight of sensor 112.

[0050] In some embodiments of this disclosure, method 500 can fine-tune based on the target identifier confidence level at predetermined intervals. For example, a confidence level greater than or equal to 0.8 is considered valid for identifying standard speed bumps. If the confidence level of a sensor is less than a preset value multiple times in a row, the weight of the sensor is reduced by a certain percentage, and the reduced portion is allocated to sensors with higher confidence levels. For example, method 500 can fine-tune based on the target identifier confidence level every 100ms. If the confidence level of a sensor is less than 0.6 three times in a row, its weight is reduced by 5%, and the reduced portion is allocated to sensors with confidence levels greater than 0.8. For example, if the confidence level of the rain camera is 0.9 and the confidence level of the LiDAR is 0.5, then 5% of the weight of the LiDAR is allocated to the camera.

[0051] In this way, Method 500, based on dynamic calibration of weights, can improve the dynamic adaptability of sensor fusion weights, thereby further improving the accuracy and confidence of the sensing results.

[0052] Figure 6 A block diagram of an apparatus 600 for data fusion according to some embodiments of the present disclosure is shown. Figure 6 As shown, the device includes an environmental state acquisition module 602, configured to determine the environmental state of the vehicle based on data collected by the vehicle. The device also includes a weight adjustment module 604, configured to adjust the weights of a first sensor and a second sensor based on the influence of the environmental state on multiple vehicle sensors. The device further includes a perception result acquisition module 606, configured to fuse the data collected by the first sensor and the second sensor based on their respective weights to obtain a perception result.

[0053] In some embodiments, the environmental state includes a first environmental state and a second environmental state, and the first sensor is more affected by the first environmental state than by the second environmental state. The device 600 also includes an influence usage module configured to adjust the weight of the first sensor in the first environmental state to be less than the weight of the first sensor in the second environmental state.

[0054] In some embodiments, the device 600 further includes a parameter adjustment module configured to adjust the parameters of the first sensor based on environmental conditions. The device 600 also includes a parameter usage module configured to acquire data from the first sensor based on the adjustment.

[0055] In some embodiments, the parameters of the first sensor include at least one of the following: point cloud noise rate, raindrop filtering, shutter speed, or sensitivity.

[0056] In some embodiments, the device 600 further includes a vehicle actuator triggering module configured to trigger one or more vehicle actuators in a vehicle based on environmental conditions, the vehicle actuators including at least one of high beam headlights or windshield wipers. The device 600 also includes a vehicle actuator usage module configured to acquire data from a first sensor when one or more vehicle actuators are activated.

[0057] In some embodiments, the weight adjustment module 604 includes a preset weight strategy usage module, configured to influence and adjust the weights of the first sensor and the second sensor using the preset weight strategy.

[0058] In some embodiments, the device 600 further includes a confidence level acquisition module configured to acquire the confidence level of the first sensor. The device 600 also includes a weight adjustment module configured to adjust the weight of the first sensor based on the confidence level.

[0059] In some embodiments, the weight adjustment module includes a weight reduction module configured to reduce the weight of the first sensor in response to the confidence level of the first sensor being less than a preset value.

[0060] In some embodiments, the first sensor and the second sensor respectively include at least one of lidar, camera or millimeter-wave radar.

[0061] In some embodiments, the environmental conditions include at least one of sunny day, rainy day, night, and nighttime downpour.

[0062] In some embodiments, the device 600 further includes a suspension adjustment module configured to control the vehicle's suspension adjustment based on the perception results.

[0063] It is understood that by utilizing the device 600 of this disclosure, at least one of the many advantages achievable by the methods or processes described above can be realized. For example, the device 600 can achieve a complete closed loop from sensing the environment to optimizing the sensor's operating state. By controlling sensor parameters and vehicle actuators, it can reduce noise and interference from the data source, lower the target misjudgment rate under adverse weather conditions, and, through dynamic weight allocation, enable the device 600 to maximize the advantages of the sensors in complex adverse environments such as "heavy rain at night," significantly improving the accuracy and confidence of the sensing results.

[0064] Figure 7 A block diagram of a controller 700 that can implement various embodiments of the present disclosure is shown. The controller 700 may, for example, be disposed in a vehicle. Reference Figure 7 As shown, the controller 700 includes a processor 701, which can perform various appropriate actions and processes based on computer program instructions loaded into random access memory (RAM) 703 according to computer program instructions stored in read-only memory (ROM) 702. The RAM 703 may also store various programs and data required for the operation of the controller 700. The processor 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0065] Processor 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 701 performs the various methods and processes described above, such as method 200. For example, in some embodiments, method 200 may be implemented as a computer software program tangibly contained in a machine-readable medium. In some embodiments, part or all of the computer program may be loaded and / or mounted to controller 700 via ROM 702. When the computer program is loaded into RAM 703 and executed by processor 701, one or more steps of method 200 described above may be performed. Alternatively, in other embodiments, processor 701 may be configured to perform method 200 by any other suitable means (e.g., by means of firmware).

[0066] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload programmable logic devices (CPLDs), and so on.

[0067] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0068] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. Furthermore, although operations are depicted in a specific order, this should be understood as requiring that such operations be performed in the specific order shown or in sequential order, or requiring that all illustrated operations be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the foregoing discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations.

[0069] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for data fusion, comprising: Based on the data collected from the vehicle, determine the environmental conditions in which the vehicle is located; Based on the influence of the environmental conditions on the multiple sensors of the vehicle, the weights of the first sensor and the second sensor are adjusted. as well as Based on the weights of the first sensor and the second sensor, the data collected by the first sensor and the second sensor are fused together to obtain the perception result.

2. The method according to claim 1, wherein the environmental state includes a first environmental state and a second environmental state, the first sensor is more affected by the first environmental state than by the second environmental state, and the method further includes: The weight of the first sensor in the first environmental state is adjusted to be less than the weight of the first sensor in the second environmental state.

3. The method according to claim 1, further comprising: Based on the environmental conditions, adjust the parameters of the first sensor; as well as Based on the aforementioned adjustments, data from the first sensor is collected.

4. The method according to claim 3, wherein the parameters of the first sensor include at least one of the following: point cloud noise rate, raindrop filtering, shutter speed, or sensitivity.

5. The method according to claim 3, further comprising: Based on the environmental conditions, one or more vehicle actuators in the vehicle are triggered, the vehicle actuators including at least one of high beam headlights or windshield wipers; The data collected by the first sensor when the one or more vehicle actuators are activated.

6. The method of claim 1, wherein adjusting the weights of the first sensor and the second sensor based on the influence of the environmental state on the multiple sensors of the vehicle comprises: Based on the aforementioned influence and a preset weighting strategy, the weights of the first sensor and the second sensor are adjusted.

7. The method of claim 6, further comprising: Obtain the confidence level of the first sensor; as well as The weight of the first sensor is adjusted based on the confidence level.

8. The method of claim 7, wherein adjusting the weights of the first sensor based on the confidence level comprises: In response to the confidence level of the first sensor being less than a preset value, the weight of the first sensor is reduced.

9. The method according to claim 1, wherein the first sensor and the second sensor respectively include at least one of lidar, camera or millimeter-wave radar.

10. The method of claim 1, wherein the environmental condition includes at least one of sunny day, rainy day, night, and nighttime downpour.

11. The method according to claim 1, further comprising: Based on the perceived results, the vehicle's suspension is adjusted.

12. An apparatus for data fusion, comprising: The environmental status acquisition module is configured to determine the environmental status of the vehicle based on the data collected by the vehicle. The weight adjustment module is configured to adjust the weights of the first sensor and the second sensor based on the influence of the environmental conditions on the multiple sensors of the vehicle. as well as The perception result acquisition module is configured to fuse the data collected by the first sensor and the data collected by the second sensor based on the weights of the first sensor and the second sensor, so as to obtain the perception result.

13. A controller, comprising: At least one processor; as well as A memory coupled to the at least one processor and having instructions stored thereon, which, when executed by the at least one processor, cause the controller to perform the method according to any one of claims 1-11.

14. A computer-readable storage medium having stored thereon computer-executable instructions, wherein the computer-executable instructions are executed by a processor to implement the method according to any one of claims 1 to 11.

Images