Foggy day driving augmented reality auxiliary method and system based on multi-mode perception

Abstract

The invention discloses a foggy day driving augmented reality auxiliary method and system based on multi-mode perception, and relates to the technical field of intelligent driving. The method comprises the following steps: constructing a three-dimensional environment map to carry out real-time driving risk prediction, and carrying out preliminary adjustment on data source weight distribution in the three-dimensional environment map according to a set rule; the PPO reinforcement learning algorithm is adopted to readjust data source weight distribution, a new three-dimensional environment map is reconstructed based on the readjusted data source weight, and the risk probability, predicted in real time, of sudden deceleration of the front vehicle and the reconstructed three-dimensional environment map are projected to a windshield through a layered projection strategy to prompt a driver. According to the method, accurate and comprehensive multi-mode sensing data can be obtained in a foggy day environment, the accuracy of decision making and risk prediction is ensured, quick response can be made according to environment changes, the safety of a driver under complex weather conditions is improved, and the method can adapt to more changing environments in practical application.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent driving, and particularly to an augmented reality assistance method and system for foggy weather driving based on multi-modal perception. Background Art

[0002] In recent years, with the rapid development of intelligent transportation and autonomous driving technologies, foggy weather driving assistance systems, as an important part of improving driving safety, have received extensive attention. Foggy weather, as a typical adverse weather condition, a significant reduction in visibility brings great driving challenges to drivers. To address this issue, data acquisition and processing technologies based on sensors have developed rapidly. Traditional foggy weather driving assistance technologies mainly rely on a single sensor (such as radar or camera) to provide perception of the environment. The single perception method still has limitations in complex fog and haze weather. Therefore, the existing technologies generally face problems such as single perception information, insufficient perception accuracy, and poor environmental adaptability.

[0003] To solve these problems, multi-modal sensor fusion technology has emerged. This technology can more comprehensively and accurately perceive environmental changes by combining multiple sensor data (such as millimeter-wave radar, lidar, infrared sensors, etc.). However, although multi-modal perception technology has made certain progress, the existing technologies still face problems such as how to effectively fuse different data sources, how to optimize the weight allocation of data sources, and how to real-time predict driving risks. In particular, the real-time performance and accuracy in complex environments still need to be further improved. Summary of the Invention

[0004] In view of the problems existing in the existing augmented reality assistance method for foggy weather driving based on multi-modal perception, the present invention is proposed. Therefore, the problem to be solved by the present invention is how to provide an augmented reality assistance method and system for foggy weather driving based on multi-modal perception.

[0005] To solve the above technical problems, the present invention provides the following technical solutions: In a first aspect, the present invention provides an augmented reality assistance method for foggy weather driving based on multi-modal perception, which includes: Collecting environmental data of foggy weather driving through sensors and performing data processing to obtain multi-modal data; Based on the multi-modal data, performing dynamic scene modeling to construct a three-dimensional environmental map, performing real-time driving risk prediction, and obtaining the real-time visibility of the road during driving, and preliminarily adjusting the weight allocation of data sources in the three-dimensional environmental map according to set rules; The performing real-time driving risk prediction includes the following content: Obtaining exhaust gas analysis data in the constructed three-dimensional environmental map, calculating the risk probability, and predicting the risk probability of the vehicle in front suddenly decelerating. The calculation formula is: ; ; Wherein: is the risk probability of sudden deceleration of the vehicle ahead, is the diffusion speed of the exhaust gas of the vehicle ahead, is the real-time following distance of the current vehicle, is the relative speed between the current vehicle and the vehicle ahead, is the driving speed of the vehicle ahead, is the change in exhaust gas concentration, is the spatial distance of the change in exhaust gas concentration; The PPO reinforcement learning algorithm is used to readjust the weight distribution of the data source. The action space is used to represent the weight adjustment, and the clipped objective function is used to optimize the policy network. The policy network is continuously updated to optimize the weight distribution of the data source. A new three-dimensional environmental map is constructed based on the readjusted data source weights; The risk probability of sudden deceleration of the vehicle ahead predicted in real time and the reconstructed three-dimensional environmental map are projected onto the windshield using a hierarchical projection strategy to prompt the driver, completing the augmented reality assistance for foggy-day driving based on multi-modal perception.

[0006] As a preferred embodiment of the augmented reality assistance method for foggy-day driving based on multi-modal perception according to the present invention, wherein: the sensors include a millimeter-wave radar, an infrared thermal imager, an exhaust gas analysis sensor, and an anti-fog lidar; the environmental data includes radar data, infrared thermal imager data, and exhaust gas analysis sensor data, and the radar data includes millimeter-wave radar point cloud data and anti-fog lidar point cloud data.

[0007] As a preferred embodiment of the augmented reality assistance method for foggy-day driving based on multi-modal perception according to the present invention, wherein: the data processing includes the following: Perform spatio-temporal calibration on the environmental data collected by multiple sensors to eliminate the differences in the time and space coordinate systems of the environmental data. A unified coordinate system is established with the vehicle centroid as the origin to align the spatial positions of the sensors; Perform preprocessing on the environmental data. Density clustering is performed on the millimeter-wave radar point cloud data and the anti-fog lidar point cloud data. Core points, boundary points, and noise points are marked. Isolated points with a distance > 50m are removed as noise points. The missing area of the millimeter-wave radar point cloud data is filled with the lidar point cloud data to obtain the filled millimeter-wave radar point cloud data; Use Gaussian filtering to smooth the thermal imaging temperature matrix in the infrared thermal imager data, and calculate the temperature gradients in the horizontal and vertical directions. The areas with a gradient amplitude > 2 °C / pixel are retained as candidate organisms; Perform gas concentration mutation detection on the exhaust gas analysis sensor data. If the gas concentration mutates , it is marked as an exhaust gas release event, and the movement direction of the exhaust gas is initially judged; The multi-modal data includes the filled millimeter-wave radar point cloud data, the thermal imaging temperature matrix, the gas concentration, and the spatio-temporal calibration parameters.

[0008] As a preferred solution of the foggy weather driving augmented reality assistance method based on multi-modal perception according to the present invention, wherein: the construction of the three-dimensional environmental map includes the following contents: Convert the filled millimeter-wave radar point cloud data into a 3D grid map, perform grid division using a set resolution, and map the reflection information into grid cells according to the spatial distribution; Use a feature extraction algorithm to extract lane lines and boundaries from the 3D grid map, mark the road shoulders and static obstacles, and determine the lane width and traffic signs through road structure analysis to generate a three-dimensional road skeleton; Adopt the YOLOHSI model to fuse the thermal imaging temperature gradient and shape contour in the thermal imaging temperature matrix to identify organism information and obtain organism thermal imaging data; Based on the gas concentration, use a simplified CFD model to calculate the diffusion speed and direction of the exhaust gas to obtain exhaust gas analysis data; Construct a three-dimensional environmental map according to the three-dimensional road skeleton, organism thermal imaging data, and exhaust gas analysis data.

[0009] As a preferred solution of the foggy weather driving augmented reality assistance method based on multi-modal perception according to the present invention, wherein: the preliminary adjustment of the data source weight distribution in the three-dimensional environmental map according to the set rules includes: When the real-time visibility > 200m, adjust the weight of the radar data in the data source to 60%, the weight of the infrared thermal imager data to 20%, and the weight of the exhaust gas analysis sensor data to 20%; When 50m ≤ real-time visibility ≤ 200m, adjust the weight of the radar data in the data source to 45%, the weight of the infrared thermal imager data to 35%, and the weight of the exhaust gas analysis sensor data to 20%; When the real-time visibility < 50m, adjust the weight of the radar data in the data source to 30%, the weight of the infrared thermal imager data to 50%, and the weight of the exhaust gas analysis sensor data to 20%.

[0010] As a preferred solution of the foggy weather driving augmented reality assistance method based on multi-modal perception according to the present invention, wherein: the hierarchical projection strategy includes the following contents: Divide different types of information into multiple layers and project them onto different areas of the windshield respectively. The layers include a basic layer, a biological layer, a dynamic layer, and a risk layer; Utilize millimeter-wave radar point cloud data and anti-fog lidar point cloud data to obtain road boundary, lane line information, and traffic sign information in real time, convert the road boundary, lane line information, and traffic sign information into an AR image and project it onto the windshield, and automatically adjust the transparency according to the vehicle speed; Adjust the pulse frequency according to the target speed and approaching distance through the identified biological information, and display the biological contour through AR projection to prompt the driver of potential dangers; Indicate the diffusion direction and speed of the exhaust gas through an orange gradient arrow, and display it through AR projection to warn the driver of the risk probability of sudden deceleration of the vehicle ahead; Utilize a three-dimensional environmental map, combine historical accident data to mark the area, display the dangerous area through AR, and broadcast potential risks in real time; Through eye-tracking technology, detect the pupil position and line-of-sight direction of the driver in real time, ensure that the AR projection is aligned with the real scene, and perform dynamic compensation through the perspective transformation algorithm according to the driver's perspective.

[0011] As a preferred solution of the foggy weather driving augmented reality assistance method based on multi-modal perception described in the present invention, wherein: the re-adjustment of the data source weight allocation by using the PPO reinforcement learning algorithm includes the following contents: Collect and analyze historical false alarm data, including mis-identification situations under different weather conditions, and define the states of the state space It is expressed as: ; Wherein, is the real-time visibility, is the radar data weight, is the infrared thermal imager data weight, is the exhaust gas analysis sensor data weight, is the false alarm rate of the sensor at the current moment, is the missed alarm rate of the sensor at the current moment; Use the action space to represent the weight adjustment, and the action space is defined as: ; Wherein: is the new weight of the radar data, is the new weight of the infrared thermal imager data, is the new weight of the exhaust gas analysis sensor data; Define the reward function as: ; ; ; In the formula, is the detection accuracy of the sensor in the current state; is the number of times the sensor correctly identifies, is the number of times the sensor misidentifies; α is the weight of the accuracy reward, β is the penalty coefficient of the false alarm rate, γ is the penalty coefficient of the missed alarm rate, and δ is the reward adjustment coefficient of the visibility condition, all of which are positive values; is the visibility condition function, indicating the visibility level of the current environment; The system selects an action according to the current state, and then calculates the corresponding reward. The process of each round is as follows: Initialize the state ; According to the current policy network , select the action ; Update the system state according to the selected action ; Calculate the reward according to the current state and the executed action ; Store the current state, action, and reward information, and train the algorithm; The advantage function is used to represent the improvement or decrease of the return of an action relative to the average level. The policy update is guided by the advantage function. The advantage function is calculated as: ; Where: is the actually obtained reward, is the expected value of the state ; Use the clipped objective function to optimize the policy network. The clipped objective function is designed as: ; ; Where: is the expected operation, representing the average value of the expected value of the objective function in time steps; represents the probability distribution of the current policy selecting the action in the state , is the probability distribution of the old policy selecting the action in the state , represents the ratio of the probability of the current policy selecting the action to the probability of the old policy selecting the action in the state ; is a clipping operation used to limit the range of variation of which is a hyperparameter for controlling the update amplitude; By continuously updating the policy network the weight distribution of the data source is automatically optimized.

[0012] In a second aspect, the present invention provides a foggy weather driving augmented reality assistance system based on multimodal perception, which includes: An acquisition module for collecting environmental data of foggy weather driving through sensors and performing data processing to obtain multimodal data; A construction module for dynamically modeling the scene based on multimodal data to construct a three-dimensional environmental map, performing real-time driving risk prediction, and obtaining the real-time visibility of the road during driving, and preliminarily adjusting the weight distribution of the data source in the three-dimensional environmental map according to set rules; using the PPO reinforcement learning algorithm to perform secondary adjustment on the weight distribution of the data source, using the action space to represent the weight adjustment, using the clipping objective function to optimize the policy network, continuously updating the policy network, optimizing the weight distribution of the data source, and constructing a new three-dimensional environmental map based on the weight of the data source after the secondary adjustment; An assistance module for using a hierarchical projection strategy to project the risk probability of the sudden deceleration of the vehicle ahead in real-time prediction and the reconstructed three-dimensional environmental map onto the windshield to prompt the driver, completing the foggy weather driving augmented reality assistance based on multimodal perception.

[0013] In a third aspect, the present invention provides a computer device, including a memory and a processor, where the memory stores a computer program, and: when the processor executes the computer program, the steps of the foggy weather driving augmented reality assistance method based on multimodal perception are implemented.

[0014] In a fourth aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored, and: when the computer program is executed by a processor, the steps of the foggy weather driving augmented reality assistance method based on multimodal perception are implemented.

[0015] The beneficial effects of the present invention are: accurate and comprehensive multimodal perception data can be obtained in foggy weather conditions, ensuring the accuracy of decision-making and risk prediction, being able to make rapid responses according to environmental changes, improving the safety of drivers in complex weather conditions, and being able to adapt to more variable environments in practical applications. Description of the Drawings

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0017] Figure 1 It is a flowchart of an augmented reality assistance method for foggy weather driving based on multi-modal perception. Specific embodiments

[0018] To make the above objects, features, and advantages of the present invention more understandable, the following will describe the specific embodiments of the present invention in detail with reference to the drawings of the specification. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0019] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below.

[0020] Secondly, the so-called "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that can be included in at least one implementation manner of the present invention. The "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor is it an embodiment that is separate or selectively exclusive to other embodiments.

[0021] Refer to Figure 1 , for the first embodiment of the present invention, an augmented reality assistance method for foggy weather driving based on multi-modal perception is provided, including: S1: Collect environmental data of foggy weather driving through sensors and perform data processing to obtain multi-modal data; Specifically, the environmental data includes radar data, infrared thermal imager data, and exhaust gas analysis sensor data. The radar data includes millimeter-wave radar point cloud data and anti-fog lidar point cloud data. Deploy multiple sensors. Millimeter-wave radar (front / lateral): Using the 77GHz frequency band, covering a 360° horizontal field of view, generating high-density point clouds (resolution 0.1°×0.1°), focusing on capturing road boundaries and static obstacle contours. 3D point clouds (including distance, azimuth angle, and speed information), in the format of (x, y, z, v), with a refresh rate of 10Hz, used to construct the road skeleton and static obstacle contours.

[0022] Infrared thermal imager (8 - 14μm band): 14 - bit temperature matrix (resolution 640×480 pixels), frame rate 25fps, temperature range 20℃ - 150℃, accuracy ±0.05℃. Extract the contour and temperature gradient features of organisms (pedestrians, animals) to distinguish living and non - living targets.

[0023] Exhaust gas analysis sensor: Detect the CO / CO2 concentration gradient (ppm level), sampling rate 1kHz, detection sensitivity 1ppm. Combine with the exhaust gas diffusion model to infer the exhaust gas diffusion direction and movement trend of the vehicle ahead.

[0024] Anti - fog lidar (1550nm wavelength): High - density 3D point cloud (point cloud density > 200 points / square meter), anti - fog scattering interference, compensate for the point cloud loss of millimeter - wave radar in foggy days, and improve the obstacle edge detection accuracy.

[0025] Data processing includes the following: Eliminate the time difference and spatial coordinate system inconsistency problems of multi - sensor data to ensure the spatio - temporal consistency of data fusion.

[0026] Hardware - level time synchronization: Protocol selection: Adopt PTP (time protocol, IEEE1588), synchronize all sensor clocks through in - vehicle Ethernet, and select the in - vehicle central controller (such as domain controller) as the master clock.

[0027] Perform clock calibration: The master clock broadcasts synchronization messages (Sync) periodically and records the transmission time .

[0028] Record the reception time after receiving the message from the sensor , and reply with a delay request message containing and local time . The master clock calculates the clock offset and compensates it to the slave device to achieve synchronization.

[0029] Software - level time alignment: Interpolate the data of low - frequency sensors (such as gas sensors, 1Hz) to the high - frequency time axis (such as 10Hz); Maintain a circular buffer for each sensor and extract the aligned data frames after sorting by timestamp.

[0030] Spatial calibration: Establish a unified coordinate system with the vehicle's centroid as the origin, and align the spatial positions of sensors through the extrinsic parameter matrix (off - line calibration of the calibration board + online iterative optimization).

[0031] Off - line calibration (acquire initial extrinsic parameters): Calibration environment: In a fog - free and static environment, use a checkerboard calibration board (size 1m×1m, side length of black and white squares 10cm).

[0032] Millimeter-wave radar calibration: Place the calibration board 5m, 10m, and 15m in front of the vehicle, and the radar scans to obtain point cloud data. Extract the corner point coordinates of the calibration board (by fitting the point cloud plane) and match them with the true physical coordinates. Use the least squares method to solve the extrinsic parameter matrix (rotation matrix , translation vector ).

[0033] Infrared thermal imager calibration: Attach a heating sheet (temperature 50°C) to the surface of the calibration board, and the infrared camera captures the thermal image. Detect the corner points, and combine with the actual coordinates of the calibration board to calculate the camera internal parameters (focal length , , principal point , ) and extrinsic parameters ( , ).

[0034] Laser radar calibration: Use an anti-fog laser radar to scan the calibration board and extract the corner point coordinates. Align the laser radar with the vehicle coordinate system through the ICP (Iterative Closest Point) algorithm.

[0035] Online calibration (dynamic extrinsic parameter optimization): Real-time obtain the point cloud of the anti-fog laser radar (reference coordinate system) and the point cloud of the millimeter-wave radar. Perform voxel filtering on the two frames of point clouds (downsample to 0.1m resolution) to reduce the computational amount. Iteratively optimize the extrinsic parameter matrix through the GICP (Generalized ICP) algorithm. If the registration error < 0.05m, update the extrinsic parameter matrix; otherwise, trigger an alarm to indicate calibration failure.

[0036] Extrinsic parameter drift compensation: Perform online ICP calibration every 5 minutes to correct the extrinsic parameter offset caused by vehicle vibration or temperature deformation.

[0037] Coordinate system transformation: Convert the points in the sensor coordinate system to the vehicle coordinate system.

[0038] Data preprocessing, removing noise, compensating for missing data, and extracting effective features to provide input for multi-modal fusion.

[0039] Data preprocessing of millimeter-wave radar and laser radar. In foggy days, the point cloud is sparse and has a lot of noise, and it is necessary to fill the holes and remove the outlier points.

[0040] Point cloud denoising (adaptive DBSCAN): Perform density clustering on the point cloud to label the core points, boundary points, and noise points.

[0041] Parameter dynamic adjustment: Adjust the clustering radius according to the visibility (estimated by the backscattering intensity of the laser radar): ; Remove noise points (such as isolated points with a distance > 50m).

[0042] Point cloud compensation (assisted by anti-fog lidar): Superimpose the anti-fog lidar point cloud and the millimeter-wave radar point cloud, and fill the missing area of the millimeter-wave radar point cloud data (such as the holes caused by fog scattering) with the lidar point cloud.

[0043] Preprocessing of infrared thermal imager data, temperature gradient filtering: Gaussian filtering: Use a 5×5 Gaussian kernel (σ = 1.5) to smooth the temperature matrix; Calculate the horizontal temperature gradient and the vertical temperature gradient : ; ; Threshold segmentation: Retain the area with a gradient amplitude > 2°C / pixel as the candidate organism.

[0044] Preprocessing of gas sensor data, perform gas concentration mutation detection, and calculate the degree of gas concentration change: ; wherein, is the detected gas concentration, is the detection time, is the change time; Event marking: If , mark it as an exhaust gas release event.

[0045] Combine the vehicle speed provided by the IMU and the concentration gradient direction to calculate the exhaust gas movement direction , and the calculation formula is: ; If , determine that the exhaust gas comes from the vehicle in front; otherwise, it is environmental interference.

[0046] The multi-modal data includes the filled millimeter-wave radar point cloud data, the thermal imaging temperature matrix, the gas concentration, and the spatio-temporal calibration parameters.

[0047] S2: Based on the multi-modal data, perform dynamic scene modeling to construct a three-dimensional environmental map, perform real-time driving risk prediction, and obtain the real-time visibility of the road during driving, and make a preliminary adjustment to the data source weight allocation in the three-dimensional environmental map according to the set rules; Specifically, collect the reflected signals in the environment through the in-vehicle millimeter-wave radar to generate point cloud data.

[0048] Convert the filled millimeter-wave radar point cloud data into a 3D grid map. Use a grid division with a resolution of 0.2m, and map the reflection information in the radar point cloud to the corresponding grid cells according to the spatial distribution.

[0049] Using feature extraction algorithms, extract lane lines and their boundaries from the 3D grid map, and label the road shoulders and static obstacles. Through road structure analysis, determine key information such as lane width and traffic signs, and generate a three-dimensional road skeleton model to assist subsequent risk assessment and prediction.

[0050] The YOLOHSI model fuses the thermal imaging temperature gradient (such as the human body feature of 37°C ± 2°C) and the shape contour (OpenPose skeletal key points) in the thermal imaging temperature matrix to distinguish organisms such as pedestrians.

[0051] Collect thermal imaging images of the environment through an infrared camera, focusing on organism features such as the human body temperature range (37°C ± 2°C).

[0052] Combine the temperature information in the thermal imaging image with the shape contour (such as human skeleton key points) extracted by the OpenPose algorithm to distinguish different organisms (such as pedestrians, animals, etc.) and obtain organism thermal imaging data.

[0053] Use the YOLOHSI model to perform real-time recognition and tracking of heat sources and organisms in the scene.

[0054] Based on the CFD simplified model, calculate the diffusion direction and velocity of the exhaust plume through the CO concentration gradient, generate motion vector arrows, predict the probability of sudden deceleration of the vehicle in front, and obtain the exhaust concentration data at the rear of the vehicle through in-vehicle sensors.

[0055] Based on the CO concentration data, use the simplified CFD model to calculate the diffusion speed and direction of the exhaust gas.

[0056] By fusing the generated road skeleton, organism thermal imaging data, exhaust gas analysis data, and lane information, construct a three-dimensional environment map.

[0057] This map includes road structures, dynamic obstacles (such as pedestrians, animals), and real-time exhaust gas diffusion conditions. Use 3D modeling software (such as Autodesk Revit, Blender, etc.) or geographic information systems (GIS) to create a three-dimensional environment map.

[0058] The environmental map needs to be continuously updated according to real-time data. For example, when new organisms enter the scene, the organism recognition layer will be automatically updated, and the exhaust gas diffusion model will also be dynamically adjusted according to the real-time vehicle state and environmental conditions. Use GPU acceleration technology to dynamically render the three-dimensional map through real-time updated sensor data (such as temperature, concentration, speed, etc.) to ensure that the displayed information is consistent with the current scene.

[0059] Obtain the real-time road visibility data during driving, and perform a preliminary adjustment of the data source weights. The set rules are as follows: When the real-time visibility > 200m, adjust the weight of radar data in the data source to 60%, the weight of infrared thermal imager data to 20%, and the weight of exhaust gas analysis sensor data to 20%; radar is dominant and focuses on road structure modeling.

[0060] When 50m ≤ real-time visibility ≤ 200m, adjust the weight of radar data in the data source to 45%, the weight of infrared thermal imager data to 35%, and the weight of exhaust gas analysis sensor data to 20%; enhance biological detection.

[0061] When the real-time visibility < 50m, adjust the weight of radar data in the data source to 30%, the weight of infrared thermal imager data to 50%, and the weight of exhaust gas analysis sensor data to 20%. Combine exhaust gas tracking to avoid misjudgment of heat sources (such as distinguishing vehicle exhaust from animals).

[0062] Use the CFD model or fluid dynamics model to calculate the spatial distribution of exhaust gas concentration, and then calculate the concentration gradient. Based on the concentration gradient, calculate the exhaust gas diffusion speed; According to the calculated exhaust gas diffusion speed, the real-time following distance, and the driving state of the leading vehicle, calculate the risk probability and predict the probability of the leading vehicle suddenly decelerating. The calculation formula is: ; ; Where: : The risk probability of sudden deceleration (between 0 and 1, the closer the value is to 1, the greater the risk), : The exhaust gas diffusion speed of the leading vehicle, indicating the impact of the exhaust gas on the leading vehicle. : The real-time following distance of the current vehicle, in meters (m), : The relative speed between the current vehicle and the leading vehicle, in meters per second (m / s). : The driving speed of the leading vehicle, dynamically obtained according to the actual vehicle speed. is the change in exhaust gas concentration (unit: ppm). is the spatial distance of the change in exhaust gas concentration (unit: m).

[0063] S3: Use the PPO reinforcement learning algorithm to re-adjust the data source weight allocation. Use the action space to represent the weight adjustment, use the clipped objective function to optimize the policy network, continuously update the policy network, optimize the data source weight allocation, and construct a new three-dimensional environmental map based on the re-adjusted data source weights; The PPO reinforcement learning algorithm is used to readjust the weight allocation of data sources. The PPO (Proximal Policy Optimization) reinforcement learning algorithm is adopted to dynamically optimize the weight policy according to historical false alarm data, enhancing the robustness in extreme foggy weather.

[0064] Collect and analyze historical false alarm data, including misidentification situations under different weather conditions. Through the PPO algorithm, optimize the weight allocation policy based on historical false alarm data, enabling the system to have higher robustness in extreme haze weather, avoiding false alarms and missed detections. The system gradually adjusts the weight allocation of sensors according to real-time data feedback to ensure accuracy in a dynamically changing environment.

[0065] The policy network is a neural network in the PPO algorithm used to select actions. By inputting the current state, it outputs the probability distribution of each action, and based on the current state, determines what action the system should take (i.e., adjust the weight of the data source).

[0066] Current policy network: State space: The state space includes real-time visibility, current weight allocation, false alarm rate, and missed detection rate.

[0067] Action space: The action space refers to the adjustment of sensor weights. The goal of the policy network is to output a new weight allocation (i.e., the adjustment of the data source weight) given the current state (such as visibility, false alarm rate, etc.).

[0068] Output: The output of the policy network is the probability or probability distribution corresponding to each action (the weight of each data source).

[0069] During the training process, PPO selects actions through the current policy network. The policy network is a neural network, usually a deep neural network composed of a fully connected layer (or convolutional layer, depending on the complexity of the problem). The policy network outputs the probability distribution of actions based on the input current state.

[0070] Old policy network: Update the previous policy network (i.e., the previous round of policy) during the previous training process. The key of the PPO algorithm is to use the ratio between the old policy and the current policy to prevent the policy from collapsing due to overly fast updates.

[0071] The old policy network is used to calculate the ratio between the current policy and the old policy to ensure the stability of the update. At the end of each training cycle, the old policy network will be replaced by the current policy network. Specifically, assume that at a certain moment t, the parameters of the policy network are , and at the next moment t + 1, the parameters of the policy network are θ, then is the policy network obtained in the previous round of training.

[0072] Use the PPO (Proximal Policy Optimization) algorithm to optimize the weight allocation of sensors. The state space represents the state of the environment and is used to describe the situation of the system at a certain moment.

[0073] Definition of the state space Is represented as a vector containing the following elements: ; Where: Is the real-time visibility, representing the visibility of the current environment (unit: meters). Is the weight of radar data, representing the current weight allocation of radar data. : Is the weight of infrared thermal imager data, representing the current weight allocation of infrared thermal imager data. Is the weight of exhaust gas analysis sensor data, representing the current weight allocation of exhaust gas analysis sensor data. Is the false alarm rate, representing the false alarm rate of the sensor at the current moment. Is the miss rate, representing the miss rate of the sensor at the current moment.

[0074] In this state space, visibility and the weight configuration of sensors affect the performance of the system and need to be passed as part of the state to the PPO algorithm for learning.

[0075] Definition of the action space In reinforcement learning, the action space represents all possible actions that an agent can take. In this problem, the action space represents the adjustment of sensor weights. Since the sensor weights are continuous (between 0 and 1), a continuous action space can be used to represent the weight adjustment.

[0076] Action space Is defined as: ; Where: Is the new weight of radar data, Is the new weight of infrared thermal imager data, Is the new weight of exhaust gas analysis sensor data; Definition of the reward function Reward function It is the core in reinforcement learning, representing the immediate reward obtained by the agent after taking a certain action in the current state. When optimizing the weight allocation, the reward should reflect the detection ability of the system and the performance of the sensors. The reward is usually based on the following aspects: Detection accuracy: If the sensor correctly identifies the target, a positive reward is given (such as pedestrians, obstacles, etc.). False alarm rate: If the sensor gives a false alarm (such as misidentifying a background object as a living organism), a negative reward is given. Miss rate: If the sensor misses the target, a negative reward is given. Visibility conditions: In the case of low visibility, the system gives more rewards to the weights of infrared sensors to better detect targets in low visibility environments.

[0077] Define the reward function as: ; ; In the formula, α is the weight of the accuracy reward, measuring the degree of reward when the sensor correctly identifies the target. Usually α is a positive value, encouraging the system to obtain rewards when correctly detecting. is the detection accuracy rate of the sensor in the current state, representing the proportion of targets successfully identified by the system. is the number of times the sensor correctly identifies, is the number of times the sensor misidentifies; β is the penalty coefficient of the false alarm rate, representing the degree of penalty when the system gives a false alarm. Usually β is a positive value, and false alarms will result in negative rewards. is the false alarm rate, representing the proportion of the system misjudging the background as the target. γ is the penalty coefficient of the miss rate, representing the degree of penalty when the system misses the target, which is a positive value. is the miss rate, representing the proportion of the system failing to detect the target. δ is the reward adjustment coefficient of the visibility condition, representing the weight adjustment under different visibilities. is the visibility condition function, representing the visibility level of the current environment: ; This value is proportional to the visibility. The lower the visibility, the corresponding increase in the weights of infrared sensors and exhaust analysis data.

[0078] Collect data. In each round of interaction, the system will select an action (adjust the weight) according to the current state, and then calculate the corresponding reward.

[0079] The process of each round is as follows: Environment initialization: Initialize the state .

[0080] Select an action: According to the current policy , select the action .

[0081] Update Status: Update the system status according to the selected action .

[0082] Calculate Reward: Calculate the reward according to the current state and the executed action .

[0083] Store Experience: Store information such as the current state, action, and reward for training.

[0084] The advantage function is used to measure the quality of taking a certain action in a given state, indicating the improvement or decline of the return of a certain action relative to the average level. The policy is updated under the guidance of the advantage function. The advantage function The calculation formula is: ; where: is the actually obtained reward. is the expected value of state predicted by the value network.

[0085] Update Policy (PPO Algorithm) PPO uses a clipped objective function to optimize the policy network to prevent excessive policy updates. The objective function is designed as: ; ; where: is the expected operation, representing the average value of the expected value of the objective function in time steps; is the current policy network, representing the probability distribution of selecting action in state . is the old policy network. is the ratio of the current policy to the old policy, representing the ratio of the probability of the current policy selecting action in state to the probability of the old policy selecting action ; is a clipping operation used to limit the range of change to prevent excessive policy updates, is a hyperparameter used to control the update amplitude, usually taking values between 0.1 and 0.3. By continuously updating the policy network , the PPO algorithm enables the system to automatically optimize the data source weight allocation, improve the detection accuracy, and reduce false positives and false negatives.

[0086] In real-time updates and online learning, in practical applications, the PPO algorithm conducts online learning and continuously updates the weights of data sources. After each round of feedback, the system continues to optimize the weights based on new data to ensure accurate perception results can be provided under various visibility and weather conditions.

[0087] S4: Use a hierarchical projection strategy to project the risk probability of the sudden deceleration of the vehicle ahead predicted in real-time and the reconstructed three-dimensional environmental map onto the windshield to prompt the driver, completing the augmented reality assistance for foggy-day driving based on multi-modal perception.

[0088] Specifically, the hierarchical projection strategy mainly divides different types of information (such as road structure, organism detection, exhaust diffusion, historical accidents, etc.) into multiple layers and projects them onto different areas of the windshield respectively.

[0089] The transparency, display method, and interaction method of each layer will be dynamically adjusted according to different environments and driver needs.

[0090] Function of the basic layer (blue semi-transparent): Dynamically display road boundaries, lane guiding lines, and traffic signs to ensure that the driver clearly knows their driving path. Transparency adjustment: The transparency is automatically adjusted according to the vehicle speed to reduce excessive visual interference during high-speed driving. For example, when the vehicle speed exceeds 80 km / h, the transparency is reduced to 30%, making the information clearer and not affecting the driver's vision. When driving at low speed, the transparency increases to ensure that the road guiding lines and obstacles are clearly visible.

[0091] Utilize millimeter-wave radar point cloud data and anti-fog lidar point cloud data to obtain information such as road boundaries and lane lines in real-time. Convert this information into an AR image and perform dynamic adjustment, then project it into the windshield.

[0092] Function of the organism layer (red pulse): Highlight the identified organisms (such as pedestrians, animals, etc.) through red pulses to remind the driver of potential collision risks. Pulse frequency: The pulse frequency (13 Hz) reflects the danger level of the target. For example, if the target is approaching at high speed, the pulse frequency will increase (such as 3 Hz indicating that the target is in a high-risk approaching state).

[0093] By combining a thermal imaging sensor and a deep learning model (such as the YOLOHSI model), identify organisms (such as pedestrians, animals, cyclists, etc.) in the scene in real-time.

[0094] Dynamically adjust the pulse frequency according to the speed and proximity of the target, and display the organism contour through AR projection to prompt the driver of potential dangers.

[0095] Function of the dynamic layer (orange gradient arrow): The arrow indicates the diffusion direction and speed of the exhaust gas, warning the driver of the possibility of sudden deceleration of the vehicle ahead. The length of the arrow is positively correlated with the CO concentration gradient: the diffusion direction and speed of the exhaust gas are obtained by calculating the CO concentration gradient at the rear of the vehicle. The length of the arrow is adjusted according to the intensity of the concentration gradient. The higher the concentration, the longer the arrow, indicating a greater risk of sudden deceleration of the vehicle ahead.

[0096] Use a CO sensor to monitor the concentration change of the gas at the rear of the vehicle. Based on the CFD simplified model, calculate the diffusion direction and speed of the exhaust gas. Display the arrow and warning information through AR projection to warn the driver of the possible sudden deceleration of the vehicle ahead in a timely manner.

[0097] Function of the risk layer (black grid): Display the historical accident areas, reminding the driver of entering potential high-risk areas. Voice prompt: Combine voice prompts (such as "dangerous curve ahead") to improve the efficiency and accuracy of information transmission, especially in complex driving environments.

[0098] Utilize high-precision map data and combine historical accident data to mark high-risk areas (such as sharp curves, accident-prone points, etc.). Display these dangerous areas through AR and real-time broadcast potential risks through voice synthesis technology.

[0099] Install an infrared camera in the vehicle to capture the movement of the driver's eyes, use eye-tracking algorithms to determine the driver's current fixation point and line of sight direction, and the system makes dynamic compensation according to the driver's perspective.

[0100] The perspective transformation algorithm adjusts the projection angle and position in real time according to the driver's line of sight and sitting posture in the vehicle to ensure that the projection position is consistent with the driver's line of sight.

[0101] According to the speed and direction of the vehicle's travel, as well as the driver's head movement, adjust the presentation method of AR information in real time to avoid unnatural visual effects.

[0102] Furthermore, this embodiment also provides an augmented reality assisted system for foggy day driving based on multimodal perception, including: An acquisition module, configured to collect environmental data of foggy day driving through sensors and perform data processing to obtain multimodal data; A construction module, configured to perform dynamic scene modeling based on multimodal data to construct a three-dimensional environmental map, perform real-time driving risk prediction, and obtain the real-time visibility of the road during driving. Make a preliminary adjustment to the data source weight allocation in the three-dimensional environmental map according to the set rules; use the PPO reinforcement learning algorithm to make a second adjustment to the data source weight allocation, use the action space to represent the weight adjustment, use the clipped objective function to optimize the policy network, continuously update the policy network, optimize the data source weight allocation, and construct a new three-dimensional environmental map based on the data source weight after the second adjustment; An auxiliary module is used to project the risk probability of the sudden deceleration of the vehicle ahead predicted in real time and the reconstructed three-dimensional environmental map onto the windshield by using a hierarchical projection strategy to prompt the driver, so as to complete the augmented reality assistance for foggy-day driving based on multi-modal perception.

[0103] This embodiment also provides a computer device applicable to the situation of the augmented reality assistance method for foggy-day driving based on multi-modal perception, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement all or part of the steps of the method described in the embodiments of the present invention as proposed in the above embodiments.

[0104] This embodiment also provides a storage medium, on which a computer program is stored. When the computer program is executed by a processor, it executes the method in any optional implementation manner of the above embodiments. Among them, the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (abbreviated as SRAM), electrically erasable programmable read-only memory (abbreviated as EEPROM), erasable programmable read-only memory (abbreviated as EPROM), programmable read-only memory (abbreviated as PROM), read-only memory (abbreviated as ROM), magnetic memory, flash memory, magnetic disk or optical disc.

[0105] The storage medium proposed in this embodiment and the data storage method proposed in the above embodiments belong to the same inventive concept. The technical details not described in detail in this embodiment can be referred to the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0106] In summary, this method can obtain accurate and comprehensive multi-modal perception data in a foggy environment, ensure the accuracy of decision-making and risk prediction, can make a quick response according to the changes in the environment, improve the safety of drivers under complex weather conditions, and can adapt to more changing environments in practical applications.

[0107] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not restrictive. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered within the scope of the claims of the present invention.

Claims

1. An augmented reality assisted method for driving in foggy weather based on multimodal perception, characterized by: include: Collect environmental data of driving in foggy weather through sensors and process the data to obtain multimodal data; Based on multimodal data, dynamic scene modeling is performed to build a three-dimensional environmental map, perform real-time driving risk prediction, and obtain real-time road visibility while driving. The weight distribution of data sources in the three-dimensional environmental map is initially adjusted according to the set rules; The real-time driving risk prediction includes the following contents: Obtain the exhaust gas analysis data in the constructed three-dimensional environmental map, perform risk probability calculation, and predict the risk probability of sudden deceleration of the preceding vehicle. The calculation formula is: ； ； in: is the risk probability of the preceding vehicle suddenly decelerating, is the diffusion speed of the exhaust gas of the preceding vehicle, is the real-time following distance of the current vehicle, is the relative speed between the current vehicle and the preceding vehicle, is the speed of the front vehicle, is the change of exhaust gas concentration, is the spatial distance over which the exhaust gas concentration changes; The PPO reinforcement learning algorithm is used to readjust the weight distribution of data sources, the action space is used to represent the weight adjustment, the clipping objective function is used to optimize the policy network, the policy network is continuously updated, the data source weight distribution is optimized, and a new 3D environment map is reconstructed based on the readjusted data source weights. A layered projection strategy is used to project the real-time predicted risk probability of sudden deceleration of the preceding vehicle and the reconstructed three-dimensional environment map onto the windshield to prompt the driver, thus completing augmented reality assistance for driving in foggy weather based on multimodal perception.

2. The method for augmented reality assistance for driving in foggy weather based on multimodal perception as claimed in claim 1, characterized in that: The sensors include millimeter wave radar, infrared thermal imager, exhaust gas analysis sensor and anti-fog laser radar; the environmental data include radar data, infrared thermal imager data and exhaust gas analysis sensor data, and the radar data include millimeter wave radar point cloud data and anti-fog laser radar point cloud data.

3. The method for augmented reality assistance for driving in foggy weather based on multimodal perception as claimed in claim 2, characterized in that: The data processing includes the following contents: Perform spatiotemporal calibration on the environmental data collected by multiple sensors for foggy driving, eliminate the time and space coordinate system differences of the environmental data, establish a unified coordinate system with the vehicle's center of mass as the origin, and align the spatial positions of the sensors; Perform environmental data preprocessing, perform density clustering on millimeter-wave radar point cloud data and anti-fog lidar point cloud data, mark core points, boundary points and noise points, remove isolated points with a distance > 50m as noise points, and use lidar point cloud data to fill in the missing areas of millimeter-wave radar point cloud data to obtain the filled millimeter-wave radar point cloud data; The thermal imaging temperature matrix in the infrared thermal imager data was smoothed using Gaussian filtering, and the temperature gradients in the horizontal and vertical directions were calculated, and the areas with gradient amplitude > 2°C / pixel were retained as candidate organisms; Perform gas concentration mutation detection on the exhaust gas analysis sensor data. If the gas concentration suddenly changes , it is marked as an exhaust gas release event, and the direction of exhaust gas movement is preliminarily determined; The multimodal data includes padded millimeter-wave radar point cloud data, thermal imaging temperature matrix, gas concentration, and spatiotemporal calibration parameters.

4. The method for augmented reality assistance for driving in foggy weather based on multimodal perception as claimed in claim 3, characterized in that: The construction of the three-dimensional environment map includes the following contents: The filled millimeter-wave radar point cloud data is converted into a 3D grid map, and the grid is divided using a set resolution, and the reflection information is mapped to the grid cells according to the spatial distribution; The feature extraction algorithm is used to extract lane lines and boundaries from the 3D grid map, and the shoulders and static obstacles are marked. Through road structure analysis, the lane width and traffic signs are determined to generate a 3D road skeleton. The YOLOHSI model is used to fuse the thermal imaging temperature gradient and shape contour in the thermal imaging temperature matrix to identify the biological information and obtain the biological thermal imaging data; Based on the gas concentration, a simplified CFD model is used to calculate the diffusion speed and direction of the exhaust gas to obtain the exhaust gas analysis data; A three-dimensional environmental map is constructed based on the three-dimensional road skeleton, biological thermal imaging data and exhaust gas analysis data.

5. The method for augmented reality assistance for driving in foggy weather based on multimodal perception as claimed in claim 4, characterized in that: The preliminary adjustment of the data source weight distribution in the three-dimensional environment map according to the set rules includes: When the real-time visibility is greater than 200m, the weight of radar data in the data source is adjusted to 60%, the weight of infrared thermal imager data is adjusted to 20%, and the weight of exhaust gas analysis sensor data is adjusted to 20%; When 50m≤real-time visibility≤200m, adjust the weight of radar data in the data source to 45%, the weight of infrared thermal imager data to 35%, and the weight of exhaust gas analysis sensor data to 20%; When the real-time visibility is less than 50m, the weight of radar data in the data source is adjusted to 30%, the weight of infrared thermal imager data is adjusted to 50%, and the weight of exhaust gas analysis sensor data is adjusted to 20%.

6. The method for augmented reality assistance for driving in foggy weather based on multimodal perception as claimed in claim 5, characterized in that: The hierarchical projection strategy includes the following: Different types of information are divided into multiple layers and projected onto different areas of the windshield. The layers include a base layer, a biological layer, a dynamic layer, and a risk layer. Using millimeter-wave radar point cloud data and anti-fog lidar point cloud data, the system obtains road boundaries, lane line information, and traffic sign information in real time, converts the road boundaries, lane line information, and traffic sign information into AR images and projects them into the windshield, automatically adjusting the transparency according to the vehicle speed; By identifying biological information, the pulse frequency is adjusted according to the target speed and approach distance, and the outline of the organism is displayed through AR projection to remind the driver of potential dangers; The diffusion direction and speed of exhaust gas are indicated by orange gradient arrows, and displayed through AR projection to warn the driver of the risk probability of sudden deceleration of the vehicle ahead. Using 3D environmental maps and combining historical accident data to mark areas, AR can be used to display dangerous areas and report potential risks in real time. Through eye tracking technology, the driver's pupil position and line of sight direction are detected in real time to ensure that the AR projection is aligned with the real scene, and dynamic compensation is performed through a perspective transformation algorithm based on the driver's perspective.

7. The method for augmented reality assistance for driving in foggy weather based on multimodal perception as claimed in claim 6, characterized in that: The use of the PPO reinforcement learning algorithm to readjust the data source weight distribution includes the following contents: Collect and analyze historical false alarm data, including false positives under different weather conditions, and define states in the state space It is expressed as: ； in, For real-time visibility, is the radar data weight, is the infrared thermal imager data weight, is the exhaust gas analysis sensor data weight, is the false alarm rate of the sensor at the current moment, is the false alarm rate of the sensor at the current moment; Use action space to represent weight adjustment, action space Defined as: ； in: is the new weight of the radar data, is the new weight of the infrared thermal imager data, New weightings for exhaust gas analysis sensor data; Defining the reward function for: ； ； ； In the formula, is the detection accuracy of the sensor in the current state; is the number of times the sensor correctly identifies is the number of times the sensor misidentifies; α is the weight of the accuracy reward, β is the penalty coefficient for the false alarm rate, γ is the penalty coefficient for the missed alarm rate, and δ is the reward adjustment coefficient for the visibility condition, all of which are positive values; is the visibility condition function, which indicates the visibility level of the current environment; The system selects actions based on the current state and then calculates the corresponding rewards. The process of each round is as follows: Initialization state ; According to the current strategy network , select action ; Updates the system status based on the selected action ; Calculate the reward based on the current state and the action performed ; Store current state, action and reward information, and train the algorithm; The advantage function is used to indicate the increase or decrease of the return of an action relative to the average level. The advantage function is used to guide the update of the strategy. The calculation formula is: ； in: is the actual reward received, Yes Status Expected value of The policy network is optimized using a clipping objective function, which Designed for: ； ； in: is the expected operation, indicating that The average value of the expected value of the objective function in time steps; Indicates that the current policy is in state Next select action The probability distribution of Is the old policy in state Next select action The probability distribution of Indicates in status Next, the current strategy selects action The probability of choosing an action with the old strategy The ratio of the probabilities of is a shear operation, used to limit The range of changes, is a hyperparameter used to control the update amplitude; By continuously updating the policy network , automatically optimize the data source weight distribution.

8. An augmented reality assistance system for driving in fog based on multimodal perception, based on the augmented reality assistance method for driving in fog based on multimodal perception according to any one of claims 1 to 7, characterized in that: include: An acquisition module is used to collect environmental data of driving in foggy weather through sensors and process the data to obtain multimodal data; A construction module is used to construct a three-dimensional environment map based on dynamic scene modeling based on multimodal data, perform real-time driving risk prediction, and obtain real-time visibility of the road while driving. The data source weight distribution in the three-dimensional environment map is initially adjusted according to the set rules; the PPO reinforcement learning algorithm is used to adjust the data source weight distribution again, the action space is used to represent the weight adjustment, the clipping objective function is used to optimize the policy network, the policy network is continuously updated, the data source weight distribution is optimized, and a new three-dimensional environment map is reconstructed based on the re-adjusted data source weight; The auxiliary module is used to use a layered projection strategy to project the real-time predicted risk probability of sudden deceleration of the preceding vehicle and the reconstructed three-dimensional environment map onto the windshield to prompt the driver, thereby completing augmented reality assistance for driving in foggy weather based on multimodal perception.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the augmented reality assistance method for foggy driving based on multimodal perception according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the augmented reality assistance method for foggy driving based on multimodal perception according to any one of claims 1 to 7 are implemented.

Images