Multimodal perception driving risk decision-making and interaction methods, devices, vehicles and media

Abstract

This invention primarily applies to the field of automotive engineering technology. It discloses a multimodal perception driving risk decision-making and interaction method, device, vehicle, and medium. The method includes: collecting driver biometric and operational behavior data, as well as environmental data of the vehicle's location; predicting driving intentions and generating intention features based on the biometric and operational behavior data; generating a spatiotemporal scene map based on the environmental data and determining the risk level of dynamic targets within it; fusing the intention features and the spatiotemporal scene map across modalities to obtain fused features; generating a dynamic risk region mapping risk level to spatial range based on the fused features, and generating a planned driving path within this region; finally, generating an augmented reality interactive screen to display the dynamic risk region and the planned path to the driver. This application significantly improves the decision-making accuracy, human-machine collaboration naturalness, and adaptability to long-tail risk scenarios in driving safety assistance systems.

Description

Technical Field

[0001] This invention relates to the field of automotive engineering technology, specifically to a multimodal perception driving risk decision-making and interaction method, device, vehicle, and medium. Background Technology

[0002] In recent years, advanced driver assistance systems (ADAS) have been rapidly adopted in the automotive industry. Perception technologies based on sensors such as cameras, millimeter-wave radar, and lidar have become relatively mature, and functions such as adaptive cruise control, lane keeping assist, and automatic emergency braking have significantly improved driving safety and comfort in structured scenarios such as highways. At the same time, driver monitoring technologies have also made significant progress. By analyzing facial expressions, gaze direction, and gestures, the system can initially identify behaviors such as fatigued driving or distracted driving.

[0003] However, existing advanced driver assistance systems (ADAS) generally employ a technical architecture that combines environmental perception with rule-based decision-making. Their core logic primarily focuses on detecting and responding to explicit targets such as static obstacles, lane lines, and traffic signs. When faced with complex and dynamic traffic scenarios, such as pedestrians suddenly turning back, adjacent vehicles cutting in, or non-motorized vehicles weaving through traffic, these events exhibit high uncertainty and long-tail distribution characteristics. Existing systems often struggle to deeply model the evolution of these scenarios, resulting in sluggish or overly conservative responses. More importantly, current systems treat the driver as a passive monitor or rely solely on low-dimensional signals such as steering wheel torque and pedal force to determine the driver's intention to take over. They fail to deeply integrate and understand the driver's implicit intentions, such as slowing down to yield, accelerating to overtake, or maneuvering to avoid obstacles, with changes in the dynamic environment.

[0004] This unidirectional chain of perception, decision-making, and execution leads to two core flaws. First, decision-making is inaccurate; the system cannot predict the driver's subjective inclinations, frequently resulting in braking or steering interventions contrary to the driver's expectations, thus causing human-machine conflict and a crisis of trust. Second, interaction is unintuitive; the system can only transmit information through audible and visual warnings or brief vibrations, lacking the ability to proactively adapt to and collaboratively interpret the driver's intentions. These flaws significantly increase the rates of false alarms, missed alarms, and erroneous actions in long-tail risk scenarios such as rain and snow, temporary construction, and irregularly shaped vehicles, making it difficult to truly support higher levels of driver assistance safety requirements. Therefore, achieving a deep integration and understanding of driver intentions and complex dynamic scenarios has become a key issue in overcoming existing technological bottlenecks. Summary of the Invention

[0005] This invention provides a multimodal perception driving risk decision-making and interaction method, device, vehicle and medium, which can achieve deep integration and understanding of driver intentions and complex dynamic scenarios, significantly improving the decision-making accuracy, human-machine collaboration and adaptability to long-tail risk scenarios of driving safety assistance systems.

[0006] This invention provides a multimodal perception driving risk decision-making and interaction method, the method comprising: Collect biometric and operational behavior data of the driver in the target vehicle, as well as environmental data of the location of the target vehicle; Based on the biometric and operational behavior data, predict the driver's driving intention of the target vehicle and generate driver intention features; Based on the environmental data, a spatiotemporal scene map is generated, and the risk level of dynamic targets in the spatiotemporal scene map is determined. The driver's intent features are fused with the spatiotemporal scene map across modalities to obtain fused features; Based on the fusion features, a dynamic risk region with a mapping relationship between risk level and spatial range is generated, and a planned driving path for the target vehicle is generated within the dynamic risk region. An augmented reality interactive screen is generated, and the dynamic risk area and the planned driving route are displayed to the driver of the target vehicle through the augmented reality interactive screen.

[0007] Optionally, generating a spatiotemporal scene map based on the environmental data and determining the risk level of dynamic targets in the spatiotemporal scene map includes: Based on the environmental data, the motion features of dynamic targets and the static road structure features are extracted; The motion characteristics are used to predict the motion trajectory of the dynamic target, and the predicted trajectory of the dynamic target is obtained. The cognitive uncertainty of the motion trajectory prediction is calculated. The predicted trajectory, the cognitive uncertainty, and the static road structure features are spatiotemporally aligned to construct the spatiotemporal scene graph, wherein the attributes of the nodes corresponding to the dynamic targets in the spatiotemporal scene graph include the predicted trajectory and the cognitive uncertainty. Based on the cognitive uncertainty, the risk level of the dynamic target is determined, wherein the risk level is positively correlated with the cognitive uncertainty.

[0008] Optionally, the step of fusing the driver's intent features with the spatiotemporal scene map across modalities to obtain fused features includes: The driver's driving style characteristics are extracted from the biometric and operational behavior data; The driver intent feature and the driving style feature are concatenated to obtain the query vector; For each node in the spatiotemporal scene graph, the node features of the node are mapped to a key vector using a first parameter matrix, and the node features of the node are mapped to a value vector using a second parameter matrix. Calculate the dot product similarity between the query vector and each of the key vectors, and normalize the dot product similarity of all the nodes in the spatiotemporal scene graph to obtain the attention weight corresponding to each node; The value vectors of all the nodes are weighted and summed according to the attention weights to obtain the driver-centric scene and intent fusion feature.

[0009] Optionally, the multimodal perception driving risk decision-making and interaction method further includes: The environmental data is spatiotemporally aligned with the driver's intention features to obtain an aligned feature vector. The aligned feature vectors are input into a preset risk decision-making model. Based on the risk entropy calculation theory, the risk decision-making model determines whether there is a potential collision risk in the current scenario. If so, a risk level is generated as the degree of risk, wherein the risk level is a low risk level, a medium risk level, or a high risk level.

[0010] Optionally, the multimodal perception driving risk decision-making and interaction method further includes: When the risk level is medium risk or high risk, the dynamic target in the spatiotemporal scene graph that causes the potential collision risk is taken as the risk source, and the location of the risk source in the real physical space is determined. The augmented reality head-up display device is controlled to project a warning box and an escape path guide line at the real physical location corresponding to the risk source, so as to guide the driver's line of sight to the risk source and prompt the avoidance path.

[0011] Optionally, the multimodal perception driving risk decision-making and interaction method further includes: When the risk level is high and the driver's intention characteristics determine that the driver has failed to avoid the potential collision risk, an active obstacle avoidance control command is generated. The active obstacle avoidance control command is sent to the drive-by-wire chassis system of the target vehicle to execute obstacle avoidance actions such as drive-by-wire braking or drive-by-wire steering.

[0012] Optionally, after generating the dynamic risk region, the multimodal perception driving risk decision-making and interaction method further includes: The risk level of the dynamic target in the spatiotemporal scene map is compared with a preset threshold. When the risk level is greater than the preset threshold, the risk area is increased along the predicted trajectory direction and the orthogonal direction of the dynamic target.

[0013] The present invention also provides a multimodal perception driving risk decision-making and interaction device, the device comprising: The data acquisition module is used to collect the biometrics and operational behavior data of the driver in the target vehicle, as well as the environmental data of the location of the target vehicle. The identification module is used to predict the driving intention of the driver of the target vehicle and generate driver intention features based on the biometric and operational behavior data. The calculation module is used to generate a spatiotemporal scene map based on the environmental data and determine the risk level of dynamic targets in the spatiotemporal scene map; The fusion module is used to perform cross-modal fusion of the driver's intention features and the spatiotemporal scene map to obtain fused features; The generation module is used to generate a dynamic risk area with a mapping relationship between risk level and spatial range based on the fusion features, and to generate the planned driving path of the target vehicle within the dynamic risk area; The display module is used to generate augmented reality interactive screens and to show the dynamic risk area and the planned driving route to the driver of the target vehicle through the augmented reality interactive screens.

[0014] The present invention also provides a vehicle, the vehicle including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the multimodal perception driving risk decision-making and interaction method as described in any of the preceding claims.

[0015] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the multimodal perception driving risk decision-making and interaction method as described in any of the preceding claims.

[0016] The present invention has at least the following beneficial effects: First, this technical solution constructs a deep fusion channel between intent and scenario, collecting not only environmental data but also the driver's biometrics and operational behavior data simultaneously, and predicting driving intent and generating intent features accordingly. Simultaneously, environmental data is transformed into a spatiotemporal scenario map containing dynamic target risk levels. Cross-modal fusion of intent features and scenario maps enables the system to understand the relationship between the driver's subjective tendencies and the objective environment, fundamentally solving the problem of discrepancies between traditional ADAS decisions and driver expectations, thereby improving decision accuracy. Second, by generating dynamic risk areas with a mapping relationship between risk level and spatial range, and planning driving paths based on these, the system transforms abstract decision-making logic into a visual spatial representation. Finally, using augmented reality interactive visuals, the risk areas and planned paths are directly displayed to the driver. This intuitive visual presentation replaces traditional audio-visual warnings, allowing the driver to quickly understand the system's intent and suggestions, significantly enhancing the naturalness of human-machine collaboration. Finally, generating dynamic risk areas based on fused features means that the system can flexibly define the spatial boundaries and levels of risk according to real-time fused intent and scenario information. This dynamic mapping mechanism does not rely on a preset rule base and can cope with complex scenarios such as pedestrians turning back and vehicles cutting in, and generate corresponding safe paths, thereby improving the system's adaptability to long-tail risk scenarios. Attached Figure Description

[0017] The accompanying drawings are provided to further understand the technical solutions of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the technical solutions of the present invention, and do not constitute a limitation on the technical solutions of the present invention.

[0018] Figure 1 This is a flowchart illustrating the steps of a multimodal perception-based driving risk decision-making and interaction method. Figure 2 This is a flowchart of step S103 in a multimodal perception driving risk decision-making and interaction method; Figure 3 This is a flowchart of step S104 in a multimodal perception driving risk decision-making and interaction method; Figure 4 It is a data processing flow for an AI risk decision-making big model in a multimodal perception driving risk decision-making and interaction method; Figure 5 This is a schematic diagram of the structure of a multimodal perception driving risk decision-making and interaction device; Figure 6 This is a schematic diagram of the overall architecture of a multimodal perception driving risk decision-making and interaction system. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0020] With the development of automotive intelligence, existing Advanced Driver Assistance Systems (ADAS) primarily rely on external environmental perception (such as cameras and millimeter-wave radar) to provide warnings or interventions (such as Automatic Emergency Braking (AEB) and Lane Departure Warning (LDW). However, existing technologies have the following drawbacks: Traditional systems cannot accurately identify the driver's instantaneous intentions (such as "emergency avoidance" or "normal lane change"), leading to frequent false alarms or erroneous interventions, which affects the driving experience.

[0021] The main interaction methods are through buzzers or dashboard icons, which makes it difficult for drivers to quickly understand the source of risk in complex road conditions.

[0022] Traditional rule-based algorithms struggle to handle long-tail scenarios such as irregularly shaped obstacles and lack context-based reasoning capabilities.

[0023] This technical solution provides a multimodal perception driving risk decision-making and interaction method, device, vehicle, and medium, which can achieve deep fusion and understanding of driver intent and complex dynamic scenarios, significantly improving the decision-making accuracy, human-machine collaboration naturalness, and adaptability to long-tail risk scenarios of the driving safety assistance system. The following are various embodiments of this technical solution.

[0024] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the steps of a multimodal perception-based driving risk decision-making and interaction method.

[0025] This embodiment provides a multimodal perception-based driving risk decision-making and interaction method, including: S101. Collect the driver's biometrics and operational behavior data in the target vehicle, as well as the environmental data of the target vehicle's location.

[0026] S102. Based on biometric and operational behavior data, predict the driver's driving intention of the target vehicle and generate driver intention features.

[0027] S103. Based on environmental data, generate a spatiotemporal scene map and determine the risk level of dynamic targets in the spatiotemporal scene map.

[0028] S104. Perform cross-modal fusion of driver intent features and spatiotemporal scene map to obtain fused features.

[0029] S105. Based on the fusion features, generate a dynamic risk area with a mapping relationship between the risk level and the spatial range, and generate the planned driving path of the target vehicle within the dynamic risk area.

[0030] S106. Generate an augmented reality interactive screen and display the dynamic risk area and planned driving route to the driver of the target vehicle through the augmented reality interactive screen.

[0031] In step S101 of some embodiments, dynamic environmental data of the vehicle's surroundings in 360 degrees is collected in real time using an external sensor group including LiDAR, millimeter-wave radar, and high-definition cameras. This environmental data specifically refers to traffic participants, obstacles, road conditions, weather, etc. Internal sensor groups, including a DMS driver monitoring camera, steering wheel grip sensor, and brake / accelerator pedal travel sensor, are used to collect real-time data on the driver's biometrics and operational behavior, such as the driver's gaze direction, facial expression, heart rate, and operational force.

[0032] In step S102 of some embodiments, the collected driver operation data and gaze data are input into an intention prediction neural network model based on the Transformer architecture. The model outputs the probability of driver intention classification within the next 1-3 seconds, including labels such as "normal cruising", "unintentional deviation", "emergency avoidance", or "aggressive lane change" as driver intention features.

[0033] Understandably, in this embodiment, firstly, the technical solution constructs a deep fusion channel between intent and scene, collecting not only environmental data but also simultaneously collecting the driver's biometric and operational behavior data, and predicting driving intent and generating intent features accordingly. Simultaneously, the environmental data is transformed into a spatiotemporal scene map containing the dynamic risk level of the target. Cross-modal fusion of intent features and scene map enables the system to understand the relationship between the driver's subjective tendencies and the objective environment, fundamentally solving the problem of discrepancies between traditional ADAS decisions and driver expectations, thereby improving decision accuracy. Secondly, by generating dynamic risk areas with a mapping relationship between risk level and spatial range, and planning driving paths based on this, the system transforms abstract decision logic into a visual spatial expression. Finally, using augmented reality interactive screens, the risk areas and planned paths are directly displayed to the driver. This intuitive visual presentation replaces traditional audio-visual warnings, enabling the driver to quickly understand the system's intent and suggestions, significantly enhancing the naturalness of human-machine collaboration. Finally, generating dynamic risk areas based on fused features means that the system can flexibly define the spatial boundaries and levels of risk according to the real-time fused intent and scene information. This dynamic mapping mechanism does not rely on a preset rule base and can cope with complex scenarios such as pedestrians turning back and vehicles cutting in, and generate corresponding safe paths, thereby improving the system's adaptability to long-tail risk scenarios.

[0034] Please refer to Figure 2 , Figure 2 This is a flowchart of step S103 in a multimodal perception driving risk decision-making and interaction method.

[0035] In some embodiments, step S103 includes: S201. Based on environmental data, extract the motion characteristics of dynamic targets and the static road structure characteristics.

[0036] S202. Utilize motion characteristics to predict the motion trajectory of a dynamic target, obtain the predicted trajectory of the dynamic target, and calculate the cognitive uncertainty of the motion trajectory prediction.

[0037] S203. Spatiotemporally align the predicted trajectory, cognitive uncertainty, and static road structure features to construct a spatiotemporal scene graph. In the spatiotemporal scene graph, the attributes of the nodes corresponding to the dynamic targets include the predicted trajectory and cognitive uncertainty.

[0038] S204. Based on cognitive uncertainty, determine the risk level of dynamic objectives, where the risk level is positively correlated with cognitive uncertainty.

[0039] Understandably, this embodiment extracts the motion features of dynamic targets and predicts their trajectories, while introducing cognitive uncertainty to quantify the reliability of the prediction. This allows the system to more accurately identify high-risk targets with long-tail characteristics such as random reversals and sudden changes in direction. By incorporating uncertainty as a node attribute into the spatiotemporal scene graph and establishing a positive correlation between risk level and uncertainty, the system generates uncertainty distributions in risk areas that more closely resemble real-world scenarios. This improvement significantly enhances the early warning capability for targets with unclear intentions, reduces false alarms and false negatives, and thus improves decision-making robustness and adaptability to long-tail risks.

[0040] Please refer to Figure 3 , Figure 3 This is a flowchart of step S104 in a multimodal perception driving risk decision-making and interaction method.

[0041] In some embodiments, step S104 includes: S301. Extract the driver's driving style characteristics from biometric and operational behavior data.

[0042] S302. Concatenate the driver's intent features with the driving style features to obtain the query vector.

[0043] S303. For each node in the spatiotemporal scene graph, the node features are mapped to key vectors using the first parameter matrix, and the node features are mapped to value vectors using the second parameter matrix.

[0044] S304. Calculate the dot product similarity between the query vector and each key vector, and normalize the dot product similarity of all nodes in the spatiotemporal scene graph to obtain the attention weight corresponding to each node.

[0045] S305. The value vectors of all nodes are weighted and summed according to the attention weights to obtain the scene and intent fusion features centered on the driver.

[0046] Understandably, this embodiment extracts driving style features and concatenates them with intent features to form a query vector, enabling the system to capture the personalized operating habits of different drivers. Simultaneously, an attention mechanism is introduced to calculate the relevance weight between the driver's query vector and each dynamic target node in the scene graph, thereby filtering out the risk factors most closely related to the current driver's concerns. This attention-weighted fusion method replaces simple feature concatenation, making the system's decisions closer to the driver's actual focus. Ultimately, driver-centric scene and intent fusion features are generated, significantly improving the adaptability to personalized driving behavior, enhancing the intuitiveness and naturalness of human-machine collaboration, and reducing human-machine conflicts caused by inconsistencies between system behavior and driver expectations.

[0047] Please refer to Figure 4 , Figure 4 It is a data processing flow for an AI risk decision-making big model in a multimodal perception driving risk decision-making and interaction method.

[0048] In some embodiments, the method further includes: The environmental data and driver intention features are spatiotemporally aligned to obtain an aligned feature vector. The aligned feature vector is then input into a pre-defined risk decision model. Based on the risk entropy calculation theory, the risk decision model determines whether there is a potential collision risk in the current scenario. If so, a risk level is generated as the degree of risk, which can be low risk, medium risk, or high risk.

[0049] Specifically, external environmental data, including point cloud data and visual semantic data, are spatiotemporally aligned with the driver's intention prediction results. The aligned feature vectors are then input into a large AI risk decision-making model trained through reinforcement learning. Based on the theory of "risk entropy" calculation, this model determines whether there is a potential collision risk in the current scenario and generates three levels of risk, such as low risk, medium risk, and high risk.

[0050] Understandably, in this embodiment, environmental data and driver intent features are first spatiotemporally aligned to eliminate temporal and spatial discrepancies between modalities, laying the foundation for joint inference. Subsequently, a large-scale risk decision model is used, based on risk entropy calculation theory, to assess potential collision risks and automatically output three risk levels: low, medium, and high. This mechanism combines the representational capabilities of deep learning models with uncertainty metrics from information theory, enabling the system to quantitatively assess latent risks in complex, long-tailed scenarios. Compared to traditional risk assessments based on fixed rules or simple classifiers, this method significantly improves the accuracy of risk identification and scenario generalization ability, and supports the system in adopting differentiated early warning and control strategies based on risk levels.

[0051] In some embodiments, when the risk level is low, the current driving status log is recorded.

[0052] In some embodiments, the method further includes: When the risk level is medium or high, the dynamic targets that cause potential collision risks in the spatiotemporal scene map are taken as risk sources, and the location of the risk sources in the real physical space is determined; the augmented reality head-up display device is controlled to project warning boxes and escape path guide lines at the real physical location corresponding to the risk source, so as to guide the driver's line of sight to the risk source and prompt the avoidance path.

[0053] Understandably, when a medium to high risk occurs, the system accurately locates the target causing the risk as the risk source and obtains its position in the real physical space. It then controls the augmented reality head-up display to project a warning frame directly onto the real location of the risk source, effectively guiding the driver's line of sight to the direction of the risk, thus solving the pain point of traditional warnings not being able to indicate location. Simultaneously, escape path guidance lines are projected, providing the driver with intuitive avoidance references. This mechanism significantly shortens the driver's time to locate and respond to the risk source, reduces the risk of collisions caused by distraction or delayed judgment, and significantly improves human-machine interaction efficiency and driving safety in emergency scenarios.

[0054] In some embodiments, the method further includes: When the risk level is high and the driver's intention characteristics determine that the driver has failed to avoid potential collision risks, an active obstacle avoidance control command is generated; the active obstacle avoidance control command is sent to the target vehicle's drive-by-wire chassis system to execute obstacle avoidance actions such as brake-by-wire or steering-by-wire.

[0055] Understandably, when the risk level reaches high risk and the system determines that the driver cannot effectively avoid the obstacle, it proactively generates obstacle avoidance control commands and sends them to the drive-by-wire chassis system to execute drive-by-wire braking or steering. This mechanism fills the gap in the capabilities of simple warnings or augmented reality guidance in extreme scenarios, achieving a gradual takeover through human-machine collaboration: the driver takes the lead, and the system only intervenes proactively when the driver's reaction is insufficient. Compared to traditional automatic emergency braking functions, this solution integrates intent judgment and scenario understanding, avoiding false triggering or excessive intervention, while providing reliable physical safety guarantees in truly dangerous emergency moments, significantly improving the survival rate of vehicles in extreme scenarios.

[0056] In some embodiments, after generating the dynamic risk region, the method further includes: The risk level of dynamic targets in the spatiotemporal scene map is compared with a preset threshold. When the risk level is greater than the preset threshold, the risk area is increased along the predicted trajectory direction and orthogonal direction of the dynamic target.

[0057] Understandably, this embodiment compares the risk level of dynamic targets in the spatiotemporal scene map with a preset threshold. When the risk level exceeds the threshold, the system actively expands the risk area along the predicted trajectory direction and its orthogonal directions. This mechanism reflects the potential threat range of high-risk targets in the forward direction and lateral offset direction, matching the geometry of the risk area with the uncertainty of the target's motion. Compared to risk areas with fixed shapes or static expansion, this method can dynamically adjust the protection boundary based on real-time risk assessment results, avoiding overly conservative area encroachment while ensuring safety. This improves the flexibility and scene adaptability of the planned path, and is particularly suitable for long-tail risk scenarios where the predicted trajectory has uncertainty.

[0058] This technical solution also provides specific embodiments of implementing multimodal perception driving risk decision-making and interaction methods in real-world scenarios, as shown below: In the first embodiment, the target vehicle is traveling at 30 km / h on an urban road. A bus is stopping on the left to pick up and drop off passengers, obstructing the driver's view. Traditional automatic emergency braking systems cannot see behind the bus using radar. When a pedestrian suddenly runs out from in front of the bus, the system's reaction distance is insufficient, easily leading to a collision. In the implementation of the multimodal perception driving risk decision-making and interaction method of this application, the system first collects information through multimodal perception. Millimeter-wave radar uses multipath reflection characteristics to capture a faint moving point cloud behind the bus, while a high-definition camera captures the shadow of a pedestrian's feet below the bus. Subsequently, the artificial intelligence risk decision-making model integrates features such as the bus stop, obstructions, and micro-Doppler effects to infer a high-risk conclusion, determining that a pedestrian is crossing behind the obstruction and calculating a collision probability exceeding 85%. At this point, although the driver has not yet seen the pedestrian, the system has already determined it to be a high-risk situation. At the interaction and execution level, an augmented reality head-up display projects a red virtual light wall onto the front edge of the bus, accompanied by text prompts indicating blind spots. At the same time, the system triggers pre-fill braking to eliminate the brake pedal's free travel, but does not apply maximum braking force temporarily to avoid false alarms that could lead to rear-end collisions. When the pedestrian actually runs out, the driver, having already recognized the risk location through augmented reality, can actively brake; if the driver's reaction is too slow, the system immediately triggers automatic emergency braking.

[0059] In the second embodiment, while the vehicle is traveling on a highway, a large cardboard box suddenly falls from the vehicle in front, which is an irregularly shaped obstacle. Traditional lane keeping systems attempt to keep the vehicle within the lane, but when an obstacle exists in the lane, the system is prone to conflicting with the driver's evasive intentions. In the implementation of the multimodal perception driving risk decision-making and interaction method of this application, the system first identifies the cardboard box obstacle ahead using a camera and calculates that it cannot be passed. At the same time, the driver monitoring camera captures the driver's gaze instantly locking onto the obstacle, and the steering wheel torque sensor detects that the driver is applying an emergency steering force exceeding a threshold. Based on this, the intention prediction model outputs an emergency evasive intention judgment. Subsequently, the artificial intelligence risk decision-making big data model receives the dual signals of the external obstacle presence and the driver's intention to evade, determines that the current lane keeping assist should be temporarily suppressed, and returns control to the driver. The system actively reduces the damping of the electric power steering to assist the driver in quickly completing a lane change to avoid the obstacle. After the avoidance is completed, the system automatically restores the lane keeping function.

[0060] Please refer to Figure 5 , Figure 5 This is a schematic diagram of the structure of a multimodal perception driving risk decision-making and interaction device.

[0061] This embodiment also provides a multimodal perception driving risk decision-making and interaction device, including: The data acquisition module 401 is used to collect the biometrics and operational behavior data of the driver in the target vehicle, as well as the environmental data of the location of the target vehicle.

[0062] The identification module 402 is used to predict the driving intention of the driver of the target vehicle and generate driver intention features based on biometric and operational behavior data.

[0063] The calculation module 403 is used to generate a spatiotemporal scene map based on environmental data and determine the risk level of dynamic targets in the spatiotemporal scene map.

[0064] The fusion module 404 is used to perform cross-modal fusion of driver intent features and spatiotemporal scene map to obtain fused features.

[0065] The generation module 405 is used to generate a dynamic risk area with a mapping relationship between risk level and spatial range based on fusion features, and to generate the planned driving path of the target vehicle within the dynamic risk area.

[0066] Display module 406 is used to generate augmented reality interactive screens and display dynamic risk areas and planned driving routes to the driver of the target vehicle through the augmented reality interactive screens.

[0067] Please refer to Figure 6 , Figure 6 This is a schematic diagram of the overall architecture of a multimodal perception driving risk decision-making and interaction system.

[0068] like Figure 6 As shown, in a multimodal perception-based driving risk decision-making and interaction system, external and internal sensor groups collect environmental data and driver biometric and operational behavior data, respectively. A driver intent prediction network outputs driver intent features based on internal sensor data. A multimodal feature fusion unit initially aligns the intent features with the environmental features extracted by the external sensors and feeds them into a Transformer temporal model, which models the long-term dependence of the dynamic target's trajectory on the vehicle's state. The fused features are ultimately input into a large-scale AI risk decision-making model, which outputs the risk level of the current scenario and generates proactive intervention or decision commands. Depending on the command type, the system, on the one hand, drives an augmented reality head-up display to show the driver the risk area and guidance information; on the other hand, it sends steer-by-wire or brake-by-wire commands to the drive-by-wire chassis system to execute obstacle avoidance actions. The entire process achieves closed-loop processing from perception, fusion, decision-making to interaction and execution.

[0069] This invention also provides a vehicle control device, including a memory, a processor, and a program stored in the memory and executable on the processor. When the program is executed by the processor, it implements the multimodal perception driving risk decision-making and interaction method described in the above embodiments.

[0070] Taking the example of a processor and memory in a vehicle controller being connected via a bus, the memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the control processor, and these remote memories can be connected to the control device via a network. The non-transitory software programs and instructions required to implement the control methods of the above embodiments are stored in the memory, and when executed by the processor, the control methods of the above embodiments are performed.

[0071] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0072] This invention also provides a vehicle, including the vehicle control device described in the above embodiments.

[0073] The vehicle can be a private car, such as a sedan, SUV, MPV, or pickup truck. It can also be a commercial vehicle, such as a van, bus, small truck, or large semi-trailer. The vehicle must have an electric motor capable of outputting power or acting as a generator to store mechanical energy. When the vehicle is a new energy vehicle, it can be a hybrid or a pure electric vehicle.

[0074] Since the vehicle applies all the technical solutions of the above-mentioned control device or vehicle controller, it has at least all the beneficial effects brought about by the technical solutions of the above embodiments, which will not be repeated here.

[0075] Furthermore, one embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions for performing the above-described multimodal perception driving risk decision-making and interaction method.

[0076] It is worth noting that, since the computer-readable storage medium of the present invention is capable of executing the multimodal perception driving risk decision-making and interaction method of any of the above embodiments, the specific implementation and technical effects of the computer-readable storage medium of the present invention can be referred to the specific implementation and technical effects of the multimodal perception driving risk decision-making and interaction method of any of the above embodiments.

[0077] Furthermore, one embodiment of the present invention also provides a computer program product, including a computer program or computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer program or computer instructions from the computer-readable storage medium and executes the computer program or computer instructions, causing the computer device to perform the above-described multimodal perception driving risk decision-making and interaction method.

[0078] It is worth noting that, since the computer program product of this embodiment can execute the multimodal perception driving risk decision-making and interaction method of any of the above embodiments, the specific implementation method and technical effect of the computer program product of this embodiment can refer to the specific implementation method and technical effect of the multimodal perception driving risk decision-making and interaction method of any of the above embodiments.

[0079] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically include computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0080] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

Claims

1. A multimodal perception-based driving risk decision-making and interaction method, characterized in that, The method includes: Collect biometric and operational behavior data of the driver in the target vehicle, as well as environmental data of the location of the target vehicle; Based on the biometric and operational behavior data, predict the driver's driving intention of the target vehicle and generate driver intention features; Based on the environmental data, a spatiotemporal scene map is generated, and the risk level of dynamic targets in the spatiotemporal scene map is determined. The driver's intent features are fused with the spatiotemporal scene map across modalities to obtain fused features; Based on the fusion features, a dynamic risk region with a mapping relationship between risk level and spatial range is generated, and a planned driving path for the target vehicle is generated within the dynamic risk region. An augmented reality interactive screen is generated, and the dynamic risk area and the planned driving route are displayed to the driver of the target vehicle through the augmented reality interactive screen.

2. The method according to claim 1, characterized in that, The step of generating a spatiotemporal scene map based on the environmental data and determining the risk level of dynamic targets in the spatiotemporal scene map includes: Based on the environmental data, the motion features of dynamic targets and the static road structure features are extracted; The motion characteristics are used to predict the motion trajectory of the dynamic target, and the predicted trajectory of the dynamic target is obtained. The cognitive uncertainty of the motion trajectory prediction is calculated. The predicted trajectory, the cognitive uncertainty, and the static road structure features are spatiotemporally aligned to construct the spatiotemporal scene graph, wherein the attributes of the nodes corresponding to the dynamic targets in the spatiotemporal scene graph include the predicted trajectory and the cognitive uncertainty. Based on the cognitive uncertainty, the risk level of the dynamic target is determined, wherein the risk level is positively correlated with the cognitive uncertainty.

3. The method according to claim 1, characterized in that, The step of fusing the driver's intent features with the spatiotemporal scene map across modalities to obtain fused features includes: The driver's driving style characteristics are extracted from the biometric and operational behavior data; The driver intent feature and the driving style feature are concatenated to obtain the query vector; For each node in the spatiotemporal scene graph, the node features of the node are mapped to a key vector using a first parameter matrix, and the node features of the node are mapped to a value vector using a second parameter matrix. Calculate the dot product similarity between the query vector and each of the key vectors, and normalize the dot product similarity of all the nodes in the spatiotemporal scene graph to obtain the attention weight corresponding to each node; The value vectors of all the nodes are weighted and summed according to the attention weights to obtain the driver-centric scene and intent fusion feature.

4. The method according to claim 1, characterized in that, The method further includes: The environmental data is spatiotemporally aligned with the driver's intention features to obtain an aligned feature vector. The aligned feature vectors are input into a preset risk decision-making model. Based on the risk entropy calculation theory, the risk decision-making model determines whether there is a potential collision risk in the current scenario. If so, a risk level is generated as the degree of risk, wherein the risk level is a low risk level, a medium risk level, or a high risk level.

5. The method according to claim 4, characterized in that, The method further includes: When the risk level is medium risk or high risk, the dynamic target in the spatiotemporal scene graph that causes the potential collision risk is taken as the risk source, and the location of the risk source in the real physical space is determined. The augmented reality head-up display device is controlled to project a warning box and an escape path guide line at the real physical location corresponding to the risk source, so as to guide the driver's line of sight to the risk source and prompt the avoidance path.

6. The method according to claim 4, characterized in that, The method further includes: When the risk level is high and the driver's intention characteristics determine that the driver has failed to avoid the potential collision risk, an active obstacle avoidance control command is generated. The active obstacle avoidance control command is sent to the drive-by-wire chassis system of the target vehicle to execute obstacle avoidance actions such as drive-by-wire braking or drive-by-wire steering.

7. The method according to claim 1, characterized in that, After generating the dynamic risk region, the method further includes: The risk level of the dynamic target in the spatiotemporal scene map is compared with a preset threshold. When the risk level is greater than the preset threshold, the risk area is increased along the predicted trajectory direction and the orthogonal direction of the dynamic target.

8. A multimodal perception driving risk decision-making and interaction device, characterized in that, The device includes: The data acquisition module is used to collect the biometrics and operational behavior data of the driver in the target vehicle, as well as the environmental data of the location of the target vehicle. The identification module is used to predict the driving intention of the driver of the target vehicle and generate driver intention features based on the biometric and operational behavior data. The calculation module is used to generate a spatiotemporal scene map based on the environmental data and determine the risk level of dynamic targets in the spatiotemporal scene map; The fusion module is used to perform cross-modal fusion of the driver's intention features and the spatiotemporal scene map to obtain fused features; The generation module is used to generate a dynamic risk area with a mapping relationship between risk level and spatial range based on the fusion features, and to generate the planned driving path of the target vehicle within the dynamic risk area; The display module is used to generate augmented reality interactive screens and to show the dynamic risk area and the planned driving route to the driver of the target vehicle through the augmented reality interactive screens.

9. A vehicle, characterized in that, The vehicle includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the multimodal perception driving risk decision-making and interaction method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the multimodal perception driving risk decision-making and interaction method as described in any one of claims 1 to 7.

Images