Vehicle driving behavior recognition and early warning method based on video sequence analysis

Abstract

This application relates to the field of intelligent driving technology and discloses a method for vehicle driving behavior recognition and early warning based on video sequence analysis, including the following steps: collecting and synchronizing multimodal data streams, wherein the multimodal data streams include in-vehicle video data, out-of-vehicle video data, and vehicle state data; parsing the synchronized multimodal data streams to generate a structured system full-state vector containing driver state, vehicle state, and environmental state at each time step; and constructing a dynamic causal graph for the current time step based on the system full-state vector and a set of predefined implicit intent nodes. By constructing a dynamic causal graph and introducing an affordance region associated with intent, this invention achieves accurate and interpretable recognition of the driver's dynamic intent and enables more targeted environmental conflict analysis for specific intents.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving technology, specifically to a method for vehicle driving behavior recognition and early warning based on video sequence analysis. Background Technology

[0002] With the development of intelligent driving technology, advanced driver assistance systems (ADAS) have been widely used in modern vehicles. Existing technologies mainly focus on real-time perception of the vehicle's surrounding environment and reactive safety intervention, such as using sensors to detect obstacles ahead to trigger emergency braking, or issuing warnings when the vehicle deviates from its lane.

[0003] However, existing technologies still fall short in understanding and predicting drivers' dynamic and uncertain driving intentions. These systems typically rely on direct observation of vehicle dynamics and have limited ability to capture the complex cognitive processes drivers undergo during decision-making, such as hesitation, evaluation, and shifts in intent. This results in a superficial understanding of driving behavior and often delayed warnings.

[0004] More importantly, existing risk assessment models typically focus on currently quantifiable collision risks. When a driver actively stops a driving action due to awareness of danger, traditional systems assume the risk has been eliminated simply because the dangerous action has ceased. This mechanism fails to conduct a deep analysis of the potential danger behind the stopped action, meaning it cannot infer the potentially serious consequences if the driver had not stopped. Therefore, existing technologies have cognitive blind spots when dealing with such near-miss events, missing crucial opportunities to provide drivers with proactive safety warnings. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a vehicle driving behavior recognition and early warning method based on video sequence analysis. This method solves the problem in existing technologies where it is difficult to accurately understand the driver's dynamic and uncertain driving intentions, thus making it difficult to assess the potential risks avoided by the driver after actively stopping the operation.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a vehicle driving behavior recognition and early warning method based on video sequence analysis, comprising the following steps: Collect and synchronize multimodal data streams, including in-vehicle video data, out-of-vehicle video data, and vehicle status data; The synchronized multimodal data stream is parsed to generate a structured system full state vector containing driver state, vehicle state, and environment state at each time step. Based on the system's full state vector and a set of predefined implicit intent nodes, construct a dynamic causal graph for the current time step; Reasoning is performed on the graph sequence of consecutive time steps, and the state of each implicit intent node in the next time step is output. Based on the inferred intention state and the dynamic causal graph at the current time step, a risk assessment is performed and an early warning signal is generated.

[0007] Preferably, the step of parsing the synchronized multimodal data stream to generate the environmental state specifically includes: Identify environmental entities from the external video data; Based on the environmental entities and the vehicle state, an Affordance zone associated with a specific driving intention is generated; and, Calculate a safety score for the Affordance zone as part of the environmental state.

[0008] Preferably, the step of constructing the dynamic causal graph at the current time step specifically includes: Instantiate the state variables in the system's total state vector as observable state nodes; and... Based on preset causal logic rules, directed edges representing perception relationships, action relationships, or intention hypothesis relationships are established between the observable state nodes and the implicit intention nodes.

[0009] Preferably, an intent state machine is associated with each of the implicit intent nodes, the intent state machine having a discrete set of states including potential, active, executing, completed, and terminated; The step of reasoning about the graph sequence of consecutive time steps specifically involves: inputting the graph sequence into a temporal graph network model to reason about the state transitions of the intent state machine, thereby determining the state of the implicit intent node in the next time step.

[0010] Preferably, the risk assessment step specifically includes: When a state transition to the aborted state is detected in any of the stated intention state machines, counterfactual reasoning is triggered.

[0011] Preferably, the counterfactual reasoning specifically includes: Based on a pre-trained driving policy network, a virtual action sequence corresponding to the intention to stop is generated; A virtual future scenario is deduced based on the virtual action sequence using a forward simulator; and Assess the potential risks of the virtual future scenario.

[0012] Preferably, the step of generating the warning signal specifically includes: When the potential risks of the virtual future scenario exceed a preset predictive risk threshold, a predictive warning signal is generated, which includes a description of the virtual future scenario.

[0013] Preferably, the calculation of the security score of the Affordance zone specifically involves: The collision time between the vehicle and other road users in the Affordance zone is quantitatively calculated based on the safe distance and time of collision.

[0014] Preferably, the step of conducting a risk assessment further includes: Calculate a real-time risk score, which includes the behavior-intent inconsistency risk calculated based on the consistency between the current behavior and the intent state, and the intent-environment conflict risk calculated based on the intent state and the security score of the Affordance zone.

[0015] A vehicle driving behavior recognition and early warning system based on video sequence analysis includes: The multimodal data processing module is used to collect and synchronize multimodal data streams, and parse the synchronized multimodal data streams to generate a structured system full state vector at each time step. The dynamic causal graph construction module is used to construct a dynamic causal graph for the current time step based on the system's full state vector and a set of predefined implicit intent nodes. The graph evolution and intent reasoning module is used to reason about the graph sequence at consecutive time steps and output the state of each implicit intent node at the next time step; and, The risk assessment and early warning module is used to perform risk assessment and generate early warning signals based on the inferred intention state and the dynamic causal graph of the current time step.

[0016] This invention provides a method for vehicle driving behavior recognition and early warning based on video sequence analysis. It has the following beneficial effects: 1. This invention explicitly establishes the causal relationships between driver state, environmental state, and implicit intention nodes by constructing a dynamic causal graph, and models the evolution of intention using an intention state machine. Compared to traditional black-box models, this structured reasoning approach can more accurately capture the dynamic changes in driver intention and provides a clear logical path for risk tracing, thereby improving the accuracy of intention recognition and the interpretability of the system.

[0017] 2. This invention introduces the concept of an affordance zone associated with specific driving intentions, shifting the focus of environmental perception from the identification of physical entities to the assessment of functional areas. By calculating the safety score of the affordance zone, the system can quantify complex environmental information into decision-making criteria directly related to the driver's intentions. This allows risk assessment to move beyond general collision warnings and enable more targeted conflict analysis based on specific intentions, thus improving the effectiveness of warnings.

[0018] 3. This invention uses a forward simulator to simulate the virtual future scenarios that might result from the continued execution of an aborted intention, and assesses its potential risks. This enables the system to detect and warn the driver of hidden dangers that they have just actively avoided but may not have fully perceived, achieving a shift from reactive warnings to proactive predictive warnings, and significantly improving active safety. Attached Figure Description

[0019] Figure 1 This is a diagram illustrating the method steps of the present invention. Detailed Implementation

[0020] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Please see the appendix Figure 1 This invention provides a method for vehicle driving behavior recognition and early warning based on video sequence analysis, comprising the following steps: Collect and synchronize multimodal data streams, including in-vehicle video data, out-of-vehicle video data, and vehicle status data; The synchronized multimodal data stream is parsed, and a structured system full state vector containing driver state, vehicle state and environment state is generated at each time step. Based on the system's full state vector and a set of predefined implicit intent nodes, construct a dynamic causal graph for the current time step; Infer the graph sequence of consecutive time steps and output the state of each implicit intent node in the next time step; Based on the inferred intention state and the dynamic causal graph of the current time step, risk assessment is performed and early warning signals are generated.

[0022] Multimodal data processing and state analysis: The data acquisition and synchronization unit is responsible for acquiring raw data from multiple sensor sources and synchronizing it in time. In one implementation, the system configuration includes: a wide-angle camera installed inside the vehicle and facing the driver to acquire in-vehicle video data; a forward-facing camera installed at the front of the vehicle to acquire external environmental video data; and a CAN bus adapter connected via a vehicle diagnostic interface such as OBD-II to read vehicle status data. To ensure precise alignment of the data streams, the system employs a software synchronization scheme based on the Network Time Protocol (NTP). All data acquisition devices are connected to the same local area network and synchronized with an NTP server. Each data entry is tagged with a high-precision synchronization timestamp, which is used by subsequent processing units for data fusion. In another implementation, a hardware synchronization mechanism can be used, where a master clock source sends synchronization pulse signals to the frame synchronization pins of all cameras and the data recording unit via GPIO pins, achieving nanosecond-level hardware synchronization.

[0023] Implementation of the structured state parsing unit This unit at each time step The synchronized data stream is converted into a structured system full-state vector. .

[0024] Driver Status Analysis The driver's state is extracted from in-vehicle video frames using a series of deep learning models. .

[0025] Visual focus vector The driver's eye region image, cropped from a video frame, is input into a pre-trained convolutional neural network (CNN), which directly regresses a three-dimensional unit vector. This vector represents the driver's instantaneous gaze direction in the vehicle coordinate system.

[0026] Head posture The algorithm uses facial landmark detection algorithms such as MediaPipeFaceMesh to locate 68 facial landmarks. Then, based on the correspondence between these landmarks and a 3D head model, a Perspective-n-Point (PnP) algorithm is used to calculate the head rotation matrix, and the pitch, yaw, and roll angles are extracted to form the head pose vector. .

[0027] Hand key points and functional area mapping A hand pose estimation model (such as MediaPipeHands) was used to detect the 3D coordinates of 21 key points on the hand, forming a key point set. Simultaneously, the system pre-defined three-dimensional spatial polygons for several functional areas within the vehicle, such as the steering wheel area, through a one-time calibration. Central control screen area By executing an algorithm to determine if a point lies within a polygon, key hand points are mapped to functional areas, generating a set of interaction relationships. .

[0028] Vehicle Status Analysis By reading and decoding messages through the CAN bus adapter, the vehicle's instantaneous speed and steering wheel angle can be directly obtained. Accelerator pedal opening Brake pedal status Turn signal status etc., constitute the vehicle status .

[0029] Analysis of Environmental Conditions and Affordance Zone This part is a technical feature of the present invention. It not only identifies objects in the environment, but also resolves them into affordance zones related to driving behavior functions.

[0030] First, a pre-trained object detection model (such as YOLOv5) on a large automotive dataset (such as WaymoOpenDataset) is used to process the forward-looking video, outputting a set of environment entities. This includes the location, size, and speed of other vehicles and pedestrians.

[0031] Secondly, based on and the condition of this vehicle Dynamically generated with specific driving intentions Related Affordance District With the intention to change lanes to the left. For example, its corresponding Affordance zone This is a virtual safety space in the left lane. The safety score for this area is... The following formula is used for quantification: ; in: This is the distance between this vehicle and the nearest vehicle in the left rear lane; This is the time-to-collision time between this vehicle and the vehicle behind it, calculated using the following formula: ,in It is the relative speed between the two vehicles, and is only calculated when the following vehicle is faster than the current vehicle; It is an activation function, such as the Sigmoid function. It maps physical quantities such as distance or time to a safety score in the (0,1) interval. It is a safety threshold; the scoring function ensures that the safety score of the Affordance zone is high only when the safe distance from the following vehicle and the TTC both meet the requirements.

[0032] Dynamic causal graph construction: Graph Node Definition and Instantiation At each time step The parsed state variables are instantiated into a graph. observable state nodes At the same time, the system predefines a set containing... A set of driving intentions serves as an implicit intention node. ,For example =Change lanes to the left, =Overtaking. The states of these intent nodes are unknown and are the targets of subsequent reasoning. The total node set of the graph is... .

[0033] Graph Edge Definition and Construction Rules: Graph Edges Dynamically generated based on a set of pre-defined causal logic rules, these rules link observed states with underlying intentions.

[0034] Perceptual edge: If the driver's visual focus vector The extension of the line intersects with a certain Affordance zone If the geometric regions intersect, then a path is established from... point to A directed edge. This edge represents the driver observing the intended action. The required environmental area.

[0035] Action side: If hands are in the steering wheel function area Inside, and the steering wheel angle If a change occurs, a path is established from the hand state node to the vehicle yaw rate node. The edge.

[0036] Intent Hypothesis Edges: If a specific set of precursory behaviors occurs, an edge is established from the state nodes of these behaviors to the corresponding intent nodes. For example, if the perception edge observes the left lane change zone and turn signal state nodes... If left turn is activated simultaneously, then establish a path from these two nodes to the intent node. The edges represent evidence that this intention has been activated.

[0037] Graph evolutionary reasoning based on intent state machine: Definition and transitions of Intent State Machine (ISM) For each implicit intent node Associate an intent state machine. The state set of this state machine is: These represent potential, active, in progress, completed, and terminated, respectively. The precise meaning of each state is defined as follows: This indicates that the intention has been formed, but the key actions have not yet begun; The key action indicating intent is in progress (e.g., the vehicle has crossed the lane line); This indicates that the intent was voluntarily abandoned by the driver before it could be completed. State transition paths are strictly limited; for example, an intent must be activated before it can be executed or aborted.

[0038] The intent-state reasoning engine based on temporal graph networks uses temporal graph networks (TGN) as the reasoning engine to model the evolution of graph sequences.

[0039] Model Architecture and Training: The TGN model comprises a memory module, a message passing network, and a state update module. During training, a driving dataset containing frame-by-frame annotations of driving videos and corresponding intent states is used. The model's task is to predict each intent node given the geographic graph and historical information at a given time step. In the next step The state. The loss function uses cross-entropy loss. : ; in, It is the sequence length. It is an intention One-hot encoding of the true state at time t+1 It is the state probability distribution predicted by the model for this intention.

[0040] Online inference and state updates: The graph is built in real time during system runtime. The input is fed into a pre-trained TGN model. The model outputs each intent. Next state probability distribution Based on this distribution, the system updates the intent state using the maximum a posteriori probability principle: .

[0041] Risk assessment and early warning generation: The real-time risk assessment unit calculates the real-time risk score at the current moment. The score is composed of two weighted parts: Risk of inconsistency between behavior and intention When a vehicle is detected to have significant lateral displacement, but the corresponding lane change intention is not... Not At that time, this risk item is activated.

[0042] Intention - Environmental Conflict Risk When the intention to change lanes is in effect. for However, its corresponding Affordance zone safety score Below the preset safety threshold This risk item is activated at that time. Its value is... The total risk is [missing information]. ,in These are the weighting coefficients.

[0043] Predictive early warning unit based on counterfactual reasoning This unit is another core technical feature of the present invention, used to assess the potential dangers of an aborted intent.

[0044] Triggering mechanism: When any intention The state machine occurs from or arrive The unit is triggered during the state transition.

[0045] Counterfactual reasoning process: Action sequence generation: The system calls an intent Pre-trained driving policy network This network is an imitation learning model that can generate actions that conform to the driving habits of experts based on the current state.

[0046] Virtual evolution: from the state before the intention was aborted Initially, the system uses a forward simulator. Perform the simulation. At each virtual time step , by policy network Generate virtual actions emulator Based on this action and the current virtual state Calculate the next virtual state .

[0047] Virtual risk assessment: for the generated virtual map sequence For each frame, apply the real-time risk function from Section 4.1. The maximum value among them is taken as the maximum potential risk of the counterfactual scenario. .

[0048] Generation and output of early warning signals If real-time risk Exceeding the high-risk threshold The system generates real-time early warning signals. If counterfactual reasoning leads to the greatest potential risk... Exceeding the predictive risk threshold The system generates predictive early warning signals. The signal is accompanied by explanatory text generated based on a counterfactual reasoning process, such as: Predictive warning: If the lane change operation that was just abandoned is continued, there is a high risk of collision with the vehicle to the side and rear.

[0049] This invention also provides a vehicle driving behavior recognition and early warning system based on video sequence analysis, comprising: The multimodal data processing module is used to collect and synchronize multimodal data streams, and parse the synchronized multimodal data streams to generate a structured system full state vector at each time step. This module serves as the system's perception entry point, responsible for extracting structured status information from heterogeneous sensor data sources. It receives multimodal data streams from onboard sensors, primarily including: driver video data collected by in-vehicle cameras, driving environment video data collected by external cameras, and vehicle status data obtained from the vehicle controller area network (CAN) bus.

[0050] In its implementation, the multimodal data processing module 100 first performs strict synchronization alignment on the different input data streams based on timestamps to ensure the temporal consistency of the state information in subsequent processing. Then, the module parses the data through a parallel processing pipeline. Driver state analysis: Using computer vision algorithms, such as human pose estimation and eye-tracking models, the in-vehicle video data is analyzed to extract the driver's head posture, visual focus (e.g., looking at the left rearview mirror, looking ahead), and hand position (e.g., holding the steering wheel, operating the gear shift lever), and other key states.

[0051] Environmental State Analysis: Utilizing object detection and semantic segmentation models, this module analyzes external video data to identify surrounding traffic participants such as other vehicles, pedestrians, and road markings like lane lines. A key step is that this module generates an affordance zone associated with a specific driving intention based on these entities and the vehicle's state, and calculates its safety score. For example, for a left lane change intention, the system generates a virtual area in the left lane that the vehicle can merge into as an affordance zone, and calculates its safety score based on the safe distance between the vehicle and vehicles behind it within that area. Colliding with Time Calculate its safety score Vehicle status analysis: Directly decodes data from the CAN bus to obtain precise vehicle dynamic parameters such as real-time speed, acceleration, steering wheel angle, and turn signal status.

[0052] Finally, this module integrates the driver state, vehicle state, and environmental state (including the affordance zone safety score) obtained from the above analysis into a unified, structured system full-state vector. Then output it to the next module.

[0053] The dynamic causal graph construction module is used to construct a dynamic causal graph for the current time step based on the system's full state vector and a set of predefined implicit intent nodes. The function of this module is based on the system's full-state vector output by the multimodal data processing module 100. Construct a graph-structured data that can explicitly express the causal relationship between each state variable and the driver's intention.

[0054] In its implementation, the dynamic causal graph construction module 200 performs this at each time step. Perform the following operations to construct a dynamic causal graph : Node instantiation: The module first instantiates the system's full state vector. Each state variable in the graph, such as visual focus = left rearview mirror and Affordance zone safety score = 0.8, is instantiated as an observable state node. Simultaneously, the module maintains a set of predefined implicit intent nodes representing the driver's potential intentions, such as the intent to change lanes to the left and the intent to follow.

[0055] Edge relationship establishment: Directed edges are established between different nodes based on a set of pre-defined causal logic rules. These rules are defined based on a driving knowledge base, for example: Perceptual edge: Points from observable state nodes representing environmental information to implicit intention nodes, indicating that the environmental information is the perceptual input that forms the intention, such as a high safety score in the left Affordance zone pointing to a left lane change intention.

[0056] Action edge: Points from the implicit intention node to the observable state node representing the driver's action, indicating that the intention may lead to the action. Left lane change intention points to turning the steering wheel left.

[0057] Intent Hypothesis Edge: An edge is established between mutually exclusive implicit intent nodes to represent the competition or inhibition relationship between them.

[0058] Through the above steps, the module transforms the unstructured state vector into a dynamic causal graph rich in semantic information. This graph provides a structured, causal-logical input for subsequent intention reasoning.

[0059] The graph evolution and intent reasoning module is used to reason about the graph sequence of consecutive time steps and output the state of each implicit intent node in the next time step. This module is the cognitive core of the system, responsible for performing temporal reasoning on the graph sequence of continuous time steps to dynamically infer the driver's true intentions.

[0060] In its implementation, the graph evolution and intent reasoning module 300 associates each implicit intent node with an intent state machine (ISM) having discrete states such as latent, activated, executing, completed, and terminated. This module receives the graph sequence output by the dynamic causal graph construction module 200. This data is then fed into a pre-trained temporal graph network model, such as GNN-LSTM or TemporalGCN. This model learns the temporal evolution patterns of the graph structure and node features to infer the most likely state transition for each intention state machine at the next time step. For example, when the model observes that the driver is continuously looking at the left rearview mirror, has activated the left turn signal, and the safety score of the left affordance zone is continuously increasing, it infers that the intention to change lanes to the left transitions from potential to active. The training objective of this module is to minimize the cross-entropy loss between the predicted state transition probabilities and the ground truth labels. ; in, Is it intended to The true state at any given moment This is the probability distribution of the states predicted by the model. Finally, this module outputs the updated states of all implicit intent nodes at the next time step. .

[0061] The risk assessment and early warning module is used to conduct risk assessment and generate early warning signals based on the inferred intention state and the dynamic causal graph of the current time step.

[0062] This module is the system's decision-making and output unit, responsible for assessing potential risks and generating corresponding early warning signals based on the inferred intention state and the current system state.

[0063] In its implementation, the risk assessment and early warning module employs a dual risk assessment mechanism: Real-time Risk Assessment: This module continuously calculates a real-time risk score, which integrates inconsistencies between behavior and intent (e.g., intending to go straight but the vehicle deviates laterally) and conflicts between intent and environment (e.g., intending to change lanes but the target affordance zone has an extremely low safety score). When the real-time risk score exceeds a preset threshold, an immediate warning is generated.

[0064] Counterfactual Risk Assessment: This module continuously monitors the intent state. When any intent state machine is detected to transition from active or executing to aborted—for example, a driver abandoning a lane-change attempt—counterfactual reasoning is triggered. This process includes: Based on a pre-trained driving policy network, a virtual action sequence corresponding to the intention to abort is generated.

[0065] Using a forward simulator, a virtual future scenario is deduced based on the virtual action sequence and the current environmental state, assuming the driver does not abort the operation.

[0066] Assess the potential risks of this virtual future scenario, such as whether it could lead to a collision.

[0067] When the risk of a virtual future scenario exceeds a high predictive risk threshold, a predictive warning signal is generated. This signal not only indicates danger but can also include a description of the virtual scenario, such as, "If you just changed lanes, you might collide with a vehicle coming from behind," thus revealing the hidden dangers that the driver should actively avoid and reinforcing their safety awareness.

[0068] Ultimately, the module outputs the generated warning signal to a human-machine interface such as a display screen, voice prompts, or the vehicle's active safety system.

[0069] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for vehicle driving behavior recognition and early warning based on video sequence analysis, characterized in that, Includes the following steps: Collect and synchronize multimodal data streams, including in-vehicle video data, out-of-vehicle video data, and vehicle status data; The synchronized multimodal data stream is parsed to generate a structured system full-state vector containing driver state, vehicle state, and environmental state at each time step. Based on the system full-state vector and a set of predefined implicit intent nodes, a dynamic causal graph for the current time step is constructed. The graph sequence of consecutive time steps is inferred to output the state of each implicit intent node in the next time step. Based on the inferred intent state and the dynamic causal graph for the current time step, a risk assessment is performed, and a warning signal is generated.

2. The vehicle driving behavior recognition and early warning method based on video sequence analysis according to claim 1, characterized in that, The step of parsing the synchronized multimodal data stream to generate the environmental state specifically includes: Identify environmental entities from the external video data; Based on the environmental entities and the vehicle state, an Affordance zone associated with a specific driving intention is generated; and, Calculate a safety score for the Affordance zone as part of the environmental state.

3. The vehicle driving behavior recognition and early warning method based on video sequence analysis according to claim 1, characterized in that, The steps for constructing the dynamic causal graph at the current time step specifically include: Instantiate the state variables in the system's total state vector as observable state nodes; and... Based on preset causal logic rules, directed edges representing perception relationships, action relationships, or intention hypothesis relationships are established between the observable state nodes and the implicit intention nodes.

4. The vehicle driving behavior recognition and early warning method based on video sequence analysis according to claim 1, characterized in that, For each of the implicit intent nodes, an intent state machine is associated, the intent state machine having a discrete set of states including potential, active, executing, completed, and aborted. The step of reasoning about the graph sequence of consecutive time steps specifically involves: inputting the graph sequence into a temporal graph network model to reason about the state transitions of the intent state machine, thereby determining the state of the implicit intent node in the next time step.

5. The vehicle driving behavior recognition and early warning method based on video sequence analysis according to claim 4, characterized in that, The steps for conducting a risk assessment specifically include: When a state transition to the aborted state is detected in any of the stated intention state machines, counterfactual reasoning is triggered.

6. The vehicle driving behavior recognition and early warning method based on video sequence analysis according to claim 5, characterized in that, The counterfactual reasoning specifically includes: Based on a pre-trained driving policy network, a virtual action sequence corresponding to the intention to stop is generated; A virtual future scenario is deduced based on the virtual action sequence using a forward simulator; and Assess the potential risks of the virtual future scenario.

7. The vehicle driving behavior recognition and early warning method based on video sequence analysis according to claim 6, characterized in that, The step of generating the early warning signal specifically includes: When the potential risks of the virtual future scenario exceed a preset predictive risk threshold, a predictive warning signal is generated, which includes a description of the virtual future scenario.

8. The vehicle driving behavior recognition and early warning method based on video sequence analysis according to claim 2, characterized in that, The calculation of the security score for the Affordance zone is specifically as follows: The collision time between the vehicle and other traffic participants in the Affordance zone is quantitatively calculated based on the safe distance and time of collision.

9. The vehicle driving behavior recognition and early warning method based on video sequence analysis according to claim 1, characterized in that, The steps for conducting a risk assessment also include: Calculate a real-time risk score, which includes the behavior-intent inconsistency risk calculated based on the consistency between the current behavior and the intent state, and the intent-environment conflict risk calculated based on the intent state and the security score of the Affordance zone.

10. A vehicle driving behavior recognition and early warning system based on video sequence analysis, used in the cooperative path game and planning method for unmanned aerial vehicles and autonomous vehicles according to claims 1-9, characterized in that, include: The multimodal data processing module is used to collect and synchronize multimodal data streams, and parse the synchronized multimodal data streams to generate a structured system full state vector at each time step. The dynamic causal graph construction module is used to construct a dynamic causal graph for the current time step based on the system's full state vector and a set of predefined implicit intent nodes. The graph evolution and intent reasoning module is used to reason about the graph sequence of consecutive time steps and output the state of each implicit intent node in the next time step. as well as, The risk assessment and early warning module is used to perform risk assessment and generate early warning signals based on the inferred intention state and the dynamic causal graph of the current time step.

Images