Traffic participant intention prediction method and device based on multi-modal time attention

Abstract

The present application relates to the technical field of multi-modal timing, and discloses a traffic participant intention prediction method and device based on multi-modal timing attention, comprising: collecting multi-modal original traffic data under a traffic scene; preprocessing the multi-modal original traffic data to obtain standardized multi-modal data; extracting features from the standardized multi-modal data to obtain a multi-dimensional feature sequence; dynamically weighting and fusing the multi-dimensional feature sequence and learning semantic association through a timing attention model to obtain a fused feature sequence; predicting the intention probability of a traffic participant based on the fused feature sequence to obtain an intention prediction result containing an intention category and a corresponding confidence; and transmitting the intention prediction result to a decision planning module of an automatic driving system, so that the decision planning module obtains a vehicle behavior decision strategy adapted to the current traffic scene based on the intention category and the corresponding confidence.

Description

Technical Field

[0001] This invention relates to the field of multimodal temporal technology, and more specifically to a method and apparatus for predicting traffic participant intentions based on multimodal temporal attention. Background Technology

[0002] Current mainstream trajectory prediction technologies (including deep learning models such as LSTM) rely on historical kinematic data (position and velocity sequences) of the target vehicle for trend extension. Their fundamental flaw lies in simplifying driving behavior as a physical inertial phenomenon, rather than a decision-making process driven by human intuition. While these technologies are well-suited for short-term, continuous motion prediction on structured roads, they completely fail in scenarios requiring "game theory and interaction," such as unprotected left turns and roundabouts. In these situations, the key to vehicle behavior lies in the driver's interpretation of the environment and social understanding / confrontation with other participants, rather than historical acceleration data. Due to a lack of perception and modeling capabilities for the high-level semantic information of "driver intent," autonomous vehicles often exhibit overly conservative (reluctant to proceed) or reckless (misunderstanding intent) non-socialized behaviors, severely impacting traffic efficiency and safety.

[0003] Driver monitoring systems (DMS), which complement trajectory prediction technology, also have significant shortcomings: First, their application is limited to the driver of the vehicle in question. If used to analyze drivers of other vehicles, issues such as long observation distances, varying angles, and facial obstruction can lead to unstable information acquisition and low accuracy. Second, the technology focuses on the driver's physiological state (fatigue, distraction), completely ignoring their cognitive intentions (driving plans, interaction tendencies). Even when the driver is awake, the "interaction signals" implied by their gaze direction and slight head movements (such as indicating yielding or preparing to cut in) are not interpreted by existing DMS technology, failing to provide crucial support for trajectory prediction by adding the "intention dimension."

[0004] Furthermore, some solutions that attempt to improve prediction performance through multi-sensor fusion are still limited by static fusion strategies: they often employ simple methods such as feature splicing and weighted averaging, failing to dynamically identify the "real-time importance" of different information in the decision-making process. For example, in a road intersection game scenario, the subtle "nodding" (indicating passage) of a target vehicle is far more valuable than the stable driving data of the previous ten seconds. However, existing technologies lack effective mechanisms to capture such fleeting "intention signals," making it difficult to dynamically focus on key information and ultimately limiting the accuracy and scene understanding of predictions in complex interaction scenarios.

[0005] In summary, existing technologies face three major constraints: mainstream trajectory prediction technologies, due to their simple model assumptions, cannot capture the social attributes of driving behavior; DMS technology, due to scenario limitations and focus bias, cannot compensate for the shortcomings in "cross-vehicle intent perception"; and multi-sensor fusion, due to its static strategy, cannot capture instantaneous key signals. These three factors together constitute the core technological gap in the "accuracy of trajectory prediction technology in complex interaction scenarios." Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes a method and apparatus for predicting traffic participant intentions based on multimodal temporal attention, in order to solve the aforementioned technical problems.

[0007] A method for predicting traffic participant intentions based on multimodal temporal attention is provided, including:

[0008] Collect multimodal raw traffic data in traffic scenarios, wherein the multimodal raw traffic data includes at least visual image data, kinematic data, and environmental context data of the target vehicle;

[0009] The original multimodal traffic data is preprocessed to obtain standardized multimodal data;

[0010] Feature extraction is performed on the standardized multimodal data to obtain a multidimensional feature sequence;

[0011] A fused feature sequence is obtained by dynamically weighting and fusing multi-dimensional feature sequences and learning semantic associations through a temporal attention model.

[0012] Based on the fused feature sequence, the probability of traffic participant intent is predicted to obtain an intent prediction result that includes intent category and corresponding confidence level;

[0013] The intent prediction results are transmitted to the decision planning module of the autonomous driving system. The decision planning module obtains a vehicle behavior decision strategy adapted to the current traffic scenario based on the intent category and corresponding confidence level.

[0014] Based on the fused feature sequence, the probability of traffic participant intent is predicted, resulting in an intent prediction outcome that includes intent category and corresponding confidence level, including:

[0015] The fused feature sequence is compressed to obtain a single-dimensional global feature vector;

[0016] The global feature vector is input into a fully connected neural network, and the global feature vector is semantically mapped by the fully connected neural network to obtain the original score of each intent category;

[0017] Applying a soft-maximization activation function to the original scores yields the probability distribution of each intent category, which includes at least the following: proceed, yield, and hesitate.

[0018] The intent category with the highest probability in the probability distribution is extracted as the intent graph, where the probability value is used as the confidence level of the intent graph, and the normalized information entropy of the probability distribution is calculated to quantify the prediction uncertainty.

[0019] By integrating the intention graph, corresponding confidence level, probability distribution, and prediction uncertainty, the intention prediction result is obtained.

[0020] Furthermore, the original multimodal traffic data is preprocessed to obtain standardized multimodal data, including:

[0021] Timestamps are added to each type of data in the multimodal raw traffic data. Based on the timestamps, the multi-type data with different sampling probabilities are aligned to a unified time axis to complete data synchronization.

[0022] The synchronized visual image data and kinematic data are converted to a unified vehicle coordinate system with the rear axle center of the autonomous vehicle as the origin, and the coordinate system is calibrated.

[0023] Distortion correction and color space conversion are performed on the calibrated visual image data, noise filtering and outlier removal are performed on the kinematic data, and format standardization is performed on the environmental context data.

[0024] The processed data of various types are integrated to obtain standardized multimodal data.

[0025] Furthermore, feature extraction is performed on the standardized multimodal data to obtain a multidimensional feature sequence, including:

[0026] For the visual image data in the standardized multimodal data, the image region corresponding to the target vehicle is extracted, input into the convolutional neural network model, and the pitch angle, yaw angle and wheel steering angle of the target vehicle are regressed respectively to obtain the micro attitude feature sequence.

[0027] For the kinematic data in the standardized multimodal data, calculate the lateral distance, longitudinal distance, lateral velocity, longitudinal velocity, lateral acceleration, longitudinal acceleration, and the lateral offset and directional angle difference of the target vehicle relative to the center line of the lane, and obtain the kinematic feature sequence.

[0028] For the environmental context data in the standardized multimodal data, the map is queried to obtain the type identifier, road geometric parameters and traffic rule information of the current scene. The type identifier and traffic rule information are encoded and processed, and combined with the road geometric parameters to obtain the environmental context feature sequence.

[0029] The microscopic posture feature sequence, kinematic feature sequence, and environmental context feature sequence are aligned by time steps to obtain a multi-dimensional feature sequence.

[0030] Furthermore, a temporal attention model is used to dynamically weight and fuse multi-dimensional feature sequences and learn semantic associations to obtain a fused feature sequence, including:

[0031] The multi-dimensional feature sequence is linearly projected to map each dimension feature to a feature space of a preset dimension, thereby obtaining an embedded feature sequence.

[0032] The position code is added to the embedded feature sequence, and temporal information is injected to obtain an embedded feature sequence with temporal information;

[0033] The embedded feature sequence with temporal information is input into the self-attention mechanism encoder block, wherein the self-attention mechanism encoder block includes a multi-head self-attention layer, a feedforward neural network layer, and a residual connection and layer normalization unit.

[0034] The attention weight matrix is ​​obtained by calculating the correlation score of the features at each time step in the feature sequence through the multi-head self-attention layer.

[0035] The feature sequence is dynamically weighted based on the attention weight matrix to obtain attention-weighted features;

[0036] The attention-weighted features are normalized and fused using the residual connection and layer normalization unit. The fusion result is then input into the feedforward neural network layer for nonlinear transformation to obtain single-round processing features.

[0037] The self-attention mechanism encoder block is processed a preset number of times, and the final output feature sequence is used as the fused feature sequence.

[0038] Furthermore, the intent prediction result is transmitted to the decision-making and planning module of the autonomous driving system. The decision-making and planning module obtains a vehicle behavior decision-making strategy adapted to the current traffic scenario based on the intent category and corresponding confidence level, including:

[0039] When the intention prediction result shows that the intention graph is yielding and the confidence level is greater than the first preset threshold and the prediction uncertainty is less than the first preset entropy value, the decision planning module will constrain the motion of the target vehicle to a deceleration or stopping model and generate a time-optimal opportunistic passage trajectory.

[0040] When the intention graph in the intention prediction result is passable and the confidence level is greater than the first preset threshold, the decision planning module regards the target vehicle as a dynamic obstacle with right-of-way and generates a defensive waiting trajectory that decelerates to a stop based on the collision time and safe distance model.

[0041] When the intention prediction result shows hesitation or the prediction uncertainty is greater than the second preset entropy value, the decision planning module starts an interactive subroutine to generate a short-term, low-speed and suspendable active probing action, and triggers the intention prediction model to perform intensive resampling and re-evaluation of the target vehicle's response, and adjusts the decision strategy based on the re-evaluation result.

[0042] Furthermore, an interactive subroutine is initiated to generate short-duration, low-speed, and abortable proactive probing actions, and to trigger the intent prediction model to intensively resample and re-evaluate the target vehicle's response. Based on the re-evaluation results, the decision-making strategy is adjusted, including:

[0043] Set the probe action parameters, which include at least the probe speed, probe displacement and duration. Based on the probe action parameters, plan the short-term movement trajectory of the autonomous vehicle. The speed of the short-term movement trajectory does not exceed a preset low-speed threshold and the displacement does not exceed a preset displacement threshold.

[0044] During the process of autonomous vehicles performing exploratory actions, the control intent prediction model increases the data sampling frequency to a preset high-frequency sampling rate, intensively collects and processes the multimodal data of the target vehicle, and generates high-frequency updated intent prediction results.

[0045] If the intention graph in the frequently updated intention prediction results changes to yielding and the confidence level meets the first preset threshold, switch to opportunistic passage trajectory planning;

[0046] If the main graph becomes passable and the confidence level meets the first preset threshold, switch to defensive waiting trajectory planning;

[0047] If the situation remains hesitant or the uncertainty of the forecast has not decreased, repeat the probing action or switch to the most conservative parking and waiting strategy.

[0048] The invention employing the above technical solution has the following advantages:

[0049] This invention enhances the active safety of autonomous driving systems by increasing decision-making certainty in unstructured interaction scenarios. Existing technologies primarily rely on posterior physical states (such as changes in speed and position) for risk assessment, which inherently suffers from time lag when facing the sudden and nonlinear behavior of human drivers. This invention achieves proactive inference of driver intentions by decoding feedforward signals such as the vehicle's micro-attitude. This capability extends the system's safety boundary from "reactive avoidance based on collision time" to "predictive intervention based on intent probability." For example, in highway merging scenarios, the system no longer reacts only after a neighboring vehicle has intruded into the lane; instead, it can identify the high-probability "forced lane change" intention in advance by analyzing features such as the vehicle's continuous, minute approach angle towards the lane line, abnormal acceleration patterns, and the driver's head orientation.

[0050] This predictive capability provides the planning module with valuable decision-making redundancy time, enabling it to generate smoother and safer avoidance trajectories. It effectively suppresses unstable behaviors such as "phantom braking" or emergency steering that may be caused by system lag, thereby significantly reducing the accident rate. Attached Figure Description

[0051] To more clearly illustrate the specific embodiments of the present invention, the accompanying drawings used in the specific embodiments will be briefly described below. In all the drawings, the elements or parts are not necessarily drawn to scale.

[0052] Figure 1 This is a structural diagram of the intent prediction system in the traffic participant intent prediction method and device based on multimodal temporal attention of the present invention;

[0053] Figure 2 This is a diagram showing the internal structure of the data acquisition and preprocessing module in the traffic participant intention prediction method and device based on multimodal temporal attention of the present invention.

[0054] Figure 3 This is a diagram showing the internal structure of the feature extraction module in the traffic participant intent prediction method and device based on multimodal temporal attention of the present invention.

[0055] Figure 4 This invention relates to the internal structure of a temporal attention prediction model in a traffic participant intent prediction method and device based on multimodal temporal attention.

[0056] Figure 5 This is a flowchart illustrating the interaction between intent information and decision planning in a traffic participant intent prediction method and device based on multimodal temporal attention, as described in this invention.

[0057] Figure 6This is a flowchart of a traffic participant intent prediction method based on multimodal temporal attention according to the present invention. Detailed Implementation

[0058] The embodiments of the technical solution of the present invention will now be described in detail with reference to the accompanying drawings. These embodiments are merely illustrative of the technical solution of the present invention and are therefore intended to limit the scope of protection of the present invention.

[0059] like Figures 1-6 As shown, the present invention provides a method for predicting traffic participant intentions based on multimodal temporal attention, comprising:

[0060] Step S01: Collect multimodal raw traffic data in traffic scenarios, wherein the multimodal raw traffic data includes at least visual image data, kinematic data, and environmental context data of the target vehicle;

[0061] Step S02: Preprocess the original multimodal traffic data to obtain standardized multimodal data;

[0062] Step S03: Extract features from the standardized multimodal data to obtain a multidimensional feature sequence;

[0063] Step S04: Dynamically weighted fusion and semantic association learning are performed on the multi-dimensional feature sequence through a temporal attention model to obtain a fused feature sequence;

[0064] Step S05: Based on the fused feature sequence, predict the probability of traffic participants' intentions to obtain the intention prediction results, which include the intention category and the corresponding confidence level.

[0065] Step S06: Transmit the intent prediction results to the decision planning module of the autonomous driving system. The decision planning module obtains a vehicle behavior decision strategy adapted to the current traffic scenario based on the intent category and corresponding confidence level.

[0066] Specifically, existing technologies mostly rely on posterior physical states (such as changes in speed and positional shifts) for risk assessment, which has an inherent lag that is difficult to avoid when dealing with sudden and nonlinear operations by human drivers. This invention, by decoding feed-forward signals such as the vehicle's micro-attitude, can achieve forward inference of the driver's intentions. This capability extends the system's safety boundary from "reactive avoidance based on time-of-collision (TTC)" to "predictive intervention based on the probability of intention."

[0067] Taking highway merging scenarios as an example, the system does not need to wait for adjacent vehicles to intrude into the lane before responding. Instead, it can identify high-probability "forced lane change" intentions in advance by analyzing features such as the minute angle at which adjacent vehicles continuously approach the lane line, abnormal acceleration patterns, and the direction of the driver's head. This predictive capability provides the planning module with valuable decision-making redundancy time, enabling it to generate smoother and safer avoidance trajectories. This effectively suppresses unstable operations such as "phantom braking" and emergency steering that may be caused by system lag, ultimately significantly reducing the accident rate.

[0068] The current bottleneck in the traffic efficiency of autonomous vehicles in areas such as unsignalized intersections stems from the high degree of uncertainty in the decision-making models' assumptions regarding the behavior of other traffic agents (such as human drivers). This uncertainty forces vehicles to adopt a conservative strategy of "waiting for a clear, conflict-free passage window," significantly limiting traffic efficiency.

[0069] This invention models the intention patterns of other vehicles, converging their behavior from a broad, random probability distribution into a high-confidence intention prediction with clear semantic labels (such as "yield" and "go straight"), thereby reducing the entropy of the decision space. Entropy is an indicator that quantifies the uncertainty and confusion of a decision when faced with multiple possible choices / outcomes. This optimization can significantly reduce decision uncertainty. When the system determines with a confidence level exceeding a threshold that the other vehicle is in a "yield" state, the decision planning module can move beyond the traditional "waiting logic based on safe distance" and instead execute an opportunistic passage strategy that conforms to social traffic regulations. This not only shortens the vehicle's path time but also avoids traffic wave interference to vehicles behind it due to its own hesitation by reducing unnecessary stops, ultimately creating a positive ripple effect on the throughput and stability of the local traffic network.

[0070] As an emerging social technology entity, autonomous vehicles must conform to the traffic cognition and interaction paradigms of human society. This invention endows the system with human-like intent understanding and interaction capabilities, transforming it from an isolated "rule-enforcing outsider" into a "socialized intelligent agent" capable of participating in traffic games: it can interpret informal communication signals conveyed by human drivers through vehicle "body language" (such as subtle steering and speed adjustments), and can also respond in a way that meets human expectations through its own driving actions (such as slightly protruding forward to indicate "testing the waters" and passing at a steady speed to indicate "understanding the intent").

[0071] This interaction model, based on a "shared mental model," can improve the predictability and explainability of autonomous vehicle behavior for human drivers. When humans perceive autonomous vehicles as collaborative partners who "understand traffic rules and can communicate effectively," their sense of trust and security will significantly increase. This plays a crucial supporting role in promoting high-level autonomous driving technology from closed testing grounds to open roads and ultimately integrating it into the human social transportation system.

[0072] In some embodiments, the raw multimodal traffic data is preprocessed to obtain standardized multimodal data, including:

[0073] Timestamps are added to each type of data in the multimodal raw traffic data. Based on the timestamps, the multi-type data with different sampling probabilities are aligned to a unified time axis to complete data synchronization.

[0074] The synchronized visual image data and kinematic data are converted to a unified vehicle coordinate system with the rear axle center of the autonomous vehicle as the origin, and the coordinate system is calibrated.

[0075] Distortion correction and color space conversion are performed on the calibrated visual image data, noise filtering and outlier removal are performed on the kinematic data, and format standardization is performed on the environmental context data.

[0076] The processed data of various types are integrated to obtain standardized multimodal data.

[0077] Specifically, such as Figure 1 The data flow and processing are shown from left to right, and consists of four main levels / modules.

[0078] Hardware layer:

[0079] This is the physical foundation of the system and the source of all information. It is not the inventive point of this invention, but it is a necessary prerequisite for the system's operation. This includes:

[0080] Forward-facing camera group: Provides high-resolution, high-frame-rate video streams and is the core sensor for extracting microscopic posture features.

[0081] Millimeter-wave radar / LiDAR: Provides precise kinematic information such as distance, speed, and angle of target vehicles, unaffected by lighting conditions.

[0082] Inertial Measurement Unit (IMU) / Satellite Positioning Data (GPS) / High-Precision Map: Provides the vehicle's precise positioning, attitude, and static information about the surrounding environment (such as lane lines, intersection types, etc.).

[0083] Data acquisition and preprocessing module:

[0084] It is responsible for processing the messy raw data from the hardware layer into a clean, organized, and usable format.

[0085] Specific steps:

[0086] (1) Data Synchronization: Since different sensors have different sampling frequencies, this module must accurately timestamp all data and align them to a unified timeline. Timestamp methods include: hardware-triggered synchronization, system clock synchronization, and sensor built-in clock. Timeline alignment methods include: time base unification, interpolation / resampling, and clock calibration correction.

[0087] (2) Coordinate system calibration and fusion: Unify the data from all sensors into the same coordinate system (usually the vehicle's own coordinate system). For example, the target in the camera pixel coordinate system is transformed into the three-dimensional spatial coordinate system of the radar and lidar through calibration parameters to achieve multi-sensor front fusion.

[0088] Intent prediction system:

[0089] This is the core device of the invention; it receives preprocessed data and performs an intent prediction task. Internally, it contains two key sub-modules:

[0090] (1) Feature extraction module: responsible for extracting meaningful features from data of different modalities. Each extractor is responsible for processing one data modality.

[0091] (2) Temporal attention prediction model: This is the system's "central processing unit". It receives the extracted features and performs in-depth analysis and final intent judgment.

[0092] Downstream application modules:

[0093] This is where the system's value is realized. It receives the intention prediction results output by the system itself and acts accordingly.

[0094] (1) Decision planning module: Based on the high-level information provided by 3, such as "the other party's intention is to give way", it plans the driving path and speed of the vehicle and makes more intelligent decisions.

[0095] (2) The vehicle control module executes the instructions of the planning module to control the steering wheel, accelerator and brake.

[0096] Detailed implementation of the data acquisition and preprocessing module:

[0097] System device diagram description, internal structure of data acquisition and preprocessing module.

[0098] Specific methods and steps for each unit:

[0099] Hardware data interface and preliminary processing (corresponding) Figure 2 (The processors on the left):

[0100] The system communicates directly with the hardware interfaces of each sensor through a driver program, receiving the raw data packets output by them. For each type of data, a dedicated processor performs preliminary "cleaning".

[0101] Camera data processor: Receives raw image frames (e.g., RAW format). First, it loads the camera's intrinsic parameter matrix, performs distortion correction on the image, and eliminates image distortion caused by the lens's physical characteristics. Then, if necessary, it may perform color space conversion (e.g., from Bayer format to RGB or grayscale).

[0102] Radar / LiDAR data processor: Receives raw point cloud data or radar target points. It applies noise filtering algorithms (such as statistical outlier removal) to remove isolated noise points. For LiDAR, target clustering (such as using the DBSCAN algorithm) is typically also performed, grouping point clouds belonging to the same object into clusters, and using multi-target tracking algorithms (such as Kalman filtering) to track these targets across consecutive frames, generating a target list with unique IDs and motion states.

[0103] Inertial Measurement Unit (IMU) / GPS / CAN Bus Data Processor: The IMU provides high-frequency angular velocity and acceleration, while GPS provides low-frequency but absolute position information. This processor fuses the two data using an extended Kalman filter (EKF) to obtain high-frequency, accurate vehicle positioning and attitude. Simultaneously, it analyzes data from the CAN bus to obtain low-level vehicle information such as wheel speed and steering wheel angle.

[0104] Timestamp synchronization unit:

[0105] The goal is to ensure that we know the precise physical time corresponding to every frame of image, every frame of point cloud, and every GPS location point at the millisecond or even microsecond level.

[0106] Specific steps:

[0107] Hardware synchronization (preferred solution) employs Precision Time Protocol (PTP). A master clock is set up in the system, and all sensors (supporting PTP) act as slave clocks, maintaining strict synchronization with the master clock via a network protocol. When sensors acquire data, a high-precision, synchronized timestamp is directly applied at the hardware level.

[0108] Software synchronization (alternative): If the hardware does not support it, the software timestamps the data when it arrives at the central computing unit. To reduce latency jitter, an event-triggered mechanism is usually used, for example, using the camera's exposure end signal as the trigger event to synchronously record data from other sensors at that moment.

[0109] Coordinate system transformation and fusion unit:

[0110] Its function is to unify all the "cleaned" and "synchronized" data into a common reference system so that subsequent joint analysis can be performed.

[0111] Specific steps:

[0112] (1) Coordinate system definition: Define a vehicle coordinate system with the rear axle center of the vehicle as the origin, the front direction of the vehicle as the X-axis, the left side as the Y-axis, and the vertical upward as the Z-axis as the Z-axis as the Z-axis, as a unified reference system.

[0113] (2) Loading of extrinsic parameter matrices: When the system starts up, it loads pre-calibrated extrinsic parameter matrices. These matrices describe the position and orientation (rotation and translation) of each sensor (such as camera, radar) relative to the vehicle coordinate system. This calibration process is completed before the vehicle leaves the factory or during maintenance using specialized calibration tools.

[0114] (3) Transformation and fusion:

[0115] For targets already tracked by radar / LiDAR, their extrinsic parameter matrices are directly used to transform their position and velocity in their own coordinate system to the vehicle coordinate system through rigid body transformation (rotation matrix multiplied by the original coordinates and then the translation vector).

[0116] For a target in a camera image, the target (e.g., a bounding box) is first detected in the image's two-dimensional pixel coordinate system. Then, using the camera's intrinsic and extrinsic parameters, as well as the target's depth information obtained from radar / LiDAR (through spatial matching), a back-projection transformation is performed to restore its three-dimensional spatial position in the vehicle coordinate system from the two-dimensional pixel coordinates. This process is called multi-sensor front fusion or tightly coupled fusion.

[0117] (4) Final output:

[0118] After this series of sophisticated operations, the data acquisition and preprocessing module will finally output a synchronized multimodal data packet. At each timestamp t, this data packet contains:

[0119] Processed video frames.

[0120] A list of targets with IDs and motion status in a unified vehicle coordinate system (from radar / LiDAR).

[0121] The precise status of this vehicle (position, speed, attitude).

[0122] In some embodiments, feature extraction is performed on standardized multimodal data to obtain a multidimensional feature sequence, including:

[0123] For visual image data in standardized multimodal data, the image region corresponding to the target vehicle is extracted, input into a convolutional neural network model, and the pitch angle, yaw angle and wheel steering angle of the target vehicle are regressed respectively to obtain a micro attitude feature sequence.

[0124] For the kinematic data in the standardized multimodal data, calculate the lateral distance, longitudinal distance, lateral velocity, longitudinal velocity, lateral acceleration, longitudinal acceleration, and the lateral offset and directional angle difference of the target vehicle relative to the center line of the lane, and obtain the kinematic feature sequence.

[0125] For the environmental context data in the standardized multimodal data, the map is queried to obtain the type identifier, road geometric parameters and traffic rule information of the current scene. The type identifier and traffic rule information are encoded and processed, and combined with the road geometric parameters to obtain the environmental context feature sequence.

[0126] By aligning the microscopic posture feature sequence, kinematic feature sequence, and environmental context feature sequence according to time steps, a multi-dimensional feature sequence is obtained.

[0127] In some embodiments, a fused feature sequence is obtained by dynamically weighting and fusing multi-dimensional feature sequences and learning semantic associations using a temporal attention model, including:

[0128] A linear projection is performed on the multi-dimensional feature sequence to map each dimension of the feature to a feature space of a preset dimension, thus obtaining the embedded feature sequence.

[0129] The positional encoding is added to the embedded feature sequence, and temporal information is injected to obtain the embedded feature sequence with temporal information;

[0130] The embedded feature sequence with temporal information is input into the self-attention mechanism encoder block, which includes a multi-head self-attention layer, a feedforward neural network layer, and residual connection and layer normalization unit.

[0131] The attention weight matrix is ​​obtained by calculating the correlation score of features at each time step in the feature sequence through a multi-head self-attention layer.

[0132] The feature sequence is dynamically weighted based on the attention weight matrix to obtain attention-weighted features;

[0133] Attention-weighted features are normalized and fused using residual connections and layer normalization units. The fusion result is then input into a feedforward neural network layer for nonlinear transformation to obtain single-round processing features.

[0134] The self-attention mechanism encoder block is processed a preset number of times, and the final output feature sequence is used as the fused feature sequence.

[0135] Specifically, the detailed implementation of the intent prediction module system:

[0136] This step can be broken down into two closely related sub-steps.

[0137] Multimodal feature extraction module: How to extract various valuable feature vectors from standardized data packets.

[0138] Temporal attention prediction model: How the intent prediction module analyzes these feature sequences and makes the final intent judgment.

[0139] Sub-step: Detailed implementation of the multimodal feature extraction module

[0140] The function of this submodule is to extract feature sequences of different dimensions but aligned in time from the input multimodal data packets in parallel, targeting the vehicle of interest.

[0141] System device diagram description, internal structure of the feature extraction module

[0142] The specific methods and steps of each feature extractor, and the target vehicle association unit:

[0143] First, the unit selects the target vehicle that needs the most attention at the moment (e.g., a vehicle about to enter the conflict zone) based on the "game object" specified by the decision planning module or automatically through an attention mechanism. Then, using the target ID provided by the preprocessing module, it associates all relevant data (position, speed, image region, etc.) of the target vehicle from the historical data cache over the past N time steps (e.g., the past 3 seconds, N=30 frames@10Hz) to form a time series data set.

[0144] Kinematic Feature Extractor:

[0145] Input a list of the target vehicle's motion states (position, velocity, acceleration, etc.) over the past N time steps [(pos_t,vel_t, acc_t), ...].

[0146] The extractor will calculate some more descriptive relative kinematic features.

[0147] For example, at each time step t, calculate:

[0148] The lateral and longitudinal distances, speed, and acceleration relative to the vehicle itself.

[0149] The lateral offset and directional angle difference relative to the centerline of its lane.

[0150] Output: At each time step t, output a kinematic feature vector Kin_vec_t.

[0151] Optional, driver behavior feature extractor.

[0152] In an enhanced embodiment of the present invention, to further improve the accuracy and confidence of intent prediction when conditions permit, the multimodal feature extraction module may also include a driver behavior feature extractor. This mainly involves the following steps:

[0153] A. Observability Assessment Unit

[0154] Input: Preprocessed panoramic video frames containing the target vehicle, as well as the target vehicle's 3D bounding box and distance information.

[0155] Specific steps:

[0156] 1. Occlusion and Reflection Detection: Run a lightweight image segmentation or classification model to determine whether there are severe physical occlusions (such as A-pillars or rearview mirrors) or strong ambient reflections in the windshield area.

[0157] 2. Resolution evaluation: Based on the distance to the target vehicle and the optical parameters of the camera, calculate the number of pixels occupied by the driver's head area in the image, and set a minimum pixel threshold (e.g., head height not less than 30 pixels).

[0158] 3. Lighting conditions assessment: Analyze the average brightness and contrast of the cockpit area to determine whether there is overexposure or severe underexposure.

[0159] Output: Boolean value – is_observable. Subsequent feature extraction steps will only be triggered if is_observable is True.

[0160] B. Driver behavioral feature extraction

[0161] When is_observable is True, the extractor will process the high-resolution cockpit RoI (Region of Interest) image provided by the preprocessing module.

[0162] Head posture estimation:

[0163] Input: A sequence of dynamic regions of interest images of the target vehicle's cockpit extracted from a high-resolution video stream.

[0164] Specific steps:

[0165] 1. Run a lightweight face or head detection module to locate the driver's head area.

[0166] 2. The cropped head image is input into a CNN model specifically designed for head pose regression. This model (e.g., based on ResNet-18 or a similar architecture) is pre-trained on a specialized head pose dataset (such as AFLW2000-3D) and its outputs are head pitch, head yaw, and head roll.

[0167] Key feature: Changes in head yaw are particularly critical. For example, a change in head yaw that rapidly turns from facing forward (0°) to the direction of the vehicle (e.g., -30°) is a strong signal that the driver is "checking for oncoming traffic," which is highly valuable for judging their intention to "give way" or "proceed."

[0168] C. Extractor Output

[0169] practice:

[0170] 1. If is_observable is True, the extractor will output a driver behavior feature vector Dri_vec_t containing the above features.

[0171] For example:

[0172] Dri_vec_t = [head_pitch, head_yaw, head_roll, gaze_direction_class] (where gaze_direction_class is the one-hot encoding of the classification result).

[0173] 2. If is_observable is False, in order to maintain the consistency of subsequent data processing, the extractor will output a vector with the same dimensions but all elements being zero.

[0174] Microscopic pose feature extractor:

[0175] Input the image cropping region of the target vehicle over the past N time steps (i.e., the bounding box of the vehicle in the video stream).

[0176] This is the key to the invention. The extractor integrates one or more pre-trained lightweight convolutional neural network (CNN) models.

[0177] (1) Estimation of vehicle front attitude:

[0178] A specialized CNN model (e.g., based on ResNet-18 or MobileNetV3 architecture and pre-trained on a large dataset with front-end pose annotations) is input to an image of the target vehicle's front end.

[0179] The output layer of this CNN is not for classification, but for regression, directly outputting two floating-point values: pitch and yaw. The changes of these two values ​​over time constitute a quantitative description of the car's "nodding" and "shaking" motion.

[0180] (2) Estimation of wheel steering angle:

[0181] Another CNN model (or a multi-task head of the same model) is specifically designed for vehicle keypoint detection. It locates the center point and contour points of the front wheels on the vehicle image.

[0182] By using geometric calculations, the steering angle of the wheels relative to the vehicle body is estimated based on the positions of these key points.

[0183] Output: At each time step t, output a micro-pose feature vector Pose_vec_t, which contains values ​​such as [pitch, yaw, wheel_angle].

[0184] Contextual Feature Extractor:

[0185] Input high-precision map data and the vehicle's location information.

[0186] This extractor retrieves static and semi-static information related to the current interaction scenario by querying a high-precision map database.

[0187] Scene type: Whether it is at an intersection, roundabout, ramp merging area, etc. (One-hot encoding can be performed).

[0188] Road geometry: the curvature, slope, and width of the lane where the target vehicle is located.

[0189] Traffic rules: Whether there are stop signs, yield signs, right-of-way, etc.

[0190] Output: At each time step t, output an environmental context feature vector Ctx_vec_t.

[0191] Feature sequence alignment and splicing unit:

[0192] The feature vectors Kin_vec_t, Pose_vec_t, Ctx_vec_t, etc., generated by the above extractors at each time step t are concatenated along the vector dimension to form a single comprehensive feature vector F_t that integrates multimodal information.

[0193] Final output: The comprehensive feature vectors F_{t-N+1}, ..., F_t from the past N time steps are arranged in chronological order to form a multimodal temporal feature sequence matrix X of shape [N, D] (see the example for an explanation of the [N, D] multimodal temporal feature sequence matrix X), where N is the sequence length and D is the total dimension of the concatenated feature vectors. This matrix X is the final input to our intent prediction module.

[0194] In some embodiments, traffic participant intent probability prediction is performed based on fused feature sequences to obtain intent prediction results including intent category and corresponding confidence level, including:

[0195] The fused feature sequence is compressed to obtain a single-dimensional global feature vector;

[0196] The global feature vector is input into a fully connected neural network, and the fully connected neural network performs semantic mapping on the global feature vector to obtain the original score of each intent category;

[0197] Apply the soft-maximization activation function to the raw scores to obtain the probability distribution of each intent category, which includes at least the following: proceed, yield, and hesitate.

[0198] The intent category with the highest probability in the probability distribution is extracted as the intent graph, where the probability value is used as the confidence level of the intent graph, and the normalized information entropy of the probability distribution is calculated to quantify the prediction uncertainty.

[0199] By integrating the intention graph, corresponding confidence level, probability distribution, and prediction uncertainty, the intention prediction result is obtained.

[0200] Specifically, the detailed implementation of the temporal attention prediction model is described in the sub-steps.

[0201] Internal structure of the temporal attention prediction model:

[0202] The Transformer architecture will be used as an example, as it excels in processing sequential data.

[0203] It should be noted that Transformer is not the focus of this invention, but is only used as an example in this embodiment.

[0204] Step 1, Input embedding layer:

[0205] The original D-dimensional feature space is mapped to a higher-dimensional d_model-dimensional representation space (e.g., d_model = 256), which is more suitable for model learning, and temporal information is injected into it.

[0206] Linear projection:

[0207] Define a weight matrix W_embed with dimensions [D, d_model]. Multiply the input matrix X by this weight matrix.

[0208] Mathematical representation:

[0209] Here, X_embed represents the embedded input tensor, which carries the core semantic information of the input data.

[0210] X represents the initial, original multimodal feature sequence matrix.

[0211] W_embed represents the weight matrix.

[0212] Result: A new matrix X_embed is obtained, with dimensions [N, d_model]. This step allows the model to learn how to combine the original D-dimensional features into a more meaningful internal representation of dimension d_model.

[0213] Location coding:

[0214] Since the Transformer itself is unordered, it cannot distinguish between the first and tenth rows. Therefore, it must be explicitly told the position of each row (each time step).

[0215] Create a position encoding matrix P with the same dimensions [N, d_model] as X_embed. This matrix is ​​pre-computed and is typically generated using sine and cosine functions. Its values ​​depend only on the position (row index pos, from 0 to N-1) and the dimension (column index i, from 0 to d_model-1).

[0216]

[0217]

[0218]

[0219]

[0220]

[0221] Mathematical representation: X_final_input = X_embed + P

[0222] Here, X_final_input represents the model input matrix, which is the input tensor that is finally fed into the Transformer encoder block.

[0223] X_embed represents the embedded input tensor.

[0224] P represents the position encoding matrix, or position encoding tensor, which integrates semantic and temporal information and is used to inject temporal or positional information into subsequent input sequences.

[0225] Result: The final model input matrix X_final_input is obtained, with dimensions still [N, d_model]. Now, each row vector contains both semantic information (from features) and temporal information (from positional encoding).

[0226] Step 2, Transformer encoder block:

[0227] This is the core of the model, and it repeats L times. Currently, we are only analyzing the internal workings of one block.

[0228] The input is X_final_input (or the output of the previous block), and the output is a new matrix of the same dimensions [N, d_model] that has been refined once.

[0229] Multi-head self-attention mechanism: "Cross-compare all clues to discover key connections":

[0230] Objective: To enable each time step (each row) in the sequence to "see" all other time steps and to calculate which time steps are most important for understanding the current time step.

[0231] (1) Generate Q, K, V vectors: Define three weight matrices Their dimensions are all [d_model, d_k], where d_k is the dimension of the Q and K vectors (usually set to d_model / h, where h is the number of heads). The output Z of the previous layer (dimension [N, d_model]) is multiplied by these three matrices respectively to obtain the query matrix Q, the key matrix K, and the value matrix V. (Note: To use a unified language to describe the computation process inside any layer (k-th layer) of the Encoder Block, the common notation Z is used to represent "the input of the current block". In this step...) )

[0232]

[0233]

[0234]

[0235] (2) Calculate the attention score: multiply the query matrix Q by the transpose K^T of the key matrix K.

[0236] Mathematical representation:

[0237] Result: A score matrix of [N, N] Scores is obtained. Each element Scores[i, j] in the matrix represents the degree of "correlation" or "match" between the i-th time step and the j-th time step when the latter two are "queried". This is the core process of the model's "thinking".

[0238] (3) Scaling:

[0239] Scaling: Divide all elements in the Scores matrix by sqrt(d_k). This is to prevent the dot product from becoming too large, which would cause the gradient of the subsequent Softmax function to become too small, thus stabilizing training.

[0240] (4) Calculate attention weights: Apply the Softmax function to the scaled score matrix row by row. That is, first scale the values ​​of each row of the "score matrix" uniformly, and then perform a Softmax operation on each row separately to make all elements in that row become a probability distribution with a sum of 1.

[0241] Mathematical representation:

[0242] Result: An attention weight matrix A of [N, N] is obtained. Each row of this matrix is ​​a probability distribution, and the row sum is 1. The value of A[i, j] represents how much information should be "drawn" from the V vector at time step j when updating the representation at time step i. This matrix A is the source of interpretability in this patent and can be visualized.

[0243] (5) Weighted summation to obtain the output: Multiply the attention weight matrix A with the value matrix V.

[0244] Mathematical representation:

[0245] Result: A matrix of [N, d_v] is obtained. Each row... All are weighted averages of all V vectors in the entire sequence, with the weights determined by the i-th row of the attention matrix A. This means that the new representation at each time step is a dynamic and focused "reorganization" of the entire sequence information based on its correlation with all other time steps.

[0246] Multi-Head: The process described in steps 1-5 above is executed in parallel h times (where h is the number of heads, e.g., 8). Each "head" has its own independent... Weight matrix. In this way, each head can learn a different attention pattern (e.g., one head focuses on velocity changes, and another head focuses on posture changes).

[0247] Finally, output the h heads. (Dimensions [N, d_v]) are concatenated along the feature dimensions to obtain a larger matrix [N, h * d_v]. Then, a linear transformation (multiplying by the weight matrix) is applied... The dimension [h*d_v, d_model] is projected back to the dimension [N, d_model].

[0248] Residual connectivity and layer normalization:

[0249]

[0250] Here, Z represents the input tensor of the current Transformer Encoder Block. N is the sequence length (number of time steps), and d_model is the feature dimension within the model. If this is the first Encoder Block, then Z is X_final_input after input embedding and position encoding. If it is a subsequent block, Z is the output of the previous block. It represents the feature representation of each position (time step) in the sequence at the current processing stage.

[0251] MultiHeadAttention(Z) represents the output tensor obtained after input Z is fed into a multi-head self-attention mechanism.

[0252] Z + MultiHeadAttention(Z) represents the residual connection operation. It adds the input Z of the multi-head attention layer to its output MultiHeadAttention(Z) element-wise.

[0253] LayerNorm represents the layer normalization operation. It is used for the results after residual connections. LayerNorm normally normalizes each row of the [N, d_model] matrix (i.e., the d_model dimensional eigenvector at each time step) independently.

[0254] Sublayer_Output_1: The final output after processing by the first sublayer (attention layer + Add & Norm). It will serve as the input to the next sublayer (feedforward neural network).

[0255] Objective: By directly adding the input Z of multi-head attention to its output, a "shortcut" is formed, greatly facilitating the backpropagation of gradients and avoiding the problems caused by excessive network depth. LayerNorm normalizes the features of each sample, accelerating convergence.

[0256] Feedforward Neural Networks:

[0257] Perform a simple two-layer fully connected network transformation on Sublayer_Output_1.

[0258]

[0259] Here, Sublayer_Output_1 represents the output of the previous step, that is, the output of the first sublayer.

[0260] W_1 represents the weight matrix of the first fully connected layer in FFN.

[0261] b_1 represents the bias vector of the first fully connected layer.

[0262] ReLU(...) represents the modified linear unit activation function.

[0263] W_2 represents the weight matrix of the second fully connected layer in FFN.

[0264] b_2 represents the bias vector of the second fully connected layer.

[0265] FFN_Output represents the output after processing through the entire feedforward neural network sublayer.

[0266] Objective: To provide the model with nonlinear transformation capabilities and increase its representational power. The dimensions of W_1 are typically [d_model, d_ff], and the dimensions of W_2 are [d_ff, d_model], where d_ff is the dimension of an intermediate layer, which is usually larger than d_model.

[0267] Second residual connection and layer normalization:

[0268]

[0269] Here, Sublayer_Output_1 represents the input of the second sublayer (FFN), which is the output of the first sublayer.

[0270] FFN_Output represents the output of FFN, the result calculated by the previous formula.

[0271] Sublayer_Output_1 + FFN_Output represents the second residual join. It adds the input Sublayer_Output_1 of FFN element-wise with its output FFN_Output.

[0272] EncoderBlock_Output represents the final output of a complete Transformer Encoder Block.

[0273] Result: The final output of a complete Transformer Encoder Block is obtained, with dimensions still [N, d_model]. This output will be used as the input of the next Encoder Block, and the above process will be repeated L times.

[0274] Step 3: Output Layer

[0275] Objective: To aggregate the sequence representations, which are refined by the L-layer encoder and contain rich contextual information, into a final intent classification result.

[0276] Sequence information aggregation:

[0277] The sequence information of [N, d_model] is compressed into a single vector that represents the entire sequence. Common methods include:

[0278] Global average pooling: Averages the N d_model dimensional row vectors element by element to obtain a vector of [1, d_model].

[0279] Using a [CLS] token: A special [CLS] (Classification) symbol is added to the beginning of the input sequence. After passing through all Encoder Blocks, only the final output vector (dimension [1, d_model]) corresponding to this [CLS] symbol is taken as the representation of the entire sequence. In practice, this method usually performs better.

[0280] Final classifier:

[0281] The convergence vector [1, d_model] obtained in the previous step is input into a simple fully connected neural network (also known as an MLP).

[0282]

[0283] Aggregated_Vector represents a single feature vector that can represent the entire input sequence after the information aggregation step.

[0284] W_final represents the weight matrix of the final classifier.

[0285] b_final represents the bias vector of the final classifier.

[0286] Logits represent the original output score vector after linear transformation, also known as "logic values," and are the inputs to the Softmax activation function in the formula below.

[0287] The dimensions of W_final are [d_model, num_classes], where num_classes is the number of intent categories of the model (e.g., 3, corresponding to "go through", "give way", "hesitate").

[0288] Result: A vector Logits of [1, num_classes] is obtained.

[0289] Softmax activation function:

[0290] Apply the Softmax function to the Logits vector.

[0291]

[0292] Probabilities represent probability distribution vectors.

[0293] The final output is a probability distribution vector of [1, num_classes]. Each element in the vector represents the probability that the model predicts the sequence corresponds to the corresponding intent. This represents a complete prediction for an input sequence (i.e., an N x D matrix X). In other words, [1, num_classes] is a "data template" defined for the model output, specifying that the output must be a row vector containing num_classes elements.

[0294] For example, the model's final output is [0.1, 0.8, 0.1] (corresponding to "go through", "yield", and "hesitate"), indicating that the model has an 80% confidence level that the target vehicle's intention is to "yield".

[0295] In some embodiments, the intent prediction result is transmitted to the decision planning module of the autonomous driving system. The decision planning module obtains a vehicle behavior decision strategy adapted to the current traffic scenario based on the intent category and corresponding confidence level, including:

[0296] When the intention graph in the intention prediction result is to give way and the confidence level is greater than the first preset threshold and the prediction uncertainty is less than the first preset entropy value, the decision planning module will constrain the motion of the target vehicle to a deceleration or stopping model and generate the opportunistic passage trajectory with the best time.

[0297] When the intention graph in the intention prediction result is passable and the confidence level is greater than the first preset threshold, the decision planning module regards the target vehicle as a dynamic obstacle with right-of-way and generates a defensive waiting trajectory to decelerate to a stop based on the collision time and safe distance model.

[0298] When the intention prediction result shows hesitation or the prediction uncertainty is greater than the second preset entropy value, the decision planning module starts an interactive subroutine to generate a short-term, low-speed and suspendable active probing action, and triggers the intention prediction model to perform intensive resampling and re-evaluation of the target vehicle's response, and adjusts the decision strategy based on the re-evaluation result.

[0299] In some embodiments, an interactive subroutine is initiated to generate short-duration, low-speed, and abortable proactive probing actions, and triggers an intent prediction model to intensively resample and re-evaluate the target vehicle's response, adjusting the decision-making strategy based on the re-evaluation results, including:

[0300] Set the probe action parameters, which include at least the probe speed, probe displacement and duration. Based on the probe action parameters, plan the short-term movement trajectory of the autonomous vehicle. The speed of the short-term movement trajectory shall not exceed the preset low speed threshold and the displacement shall not exceed the preset displacement threshold.

[0301] During the process of autonomous vehicles performing exploratory actions, the control intent prediction model increases the data sampling frequency to a preset high-frequency sampling rate, intensively collects and processes the multimodal data of the target vehicle, and generates high-frequency updated intent prediction results.

[0302] If the intention graph in the frequently updated intention prediction results changes to yielding and the confidence level meets the first preset threshold, switch to opportunistic passage trajectory planning;

[0303] If the main graph becomes passable and the confidence level meets the first preset threshold, switch to defensive waiting trajectory planning;

[0304] If the situation remains hesitant or the uncertainty of the forecast has not decreased, repeat the probing action or switch to the most conservative parking and waiting strategy.

[0305] Specifically, the intended output is the integration and application in advanced driving systems.

[0306] The objective of this invention is to seamlessly integrate the high-level semantic intent information generated above into an autonomous driving system. The core of this step lies in defining the data interface output by this invention and explaining how downstream decision-making and planning modules can utilize this information to optimize their behavioral strategies.

[0307] The interaction process between intent information and decision-making planning, and the specific practices and steps of each unit.

[0308] Intent parsing and uncertainty quantification unit:

[0309] Input: The intent probability distribution vector P = [p_1, p_2, ..., p_num_classes] output by the core model of this invention.

[0310] practice:

[0311] (1) Extraction of the highest probability intent and confidence assessment: The argmax function is used to find the intent category with the highest probability as Dominant_Intent, and the probability value itself is Confidence.

[0312] (2) Uncertainty quantification: The key to this unit is to accurately quantify the uncertainty of model predictions, so as to provide risk measurement for decision-making.

[0313] Normalized information entropy is used as the core indicator, and the calculation formula is as follows:

[0314]

[0315] Here, Uncertainty represents uncertainty (or normalized entropy), which measures the degree of "uncertainty" the model has about the prediction results.

[0316] K represents the total number of classes (num_classes), which is the total number of all possible classes in the classification task.

[0317] Pk represents the probability that the model predicts the class to be class k.

[0318] The normalized entropy value ranges from 0 to 1, where 0 represents extreme certainty and 1 represents complete uncertainty (uniform distribution), which facilitates the setting of decision thresholds downstream.

[0319] Output: Generate a standardized, structured intent information package, which is published to the decision planning module via an internal bus (such as a Robot Operating System Topic (ROS Topic) or Data Distribution Service (DDS)). This information package must contain at least:

[0320] dominant_intent: e.g., "Yield".

[0321] Confidence level: e.g., 0.8

[0322] Uncertainty: e.g., 0.3 (normalized entropy value)

[0323] full_distribution (full probability distribution): [0.1, 0.8, 0.1] (complete probability distribution for use in more complex decision-making logic).

[0324] Decision-making and planning module:

[0325] This directly embodies the value of the invention. Its behavioral decision-making layer upgrades traditional physical rule-based decision-making to intelligent decision-making based on social cognition.

[0326] The inputs to the behavioral decision layer include: the intent information package provided by this invention, the environmental model (information from the perception module about road structure, static obstacles, traffic light status, etc.), and traffic rules (whether there are stop signs, yield signs, right-of-way, etc.).

[0327] Scenario: This vehicle is preparing to turn left at an unprotected intersection, with oncoming traffic going straight.

[0328] Strategy 1: Opportunistic Passage Strategy

[0329] Triggering conditions: The received intent information packet satisfies intent == "Yield" and confidence > 0.9 and uncertainty < 0.15.

[0330] Implementation: Under this condition, the planner constrains the predicted trajectory of oncoming vehicles to an explicit deceleration or stopping model. This simplifies the constraint space of the planning problem, enabling it to quickly solve a time-optimal and comfortable travel trajectory without unnecessary deceleration or waiting, thereby improving traffic efficiency.

[0331] Strategy 2: Defensive waiting strategy.

[0332] Triggering conditions: intent == "Go" and confidence > 0.9.

[0333] Implementation: The planner treats oncoming vehicles as dynamic obstacles with absolute right-of-way that cannot be infringed upon. The generated trajectory will plan a smooth deceleration to stop within the lane while ensuring an absolute safe distance (e.g., based on Time-of-Collision (TTC) or Distance-of-Safety (DSM) model).

[0334] Strategy 3: Proactive exploration and testing strategy, reflecting core intelligence.

[0335] Triggering conditions: intent == "Hesitate" or uncertainty > 0.7 (i.e. the model explicitly reports "hesitate" or is extremely unconfident in any judgment).

[0336] Implementation: Unlike traditional planners, the behavioral decision layer will initiate an interactive subroutine:

[0337] (1) Planning small movements: The trajectory generation layer no longer plans a complete path, but generates a short-duration, low-speed "probe" movement that can be stopped at any time, such as moving forward 0.5 meters at a speed of 0.5 m / s. The physical intention of this movement is to convey a clear interactive signal to the other driver that requires a response.

[0338] (2) Triggering synchronous re-evaluation: While planning this action, the system will send a high-priority request to the intent prediction module of the present invention, forcing it to perform intensive resampling and re-evaluation of the target vehicle's reaction in the next 1-2 seconds.

[0339] (3) Closed-loop decision-making: The decision-making system waits for the updated intent information packet returned after re-evaluation. If the new intent becomes clear (e.g., the other party is prompted by the "probing" action and clearly slows down to give way), it switches to strategy one or strategy two. If it is still uncertain, it can repeat the probing once or finally switch to the most conservative defensive waiting strategy. This closed loop of "planning-action-perception-replanning" is the key to enabling the system of this invention to achieve advanced social interaction.

[0340] It is worth noting that the three strategies proposed in this invention—opportunistic passage, defensive waiting, and proactive probing—are merely typical examples used to clearly and intuitively illustrate how the intent prediction results of this invention fundamentally change the behavioral pattern of decision-making and planning, aiming to reveal a novel design paradigm based on social cognition. In practical, more complex autonomous driving systems, based on the intent prediction capabilities of this invention, a much richer and more granular set of intelligent decision-making strategies can be derived.

[0341] Vehicle control module:

[0342] Role positioning: As a downstream actuator of the technical solution of this invention, it is not an innovative point of this invention.

[0343] Interface and Implementation: This module receives the target spatiotemporal trajectory (typically represented as a series of path points with pose, velocity, acceleration, and curvature constraints) output by the decision planning module. To accurately track this trajectory, this module can employ any advanced control algorithm. Its implementation is not limited to classic MPC (Model Predictive Control) or LQR (Linear Quadratic Regulator), but can also utilize a deep learning-based end-to-end controller capable of learning more complex vehicle dynamics and more human-like driving styles. The effectiveness of this invention does not depend on the selection of specific control technologies in the vehicle control module; its output high-level semantic information has guiding value and performance-enhancing effects for any advanced controller.

[0344] The specific implementation method describes the formation process of the multimodal temporal feature sequence matrix X.

[0345] Assumption:

[0346] The system's sampling frequency is 10Hz (i.e., one data packet is generated every 0.1 seconds).

[0347] The set historical backtracking time window length N is 30 (i.e., analyzing behavior over the past 3 seconds).

[0348] Therefore, this matrix X will have 30 rows. Each row represents a time step (t-29, t-28, ..., t).

[0349] If at any time step t, each feature extractor produces the following feature vector:

[0350] The kinematic feature vector Kin_vec_t (assuming a dimension of 8).

[0351] The extractor calculates:

[0352] * Longitudinal distance relative to this vehicle (1D)

[0353] * Lateral distance relative to this vehicle (1D)

[0354] * Relative longitudinal velocity of the vehicle (1-dimensional)

[0355] * Lateral velocity relative to the vehicle (1-dimensional)

[0356] * Longitudinal acceleration relative to the vehicle (1-dimensional)

[0357] * Lateral acceleration relative to the vehicle (1-dimensional)

[0358] * Lateral offset from the lane centerline (1D)

[0359] * Direction angle difference from the lane centerline (1D)

[0360] Then: Kin_vec_t = [d_lon, d_lat, v_lon, v_lat, a_lon, a_lat, lane_off,angle_diff]

[0361] Microscopic pose feature vector Pose_vec_t (assuming a dimension of 3).

[0362] The extractor is obtained through CNN regression and computation:

[0363] * Pitch angle (1D)

[0364] * Yaw angle (1D)

[0365] * Estimated wheel angle (1D)

[0366] Then: Pose_vec_t = [pitch, yaw, wheel_angle].

[0367] The environmental context feature vector Ctx_vec_t (assuming a dimension of 5).

[0368] The extractor retrieves the following from the high-precision map:

[0369] * Is it at an intersection? (One-hot encoding: Yes = 1, No = 0) (1D)

[0370] * Is it in a roundabout? (One-hot encoding: Yes = 1, No = 0) (1D)

[0371] * Is there a stop flag? (One-hot encoding: Yes = 1, No = 0) (1D)

[0372] * Lane curvature (1D)

[0373] * Lane slope (1D)

[0374] Then: Ctx_vec_t = [is_intersection, is_roundabout, has_stop_sign,curvature, slope].

[0375] Now, the task of the feature sequence alignment and splicing unit at time step t is to splice these feature vectors at the same time laterally to form a longer comprehensive feature vector F_t.

[0376] F_t = [Kin_vec_t, Pose_vec_t, Ctx_vec_t]

[0377] F_t =[d_lon, d_lat, ..., angle_diff, pitch, yaw, wheel_angle, is_intersection, ..., slope]

[0378] The total dimension D of this comprehensive feature vector F_t is the sum of the dimensions of all subvectors: D = 8 + 3 + 5 = 16.

[0379] Finally, the combined feature vectors F_t from the past N=30 time steps are stacked vertically to form the final input matrix X.

[0380] <--------------------- Dimension D = 16 --------------------->

[0381]

[0382] Matrix X has a shape of [30, 16], or [N, D]. In other words, matrix X is a highly structured and information-rich mathematical object that fully encapsulates the multi-dimensional behavioral history of the target vehicle over a period of time, and can be efficiently processed and analyzed by the Transformer model.

[0383] In other embodiments, a traffic participant intention prediction device based on multimodal temporal attention is provided, characterized in that a traffic participant intention prediction method based on multimodal temporal attention, based on any one of the preceding claims, includes:

[0384] The data acquisition module is configured to collect multimodal raw traffic data in traffic scenarios.

[0385] The preprocessing module is configured to preprocess the raw multimodal traffic data to obtain standardized multimodal data.

[0386] The feature extraction module is configured to extract features from standardized multimodal data to obtain multidimensional feature sequences;

[0387] The weighted fusion module is configured to dynamically weight and fuse multi-dimensional feature sequences and learn semantic associations through a temporal attention model to obtain a fused feature sequence.

[0388] The probability prediction module is configured to predict the probability of traffic participants' intentions based on the fused feature sequence, and obtain the intention prediction results including the intention category and the corresponding confidence level.

[0389] The transmission module is configured to transmit the intent prediction results to the decision planning module of the autonomous driving system. The decision planning module obtains a vehicle behavior decision strategy adapted to the current traffic scenario based on the intent category and corresponding confidence level.

[0390] In other embodiments, a terminal is provided, characterized in that it includes a processor, an input device, an output device, and a memory, the processor, the input device, the output device, and the memory being interconnected, wherein the memory is used to store a computer program, the computer program including program instructions, and the processor is configured to invoke the program instructions to execute a traffic participant intent prediction method based on multimodal temporal attention as described above.

[0391] In other embodiments, a computer-readable storage medium is provided, characterized in that the computer-readable storage medium stores a computer program, the computer program including program instructions, which, when executed by a processor, cause the processor to perform a traffic participant intent prediction method based on multimodal temporal attention as described above.

[0392] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.

Claims

1. A method for predicting traffic participant intentions based on multimodal temporal attention, characterized in that, include: Collect multimodal raw traffic data in traffic scenarios, wherein the multimodal raw traffic data includes at least visual image data, kinematic data, and environmental context data of the target vehicle; The original multimodal traffic data is preprocessed to obtain standardized multimodal data; Feature extraction is performed on the standardized multimodal data to obtain a multidimensional feature sequence; A fused feature sequence is obtained by dynamically weighting and fusing multi-dimensional feature sequences and learning semantic associations through a temporal attention model. Based on the fused feature sequence, the probability of traffic participant intent is predicted to obtain an intent prediction result that includes intent category and corresponding confidence level; The intent prediction results are transmitted to the decision planning module of the autonomous driving system. The decision planning module obtains a vehicle behavior decision strategy adapted to the current traffic scenario based on the intent category and corresponding confidence level. The fused feature sequence is compressed to obtain a single-dimensional global feature vector; The global feature vector is input into a fully connected neural network, and the fully connected neural network performs semantic mapping on the global feature vector to obtain the original score of each intent category; Apply the soft-maximization activation function to the raw scores to obtain the probability distribution of each intent category, which includes at least the following: proceed, yield, and hesitate. The intent category with the highest probability in the probability distribution is extracted as the intent graph, where the probability value is used as the confidence level of the intent graph, and the normalized information entropy of the probability distribution is calculated to quantify the prediction uncertainty. By integrating the idea graph, corresponding confidence level, probability distribution, and prediction uncertainty, the intention prediction result is obtained. When the intention graph in the intention prediction result is to give way and the confidence level is greater than the first preset threshold and the prediction uncertainty is less than the first preset entropy value, the decision planning module will constrain the motion of the target vehicle to a deceleration or stopping model and generate the opportunistic passage trajectory with the best time. When the intention graph in the intention prediction result is passable and the confidence level is greater than the first preset threshold, the decision planning module regards the target vehicle as a dynamic obstacle with right-of-way and generates a defensive waiting trajectory to decelerate to a stop based on the collision time and safe distance model. When the intention graph in the intention prediction result is hesitant or the prediction uncertainty is greater than the second preset entropy value, the decision planning module starts an interactive subroutine to generate a short-term, low-speed and suspendable active probing action, and triggers the intention prediction model to perform intensive resampling and re-evaluation of the target vehicle's response, and adjusts the decision strategy based on the re-evaluation results. Set the probe action parameters, which include at least the probe speed, probe displacement and duration. Based on the probe action parameters, plan the short-term movement trajectory of the autonomous vehicle. The speed of the short-term movement trajectory shall not exceed the preset low speed threshold and the displacement shall not exceed the preset displacement threshold. During the process of autonomous vehicles performing exploratory actions, the control intent prediction model increases the data sampling frequency to a preset high-frequency sampling rate, intensively collects and processes the multimodal data of the target vehicle, and generates high-frequency updated intent prediction results. If the intention graph in the frequently updated intention prediction results changes to yielding and the confidence level meets the first preset threshold, switch to opportunistic passage trajectory planning; If the main graph becomes passable and the confidence level meets the first preset threshold, switch to defensive waiting trajectory planning; If the situation remains hesitant or the uncertainty of the forecast has not decreased, repeat the probing action or switch to the most conservative parking and waiting strategy.

2. The traffic participant intent prediction method based on multimodal temporal attention according to claim 1, characterized in that, The original multimodal traffic data is preprocessed to obtain standardized multimodal data, including: Timestamps are added to each type of data in the multimodal raw traffic data. Based on the timestamps, the multi-type data with different sampling probabilities are aligned to a unified time axis to complete data synchronization. The synchronized visual image data and kinematic data are converted to a unified vehicle coordinate system with the rear axle center of the autonomous vehicle as the origin, and the coordinate system is calibrated. Distortion correction and color space conversion are performed on the calibrated visual image data, noise filtering and outlier removal are performed on the kinematic data, and format standardization is performed on the environmental context data. The processed data of various types are integrated to obtain standardized multimodal data.

3. The traffic participant intent prediction method based on multimodal temporal attention according to claim 1, characterized in that, Feature extraction is performed on the standardized multimodal data to obtain a multidimensional feature sequence, including: For the visual image data in the standardized multimodal data, the image region corresponding to the target vehicle is extracted, input into the convolutional neural network model, and the pitch angle, yaw angle and wheel steering angle of the target vehicle are regressed respectively to obtain the micro attitude feature sequence. For the kinematic data in the standardized multimodal data, calculate the lateral distance, longitudinal distance, lateral velocity, longitudinal velocity, lateral acceleration, longitudinal acceleration, and the lateral offset and directional angle difference of the target vehicle relative to the center line of the lane, and obtain the kinematic feature sequence. For the environmental context data in the standardized multimodal data, the map is queried to obtain the type identifier, road geometric parameters and traffic rule information of the current scene. The type identifier and traffic rule information are encoded and processed, and combined with the road geometric parameters to obtain the environmental context feature sequence. The microscopic posture feature sequence, kinematic feature sequence, and environmental context feature sequence are aligned by time steps to obtain a multi-dimensional feature sequence.

4. The traffic participant intent prediction method based on multimodal temporal attention according to claim 1, characterized in that, By using a temporal attention model to dynamically weight and fuse multi-dimensional feature sequences and learn semantic associations, a fused feature sequence is obtained, including: Linear projection is performed on the multi-dimensional feature sequence to map each dimension feature to a feature space of a preset dimension, thereby obtaining an embedded feature sequence. The position code is added to the embedded feature sequence, and temporal information is injected to obtain an embedded feature sequence with temporal information; The embedded feature sequence with temporal information is input into the self-attention mechanism encoder block, wherein the self-attention mechanism encoder block includes a multi-head self-attention layer, a feedforward neural network layer, and a residual connection and layer normalization unit. The attention weight matrix is ​​obtained by calculating the correlation score of the features at each time step in the feature sequence through the multi-head self-attention layer. The feature sequence is dynamically weighted based on the attention weight matrix to obtain attention-weighted features; The attention-weighted features are normalized and fused using the residual connection and layer normalization unit. The fusion result is then input into the feedforward neural network layer for nonlinear transformation to obtain single-round processing features. The self-attention mechanism encoder block is processed a preset number of times, and the final output feature sequence is used as the fused feature sequence.

5. A traffic participant intention prediction device based on multimodal temporal attention, characterized in that, A traffic participant intent prediction method based on multimodal temporal attention, according to any one of claims 1 to 4, includes: The data acquisition module is configured to collect multimodal raw traffic data in traffic scenarios. The preprocessing module is configured to preprocess the original multimodal traffic data to obtain standardized multimodal data. The feature extraction module is configured to extract features from the standardized multimodal data to obtain a multidimensional feature sequence; The weighted fusion module is configured to dynamically weight and fuse multi-dimensional feature sequences and learn semantic associations through a temporal attention model to obtain a fused feature sequence. The probability prediction module is configured to predict the probability of traffic participants' intentions based on the fused feature sequence, and obtain an intention prediction result that includes the intention category and the corresponding confidence level. The transmission module is configured to transmit the intent prediction result to the decision planning module of the autonomous driving system, wherein the decision planning module obtains a vehicle behavior decision strategy adapted to the current traffic scenario based on the intent category and corresponding confidence level.

6. A terminal, characterized in that, The system includes a processor, an input device, an output device, and a memory, which are interconnected. The memory stores a computer program, which includes program instructions. The processor is configured to invoke the program instructions to execute a traffic participant intent prediction method based on multimodal temporal attention as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform a traffic participant intent prediction method based on multimodal temporal attention as described in any one of claims 1 to 4.

Images