A target behavior prediction method and related devices

Abstract

The present application discloses a target behavior prediction method and related equipment, which relates to the field of target prediction technology. The method includes: obtaining multimodal historical behavior data and real-time environmental semantic information of the target object; determining key causal event nodes based on the multimodal historical behavior data and the spatiotemporal causal model; determining a dynamic causal graph based on the key causal event nodes and the real-time environmental semantic information; determining future potential event types and corresponding event occurrence probabilities based on the dynamic causal graph; determining event triggering constraints based on the event occurrence probabilities; determining node weight update parameters of the dynamic causal graph based on an incremental learning framework and real-time input multimodal historical behavior data; and determining the predicted behavior trajectory of the target object based on the event triggering constraints and the updated dynamic causal graph. The present application improves the accuracy and environmental adaptability of target behavior prediction by integrating multimodal data with spatiotemporal causal reasoning.

Description

Technical Field

[0001] The present application relates to the field of target prediction technology, and in particular to a target behavior prediction method and related equipment. Background Art

[0002] In the field of target behavior prediction, existing technologies often rely on static environment modeling and simple time-series analysis of historical behavior data. Traditional methods typically employ prediction frameworks based on rules or statistical models. For example, in autonomous driving scenarios, the sudden behavior of pedestrians or vehicles often leads to prediction errors, while existing models lack the ability to deeply model the causal relationship between environmental changes and target behavior. Therefore, a target behavior prediction method is urgently needed to address the aforementioned technical issues. Summary of the Invention

[0003] The Summary of the Invention introduces a series of simplified concepts that will be further described in the Detailed Description of the Invention. The Summary of the Invention of this application is not intended to limit the key features and essential technical features of the claimed technical solution, nor is it intended to determine the scope of protection of the claimed technical solution.

[0004] In a first aspect, the present application provides a target behavior prediction method, the method comprising:

[0005] Obtain multimodal historical behavior data and real-time environmental semantic information of the target object;

[0006] Determine key causal event nodes based on multimodal historical behavior data and spatiotemporal causal models;

[0007] Determine a dynamic causal graph containing environmental interaction relationships based on key causal event nodes and real-time environmental semantic information;

[0008] Based on the dynamic causal diagram, determine the types of potential future events and the corresponding probability of occurrence;

[0009] Determine event triggering constraints based on event occurrence probability;

[0010] Determine the node weight update parameters of the dynamic causal graph based on the incremental learning framework and real-time input multimodal historical behavior data;

[0011] Based on the event triggering constraints and the updated dynamic causal graph, the predicted behavior trajectory of the target object is determined.

[0012] In some embodiments, obtaining multimodal historical behavior data and real-time environmental semantic information of a target object includes:

[0013] Determine the target object's movement speed and direction angle data based on the detection signal of the millimeter-wave radar;

[0014] Based on the image acquisition results of the visual sensor, determine the contour features of the target object and the local environment image;

[0015] Determine a three-dimensional environment topology map based on the laser radar point cloud data, wherein the three-dimensional environment topology map includes movement direction annotation information of dynamic obstacles;

[0016] Based on the preset timestamp synchronization rules, the motion speed, directional angle data, contour features, local environment images and 3D environment topology maps are aligned to determine the multimodal historical behavior data;

[0017] Based on the motion direction annotation information, the real-time environment semantic information is determined.

[0018] In some embodiments, determining key causal event nodes based on multimodal historical behavior data and a spatiotemporal causal model includes:

[0019] Based on the time series characteristics of multimodal historical behavior data, multiple historical time segments are determined;

[0020] Based on the event detection rules in the spatiotemporal causal model, feature extraction is performed on each historical time segment to determine speed mutation points, direction turning points, and environmental interaction events;

[0021] Based on the preset causal association threshold, speed mutation points, direction turning points and environmental interaction events are marked to determine the set of candidate causal event nodes;

[0022] Based on the Granger causality test algorithm in the spatiotemporal causal model, the candidate causal event node set is eliminated and the key causal event nodes are determined.

[0023] In some embodiments, determining a dynamic causal graph including environmental interaction relationships based on key causal event nodes and real-time environmental semantic information includes:

[0024] Based on the key causal event nodes, determine the event node set;

[0025] Determine the movement direction and position of dynamic obstacles based on real-time environmental semantic information;

[0026] Determine the environmental interaction weight based on the event node set, motion direction, and position, where the environmental interaction weight is used to characterize the strength of the association between the event node and the dynamic obstacle;

[0027] Determine the initial weighted causal graph based on the event node set and the environment interaction weights;

[0028] Based on a preset causal redundancy threshold, the connection edges in the initial causal graph whose weights are less than the causal redundancy threshold are removed to determine the dynamic causal graph.

[0029] In some embodiments, determining the types of potential future events and the corresponding probabilities of occurrence of the events based on the dynamic causal graph includes:

[0030] Determine a set of candidate event types based on a set of event nodes in a dynamic causal graph;

[0031] Based on the connection weights of each event node in the dynamic causal graph, the node activation probability of each candidate event type is determined;

[0032] Based on the preset probability threshold, the node activation probability is screened and the candidate event types that meet the preset probability threshold are determined as future potential event types;

[0033] Based on the node activation probability corresponding to the future potential event type, the event occurrence probability is determined, where the event occurrence probability is positively correlated with the node activation probability.

[0034] In some embodiments, determining an event triggering constraint based on the event occurrence probability includes:

[0035] Based on the probability of event occurrence, determine the probability distribution parameters of each potential future event type;

[0036] Determine the time window and spatial area for event triggering based on preset triggering rules and probability distribution parameters;

[0037] Determine the dynamic threshold conditions for event triggering based on the time window and spatial region, where the dynamic threshold conditions include the triggering time interval, the triggering spatial boundary and the minimum probability threshold;

[0038] Based on dynamic threshold conditions and event occurrence probability, event triggering constraints are determined.

[0039] In some embodiments, determining node weight update parameters of a dynamic causal graph based on an incremental learning framework and real-time input multimodal historical behavior data includes:

[0040] Determine the incremental input data set based on multimodal historical behavior data input in real time;

[0041] Based on the distribution difference detection algorithm in the incremental learning framework, the distribution difference value between the incremental input data set and the historical data set is determined;

[0042] Determining a model update trigger signal based on a preset distribution difference threshold and a distribution difference value;

[0043] Determine the knowledge transfer parameters of the online knowledge distillation model based on the model update trigger signal and the incremental input dataset;

[0044] Based on the knowledge transfer parameter and the current node weight of the dynamic causal graph, a node weight update parameter is determined, wherein the node weight update parameter is used to adjust the association strength between the event node and the dynamic obstacle in the dynamic causal graph.

[0045] In some embodiments, determining a predicted behavior trajectory of a target object based on the event triggering constraint and the updated dynamic causal graph includes:

[0046] Determine multiple candidate trajectory branches based on time windows and spatial regions;

[0047] Determine the posterior probability of each candidate trajectory branch based on the node weights in the updated dynamic causal graph;

[0048] Based on the maximum value of the posterior probability, the main prediction trajectory is determined;

[0049] Determine the trajectory conflict probability based on the main predicted trajectory and the dynamic obstacle positions in the real-time environment semantic information;

[0050] Determining a trajectory correction vector based on a trajectory conflict probability and a preset conflict threshold;

[0051] Based on the trajectory correction vector and the preset smoothing algorithm, the predicted behavior trajectory of the target object is determined.

[0052] In a second aspect, the present application proposes a target behavior prediction device, comprising:

[0053] A target object data acquisition unit, used to acquire the target object's multimodal historical behavior data and real-time environmental semantic information;

[0054] The key event node determination unit determines the key causal event nodes based on multimodal historical behavior data and spatiotemporal causal models;

[0055] A dynamic causal graph generation unit determines a dynamic causal graph including environmental interaction relationships based on key causal event nodes and real-time environmental semantic information;

[0056] A future potential event determination unit determines the type of future potential events and the corresponding probability of occurrence of the event based on the dynamic causal graph;

[0057] An event constraint determination unit determines event triggering constraints based on the probability of event occurrence;

[0058] The node weight parameter update unit determines the node weight update parameters of the dynamic causal graph based on the incremental learning framework and real-time input multimodal historical behavior data;

[0059] The target object behavior prediction unit determines the predicted behavior trajectory of the target object based on the event triggering constraints and the updated dynamic causal graph.

[0060] In a third aspect, an electronic device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is configured to implement the steps of the target behavior prediction method of any one of the first aspects when executing the computer program stored in the memory.

[0061] In summary, this application improves the accuracy and environmental adaptability of target behavior prediction by integrating multimodal data with spatiotemporal causal reasoning. First, by modeling environmental interactions based on dynamic causal graphs, it can explicitly capture the causal event association between target objects and obstacles, solving the problem of prediction bias in complex scenarios caused by traditional methods. Second, through the incremental learning framework, node weights are updated in real time, allowing the model to continuously adapt to new data distributions and avoid prediction failures caused by environmental changes. Finally, by combining event-triggered constraints to generate probabilistic trajectories, dynamic avoidance and smooth correction of sudden obstacle conflicts are achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present description. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0063] Figure 1 A flow chart of a target behavior prediction method provided in an embodiment of the present application;

[0064] Figure 2 A schematic diagram of the structure of a target behavior prediction device provided in an embodiment of the present application;

[0065] Figure 3 A structural diagram of a target behavior prediction electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0066] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or that are inherent to these processes, methods, products or devices. The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the embodiments described are only part of the embodiments of the present application, not all of the embodiments.

[0067] See also Figure 1 , is a schematic diagram of a target behavior prediction process provided in an embodiment of the present application, which may specifically include:

[0068] S110, obtaining multimodal historical behavior data and real-time environmental semantic information of the target object;

[0069] For example, the historical behavior data and real-time environmental status of the target object are collected through multi-source sensors. The acquisition of multimodal historical behavior data involves equipment such as millimeter-wave radar, visual sensors, and lidar, which are used to capture the target object's motion parameters (such as speed and direction angle), morphological features (such as outlines, local environmental images), and three-dimensional spatial information (such as the movement direction of dynamic obstacles). These heterogeneous data are aligned and fused through preset timestamp synchronization rules to form a unified multimodal data set, ensuring temporal consistency and spatial correlation, and providing a comprehensive behavioral representation basis for subsequent causal reasoning.

[0070] The acquisition of real-time environmental semantic information focuses on real-time status monitoring of dynamic obstacles. Based on the three-dimensional environmental topology map generated by LiDAR and combined with the movement direction annotations of dynamic obstacles, the spatial structure of the target object's environment and the behavioral characteristics of the obstacles are updated in real time. This semantic information not only includes static environmental elements but also emphasizes the capture of dynamic interactions, such as obstacle movement trends and potential conflict areas. This provides critical environmental context input for the subsequent construction of dynamic cause-and-effect diagrams, ensuring that the predictive model can adapt to the real-time changes in complex scenarios.

[0071] S120. Determine key causal event nodes based on multimodal historical behavior data and spatiotemporal causal models;

[0072] For example, in the process of identifying key causal event nodes, a structured analysis of the time series characteristics of multimodal historical behavior data is required. Based on the preset historical time segment division rules, the continuous behavior data is discretized into time units with independent semantics. Then, through the event detection rules in the spatiotemporal causal model, significant behavioral features within each time segment are extracted, such as speed mutation points, direction turning points, and interaction events with environmental obstacles. These features reflect the abnormal behavior or pattern changes of the target object within a specific spatiotemporal range, providing a preliminary set of candidate event nodes for subsequent causal reasoning.

[0073] Furthermore, the causal association evaluation mechanism within the spatiotemporal causal model is used to screen and optimize candidate event nodes. Based on a preset causal association threshold, isolated or weakly correlated events are eliminated, while key nodes with significant causal relationships with the target object's behavioral evolution are retained. Simultaneously, a statistical causal verification algorithm is employed to verify the temporal causality between event nodes, eliminating spurious correlations. Ultimately, the core causal event nodes that characterize the dynamic evolution of the target behavior are identified, laying the foundation for constructing a dynamic causal graph.

[0074] S130, determining a dynamic causal graph including environmental interaction relationships based on key causal event nodes and real-time environmental semantic information;

[0075] For example, when constructing a dynamic causal graph, the first step is to integrate key causal event nodes with real-time environmental semantic information and establish the interactive relationship between event nodes and dynamic obstacles. Based on the obstacle movement direction and position data provided by the real-time environmental semantic information, combined with the set of key causal event nodes, environmental interaction weights are calculated to quantify the strength of the association between event nodes and obstacles. By assigning weights, an initial weighted causal graph is generated, where nodes represent key events and connecting edges reflect the causal influence of environmental interactions, thus preliminarily characterizing the interaction network between the target object and the dynamic environment.

[0076] Furthermore, the initial causal graph is dynamically optimized using a preset causal redundancy threshold, removing redundant connections with weights below the threshold. This process preserves strongly correlated causal paths, ensuring that the causal graph only includes significant environmental interactions and avoiding noise interference. The resulting dynamic causal graph can reflect the evolving associations between target object behavior and obstacle dynamics in real time, providing a structured causal reasoning framework for subsequent event probabilistic prediction.

[0077] S140. Determine the types of potential future events and the corresponding probability of occurrence of the events based on the dynamic causal graph;

[0078] For example, when predicting potential future event types, the activation probability of candidate event types is derived based on the topological structure and connection weights of event nodes in a dynamic causal graph. Event nodes in a dynamic causal graph represent the causal relationship between historical behavior and environmental interactions, and their connection weights reflect the strength of the causal influence between events. By aggregating the weight transfer effects of adjacent nodes, the node activation probability of each candidate event type is calculated, initially forming a set of possible future event types and their probability distribution, providing a basis for subsequent probability screening.

[0079] Furthermore, candidate event types are probabilistically screened and ranked using a preset probability threshold. By comparing node activation probabilities with the threshold, low-probability events are eliminated, while potential event types with a significant likelihood of occurrence are retained. Finally, a quantitative expression of the event probability is generated based on the screened event types and their corresponding activation probabilities, ensuring that the prediction results both conform to the dynamic evolution of the causal graph and meet the feasibility constraints of the actual scenario.

[0080] S150: Determine event triggering constraints based on the event occurrence probability;

[0081] For example, when determining the event triggering constraints, it is first necessary to extract the probability distribution parameters (such as mean, variance or confidence interval) of future potential event types based on the distribution characteristics of the event probability, and quantify the spatiotemporal triggering boundaries of the event in combination with the preset triggering rules (such as safety thresholds, priority strategies). For example, high-probability events may be assigned shorter time windows and more precise spatial regions to ensure that the prediction results match the real-time environment dynamics. By integrating the probability distribution parameters with the triggering rules, the time window and spatial region constraints for event triggering are preliminarily generated, providing a quantitative basis for the definition of dynamic threshold conditions.

[0082] Furthermore, by integrating time windows, spatial regions, and minimum probability thresholds, we construct multidimensional dynamic threshold conditions. The time window constrains the timeliness of event triggering, the spatial region limits the physical range of event occurrence, and the minimum probability threshold selects high-confidence candidate events. Together, these threshold conditions form a comprehensive constraint framework for event triggering, ensuring consistency between predicted behavior and probability distribution while also addressing the need for real-time adaptation to environmental changes, providing actionable trigger logic boundaries for subsequent predictions.

[0083] S160, determining node weight update parameters of the dynamic causal graph based on the incremental learning framework and real-time input multimodal historical behavior data;

[0084] For example, during the node weight update process in a dynamic causal graph, multimodal historical behavioral data is first accessed in real time through an incremental learning framework to detect distribution differences between the new data and the historical dataset. When the distribution difference of the incremental input data exceeds a preset threshold, a model update mechanism is triggered, generating knowledge transfer parameters. This mechanism ensures that the model can adapt to environmental changes, avoiding predictive performance degradation caused by data distribution shifts, while maintaining real-time consistency between node weights in the dynamic causal graph and the intensity of environmental interactions.

[0085] Furthermore, based on an online knowledge distillation model, new knowledge from incremental data is transferred to the dynamic causal graph. By integrating knowledge transfer parameters with current node weights, the association strength between event nodes and dynamic obstacles is readjusted. This process locally optimizes node weight parameters without destroying the original causal structure, enabling the dynamic causal graph to continuously reflect the latest interaction characteristics between the target object and the environment, providing a dynamically updated causal reasoning foundation for the subsequent generation of predicted behavioral trajectories.

[0086] S170: Determine the predicted behavior trajectory of the target object based on the event triggering constraint conditions and the updated dynamic causal graph.

[0087] For example, when generating a predicted behavioral trajectory for a target object, multiple candidate trajectory branches are first generated based on the time window and spatial region specified in the event triggering constraints. The posterior probability of each candidate trajectory branch is calculated based on the weight distribution of event nodes in the updated dynamic causal graph, reflecting its degree of match with historical causal relationships and the intensity of environmental interactions. Ultimately, the candidate trajectory with the highest posterior probability is selected as the primary predicted trajectory, preliminarily representing the target object's most likely behavioral path within the future spatiotemporal range.

[0088] Furthermore, the collision probability between the main predicted trajectory and the obstacle is evaluated by combining the real-time semantic information about the location of dynamic obstacles. If the collision probability exceeds a preset threshold, the spatial path of the main predicted trajectory is adjusted using a trajectory correction vector, and a smoothing algorithm is used to eliminate sudden changes in the trajectory. This ensures that the corrected predicted behavior trajectory meets both the high probability of causal association and the requirements for safe obstacle avoidance in a dynamic environment. The final output is a smooth and environmentally adaptive behavior prediction result.

[0089] In summary, in the embodiments of the present application, by fusing multimodal historical behavior data with real-time environmental semantic information and combining the spatiotemporal causal model to construct a dynamic causal graph, the accuracy of target behavior prediction and scene adaptability are improved. The dynamic causal graph explicitly models the environmental interaction relationship between the target object and the dynamic obstacle, and effectively captures the key causal links in complex scenarios through the weight association and redundancy elimination mechanism of the causal event nodes, overcoming the prediction bias problem caused by the traditional method ignoring the dynamic interaction of the environment. The incremental learning framework updates the node weight parameters in real time, so that the model continuously adapts to the new data distribution, avoids the prediction failure caused by the sudden change of the environment, and ensures the stability of the causal structure. Based on the triggering constraints of the probability of event occurrence and the posterior probability screening of candidate trajectories, high-confidence prediction and real-time environmental conflict avoidance are taken into account. Finally, a safe and coherent behavior trajectory is generated through trajectory correction and smoothing algorithms, realizing the robustness of dynamic avoidance and behavior prediction in sudden obstacle scenarios.

[0090] In some instances, the multimodal historical behavior data and real-time environment semantic information of the target object are obtained, including:

[0091] Determine the target object's movement speed and direction angle data based on the detection signal of the millimeter-wave radar;

[0092] Based on the image acquisition results of the visual sensor, determine the contour features of the target object and the local environment image;

[0093] Determine a three-dimensional environment topology map based on the laser radar point cloud data, wherein the three-dimensional environment topology map includes movement direction annotation information of dynamic obstacles;

[0094] Based on the preset timestamp synchronization rules, the motion speed, directional angle data, contour features, local environment images and 3D environment topology maps are aligned to determine the multimodal historical behavior data;

[0095] Based on the motion direction annotation information, the real-time environment semantic information is determined.

[0096] For example, millimeter-wave radar transmits high-frequency electromagnetic waves and receives signals reflected by the target object, calculating the radial velocity of the target object based on the Doppler effect. Specifically, when the electromagnetic wave encounters a moving target, the frequency of its reflected signal will shift due to the radial motion of the target relative to the radar. The frequency offset is linearly related to the target's velocity, and the instantaneous radial velocity of the target can be obtained by solving the frequency offset. To obtain the target's angular direction information, the millimeter-wave radar uses a multiple-input multiple-output (MIMO) array antenna structure, combined with beamforming technology, to calculate the spatial distribution of the target in the horizontal azimuth angle by analyzing the phase difference between different receiving antennas. The beamforming algorithm generates a directional beam by weighted synthesis of the signals received by each antenna. When the target is located at different azimuths, the peak intensity of the received signal corresponds to its angular direction. In addition, the time domain signal is spectrally analyzed by fast Fourier transform (FFT) to extract the characteristics of the target in the range and velocity domains, further separating the velocity and angle information in multi-target scenarios. Ultimately, the motion parameters output by the millimeter-wave radar include the real-time speed value and angular data of the target object. These parameters are stored in a sequence with timestamps as indexes, forming a continuous time series data set, providing high-precision, low-latency underlying motion features for subsequent behavioral analysis.

[0097] Visual sensors (such as RGB cameras or infrared cameras) capture visible light or infrared spectrum images of target objects to obtain their contour features and local environment images. Specifically, the original images collected by the visual sensor first undergo preprocessing steps (including denoising, contrast enhancement, and geometric correction) to eliminate the interference of illumination changes, motion blur, or sensor noise on feature extraction. Subsequently, edge detection algorithms (such as the Canny operator) or deep learning-based feature extraction models (such as convolutional neural networks) are used to identify the contour features of the target object from the preprocessed image. For targets such as pedestrians or vehicles, contour features are specifically manifested as key morphological structures such as limb joint connection points and vehicle outline geometry. These features are characterized by pixel-level coordinates or vectorized geometric parameters. At the same time, the acquisition of the local environment image depends on the focusing and segmentation of the area around the target object in the image. For example, a region of interest (ROI) delineation algorithm is used to crop a local image block containing the target object and its surrounding environment from the complete image. This local image not only preserves the spatial relative position of the target and surrounding objects (such as road markings and nearby obstacles), but also describes environmental details through visual features such as texture and color, forming contextual information associated with the target's behavior. Through these steps, the contour features and local environment images output by the visual sensor provide a visual representation of the multimodal historical behavior data, complementing the physical parameters of millimeter-wave radar and lidar.

[0098] The determination of a 3D environmental topology map is based on high-density 3D point cloud data collected by lidar. First, the lidar emits laser pulses and receives reflected signals, acquiring distance information about the surfaces of various objects in the target scene and generating raw point cloud data. After preprocessing (e.g., denoising and ground segmentation), this point cloud data is then grouped using a point cloud clustering algorithm (e.g., the density-based DBSCAN algorithm) to isolate independent target entities (e.g., vehicles, pedestrians, trees, etc.). The DBSCAN algorithm clusters densely distributed point clouds into independent objects using a preset neighborhood radius and minimum point count threshold, thereby distinguishing static environmental structures from dynamic obstacles. Subsequently, dynamic target tracking techniques (e.g., Kalman filtering or particle filtering) are used to calculate the motion vector (including instantaneous velocity, acceleration, and azimuth) of dynamic obstacles, combining the temporal correlation of consecutive frame point cloud data. This process uses a prediction and correction model to update the position and motion state of obstacles in real time, and eliminates noise interference based on preset motion consistency criteria (e.g., acceleration limit threshold). Ultimately, the real-time positions and motion direction vectors of static environmental structures (such as road boundaries and traffic signs) and dynamic obstacles are integrated into a three-dimensional coordinate system to form a topological map with temporal and spatial correlation. This map, through coordinate system conversion and rasterization, transforms point cloud data into a structured environmental model. The motion directions of dynamic obstacles are annotated with vector arrows, and static structures are presented as geometric grids, providing a three-dimensional environmental representation foundation for subsequent behavior prediction.

[0099] In the process of determining multimodal historical behavioral data, a pre-set timestamp synchronization rule is used to achieve spatiotemporal alignment of the multi-source data, addressing the differences in data acquisition frequencies between millimeter-wave radar, vision sensors, and lidar. Specifically, millimeter-wave radar samples the target's velocity and azimuth data at a high frequency, vision sensors capture contour features and local environmental images at a fixed frame rate, and lidar generates a discrete 3D topological map of the environment through pulse scanning. Due to the varying data acquisition cycles of different sensors, a hardware clock synchronization mechanism is employed to assign a unified timestamp reference to each sensor's data, ensuring initial temporal alignment of the raw data. For asynchronous data segments caused by inconsistent sensor sampling intervals, such as when the lidar does not complete a full scan at a certain moment while the vision sensor has already generated an image frame, a sliding window alignment strategy is implemented to define a time window (e.g., ±10ms). The parameter values at the missing moments within the window are linearly interpolated to ensure that the velocity, azimuth, contour features, and 3D map data are strictly aligned on a unified time axis. Furthermore, the direction-of-motion annotations of dynamic obstacles are combined with their timestamps and the target's behavioral data to fit the continuous motion trajectory using a quadratic interpolation algorithm, eliminating data gaps caused by discrete sampling. Through the above synchronization rules, heterogeneous sensor data are fused into a temporally consistent and spatially correlated multimodal historical behavior dataset, ensuring the temporal consistency and spatial integrity of subsequent analysis.

[0100] Real-time environmental semantic information is determined based on the annotation of the movement directions and real-time updated position coordinates of dynamic obstacles in the 3D environmental topology map. Specifically, using high-density point cloud data acquired by LiDAR, the obstacle's movement direction is identified and annotated in real time using methods such as Kalman filtering, generating directional parameters including the velocity vector modulus and angle. Simultaneously, the obstacle's real-time position coordinates in 3D space are combined with spatial geometry calculations to infer the instantaneous state of its motion trajectory. Furthermore, through vector decomposition and kinematic modeling, the obstacle's real-time motion characteristics (such as velocity, acceleration, and steering angle) are quantified and encoded into semantic labels, such as "constant speed straight ahead" and "accelerated left turn." Furthermore, based on the spatiotemporal projection of the obstacle's current position and movement direction, its potential short-term future motion path is calculated. Combined with the predicted trajectory of the target object, interaction areas between the two (such as collision risk areas or intersection points) are identified. Ultimately, by integrating these movement directions, position coordinates, behavior patterns, and interaction areas, a spatiotemporally consistent real-time semantic description of the environment is constructed, providing quantitative interaction characteristics of dynamic obstacles for subsequent causal reasoning.

[0101] In some instances, key causal event nodes are identified based on multimodal historical behavior data and spatiotemporal causal models, including:

[0102] Based on the time series characteristics of multimodal historical behavior data, multiple historical time segments are determined;

[0103] Based on the event detection rules in the spatiotemporal causal model, feature extraction is performed on each historical time segment to determine speed mutation points, direction turning points, and environmental interaction events;

[0104] Based on the preset causal association threshold, speed mutation points, direction turning points and environmental interaction events are marked to determine the set of candidate causal event nodes;

[0105] Based on the Granger causality test algorithm in the spatiotemporal causal model, the candidate causal event node set is eliminated and the key causal event nodes are determined.

[0106] Exemplarily, when determining historical time segments, a sliding window algorithm is first used to segment the continuous time series data based on the time series characteristics of the multimodal historical behavioral data. Specifically, a preset time segment length serves as the sliding reference for the window, and heterogeneous data such as the target object's motion speed, azimuth, profile features, and environmental topology information are divided into multiple independent time segments of fixed duration. Each time segment contains a complete behavioral representation of the target object during that period, such as the continuous sampling values of the speed sequence, the instantaneous change curve of the azimuth, the geometric evolution trajectory of the profile features, and the relative position and motion trend of dynamic obstacles in the environmental topology information. To ensure the continuity of the time segment division and avoid the loss of key behavioral features at segment boundaries, a preset overlap region is set between adjacent time segments. Through the progressive advancement of the sliding window, the complete time series data stream is covered. This overlapping mechanism effectively eliminates the risk of data fragmentation, allowing the target object's behavioral patterns in the transition phases between adjacent segments (such as sudden speed changes during acceleration or slight angle adjustments in the initial stage of steering) to be continuously captured, thereby providing a spatiotemporally coherent multimodal data foundation for the subsequent extraction of causal event nodes.

[0107] When extracting candidate event features, speed mutation points, directional turning points, and environmental interaction events are identified based on the divided historical time segments and the event detection rules preset by the spatiotemporal causal model. The spatiotemporal causal model is a mathematical model that combines time series analysis with spatial correlation reasoning, aiming to reveal the causal relationship between the target object's behavior and dynamic obstacles in the environment. To detect speed mutation points, a differential threshold method is used to analyze the target object's speed sequence. This method calculates the absolute value of the speed difference between adjacent time points. If this difference exceeds the preset mutation threshold, a speed mutation event is determined to have occurred at the current moment, and the occurrence timestamp and mutation amplitude are recorded. Identification of directional turning points is achieved by calculating the rate of change of the directional angle, specifically the change in the directional angle per unit time. If this change exceeds the preset turning threshold, it is marked as a directional turning event, and the turning angle and duration are also recorded. Detecting environmental interaction events requires considering the spatiotemporal relationship between the target object and the dynamic obstacle. This involves calculating the Euclidean distance between the two in real time. If this distance decreases continuously within a time segment and falls below a preset safety distance, and the angle between their motion vectors is less than a preset angle threshold, an environmental interaction event is identified and the obstacle identifier, interaction distance, and vector angle parameters are recorded. These detection results collectively constitute a set of event features within the time segment, providing foundational data for subsequent causal analysis.

[0108] When determining the set of candidate causal event nodes, the extracted speed mutation points, directional turning points, and environmental interaction events are evaluated for causal relevance. Specifically, based on a preset causal relevance threshold (e.g., using the Pearson correlation coefficient or mutual information as a quantitative indicator), the strength of the correlation between the event and the target behavior evolution is analyzed. For a speed mutation point, if the time interval between its occurrence and the subsequent directional turning event or environmental interaction event falls within a preset time window, and the correlation coefficient between the two exceeds a preset threshold, the speed mutation point is considered to be causally related to the subsequent event. Similarly, if a directional turning event is closely temporally and spatially correlated with a subsequent environmental interaction event (e.g., the obstacle position changes synchronously within the same time window), and the correlation coefficient between the two exceeds a preset threshold, the node is marked as a candidate node with a causal relationship. By traversing all historical time segments, event nodes that meet the above causal relevance criteria (including event type, occurrence timestamp, and correlation strength parameter) are aggregated into a set of candidate causal event nodes. This set, by integrating related events across multiple time segments, forms a preliminary causal network structure, providing data support for the subsequent screening of key nodes.

[0109] When identifying key causal event nodes, the generated set of candidate causal event nodes is causally verified and redundantly eliminated based on the Granger causality test algorithm integrated into the spatiotemporal causal model. Specifically, the spatiotemporal causal model introduces Granger causality tests to quantify the temporal causal relationship between event nodes and determine whether one event statistically significantly affects the occurrence of another. For each pair of candidate event nodes (e.g., event A and event B), a vector autoregression (VAR) model is first constructed, using the historical time series data of event A as the independent variable and the future time series data of event B as the dependent variable. The model parameters are fitted using the least squares method. Subsequently, the model is hypothesized to be tested using the F-test, calculating the significance level (p-value) of the predictive power of event A's time series information on event B. If the p-value of the test result is lower than the preset significance level (e.g., 0.05), event A is determined to be a Granger cause of event B, indicating that event A has a temporal causal driving effect on event B. Conversely, if the p-value does not meet the significance criteria, the causal relationship is eliminated to avoid spurious correlation interference. By traversing all event pairs in the candidate node set and iteratively executing the aforementioned testing and screening process, we ultimately retain statistically significant causal nodes, forming key causal event nodes. These nodes not only contain event types, timestamps, and correlation strength parameters, but also reinforce the credibility of causal relationships through Granger causality test results, providing a high-confidence spatiotemporal causal reasoning foundation for the construction of dynamic causal graphs.

[0110] In some instances, a dynamic causal graph including environmental interaction relationships is determined based on key causal event nodes and real-time environmental semantic information, including:

[0111] Based on the key causal event nodes, determine the event node set;

[0112] Determine the movement direction and position of dynamic obstacles based on real-time environmental semantic information;

[0113] Determine the environmental interaction weight based on the event node set, motion direction, and position, where the environmental interaction weight is used to characterize the strength of the association between the event node and the dynamic obstacle;

[0114] Determine the initial weighted causal graph based on the event node set and the environment interaction weights;

[0115] Based on a preset causal redundancy threshold, the connection edges in the initial causal graph whose weights are less than the causal redundancy threshold are removed to determine the dynamic causal graph.

[0116] Exemplarily, key causal event nodes are obtained by screening the spatiotemporal causal model through the Granger causality test, and each node represents an event that has a significant causal relationship with the behavioral evolution of the target object (such as a speed mutation, a direction change, or an environmental interaction event). The event node set is formed by integrating all key causal event nodes that have passed the test, where each node contains the event type (such as "speed mutation"), the occurrence timestamp, the association strength parameter (such as the p-value of the Granger causality test), and the quantitative characteristics corresponding to the event (such as the mutation amplitude or the turning angle). For example, if a speed mutation event is verified to be the Granger cause of a subsequent direction change event, the speed mutation event will be included in the event node set, and its interaction relationship with the obstacle will be marked. The determination of the event node set provides the basic elements for the construction of the dynamic causal graph.

[0117] Real-time environmental semantic information includes the real-time status of dynamic obstacles marked in the three-dimensional environmental topology map generated by the lidar, including the obstacle's position coordinates (such as the x, y, and z axis positions in three-dimensional space) and the motion direction vector (such as the modulus and angle of the velocity vector). By parsing the motion direction annotations in the semantic information, the instantaneous motion parameters and spatial position of each dynamic obstacle are extracted and mapped to a unified three-dimensional coordinate system. For example, the obstacle's motion direction is represented by a vector arrow, and its position is accurately marked by rasterized coordinates, forming a real-time state description of the dynamic obstacle, providing input for the subsequent calculation of the environmental interaction weight.

[0118] Environmental interaction weights are used to quantify the strength of the association between event nodes and dynamic obstacles. For each event node, the Euclidean distance between the target object and the dynamic obstacle at the time of occurrence is calculated. The closer the distance, the higher the weight. Secondly, the angle between the target object and the obstacle's motion vectors is analyzed. If the angle is less than a preset angle threshold, indicating a potential conflict between their motion directions, the weight is increased accordingly. Furthermore, the type of event node is considered (e.g., environmental interaction events are weighted higher than speed mutation events) and weighted using a preset weight distribution coefficient. Finally, the environmental interaction weights are mapped to the [0, 1] interval using a normalization formula (e.g., a sigmoid function) to ensure comparability and consistency of the weight values.

[0119] The initial causal graph is constructed as a directed weighted graph with the set of event nodes as vertices and the environmental interaction weights as the edge weights. Specifically, if two event nodes are causally related in time and space (such as event A is the Granger cause of event B), a directed edge is drawn from event A to event B, and the environmental interaction weight is assigned to the edge. The edge weight value comprehensively reflects the strength of the causal relationship between events and the degree of influence of environmental interaction. For example, if event A (speed mutation) causes event B (environmental interaction) and the two are close to each other and their movement directions conflict, the edge weight value approaches 1; conversely, if the association is weak, the weight value approaches 0. The initial causal graph is stored in an adjacency matrix or graph structure database to support subsequent dynamic optimization and real-time updates.

[0120] The connection edges in the initial causal graph are screened using a preset causal redundancy threshold. Specifically, all directed edges are traversed, and if the weight value of an edge is lower than the causal redundancy threshold, it is determined to be a redundant connection and removed from the graph. For example, if the edge weight from event A to event B is 0.25, which is lower than the threshold of 0.3, the edge is deleted to eliminate noise interference. This process retains strong causal paths with high weights, ensuring that the dynamic causal graph only contains significant environmental interactions. The final generated dynamic causal graph is continuously optimized through a real-time update mechanism (such as an incremental learning framework), which can reflect the dynamic evolution of the causal relationship between the target object and the obstacle, and provide a structured reasoning framework for subsequent event probability prediction and behavior trajectory generation.

[0121] In some examples, based on the dynamic causal graph, potential future event types and corresponding event probabilities are determined, including:

[0122] Determine a set of candidate event types based on a set of event nodes in a dynamic causal graph;

[0123] Based on the connection weights of each event node in the dynamic causal graph, the node activation probability of each candidate event type is determined;

[0124] Based on the preset probability threshold, the node activation probability is screened and the candidate event types that meet the preset probability threshold are determined as future potential event types;

[0125] Based on the node activation probability corresponding to the future potential event type, the event occurrence probability is determined, where the event occurrence probability is positively correlated with the node activation probability.

[0126] Exemplarily, a dynamic causal graph consists of a set of event nodes (including speed mutations, direction turnings, and environmental interaction events) and their weighted connecting edges. First, traverse all event nodes in the dynamic causal graph, extract their event types (such as "environmental interaction" or "speed mutation"), and classify them according to preset event classification rules (such as semantic division based on event labels) to form a set of candidate event types. For example, if the causal graph contains multiple environmental interaction event nodes, they are classified as "environmental interaction" type; if there are multiple direction turning event nodes, they are classified as "direction turning" type. The set of candidate event types needs to cover all event types that may be triggered by the target object in historical behavior to ensure the comprehensiveness of the prediction.

[0127] For each candidate event type, its node activation probability is calculated by aggregating the connection weights of related event nodes in the dynamic causal graph. Specifically, for a certain candidate event type (such as "environmental interaction"), all event nodes of this type are traversed in the causal graph, and the weight values of all its incoming edges (that is, the causal connection edges pointing to the node) are extracted. The comprehensive activation probability of the event type is obtained by weighted summation or mean calculation (for example, normalization using the Softmax function). For example, if an environmental interaction event node has two incoming edges with weights of 0.8 and 0.6 respectively, its activation probability can be calculated as (0.8+0.6) / 2=0.7. This process reflects the cumulative impact of historical causal associations on the future occurrence of event types.

[0128] The node activation probabilities of candidate event types are screened using a preset probability threshold. If the activation probability of a certain event type exceeds the threshold, it is determined to be a potential future event type; otherwise, it is eliminated. For example, if the activation probability of the "environmental interaction" type is 0.7 (exceeding the threshold of 0.5), it is retained as a potential event type; if the activation probability of the "speed mutation" type is 0.4 (below the threshold), it is excluded. This step ensures that only high-confidence event types are retained, preventing low-probability noise from interfering with the prediction results. The screened set of potential event types will serve as input for subsequent probability quantification.

[0129] Based on the screened future potential event types and their corresponding node activation probabilities, a quantitative expression of the event occurrence probability is generated. The event occurrence probability is positively correlated with the node activation probability, which is specifically achieved through linear mapping or probability distribution models (such as Poisson distribution). For example, if the activation probability of the "environmental interaction" type is 0.7, its occurrence probability is preset to the normalized result of the activation probability (such as 0.7 / total activation probability), or the activation probability is directly used as the event occurrence probability. The final generated probability value will be used to define the subsequent event triggering constraints to ensure that the prediction results meet both the strength of the causal relationship and the feasibility requirements of the actual scenario.

[0130] In some examples, event triggering constraints are determined based on the probability of event occurrence, including:

[0131] Based on the probability of event occurrence, determine the probability distribution parameters of each potential future event type;

[0132] Determine the time window and spatial area for event triggering based on preset triggering rules and probability distribution parameters;

[0133] Determine the dynamic threshold conditions for event triggering based on the time window and spatial region, where the dynamic threshold conditions include the triggering time interval, the triggering spatial boundary and the minimum probability threshold;

[0134] Based on dynamic threshold conditions and event occurrence probability, event triggering constraints are determined.

[0135] For example, for the screened future potential event types (such as "environmental interaction" or "directional turning"), the distribution characteristics of the probability of occurrence of the event are extracted, including parameters such as mean, variance and confidence interval. For example, if the probability of occurrence of a certain event type is 0.7, its mean can be directly taken as the probability value, and the variance reflects the fluctuation range of the probability in the historical data. For the confidence interval, the upper and lower bounds of the probability value are calculated by statistical methods (such as Bootstrap sampling). For example, at a 95% confidence level, the probability interval is [0.65, 0.75]. These parameters quantify the stability and reliability of event occurrence and provide a basis for the formulation of subsequent trigger rules.

[0136] Based on preset triggering rules (such as safety thresholds and priority strategies), probability distribution parameters are mapped to spatiotemporal constraints for event triggering. For example, if the preset rules require a short response time for high-probability events, a shorter time window (such as the next 2 seconds) is allocated based on the probability mean (such as 0.7). At the same time, the accuracy range of the spatial region (such as the triggering area within a radius of 1 meter) is determined based on the upper limit of the confidence interval (such as 0.75). For low-priority events, the time window may be relaxed (such as the next 5 seconds) and the spatial region may be expanded (such as a radius of 3 meters). By integrating probability parameters with triggering rules, a preliminary time window and spatial region are generated to ensure that the prediction results are adaptable to the real-time environment dynamics.

[0137] By combining a time window, a spatial region, and a preset minimum probability threshold (e.g., 0.6), a multi-dimensional dynamic threshold condition is constructed. This condition includes a triggering time interval, a triggering spatial boundary, and a minimum probability threshold. The triggering time interval limits the event to within the time window; the triggering spatial boundary defines the physical range within which the event occurs, for example, triggering only when the target object enters the spatial region; and the minimum probability threshold filters out event types with a probability higher than the threshold, for example, retaining only events with a probability ≥ 0.6. Dynamic threshold conditions combine these dimensions through logical relationships to ensure the stringency and operability of the triggering conditions.

[0138] The dynamic threshold conditions are matched with the probability of event occurrence to generate the final event trigger constraint conditions. Specifically, all future potential event types are traversed. If the probability distribution parameters of an event (such as a mean of 0.7) meet the dynamic threshold conditions (such as time window, spatial area and minimum probability threshold), it will be included in the trigger constraint set. For example, if the probability of an environmental interaction event is 0.7, it is within the time window and located in the preset spatial area, then it is determined to meet the trigger conditions and its trigger priority is recorded (such as sorting by probability value). The final generated trigger constraint conditions are stored in structured data (such as JSON or XML format), including event type, trigger time, spatial range and probability threshold, providing a clear trigger logic boundary for subsequent behavior trajectory prediction.

[0139] In some instances, based on an incremental learning framework and real-time input multimodal historical behavior data, node weight update parameters of a dynamic causal graph are determined, including:

[0140] Determine the incremental input data set based on multimodal historical behavior data input in real time;

[0141] Based on the distribution difference detection algorithm in the incremental learning framework, the distribution difference value between the incremental input data set and the historical data set is determined;

[0142] Determining a model update trigger signal based on a preset distribution difference threshold and a distribution difference value;

[0143] Determine the knowledge transfer parameters of the online knowledge distillation model based on the model update trigger signal and the incremental input dataset;

[0144] Based on the knowledge transfer parameter and the current node weight of the dynamic causal graph, the node weight update parameter is determined. The node weight update parameter is used to adjust the association strength between the event node and the dynamic obstacle in the dynamic causal graph.

[0145] Exemplarily, the incremental input data set is generated by preprocessing and alignment of the real-time input multimodal historical behavior data (including the motion parameters of the millimeter-wave radar, the contour features of the visual sensor, and the three-dimensional environmental topology data of the lidar). Specifically, through the preset timestamp synchronization rules, the real-time collected motion speed, direction angle, contour features and environmental topology information are aligned according to a unified time axis to eliminate the timing misalignment caused by the difference in sensor sampling frequency. For missing data segments, linear interpolation or sliding window completion algorithms are used to fill in and ensure the temporal integrity of the incremental data set. The final generated incremental input data set contains the behavioral characteristics and environmental interaction status of the target object in the latest time window, providing input for subsequent distribution difference detection.

[0146] In the incremental learning framework, the Kullback-Leibler (KL) divergence or Jensen-Shannon (JS) divergence algorithm is used to quantify the distribution difference between the incremental input dataset and the historical dataset. Specifically, the probability distribution of the historical dataset and the incremental dataset in key feature dimensions (such as speed mutation frequency, angular change rate, and obstacle interaction distance) is extracted, and the distribution difference value is obtained through divergence calculation. For example, if the mean of speed mutations in the historical dataset is 1.5, while the mean of the incremental dataset is 2.0, the KL divergence will reflect this offset. The distribution difference value is mapped to the [0,1] interval through normalization processing to determine whether the data distribution has changed significantly.

[0147] A preset distribution difference threshold (for example, 0.3) is set. When the calculated distribution difference value exceeds the threshold, it is determined that there is a significant deviation between the incremental data and the historical data distribution, and the model update mechanism is triggered. For example, if the distribution difference value is 0.45 (exceeding the threshold value of 0.3), an update trigger signal is generated; conversely, if the difference value is 0.2, the current model is maintained. The trigger signal is passed to the subsequent processing module through a binary flag (such as "1" for update and "0" for maintenance) or a priority parameter (such as the larger the difference value, the higher the update priority), ensuring that the model only starts the update process when necessary, avoiding stability problems caused by frequent adjustments.

[0148] When the model update trigger signal takes effect, the online knowledge distillation model is used to extract new knowledge from the incremental data set. Specifically, the incremental data is input into the teacher model (such as a pre-trained deep neural network) to generate soft labels (SoftLabels) or feature embedding vectors as representations of the new knowledge. At the same time, the student model (i.e., the dynamic causal graph) calculates the knowledge transfer loss (such as cross entropy loss or mean square error) by comparing the output of the teacher model with its own prediction results. The student model optimizes the loss function through the gradient descent algorithm to obtain the knowledge transfer parameters (such as weight adjustment coefficients or bias terms) to quantify the strength of the correction of the new data to the original causal relationship. For example, if the incremental data shows that the frequency of obstacle interactions has increased significantly, the migration parameters will increase the weight coefficient of the environmental interaction event.

[0149] The knowledge transfer parameters are integrated with the current node weights of the dynamic causal graph to recalculate the association strength between event nodes and dynamic obstacles. Specifically, for each event node, the updated weight is calculated as follows: New Weight = Original Weight × (1 - Decay Coefficient) + Transfer Parameter × Learning Rate. The decay coefficient (e.g., 0.1) controls the proportion of historical weights retained, and the learning rate (e.g., 0.05) regulates the rate of incorporation of new knowledge. For example, if the original weight of an environmental interaction node is 0.8 and the transfer parameter is 0.3, the new weight is 0.8 × 0.9 + 0.3 × 0.05 = 0.735. This process locally adjusts node weights without disrupting the original causal graph topology, enabling the model to adapt to dynamic environmental changes. The resulting node weight update parameters are written to the dynamic causal graph via a graph database or adjacency matrix update interface, completing the real-time optimization of the model.

[0150] In some instances, based on the event triggering constraints and the updated dynamic causal graph, a predicted behavior trajectory of the target object is determined, including:

[0151] Determine multiple candidate trajectory branches based on time windows and spatial regions;

[0152] Determine the posterior probability of each candidate trajectory branch based on the node weights in the updated dynamic causal graph;

[0153] Based on the maximum value of the posterior probability, the main prediction trajectory is determined;

[0154] Determine the trajectory conflict probability based on the main predicted trajectory and the dynamic obstacle positions in the real-time environment semantic information;

[0155] Determining a trajectory correction vector based on a trajectory conflict probability and a preset conflict threshold;

[0156] Based on the trajectory correction vector and the preset smoothing algorithm, the predicted behavior trajectory of the target object is determined.

[0157] For example,

[0158] First, multiple candidate trajectory branches are generated based on the time window and spatial area defined in the event triggering constraints. The generation of candidate trajectories depends on the historical behavior pattern of the target object and the causal correlation of event nodes in the dynamic causal graph. For example, based on the current movement speed and angular data of the target object, its possible path within the time window is deduced through a kinematic model (such as a uniform velocity model or an acceleration model), and the obstacle distribution in the environmental semantic information is combined to exclude invalid paths that overlap with static obstacles. Each candidate trajectory branch is represented by a parameter, recording its time and space coordinate sequence, forming a trajectory set covering the potential movement direction of the target object.

[0159] For each candidate trajectory branch, the degree of match with the historical causal relationship is calculated by the weight distribution of the event nodes in the updated dynamic causal graph. Specifically, the spatiotemporal region through which the trajectory branch passes is traversed, and the event nodes and their connection weights of the dynamic causal graph in the region are extracted. For example, if a trajectory passes through an environmental interaction event node (weight 0.8) and a speed mutation event node (weight 0.6), the posterior probability of the trajectory is calculated by weighted summation or Bayesian network inference. The quantitative formula of the posterior probability is:

[0160] in, For the The posterior probability of a candidate trajectory branch, that is, the possibility of the trajectory occurring in the future, quantifies the degree of matching between the trajectory and the historical causal relationship and the intensity of environmental interaction. The higher the probability value, the more the trajectory conforms to the evolution law of the dynamic causal graph; for all event nodes in the dynamic causal graph Perform a summation operation to integrate the impact of all relevant event nodes on the current trajectory, reflecting the cumulative effect of causal relationships; For the The weight of an event node comes from the updated edge weight in the dynamic causal graph. The strength of the association with the target object's behavioral evolution. The higher the weight, the more significant the causal impact of the event on the trajectory; Used to judge candidate trajectories Whether to cover the event node The space-time range of Space-time paths and events If there is an intersection between the time and space ranges of , it is 1, otherwise it is 0. The causal event nodes related to the trajectory are screened. Only when the trajectory passes through the time and space region of the event node, the weight of the event will be included in the posterior probability.

[0161] By comparing the posterior probabilities of all candidate trajectory branches, the one with the highest probability is selected as the primary predicted trajectory. For example, if the posterior probability of candidate trajectory A is 0.85 and that of trajectory B is 0.72, trajectory A is determined to be the primary predicted trajectory. The spatiotemporal coordinate sequence of the primary predicted trajectory represents the most likely path of the target object within the future time window, including trends in speed, direction, and position. The probability value of the primary predicted trajectory and associated event node information are recorded to provide input for subsequent conflict detection.

[0162] Based on the dynamic obstacle position (such as 3D coordinates and movement direction) in the real-time environment semantic information, the collision probability between the main predicted trajectory and the obstacle is calculated. Specifically, through the spatial geometric model (such as bounding box detection or distance field analysis), it is determined whether the main predicted trajectory intersects with the predicted path of the obstacle within the time window. For example, if the obstacle is moving at a speed of When moving in the direction of the main predicted trajectory, the relative distance to the target will decrease over time, and the collision risk should increase with increasing speed. The collision time formula for calculating the collision probability is expressed as: in, is the preset safety distance threshold, which indicates the minimum safety distance allowed between the target object and the obstacle; The Euclidean distance calculated in real time is the actual distance between the target object and the obstacle; To adjust the coefficient, control the slope of the Sigmoid function, and determine the sensitivity of the probability to distance changes ( When , the distance difference has a more significant impact on the probability); is the speed of the obstacle relative to the target object; is the prediction time window;

[0163] The above formula calculates the obstacle The distance that can be shortened in time ; The actual distance after correction is (If an obstacle approaches, is a positive value, shortening the distance); when the corrected actual distance is less than or equal to the preset safety distance threshold The probability of conflict increases significantly.

[0164] When the trajectory conflict probability exceeds a preset threshold (0.2), a trajectory correction vector is generated through an optimization algorithm (such as gradient descent or geometric projection). The direction of the correction vector is determined by the relative relationship between the obstacle position and the main predicted trajectory: if the obstacle is on the left side of the trajectory, a lateral offset vector to the right is generated to avoid the conflict. If it is in front, a deceleration or longitudinal avoidance vector is generated. The modulus of the correction vector is controlled by the difference ratio between the conflict probability and the preset threshold. For example, when the conflict probability is 0.8, the modulus of the correction vector is (0.8-0.2) / 0.2×maximum correction amount. Through vector superposition, the correction vector is applied to the path points of the main predicted trajectory to generate an adjusted candidate trajectory.

[0165] The corrected trajectory may exhibit path abrupt changes or discontinuities. Preset smoothing algorithms (such as Kalman filtering or spline interpolation) are used to optimize the corrected trajectory, eliminating path abrupt changes and enhancing trajectory continuity. For example, a cubic spline interpolation algorithm is employed, with the corrected key path points serving as control points, to generate a smooth trajectory curve. The smoothed trajectory must simultaneously maintain a safe distance from dynamic obstacles within the time window; conform to the kinematic characteristics of the target object (e.g., speed and acceleration limits); and maintain a causal relationship with high-weight event nodes in the dynamic causal graph. The resulting predicted behavior trajectory is represented by a sequence of spatiotemporal coordinates, providing highly reliable behavior predictions for autonomous driving or robotic navigation.

[0166] See also Figure 2 , is a schematic diagram of the structure of a target behavior prediction device provided in an embodiment of the present application, comprising:

[0167] The target object data acquisition unit 21 is used to acquire the multimodal historical behavior data and real-time environment semantic information of the target object;

[0168] A key event node determination unit 22 determines key causal event nodes based on multimodal historical behavior data and a spatiotemporal causal model;

[0169] A dynamic causal graph generating unit 23 determines a dynamic causal graph including environmental interaction relationships based on key causal event nodes and real-time environmental semantic information;

[0170] A future potential event determination unit 24 determines the type of future potential event and the corresponding probability of occurrence of the event based on the dynamic causal graph;

[0171] An event constraint determination unit 25 determines an event trigger constraint based on the event occurrence probability;

[0172] A node weight parameter updating unit 26 determines node weight update parameters of the dynamic causal graph based on an incremental learning framework and multimodal historical behavior data input in real time;

[0173] The target object behavior prediction unit 27 determines the predicted behavior trajectory of the target object based on the event triggering constraint conditions and the updated dynamic causal graph.

[0174] See also Figure 3 An embodiment of the present application also provides an electronic device 300, including a memory 310, a processor 320, and a computer program 311 stored in the memory 310 and executable on the processor. When the processor 320 executes the computer program 311, the steps of any method for predicting target behavior are implemented.

[0175] Since the electronic device introduced in this embodiment is a device used to implement a target behavior prediction device in the embodiment of the present application, based on the method introduced in the embodiment of the present application, technical personnel in this field can understand the specific implementation of the electronic device of this embodiment and its various variations. Therefore, how the electronic device implements the method in the embodiment of the present application is no longer introduced in detail. As long as the equipment used by technical personnel in this field to implement the method in the embodiment of the present application falls within the scope of protection of this application.

[0176] During the specific implementation process, when the computer program 311 is executed by the processor, any implementation method of the embodiments corresponding to the first aspect can be implemented.

[0177] It should be noted that, in the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0178] Those skilled in the art will appreciate that embodiments of the present application may provide methods, systems, or computer program products. Thus, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-readable storage media containing computer-readable program code.

[0179] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0180] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0181] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0182] The present application also provides a computer program product, which includes computer software instructions. When the computer software instructions are executed on a processing device, the processing device executes Figure 1 The process of a target behavior prediction method in the corresponding embodiment.

[0183] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium can be a magnetic medium, an optical medium or a semiconductor medium, etc.

[0184] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0185] In the several embodiments provided in this application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0186] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0187] In addition, the functional units in the various embodiments of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The above-mentioned integrated units may be implemented in the form of hardware and / or software functional units.

[0188] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device to execute all or part of the steps of the various embodiments of the method of the present application.

[0189] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0190] Although the preferred embodiments of this specification have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of this specification.

[0191] Obviously, those skilled in the art may make various changes and modifications to this specification without departing from the spirit and scope of this specification. Thus, if such changes and modifications fall within the scope of the claims of this specification and their equivalents, this specification is intended to include such changes and modifications.

Claims

1. A target behavior prediction method, characterized in that: include: Obtain multimodal historical behavior data and real-time environmental semantic information of the target object; Determining key causal event nodes based on the multimodal historical behavior data and the spatiotemporal causal model; Determining key causal event nodes based on the multimodal historical behavior data and the spatiotemporal causal model includes: Determining a plurality of historical time segments based on the time series characteristics of the multimodal historical behavior data; Based on the event detection rules in the spatiotemporal causal model, feature extraction is performed on each of the historical time segments to determine speed mutation points, direction turning points, and environmental interaction events; Based on a preset causal association threshold, the speed mutation point, the direction turning point and the environment interaction event are marked to determine a set of candidate causal event nodes; Based on the Granger causality test algorithm in the spatiotemporal causal model, the candidate causal event node set is subjected to association elimination to determine the key causal event node; Determining a dynamic causal graph including environmental interaction relationships based on the key causal event nodes and the real-time environmental semantic information; Determining future potential event types and corresponding event probabilities based on the dynamic causal graph; Determining event triggering constraints based on the event occurrence probability; Determining node weight update parameters of the dynamic causal graph based on an incremental learning framework and real-time input multimodal historical behavior data; Based on the event triggering constraints and the updated dynamic causal graph, a predicted behavior trajectory of the target object is determined.

2. The method according to claim 1, characterized in that The step of obtaining the multimodal historical behavior data and real-time environment semantic information of the target object includes: Determining the speed and direction of the target object based on the detection signal of the millimeter-wave radar; Determine the contour features of the target object and the local environment image based on the image acquisition results of the visual sensor; Determine a three-dimensional environment topology map based on the laser radar point cloud data, wherein the three-dimensional environment topology map includes movement direction annotation information of dynamic obstacles; Based on a preset timestamp synchronization rule, the movement speed, the directional angle data, the contour features, the local environment image, and the three-dimensional environment topology map are aligned to determine multimodal historical behavior data; Based on the movement direction annotation information, real-time environmental semantic information is determined.

3. The method according to claim 1, characterized in that The determining of a dynamic causal graph including environmental interaction relationships based on the key causal event nodes and the real-time environmental semantic information includes: Determining an event node set based on the key causal event nodes; Determining the movement direction and position of the dynamic obstacle based on the real-time environmental semantic information; Determining an environment interaction weight based on the event node set, the motion direction, and the position, wherein the environment interaction weight is used to characterize the strength of association between the event node and the dynamic obstacle; Determining an initial causal graph with weights based on the event node set and the environment interaction weights; Based on a preset causal redundancy threshold, connecting edges whose weights are less than the causal redundancy threshold in the initial causal graph are removed to determine a dynamic causal graph.

4. The method according to claim 1, wherein Determining the types of potential future events and the corresponding probability of occurrence of the events based on the dynamic causal graph includes: Determining a set of candidate event types based on a set of event nodes in the dynamic causal graph; Determining a node activation probability for each candidate event type based on the connection weights of the event nodes in the dynamic causal graph; Based on a preset probability threshold, the node activation probability is screened to determine the candidate event type that meets the preset probability threshold as the future potential event type; Based on the node activation probability corresponding to the future potential event type, the event occurrence probability is determined, wherein the event occurrence probability is positively correlated with the node activation probability.

5. The method according to claim 1, characterized in that The determining of event triggering constraint conditions based on the event occurrence probability includes: Determining probability distribution parameters for each future potential event type based on the probability of occurrence of the event; Determine the time window and spatial area for event triggering based on preset triggering rules and the probability distribution parameters; Determining a dynamic threshold condition for event triggering based on the time window and the spatial region, wherein the dynamic threshold condition includes a triggering time interval, a triggering spatial boundary, and a minimum probability threshold; An event triggering constraint condition is determined based on the dynamic threshold condition and the event occurrence probability.

6. The method according to claim 5, characterized in that The method of determining the node weight update parameters of the dynamic causal graph based on the incremental learning framework and the real-time input multimodal historical behavior data includes: Determine the incremental input data set based on the multimodal historical behavior data input in real time; Determining the distribution difference value between the incremental input data set and the historical data set based on the distribution difference detection algorithm in the incremental learning framework; determining a model update trigger signal based on a preset distribution difference threshold and the distribution difference value; Determining knowledge migration parameters of an online knowledge distillation model based on the model update trigger signal and the incremental input data set; Based on the knowledge transfer parameter and the current node weight of the dynamic causal graph, a node weight update parameter is determined, wherein the node weight update parameter is used to adjust the association strength between the event node and the dynamic obstacle in the dynamic causal graph.

7. The method according to claim 6, characterized in that Determining the predicted behavior trajectory of the target object based on the event triggering constraint condition and the updated dynamic causal graph includes: Determining a plurality of candidate trajectory branches based on the time window and the spatial region; Determining the posterior probability of each candidate trajectory branch based on the node weights in the updated dynamic causal graph; Determining a primary prediction trajectory based on the maximum value of the posterior probability; determining a trajectory conflict probability based on the primary predicted trajectory and a position of a dynamic obstacle in the real-time environment semantic information; determining a trajectory correction vector based on the trajectory conflict probability and a preset conflict threshold; Based on the trajectory correction vector and a preset smoothing algorithm, a predicted behavior trajectory of the target object is determined.

8. A target behavior prediction device, characterized in that: include: A target object data acquisition unit, used to acquire the target object's multimodal historical behavior data and real-time environmental semantic information; a key event node determination unit, which determines key causal event nodes based on the multimodal historical behavior data and the spatiotemporal causal model; Determining key causal event nodes based on the multimodal historical behavior data and the spatiotemporal causal model includes: Determining a plurality of historical time segments based on the time series characteristics of the multimodal historical behavior data; Based on the event detection rules in the spatiotemporal causal model, feature extraction is performed on each of the historical time segments to determine speed mutation points, direction turning points, and environmental interaction events; Based on a preset causal association threshold, the speed mutation point, the direction turning point and the environment interaction event are marked to determine a set of candidate causal event nodes; Based on the Granger causality test algorithm in the spatiotemporal causal model, the candidate causal event node set is subjected to association elimination to determine the key causal event node; A dynamic causal graph generating unit, which determines a dynamic causal graph including environmental interaction relationships based on the key causal event nodes and the real-time environmental semantic information; A future potential event determination unit, which determines the type of future potential events and the corresponding probability of occurrence of the events based on the dynamic causal graph; An event constraint condition determination unit, which determines an event triggering constraint condition based on the event occurrence probability; A node weight parameter updating unit, which determines the node weight update parameters of the dynamic causal graph based on an incremental learning framework and multimodal historical behavior data input in real time; The target object behavior prediction unit determines the predicted behavior trajectory of the target object based on the event triggering constraint condition and the updated dynamic causal graph.

9. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is configured to implement the steps of the target behavior prediction method as described in any one of claims 1 to 7 when executing the computer program stored in the memory.

Images