An autonomous driving context awareness and decision method and system

Abstract

The application provides an automatic driving context perception and decision method and system. The application generates standardized scene data through multi-source data fusion, constructs an adaptive digital twin environment, generates context labels through semantic reasoning using a knowledge graph, generates a decision evaluation matrix based on multi-hypothesis parallel simulation, selects the optimal action in combination with user preferences, and finally realizes system self-evolution through closed-loop learning optimization. Thus, the safety and robustness of the system in complex scenarios are improved. The application can solve the technical problems of existing automatic driving technologies, such as unstable prediction caused by lack of semantic context understanding, and behavior conservation and inefficiency caused by rigid decoupling of prediction and planning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, in particular to an automatic driving situation perception and decision method and system. BACKGROUND

[0002] The trajectory prediction and planning decision technology based on interactive perception is one of the core decision frameworks of the current mainstream high-level automatic driving system, which follows a modularized "perception-prediction-planning" pipeline. First, a multi-sensor fusion module constructs a dynamic bird's eye view of the vehicle's surroundings and identifies other traffic participants. Then, a prediction module uses a deep learning model to predict the possible trajectories of these traffic participants within the next 3-8 seconds. In order to handle uncertainty, advanced prediction models can output multiple modal trajectories, i.e., predicting several different probability future paths for a pedestrian or vehicle. Finally, the planning module receives these predicted trajectories as "dynamic obstacles" and inputs them into a cost function. The cost function considers safety, comfort, efficiency, and rule compliance, and the planning module searches for a self-driving path with the lowest cost that avoids all predicted trajectories through optimization algorithms. However, this planning and decision-making method has the technical problems of lack of semantic situation understanding, unstable prediction, and rigid decoupling of prediction and planning, leading to conservative and inefficient behavior.

[0003] Therefore, how to make the automatic driving decision system go beyond simple kinematic prediction and achieve deep semantic understanding and forward-looking risk prediction of complex unstructured scenes is a technical problem that needs to be solved in the field. SUMMARY

[0004] In view of the deficiencies of the prior art, the present application proposes an automatic driving situation perception and decision method and system to solve the technical problems of lack of semantic situation understanding, unstable prediction, and rigid decoupling of prediction and planning in the prior art, leading to conservative and inefficient behavior.

[0005] The technical solution adopted by the present application is an automatic driving situation perception and decision method, comprising:

[0006] Time synchronization, spatial registration, and semantic fusion of multi-source heterogeneous data from vehicle-mounted sensors to generate standardized scene data;

[0007] Generating an adaptive digital twin environment based on the standardized scene data and environmental prior constraints;

[0008] Performing graph reasoning on the standardized scene data to generate situation semantic labels;

[0009] Based on the adaptive digital twin environment and situation semantic labels, generating a set of probabilistic behavior hypotheses and a set of candidate actions corresponding to key dynamic targets;

[0010] Parallel simulation deduction is performed in combination with the set of probabilistic behavior hypotheses and the set of candidate actions to generate a decision evaluation matrix;

[0011] Based on the decision evaluation matrix, the optimal driving action is selected and issued in combination with the user driving preference and the preset decision criterion.

[0012] Further, the standardized scene data is generated, including:

[0013] A uniform timestamp is assigned to the received multi-source heterogeneous data;

[0014] The timestamp of the data frame is read, and the vehicle attitude is obtained by interpolation based on the inertial measurement unit data;

[0015] In combination with the preset sensor external parameters, the multi-source heterogeneous data is unified to the ego vehicle coordinate system through rigid body transformation;

[0016] The converted multi-source heterogeneous data is subjected to data fusion to generate a fusion object;

[0017] The fusion object is subjected to validity detection and abnormality rejection, and the cleaned fusion object is encapsulated as a data structure with uniform fields.

[0018] Further, an adaptive digital twin environment is generated, including:

[0019] The standardized scene data published to different data topics is subscribed and received, wherein: the vehicle state topic provides a vehicle state vector, the fusion object topic provides a dynamic target list, and the map update topic provides road and environment information;

[0020] The vehicle state, spatial position and motion state of the traffic participants are updated according to the standardized scene data, and an environment factor vector is constructed;

[0021] According to the environment factor vector, an adaptive performance decay mathematical model is used to calculate a set of performance decay coefficients of the environment factor vector at the current time;

[0022] Based on the vehicle state, spatial position and motion state, a digital twin scene snapshot is generated, and an adaptive digital twin environment is generated in combination with the set of performance decay coefficients.

[0023] Further, the performance decay mathematical model includes:

[0024] LiDAR attenuation model adapted to rainfall:

[0025]

[0026] wherein, represents the intensity of LiDAR signal after propagation distance d, represents the initial signal intensity, represents the extinction coefficient;

[0027] Camera attenuation model adapting to light and sensor working state:

[0028]

[0029] wherein K represents a camera performance attenuation coefficient, represents a light mapping function describing the relationship between light intensity and image signal-to-noise ratio, represents a weight coefficient of the light mapping function, represents a lens dirt confidence output by a visual algorithm for detecting camera lens dirt, represents a weight coefficient of the lens dirt confidence.

[0030] Further, the standardized scene data is subjected to graph reasoning to generate a situational semantic label, including:

[0031] mapping the standardized scene data to a pre-constructed vehicle-mounted knowledge graph entity;

[0032] determining a risk path from the matched entity to a risk entity by performing graph traversal and rule matching on the vehicle-mounted knowledge graph;

[0033] confidence fusion is performed on the results pointing to the same risk entity based on the risk path to generate a probability correction factor corresponding to the risk entity;

[0034] the risk path and the probability correction factor corresponding to the risk entity are packaged as a situational semantic label.

[0035] Further, a decision evaluation matrix is generated, including:

[0036] based on the current vehicle state and task, sampling in the two-dimensional space of acceleration and curvature to generate a set of candidate action sequences;

[0037] generating a set of basic behavior hypotheses for the key dynamic target in the digital twin scene snapshot according to a preset dynamic target basic behavior model library and a pre-defined behavior library;

[0038] adjusting the prior probability of the basic behavior hypothesis based on the probability correction factor in the situational semantic label;

[0039] combining the prior probability with the basic behavior hypothesis to generate a set of probabilistic behavior hypotheses;

[0040] loading the latest digital twin scene and attenuation parameters, and constructing a simulation task tree with combinations of all elements in the probabilistic behavior hypothesis set, and distributing to individual computing cores of HPC;

[0041] The computing core performs simulation deduction of future scenarios in parallel according to a preset time step;

[0042] The computing core records key simulation data including vehicle spacing, relative speed and virtual sensor perception results during simulation;

[0043] Calculate the single task cost in the simulation task tree based on the key simulation data;

[0044] Weight the single task cost and the corresponding hypothesis probability in the probabilistic behavior hypothesis set to calculate the expected total cost of each candidate action;

[0045] The candidate action sequence and its corresponding expected total cost form a decision evaluation matrix.

[0046] Further, based on the key simulation data, the single task cost in the simulation task tree is calculated, including:

[0047] According to the key simulation data, the safety cost, efficiency cost, comfort cost, compliance cost and social acceptance cost are calculated respectively;

[0048] By weighting and summing the total cost, efficiency cost, comfort cost, compliance cost and social acceptance cost according to the cost function, the single task cost is obtained:

[0049]

[0050] where J represents the single task cost, represents the coefficient of the cost function weight, represents the cost component.

[0051] Further, selecting and issuing the optimal driving action includes:

[0052] Based on the highest risk value in the decision evaluation matrix and its corresponding scene semantic label, combined with user preference configuration parameters, the decision criterion of this decision cycle is selected;

[0053] According to the decision criterion, the corresponding optimal candidate action is selected from the decision evaluation matrix.

[0054] Further, it also includes: based on the execution result of the control instruction, the cost function weight and the decision criterion are closed-loop optimized and adaptively adjusted.

[0055] Further, the cost function weight and decision criteria are closed-loop optimized and adaptively adjusted, including:

[0056] The standardized scene data is taken as real-world observation data, and the decision evaluation matrix, the set of probabilistic behavior assumptions, and the control instruction are taken as prediction data;

[0057] After the real-world observation data and the prediction data are aligned in time and space, the inconsistency score of the real-world observation data and the prediction data is calculated according to a preset metric function library;

[0058] The inconsistency score is compared with a dynamic threshold corresponding to the current scene;

[0059] In response to the inconsistency score being greater than the dynamic threshold, a case file of data related to the current cognitive bias event is traced back and collected;

[0060] The light-weight model is fine-tuned according to the metadata of the case file.

[0061] From the above technical solutions, the beneficial technical effects of the present application are as follows:

[0062] 1. By using the vehicle-mounted knowledge graph reasoning and the probabilistic correction mechanism, the semantic information in the complex traffic scene is combined with the dynamic target behavior prediction, the limitations of the prior art that only rely on kinematics history and lack of robustness are overcome, and the perception and prediction ability of the automatic driving in unstructured and sudden scenes is improved.

[0063] 2. By using the adaptive digital twin environment, multi-hypothesis simulation deduction, and closed-loop learning optimization, the prediction and planning are closely coupled, and the cost function weight and model parameters are dynamically corrected, thereby breaking through the bottleneck of the prior art that the prediction and planning are rigidly decoupled and the simulation is limited to offline verification, so as to realize the flexibility, safety, and efficiency of online decision-making. BRIEF DESCRIPTION OF DRAWINGS

[0064] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the drawings needed in the specific embodiments or prior art description will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, each element or part is not necessarily drawn according to the actual scale.

[0065] Figure 1 The method flowchart of the embodiment 1 of the present application;

[0066] Figure 2 The device connection diagram of the embodiment 2 of the present application;

[0067] Figure 3 The multi-source data fusion module structure diagram of the embodiment 2 of the present application;

[0068] Figure 4 The running flow chart of the multi-source data fusion module of the embodiment 2 of the present application;

[0069] Figure 5 The structural diagram of the dynamic digital twin construction module of the embodiment 2 of the present application;

[0070] Figure 6 The running flow chart of the dynamic digital twin construction module of the embodiment 2 of the present application;

[0071] Figure 7 The structural diagram of the semantic context reasoning module of the embodiment 2 of the present application;

[0072] Figure 8 The running flow chart of the semantic context reasoning module of the embodiment 2 of the present application;

[0073] Figure 9 The structural diagram of the multi-hypothesis simulation prediction module of the embodiment 2 of the present application;

[0074] Figure 10 The running flow chart of the multi-hypothesis simulation prediction module of the embodiment 2 of the present application;

[0075] Figure 11 The structural diagram of the optimal behavior decision module of the embodiment 2 of the present application;

[0076] Figure 12 The running flow chart of the optimal behavior decision module of the embodiment 2 of the present application;

[0077] Figure 13 The structural diagram of the closed-loop learning optimization module of the embodiment 2 of the present application;

[0078] Figure 14 The running flow chart of the closed-loop learning optimization module of the embodiment 2 of the present application. DETAILED DESCRIPTION

[0079] The embodiments of the technical solutions of the present application will be described in detail below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present application, and therefore only serve as examples, and cannot limit the protection scope of the present application.

[0080] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in the present application should be understood as the usual meanings understood by the skilled in the art to which the present application belongs.

[0081] Embodiment 1

[0082] The present embodiment provides an automatic driving context perception and decision method, and the working principle of embodiment 1 is described in detail as follows:

[0083] The method flowchart of the embodiment is as shown in Figure 1 includes time synchronization, spatial registration and semantic fusion of multi-source heterogeneous data from vehicle-mounted sensors to generate standardized scene data.

[0084] The standardized scene data is combined with environmental prior constraints to generate an adaptive digital twin environment.

[0085] Graph reasoning is performed on the standardized scene data to generate situational semantic labels.

[0086] Based on the adaptive digital twin environment and situational semantic labels, a set of probabilistic behavior hypotheses is generated for key dynamic targets in the scene, and parallel simulation deduction is performed in combination with candidate actions to generate a decision evaluation matrix.

[0087] Based on the decision evaluation matrix, in combination with user driving preferences and preset decision criteria, the optimal driving action is selected and issued, and the driving action is converted into a control instruction executable by the vehicle.

[0088] Based on the execution result of the control instruction, the cost function weight and decision parameter are closed-loop optimized and adaptively adjusted.

[0089] In the embodiment, further, multi-source heterogeneous data from vehicle-mounted sensors are time-synchronized, spatially registered and semantically fused to generate standardized scene data, including:

[0090] The received multi-source heterogeneous data is assigned an accurate timestamp based on a system unified time reference.

[0091] The data frames containing spatial position information in the multi-source heterogeneous data are input into a coordinate transformation engine.

[0092] The timestamp of the data frame is read, and the vehicle attitude is obtained by interpolation based on inertial measurement unit data.

[0093] In combination with the preset sensor external parameters describing the installation position and attitude of the sensor, the multi-source heterogeneous data in the coordinate system of each sensor is transformed to a unified ego vehicle coordinate system through rigid body transformation.

[0094] The converted multi-source heterogeneous data is subjected to data fusion to generate a fusion object, including:

[0095] It is identified which measurements from different sensors actually point to the same physical object, such as the system determining that the front radar target A, LiDAR cluster B and camera-recognized vehicle C are highly consistent in spatial position and speed, and therefore associating them together.

[0096] For each successfully associated object, the system uses a Kalman filter algorithm to combine measurements from multiple sensors with different noise characteristics into a single, more robust, more accurate state description.

[0097] The converted multi-source heterogeneous data is subjected to validity detection and abnormality elimination, and the cleaned multi-source heterogeneous data is encapsulated into a data structure with unified fields, including identification information, a timestamp, a position, a speed vector, a size, a category, a source sensor, and a confidence level.

[0098] The data structure with unified fields is stored in a preset cache queue.

[0099] Standardized scene data is extracted from the cache queue at a fixed frequency and published to a corresponding predefined data topic, so as to be subscribed and used by subsequent modules.

[0100] The converted multi-source heterogeneous data is subjected to validity detection and abnormality elimination, for ensuring that data processed by subsequent modules has basic authenticity, integrity, and timeliness, and the entire process includes:

[0101] Multi-source heterogeneous data is subjected to syntax and format verification, aiming to ensure that a data packet itself is structurally complete and can be parsed, including:

[0102] A protocol layer of a received data packet is parsed, and any data packet that cannot pass syntax parsing will be discarded immediately, and a parsing error log is recorded, so as to prevent format error data from causing a subsequent processing module to crash;

[0103] After successful parsing, it is checked whether key fields exist and whether their values are within an expected physical or logical range, and data that lacks key fields or whose field values are obviously beyond a reasonable range will be marked as “low quality” or discarded directly, and a specific strategy depends on the importance of the data.

[0104] For each received V2X message (such as BSM and SPaT), a corresponding authentication algorithm is executed for verification, including:

[0105] A certificate in the message is extracted, and it is verified whether the certificate is issued by a trusted root certificate authority (CA) and whether it is within a valid period (usually through a preloaded certificate trust list CTL);

[0106] A public key in the certificate is used to decrypt and hash compare a signature and content of the message, and any message with a failed signature verification or invalid certificate will be discarded immediately and a security alert will be triggered, because it may be a spoofing or forgery attack;

[0107] For cloud data transmitted through protocols such as TCP / IP, verify the checksums of its transport layer and application layer (such as TCP Checksum, SHA-256 Hash) to ensure that the data has not been tampered with accidentally or maliciously during transmission. Data packets that fail the verification will be discarded and may be requested to be retransmitted.

[0108] Phase Three: Timeliness Verification

[0109] Ensure that the system uses "fresh" data to prevent replay attacks and incorrect decisions based on outdated information, including:

[0110] Compare the timestamp of the data packet with the current high-precision system timestamp stamped by SDFM. Calculate the time difference between the two. If the time difference exceeds a validity window preset for the data type (for example, 100 ms for BSM messages and 5 minutes for weather data), the data is considered outdated and discarded.

[0111] Maintain a cache of recently received messages that stores the unique identifier of the message. When a new message is received, first query the cache. If the message's identifier already exists, it is a duplicate message and is immediately discarded, which effectively prevents simple replay attacks.

[0112] Use physical laws and prior knowledge to eliminate obviously unreasonable data points, including:

[0113] Define a precise 3D ego mask that includes the vehicle body, rearview mirrors, roof racks, and other components. Traverse each point in the point cloud. If its coordinates are within the ego mask, mark and remove it.

[0114] Apply a combination of filters to remove points with a reflection intensity below a preset threshold, which are usually suspended in the air (rain, snow, dust); remove points beyond the LiDAR's effective working range; calculate the distance distribution of each point with other points in its neighborhood, and remove those isolated points that significantly deviate from the distribution. Through the above combination method, a cleaner and more focused point cloud on real environmental obstacles can be obtained.

[0115] Continuously track the Doppler velocity of each target. If a target's velocity is consistently close to zero over multiple consecutive frames, and its position matches a static element in the high-definition map (such as a guardrail, streetlight pole), it is classified as static clutter, reducing its threat level or removing it from the dynamic target list.

[0116] A preliminary cross-source plausibility check is performed, for example, if the cloud API reports that it is currently "sunny", but the camera image detects that the windshield wipers are working at maximum speed, the weather API data is then reduced in reliability score; if a V2X message claims that a car is at a certain location, but the LiDAR does not detect any object at that location at all, the reliability of the V2X message is also reduced, the result of this cross-validation is not immediately discarded, but is attached to the data in the form of a "reliability score" for the downstream fusion and decision-making module to reference.

[0117] Through the above four-stage cleaning process, noise, invalid, outdated and false information can be effectively filtered out, providing a high-quality and high-reliability information foundation for the entire decision-making system.

[0118] In this embodiment, the standardized scene data is further combined with environmental prior constraints to generate an adaptive digital twin environment, including:

[0119] Subscribing to and receiving standardized scene data published to different data topics, wherein: the vehicle state topic provides a vehicle state vector, the fusion object topic provides a dynamic target list containing dynamic traffic participant data, and the map update topic provides road and environmental information.

[0120] Updating the vehicle state according to the vehicle state vector, including the pose and dynamics state of the vehicle.

[0121] Updating the spatial position and motion state of the traffic participants in the digital twin environment according to the dynamic target list.

[0122] Extracting rainfall, illumination, and sensor working state from the map update topic to construct an environmental factor vector.

[0123] Inputting the environmental factor vector into the decay model library to calculate a set of performance decay coefficients of the environmental factor vector at the current time.

[0124] Generating a digital twin scene snapshot containing the vehicle state and the digital twin environment, and publishing it to the data topic of the twin scene.

[0125] Publishing the set of performance decay coefficients to the data topic of the decay parameter for subsequent simulation prediction calls.

[0126] After the above steps are completed, the following steps are performed in a loop: waiting for the next data update, returning to the subscription and receiving step, and entering a new twin construction loop.

[0127] Further, the performance degradation model library pre-stores performance degradation mathematical models for different environmental factor vectors, for mapping environmental factors to performance degradation coefficients, including: mapping the performance degradation coefficient of rainfall according to the LiDAR degradation model:

[0128]

[0129] wherein, represents the intensity of the LiDAR signal after a propagation distance d, represents the initial signal intensity, represents the extinction coefficient.

[0130] mapping the performance degradation coefficient of light and sensor working state according to the camera degradation model:

[0131]

[0132] wherein, K represents the camera performance degradation coefficient, represents a nonlinear light mapping function describing the relationship between light intensity and image signal-to-noise ratio, represents a weight coefficient of the light mapping function, represents a lens dirt confidence output by a visual algorithm for detecting camera lens dirt, represents a weight coefficient of the lens dirt confidence.

[0133] Further, performing graph reasoning on the standardized scene data to generate situational semantic labels, including:

[0134] continuously monitoring the standardized scene data in the data theme.

[0135] in response to any of the standardized scene data changing significantly, such as a vehicle entering a new geofence, a time entering a new preset time period, receiving a new weather or event alert, mapping the standardized scene data to a pre-constructed vehicle knowledge graph entity.

[0136] the vehicle knowledge graph entity includes: spatial mapping matches vehicle GPS coordinates to geographic entities, time mapping matches system time to time entities, event mapping matches event data to event entities, and environment mapping matches environmental data such as rainfall to environmental entities.

[0137] generating a structured query request based on the mapping result, i.e., the vehicle knowledge graph entity.

[0138] performing graph traversal and rule matching on the vehicle knowledge graph according to the structured query request, finding potential risks associated with risk path from matching entities to risk entities, and storing to a risk type list.

[0139] The confidence fusion of the results pointing to the same risk entity is performed based on the risk path, and a probability correction factor corresponding to the risk entity is generated. The confidence fusion includes: when multiple paths point to the same risk entity, the inference engine fuses according to a preset weight or a simple logic (OR) to obtain a comprehensive risk judgment.

[0140] The risk type list and the probability correction factor are packaged into a structured scenario semantic label in a standard format. The format includes a context ID, a timestamp, a scenario name, a contributing entity, an inferred risk list containing a risk type and a probability modifier, and a recommended posture.

[0141] The inference engine packages the final result into a structured scenario semantic label in a standard format, and publishes the scenario semantic label to the data topic of the internal data bus.

[0142] In response to any of the standardized scenario data not changing significantly, the published scenario semantic label is published again to the data topic of the internal data bus. If the key input data does not change at the next time step, the vehicle is still in the same area, and the time period does not change, the last semantic label is directly reused and published to avoid unnecessary repeated calculation, thereby ensuring system efficiency. Only when the trigger condition is met again, the complete inference process is re-executed.

[0143] In the embodiment, the system constructs a multi-dimensional and hierarchical event-driven trigger mechanism for judging whether the standardized scenario data has changed significantly. The mechanism continuously monitors the standardized data stream, and only when the change in one or more dimensions meets the preset quantitative condition, the complete semantic scenario re-evaluation is triggered. The mechanism mainly includes the following four types of quantitative triggers:

[0144] The spatial and geographical trigger is used to monitor the change of the position and topological relationship of the vehicle in the geographical space, including: the system preloads a database containing a plurality of semantic geographical fences, the trigger calculates the relationship between the current coordinates of the vehicle and the polygons in real time, and triggers when the geographical fence state of the vehicle changes discretely;

[0145] The system defines different "influence radius" thresholds for different types of POIs, and triggers when the Euclidean distance of the vehicle to a high-priority POI first crosses the influence radius threshold;

[0146] Based on high-precision map positioning, the system can obtain the road attribute to which the current lane belongs, and triggers when the type enumeration value of the road changes.

[0147] Time trigger, based on system clock, monitors time entering predefined semantic time period, including: defining a time period configuration file within the system, dividing a day into multiple semantic time periods, triggering when the system clock crosses any defined time period start or end time point.

[0148] Dynamic environment data trigger, monitoring continuously changing external environment data, triggering when its change amplitude or state exceeds a threshold, including: mapping continuous weather parameters to discrete state levels; triggering when the mapped weather state enumeration value changes;

[0149] By fusing sensor data, calculating the vehicle density around the vehicle, and dividing it into different levels, triggering when the traffic density level changes and the duration of the new level exceeds a small hysteresis window.

[0150] Explicit event triggers are the highest priority triggers, which are directly triggered by specific, high-value information sent by external systems, including:

[0151] The system maintains a "high-priority message type" list, and when a V2X or cloud message of a type in the list is received and verified, the semantic context reasoning is triggered immediately and unconditionally.

[0152] Therefore, through the multi-dimensional trigger mechanism, "on-demand evaluation" is realized, which concentrates computing resources at the right moment, ensuring the system's rapid response to key context changes and greatly optimizing the overall computing load. The above quantitative indicators are designed as configurable, calibratable and adjustable parameters.

[0153] In this embodiment, further, based on the adaptive digital twin environment and scenario semantic label, a set of probabilistic behavior hypotheses for key dynamic targets in the scene are generated, and parallel simulation deduction is performed in combination with candidate actions to generate a decision evaluation matrix, including: subscribing to and receiving the decay parameters, digital twin scene snapshots and scenario semantic labels.

[0154] Based on the current vehicle state and task, sampling in the two-dimensional space of acceleration and curvature generates a set of candidate action sequences.

[0155] Based on vehicle dynamics constraints and driving tasks such as navigation targets, sampling in the two-dimensional space of acceleration and curvature generates a set of discrete, feasible candidate action sequences in a short time (such as 3-5 seconds) in the future.

[0156] According to the pre-set dynamic target basic behavior model library and pre-defined behavior hypotheses, a set of basic behavior hypotheses for key dynamic targets in the digital twin scene snapshot are generated.

[0157] adjusting a prior probability of the base behavior hypothesis based on a probability modifier factor in the situational semantic label.

[0158] normalizing the prior probability and merging with the base behavior hypothesis to generate a set of probabilistic behavior hypotheses.

[0159] the parallel simulation engine loads the latest digital twin scenario and attenuation parameters.

[0160] the candidate action sequence is combined with all elements in the set of probabilistic behavior hypotheses to build a simulation task tree and distributed to individual computing cores of HPC (High Performance Computing).

[0161] the computing cores perform simulation deduction at time steps (Δt = 0.1 seconds) far beyond real time until a preset simulation duration is reached. In each time step, the future scenario is deduced in parallel, including:

[0162] updating the ego vehicle state based on the candidate action sequence.

[0163] updating the target state based on the set of probabilistic behavior hypotheses.

[0164] calling a virtual sensor model to apply the attenuation parameters to simulate ego vehicle perception.

[0165] the computing cores record key simulation data including inter-vehicle distance, relative speed, and virtual sensor perception results during the simulation process.

[0166] calculating the single-task cost in the simulation task tree based on the key simulation data.

[0167] weighting the single-task cost and the corresponding hypothesis probability in the set of probabilistic behavior hypotheses to calculate the expected total cost of each candidate action.

[0168] the candidate action sequence and its corresponding expected total cost are composed into a decision evaluation matrix and published to the decision evaluation topic of the internal data bus for downstream decision invocation.

[0169] Further, in the embodiment, calculating the single-task cost in the simulation task tree based on the key simulation data includes:

[0170] calculating the single-task cost according to the following cost function:

[0171]

[0172] where J represents the single-task cost, represents the weight coefficient, represents the cost component;

[0173] the cost component includes:

[0174] C_safety: safety cost calculated based on the minimum TTC (Time-to-Collision) or minimum inter-vehicle distance in the simulation, which is infinite if a collision occurs.

[0175] C_efficiency: efficiency cost measured by the time, distance or average speed deviation required for the vehicle to reach the target point.

[0176] C_comfort: comfort cost calculated based on longitudinal acceleration and lateral acceleration.

[0177] C_legality: legality cost based on the detection of whether the candidate action violates traffic rules.

[0178] C_social: social acceptance cost generated by combining semantic context labels to socially evaluate actions.

[0179] In this embodiment, further, based on the decision evaluation matrix, the user driving preferences and the preset decision criteria are combined to select and issue the optimal driving action, and the driving action is converted into control instructions executable by the vehicle, including: subscribing to and receiving the decision evaluation matrix.

[0180] Based on the highest risk value in the decision evaluation matrix and its corresponding scenario semantic label, the user preference configuration parameters are combined to select the decision criteria for this decision cycle. When the safety cost in the decision evaluation matrix exceeds the preset threshold, or the scenario semantic label contains high-risk, the decision criteria of minimizing the maximum loss is selected. When the user preference configuration parameter indicates a balanced driving mode, the decision criteria of maximizing expected utility is selected. When the user preference configuration parameter indicates a conservative driving mode, the triggering sensitivity of the minimax decision criteria is increased.

[0181] According to the decision criteria, the decision evaluation matrix is calculated to select the corresponding optimal candidate action.

[0182] The target trajectory corresponding to the optimal candidate action is converted into original control instructions recognizable by the vehicle bottom layer actuator, including target acceleration and target steering angle.

[0183] The original control instructions are filtered and smoothed, and the amplitude is limited within the vehicle dynamics constraint range to obtain the control instructions.

[0184] The final control instructions are issued to the vehicle motion control module to drive the physical actuator to complete the optimal driving action.

[0185] Further, based on the execution result of the control instruction, the cost function weight and the decision parameter are closed-loop optimized and adaptively adjusted, including: subscribing and receiving the standardized scene data as real world observation data.

[0186] Subscribing and receiving the decision evaluation matrix, the set of probabilistic behavior assumptions, and the control instruction as prediction data.

[0187] After aligning the real world observation data and the prediction data in time and space, the inconsistency score of the real world observation data and the prediction data is calculated according to the preset metric function library. The inconsistency metric function library provides multiple methods for calculating inconsistency.

[0188] The inconsistency score is compared with the dynamic threshold value corresponding to the current scene.

[0189] In response to the inconsistency score being greater than the dynamic threshold value, the case file related to the current cognitive bias event is traced back and collected.

[0190] The case file includes:

[0191] Input layer: including standardized scene data.

[0192] Twin layer: including digital twin scene snapshot and decay parameter.

[0193] Cognitive layer: including scenario semantic label.

[0194] Prediction layer: including set of probabilistic behavior assumptions and prior probability.

[0195] Decision layer: including optimal candidate action and evaluation matrix.

[0196] Result layer: real trajectory of all targets in the physical world.

[0197] Labeling: the engine labels this case file with metadata such as [event ID, timestamp, inconsistency score, bias type: horizontal prediction failure, trigger context: rainy intersection].

[0198] According to the metadata of the case file, a model optimization task is triggered, including online fine-tuning of a lightweight model and joining an offline training pool in the cloud.

[0199] In response to the inconsistency score being less than the dynamic threshold value, the cache is emptied.

[0200] The beneficial technical effects of the embodiment are: by combining semantic information in complex traffic scenarios with dynamic target behavior prediction through vehicle-mounted knowledge graph reasoning and probability correction mechanism, the limitations of relying only on kinematic history and lacking robustness in the prior art are overcome, and the perception and prediction ability of autonomous driving in unstructured and unexpected scenarios is improved; by using adaptive digital twin environment, multi-hypothesis simulation deduction and closed-loop learning optimization, prediction and planning are tightly coupled, and the cost function weight and model parameters are dynamically corrected, breaking through the bottleneck of rigid decoupling of prediction and planning and offline validation of simulation in the prior art, thereby realizing flexibility, safety and efficiency of online decision-making.

[0201] Embodiment 2

[0202] In combination with Embodiment 1, in Embodiment 2, an automatic driving situational awareness and decision-making system is included, which is designed as a software and hardware cooperative system deployed on an intelligent connected vehicle, mainly composed of six core modules that interact at high speed and low delay through standardized data interfaces. The six modules are: a multi-source data fusion module (SDFM), a dynamic digital twin construction module (DDTCM), a semantic situation reasoning module (SCRM), a multi-hypothesis simulation prediction module (MHSPM), an optimal behavior decision-making module (OBDM), and a closed-loop learning optimization module (CLOM). The device connection diagram of the embodiment is shown in Figure 2 , which includes:

[0203] The multi-source data fusion module is used for time synchronization, spatial registration and semantic fusion of multi-source heterogeneous data from vehicle-mounted sensors to generate standardized scene data.

[0204] The dynamic digital twin construction module is used to combine the standardized scene data with environmental prior constraints to generate an adaptive digital twin environment.

[0205] The semantic situation reasoning module is used to perform graph reasoning on the standardized scene data to generate situational semantic labels.

[0206] The multi-hypothesis simulation prediction module is used to generate a set of probabilistic behavior hypotheses for key dynamic targets in the scene based on the adaptive digital twin environment and situational semantic labels, and to perform parallel simulation deduction in combination with candidate actions to generate a decision evaluation matrix.

[0207] The optimal behavior decision-making module is used to select and issue optimal driving actions based on the decision evaluation matrix in combination with user driving preferences and preset decision criteria, and to convert the driving actions into vehicle executable control instructions.

[0208] The closed-loop learning optimization module is used to perform closed-loop optimization and adaptive adjustment of cost function weights and decision parameters based on the execution results of the control instructions.

[0209] The input layer of the system is: vehicle-mounted sensors, V2X OBU, GNSS / IMU, cloud platform interface (for weather API, traffic event API, etc.).

[0210] The processing layer of the system is: SDFM receives all inputs, performs timestamp alignment and coordinate system unification. DDTCM obtains data from SDFM, internally containing a vehicle twin sub-module, an environment twin sub-module, and a key sensor attenuation sub-module. SCRM receives data from SDFM and DDTCM, and its core is a traffic situation knowledge graph. MHSPM receives output from DDTCM and SCRM, internally containing a behavior hypothesis generator, a parallel simulation engine, and a multi-object evaluator. OBDM receives the evaluation results of MHSPM and outputs control instructions. CLOM runs in parallel and continuously monitors the data flow of SDFM and MHSPM, and its core is an inconsistency monitoring unit.

[0211] The output layer of the system is: vehicle control instructions (output to CAN / Ethernet bus), high-value learning samples (output to cloud training platform).

[0212] In this embodiment, further, the multi-source data fusion module (SDFM - Sensor & Data Fusion Module) is a software service running on the vehicle-mounted high-performance computing unit (HPC). It contains multiple data adapters corresponding to cameras, lidar, millimeter wave radar, GNSS / IMU, V2X OBU, and cloud data interfaces. Its role is to convert heterogeneous data sources from different physical characteristics, different sampling frequencies, different coordinate systems, and different communication protocols into unified, synchronized, and formatted data streams, providing a clean, reliable, and ready-to-use data base for all subsequent advanced processing modules.

[0213] SDFM is designed as a multi-level software architecture running on the vehicle-mounted high-performance computing unit (HPC) and tightly coupled with the vehicle's hardware interface. The structure diagram of the multi-source data fusion module is shown in Figure 3 The internal structure can be divided into three layers:

[0214] The hardware adaptation layer (Hardware Adaptation Layer - HAL) includes a series of dedicated device drivers and software adapters:

[0215] Sensor driver: the hardware adaptation layer directly communicates with the ECU of the vehicle-mounted sensor, such as receiving image data from the camera, point cloud data from the lidar, target list data from the millimeter wave radar, and distance data from the ultrasonic radar through the CAN bus or vehicle-mounted Ethernet.

[0216] GNSS / IMU Adapter: Through serial port or UDP protocol, parse NMEA messages or custom format data from high-precision positioning units, extract latitude, longitude, altitude, speed, heading, attitude angle (roll, pitch, yaw) and other information.

[0217] V2X OBU Adapter: Listen to the network interface from the on-board communication unit (OBU), parse standard-compliant V2X messages such as Basic Safety Message (BSM), Map Message (MAP), Signal Phase and Timing Message (SPAT), etc.

[0218] Cloud Platform Interface: Responsible for communicating with cloud servers through the 4G / 5G connection of the on-board T-Box. It actively requests or passively receives data services from the cloud, such as weather APIs, real-time traffic event APIs, high-precision map dynamic layer updates, etc.

[0219] Data Processing Core is the core logic of SDFM, composed of three key components:

[0220] Time Synchronization Manager: Responsible for maintaining a unified and high-precision time reference for the system. It synchronizes with the PTP (Precision Time Protocol) master clock on the vehicle or the PPS (Pulse Per Second) signal provided by the GNSS module to ensure the accuracy of all timestamps.

[0221] Coordinate Transformation Engine: Stores precise calibration parameters for the vehicle, including the precise three-dimensional position (x, y, z) and installation attitude (roll, pitch, yaw) of each sensor relative to the vehicle coordinate system origin, which is usually the rear axle center.

[0222] Data Formatting and Buffer Queue: Defines standard data structures for interaction between all modules in the system, and sets up thread-safe ring buffer areas with fixed lengths for each data type.

[0223] Data Service Publication Layer is implemented using a high-efficiency data distribution mechanism in the Publish / Subscribe mode, including:

[0224] Data Topic: Define a "topic" for each processed data stream, such as: topic_camera_front_image, topic_lidar_points, topic_fused_objects, topic_vehicle_state.

[0225] Publisher: Publish formatted data to the corresponding topic.

[0226] Subscriber interface: Provide APIs to other modules in the system, such as DDTCM, SCRM, allowing them to subscribe to the data topics of interest on demand.

[0227] The running of SDFM is a continuous, high-concurrency pipeline process. A typical running cycle is as follows, as shown in Figure 4

[0228] Step 1 is parallel data acquisition and timestamp calibration:

[0229] The various adapters and drivers of the HAL layer are independent threads or processes that receive raw data from their respective hardware interfaces in parallel, such as a frame of image, a packet of point cloud, or a BSM message.

[0230] At the first time the data arrives at the HAL, the time synchronization manager immediately assigns it an accurate timestamp based on the system unified time reference. This step is crucial because it minimizes the time error caused by operating system scheduling delays.

[0231] Step 2 is coordinate system unification processing:

[0232] All data containing spatial position information, such as laser radar point cloud, radar target, and location in V2X message, are sent to the coordinate transformation engine.

[0233] The engine reads the timestamp of the data frame and interpolates the vehicle pose at that precise moment from the IMU data stream. Then, using the pre-stored sensor external parameters including installation position and pose, it converts all point coordinates from their respective sensor coordinate systems to the unified ego vehicle coordinate system through standard rigid body transformation (rotation + translation) mathematical operations.

[0234] Step 3 is data preliminary cleaning and formatting:

[0235] Before entering the buffer queue, the data will undergo a preliminary cleaning. This includes removing invalid laser radar points, such as reflection points from the inside of the vehicle, and verifying the signature of V2X messages to ensure their reliable source.

[0236] The cleaned data is encapsulated into a pre-defined standard data structure. In this embodiment, a FusedObject structure contains fields such as id, timestamp, position_in_vehicle_frame, velocity_vector, dimensions, classification (vehicle / pedestrian), source_sensor_list, confidence, etc.

[0237] ​Step 4 is data write to buffer queue:

[0238] The formatted data object is written into the corresponding ring buffer of the data processing core layer. The advantage of using a ring buffer is that it can guarantee the freshness of data, automatically overwrite the oldest data, while avoiding the overhead and uncertainty brought by dynamic memory allocation.

[0239] Step 5 is data publishing:

[0240] The publisher thread of the data service publishing layer takes the latest data from each buffer queue at a fixed high frequency (100 Hz in this embodiment).

[0241] The data object is published to its corresponding data topic. In this embodiment, the latest FusedObject list is published to the topic_fused_objects topic. At this time, all other modules that subscribe to this topic (such as the environment twin sub-module of the DDTCM) will receive a data update notification almost simultaneously and can immediately access these fresh, synchronized, and formatted data.

[0242] Further, in this embodiment, the dynamic digital twin construction module (DDTCM - Dynamic Digital Twin Construction Module) is a "high-fidelity real-time sand table" before the behavior decision of this embodiment. Its core task is no longer to create a general virtual world for offline testing, but to construct and maintain a high-fidelity, adaptive online digital mirror for the current physical vehicle in the specific environment at this moment. It will reproduce the geometric and physical state of the vehicle and the environment, and can synchronize the performance fluctuations of the physical vehicle caused by internal wear and external environmental influences in real time, especially the performance degradation of sensors.

[0243] The DDTCM is composed of three interrelated and dynamically updated sub-modules, which together build a complete digital twin scene that coexists with the real world. Its structure diagram is shown in Figure 5 , including the vehicle twin sub-module, the environment twin sub-module, and the sensor degradation sub-module.

[0244] Vehicle Twin Sub-module: Contains a vehicle dynamics model (VDM), a vehicle state vector (VSV), and a slowly varying parameter database (SPDB).

[0245] The vehicle dynamics model employs a simplified, yet highly sensitive mathematical model to key decision-making parameters (including braking distance, steering response), including the Single-track Bicycle Model and more complex multi-body dynamics models. The model receives control commands (target acceleration, steering angle), and outputs the vehicle's motion state.

[0246] The vehicle state vector (VSV) is a real-time updated data structure that stores the vehicle's instantaneous state, such as [x, y, z, yaw, pitch, roll, vx, vy, ax, ay, steering_angle, brake_pressure], obtained from the SDFM. This vector is used to initialize the VDM's state at the beginning of each simulation step, ensuring that the simulation starts exactly as the physical vehicle.

[0247] The slow-varying parameter database (SPDB) stores vehicle performance parameters that do not change dramatically with a single drive, but change over time due to wear and tear. The VDM dynamically loads parameters from the SPDB during simulation calculations, allowing the simulation results to reflect the vehicle's long-term health. The vehicle performance parameters include:

[0248] tire_friction_coefficient: tire friction coefficient, which can be learned and updated by the CLOM over time based on historical slip data.

[0249] brake_system_delay: brake system response delay, which can be calibrated offline.

[0250] power_consumption_model: battery / engine power consumption model.

[0251] Environment Twin Sub-module: contains a static environment layer and a dynamic entity layer.

[0252] The static environment layer is based on high-definition map (HD Map) data. It contains the road's geometric topology (lane lines, road boundaries), curvature, slope, and three-dimensional models of fixed elements such as traffic signs, signals, and buildings. This layer is relatively stable and only modified when HD Map updates are received.

[0253] The dynamic entity layer is the most dynamic part of the scene, maintaining a dynamic target list, with each object in the list being a data structure containing information obtained from different topics published by the SDFM. This layer is refreshed in real-time at a high frequency (e.g., 10-30 Hz) to ensure that traffic participants in the virtual scene are fully synchronized with the physical world.

[0254] Sensor Degradation Sub-module: Its core is a configurable Degradation Model Library (DML) and a Performance Applicator Engine (PAE). The DML pre-stores the performance degradation mathematical models for different sensors. These models map environmental factors to performance degradation coefficients, including:

[0255] According to the LiDAR degradation model, map the performance degradation coefficient of rainfall:

[0256]

[0257] where, represents the intensity of the LiDAR signal after a propagation distance d, represents the initial signal intensity, represents the extinction coefficient, which is a function related to the rainfall intensity (mm / h) or visibility (m), and can be implemented by lookup table method (Lookup Table) or a small neural network f(rain, fog);

[0258] According to the camera degradation model, map the performance degradation coefficient of light and sensor working state:

[0259]

[0260] where K represents the camera performance degradation coefficient, represents a (non-linear) light mapping function that describes the relationship between light intensity and image signal-to-noise ratio, which is a non-linear function that describes the relationship between light intensity and image signal-to-noise ratio, represents the weight coefficient of the light mapping function, from a visual algorithm specially designed to detect camera lens dirt, lens dirt confidence, represents the weight coefficient of the lens dirt confidence.

[0261] The Performance Applicator Engine (PAE) applies the degradation coefficients calculated by the DML to the virtual sensor models called by the subsequent simulation module (MHSPM). When the MHSPM simulates the "perception" process in the digital twin, the PAE will intercept this process and modify the behavior of the virtual sensor. For LiDAR, the PAE will reduce the effective detection distance of the virtual LiDAR point cloud according to the extinction coefficient and increase the random noise points according to the rain and fog model.

[0262] For cameras, PAE will "degrade" the images generated by virtual cameras through image processing algorithms, including reducing contrast, increasing Gaussian noise, simulating glare, and then sending them to the virtual perception algorithm for processing.

[0263] DDTCM runs in a continuous cycle to ensure the close coupling of digital twins with the physical world. A typical running cycle is as follows, as shown in Figure 6

[0264] Each submodule of DDTCM subscribes to the relevant topics published by SDFM. The vehicle state topic provides the vehicle state vector, the fusion object topic provides the dynamic target list, and the map update topic provides road and environmental information.

[0265] The vehicle twin submodule updates its internal vehicle state according to the latest VSV data received.

[0266] The environment twin submodule refreshes the location, speed, and other information of traffic participants in the virtual scene according to the latest dynamic target list.

[0267] The sensor attenuation submodule extracts key environmental factors from the map update topic, including rainfall, visibility, and illumination.

[0268] The sensor attenuation submodule inputs the extracted environmental factor vector into the corresponding model in the DML, and the model calculates the performance attenuation coefficient of each sensor at the current time.

[0269] DDTCM publishes the constructed digital twin scene snapshot containing the complete vehicle state and environmental state to the data topic of the twin scene for MHSPM to call, and publishes the calculated sensor attenuation coefficient set to the data topic of the attenuation parameter.

[0270] Finally, the module returns to the subscription and reception step, waits for the next data update from SDFM, and enters a new construction cycle.

[0271] ​In this embodiment, further, the semantic context reasoning module (SCRM) is the primary task to transform the massive, heterogeneous, low-level data stream provided by the SDFM into high-level, structured, and machine-understandable semantic context. It does not care about "there is a moving object 50 meters ahead", but strives to answer "we are in a 'rainy night construction section', which means that the moving object ahead is most likely a tired worker, and the ground is wet, so the braking distance will increase". SCRM performs deep reasoning on the current scene through a pre-built knowledge base, provides a "scene characterization" conclusion with a clear risk direction for the subsequent risk prediction and decision-making module, so that the vehicle's behavior decision-making has similar human "common sense" and "predictability". The core of SCRM is a vehicle-mounted, optimized knowledge reasoning system. It mainly consists of three tightly coupled sub-units: a vehicle-mounted traffic situation knowledge graph, a data abstraction and query generation unit, and a reasoning engine and label generator, as shown in Figure 7

[0272] The vehicle-mounted traffic situation knowledge graph is a graph database stored in the non-volatile memory of the vehicle-mounted HPC. It is composed of entities and relationships, and the entities include:

[0273] Geographical entities are derived from high-precision map POIs, including: schools, hospitals, shopping malls, bar streets, highway entrance, construction zones.

[0274] Time entities are predefined time periods, including: weekday morning rush hour, night, holiday, weekend afternoon.

[0275] Event entities are dynamically updated through cloud services, including: sports events, concerts, temporary traffic control.

[0276] Environmental entities are abstracted from environmental data, including: heavy rain, dense fog, icy road, strong backlight.

[0277] Risk entities are predefined typical risk patterns, including: pedestrian suddenly rushing out, vehicle ghost head, drunk driving, non-motor vehicle reverse.

[0278] The relationship is used to define the association between entities, including:

[0279] locatedAt represents the "located at" relationship, which is used to describe the association between geographical entities and specific geographical locations. For example, locatedAt (school, GPS polygon area) indicates that the "school" entity is located within a certain GPS polygon area.

[0280] ​hasProperty represents the has property relationship, which is used to describe that an entity has certain specific properties or characteristics, for example, hasProperty (school, {dismissal time: '15:00-16:30'}) represents that the school entity has an attribute, that is, the dismissal time is from 3:00 pm to 4:30 pm.

[0281] triggersRisk represents the triggers risk relationship, which is used to describe that a certain entity directly triggers a certain risk event. For example, triggersRisk (bar street, drunk driving) represents that the bar street entity is prone to trigger the drunk driving risk.

[0282] increasesRisk represents the increases risk relationship, which is used to describe that a certain entity increases the probability or intensity of another risk event, for example, increasesRisk (heavy rain, pedestrian suddenly rushing out) represents that the heavy rain environment entity will increase the probability of occurrence of the pedestrian suddenly rushing out risk.

[0283] co_occursWith represents the co-occurrence or simultaneous occurrence relationship, which is used to describe that two entities often appear at the same time in time or space, for example, co_occursWith (sports event, temporary traffic control) represents that the sports event entity usually occurs at the same time as the temporary traffic control event entity.

[0284] The basic knowledge graph is pre-installed when the vehicle is shipped, and the event entity and part of the relationship can be updated periodically and incrementally through cloud services.

[0285] The data abstraction and query generation unit is a series of data classifiers and logical rule engines. This unit is responsible for translating continuous and raw data streams from SDFM into discrete entities that the knowledge graph can understand:

[0286] First, match the current GPS coordinates of the vehicle with the geographical entities in the knowledge graph, for example, through the geographic fence to determine whether the vehicle has entered the "school" area.

[0287] Further, match the current system time to the pre-defined time entity, for example, 3:30 pm belongs to the afternoon of weekdays.

[0288] Further, check the event information obtained from the cloud to see if the current time and space coincide with a certain event entity.

[0289] Further, classify numerical data such as rainfall > 25mm / h as a heavy rain environment entity.

[0290] Based on the mapping result above, a structured query request is generated, for example: Find_Context({Location:'school', Time: 'weekday afternoon', Weather: 'heavy rain'}).

[0291] The reasoning engine and the label generator are an inference engine based on graph traversal and rule matching. After receiving the query request, the engine performs the inference task on the knowledge graph, which finds the path from the currently matched entity to the risk entity. For example, for the matched entities (school, heavy rain), it finds two paths:

[0292] Path 1: school-hasProperty({dismissal time})-[match with current time]-triggersRisk-sudden rush of pedestrians.

[0293] Path 2: heavy rain-increasesRisk-sudden rush of pedestrians.

[0294] When multiple paths point to the same risk entity, the reasoning engine will fuse according to the preset weight or simple logic to obtain a comprehensive risk judgment.

[0295] Finally, the reasoning engine packages the inference result into a structured semantic label with rich content. This label is the core output of SCRM.

[0296] The running of SCRM is an event-driven and periodic polling combined process. The running process of SCRM is shown in Figure 8 , which includes:

[0297] Step 1: After SCRM starts, it continuously listens to the relevant topics published by SDFM.

[0298] Step 2: When any key input data changes significantly, such as a vehicle entering a new geographic fence, the time entering a new preset period, receiving a new weather or event alert, the data abstraction unit is triggered.

[0299] Step 3: The data abstraction unit executes its mapping logic to convert the current raw data into a set of matched knowledge graph entities, and constructs an internal query request.

[0300] Step 4: The reasoning engine receives the query request and performs graph traversal and rule matching on the vehicle-mounted knowledge graph to find potential risks associated with the current entity. Finally, the results of multiple inference paths are fused and quantified.

[0301] Step 5: The reasoning engine packages the final result into a structured semantic label in a standard format, which is published to the data topic of the internal data bus for MHSPM subscription.

[0302] Step 6: If at the next time step, the key input data has not changed, i.e., the vehicle is still in the same region, the time period has not changed, the SCRM will directly reuse and publish the last semantic label, avoiding unnecessary repeated calculations to ensure system efficiency. Only when the trigger condition is met again will the complete reasoning process be re-executed.

[0303] In this embodiment, further, the Multi-Hypothesis Simulation & Prediction Module (MHSPM) is a highly realistic, real-time digital twin environment that simulates the "future" before the vehicle makes any physical action. This module automatically generates multiple behavior hypotheses that are consistent with the situational logic and evaluates the consequences of the vehicle in response to each hypothesis. Through this analysis, the MHSPM can identify driving strategies that are robust under various possibilities, i.e., the driving strategies with the lowest risk and the highest return, thereby making the vehicle's final decision robust and intelligent.

[0304] The structure diagram of the MHSPM is shown in Figure 9 The MHSPM is a computationally intensive module that consists of four interdependent sub-units: a candidate action generator, a probabilistic behavior hypothesis generator, a parallel simulation engine, and a multi-objective evaluator.

[0305] The candidate action generator consists of a sampler based on vehicle dynamics constraints and driving tasks, responsible for generating a set of discrete, feasible driving action sequences for the vehicle in the near future (3-5 seconds). Actions are usually sampled in the two-dimensional space of acceleration and curvature. Its sampling strategy is lattice sampling, i.e., uniformly generating N candidate actions in the action space, for example:

[0306]

[0307] Further, the candidate action generator, in combination with the high-level navigation task, generates more task-related actions with bias, and finally outputs a set of candidate action sequences, for example:

[0308] {A1: maintain lane speed, A2: slightly decelerate, A3: prepare to change lane to the right...}

[0309] The probabilistic behavior hypothesis generator consists of a context-driven model library and a probability adjustment engine. It generates a set of probabilistic, discrete behavior hypotheses for key dynamic targets in the scene, especially those with high uncertainty, such as pedestrians and non-motor vehicles, instead of a single trajectory, based on a pre-set dynamic target basic behavior model library and pre-defined behavior macros. Further, it generates a decision evaluation matrix by combining candidate actions and performing parallel simulation deduction.

[0310] The model library stores a variety of basic behavior models, including: CVM (Constant Velocity Model): Constant Velocity Model; Social-LSTM / Social-GAN: Deep Learning Prediction Model Considering Interaction.

[0311] The pre-defined behavior macros (Maneuver Templates) include parameterized templates of typical dangerous behaviors such as "sudden crossing" and "emergency braking".

[0312] In the hypothesis generator, a set of basic behavior hypotheses (such as "walking along the roadside" and "crossing") is first generated for the target, and then the probability modification factor in the semantic label is used to adjust the prior probability of these hypotheses to generate a set of probabilistic, discrete behavior hypotheses.

[0313] The parallel simulation engine is a simulation kernel manager that can run on GPU or dedicated hardware. The parallel simulation engine is the computing core of the system, responsible for performing large-scale simulation deduction. The engine first obtains the latest digital twin scene snapshot and sensor attenuation parameters from the DDTCM; further, the engine forms a large simulation task tree by combining all combinations of N candidate actions of the ego vehicle and K behavior hypotheses of M targets, and distributes them to multiple computing cores of the HPC. On each computing core, the simulator performs forward deduction at a time step (Δt=0.1s) much faster than real time, and in each step:

[0314] The ego vehicle updates its state according to its candidate action sequence.

[0315] Other targets update their states according to their behavior hypotheses.

[0316] Call the virtual sensor model with attenuation to simulate the ego vehicle's perception process.

[0317] Record key simulation data such as inter-vehicle distance and relative speed.

[0318] The multi-target evaluator consists of a cost function calculation unit and a risk aggregator. It is used to quantitatively score the results of each simulation after it is completed.

[0319] For each candidate action of the ego vehicle, it corresponds to a set of simulation results and their costs combined with all target behavior hypotheses, the aggregator needs to calculate the comprehensive expected cost of the action.

[0320] The MHSPM runs in a high-frequency decision-making cycle, usually consistent with the control frequency of the vehicle (10 Hz), and its running flowchart is shown in Figure 10 , including:

[0321] Step 1: The MHSPM subscribes to and receives the relevant topics of the decay parameters, digital twin scene snapshots, and scenario semantic labels. When a new round of input is received, the decision-making cycle begins.

[0322] Step 2: Receive semantic labels from SCRM, generate a set of candidate actions {A_i} of the ego vehicle based on the current vehicle state and task;

[0323] The probabilistic behavior hypothesis generator generates a set of probabilistic behavior hypotheses {H_j, P(H_j)} for key targets in the scene based on the latest semantic labels.

[0324] Step 3: The parallel simulation engine loads the latest digital twin scene and sensor decay parameters;

[0325] Build a simulation task tree and distribute (A_i, H_j) combinations to each computing core of the HPC.

[0326] Step 4: All computing cores start simultaneously and forward the future scene assigned to them at high speed until the preset simulation duration is reached.

[0327] Step 5: After each simulation task is completed, the multi-objective evaluator immediately calculates its corresponding cost Cost(A_i, H_j).

[0328] Step 6: After all simulation tasks are completed, the evaluator calculates the expected total cost of each candidate action A_i of the ego vehicle.

[0329] Step 7: The MHSPM packages the evaluation matrix containing all candidate actions and their corresponding expected costs into a structured scenario semantic label in a standard format, and publishes the scenario semantic label to the data topic of the internal data bus for the OBDM module to make the final decision.

[0330] This process ensures that the system has fully "thought" about the various possibilities of the future and their probabilities in the current situation before making a decision, so that it can make intelligent decisions that are both safe and efficient.

[0331] In this embodiment, further, the optimal behavior decision module (OBDM - Optimal Behavior Decision Module) is based on the evaluation matrix provided by the MHSPM, according to the preset, clear decision criteria, select and issue a final, optimal driving action. This module directly reflects the driving personality and moral ethics orientation in extreme situations, OBDM will complex, probabilistic evaluation results, convergence to a single, determined physical world control instruction, complete the key implementation from thinking to action.

[0332] OBDM is a lightweight but highly logical module, its structure diagram is shown in Figure 11 It consists of a user preference configuration unit, a decision criteria selection engine and an instruction generation and smoothing unit.

[0333] The user preference configuration unit consists of a parameter set storing the user's driving style preference, which can be set through the vehicle-mounted human-computer interaction interface. The parameter set stores a set of configurable driving modes and custom cost function weights. In this embodiment, the driving modes include conservative, balanced and sporty modes, and the custom cost function weights include safety, comfort and energy saving weights. These parameters will directly affect the selection of subsequent decision criteria or the weighting method of the cost function, for example, selecting the sporty mode may make the system more inclined to select actions with lower energy-saving cost, even if the comfort cost is slightly higher. The user preference configuration unit provides an API for HMI to call, allowing the driver to switch between safety, comfort and energy saving modes.

[0334] The decision criteria selection engine embeds multiple classical decision theory models. According to the current safety state of the vehicle and user preferences, select the most suitable decision criteria from its model library. Decision theory models include Minimax criteria, expected utility maximization, regret minimization and user preference influence:

[0335] Minimax (Minimax) criteria: This is the ultimate criterion for safety first. Its logic is to choose the scheme that minimizes our loss in all possible worst-case scenarios. It does not care how good an action's average performance is, only its bottom line in the worst case. Its calculation idea is that for each candidate action A_i, find its corresponding cost value max(Cost(A_i, H_j)) in all simulations, then select the action A_i with the smallest max(Cost).

[0336] Maximize Expected Utility: This is the commonly used criterion for equilibrium type, whose logic is to choose the one that has the highest overall expected benefit under all possibilities, and its calculation idea is to directly select the action A_i with the lowest ExpectedCost(A_i) calculated by MHSPM, that is, the one with the highest expected utility.

[0337] Minimize Regret: A more complex criterion, aiming to choose the action that is least likely to regret later, whose selection logic defaults to the Maximize Expected Utility criterion; when the semantic label in the SCRM contains high-risk or there is an extremely high safety cost in any one simulation, the engine automatically switches to the Minimax criterion and enforces the most conservative and safest decision.

[0338] User preference influence: User selection of Conservative mode will increase the sensitivity of triggering the Minimax criterion.

[0339] The instruction generation and smoothing unit has an action-to-control instruction converter and an instruction smoothing filter, which converts the selected discrete and abstract candidate action, such as "gentle deceleration and slight right turn", into a continuous and smooth control instruction that the vehicle's underlying actuators can understand. The selected candidate action is essentially a short-term target trajectory or state sequence, and this unit uses the inverse dynamics model of the vehicle to calculate the longitudinal target acceleration and lateral target curvature required to achieve this target trajectory.

[0340] In order to avoid the sudden change of control instruction causing the vehicle to shake or the passengers to be uncomfortable, the output control instruction sequence will be smoothed by a low-pass filter or PID controller. At the same time, this unit will check whether the instruction is within the physical capability range of the vehicle, such as maximum acceleration and maximum steering angular velocity, and perform necessary clipping. Finally, the instruction is output to the Vehicle Motion Control (VMC) module.

[0341] The operation flow of OBDM follows MHSPM and constitutes the last stage of the decision-making cycle, whose flow chart is shown in Figure 12 , including:

[0342] Step 1: OBDM subscribes to the decision evaluation matrix published by MHSPM to the data topic in the internal data bus, and obtains the current settings of the user preference configuration unit.

[0343] Step 2: The decision criterion selection engine analyzes the highest risk value and semantic label in the evaluation matrix, and selects the criterion to be used in this decision-making cycle in combination with user preferences.

[0344] Step 3: The module calculates the evaluation matrix according to the selected criteria;

[0345] Step 4: The instruction generation and smoothing unit receives the selected optimal action ID and finds its corresponding target trajectory from the candidate action list.

[0346] Step 5: The original control instructions required to achieve the trajectory are calculated through the inverse dynamics model, and are filtered, smoothed, and amplitude-limited.

[0347] Step 6: The final, smoothed control instructions are published to their corresponding topics, received by the vehicle's VMC module, and drive the physical actuators, i.e., the throttle, brake, and steering system, to complete the action.

[0348] This process ensures that the vehicle's final behavior is not only based on future considerations, but also meets the preset safety policy and user expectations.

[0349] In this embodiment, further, the closed-loop learning optimization module (CLOM) is the reflection and evolution engine of this embodiment. Its core mission is to establish an intelligent feedback loop from the physical world results to the digital twin model, achieving continuous self-optimization of the system. Unlike traditional methods that rely on massive amounts of undifferentiated data for offline training, CLOM captures high-value data samples that best expose the current model's flaws by monitoring the inconsistencies between predictions and reality. These samples are then used to specifically correct and optimize the core models of the system, such as the behavior prediction model and the sensor degradation model, thereby driving the iterative evolution of the system's cognitive abilities in the most efficient way.

[0350] CLOM is a background running, event-driven module. It starts working after a physical action occurs and is composed of three sub-units: a prediction-reality inconsistency monitoring unit, a high-value sample screening and labeling unit, and a model optimization task distribution unit. Its structure diagram is shown in Figure 13 .

[0351] The prediction-reality inconsistency monitoring unit is composed of a data aligner, a state comparator, and a set of inconsistency measurement functions, and is used to output a quantitative inconsistency score within a time window.

[0352] The data aligner caches the prediction data generated by the MHSPM before decision-making and the selected action of the ego vehicle by the OBDM. After the vehicle executes the action, it continuously receives real data from the SDFM, i.e., the actual observation trajectory of the key target. The data aligner is responsible for accurately aligning the two sets of data in time and space;

[0353]  For each target being monitored, the comparator compares its predicted state with the actual state at each time step.

[0354]  The library of inconsistency metrics provides a variety of ways to compute inconsistency, including:

[0355]  Lateral / Longitudinal Error: computes the lateral and longitudinal distance between the predicted and actual positions in the Frenet coordinate system.

[0356]  Mahalanobis Distance: a more advanced metric that takes into account the uncertainty of the prediction (covariance matrix). The Mahalanobis distance is large if the actual position falls outside the high-confidence region of the predicted probability distribution.

[0357]  Trajectory Similarity: such as Dynamic Time Warping (DTW) or Fréchet Distance, used to assess the shape similarity of the entire predicted trajectory with the actual trajectory.

[0358] The high-value sample screening and labeling unit consists of a dynamic threshold comparator and a data packaging and labeling engine. The dynamic threshold comparator is used to determine whether the inconsistency score is large enough to consider that a significant cognitive bias has occurred. The dynamic threshold is not fixed, but is context-dependent. For example, in the simple scenario of highway following, the dynamic threshold will be small, and slight deviations will be labeled; while at a chaotic intersection, the dynamic threshold will be larger to tolerate the high uncertainty that exists by itself. The dynamic threshold can be adjusted by the semantic labels output by the SCRM.

[0359] The data packaging and labeling engine is activated when the inconsistency score is greater than the dynamic threshold. It will backtrack and collect all the data related to this cognitive bias event, forming a complete case file. This case file contains:

[0360] Input layer: the complete SDFM raw data segment.

[0361] Twin layer: the scene snapshot and sensor attenuation parameters constructed by the DDTCM.

[0362] Cognitive layer: the semantic labels generated by the SCRM.

[0363] Prediction layer: the complete set of behavior hypotheses and probabilities generated by the MHSPM.

[0364] Decision layer: the actions selected by the OBDM and the final evaluation matrix.

[0365] Result layer: Real trajectory of all targets in the physical world.

[0366] The data packaging and labeling engine tags this case file with meta data such as: [Event ID, Timestamp, Inconsistency Score, Bias Type: Lateral Prediction Failure, Triggering Context: Rainy Intersection].

[0367] The model optimization task distribution unit consists of a sample storage manager and a model training / fine-tuning task scheduler. The sample storage manager is used to store the packaged high-value samples in the "mistake book" database on the vehicle or upload them to the cloud sample library through the T-Box. The task scheduler is used to determine how to use the sample based on its labeling information. If the bias is small and the computing resources allow, some lightweight models such as the lookup table in the sensor attenuation model can be updated slightly on the vehicle side. For most cases, the sample is uploaded to the cloud. The MLOps platform in the cloud will automatically add these high-value samples to the training set for the next round of retraining or reinforcement learning of heavy models. If it is found that the prediction under a certain context fails continuously, the system can generate a report to suggest developers to add new entities or relationship rules to the knowledge graph.

[0368] The running process of CLOM is triggered asynchronously after decision execution, and its flowchart is shown in Figure 14 , which includes:

[0369] Step 1: At the same time as the OBDM issues control instructions, the inconsistency monitoring unit of CLOM starts to cache the prediction results of MHSPM and related context.

[0370] Step 2: Continuous monitoring and data alignment: In the next few seconds, the monitoring unit continuously receives and aligns real-world observation data from SDFM.

[0371] Step 3: At the end of the monitoring time window, the monitoring unit calls its metric function library to calculate the inconsistency score between prediction and reality.

[0372] Step 4: The high-value sample screening unit compares the calculated inconsistency score with the dynamic threshold under the current context.

[0373] Step 5: In response to the inconsistency score being less than or equal to the dynamic threshold, indicating that the prediction is within an acceptable range, this cycle ends and the cache is emptied.

[0374] In response to the inconsistency score being greater than the dynamic threshold, the packaging and labeling engine is triggered to backtrack and collect full-chain data to generate a high-value sample with detailed metadata.

[0375] Step 6: The model optimization task distribution unit stores the generated high-value samples in the database or uploads them to the cloud; according to the metadata of the samples, the scheduler decides whether to trigger an online fine-tuning task or add it to the offline training pool in the cloud.

[0376] This process creates an efficient, intelligent, and self-driven evolution mechanism, ensuring that the system can learn from every accident and continuously improve its understanding and prediction of the world.

[0377] The above six modules are not a simple linear pipeline, but a tightly coupled and iterative cognitive architecture. Its core operating logic can be summarized as three nested stages: generation, deduction, and evaluation. Through this cognitive cycle, the system upgrades the vehicle from a passive reactor to an active thinker, providing the autonomous driving system with quantifiable, real-time state-based decision-making, enhancing the system's explainability and credibility.

[0378] Finally, it should be noted that the above examples are used to illustrate the technical solutions of the present application, but are not limited thereto; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions described in the foregoing examples can still be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application, and they should be covered within the scope of the claims and the specification of the present application.

Claims

1. An automatic driving context awareness and decision making method, characterized in that, The method comprises the following steps: Time synchronization, spatial registration and semantic fusion are performed on multi-source heterogeneous data from vehicle-mounted sensors to generate standardized scene data; Based on the standardized scene data and environmental prior constraints, an adaptive digital twin environment is generated; Graph reasoning is performed on the standardized scene data to generate situational semantic labels; Based on the adaptive digital twin environment and situational semantic labels, a set of probabilistic behavior hypotheses and a set of candidate actions corresponding to key dynamic targets are generated; Parallel simulation deduction is performed in combination with the set of probabilistic behavior hypotheses and the set of candidate actions to generate a decision evaluation matrix; Based on the decision evaluation matrix, the optimal driving action is selected and issued in combination with user driving preferences and preset decision criteria; The decision evaluation matrix is generated, comprising: Based on the current vehicle state and task, sampling is performed in the two-dimensional space of acceleration and curvature to generate a set of candidate action sequences; According to a preset dynamic target basic behavior model library and a predefined behavior library, a set of basic behavior hypotheses is generated for key dynamic targets in the digital twin scene snapshot; The prior probability of the basic behavior hypothesis is adjusted based on the probability correction factor in the situational semantic label; The set of probabilistic behavior hypotheses is generated by merging the prior probability and the basic behavior hypothesis; The latest digital twin scene and decay parameters are loaded, and the combination of the candidate action sequence and all elements in the set of probabilistic behavior hypotheses is used to construct a simulation task tree, which is distributed to each computing core of the HPC; The computing core performs parallel simulation deduction of the future scene according to the preset time step; The computing core records key simulation data including vehicle spacing, relative speed between the vehicle and the key dynamic target, and virtual sensor perception results during the simulation process; The single-task cost in the simulation task tree is calculated based on the key simulation data; The expected total cost of each candidate action is calculated by weighting the single-task cost and the corresponding hypothesis probability in the set of probabilistic behavior hypotheses; The candidate action sequence and its corresponding expected total cost constitute the decision evaluation matrix.

2. The method of claim 1, wherein, The standardized scene data is generated, comprising: A uniform timestamp is assigned to the received multi-source heterogeneous data; The timestamp of the data frame is read, and the vehicle attitude is obtained by interpolation based on inertial measurement unit data; Through rigid body transformation, the multi-source heterogeneous data is unified to the ego vehicle coordinate system in combination with the preset sensor external parameters; Data fusion is performed on the converted multi-source heterogeneous data to generate a fusion object; The fusion object is subjected to effectiveness detection and abnormality rejection, and the cleaned fusion object is encapsulated into a data structure with uniform fields.

3. The method of claim 1, wherein, The adaptive digital twin environment is generated, comprising: Subscribing to and receiving standardized scene data published to different data topics, wherein: the vehicle state topic provides a vehicle state vector, the fusion object topic provides a dynamic target list, and the map update topic provides road and environmental information; The vehicle state, spatial position and motion state of traffic participants, and the environmental factor vector are updated according to the standardized scene data; According to the environmental factor vector, the performance decay coefficient set of the environmental factor vector at the current time is calculated using an adaptive performance decay mathematical model; Generate a digital twin scene snapshot based on the vehicle state, spatial position, and motion state, and generate an adaptive digital twin environment in combination with the performance attenuation coefficient set.

4. The method of claim 3, wherein, The performance attenuation mathematical model comprises: A LiDAR attenuation model adapted to rainfall: wherein, represents the intensity of the LiDAR signal after a propagation distance d, represents the initial signal intensity, represents the extinction coefficient; A camera attenuation model adapted to light and sensor working state: wherein K represents a camera performance decay coefficient, represents an illumination mapping function describing the relationship between the illumination intensity and the image signal-to-noise ratio, represents a weight coefficient of the illumination mapping function, represents a lens dirt confidence output by a visual algorithm for detecting camera lens dirt, represents a weight coefficient of the lens dirt confidence.

5. The method of claim 1, wherein, Perform graph reasoning on the standardized scene data to generate scenario semantic labels, comprising: Map the standardized scene data to a pre-constructed vehicle-mounted knowledge graph entity; Determine a risk path from the matching entity to the risk entity by performing graph traversal and rule matching on the vehicle-mounted knowledge graph; Based on the risk path, fuse the results pointing to the same risk entity based on confidence, and generate a probability correction factor corresponding to the risk entity; Pack the risk path and probability correction factor corresponding to the risk entity into a scenario semantic label.

6. The method of claim 1, wherein, Calculate the single-task cost in the simulation task tree based on the key simulation data, comprising: According to the key simulation data, calculate the safety cost, efficiency cost, comfort cost, compliance cost, and social acceptance cost respectively; By weighting and summing the safety cost, efficiency cost, comfort cost, compliance cost, and social acceptance cost according to the cost function, the single-task cost is obtained: where J denotes a single-task cost, a coefficient representing a cost function weight, denotes a cost component.

7. The method of claim 1, wherein, Select and issue the optimal driving action, comprising: Based on the highest risk value in the decision evaluation matrix and its corresponding scenario semantic label, combined with the user preference configuration parameter, select the decision criterion of this decision cycle; According to the decision criterion, select the corresponding optimal candidate action from the decision evaluation matrix.

8. The method of claim 6, wherein, Based on the decision evaluation matrix, combined with the user driving preference and the preset decision criterion, after selecting and issuing the optimal driving action, it further comprises: Convert the driving action into a control instruction executable by the vehicle; Based on the execution result of the control instruction, the cost function weight and the decision criterion are closed-loop optimized and adaptively adjusted.

9. The method of claim 8, wherein, The closed-loop optimization and adaptive adjustment of the cost function weight and the decision criterion comprise: The standardized scene data is used as real-world observation data, and the decision evaluation matrix, the probabilistic behavior assumption set, and the control instruction are used as prediction data; After aligning the real-world observation data and the prediction data in time and space, calculate the inconsistency score of the real-world observation data and the prediction data according to the preset metric function library; Compare the inconsistency score with the dynamic threshold value corresponding to the current scene; In response to the inconsistency score being greater than the dynamic threshold value, backtrack and collect the case files related to the current cognitive bias event; According to the metadata of the case file, fine-tune the lightweight model.

Images