A causal reasoning method and device for autonomous driving decision

Abstract

The application provides a kind of causal reasoning method and device for automatic driving decision, language description to traffic police gesture and traffic sign is identified and detected to multi-modal driving input data, join multi-modal data and align timestamp after, utilize visual language model to carry out semantic extraction and output standardization result, entity and relationship are mapped into causal scene graph, do operator causal intervention is carried out to causal model using traffic rule constraint, and candidate driving action set that meets rule constraint is generated, the optimal candidate action is screened by executing counterfactual inference calculation comprehensive action score.This application combines structured semantics output by visual language model with causal reasoning model deeply, realizes pre-risk prediction of driving action by counterfactual inference, significantly improves decision safety and traffic rule compliance rate, and at the same time, with the help of standardized entity relationship output and traceable evidence index, the explainability of decision process is ensured, and a reliable decision mechanism is provided for automatic driving system.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a causal reasoning method and apparatus for autonomous driving decision-making. Background Technology

[0002] Autonomous driving technology is a core technology direction for the integrated development of intelligent transportation and new energy vehicle industries. End-to-end autonomous driving, with its integrated perception, prediction, planning and control architecture, has greatly simplified the modular link of traditional autonomous driving technology and effectively reduced decision bias caused by the accumulation of errors between modules. It has become the mainstream technology research and development and implementation direction in the global autonomous driving field. With the rapid iteration and industrial application of multimodal large model technology, the integration of visual language large model and end-to-end autonomous driving architecture has also gradually become the core path for breakthroughs in high-level autonomous driving technology.

[0003] Currently, the mainstream autonomous driving technology routes in the industry are mainly divided into three categories. The first category is the traditional modular autonomous driving solution based on detection and rules. The second category is the end-to-end autonomous driving strategy network driven purely by vision. The third category is a technical solution that decouples visual language large-scale model semantic understanding from independent driving control strategies. Among them, the traditional modular solution relies heavily on manually written rule logic, and its scene generalization ability has inherent limitations. It cannot adapt to complex and ever-changing real-world road driving scenarios. In long-tail scenarios such as urban roads with dense human-vehicle interactions and temporary construction sections, it has a lag in response and is unable to cope with sudden traffic situations. The end-to-end autonomous driving strategy network driven purely by vision lacks language priors and general common sense reasoning ability. It can only complete pattern matching at the data level and cannot deeply understand non-standard traffic information such as traffic police gestures, temporary traffic signs, and lane text semantics in road scenarios. The model decision-making process is highly black-box, and it cannot clearly explain the decision-making basis to drivers and regulatory agencies. It is prone to misjudgment when facing complex scenarios such as target occlusion and sudden changes in lighting, which poses significant driving safety hazards. The current technical solution of decoupling the visual language model from the control strategy only uses the visual language model to generate scene description text, completely disconnecting it from the downstream driving control process. This fails to form a closed-loop decision-making system driven by causal reasoning, and the consistency between the generated explanations and the actual driving actions cannot be guaranteed. Furthermore, it cannot solve the compliance and liability issues arising from black-box decision-making. Therefore, a new solution for generating autonomous driving decisions is urgently needed. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a causal reasoning method and apparatus for autonomous driving decision-making, which solves the technical problems of insufficient semantic understanding, unexplainable black box decision-making, and poor rule compliance in existing end-to-end autonomous driving systems due to the lack of causal reasoning capabilities.

[0005] One aspect of the present invention provides a causal reasoning method for autonomous driving decision-making, the method comprising the following steps: The system acquires multimodal driving input data collected from the vehicle, including: multi-view radar and camera data sequences, vehicle CAN bus status data, V2X vehicle-to-infrastructure communication messages, map data, and driver commands in voice or text form; based on a pre-trained target detection model, it processes the image data in the multimodal driving input data to identify key frames containing traffic police or traffic signs, obtains the language description of the key frames through action recognition and optical character recognition, and performs word vector encoding to obtain the language description vector; the language description vector is then used as a supplementary semantic modality and aligned with the multimodal driving input data according to the timestamp to form enhanced multimodal driving input data; The enhanced multimodal driving input data is input into the visual language model after fine-tuning the driving scenario for semantic extraction, and the output is a standardized result containing a set of scene entities, a set of entity relationships, a scene text summary, and an evidence fragment index. The set of scene entities and the set of entity relationships are mapped to a causal scene graph with a directed causal graph structure, where nodes represent scene entities and driving actions, and edges represent the relationships between scene entities. The do-operator causal intervention is performed on the pre-trained causal model using traffic rule constraints pre-generated by a traffic rule compiler, generating a set of candidate driving actions that conform to the rule constraints for the current causal scene graph; the causal model includes a structure equation that is pre-trained based on multiple sample causal scene graphs and is dynamically updated. For each candidate driving action in the candidate driving action set, counterfactual inference is performed based on the causal model to calculate a multi-dimensional score after the candidate action is executed, and a comprehensive action score is obtained by weighted summation based on preset weights, and the optimal candidate action is selected and output; the multi-dimensional score includes collision probability, traffic rule violation probability, ride comfort, traffic efficiency, and impact on surrounding traffic participants.

[0006] In some embodiments, after filtering and outputting the optimal candidate action, the method further includes: The optimal candidate action is subjected to multidimensional hard constraint verification based on a pre-built secure shell module, including: Perform a traffic regulation compliance verification to check whether the optimal candidate action complies with the traffic rule constraints; Perform semantic consistency verification by detecting the semantic similarity between the optimal candidate action and the multimodal driving input data based on the visual language model, in order to check for prompt injection attacks; Perform a safety verification to check whether the collision probability corresponding to the optimal candidate action is lower than a preset safety threshold. Perform reliability verification to check whether the uncertainty of the optimal candidate action output by the visual language model is lower than a preset uncertainty threshold, and whether the evidence coverage rate is higher than a preset coverage rate threshold; A standardized verification report is generated based on the verification results.

[0007] In some embodiments, the method further includes: If all the multidimensional hard constraint verifications pass, the optimal candidate action is output to the end-to-end vehicle controller for execution; if any verification in the multidimensional hard constraint verification fails, a graded degradation mechanism is triggered, including: According to the first triggering condition, a first-level downgrade is executed, discarding the reasoning results of the visual language model and switching to the output control instructions of the planner based on rules and constraints; According to the second trigger condition, the second-level downgrade is executed, and an audible and visual alarm is issued to prompt personnel to take over. If the personnel do not take over within the time limit, the preset fallback operation is executed.

[0008] In some embodiments, the method further includes: The observable explanatory information for the optimal candidate action decision is output to the outside world through a preset communication link or in-vehicle human-machine interface, including reasoning keywords, key evidence, trigger cause codes and driving intention statements; the observable explanatory information is written into the in-vehicle driving recorder and diagnostic log. The reasoning keywords are used to describe the core reasons for the decision, the key evidence is used to label and explain the decision elements that generate the core reasons based on the multimodal driving input data, the triggering reason code is the event code corresponding to the core reasons, and the driving intention statement is used to record the target operation result.

[0009] In some embodiments, the method further includes: For each decision execution cycle that generates the optimal candidate action, a standard structured audit record is generated, and the audit record uses a hash chain structure to prevent tampering. The audit log includes a global identifier field, a model and version field, an input and output field, a reasoning and evidence field, and an intent statement field; the reasoning and evidence field includes: a scene text summary, a causal scene graph summary hash, a trigger reason code, an evidence fragment index, a model confidence level, and / or a secure shell verification result; The global identifier field includes a globally unique audit ID, a scenario unique ID, a transaction record number, and a millisecond-level timestamp; The model and version fields include the visual language model version, traffic regulation version, and driving strategy version; The input and output fields include an input data digest hash and an output action digest hash; The intent statement field is used to record structured driving intent statements.

[0010] In some embodiments, the training process of the object detection model includes: Obtain a first training sample set, wherein each sample in the first sample set contains sample image frames captured by an in-vehicle camera in a driving scenario, and the location and category of traffic police personnel and traffic signs are marked as labels; The first training sample set is used to train the initial model based on YOLOv8-nano, YOLOv10-nano or YOLO-Lite model base. The sample image frames are used as input and the predicted values ​​of the location and category of the traffic police body and the traffic sign are output. The loss is constructed based on the deviation between the predicted value and the label, and the initial model parameters are updated to obtain the target detection model. The language description of the key frame is obtained through motion recognition and optical character recognition processing, including: using a BlazePalm lightweight detector to locate key points of the traffic police officer's hand, and recognizing the traffic police officer's gestures based on a preset gesture rule engine to obtain the corresponding gesture language description; detecting and outputting traffic signs based on optical character recognition, and matching the traffic sign database to output the traffic sign language description.

[0011] In some embodiments, the causal model is pre-constructed using the DoWhy causal inference library based on the sample causal scenario graph, and the form of the sample causal scenario graph is similar to that of the causal scenario. Figure 1 Therefore, the structural equations of the causal model include road accessibility equations, vehicle priority equations, collision risk equations, and traffic rule compliance equations.

[0012] On the other hand, the present invention also provides a causal reasoning device for autonomous driving decision-making, comprising: a multimodal input preprocessing module, a visual language model semantic extraction module, a causal reasoning module, an action planning and evaluation module, a safe shell and degradation module, an observable evidence module, and an audit evidence storage module. The device is used to implement the steps of the above method when a computer program or instruction is executed.

[0013] On the other hand, the present invention also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0014] On the other hand, the present invention also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0015] The causal reasoning method and apparatus for autonomous driving decision-making described in this invention acquires multimodal driving input data, processes image recognition using a target detection model to identify keyframes containing traffic police or traffic signs, obtains linguistic descriptions of traffic police gestures and traffic signs through action recognition and optical character recognition, incorporates multimodal data and aligns timestamps, performs semantic extraction using a visual language model, outputs standardized results, maps the entities and relationships therein to a causal scene graph, and applies traffic rule constraints to the causal model to perform causal intervention using the do operator. For the current causal scene graph, a set of candidate driving actions conforming to the rule constraints is generated, and counterfactual inference is performed to calculate a comprehensive action score for the candidate driving actions to select the optimal candidate action. This invention deeply integrates the structured semantics output by the visual language model with the causal reasoning model, using counterfactual inference to achieve pre-emptive risk prediction of driving actions, significantly improving decision-making safety and traffic rule compliance rates. Simultaneously, by leveraging standardized entity relationship output and traceable evidence fragment indexes, it ensures the interpretability of the decision-making process and the consistency between the decision basis and driving actions, providing a reliable decision-making mechanism for autonomous driving systems that meets functional safety requirements.

[0016] Furthermore, by setting hard constraints on the safety shell and a graded downgrade fallback mechanism, the optimal candidate actions for autonomous driving decisions can be verified in multiple dimensions for compliance, safety and reliability. This effectively prevents risks caused by prompt injection attacks and low-confidence outputs from the model. When verification fails, the control strategy can be switched in a graded manner or manual takeover can be requested to achieve a safety fallback for driving, ensuring the safety, compliance and stability of driving decision outputs and meeting automotive-grade functional safety requirements.

[0017] Furthermore, by establishing a full-chain anti-tampering audit system, the compliance requirements of autonomous driving regulation are met, providing an irrefutable legal basis for determining liability in accidents.

[0018] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0019] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, are not intended to limit the scope of the invention. In the drawings: Figure 1This is a flowchart illustrating a causal reasoning method for autonomous driving decision-making according to an embodiment of the present invention.

[0021] Figure 2 This is a schematic diagram of the causal reasoning device for autonomous driving decision-making according to an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0023] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0024] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0025] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0026] Currently, none of the publicly available autonomous driving technologies can simultaneously achieve closed-loop decision-making driven by causal reasoning of large visual language models, real-time externally observable decision-making evidence, and end-to-end auditable and traceable compliance management. They cannot simultaneously address the core shortcomings of existing end-to-end autonomous driving technologies, such as insufficient semantic understanding and common-sense reasoning capabilities, unexplainable and unprovable black-box decision-making, uncontrollable rule compliance and driving safety, and the difficulty of mass production and deployment at the automotive grade. They are also unable to meet the core requirements of functional safety, regulatory compliance, and liability determination in the process of large-scale deployment of advanced autonomous driving.

[0027] In view of this, the present invention provides a causal reasoning method for autonomous driving decision-making, such as... Figure 1 As shown, the method includes the following steps S101~S105: Step S101: Acquire multimodal driving input data collected by the vehicle, including: multi-view radar and camera data sequences, vehicle CAN bus status data, V2X vehicle-to-infrastructure communication messages, map data, and driver commands in voice or text form; process the image data in the multimodal driving input data based on a pre-trained target detection model to identify key frames containing traffic police or traffic signs, obtain language descriptions of the key frames through action recognition and optical character recognition, and perform word vector encoding to obtain language description vectors; align the language description vectors as supplementary semantic modalities with the multimodal driving input data according to timestamps to form enhanced multimodal driving input data.

[0028] Step S102: Input the enhanced multimodal driving input data into the visual language model after fine-tuning the driving scenario for semantic extraction, and output a standardized result containing a set of scene entities, a set of entity relationships, a scene text summary, and an evidence fragment index.

[0029] Step S103: Map the set of scene entities and the set of entity relationships into a causal scene graph with a directed causal graph structure, where nodes represent scene entities and driving actions, and edges represent relationships between scene entities.

[0030] Step S104: Apply the do operator causal intervention to the pre-trained causal model using traffic rule constraints pre-generated by the traffic rule compiler, and generate a set of candidate driving actions that conform to the rule constraints for the current causal scene graph; the causal model contains a structure equation that is pre-trained based on multiple sample causal scene graphs and is dynamically updated.

[0031] Step S105: Perform counterfactual inference on each candidate action in the candidate driving action set based on the causal model, calculate the multi-dimensional score after the candidate action is executed, and obtain the comprehensive action score by weighted summation based on preset weights, and filter and output the optimal candidate action; the multi-dimensional score includes collision probability, traffic rule violation probability, ride comfort, traffic efficiency, and impact on surrounding traffic participants.

[0032] In step S101, the vehicle-mounted autonomous driving system needs to simultaneously collect input data from multiple sources. Among these, multi-view camera data sequences typically include continuous frame images from multiple perspectives such as forward-looking and surround-view, used to perceive the surrounding traffic environment; radar data includes lidar point clouds or millimeter-wave radar targets, used to obtain accurate distance and speed information; vehicle CAN bus status data records the vehicle's own dynamics in real time, such as vehicle speed, throttle opening, brake pedal travel, and steering angle; V2X vehicle-to-infrastructure cooperative messages come from roadside units or other traffic participants, providing beyond-line-of-sight traffic light status, construction warnings, or cooperative traffic information; map data includes high-precision lane lines, intersection topology, and traffic sign locations; and driver commands are input through the vehicle microphone or touchscreen, expressing the driver's intentions in voice or text form.

[0033] After the aforementioned multi-source data is collected, key information needs to be extracted. For example, for image data, the system will call a pre-trained target detection model to specifically identify key frames containing traffic police or traffic signs. Once a traffic police officer is detected, the system will further analyze the specific meaning of the traffic police officer's gestures through pose estimation or action recognition models. If a traffic sign is detected, the system will extract the text content of the sign through optical character recognition technology, and transform these unstructured visual information into structured language descriptions, such as descriptions of traffic police officer stop gestures, or indicative descriptions such as "Speed ​​limit of 20 km / h ahead due to construction." Then, word embedding technology will be used to transform the information into language description vectors.

[0034] Ultimately, all modal data, including raw sensor data, processed language description vectors, and driver commands, need to be precisely aligned based on a unified millisecond-level timestamp to ensure that each data source reflects the physical world state at the same moment within the same decision-making cycle.

[0035] In some embodiments, the training process of the object detection model includes steps S1011 to S1013: Step S1011: Obtain the first training sample set. Each sample in the first sample set contains sample image frames captured by the vehicle-mounted camera in a driving scenario, and labels the location and category of the traffic police officer and traffic signs.

[0036] Step S1012: Use the first training sample set to train the initial model based on the YOLOv8-nano, YOLOv10-nano or YOLO-Lite model base. Take the sample image frames as input and output the predicted values ​​of the location and category of traffic police and traffic signs. Construct the loss based on the deviation between the predicted values ​​and the labels, and update the initial model parameters to obtain the target detection model.

[0037] Step S1013: Obtain the language description of the key frame through motion recognition and optical character recognition processing, including: using BlazePalm lightweight detector to locate key points of the traffic police officer's hand, and recognizing the traffic police officer's gestures based on a preset gesture rule engine to obtain the corresponding gesture language description; detecting and outputting traffic signs based on optical character recognition, and matching the traffic sign database to output the traffic sign language description.

[0038] Steps S1011 to S1013 transform the traffic police and traffic signs in the original images into structured language descriptions that can be directly used by the subsequent causal inference module. Step S1011 first constructs a first training sample set by accurately labeling the location and category of traffic police personnel and traffic signs in driving scene images, providing a high-quality label foundation for supervised learning and ensuring that the subsequent model can accurately learn the visual features of these key targets. Step S1012 uses lightweight model bases such as YOLOv8-nano, YOLOv10-nano, or YOLO-Lite for training. These models are designed to balance detection accuracy and inference speed. They extract image features and predict target location and category through convolutional neural networks, construct a loss function based on the deviation between the predicted value and the label, and iteratively update the parameters to finally obtain a target detection model that can run in real time on the vehicle and has high recall and accuracy for traffic police and traffic signs, providing reliable target regions for subsequent semantic extraction. Step S1013 further refines the detected key areas. For traffic police gestures, a BlazePalm lightweight detector is used to locate key hand points, and matching and recognition are performed based on a preset gesture rule engine. The gesture posture is converted into a standardized linguistic description, such as "stop" or "turn left." The principle is to compare the geometric structure and temporal change features of the hand with predefined rules, avoiding the tedious work of annotating a large number of gesture samples. For traffic signs, optical character recognition technology is used to extract the text information in the signs and match it with a traffic sign database to obtain a linguistic description containing specific meanings and rules. Through the combination of lightweight detection and rule post-processing, accurate semantic understanding of traffic police gestures and traffic signs is achieved under limited vehicle-side computing power. Visual information is transformed into structured symbolic input, which not only reduces the processing burden of subsequent large models but also provides a traceable and verifiable visual evidence foundation for the entire causal reasoning chain, significantly improving the system's ability to understand long-tail scenarios and the reliability of decision-making.

[0039] In step S102, the aligned multimodal data is input into a large-scale visual language model that has been fine-tuned specifically for driving scenarios. This model, with its powerful generalization ability obtained by pre-training on massive amounts of image and text data, can jointly understand complex visual scenes and auxiliary information and output structured semantic results.

[0040] The scene entity set is a list detailing all key objects in the current scene, such as vehicle A, pedestrian B, traffic police officer C, and construction cone D, along with attributes for each entity such as color, type, motion state, and 2D or 3D location. The entity relationship set describes the logical and spatiotemporal connections between these entities; for example, vehicle A is in front of traffic police officer C, pedestrian B partially obstructs vehicle A's view, and traffic police officer C's gesture grants vehicle A priority right-of-way. The scene text summary is a human-readable natural language description, such as a traffic police officer directing traffic at an intersection ahead, whose stop gesture requires vehicles in this lane to stop and wait. The evidence fragment index is the original data source information corresponding to the above outputs, recording which frame and region of the image the key evidence supporting the current semantic conclusion comes from; for example, the traffic police officer's gesture recognition result corresponds to the coordinate frame region of the third frame image. This indexing mechanism provides direct data anchors for subsequent evidence presentation and auditing.

[0041] In step S103, each object in the entity set is defined as a node in the causal graph, and each relationship in the entity relationship set is defined as a directed edge connecting the nodes, thereby constructing a causal scenario graph for the current scenario. For example, a traffic police node has a directed edge pointing to the vehicle node, indicating that the traffic police's actions affect the vehicle's behavior; a preceding vehicle node has a directed edge pointing to the vehicle node, indicating that the preceding vehicle's state affects the vehicle's decision; the vehicle node itself is connected to driving action nodes. Based on this, a structural causal model is further instantiated based on this causal scenario graph, that is, using pre-trained structural equations to quantify the causal relationships in the graph. These structural equations are mathematical functions learned in advance through a large amount of driving data. For example, the road drivability equation can calculate the drivability probability of the current lane based on lane line conditions, obstacle positions, and traffic signals, and the collision risk equation can calculate the rear-end collision probability based on the relative speed and distance between the vehicle and the preceding vehicle and the road friction coefficient. Through this step, the abstract causal relationship is given concrete quantitative calculation capabilities, enabling the model not only to know that the deceleration of the vehicle in front will cause the vehicle to decelerate, but also to accurately calculate how much deceleration the vehicle needs to maintain safety at the current distance.

[0042] In step S104, the traffic regulations written in natural language are first converted into machine-executable finite state machines or constraint functions using a traffic rule compiler. For example, the rule that all vehicles facing the traffic light must stop when the red light is on can be compiled into a constraint: when the traffic light node is red and the vehicle is in the corresponding lane, the candidate action set must not include the action of passing through the intersection. Based on these compiled rule constraints, the do operator intervention is performed on the currently instantiated causal model. The do operator operation refers to forcibly changing the value of a variable in the model and cutting off its influence from other variables. For example, the system can force the vehicle's action to accelerate and then observe how other nodes in the model, such as collision risk and compliance status, change under this intervention. By traversing various possible action interventions, the system filters out all actions that still satisfy the rule constraints after intervention, forming a safe candidate action set. For example, in the current scenario, the causal model shows that the three actions of slowing down to yield, passing at a constant speed, and accelerating to overtake do not trigger any rule violations, and they proceed to the next evaluation step.

[0043] In step S105, for each action in the candidate action set, counterfactual inference is performed on the causal model. This is a hypothetical deduction that answers the question of what the state would be if the action were performed. For example, for the candidate action of slowing down and yielding, the future state after performing the action is deduced based on the current observed state and the structural equations in the causal model, and scores are calculated for multiple dimensions such as collision probability, traffic rule violation probability, ride comfort, traffic efficiency, and impact on surrounding traffic participants in that state.

[0044] The collision probability is given by the collision risk equation, the violation probability is evaluated by the traffic rule constraint function, comfort is measured by acceleration, and traffic efficiency is estimated by the time required to cross the intersection. The scores for each dimension are weighted and summed according to preset weights to obtain a comprehensive action score. For example, under a safety-first strategy, the weights for collision probability and violation probability are set very high. After scoring all candidate actions, the action with the highest comprehensive score is selected as the optimal candidate action and output to the downstream execution agency. This mechanism ensures that the final output driving action is not only compliant and safe, but also achieves an optimal balance between comfort and efficiency.

[0045] In some embodiments, after filtering and outputting the optimal candidate action, the method further performs multidimensional hard constraint verification on the optimal candidate action based on a pre-built safe shell module, including steps S201~S205: Step S201: Perform traffic regulation compliance verification to check whether the optimal candidate action complies with traffic rule constraints.

[0046] Step S202: Perform semantic consistency verification by detecting the semantic similarity between the optimal candidate action and the multimodal driving input data based on the visual language model, in order to check for injection attacks.

[0047] Step S203: Perform a safety verification to check whether the collision probability corresponding to the optimal candidate action is lower than a preset safety threshold.

[0048] Step S204: Perform reliability verification, check whether the uncertainty of the optimal candidate action output by the visual language model is lower than the preset uncertainty threshold, and whether the evidence coverage rate is higher than the preset coverage rate threshold.

[0049] Step S205: Generate a standardized verification report based on the verification results.

[0050] In step S201, the traffic rule compiler has pre-converted the natural language forms of road traffic safety laws, local traffic control regulations, and enterprise specifications for specific operational scenarios into machine-executable finite state machines or hard constraint functions. When a candidate action is sent to the safe shell module, the verification engine uses that action as input, substituting it into the current traffic rule constraint function for calculation. For example, if the current traffic light is red and the vehicle is before the stop line, a straight-line action will be directly judged as a violation by the constraint function; if the traffic police hand signal is "stop" and the candidate action is "slow down and stop," it will be judged as compliant and allowed to pass. This verification process has absolute priority; any candidate action that fails the traffic rule compliance verification will be directly intercepted and cannot enter subsequent verification stages. This mechanism ensures that vehicles always meet the basic requirements of road traffic regulations, fundamentally preventing violations caused by model illusions or decision-making errors.

[0051] In step S202, the high-level semantic outputs generated by the visual language model based on the current multimodal input data, such as scene text summaries and entity relationship sets, are used to calculate semantic similarity with the optimal candidate action to be executed. For example, if the model outputs a scene description of "a pedestrian is crossing the zebra crossing ahead, please stop and yield," while the candidate action is "accelerate through the intersection," the similarity between the two in the semantic space will be extremely low, triggering a consistency alarm. To achieve this comparison, the system typically uses a pre-trained semantic encoder to map the text description and action token to the same vector space, quantifying the degree of consistency by calculating cosine similarity or Euclidean distance. If the similarity is lower than a preset threshold, it is determined that there may be a hint injection attack, i.e., external malicious input induces the model to output a dangerous action that contradicts the scene understanding. At this time, the secure shell module will immediately trigger a degradation mechanism, discarding the current VLM inference result and switching to an alternative planning scheme. This verification step effectively prevents adversarial attacks against large models and ensures logical alignment between semantic understanding and underlying execution.

[0052] In step S203, this verification step directly calls the collision probability value calculated by the causal model in step S105 and compares this value with a preset safety threshold. For example, if the set safety threshold is that the collision probability must not exceed one percent, and the collision probability corresponding to the current optimal candidate action is inferred to be 0.5 percent by counterfactual reasoning, then the verification passes; otherwise, if the collision probability reaches two percent, then the action is determined to lack physical safety.

[0053] The safety thresholds here are typically set according to the requirements of the ISO 26262 functional safety standard and are dynamically adjusted based on different vehicle speeds and scenario types. For example, in high-speed driving scenarios, the safety thresholds will be set more strictly. Safety verification is a crucial link in the entire verification chain that is directly related to the safety of life and property. It transforms abstract causal reasoning results into actionable safety decision-making criteria.

[0054] Step S204 includes two dimensions: uncertainty verification and evidence coverage verification. Uncertainty verification aims to quantify the confidence level of the visual language model in the current output result. Its calculation can be achieved using various techniques, such as calculating the entropy value based on the probability distribution of the model's output. If the probability distribution of each candidate word is evenly dispersed when the model generates scene text summaries, the entropy value is high, indicating that the model lacks confidence in its own output; or the Monte Carlo dropout method can be used to perform multiple forward inferences and estimate the model's uncertainty by statistically analyzing the variance of the output results.

[0055] Evidence coverage verification focuses on evaluating whether the model output is supported by sufficient visual evidence. The specific calculation method is to compare the evidence fragment index output by VLM with the preset evidence requirement list. For example, for a decision that requires yielding to pedestrians, the preset evidence requirement list requires that pedestrian entities, zebra crossing areas, and the relative positional relationship between the vehicle and the pedestrian must be detected. If the evidence fragments output by the current model only cover pedestrian entities and lack zebra crossing detection, then the evidence coverage is insufficient.

[0056] The reliability verification is passed only when the uncertainty is below the preset threshold and the evidence coverage is above the preset threshold. This dual verification mechanism effectively prevents the model from blindly outputting decisions when there is perceptual ambiguity or insufficient evidence.

[0057] Step S205 is responsible for standardizing and integrating the results of all the aforementioned verification steps to generate a structured verification report. In practice, this report is not a simple pass or fail flag, but a standardized data structure containing multi-dimensional verification fields, which records in detail the execution result, verification threshold, actual measurement value, and verification timestamp for each verification item.

[0058] In some embodiments, the method further includes, if all multidimensional hard constraint verifications pass, outputting the optimal candidate action to the end-to-end vehicle controller for execution; if any verification in the multidimensional hard constraint verification fails, triggering a graded degradation mechanism, including steps S301 and S302: Step S301: According to the first triggering condition, perform a first-level downgrade, discard the reasoning results of the visual language model, and switch to the output control instructions of the planner based on rules and constraints.

[0059] Step S302: According to the second triggering condition, perform the second-level downgrade and issue an audible and visual alarm to prompt personnel to take over. If the personnel do not take over within the time limit, perform the preset fallback operation.

[0060] In step S301, the triggering conditions for Level 1 degradation can be set as follows: semantic consistency verification fails, uncertainty in reliability verification is too high or evidence coverage is insufficient, and the collision probability in safety verification slightly exceeds the standard but has not yet reached the level of immediate danger. When these conditions are met, it is determined that the output of the current visual language model is no longer reliable, but the overall driving environment has not deteriorated to the point where manual intervention is necessary. Therefore, the candidate actions generated by the VLM are immediately discarded, and the pre-configured planner is activated. This planner is based on the traditional modular architecture of autonomous driving and relies on high-precision maps, LiDAR, and millimeter-wave radar data. It generates safe and feasible control commands through finite state machines and cost function search. For example, when the VLM outputs an acceleration command that contradicts the scene understanding due to a cue injection attack, the Level 1 degradation mechanism will decisively switch to the rule planner. The latter calculates a safe following deceleration command based on the distance to the vehicle in front and the current vehicle speed to ensure that the vehicle continues to drive smoothly. This degradation strategy preserves the vehicle's autonomous driving capability while ensuring safety, avoids frequent manual intervention due to model failure, and reflects the fault tolerance and degradation design concept in functional safety design.

[0061] The secondary downgrade mechanism implemented in step S302 is activated under more severe failure modes. Its triggering conditions can be set as follows: failure to pass traffic compliance verification (i.e., the vehicle is about to commit a clear violation), collision probability exceeding a preset high-risk threshold during safety verification, or the planner still failing to generate valid instructions after primary downgrade. Once these conditions are met, it is determined that driving safety cannot be guaranteed through algorithmic switching, and human driver judgment and intervention must be introduced. In practice, the vehicle will immediately issue audible and visual alarms, including an in-vehicle buzzer, a voice prompt requesting immediate steering wheel control, and a flashing red warning light on the dashboard. Simultaneously, the reason code for the takeover request will be displayed on the in-vehicle HMI screen, such as excessively high traffic violation risk or collision risk. The system initiates a preset takeover timer. If the driver indicates takeover control within three seconds via the steering wheel or pedals, the system exits autonomous mode and returns to human driving. If no takeover action is detected within the time limit, the driver is deemed incapacitated or unresponsive, and the vehicle automatically executes preset fallback maneuvers, typically involving activating hazard lights, gradually slowing down, and ultimately pulling over safely. Simultaneously, an emergency status signal is sent to the cloud monitoring platform via V2X or cellular network. This tiered degradation mechanism extends from the primary degradation algorithm redundancy switching to the secondary degradation human-machine collaborative fallback, constructing a functional safety protection chain covering all scenarios from minor anomalies to severe failures, ensuring the vehicle can return to a safe state under any extreme circumstances.

[0062] In some embodiments, the method further includes: outputting observable explanatory information for the optimal candidate action decision through a preset communication link or an in-vehicle human-machine interface, including inference keywords, key evidence, trigger cause codes, and driving intention statements; and writing the observable explanatory information into an in-vehicle dashcam and diagnostic log. Specifically, inference keywords describe the core reasons for the decision, key evidence is used to label and explain the decision elements that generate the core reasons based on multimodal driving input data, trigger cause codes are the event codes corresponding to the core reasons, and driving intention statements are used to record the target operation result.

[0063] In some embodiments, the method further includes step S401: for each decision execution cycle that generates the optimal candidate action, a standard structured audit record is generated, wherein the audit record adopts a hash chain structure to prevent tampering.

[0064] Audit logs include global identifier fields, model and version fields, input / output fields, reasoning and evidence fields, and intent statement fields. The global identifier field includes a globally unique audit ID, a scenario unique ID, a log entry number, and a millisecond-level timestamp. The model and version field includes the visual language model version, traffic regulation version, and driving strategy version. The input / output fields include input data digest hashes and output action digest hashes. The reasoning and evidence fields include: scenario text digest, causal scenario graph digest hash, trigger cause code, evidence fragment index, model confidence, and / or safe shell verification results. The intent statement field records structured statements of driving intent.

[0065] In some embodiments, the causal model is pre-built using the DoWhy causal inference library based on sample causal scenario graphs. The form of the sample causal scenario graphs is similar to that of the causal scenarios. Figure 1 The structural equations of the causal model include the road accessibility equation, the vehicle priority equation, the collision risk equation, and the traffic rule compliance equation.

[0066] Specifically, the essential feature that distinguishes the DoWhy library from general machine learning libraries is that it takes causal hypotheses as explicit inputs, supports the construction of causal graphs in a graphical way, and performs identification and estimation based on the do operator. Its core design concept is to decompose the causal reasoning problem into four distinct stages: modeling, identification, estimation, and refutation. This allows developers to robustly learn the causal structural relationships and quantitative parameters between variables based on a large number of sample causal scene graphs containing various driving scenarios, using the backdoor criteria, instrumental variables, and other identification methods provided by DoWhy, as well as rich statistical testing tools. Building upon this, the causal model includes a road accessibility equation to quantify the probability of whether a lane is currently passable. This equation typically incorporates factors such as lane detection results, obstacle occupancy, and traffic signal status. The vehicle priority equation dynamically calculates the vehicle's priority relative to other traffic participants based on right-of-way rules and interactive dynamics. The collision risk equation predicts the probability of collision after performing different actions based on physical quantities such as the relative speed, distance, heading angle, and road friction coefficient between the vehicle and the target through counterfactual reasoning. The traffic rule compliance equation directly transforms the constraints output by the traffic rule compiler into quantifiable evaluation functions to assess the likelihood of candidate actions violating traffic regulations.

[0067] On the other hand, the present invention also provides a causal reasoning device for autonomous driving decision-making, such as... Figure 2 As shown, the device includes: a multimodal input preprocessing module, a visual language model semantic extraction module, a causal reasoning module, an action planning and evaluation module, a safe shell and degradation module, an observable evidence module, and an audit evidence storage module. The device is used to implement the steps of the above method when a computer program or instruction is executed.

[0068] On the other hand, the present invention also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0069] On the other hand, the present invention also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0070] The present invention will now be described in conjunction with specific embodiments: Example 1 This embodiment provides an end-to-end autonomous driving scene understanding and decision-making method based on VLM causal reasoning. VLM transforms visual scenes into standardized textual semantics and structured scene graphs, solving the problem of stable mapping from visual information to linguistic semantics and addressing the common-sense reasoning shortcomings of E2E models. A Structural Causal Model (SCM) is constructed based on the scene graph. Through causal intervention and counterfactual evaluation, it enables pre-judgment of the consequences of implemented decisions, fundamentally resolving safety defects caused by misjudgments of correlations. Through hard constraints of a secure shell and a tiered degradation fallback mechanism, it addresses the defects of uncontrollable output and susceptibility to hint-based injection attacks in large models, meeting the ISO 26262 automotive-grade functional safety requirements. By displaying key decision-making evidence in real-time through an onboard HMI, coupled with tamper-proof hash chain audit records bound to a global audit ID, it achieves end-to-end observability, evidence-gathering, and traceability of the decision-making process, completely resolving the compliance and evidence deficiencies of black-box decision-making.

[0071] The complete method flow of this invention includes the following 8 core steps S1 to S8: S1: Multimodal input acquisition and preprocessing The vehicle-mounted autonomous driving system collects multimodal driving input data, including: multi-frame sequences of forward / surround view cameras; vehicle CAN bus status data, such as vehicle speed, throttle, brake, and steering angle; V2X vehicle-to-infrastructure communication messages received by the vehicle; high-precision maps / lane line information; and driver voice / text commands.

[0072] Standardized preprocessing of input data: OCR text recognition is performed on road signs, lane text, and temporary signs; key point detection is performed on traffic police gestures and traffic signs and standardized language description tokens are generated; millisecond-level timestamp alignment is performed on all data with an alignment error ≤10ms; and a unified multimodal observation sequence is generated.

[0073] For example, standardized preprocessing is performed on the input data: PaddleOCR text recognition is performed on road signs, lane text, and temporary construction signs; MediaPipe key point detection is performed on traffic police gestures to extract the coordinates of key points on the arm and palm and generate standardized language description tokens; millisecond-level timestamp alignment is performed on all data with an alignment error of no more than 10ms to generate a unified multimodal observation sequence.

[0074] S2: Visual Tokenization and VLM Semantic Extraction The aligned multi-frame visual input is encoded into a visual token sequence, which is then input into a pre-trained visual-language large model (VLM) fine-tuned for driving scenarios. The VLM performs joint semantic extraction on the visual token sequence and multimodal auxiliary information, and outputs the standardized results as follows: Scene entity set E: includes core entities such as vehicles, pedestrians, traffic police, non-motorized vehicles, construction cones, traffic signs, lane lines, and drivable areas, as well as the attributes, location, and motion state of each entity.

[0075] The entity relationship set R includes spatiotemporal and causal relationships between entities such as yielding, blocking, priority, following, intersection, and prohibition.

[0076] Scene text summary: A standardized scene description that is readable by humans.

[0077] Evidence fragment index: The keyframe number and image region bounding box corresponding to the semantic extraction result, used for subsequent evidence presentation.

[0078] For example, a Qwen-VL multimodal large model fine-tuned for driving scenarios is used. The aligned four consecutive frames of visual input are encoded into a 256-dimensional visual token sequence, which is then concatenated with multimodal auxiliary information and input into the model. The model fine-tuning adopts the LoRA low-rank adaptation method and is performed based on the public driving datasets BDD100K and nuScenes. This can be directly reproduced by those skilled in the art.

[0079] The model outputs standardized semantic extraction results, such as entity set E, which includes entities such as the vehicle itself, the vehicle in front, the pedestrian on the left, the traffic police at the intersection, the construction cones, the stop line, and the left-turn lane, as well as the position, motion state, and attribute information of each entity.

[0080] The set of relations R includes the spatiotemporal and causal relationships between entities such as traffic police directing their own vehicle to stop, construction cones closing the straight lane, and vehicles in front remaining stationary.

[0081] Scene text summary: At the intersection, a traffic police officer is issuing a stop signal. The straight lane is closed by construction cones, and the vehicles ahead are stationary. This vehicle needs to stop and wait.

[0082] Evidence fragment index: Frame 2: Traffic police hand gesture detection area of ​​the forward-looking camera; Frame 3: Construction cone detection area.

[0083] S3: Causal Scenario Graph Construction and Structural Causal Model (SCM) Generation The entity set E and relation set R output by VLM are mapped to a directed causal graph structure to construct a causal scene graph. Nodes represent scene entities and driving actions, and edges represent causal relationships between entities, such as a traffic police officer's stop gesture causing the vehicle to stop, and the vehicle in front slowing down causing the vehicle to slow down.

[0084] Based on the causal scenario graph, a structural causal model (SCM) is constructed, and structural equations that can be updated in real time are maintained, including: road accessibility equation, vehicle priority equation, collision risk equation, and traffic rule compliance equation, providing a basis for subsequent causal intervention and counterfactual assessment.

[0085] S4: Rule compilation and causal intervention to generate candidate action sets The traffic rule compiler transforms natural language texts such as national / local traffic regulations, temporary traffic control requirements, and vehicle operation strategies into executable finite state machines (FSMs) or hard constraint functions, which then serve as traffic rule constraints.

[0086] Based on the compiled traffic rule constraints, the do operator causal intervention is performed on the SCM to generate a set of candidate driving actions that conform to the rule constraints. Each candidate action corresponds to a standardized action token, including driving behaviors such as acceleration, deceleration, steering, lane changing, yielding, and stopping.

[0087] S5: Counterfactual Evaluation of Candidate Actions and Selection of Optimal Action For each candidate action Based on SCM, counterfactual inference is performed to calculate multi-dimensional scores after the action is performed, including: collision probability, traffic rule violation probability, ride comfort, traffic efficiency, and impact on surrounding traffic participants. Based on a preset weight system, the multi-dimensional scores of each candidate action are weighted and summed to generate a comprehensive action score. The optimal action with the highest comprehensive score is selected as the candidate action to be output.

[0088] For example, the "Road Traffic Safety Law of the People's Republic of China" and temporary traffic control requirements at intersections are transformed into an executable finite state machine (FSM) and constraint functions using a traffic rule compiler. Core constraints include prioritizing police hand signals when given instructions, ignoring traffic lights, prohibiting passage through construction zones, and prohibiting crossing the stop line at intersections. Based on the compiled rule constraints, the do operator is used to intervene causally in the SCM, generating a set of candidate actions that conform to the rules: {decelerate and stop, change lanes to the left and detour}. Counterfactual evaluation is performed on each candidate action, and multi-dimensional scores are calculated. The results are shown in Table 1 below. Table 1 Based on the comprehensive score, deceleration and stopping were selected as the optimal candidate action.

[0089] S6: Containment Validation and Graded Degradation Fallback Mechanism The containment module performs multi-dimensional hard constraint verification on the candidate actions to be output. The verification items include: 1. Traffic regulation compliance verification: Whether the action complies with traffic rule constraint functions.

[0090] 2. Semantic Consistency Verification: The semantic similarity between the semantic reasoning results output by the computer vision-language large model and the optimal candidate action is used to detect hint injection attacks and ensure that the action and the reasoning results are consistent.

[0091] 3. Safety verification: Whether the collision risk after the action is executed is lower than the preset safety threshold.

[0092] 4. Reliability verification: Whether the uncertainty of the VLM output is lower than the preset threshold and whether the evidence coverage is higher than the preset threshold.

[0093] If all verifications pass, the optimal action token is output to the end-to-end vehicle controller. If any verification fails, a tiered degradation mechanism is immediately triggered: Level 1 degradation discards the VLM inference results and switches to the traditional rule- and constraint-based planner to output control commands. Level 2 degradation triggers an audible and visual alarm, requesting human driver intervention. If intervention is not initiated within the specified time, an emergency pullover is executed as a fallback strategy. The triggering cause, time, and execution result of all degradation events are recorded in an interpretable chain of evidence and audit logs.

[0094] For example, four hard constraint verifications are performed on the deceleration and stopping action using the containment module: 1. Compliance verification: The action complies with traffic rules and passes verification.

[0095] 2. Semantic consistency verification: The semantic similarity between the semantic reasoning result and the action is 0.92, which is higher than the preset threshold of 0.7. There is no indication of injection risk, and the verification is successful.

[0096] 3. Safety verification: The collision probability is 0.1%, which is lower than the preset threshold of 1%, and the verification is passed.

[0097] 4. Reliability verification: The model uncertainty is 0.12, which is lower than the preset threshold of 0.35, and the evidence coverage is 0.95, which is higher than the preset threshold of 0.6. The verification is successful.

[0098] All verifications passed, and the deceleration and stopping action was encoded as an 8-bit discrete action token, which was output to the end-to-end vehicle controller to control the vehicle to perform smooth deceleration and stopping.

[0099] If any verification fails, a first-level degradation is immediately triggered, switching to the traditional rule- and constraint-based planner output control commands. If three consecutive verifications fail, a second-level degradation is triggered, requesting driver takeover. If takeover is not initiated within 10 seconds, an emergency pullover is executed. All degradation events are recorded in the audit log.

[0100] S7: Real-time output of externally observable interpretable information In the vehicle's human-machine interface (HMI), the reasoning keywords, key evidence, trigger code, and driving intention statement for this decision are displayed in real time in the form of text bubbles, prompts, or HUD projections.

[0101] Inference keywords could include a traffic police officer's hand gesture directing a vehicle to stop, a vehicle cutting in from the left, and the need to slow down and yield. Key evidence could include the traffic police officer's hand gesture detection result from the third frame of the forward-facing camera and the identification result of a construction area 50 meters ahead. The trigger cause code is a standardized numerical code that corresponds to the core triggering event of the decision, facilitating rapid identification and evidence collection.

[0102] The above-mentioned content is simultaneously written to the vehicle's dashcam and diagnostic logs, enabling external evidence to be presented during the decision-making process.

[0103] For example, the text bubble is displayed in real time on the vehicle's central control screen (HMI), as follows: Reason for decision: Traffic police at the intersection ahead directed people to stop, and the straight lane was closed for construction.

[0104] Key evidence: The forward-facing camera detected a traffic police officer's stop gesture and construction cones.

[0105] Trigger code: TR-001, traffic police command takes priority.

[0106] The displayed content is simultaneously written to the vehicle dashcam and diagnostic interface logs, and evidence can be obtained directly through screen recording and diagnostic interface packet capture.

[0107] S8: Generation and storage of end-to-end tamper-proof audit records For each decision execution cycle (100ms / frame), generate a standardized structured audit log. The audit log must contain at least the following fields: Global identifier fields: X-Audit-Trace-ID (globally unique audit ID), scene_id (scene unique ID), record_id (record serial number), timestamp (millisecond-level timestamp); Model and version fields: vlm_version (VLM model version), rule_version (traffic rule version), strategy_version (driving strategy version); Input and output fields: input_hash (input data digest hash), output_hash (output action digest hash), action_token (action token digest); Inference and Evidence Fields: scene_caption (scene text summary), causal_graph_hash (causal scene graph summary hash), reason_code (trigger reason code), evidence_snippet_id (evidence snippet index), confidence (model confidence), safety_gate (safety gate verification result); Intent Statement field: Intent_Statement (structured human-readable driving intent statement).

[0108] Audit records use a hash chain structure to prevent tampering. The hash chain generation formula is as follows: .

[0109] in, The hash value of the current record. The hash value recorded in the previous period. This refers to all fields of the audit records for the current period. It uses the national cryptographic hash algorithm. Each audit record is signed using the national cryptographic SM2 algorithm and simultaneously stored in the vehicle's Trusted Execution Environment (TEE), the vehicle's black box, and the cloud platform to ensure that the records are tamper-proof and non-repudiable, supporting post-incident review, liability evidence collection, and independent third-party verification.

[0110] For example, for this decision-making cycle, a structured audit record is generated and bound to a globally unique X-Audit-Trace-ID: AUD-20240520-001256. The core fields include: scene_id: SCENE-20240520-0897; record_id: REC-001256; timestamp: 2024-05-20 14:32:15.125.

[0111] vlm_version: Qwen-VL-Drive-v1.0; rule_version: 2024 version of traffic regulations V2.1.

[0112] input_hash: 0x7a8f9d2c4e6b8a0d; output_hash: 0x2b3c4e5f6d7a8b9c; action_token: 0x00110101.

[0113] scene_caption: Traffic police at the intersection are issuing a stop signal; the straight lane is closed for construction. causal_graph_hash: 0x9d8e7f6a5b4c3d2e.

[0114] reason_code: TR-001; confidence: 0.95; safety_gate: PASS.

[0115] Intent_Statement: This vehicle was stopped at an intersection by traffic police, and this action was taken to slow down and stop in order to ensure compliance with traffic rules and driving safety.

[0116] Audit records are stored using the national cryptographic standard SM3 hash chain structure. It is signed using the SM2 algorithm and synchronously stored in the vehicle-side TEE, the vehicle-mounted black box, and the cloud platform to ensure that it is tamper-proof and non-repudiable, and supports post-accident review, third-party verification, and legal evidence.

[0117] To verify the technical effectiveness of this embodiment, the official validation set of the publicly available driving dataset nuScenes and the BDD100K traffic rule compliance subset were used for testing. The test scenarios covered four core long-tail scenarios: traffic police command, road construction detours, temporary stops to avoid obstacles, and unprotected left turns. Each scenario had 1000 test cases. The evaluation metrics included rule compliance rate, risk event rate, and interpretation-action consistency. The test results are shown in Table 2 below. Table 2 Experimental conclusion: This embodiment improves traffic rule compliance rate and interpretation-action consistency by using causal reasoning and consistency gating, reduces the incidence of risk events, and solves the core defects of existing solutions.

[0118] Furthermore, in this embodiment, the VLM model can be replaced by a multimodal Transformer, a video language model, or a graph neural network model. As long as it can output standardized entity, relation, and semantic information, it can achieve the core objective of this invention. Causal relationship representation can be equivalently replaced by event chains, key evidence lists, causal scoring vectors, or Bayesian networks. As long as causal intervention and counterfactual assessment can be achieved, they can replace causal scene graphs and SCM. Externally observable evidence presentation methods can be replaced by voice broadcasts, light prompts, in-vehicle screen pop-ups, or HUD projections. Any form of decision evidence display that is externally observable and recordable can replace HMI text bubbles / prompt boxes. Tamper-proof evidence storage methods can be replaced by trusted timestamps, WORM read-only storage, or blockchain evidence storage. As long as audit records are tamper-proof and non-repudiable, they can replace hash chains and national cryptographic signature schemes. Vehicle-side deployment can be replaced by an edge-cloud collaborative inference architecture, where the low-latency control module executes on the vehicle side, and the high-semantic VLM inference executes on edge nodes / the cloud, meeting the real-time requirements of the vehicle side.

[0119] Example 1 This embodiment provides an autonomous driving device that implements the method described in Embodiment 1. The system architecture corresponds to the appendix to the specification. Figure 2 Deployed on the vehicle side, including: Multimodal input preprocessing module: used to perform step S1 as described in Example 1; VLM semantic extraction module: used to perform step S2 as described in Example 1; Causal reasoning module: used to execute step S3 as described in Example 1; Motion planning and evaluation module: used to execute step S4 as described in Example 1; Containment and Degradation Module: Used to perform step S5 as described in Example 1; Observable evidence module: used to execute step S6 as described in embodiment 1, and control the vehicle-mounted HMI8 to display observable explanatory information; Audit evidence storage module: used to perform step S7 as described in embodiment 1, and store audit records to trusted storage unit 10.

[0120] Each module is connected via an in-vehicle Ethernet communication system, meeting the automotive-grade functional safety ASIL-D level requirements, and collaboratively implementing all the method steps described in Example 1.

[0121] The core advantages of Examples 1 and 2 are: 1. Causal reasoning drives decision-making, enhancing safety and robustness. This embodiment uses VLM semantic extraction as a foundation to construct a structural causal model (SCM). Through causal intervention and counterfactual assessment, it achieves pre-emptive risk prediction of driving actions. Unlike existing solutions that only learn data correlations, this embodiment can effectively distinguish between causality and correlation, avoid misjudgments in occluded and long-tail scenarios, increase traffic rule compliance rate from 86.5% to 92.7%, and reduce risk event rate from 3.4% to 2.2%, thereby improving robustness and driving safety in complex interaction scenarios.

[0122] 2. Deep integration of decision-making and explanation enables external observability and verifiability. This embodiment binds the VLM causal reasoning results with the entire driving decision-making process to ensure consistency between explanation and action. At the same time, it displays key evidence and cause codes of the decision in real time through the vehicle-mounted HMI, forming externally observable explanatory information. This is different from the existing solution's mode of decoupling explanation and control and only storing logs internally. It increases the consistency between explanation and action from 54% to 81%, solving the defects of autonomous driving black box decision-making that cannot provide evidence or determine responsibility.

[0123] 3. A full-chain anti-tampering audit system to meet regulatory compliance requirements. This embodiment constructs standardized audit records bound to a globally unique audit ID, and achieves tamper-proofing through national cryptographic hash chains and signatures, covering the entire chain from input to reasoning to decision-making to output. Unlike existing solutions that lack unified audit standards and are prone to log tampering, this embodiment meets the compliance requirements of autonomous driving regulation and provides an irrefutable legal basis for determining liability in accidents.

[0124] 4. Automotive-grade safety design, supporting mass production implementation. This embodiment addresses the shortcomings of uncontrollable large model output and latency not meeting automotive-grade requirements through hard constraints on the containment structure, a graded degradation fallback mechanism, and a layered inference architecture. It is compatible with the ISO 26262 functional safety ASIL-D level requirements and differs from existing solutions that only focus on algorithm performance and ignore automotive-grade implementation. It has the capability for pre-installed mass production.

[0125] In summary, the causal reasoning method and apparatus for autonomous driving decision-making described in this invention acquires multimodal driving input data, processes image recognition using a target detection model to identify keyframes containing traffic police or traffic signs, obtains linguistic descriptions of traffic police gestures and traffic signs through action recognition and optical character recognition, incorporates multimodal data and aligns timestamps, performs semantic extraction using a visual language model, outputs standardized results, maps the entities and relationships therein to a causal scene graph, and applies traffic rule constraints to the causal model to perform causal intervention using the do operator. For the current causal scene graph, a set of candidate driving actions conforming to the rule constraints is generated, and counterfactual inference is performed to calculate the comprehensive action score of the candidate driving actions to select the optimal candidate action. This invention deeply integrates the structured semantics output by the visual language model with the causal reasoning model, using counterfactual inference to achieve pre-emptive risk prediction of driving actions, significantly improving decision-making safety and traffic rule compliance rate. Simultaneously, by leveraging standardized entity relationship output and traceable evidence fragment indexes, it ensures the interpretability of the decision-making process and the consistency between the decision basis and driving actions, providing a reliable decision-making mechanism for autonomous driving systems that meets functional safety requirements.

[0126] Furthermore, by setting hard constraints on the safety shell and a graded downgrade fallback mechanism, the optimal candidate actions for autonomous driving decisions can be verified in multiple dimensions for compliance, safety and reliability. This effectively prevents risks caused by prompt injection attacks and low-confidence outputs from the model. When verification fails, the control strategy can be switched in a graded manner or manual takeover can be requested to achieve a safety fallback for driving, ensuring the safety, compliance and stability of driving decision outputs and meeting automotive-grade functional safety requirements.

[0127] Furthermore, by establishing a full-chain anti-tampering audit system, the compliance requirements of autonomous driving regulation are met, providing an irrefutable legal basis for determining liability in accidents.

[0128] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0129] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0130] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0131] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A causal reasoning method for autonomous driving decision-making, characterized in that, The method includes the following steps: The system acquires multimodal driving input data collected from the vehicle, including: multi-view radar and camera data sequences, vehicle CAN bus status data, V2X vehicle-to-infrastructure communication messages, map data, and driver commands in voice or text form; based on a pre-trained target detection model, it processes the image data in the multimodal driving input data to identify key frames containing traffic police or traffic signs, obtains the language description of the key frames through action recognition and optical character recognition, and performs word vector encoding to obtain the language description vector; the language description vector is then used as a supplementary semantic modality and aligned with the multimodal driving input data according to the timestamp to form enhanced multimodal driving input data; The enhanced multimodal driving input data is input into the visual language model after fine-tuning the driving scenario for semantic extraction, and the output is a standardized result containing a set of scene entities, a set of entity relationships, a scene text summary, and an evidence fragment index. The set of scene entities and the set of entity relationships are mapped to a causal scene graph with a directed causal graph structure, where nodes represent scene entities and driving actions, and edges represent the relationships between scene entities. The do-operator causal intervention is performed on the pre-trained causal model using traffic rule constraints pre-generated by a traffic rule compiler, generating a set of candidate driving actions that conform to the rule constraints for the current causal scene graph; the causal model includes a structure equation that is pre-trained based on multiple sample causal scene graphs and is dynamically updated. For each candidate driving action in the candidate driving action set, counterfactual inference is performed based on the causal model to calculate a multi-dimensional score after the candidate action is executed, and a comprehensive action score is obtained by weighted summation based on preset weights, and the optimal candidate action is selected and output; the multi-dimensional score includes collision probability, traffic rule violation probability, ride comfort, traffic efficiency, and impact on surrounding traffic participants.

2. The causal reasoning method for autonomous driving decision-making according to claim 1, characterized in that, After filtering and outputting the optimal candidate actions, multi-dimensional hard constraint verification is performed on the optimal candidate actions based on the pre-built safe shell module, including: Perform a traffic regulation compliance verification to check whether the optimal candidate action complies with the traffic rule constraints; Perform semantic consistency verification by detecting the semantic similarity between the optimal candidate action and the multimodal driving input data based on the visual language model, in order to check for prompt injection attacks; Perform a safety verification to check whether the collision probability corresponding to the optimal candidate action is lower than a preset safety threshold. Perform reliability verification to check whether the uncertainty of the optimal candidate action output by the visual language model is lower than a preset uncertainty threshold, and whether the evidence coverage rate is higher than a preset coverage rate threshold; A standardized verification report is generated based on the verification results.

3. The causal reasoning method for autonomous driving decision-making according to claim 2, characterized in that, The method further includes: If all the multidimensional hard constraint verifications pass, the optimal candidate action is output to the end-to-end vehicle controller for execution; if any verification in the multidimensional hard constraint verification fails, a graded degradation mechanism is triggered, including: According to the first triggering condition, a first-level downgrade is executed, discarding the reasoning results of the visual language model and switching to the output control instructions of the planner based on rules and constraints; According to the second trigger condition, the second-level downgrade is executed, and an audible and visual alarm is issued to prompt personnel to take over. If the personnel do not take over within the time limit, the preset fallback operation is executed.

4. The causal reasoning method for autonomous driving decision-making according to claim 1, characterized in that, The method further includes: The observable explanatory information for the optimal candidate action decision is output to the outside world through a preset communication link or in-vehicle human-machine interface, including reasoning keywords, key evidence, trigger cause codes and driving intention statements; the observable explanatory information is written into the in-vehicle driving recorder and diagnostic log. The reasoning keywords are used to describe the core reasons for the decision, the key evidence is used to label and explain the decision elements that generate the core reasons based on the multimodal driving input data, the triggering reason code is the event code corresponding to the core reasons, and the driving intention statement is used to record the target operation result.

5. The causal reasoning method for autonomous driving decision-making according to claim 2, characterized in that, The method further includes: For each decision execution cycle that generates the optimal candidate action, a standard structured audit record is generated, and the audit record uses a hash chain structure to prevent tampering. The audit log includes a global identifier field, a model and version field, an input and output field, a reasoning and evidence field, and an intent statement field; the reasoning and evidence field includes: a scene text summary, a causal scene graph summary hash, a trigger reason code, an evidence fragment index, a model confidence level, and / or a secure shell verification result; The global identifier field includes a globally unique audit ID, a scenario unique ID, a transaction record number, and a millisecond-level timestamp; The model and version fields include the visual language model version, traffic regulation version, and driving strategy version; The input and output fields include an input data digest hash and an output action digest hash; The intent statement field is used to record structured driving intent statements.

6. The causal reasoning method for autonomous driving decision-making according to claim 1, characterized in that, The training process of the target detection model includes: Obtain a first training sample set, wherein each sample in the first sample set contains sample image frames captured by an in-vehicle camera in a driving scenario, and the location and category of traffic police personnel and traffic signs are marked as labels; The first training sample set is used to train the initial model based on YOLOv8-nano, YOLOv10-nano or YOLO-Lite model base. The sample image frames are used as input and the predicted values ​​of the location and category of the traffic police body and the traffic sign are output. The loss is constructed based on the deviation between the predicted value and the label, and the initial model parameters are updated to obtain the target detection model. The language description of the key frame is obtained through motion recognition and optical character recognition processing, including: using a BlazePalm lightweight detector to locate key points of the traffic police officer's hand, and recognizing the traffic police officer's gestures based on a preset gesture rule engine to obtain the corresponding gesture language description; detecting and outputting traffic signs based on optical character recognition, and matching the traffic sign database to output the traffic sign language description.

7. The causal reasoning method for autonomous driving decision-making according to claim 1, characterized in that, The causal model is pre-constructed using the DoWhy causal inference library based on the sample causal scenario graph. The sample causal scenario graph has the same form as the causal scenario graph. The structural equations of the causal model include road accessibility equation, vehicle priority equation, collision risk equation, and traffic rule compliance equation.

8. A causal reasoning device for autonomous driving decision-making, characterized in that, include: The device comprises a multimodal input preprocessing module, a visual language model semantic extraction module, a causal reasoning module, an action planning and evaluation module, a safe shell and degradation module, an observable evidence module, and an audit evidence storage module. The device is used to implement the steps of the method as described in any one of claims 1 to 7 when a computer program or instruction is executed.

9. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 7.

Images