End-to-end automatic driving decision control method and system

Abstract

The invention provides an end-to-end automatic driving decision control method and system, and belongs to the technical field of automatic driving. The invention relates to an end-to-end automatic driving decision control method based on multi-modal perception and hierarchical trajectory optimization, and the method comprises the steps: constructing an end-to-end decision closed loop through combining the zero sample migration capability of a vision-language-action (VLA) model with a hierarchical optimization architecture: analyzing multi-modal input (vision, language and point cloud) by using a pre-trained VLA model to generate path points; the vehicle pose is dynamically adjusted through upper-layer optimization to expand a feasible solution space, a smooth track meeting dynamics and collision avoidance constraints is solved in real time through lower-layer optimization, and finally a vehicle control instruction is output. According to the method, a multi-modal sensing and hierarchical trajectory optimization mechanism is fused, and the sensing generalization ability, the path planning feasibility and the control execution robustness of the system in a complex traffic environment are effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, in particular to an end-to-end automatic driving decision control method and system, and particularly to an end-to-end automatic driving decision control method and system based on multi-modal perception and hierarchical trajectory optimization. BACKGROUND

[0002] Current automatic driving systems generally adopt a modular architecture, separating the perception, planning and control functions for processing. Such methods have inherent defects: the recognition errors of the perception module will be passed to the planning and control layer, causing cumulative decision bias; traditional rule-based path planning relies on pre-defined maps and fixed scene logic, making it difficult to effectively respond to unexpected obstacles or unknown road conditions. In recent years, end-to-end methods directly map perception inputs to control signals through neural networks, although they simplify the processing flow, they still face two major bottlenecks: one is the need to rely on large-scale labeled data sets for training, the model generalization ability is limited, and it is difficult to cover long-tail scenarios; the second is the lack of explicit modeling of vehicle dynamics constraints, which is prone to generate physically infeasible trajectories, posing a safety hazard.

[0003] Although the vision-language-action large model has shown cross-task generalization ability in the field of fixed-base robots, its application is still difficult to directly adapt to the cooperative control requirements of vehicle chassis and actuators. In the prior art, although the trajectory optimization algorithm can embed the dynamics constraints, it fails to deeply integrate with multi-modal perception: a single sensor input cannot fully understand the complex semantic environment, and the traditional optimizer has low computational efficiency, which cannot meet the real-time requirements of high-speed driving. Therefore, there is an urgent need for an integrated framework that integrates multi-modal generalization perception, real-time physical constraint optimization and vehicle motion control to achieve safe and reliable automatic driving decisions in open scenarios. SUMMARY

[0004] In view of this, in order to solve the technical problems that the trajectory optimization algorithm in the prior art fails to deeply integrate with multi-modal perception, a single sensor input cannot fully understand the complex semantic environment, and the traditional optimizer has low computational efficiency, which cannot meet the real-time requirements of high-speed driving, the present application provides an end-to-end automatic driving decision control method and system based on multi-modal perception and hierarchical trajectory optimization, which constructs an end-to-end decision loop by combining the zero-shot transfer ability of the vision-language-action (VLA) model and the hierarchical optimization architecture: uses a pre-trained VLA model to analyze multi-modal inputs (vision, language, point cloud) to generate path points, dynamically adjusts the vehicle pose through the upper layer optimization to expand the feasible solution space, and the lower layer optimization solves the smooth trajectory that meets the dynamics and collision avoidance constraints in real time, and finally outputs the vehicle control instructions. This method breaks through the error accumulation bottleneck of the modular system, realizes complex scene generalization with ultra-low data requirements, while ensuring the physical feasibility of the trajectory and passenger comfort.

[0005] To achieve the above object, the present application provides the technical solutions as follows:

[0006] In the first aspect, the present application provides an end-to-end automatic driving decision control method based on multi-modal perception and hierarchical trajectory optimization, comprising the following steps:

[0007] Step S1, multi-modal perception and high generalization path point generation, using a pre-trained VLA large model to analyze multi-modal input data and generate a high generalization path point sequence, comprising the following steps:

[0008] Step S1.1, multi-modal environment semantic modeling;

[0009] Step S1.2, cross-scene path point generation;

[0010] Step S1.3, dynamic feasibility pre-screening cross-scene path point generation;

[0011] Step S2, double-layer real-time trajectory optimization framework, based on the path points generated by VLA, realizing physical feasible end-to-end control execution and real-time planning of closed-loop monitoring trajectory through hierarchical optimization, comprising the following steps:

[0012] Step S2.1, upper-layer vehicle pose candidate sampling and searching;

[0013] Step S2.2, lower-layer trajectory dynamic optimization solving;

[0014] Step (3), end-to-end control execution and closed-loop monitoring, converting the optimized trajectory into vehicle control instructions and realizing real-time safety monitoring, comprising the following steps:

[0015] Step S3.1, control instruction decoding and execution mapping;

[0016] Step S3.2, real-time feedback perception and fault compensation;

[0017] Step S3.3, closed-loop dynamic re-planning coordination mechanism.

[0018] In the second aspect, the present application further provides an end-to-end automatic driving decision control system based on multi-modal perception and hierarchical trajectory optimization, comprising:

[0019] A multi-modal perception and path point generation module is configured to receive visual, point cloud, language instruction and vehicle pose data, generate a cross-scene generalization path point sequence through a pre-trained visual-language-action model, and output physically feasible path points through dynamic feasibility screening;

[0020] A double-layer trajectory optimization module is configured to perform upper-layer pose candidate sampling and searching, lower-layer trajectory dynamic optimization solving based on the path point sequence, and generate an optimal trajectory satisfying vehicle dynamics constraints and collision avoidance constraints.

[0021] The control execution and closed-loop monitoring module is configured to decode the optimal trajectory into steering angle and acceleration control instructions, perceive the vehicle state through real-time feedback, detect abnormalities and trigger fault compensation, realize dynamic re-planning and seamless switching of the trajectory.

[0022] The present application combines the zero-shot transfer ability of the visual-language-action (VLA) model and the hierarchical optimization architecture to build an end-to-end decision-making closed loop: a pre-trained VLA model is used to analyze multi-modal input (vision, language, point cloud) to generate path points, the upper layer optimization dynamically adjusts the vehicle pose to expand the feasible solution space, and the lower layer optimization solves the smooth trajectory that meets the dynamics and collision avoidance constraints in real time, and finally outputs the vehicle control instructions. The present application combines multi-modal perception and hierarchical trajectory optimization mechanism, effectively improves the perception generalization ability, path planning feasibility and control execution robustness of the system in complex traffic environment, has the following beneficial effects:

[0023] The present application introduces a pre-trained visual-language-action large model to build a unified multi-modal semantic representation mechanism, realizing deep fusion and dynamic perception of various heterogeneous data such as images, laser radars and language instructions. This mechanism can accurately depict the semantic structure and dynamic target state of the traffic scene, and has the ability to express task relevance, significantly enhancing the semantic guidance and scene adaptation ability in the path generation stage.

[0024] The present application builds a hierarchical and collaborative decision-making and control architecture of "path point generation-trajectory optimization-control execution". Among them, the path point generation module is based on the joint modeling of semantics and tasks, and has cross-scene generalization ability; the trajectory optimization module adopts a double-layer structure, optimizes the vehicle pose and control input decoupling, and fully considers the vehicle dynamics constraints and environmental constraints, generates a control trajectory with high physical feasibility and good continuity; the control execution module realizes accurate mapping from trajectory state to low-level execution instruction, ensuring real-time performance and instruction controllability.

[0025] The present application introduces a real-time feedback perception and fault detection mechanism, which continuously monitors the trajectory tracking error, control instruction execution deviation and perception state abnormality during system operation. When a key fault or task deviation is detected, the system can automatically trigger the path point regeneration and trajectory re-planning module to realize seamless transition of the control trajectory and self-recovery of the system running state, significantly improving the adaptive ability and safety guarantee ability of the system in dynamic and complex scenes.

[0026] In summary, the application constructs a closed-loop, high-robustness, high-generalization capability end-to-end automatic driving decision control system, effectively overcoming the technical bottlenecks of information fragmentation, insufficient dynamic response capability and limited generalization capability in traditional modular systems. The system is suitable for automatic driving applications under various road types and task instruction conditions, and has broad engineering application prospects and promotion value. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 The method flowchart of the application is shown in the figure.

[0028] Figure 2 The multi-modal perception and path point generation module structure diagram is shown in the figure.

[0029] Figure 3 The double-layer trajectory optimization module structure diagram is shown in the figure.

[0030] Figure 4 The control execution and feedback compensation module structure diagram is shown in the figure.

[0031] Figure 5 The system result display diagram is shown in the figure. DETAILED DESCRIPTION

[0032] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the application.

[0033] As shown in the figure, Figure 1 The application provides an end-to-end automatic driving decision control method based on multi-modal perception and hierarchical trajectory optimization, characterized by comprising the following steps:

[0034] Step S1, multi-modal perception and high-generalization path point generation, using a pre-trained VLA large model to analyze multi-modal input data and generate high-generalization path point sequences, comprising the following steps:

[0035] Step S1.1, multi-modal environment semantic modeling;

[0036] Step S1.2, cross-scenario path point generation;

[0037] Step S1.3, dynamic feasibility pre-screening cross-scenario path point generation;

[0038] Step S2, double-layer real-time trajectory optimization framework, based on the path points generated by VLA, realizing physical feasible end-to-end control execution and real-time planning of closed-loop monitoring trajectory through hierarchical optimization, comprising the following steps:

[0039] Step S2.1, upper-layer vehicle pose candidate sampling and search;

[0040] Step S2.2, lower-layer trajectory dynamic optimization solving;

[0041] Step (3), end-to-end control execution and closed-loop monitoring, converting the optimized trajectory into vehicle control instructions and realizing real-time safety monitoring, including the following steps:

[0042] Step S3.1, control instruction decoding and execution mapping;

[0043] Step S3.2, real-time feedback perception and fault compensation;

[0044] Step S3.3, closed-loop dynamic re-planning coordination mechanism.

[0045] In the present application, step S1.1, multi-modal environment semantic modeling, through hierarchical feature extraction and dynamic fusion mechanism, multi-source sensor data of the vehicle is converted into unified spatio-temporal semantic representation, providing an interpretable environment understanding basis for path point generation, that is, receiving image I t , point cloud L t , task language semantic vector E L and vehicle pose P t ={x t ,y t ,θ t}, through image encoder, point cloud encoding network and language model to extract features E L , through semantic fusion module to generate unified enhanced spatio-temporal semantic representation for describing the structure and task intention in the scene, specifically including the following steps:

[0046] Step S1.1.1, standardization processing and spatial alignment of multi-modal perception data

[0047] Receiving multi-source perception data from camera (preferably RGB camera), laser radar (LiDAR) and inertial navigation system, respectively denoted as image I t , point cloud L t and pose information P t ={x t ,y t ,θ t}, due to different sampling frequencies and coordinate reference systems, all data need to be converted to vehicle reference coordinate system F s→v through the extrinsic matrix T v , that is:

[0048]

[0049] The process ensures that the subsequent multi-modal features have a consistent spatial reference basis, avoiding semantic drift and structural distortion problems.

[0050] Step S1.1.2, hierarchical semantic feature extraction and fusion

[0051] For the aligned image And the point cloud Use the visual encoder and the voxel point cloud network respectively to extract semantic features, and the image is encoded by ViT to obtain the visual semantic tensor The point cloud obtains the structural feature by sparse convolution network If there is a task language semantic vector E L , then use the language model to encode and cross-attention fuse with the image feature to construct the visual-language expression:

[0052]

[0053] Then the image-language feature and the point cloud feature are unified embedded through the graph attention fusion network to generate the full-scene enhanced spatio-temporal semantic representation Φ t , which is used to describe the key environmental information such as road, obstacle and traffic element;

[0054] Step S1.1.3, dynamic time sequence modeling to generate spatio-temporal semantic representation

[0055] Use the multi-frame feature sequence in the time window Extract the motion trend and behavior context through the time sequence Transformer encoder to obtain the enhanced spatio-temporal semantic representation:

[0056]

[0057] In the present application, step S1.2, cross-scene path point generation, generates a path point sequence with generalization ability through a semantic guided path sampling and candidate point screening mechanism, which is called by the subsequent trajectory optimization module, and specifically includes the following steps:

[0058] Step S1.2.1, path point condition generation based on semantic embedding

[0059] According to the enhanced spatio-temporal semantic representation output by step S1.1 And the task language semantic vector E L , the path point generation is modeled as a conditional generation process through the path generation function: Generate a set of path point sequences w i =(x i ,y i ,θ i ), wherein f genA path generation network that accepts semantic representation and task guidance as input, outputs structured path sequence, w i represents the i-th path point, containing two-dimensional plane coordinates (x i ,y i ) and heading angle θ i .

[0060] Step S1.2.2, multi-stage path point sampling and screening mechanism

[0061] The path generation network adopts a multi-stage sampling mechanism, first performs coarse-grained candidate sampling in free space based on the global semantic graph, and obtains an initial path point set Then, combined with the following constraints, screening is performed:

[0062] Semantic consistency constraint: the location of each candidate point needs to be marked as "passable area" in the semantic graph, that is, satisfy: Where C free represents the free space class label set;

[0063] Task relevance constraint: according to the task language semantic vector E L , select the candidate points that are highly matched with the instruction intent, and use attention matching score to sort;

[0064] Finally, select the path points that are structurally continuous and directionally consistent from the high-score area to form the refined path

[0065] Step S1.2.3, scene distribution normalization and path point standardization encoding

[0066] To ensure the stability of the path points under cross-scene conditions, the system performs normalization processing on the generated path point sequence W * . With the current vehicle state P t ={x t ,y t , θ t} as the reference, the relative position of each path point is defined as:

[0067]

[0068] Where x t , y t are the two-dimensional position coordinates of the current vehicle, θ t is the heading angle of the current vehicle, and R(-θ t ) is the inverse rotation matrix of the vehicle orientation.

[0069] In the present application, step S1.3, the dynamic feasibility pre-screening cross-scene path point generation, pre-screens the dynamic feasibility of the candidate path point sequence, eliminates the path points that do not meet the vehicle motion constraints, improves the efficiency and stability of subsequent trajectory optimization, and specifically includes the following steps:

[0070] Step S1.3.1, construct a vehicle dynamics feasibility constraint model, apply vehicle steering curvature, speed change rate and other constraints to each path point in W * , eliminate infeasible path points, and complete the post-screening sequence W final by interpolation, specifically as follows:

[0071] Let the current state of the vehicle be P t =(x t ,y t ,θ t ,v t ,a t ), where v t is the longitudinal speed and a t is the acceleration, a simplified single-track model is used to establish the feasibility judgment constraint, and the reachability of the candidate path point w i =(x i ,y i ,θ i ) must satisfy the following conditions:

[0072] Maximum curvature constraint:

[0073] Maximum acceleration and speed change rate constraint: |v i -v i-1 |≤Δv max ,|a i |≤a max ;

[0074] Where κ i represents the curvature, v i , a i are the expected speed and acceleration at the current path point κ max , Δv max , a max are the physical constraint thresholds of the vehicle;

[0075] Step S1.3.2, dynamic feasibility test of candidate path points

[0076] The path point sequence W is tested for dynamic conditions point by point, and a Boolean screening function δ(w i ) is constructed, defined as follows:

[0077]

[0078] According to the output of δ(w i ), the feasible path points are retained, and the path points not satisfying the condition are eliminated, to obtain a screened path point set:

[0079] Step S1.3.3, continuity correction and boundary interpolation completion

[0080] In order to prevent the discontinuity or sparseness of the path points caused by the screening operation, an interpolation method is used to insert compensation points in the screening result The compensation points are obtained by performing spline interpolation between the front and rear effective points w i , w i+1 , and whether the interpolation points satisfy the feasibility function δ(·) is checked point by point, and only the compensation points satisfying the condition are retained to construct the final feasible path sequence:

[0081] In the present application, step S2.1, upper vehicle pose candidate sampling and searching, performs upper vehicle pose candidate sampling and searching on the path point sequence, determines a global navigation pose set satisfying the physical constraint, and provides a feasible solution space for the lower trajectory optimization, a plurality of perturbed poses i are sampled at each path point w to form a candidate set P cand , the effective poses P valid are screened according to the physical feasibility, and the optimal navigation sequence P top is selected based on the target cost function, which specifically includes the following steps:

[0082] Step S2.1.1, constructing a candidate vehicle pose set based on the path points

[0083] Let the path point sequence output by step S1.3 be where each path point w i =(x i ,y i ,θ i ), x i and y i represent two-dimensional position coordinates, and θ i represents a heading angle, M vehicle pose perturbation samples are generated at each path point to expand the vehicle feasible attitude space, and a candidate vehicle pose set is constructed:

[0084]

[0085] where Δθ (m) ∈[-Δθ max ,Δθ max ] is the heading perturbation angle, and M is the sampling number hyperparameter;

[0086] The sampling results of all path points are combined to form an upper layer pose candidate set:

[0087]

[0088] Step S2.1.2, pose physical feasibility screening and search space compression

[0089] For each candidate pose According to the vehicle dynamics model, its physical feasibility is evaluated to ensure that the following constraint conditions are met:

[0090] (1) Steering radius constraint: whether the corresponding heading change is within the minimum turning radius R min ;

[0091] (2) Boundary condition constraint: whether the vehicle boundary polygon overlaps with the obstacle region;

[0092] (3) Connectivity constraint: whether there is a coherent motion path between the current pose and the last valid pose;

[0093] Define a feasibility judgment function δ p (p)∈{0,1}, the screened pose set is:

[0094] P valid ={p∈P cand ∣δ p (p)=1

[0095] Step S2.1.3, global search and sorting based on target driving

[0096] For each candidate pose in the feasible set P valid , calculate the state deviation cost to the target path end point ω N :

[0097]

[0098] Where (x p , y p , θ p ) represents the candidate pose, and α, β are weighting coefficients;

[0099] Sort P valid according to the cost function J(p), and select the top K pose sequence P top ={p1,…,p K} as the upper layer reference path input for trajectory optimization.

[0100] In the present application, step S2.2, lower layer trajectory dynamic optimization solving, based on the upper layer selected reference pose sequence, performs lower layer trajectory dynamic optimization solving, generates a continuous feasible control trajectory, meets the physical constraints and path point guiding requirements, and outputs the optimal trajectory T top as a constraint, defines a state sequence T={s t =(x t ,y t ,θ t ,v t ,a t )}, constructs a cost function J total , includes tracking error J track , smoothness J smooth and safety distance term J safe , solves using a constrained optimizer, and outputs the optimal trajectory T * , specifically including the following steps:

[0101] Step S2.2.1, trajectory optimization problem modeling

[0102] Let the candidate navigation pose sequence output by the upper layer be where each p k =(x k ,y k ,θ k ) is the target reference state of the vehicle at a discrete time, and the optimized trajectory is defined as a set of continuous state sequences:

[0103]

[0104] where s t represents the vehicle state at time t, x t , y t are the two-dimensional position coordinates of the vehicle at time t, θ t is the heading at time t, v t is the speed at time t, and a t is the acceleration at time t;

[0105] An optimization objective function is constructed:

[0106]

[0107] where J track represents the tracking error of the trajectory with respect to the upper layer reference pose;

[0108]

[0109] J smooth represents the smoothness loss of the trajectory state change;

[0110]

[0111] J safe represents the trajectory and environment collision risk measure;

[0112]

[0113] wherein, is the jth sampling boundary point of the vehicle at time t, D(·,Ω) is the distance from the point to the obstacle surface, ε0 is a safety threshold, N q is the number of sampling points, λ1, λ2, λ3 are loss weight parameters;

[0114] Step S2.2.2, optimization solving and trajectory generation

[0115] The objective function is iteratively solved by using a numerical optimizer with constraints to obtain the optimal trajectory T * The optimization initial value is generated by linear interpolation, the boundary state is determined by the current vehicle state and the end of the upper path, and the final output trajectory is:

[0116]

[0117] In the present application, step S3.1 controls instruction decoding and execution mapping, controls instruction decoding and execution mapping on the state sequence in the optimized trajectory, generates bottom layer control instructions executable by the vehicle, and realizes trajectory tracking control, specifically including the following steps:

[0118] Step S3.1.1, mapping modeling of trajectory state to control space

[0119] Let the optimized trajectory be wherein each state is:

[0120]

[0121] The system converts the state sequence into the corresponding control quantity sequence based on the vehicle dynamics model wherein: u t =(δ t ,a t ) represents the front wheel steering angle δ t and acceleration a t at time t;

[0122] The steering angle is calculated from the expected trajectory curvature:

[0123] δ t =arctan(L·κ t )

[0124] After discretization and synchronization, the control instructions are sent to the vehicle actuators, wherein κ tdenotes the curvature of the trajectory at time t, L is the wheelbase of the vehicle, and the curvature can be solved from the discrete points of the trajectory:

[0125]

[0126] Step S3.1.2, time synchronization of control instructions and reconstruction of discretization

[0127] Since the time resolution of the optimized trajectory is higher than the control frequency, the system resamples the control quantity U according to the control period Δt c through the time synchronization module to generate a synchronous control instruction stream:

[0128]

[0129] where T c is the total number of steps in the control period, is the kth control instruction, which is obtained based on trajectory interpolation and time alignment, and meets the controller scheduling frequency requirement;

[0130] Step S3.1.3, interface packaging and execution driving of control instructions

[0131] The control instructions are converted into vehicle execution layer interface commands by the execution module, including:

[0132] Steering system input: δ k →SteerActuator

[0133] Drive / brake system input: a k →Throttle / BrakeActuator

[0134] The instructions are packaged into data structures supported by the vehicle control bus and sent to the vehicle execution module to complete the “state-control” conversion in the control closed loop.

[0135] In the present application, step S3.2, real-time feedback sensing and fault compensation, establishes a real-time feedback sensing mechanism, and combines vehicle state and environment abnormality discrimination to execute fault compensation strategies to ensure the safety and robustness of system operation. The system continuously collects vehicle feedback information calculates error indicators and detects control deviation or sensing abnormalities, and triggers a redundant compensation mechanism if necessary, which specifically includes the following steps:

[0136] Step S3.2.1, real-time sensing and feedback collection of running state

[0137] During the execution process of the control module, the system continuously receives vehicle low-level feedback data, including the current position current speed actual acceleration Performing cornering and sensor state data S t All feedback data are collected in real-time through standard vehicle communication interfaces and updated at sampling period Δt s ;

[0138] The system calculates the tracking error between the current vehicle position and the reference trajectory state:

[0139]

[0140] where, is the reference trajectory state at the corresponding time, is the two-dimensional position coordinate of the reference trajectory at time t, is the heading angle of the reference trajectory at time t;

[0141] Step S3.2.2, anomaly detection and fault mode identification

[0142] The system sets multiple operating index thresholds to monitor control execution deviation and perception anomalies and identify the following typical fault modes:

[0143] Trajectory deviation fault: when or , it is judged as trajectory deviation;

[0144] Perception failure fault: when key sensor data is missing or constant, it is recorded as a perception anomaly;

[0145] Control lag or distortion: if the difference between and the command u t is greater than the threshold for consecutive time points, it is marked as an execution anomaly;

[0146] Define the fault discrimination function:

[0147]

[0148] Step S3.2.3, fault compensation mechanism and emergency handling strategy

[0149] When the system detects δ fault (t) = 1, the corresponding compensation strategy is triggered immediately, including:

[0150] (1) Trajectory correction compensation: taking the current vehicle state as the starting point, a short period of smooth trajectory is locally reconstructed to regenerate the control command

[0151] (2) Perception degradation compensation: activate the redundant sensor fusion module and switch to a low-precision positioning source to maintain continuous control;

[0152] (3) Control fault-tolerant mode: switch control to safe redundancy strategy, such as deceleration parking, entering protection state, and sending fault code F to the upper layer of the system t ;

[0153] During the execution of all compensation strategies, the system continuously records key states and abnormal indicators for subsequent closed-loop re-planning.

[0154] In the present application, step S3.3, the closed-loop dynamic re-planning coordination mechanism, continuously evaluates the current task execution state during system operation, and implements path point update and trajectory reconstruction when key conditions are triggered, improving system robustness and task completion rate. When the system detects abnormal state, the path point W new is reconstructed and the trajectory is re-planned Dynamic switching is achieved through trajectory splicing, maintaining control loop continuity and safety, which includes the following steps:

[0155] Step S3.3.1, re-planning trigger condition judgment and reconstruction request generation

[0156] During system execution, the following three types of signals are continuously received to determine whether re-planning is needed:

[0157] Control deviation exceeds threshold: if the trajectory tracking error or continuously exceeds the set threshold time window τ dev , that is:

[0158]

[0159] wherein, and are the tracking deviations of the vehicle in position and heading, respectively, τ dev represents the time window length for control deviation judgment, ε p and ε θ are the maximum allowable tolerance thresholds for trajectory position deviation and heading angle deviation, respectively. If the deviation exceeds the tolerance threshold ε dev or ε p at any time within the time window τ θ , the system determines that the current control result is invalid and needs to trigger the re-planning process.

[0160] Environment mutation detection: new dynamic obstacles or key semantic changes reported by the perception module, causing path conflicts;

[0161] System fault compensation failure: if the fault code F t ∈F critical defined in S3.2 cannot be stabilized, a re-planning request is immediately initiated;

[0162] Once any of the triggering conditions is met, the system sets the re-planning flag δ replan = 1, and sends the current state history state sequence environment update semantics to the waypoint generation module (S1.2) as context input;

[0163] Step S3.3.2, semantic-aware feedback-guided waypoint re-generation

[0164] Upon receiving the reconfiguration request, the S1.2 module re-executes the cross-scene waypoint generation process and generates a new waypoint sequence W based on the current real-time semantic map L , which is constructed in the same format as the initial path to ensure format consistency and interface compatibility; new

[0165] Step, followed by the automatic re-execution of the entire process S1.3-S2.2, including feasibility screening, trajectory optimization, and control decoding, to obtain a new execution trajectory and control instruction sequence U new ;

[0166] Step S3.3.3, seamless switching of reconfigured trajectory and system state synchronization

[0167] To ensure continuous and safe trajectory switching, the system introduces a time alignment mechanism to interpolate the transition segment between the end state of the old trajectory and the initial state of the new trajectory :

[0168]

[0169] The final trajectory is spliced as:

[0170]

[0171] Meanwhile, the execution instruction queue in the control buffer is updated to the newly generated U new , the system state returns to normal, and the re-planning flag δ replan is reset to 0.

[0172] At this point, the control execution and path planning form a complete closed-loop adaptive mechanism, realizing the entire process from high-generalization waypoint generation, hierarchical trajectory optimization to multi-source feedback-driven execution steady-state control.

[0173] The present application exhibits significant advantages in the field of autonomous driving decision control by integrating multi-modal perception, hierarchical trajectory optimization, and closed-loop monitoring mechanisms, specifically in the following aspects:

[0174] (1) Multi-modal fusion improves scene generalization ability​

[0175] The system adopts a pre-trained visual-language-action (VLA) large model, deeply fuses images, laser radar point clouds and task language instructions, and generates unified space-time semantic representations. Through semantic-guided path point generation and dynamic pre-screening, low-data-dependent cross-scene adaptation (such as urban roads, highways, and special scenes) is achieved, solving the problem of high demand for labeled data and insufficient generalization of long-tail scenes in traditional modular systems.

[0176] (2) Hierarchical optimization ensures trajectory physical feasibility

[0177] Innovatively, a two-layer optimization architecture of "upper-layer pose sampling-lower-layer trajectory solving" is constructed: the upper layer generates a global navigation pose sequence through path point perturbation sampling and physical constraint screening; the lower layer constructs an optimization objective function based on tracking error, smoothness and safety distance, and uses a constrained numerical optimizer to solve a trajectory that meets the vehicle dynamics (steering curvature, acceleration rate of change) and collision avoidance constraints in real time. This structure decouples search and optimization, balances task guidance and physical feasibility, and improves computational efficiency.

[0178] (3) End-to-end closed-loop control enhances dynamic response

[0179] Breaking through the information fragmentation bottleneck of traditional modular systems "perception-planning-control", an end-to-end closed loop from multi-modal input to actuator instructions is constructed. The control instruction decoding module directly converts the trajectory state into steering angle (based on curvature and wheelbase calculation) and acceleration, and after time synchronization and discretization processing, it ensures that the control frequency matches the vehicle actuator scheduling, reducing dynamic response delay.

[0180] (4) Real-time monitoring and fault tolerance improve system robustness

[0181] Through a closed-loop monitoring mechanism, full-process safety is ensured: real-time collection of vehicle position, speed and actuator state, calculation of trajectory tracking error; for trajectory deviation, perception failure and other faults, trajectory correction, redundant sensor switching or safety fault tolerance strategies (such as deceleration and parking) are triggered; the dynamic re-planning module quickly updates the path points through semantic feedback, and the new and old trajectories are smoothly transitioned through spline interpolation, ensuring control continuity in abnormal scenarios.

[0182] (5) Balance between low data dependence and engineering practicality

[0183] Without large-scale scene labeled data, relying on the zero-shot transfer ability of the VLA model and the hierarchical optimization architecture, high-precision path planning and control are achieved in complex dynamic environments. The system is compatible with standard vehicle bus protocols, and the control instruction interface supports mainstream chassis systems, with the advantage of plug-and-play engineering deployment, suitable for various autonomous vehicles and application scenarios.

[0184] In summary, the application effectively solves the core problems of weak generalization ability, poor physical feasibility, and dynamic response lag of traditional automatic driving systems through the full-link innovation of the "perception generalization-trajectory optimization-execution closed loop", providing technical support for safe and reliable driving in complex traffic environments.

[0185] In a second aspect, the application also provides an end-to-end automatic driving decision control system based on multi-modal perception and hierarchical trajectory optimization (as shown in Figure 2 The system comprises:

[0186] A multi-modal perception and path point generation module configured to receive visual, point cloud, language instruction and vehicle pose data, generate a cross-scene generalization path point sequence through a pre-trained visual-language-action model, and output a physically feasible path point after dynamic feasibility screening.

[0187] A double-layer trajectory optimization module configured to perform upper-layer pose candidate sampling and search and lower-layer trajectory dynamic optimization solution based on the path point sequence, and generate an optimal trajectory that meets vehicle dynamics constraints and collision avoidance constraints.

[0188] A control execution and closed-loop monitoring module configured to decode the optimal trajectory into steering angle and acceleration control instructions, detect abnormalities and trigger fault compensation through real-time feedback perception vehicle state, and realize dynamic re-planning and seamless trajectory switching.

[0189] As shown in Figure 2 The multi-modal perception and path point generation module comprises a multi-modal sensor input layer, a perception preprocessing module, a feature encoding module, a semantic fusion module and a path point generation module, which are connected in sequence in terms of function and data flow, and are used to parse heterogeneous perception information into unified spatio-temporal semantic representation and generate path point sequences with task relevance.

[0190] Wherein:

[0191] The multi-modal input layer is responsible for receiving various perception data including camera images, laser radar point clouds, vehicle position and attitude information, and optional task language instructions. All input data are time-synchronized and coordinate-converted in this module, unified to the vehicle's own reference system, ensuring spatial consistency in subsequent processing.

[0192] The preprocessing module is used to filter out noise, crop redundant areas, and complete point cloud and image registration, providing structured input for subsequent feature extraction.

[0193] The feature encoding module inputs image, point cloud and language information into a visual encoding network, a point cloud encoding network and a language model respectively, and extracts their respective intermediate feature representations. The image side focuses on semantic details, the point cloud side extracts geometric structures, and the language model expresses task intent or behavior goals.

[0194] The semantic fusion module dynamically fuses the features of different modalities through an attention mechanism and a graph neural structure to generate a unified semantic representation graph. The semantic graph not only reflects static structures such as roads, obstacles, and target objects, but also fuses driving intentions and local dynamic semantic information.

[0195] The path point generation module samples spatial regions based on the semantic graph and screens path candidates in combination with language guidance. The finally output path point sequence has continuity, target orientation, and passability, providing input for a subsequent trajectory planning module.

[0196] The module can achieve cross-scene adaptation and dynamic semantic understanding in the path generation process, laying a foundation for realizing high generalization and high robustness of automatic driving decision control.

[0197] As shown in Figure 3 The double-layer trajectory optimization module includes an upper-layer vehicle pose sampling and searching module and a lower-layer trajectory dynamic optimization module, which form an upper-lower structure in order of processing flow, and are used to convert the path point sequence into a set of optimal trajectories that satisfy physical feasibility and task constraints.

[0198] Among them:

[0199] The upper-layer pose sampling and searching module receives the path point sequence generated in the previous step, and generates a plurality of candidate poses with heading disturbance at each path point. The feasibility of the candidate poses is judged by a vehicle kinematics model, and invalid poses that do not satisfy constraints such as turning radius and attitude change amplitude are eliminated. Then, the system sorts according to the current task target and cost function, and selects a coherent reference pose sequence as the target skeleton of trajectory optimization.

[0200] The lower-layer trajectory dynamic optimization module constructs a set of trajectory state sequences based on the upper-layer candidate poses, including parameters such as position, speed, heading, and acceleration of the vehicle within a period of time. The module defines multiple optimization objectives, including tracking accuracy of the reference pose, smoothness of the motion process, and safety of obstacle avoidance in the environment. Under the premise of satisfying the vehicle dynamics constraints, the optimal trajectory is solved by using a numerical optimization method, and a continuous, smooth, and executable trajectory result is output.

[0201] The double-layer optimization structure decouples search and optimization, on the one hand guaranteeing the task guidance and environmental adaptability of trajectory generation, and on the other hand significantly improving the physical feasibility of the trajectory and the calculation efficiency of the algorithm, providing a stable input for real-time control execution.

[0202] As shown in Figure 4As shown, the control execution and closed-loop monitoring module includes a control instruction decoding module, an execution mapping module, a feedback sensing module, a fault detection module, and a dynamic re-planning module. The module as a whole constitutes a closed-loop structure of the control loop, realizing a complete control link from trajectory to instruction, from execution to feedback.

[0203] Wherein:

[0204] The control instruction decoding module receives the optimal trajectory state sequence output by the trajectory optimization module, extracts the expected position, speed, and heading at each time, and calculates the corresponding control instruction parameters, including the front wheel steering angle and longitudinal acceleration of the vehicle. The decoding result is interpolated and frequency matched to be converted into a continuous control instruction stream.

[0205] The execution mapping module encapsulates the control instructions generated in the previous step into an interface format recognizable by the vehicle chassis control system, and sends them to the steering system, driving system, and braking system respectively, to drive the vehicle to run according to the trajectory. This module supports standard protocol communication and has real-time performance and system compatibility.

[0206] The feedback sensing module collects the current motion state of the vehicle in real time, including actual position, speed, heading angle, and the state of each actuator, etc. By comparing with the reference trajectory, the key error indicators such as lateral deviation and heading deviation are calculated.

[0207] The fault detection module continuously monitors the feedback error and sensor health status. When there are problems such as trajectory deviation, control lag, or sensor abnormalities, the system immediately marks it as an abnormal state and records the current running context.

[0208] The dynamic re-planning module automatically calls the path point generation and trajectory optimization module when the fault detection module is triggered, quickly reconstructs the path point sequence and new trajectory based on the current vehicle state and environmental semantic information. The new trajectory is smoothly spliced with the original trajectory to realize uninterrupted trajectory switching and control transition.

[0209] Through the cooperative operation of the above modules, the system can realize an end-to-end closed-loop control process, has dynamic adaptive ability and fault recovery ability, and significantly improves the stability, safety, and practicality of the automatic driving system.

[0210] As shown in Figure 5 The system interface is composed of four parts. Frame ① shows the current running state of the vehicle, including driving, stopping, braking, accelerating, starting, etc. Frame ② shows the current time, signal state of the vehicle, and remaining power at the current time. Frame ③ shows the keys of the two display interfaces of the trajectory prediction system: "Vehicle Trajectory Prediction" and "Real-time Control Execution", which can be used to view the specific display information. Frame ④ shows the vehicle-related information corresponding to the keys in frame ③.

[0211] The "vehicle trajectory prediction" interface displays the predicted future trajectories of the target vehicle and its surrounding vehicles according to the input information at the current time. When the "vehicle trajectory prediction" button is clicked, the area displays the trajectory prediction results of the target vehicle and surrounding traffic participants generated according to the current environmental input, path points, and semantic map, including trajectory path curves, motion trend arrows, relative heading changes, etc. The system simultaneously labels the prediction time window and trajectory smoothness indicators, and distinguishes the predicted trajectories of the ego vehicle and other vehicles by color or solid and dashed lines.

[0212] The "vehicle trajectory prediction" interface displays the results of the system predicting the future motion trajectories of the autonomous vehicle and its surrounding traffic participants within a few seconds based on the current multi-modal perception information and path point sequence. The interface distinguishes the ego vehicle and other vehicle trajectories by color and line type, showing their position changes, speed trends, and heading changes. Combined with the semantic map and obstacle information, it can clearly present the motion intentions and potential interaction relationships of each subject in the short future, which is used to assist trajectory optimization and risk prediction.

[0213] The description of the specific embodiments of the present application is only used to illustrate the technical principles and implementation effects of the present application, and does not limit the protection scope of the present application. Any equivalent deformation or replacement based on the technical scheme of the present application shall fall within the protection scope of the present application.

Claims

1. An end-to-end autonomous driving decision control method based on multi-modal perception and hierarchical trajectory optimization, characterized in that, Comprise the following steps: Step S1, multi-modal perception and high generalization path point generation, using a pre-trained VLA large model to analyze multi-modal input data, generating high generalization path point sequences, comprising the following steps: Step S1.1, multi-modal environment semantic modeling; Step S1.2, cross-scene path point generation; Step S1.3, dynamic feasibility pre-screening cross-scene path point generation; Step S2, double-layer real-time trajectory optimization framework, based on the path points generated by VLA, through hierarchical optimization to realize physical feasible end-to-end control execution and real-time planning of closed-loop monitoring trajectory, comprising the following steps: Step S2.1, upper layer vehicle pose candidate sampling and search; Step S2.2, lower layer trajectory dynamic optimization solution; Step (3), end-to-end control execution and closed-loop monitoring, converting the optimized trajectory into vehicle control instructions and realizing real-time safety monitoring, comprising the following steps: Step S3.1, control instruction decoding and execution mapping; Step S3.2, real-time feedback perception and fault compensation; Step S3.3, closed-loop dynamic re-planning coordination mechanism.

2. The end-to-end autonomous driving decision control method based on multi-modal perception and hierarchical trajectory optimization according to claim 1, characterized in that, Step S1.1, multi-modal environment semantic modeling, through hierarchical feature extraction and dynamic fusion mechanism, converting vehicle multi-source sensor data into unified spatio-temporal semantic representation, providing explainable environment understanding basis for path point generation, specifically comprising the following steps: Step S1.1.1, standardization processing and spatial alignment of multi-modal perception data Receive multi-source perception data from camera, lidar and inertial navigation system, respectively denoted as image I t , point cloud L t and pose information P t ={x t , y t , θ t}, each modality data needs to be converted to vehicle reference coordinate system F v through extrinsic matrix T s→v due to different sampling frequencies and coordinate reference systems, that is: Step S1.1.2, hierarchical semantic feature extraction and fusion For the aligned image and point cloud Respectively use visual encoder and voxel point cloud network to extract semantic features, image is encoded by ViT to obtain visual semantic tensor Point cloud obtains structural features by sparse convolution network If there is a task language semantic vector E L , cross attention fusion is carried out with image features after using language model coding, and visual-language expression is constructed: The image-language features and the point cloud features are then unified embedded through a graph attention fusion network to generate a full-scene enhanced spatiotemporal semantic representation Φ t for describing key environmental information; Step S1.1.3, dynamic time series modeling to generate spatio-temporal semantic representation Using multiple frame feature sequences within a time window Extract motion trends and behavioral context through a temporal Transformer encoder, resulting in an enhanced spatio-temporal semantic representation:

3. The end-to-end autonomous driving decision control method based on multi-modal perception and hierarchical trajectory optimization according to claim 1, characterized in that, Step S1.2, cross-scene path point generation, through semantic guided path sampling and candidate point screening mechanism, generating path point sequences with generalization ability, for subsequent trajectory optimization module to call, specifically comprising the following steps: Step S1.2.1, path point condition generation based on semantic embedding enhanced spatio-temporal semantic representation output according to step S1.1 with task language semantic vector E L The path point generation is modeled as a conditional generation process by a path generation function: generating a set of path point sequences w i = (x i , y i , θ i ), where f gen is a path generation network that takes semantic representation and task guidance as input and outputs a structured path sequence, w i denotes the i-th path point, containing two-dimensional plane coordinates (x i , y i ) and a heading angle θ i ; Step S1.2.2, multi-stage path point sampling and screening mechanism The path generation network adopts a multi-stage sampling mechanism. First, coarse-grained candidate sampling is performed in free space based on a global semantic graph to obtain an initial path point set Subsequently, screening is performed in combination with the following constraints: semantic consistency constraint: each candidate point its location needs to be labeled as "drivable area" in the semantic map, i.e. satisfy: where C free denotes the free space class label set; Task-related constraints: according to the task language semantic vector E L , select candidate points with high matching with instruction intent, and use attention matching score to sort; The final refined path is composed of the path points selected from the high-score region with continuous structure and consistent direction Step S1.2.3, scene distribution normalization and path point standardization encoding To ensure the stability of the path points under cross-scene conditions, the system performs normalization processing on the generated path point sequence W * ; with the current vehicle state P t ={x t ,y t ,θ t} as a reference, the relative position of each path point is defined as: where x t ,y t are the current vehicle two-dimensional position coordinates, θ t is the current vehicle heading angle, and R(−θ t ) is the inverse rotation matrix of the vehicle orientation.

4. The end-to-end autonomous driving decision control method based on multi-modal perception and hierarchical trajectory optimization according to claim 1, characterized in that, Step S1.3, dynamic feasibility pre-screening cross-scene path point generation, pre-screening the candidate path point sequence for dynamic feasibility, eliminating path points that do not meet the vehicle motion constraints, improving the efficiency and stability of subsequent trajectory optimization, specifically comprising the following steps: Step S1.3.1, constructing vehicle dynamic feasibility constraint model Let the current state of the vehicle be P t = (x t , y t , θ t , v t , a t ), where v t is the longitudinal velocity and a t is the acceleration. A simplified single-track model is used to establish the feasibility judgment constraint, and the reachability of the candidate path point w i = (x i , y i , θ i ) must satisfy the following conditions: Maximum curvature constraint: Maximum acceleration and jerk constraints: |a i -v i-1 |≤Δv max ,|a i |≤a max ; where κ i represents the curvature, v i , a i are the desired speed and acceleration at the current path point, respectively, κ max , Δv max , a max are vehicle physical constraint thresholds; Step S1.3.2, dynamic feasibility test of candidate path points On a sequence of waypoints The dynamic condition check is performed point by point, constructing a Boolean screening function δ(w i ) defined as follows: According to the output of δ(w i ), the feasible path points are retained and the path points not satisfying the condition are eliminated to obtain a screened path point set: Step S1.3.3, continuity correction and boundary interpolation completion To prevent the path points from being discontinuous or sparse due to the screening operation, an interpolation method is used to insert compensation points in the screening result The compensation points are obtained by performing spline interpolation on the effective points w i , w i+1 in front and behind, and checking whether the interpolation points satisfy the feasibility function δ(·) point by point, only keeping the compensation points satisfying the condition, and constructing the final feasible path sequence:

5. The end-to-end autonomous driving decision control method based on multi-modal perception and hierarchical trajectory optimization according to claim 1, characterized in that, Step S2.1, upper layer vehicle pose candidate sampling and search, performing upper layer vehicle pose candidate sampling and search on the path point sequence, determining a global navigation pose set that meets the physical constraints, providing a feasible solution space for lower layer trajectory optimization, specifically comprising the following steps: Step S2.1.1, constructing candidate vehicle pose set based on path points Let the sequence of waypoints outputted by step S1.3 be where each waypoint w i = (x i , y i , θ i ), x i , y i denote two-dimensional position coordinates, and θ i denote a heading angle, M pose perturbation samples are generated at each waypoint, and a candidate vehicle pose set is constructed: where Δθ (m) ∈ [-Δθ max , Δθ max ] is the heading disturbance angle, M is the number of samples hyperparameter; Jointly sampling all path points to form an upper layer pose candidate set: Step S2.1.2, physical feasibility screening and search space compression For each candidate pose The physical feasibility is evaluated according to the vehicle dynamics model, ensuring that the following constraints are met: (1) Turn radius constraint: whether the corresponding heading change is within the minimum turn radius R min turn radius Rmin. (2) Boundary condition constraint: whether the vehicle boundary polygon overlaps with the obstacle region; (3) Connectivity constraint: whether there exists a continuous motion path between the current pose and the last valid pose; Definition of feasibility decision function δ p (p) e {0,1}, the filtered pose set is: P valid = { p e P cand | δ p (p) = 1 Step S2.1.3, global search and sorting based on target driving For each candidate pose in the feasible set P valid compute the state deviation cost from the end point ω of the target path: N of the target path: where (x p ,y p ,θ p ) represents a candidate pose, and α, β are weighting coefficients; According to the cost function J(p) for P valid Sort the sequences and select the top K pose sequences P. top ={p1,…,p K } serves as the upper-level reference path input for trajectory optimization.

6. The end-to-end autonomous driving decision control method based on multi-modal perception and hierarchical trajectory optimization according to claim 1, characterized in that, Step S2.2, lower-level trajectory dynamic optimization solving, based on the reference pose sequence selected by the upper layer, the lower-level trajectory dynamic optimization solving is performed to generate a continuous feasible control trajectory, which meets the physical constraints and path point guidance requirements, specifically including the following steps: Step S2.2.1, trajectory optimization problem modeling Let the candidate navigation pose sequence of the upper layer output be where each p k = (x k , y k , θ k ) is the target reference state of the vehicle at a discrete time, and define the optimized trajectory as a sequence of continuous states: s t = (x t , y t , θ t , v t , a t ) where s t represents the vehicle state at time t, x t , y t are the two-dimensional position coordinates of the vehicle at time t, θ t is the heading at time t, v t is the speed at time t, and a t is the acceleration at time t. Construct the optimization objective function: where J track denotes the tracking error of the trajectory pair to the upper reference pose; J smooth a smoothness loss representing a change in trajectory state; J safe represents a measure of the risk of collision of the trajectory with the environment; wherein, is the jth sampling boundary point of the vehicle at time t, D(·, Ω) is the distance from the point to the obstacle surface, ε0is a safety threshold, N q is the number of sampling points, λ1, λ2, λ3are loss weight parameters; Step S2.2.2, optimization solving and trajectory generation The numerical optimizer with constraints is used to iteratively solve the objective function to obtain the optimal trajectory T * The optimization initial value is generated by linear interpolation, the boundary state is determined by the current vehicle state and the end of the upper path, and the final output trajectory is:

7. The end-to-end autonomous driving decision control method based on multi-modal perception and hierarchical trajectory optimization according to claim 1, characterized in that, Step S3.1 control command decoding and execution mapping, the state sequence in the optimized trajectory is decoded and mapped to the control command, and the vehicle executable bottom-level control command is generated to realize the trajectory tracking control, specifically including the following steps: Step S3.1.1, mapping modeling from trajectory state to control space Let the optimized trajectory be where each state is: The system converts the state sequence into a corresponding control quantity sequence based on a vehicle dynamics model wherein: u t = (δ t , a t ) denotes the front wheel steering angle δ t and the acceleration a t at time t; The turning angle is calculated from the expected trajectory curvature: δ t = arctan(L - K t ) where κ t represents the curvature of the trajectory at time t, L is the wheelbase of the vehicle, and the curvature can be solved from the discrete points of the trajectory: Step S3.1.2, control command time synchronization and discrete reconstruction Since the time resolution of the optimized trajectory is higher than the control frequency, the system generates a synchronous control instruction stream by resampling the control quantity U according to the control period Δt c through the time synchronization module: wherein T c is the total number of steps in the control period, is the kth control command, obtained based on trajectory interpolation and time alignment, satisfying the controller scheduling frequency requirement; Step S3.1.3, control command interface packaging and execution driving Control instructions By executing the module transformation into a vehicle execution layer interface command, including: Steering system input: δ k → Steer Actuator Drive / Brake System Input: a k → Throttle / Brake Actuator The command is packaged into a data structure supported by the vehicle control bus and sent to the vehicle execution module to complete the "state-control" conversion in the control closed loop.

8. The end-to-end autonomous driving decision control method based on multi-modal perception and hierarchical trajectory optimization according to claim 1, characterized in that, Step S3.2, real-time feedback sensing and fault compensation, a real-time feedback sensing mechanism is established, and fault compensation strategies are executed in combination with vehicle state and environment anomaly discrimination to ensure the safety and robustness of system operation, specifically including the following steps: Step S3.2.1, real-time sensing and feedback collection of running state During the execution of the control module, the system continuously receives low-level feedback data from the vehicle, including the current position current speed actual acceleration turning angle and sensor status data S t All feedback data is collected in real time through standard vehicle communication interfaces and updated at a sampling period Δt s . The system calculates the tracking error between the current position of the vehicle and the reference trajectory state: wherein, is the reference trajectory state at the corresponding time instant, is the two-dimensional position coordinate of the reference trajectory at time instant t, is the heading angle of the reference trajectory at time instant t; Step S3.2.2, anomaly detection and fault mode identification The system sets multiple running index thresholds to monitor control execution deviation and sensing anomalies, and identifies the following typical fault modes: Trajectory deviation fault: when or trajectory deviation is judged. Sensing failure fault: when the key sensor data is lost or constant, it is recorded as a sensing anomaly; Control lag or distortion: if a number of consecutive time instants with the instruction u t Difference greater than threshold, flag execution anomaly; Define the fault discrimination function: Step S3.2.3, fault compensation mechanism and emergency handling strategy When the system detects that δ fault (t) = 1, the corresponding compensation strategy is triggered immediately, including: (1) Trajectory deviation correction compensation: taking the current vehicle state as the starting point, locally reconstructing the short-term smooth trajectory, and regenerating the control command ​ (2) Sensing degradation compensation: activate the redundant sensor fusion module and switch to a low-precision positioning source to maintain continuous control; (3) Control fault-tolerant mode: switch control to a safe redundancy strategy, such as deceleration stop, enter protection state, and send fault code F to the upper layer of the system t ; During the execution of all compensation strategies, the system continuously records key states and anomaly indicators for subsequent closed-loop re-planning.

9. The end-to-end autonomous driving decision control method based on multi-modal perception and hierarchical trajectory optimization according to claim 1, characterized in that, Step S3.3, closed-loop dynamic re-planning coordination mechanism, continuously evaluate the current task execution state during system operation, and implement path point update and trajectory reconstruction when key conditions are triggered to improve system robustness and task completion rate, specifically including the following steps: Step S3.3.1, re-planning trigger condition determination and reconstruction request generation During system execution, the following three types of signals are continuously received to determine whether re-planning is needed: Controlled bias overshoot: if the trajectory tracking error or continues to exceed a set threshold time window τ dev i.e.: or wherein, with respectively the tracking deviation of the vehicle in position and heading, τ dev denotes the length of the time window for control deviation judgment, ε p with θ respectively the allowed maximum tolerance threshold of the trajectory position deviation and the heading angle deviation, if at any time within the time window τ dev the deviation exceeds the tolerance threshold ε p or ε θ , the system judges that the current control result is invalid, and the re-planning process needs to be triggered. Environment mutation detection: new dynamic obstacles or key semantic changes reported by the sensing module cause path conflicts; System failure compensation failure: if the fault tolerance mechanism defined in S3.2 still cannot run stably, feedback the fault code F t ∈ F critical Then immediately initiate a re-planning request; Once any of the triggering conditions is true, the system sets a re-planning flag δ replan = 1 and sends the current state history of state sequences environment update semantics together with the current state sequence and the history of state sequences to the waypoint generation module (S1.2) as context input; Step S3.3.2, path point regeneration guided by semantic sensing feedback Upon receiving the reconstruction request, the S1.2 module re-executes the cross-scene path point generation process and updates the path point sequence W with the task instruction E L , the path point sequence W new is updated, and the construction is the same as the initial path, ensuring format consistency and interface compatibility; Step, then S1.3-S2.2 full flow automatic re-execution, including feasibility screening, trajectory optimization and control decoding, get a new execution trajectory with control instruction sequence U new ; Step S3.3.3, seamless switching of reconstructed trajectory and system state synchronization To ensure the continuity and safety of trajectory switching process, the system introduces a time alignment mechanism to transition the end state of the old trajectory to the initial state of the new trajectory and constructs a transition segment interpolation The final trajectory is spliced as: The execution instruction queue in the control buffer is updated at the same time to the newly generated U new , the system state returns to normal, and the reset re-planning flag δ replan ← 0.

10. An end-to-end autonomous driving decision control system based on multi-modal perception and hierarchical trajectory optimization, characterized in that, Including: The multi-modal perception and path point generation module is configured to receive visual, point cloud, language instruction and vehicle pose data, generate a cross-scene generalization path point sequence through a pre-trained visual-language-action model, and output a physically feasible path point through dynamic feasibility screening; The double-layer trajectory optimization module is configured to perform upper-layer pose candidate sampling and search and lower-layer trajectory dynamic optimization solving based on the path point sequence, and generate an optimal trajectory satisfying vehicle dynamics constraints and collision avoidance constraints; The control execution and closed-loop monitoring module is configured to decode the optimal trajectory into steering angle and acceleration control instructions, perceive vehicle state through real-time feedback, detect abnormalities and trigger fault compensation, and realize dynamic re-planning and trajectory seamless switching.

Images