A man-machine co-driving closed-loop optimization method and system based on end-to-end and world model fusion

Abstract

The application provides a man-machine co-driving closed-loop optimization method and system based on end-to-end and world model fusion, and belongs to the technical field of intelligent driving. In view of the technical problems that the existing man-machine cooperation signal cannot be fully utilized, the closed-loop optimization mechanism is imperfect, and the world model scene generation lacks risk assessment and strong man-machine cooperation constraint, the end-to-end automatic driving model is deeply fused with the multi-level dynamic world model, and a data and cognition double-driven man-machine co-driving cooperative interaction framework is constructed. Through the man-machine bidirectional mutual feedback closed-loop structure, the driver takeover, intervention and preference signal is fed into the system in real time to drive the decision optimization and model iteration, and based on the risk state, scene prediction and cause inversion result, the active safety bottom protection and dynamic interaction guidance are provided to the driver. Through the prediction and planning iteration mechanism coupled with the world model and the double-loop closed-loop evolution mechanism, the system is continuously optimized.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving technology, and more particularly to a human-machine co-driving closed-loop optimization method and system based on end-to-end fusion with a world model. Background Technology

[0002] Autonomous driving technology can be classified into six levels, from L0 to L5, according to the Society of Automotive Engineers (SAE) classification standard. Levels L0 to L2 represent driver assistance, where the system provides steering, acceleration, or braking assistance in specific scenarios, but the driving entity remains a human driver. Level L3 is conditional automation, where the system undertakes all dynamic driving tasks under its designed operating conditions, but requires a human driver to take over when requested. Level L4 is highly automated driving, where the system can complete driving tasks without human intervention in specific scenarios. Level L5 is fully automated driving, where the system can independently complete driving tasks in all scenarios. In recent years, end-to-end autonomous driving systems have simplified the traditional modular pipeline of perception, prediction, and planning by directly mapping raw sensor data to planned trajectories, demonstrating superior performance. Meanwhile, world models, with their high-fidelity world state prediction capabilities, have received widespread attention and are being used to improve the performance of autonomous driving systems.

[0003] Current autonomous driving technologies still have significant shortcomings. First, in the transition from L2 to L3, human-machine cooperative driving is the core paradigm, but existing systems lack efficient human-machine interaction and collaboration mechanisms. Second, the human-machine cooperative signals generated during actual driving (such as takeover actions and interventions) contain rich driving knowledge and safety information, but existing systems have failed to fully utilize these signals to improve driving strategies. Finally, existing systems mostly adopt an open-loop model of offline training and online inference, lacking an effective closed-loop optimization mechanism. This leads to a disconnect between scenario generation and actual driving needs, making it difficult to achieve continuous performance improvement and personalized adaptation. These problems limit the performance and user experience of autonomous driving systems in actual deployment. Summary of the Invention

[0004] To address the technical challenges of underutilized human-machine collaboration signals, imperfect closed-loop optimization mechanisms in autonomous driving systems, and the lack of end-to-end model risk assessment and strong human-machine collaboration constraints in world model scenario generation, this invention provides a human-machine co-driving closed-loop optimization method and system based on end-to-end and world model fusion. This invention primarily utilizes the deep fusion of an end-to-end autonomous driving model and a multi-level dynamic world model to construct a data- and cognition-driven human-machine co-driving collaborative interaction network. Through a two-way human-machine feedback closed-loop structure and a dual-loop closed-loop evolution mechanism, human-machine collaboration signals such as driver takeover, intervention, and preferences are fed into the system in real time to drive decision optimization and model iteration. Simultaneously, based on risk status, scenario prediction, and causal inversion results, it provides the driver with proactive safety protection and dynamic interactive guidance. This enables joint reasoning of perception, prediction, and planning, multi-level dynamic human-machine collaboration, and continuous closed-loop evolution of the system in complex dynamic driving environments, significantly improving the safety, adaptability, and interpretability of intelligent driving systems in long-tail scenarios.

[0005] The technical means employed in this invention are as follows: A closed-loop optimization method for human-machine co-driving based on end-to-end and world model fusion includes: S1. Acquire multimodal sensor data, preprocess and align the multimodal sensor data, extract multimodal features and map them to a shared implicit feature space, project them to the bird's-eye view coordinate system after spatiotemporal information interaction and aggregation, and generate a unified representation vector of driving scene based on BEV feature network. S2. Construct a multi-level dynamic world model, and use the generated unified representation vector of the driving scene and the BEV feature grid as scene conditions input to the multi-level dynamic world model to generate visual images of future driving scenes and movement trajectories of traffic participants. S3. Obtain human-machine collaboration signals and expected risk assessment representations, encode the human-machine collaboration signals into collaboration signal semantic vectors, and use cross-attention mechanism to interactively fuse the collaboration signal semantic vectors with the expected risk assessment representations to obtain scene generation constraints. Introduce the scene generation constraints into a multi-level dynamic world model to guide the multi-level dynamic world model to generate driving scenarios in a targeted manner. S4. Construct an end-to-end autonomous driving model based on BEV feature grid. The end-to-end autonomous driving model generates candidate planning trajectories through an integrated prediction and planning network, and iteratively optimizes the candidate planning trajectories through a prediction and planning iterative optimization mechanism coupled with the multi-level dynamic world model, and outputs the optimal planning trajectory. S5. Collect information on human driver takeover behavior, intervention operations, driving preferences, and system active takeover to construct a trajectory preference pair dataset, including trajectory pairs of human-takeover vehicles, system-takeover vehicles, and normal driving trajectories; use the trajectory preference pair dataset to train an implicit reward function model and a preference evaluation network, respectively; use the preference evaluation network to evaluate the social rationality of the optimal planned trajectory and output a preference score; optimize the end-to-end autonomous driving model based on the preference score and the implicit reward function model, and output the optimized planning strategy. S6. Perform real-time multi-dimensional risk assessment on the end-to-end autonomous driving model, and output risk score, risk level and safety constraints; when a risk event is detected, save a full-dimensional state snapshot, use causal Transformer and time-series backtracking algorithm to decompose the risk causes, output standardized cause labels and label confidence level and cause elements. S7. Real-time acquisition of driver takeover timing, driving behavior, driving preferences, attention state, and system active takeover trigger signals. Human-machine interaction semantic vectors are generated by the collaborative signal encoder. The human-machine interaction semantic vectors are input into the conditional embedding layer of the end-to-end autonomous driving model and the multi-level dynamic world model. Based on risk level, standardized cause labels, future scenario prediction, and control transfer direction, graded warnings, takeover suggestions, system takeover notifications, permission adjustments, and interactive guidance are output to the driver to establish a human-machine interaction time sequence record chain.

[0006] Further, step S1 includes: S11. Select the forward-looking and surround-view image sequences and the vehicle's historical trajectory data as the multimodal sensor data input source, and perform time alignment and spatial registration on the multimodal sensor data to form a time-consistent multimodal input stream; S12. Preliminary features are extracted from the original data of each modality through their respective dedicated backbone networks, and all modal features are uniformly mapped to a shared implicit feature space through a learnable projection layer. S13. Input the aligned multimodal feature sequence into the spatiotemporal Transformer encoder to perform information interaction and aggregation simultaneously in both time and space dimensions, and output the fused feature tensor. S14. The fused feature tensor is uniformly projected onto the bird's-eye view coordinate system centered on the vehicle through a differentiable view transformation module to generate a discretized, grid-structured BEV feature grid, with each grid encoding the multimodal spatiotemporal semantic information of the corresponding physical region. S15. Apply semantically aware adaptive pooling to the BEV feature grid, use a lightweight semantic segmentation head to perform grid-by-grid semantic category prediction, use a smaller pooling kernel to preserve spatial details for regions containing key dynamic traffic participants, use a larger pooling kernel to compress redundant information for background regions, and output a fixed-dimensional unified representation vector of the driving scene.

[0007] Further, step S2 includes: S21. A multi-level dynamic world model is constructed using a decoupled spatiotemporal diffusion network, including a causal Transformer, a diffusion Transformer, a temporal dynamic modeling module, and a spatial detail and behavior generation module, wherein: The causal Transformer models the sequence of historical scene states in an autoregressive manner and extracts temporal causal dependencies; The Diffusion Transformer iteratively generates refined spatial features and behavioral distributions of future states through forward diffusion and backward denoising processes. The temporal dynamic modeling module captures the long-term dependencies of driving scenarios as they evolve over time. The spatial detail and behavior generation module generates high-resolution spatial details and traffic participant behavior; S22. The unified representation vector of the driving scenario and the BEV feature grid are used as scene conditions input to the multi-level dynamic world model. The driving scenario condition information is encoded into a semantic vector through the condition embedding layer. The condition embedding layer includes physical layer information, behavioral layer information and rule layer information, which correspond to the physical kinematic state, the behavioral intention of traffic participants and the scene condition information under traffic rule constraints, respectively. S23. Using VisDiT Visual Diffusion Transformer and TrajDiT Trajectory Diffusion Transformer, the semantic vector, temporal causal dependency, future spatial features, and behavioral distribution are accepted as conditional inputs to generate high-fidelity visual images of future driving scenes and motion trajectories of traffic participants, respectively.

[0008] Further, step S3 includes: S31. The driver's takeover timing, intervention operation, and driving preference information are extracted and semantically encoded by a cooperative signal encoder to generate a fixed-dimensional cooperative signal semantic vector; wherein, the cooperative signal encoder includes a temporal coding branch and a behavioral coding branch, which encode continuous temporal operation signals and discrete intervention type signals, respectively. S32. Input the collaborative signal semantic vector and the expected risk assessment representation into the cross-attention layer. In the cross-attention layer, use the expected risk assessment representation as the query vector and the collaborative signal semantic vector as the key vector to calculate the attention weight and perform weighted fusion to generate scene generation constraints that include risk orientation and cognition orientation. S33. The scene generation constraints are injected into the diffusion Transformer of the multi-level dynamic world model through the condition embedding layer. During the reverse denoising process, the scene generation constraints guide the gradual elimination of noise, and generate high-risk, high-difficulty or high-value driving scenarios that match the cooperative signal semantic vector and the expected risk assessment representation.

[0009] Further, step S4 includes: S41. Input the BEV feature mesh into the interaction modeling network to generate a graph structure of traffic participants, where nodes represent the state of traffic participants and edges represent interaction relationships; extract interaction features through a graph attention mechanism and fuse them with BEV features to obtain an interaction-enhanced scene representation. S42. Construct an intent latent space through cross-modal contrastive learning and encode the behavioral intent of traffic participants as policy latent variables; S43. Input the interaction enhancement scenario representation and the policy latent variables into the prediction planning integrated network to generate multiple candidate planning trajectories; S44. Construct a multi-objective reward function for the candidate planned trajectories, including safety, smoothness, regularity, task completion, and human-like indicators; dynamically weight different objectives through an attention weight matrix, and introduce conservative regularization constraints to limit policy offset, thereby obtaining an optimized set of candidate trajectories. S45. Based on real driving data, the integrated prediction and planning network is pre-trained using imitation learning, and then trained using an offline reinforcement learning method with a multi-objective reward function as the optimization objective. S46. Input the candidate planning trajectory and the unified representation of the driving scene into the multi-level dynamic world model to predict the scene state in future multiple frames; S47. Based on the predicted future multi-frame scene state, evaluate the safety, interaction rationality and risk level of the candidate trajectory in the long time series, and feed the evaluation results back to the prediction and planning integrated network to iteratively optimize the trajectory and output the optimal planned trajectory.

[0010] Further, step S5 includes: S51. Collect information on human drivers' takeover behaviors, interventions, driving preferences, and system-initiated takeovers during real-world driving processes. Clean and filter the collected information to distinguish between human-led takeover events, system-led takeover events, and normal driving events. For system-led takeover events, retain trajectory segments within a preset time window before and after the system-triggered takeover. For normal driving events, retain reference trajectory segments of equal length. Construct pairs of human-led takeover trajectories containing the system-planned trajectory before takeover and the driver's actual trajectory after takeover, pairs of system-led takeover trajectories containing the driver's trajectory before system-triggered takeover and the system-planned trajectory after takeover, and pairs of normal driving reference trajectories. Perform data augmentation and distribution balancing on the trajectory preference dataset. S52. Based on the trajectory preference dataset, an implicit reward function model is trained using a ranking loss function. The ranking loss function aims to maximize the reward difference between the trajectory after takeover and the trajectory before takeover. It learns the driver's driving preferences, system safety preferences, and social value constraints, and outputs the implicit reward value of the trajectory. S53. Construct a CPEN preference evaluation network based on contrastive learning, including a trajectory feature encoder, a scene context encoder, and a preference score fusion machine. Input the candidate planned trajectory and the current scene features into the trajectory feature encoder and the scene context encoder, respectively, and output the preference score through the preference score fusion machine. S54. Construct a composite optimization objective using the implicit reward value and the preference score. Introduce safety boundary constraints and conservative regularization terms into the composite optimization objective. Perform safety reinforcement learning fine-tuning on the end-to-end autonomous driving model. Introduce safety negative samples, including collision risks and violations, for adversarial training. S55. The elastic weight consolidation method is used to apply secondary penalty constraints to the key network parameters related to social reasoning in the end-to-end autonomous driving model, freeze the basic network parameters that have been trained and converged, and update the relevant parameters to optimize the strategy. S56. Construct an experience replay buffer, prioritize sampling according to scene difficulty and preference intensity, and mix the trajectory preference dataset with the high-value scenes generated by the multi-level dynamic world model in a preset ratio, and use them together to retrain the end-to-end autonomous driving model to output the optimized planning strategy.

[0011] Further, step S6 includes: S61. Input the current operating status of the end-to-end autonomous driving model into the risk assessment network, and calculate the scenario complexity risk score, interaction risk score, decision compliance risk score, trajectory safety risk score and takeover urgency risk score respectively. S62. The risk scores for scenario complexity, interaction, decision-making compliance, trajectory safety, and takeover urgency are weighted and integrated to obtain a comprehensive risk score. S63. Map the comprehensive risk score to a risk level according to a preset risk level threshold, and generate safety constraints corresponding to the risk level. S64. Collect the operation data and risk event data of the end-to-end autonomous driving model as the data driving source, collect human driver takeover behavior, intervention operation, driving preference information and system active takeover information, and encode them into cognitive semantic vectors as the cognitive driving source. S65. Input the data-driven source and the cognitive-driven source into the fusion network to generate a human-machine collaborative optimization signal; S66. Monitor the operating status of the end-to-end autonomous driving model and the driver's operation status. When a risk trigger, system anomaly, or driver forced takeover or system active takeover is detected, trigger a full-dimensional state snapshot saving instruction. The full-dimensional state snapshot includes raw sensor data, perception output results, prediction output results, planning output results, control output results, control instruction sequence, vehicle state parameters, driver operation sequence, traffic participant behavior data, and environmental context information. S67. Input the full-dimensional state snapshot into the causal Transformer to extract the state transition relationship and temporal dependency features within the preceding time window of the risk event; locate the key nodes of risk evolution through the temporal backtracking algorithm, and distinguish between system-side causes and driver-side causes; label system-side causes with tags such as perception miss, prediction deviation, unreasonable planning, control delay, or module coordination failure; label driver-side causes with tags such as operational error, lack of concentration, misjudgment of intent, or personal driving style; output standardized cause labels containing cause type, confidence score, and cause elements.

[0012] Further, step S7 includes: S71: Real-time data collection of driver takeover timing, driving behavior, driving preferences and attention status is achieved through in-vehicle cameras, steering wheel torque sensors, pedal travel sensors and eye-tracking devices, while monitoring system active takeover trigger signals. S72. The collected driver status signal and system active takeover signal are input into the cooperative signal encoder. The cooperative signal encoder performs sliding window segmentation and feature extraction on the continuous time-series signal, performs one-hot encoding on the discrete event signal, and fuses them to generate a fixed-dimensional human-machine interaction semantic vector. S73. The human-machine interaction semantic vector is input into the policy optimization parameter space of the end-to-end autonomous driving model and the conditional embedding layer of the multi-level dynamic world model through the interface layer, respectively, wherein: In the end-to-end autonomous driving model, the human-machine interaction semantic vector and the scene representation vector are concatenated and then input into the prediction-planning integrated network. In the multi-layered dynamic world model, the human-machine interaction semantic vector is fused with the physical layer, behavioral layer, and rule layer condition vectors to guide the direction of scene generation. S74. Obtain the risk level, the standardized causative label, the future scenario prediction result output by the multi-level dynamic world model, and the current control ownership status, and input them into the human-machine collaborative feedback decision network. The human-machine collaborative feedback decision network matches and outputs content according to a preset hierarchical mapping table. When the risk level is emergency and control is transferred from the system to the human, output a takeover suggestion; when the risk level is warning, output a tiered warning; when the system triggers active takeover, output a system takeover notification; when the cause label points to the system side, output a permission adjustment instruction; when the cause label points to the driver side, output interactive guidance information. S75. The timing of the driver's takeover, driving behavior, driving preferences and attention state are matched one-to-one with the hierarchical warnings, takeover suggestions, permission adjustments and interactive guidance output by the system according to the timestamp, and the data is stored in alignment to establish a human-machine interaction time sequence record chain.

[0013] This invention also provides a human-machine co-driving closed-loop optimization system based on the aforementioned human-machine co-driving closed-loop optimization method based on end-to-end and world model fusion, comprising: a unified representation module for driving scenarios, a multi-level dynamic world model construction module, a scenario orientation generation module, an end-to-end autonomous driving model construction module, a driver feedback-driven social reasoning optimization module, a real-time risk assessment and causal inversion module, and a human-machine two-way feedback interaction module, wherein: The unified representation module for driving scenarios is used to acquire multimodal sensor data, preprocess and align the multimodal sensor data, extract multimodal features and map them to a shared implicit feature space, project them to the bird's-eye view coordinate system after spatiotemporal information interaction and aggregation, and generate a unified representation vector for driving scenarios based on the BEV feature network. The multi-level dynamic world model construction module is used to construct a multi-level dynamic world model. The generated unified representation vector of the driving scene and the BEV feature grid are used as scene conditions input into the multi-level dynamic world model to generate a visual image of the future driving scene and the movement trajectory of traffic participants. The scene-oriented generation module is used to acquire human-machine collaboration signals and expected risk assessment representations, encode the human-machine collaboration signals into collaboration signal semantic vectors, and interact and fuse the collaboration signal semantic vectors with the expected risk assessment representations through a cross-attention mechanism to obtain scene generation constraints. The scene generation constraints are then introduced into a multi-level dynamic world model to guide the multi-level dynamic world model to generate driving scenarios in an oriented manner. The end-to-end autonomous driving model construction module is used to construct an end-to-end autonomous driving model based on the BEV feature grid. The end-to-end autonomous driving model generates candidate planning trajectories through an integrated prediction and planning network, and iteratively optimizes the candidate planning trajectories through a prediction and planning iterative optimization mechanism coupled with the multi-level dynamic world model, and outputs the optimal planning trajectory. The driver feedback-driven social reasoning optimization module is used to collect human driver takeover behavior, intervention operations, driving preference information, and system active takeover information to construct a trajectory preference pair dataset, including trajectory pairs of human-takeover vehicles, trajectory pairs of system-takeover vehicles, and normal driving trajectory pairs. The implicit reward function model and preference evaluation network are trained using the trajectory preference pair dataset. The preference evaluation network evaluates the social rationality of the optimal planned trajectory and outputs a preference score. Based on the preference score and the implicit reward function model, the end-to-end autonomous driving model is optimized, and the optimized planning strategy is output. The real-time risk assessment and cause inversion module is used to perform real-time multi-dimensional risk assessment on the end-to-end autonomous driving model, and output risk scores, risk levels and safety constraints. When a risk event is detected, a full-dimensional state snapshot is saved, and the causes of the risk are decomposed using causal Transformer and time-series backtracking algorithms. Standardized cause labels are output and confidence and cause elements are marked. The human-machine two-way feedback interaction module is used to collect driver takeover timing, driving behavior, driving preferences, attention state, and system active takeover trigger signals in real time. The collaborative signal encoder generates human-machine feedback semantic vectors, which are then input into the conditional embedding layer of the end-to-end autonomous driving model and the multi-level dynamic world model. Based on risk level, standardized cause labels, future scenario predictions, and control transfer direction, the module outputs graded warnings, takeover suggestions, system takeover notifications, permission adjustments, and interactive guidance to the driver, establishing a human-machine interaction time sequence record chain.

[0014] Compared with the prior art, the present invention has the following advantages: 1. The human-machine co-driving closed-loop optimization method based on end-to-end and world model fusion provided by this invention feeds driver takeover, intervention and preference behavior signals into the system in real time to drive decision optimization and model iteration. At the same time, based on risk status, scenario prediction and cause inversion results, it outputs graded warnings, takeover suggestions and interactive guidance to the driver, realizing human-machine intention alignment, clear responsibility boundaries and controllable risk co-driving closed loop.

[0015] 2. The human-machine co-driving closed-loop optimization method based on end-to-end and world model fusion provided by this invention performs full-dimensional causal inversion of risk events through causal Transformer and time-series backtracking algorithm, accurately distinguishes between driver human factors and system failure factors, realizes human-machine responsibility definition and problem tracing, and provides accurate basis for workload optimization, safety strategy adjustment and system continuous evolution.

[0016] 3. The human-machine co-driving closed-loop optimization method based on end-to-end fusion with the world model provided by this invention encodes the driver's takeover behavior, intervention operations and driving preferences into collaborative signal semantic vectors, and deeply integrates them with a multi-level dynamic world model through a cross-attention mechanism, thereby achieving the organic unity of data-driven and cognitive-driven approaches, and guiding the world model to generate high-risk, high-difficulty or high-value driving scenarios.

[0017] 4. The human-machine co-driving closed-loop optimization method based on end-to-end fusion with world model provided by this invention generates high-risk and high-value scenarios through multi-level dynamic world model to expand virtual data, and continuously supplements training data through human-machine collaborative signals in real driving process, thereby realizing the long-term evolution capability of autonomous driving system.

[0018] 5. The human-machine co-driving closed-loop optimization method based on end-to-end fusion with world model provided by the present invention predicts the future multi-frame scene state by inputting candidate planning trajectories into a multi-level dynamic world model, evaluates the safety, interaction rationality and risk level of candidate trajectories in long time series, and feeds the evaluation results back to the prediction and planning integrated network for iterative optimization, thereby realizing closed-loop coupled reasoning of prediction, planning and evaluation.

[0019] 6. The human-machine co-driving closed-loop optimization method based on end-to-end and world model fusion provided by this invention constructs a trajectory preference pair dataset by collecting trajectories before and after takeover, trains an implicit reward function model and CPEN preference evaluation network to learn human driving preferences and social value constraints, realizes the social rationality assessment and strategy optimization of the planned trajectory, and improves the model's understanding and adaptation to the driver's driving habits.

[0020] In summary, the technical solution of this invention solves the problems in the prior art where human-machine collaborative signals are not fully utilized, the closed-loop optimization mechanism of the autonomous driving system is imperfect, and the world model scene generation is disconnected from the actual scene and human driving habits.

[0021] Based on the above reasons, this invention can be widely applied in fields such as intelligent driving, human-machine co-driving system development, autonomous driving model testing and verification, and continuous evolution. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of the method of the present invention.

[0024] Figure 2 This is a block diagram of the system structure of the present invention. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "comprising" and "having" and any variations thereof in the specification, claims and accompanying drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product or device.

[0027] like Figure 1 As shown, this invention provides a human-machine co-driving closed-loop optimization method based on end-to-end fusion with a world model, including: S1. Acquire multimodal sensor data, preprocess and align the multimodal sensor data, extract multimodal features and map them to a shared implicit feature space, project them to the bird's-eye view coordinate system after spatiotemporal information interaction and aggregation, and generate a unified representation vector of driving scene based on BEV feature network. S2. Construct a multi-level dynamic world model, and use the generated unified representation vector of the driving scene and the BEV feature grid as scene conditions input to the multi-level dynamic world model to generate visual images of future driving scenes and movement trajectories of traffic participants. S3. Obtain human-machine collaboration signals and expected risk assessment representations, encode the human-machine collaboration signals into collaboration signal semantic vectors, and use cross-attention mechanism to interactively fuse the collaboration signal semantic vectors with the expected risk assessment representations to obtain scene generation constraints. Introduce the scene generation constraints into a multi-level dynamic world model to guide the multi-level dynamic world model to generate driving scenarios in a targeted manner. S4. Construct an end-to-end autonomous driving model based on BEV feature grid. The end-to-end autonomous driving model generates candidate planning trajectories through an integrated prediction and planning network, and iteratively optimizes the candidate planning trajectories through a prediction and planning iterative optimization mechanism coupled with the multi-level dynamic world model, and outputs the optimal planning trajectory. S5. Collect information on human driver takeover behavior, intervention operations, driving preferences, and system active takeover to construct a trajectory preference pair dataset, including trajectory pairs of human-takeover vehicles, system-takeover vehicles, and normal driving trajectories; use the trajectory preference pair dataset to train an implicit reward function model and a preference evaluation network, respectively; use the preference evaluation network to evaluate the social rationality of the optimal planned trajectory and output a preference score; optimize the end-to-end autonomous driving model based on the preference score and the implicit reward function model, and output the optimized planning strategy. S6. Perform real-time multi-dimensional risk assessment on the end-to-end autonomous driving model, and output risk score, risk level and safety constraints; when a risk event is detected, save a full-dimensional state snapshot, use causal Transformer and time-series backtracking algorithm to decompose the risk causes, output standardized cause labels and label confidence level and cause elements. S7. Real-time acquisition of driver takeover timing, driving behavior, driving preferences, attention state, and system active takeover trigger signals. Human-machine interaction semantic vectors are generated by the collaborative signal encoder. The human-machine interaction semantic vectors are input into the conditional embedding layer of the end-to-end autonomous driving model and the multi-level dynamic world model. Based on risk level, standardized cause labels, future scenario prediction, and control transfer direction, graded warnings, takeover suggestions, system takeover notifications, permission adjustments, and interactive guidance are output to the driver to establish a human-machine interaction time sequence record chain.

[0028] In a specific implementation, as a preferred embodiment of the present invention, step S1 includes: S11. Select the forward-looking and surround-view image sequences and the vehicle's historical trajectory data as the multimodal sensor data input source, and perform time alignment and spatial registration on the multimodal sensor data to form a time-consistent multimodal input stream; S12. Preliminary features are extracted from the original data of each modality through their respective dedicated backbone networks, and all modal features are uniformly mapped to a shared implicit feature space through a learnable projection layer. S13. Input the aligned multimodal feature sequence into the spatiotemporal Transformer encoder to perform information interaction and aggregation simultaneously in both time and space dimensions, and output the fused feature tensor. S14. The fused feature tensor is uniformly projected onto the bird's-eye view coordinate system centered on the vehicle through a differentiable view transformation module to generate a discretized, grid-structured BEV feature grid, with each grid encoding the multimodal spatiotemporal semantic information of the corresponding physical region. S15. Apply semantically aware adaptive pooling to the BEV feature grid, use a lightweight semantic segmentation head to perform grid-by-grid semantic category prediction, use a smaller pooling kernel to preserve spatial details for regions containing key dynamic traffic participants, use a larger pooling kernel to compress redundant information for background regions, and output a fixed-dimensional unified representation vector of the driving scene.

[0029] In a specific implementation, as a preferred embodiment of the present invention, step S2 includes: S21. A multi-level dynamic world model is constructed using a decoupled spatiotemporal diffusion network, including a causal Transformer, a diffusion Transformer, a temporal dynamic modeling module, and a spatial detail and behavior generation module, wherein: The causal Transformer models the sequence of historical scene states in an autoregressive manner and extracts temporal causal dependencies; The Diffusion Transformer iteratively generates refined spatial features and behavioral distributions of future states through forward diffusion and backward denoising processes. The temporal dynamic modeling module captures the long-term dependencies of driving scenarios as they evolve over time. The spatial detail and behavior generation module generates high-resolution spatial details and traffic participant behavior; S22. The unified representation vector of the driving scenario and the BEV feature grid are used as scene conditions input to the multi-level dynamic world model. The driving scenario condition information is encoded into a semantic vector through the condition embedding layer. The condition embedding layer includes physical layer information, behavioral layer information and rule layer information, which correspond to the physical kinematic state, the behavioral intention of traffic participants and the scene condition information under traffic rule constraints, respectively. S23. Using VisDiT Visual Diffusion Transformer and TrajDiT Trajectory Diffusion Transformer, the semantic vector, temporal causal dependency, future spatial features, and behavioral distribution are accepted as conditional inputs to generate high-fidelity visual images of future driving scenes and motion trajectories of traffic participants, respectively.

[0030] In a specific implementation, as a preferred embodiment of the present invention, step S3 includes: S31. The driver's takeover timing, intervention operation, and driving preference information are extracted and semantically encoded by a cooperative signal encoder to generate a fixed-dimensional cooperative signal semantic vector; wherein, the cooperative signal encoder includes a temporal coding branch and a behavioral coding branch, which encode continuous temporal operation signals and discrete intervention type signals, respectively. S32. Input the collaborative signal semantic vector and the expected risk assessment representation into the cross-attention layer. In the cross-attention layer, use the expected risk assessment representation as the query vector and the collaborative signal semantic vector as the key vector to calculate the attention weight and perform weighted fusion to generate scene generation constraints that include risk orientation and cognition orientation. S33. The scene generation constraints are injected into the diffusion Transformer of the multi-level dynamic world model through the condition embedding layer. During the reverse denoising process, the scene generation constraints guide the gradual elimination of noise, and generate high-risk, high-difficulty or high-value driving scenarios that match the cooperative signal semantic vector and the expected risk assessment representation.

[0031] In a specific implementation, as a preferred embodiment of the present invention, step S4 includes: S41. Input the BEV feature mesh into the interaction modeling network to generate a graph structure of traffic participants, where nodes represent the state of traffic participants and edges represent interaction relationships; extract interaction features through a graph attention mechanism and fuse them with BEV features to obtain an interaction-enhanced scene representation. S42. Construct an intent latent space through cross-modal contrastive learning and encode the behavioral intent of traffic participants as policy latent variables; S43. Input the interaction enhancement scenario representation and the policy latent variables into the prediction planning integrated network to generate multiple candidate planning trajectories; S44. Construct a multi-objective reward function for the candidate planned trajectories, including safety, smoothness, regularity, task completion, and human-like indicators; dynamically weight different objectives through an attention weight matrix, and introduce conservative regularization constraints to limit policy offset, thereby obtaining an optimized set of candidate trajectories. S45. Based on real driving data, the prediction and planning integrated network is pre-trained by imitation learning, and then trained by offline reinforcement learning with a multi-objective reward function as the optimization objective. S46. Input the candidate planning trajectory and the unified representation of the driving scene into the multi-level dynamic world model to predict the scene state in future multiple frames; S47. Based on the predicted future multi-frame scene state, evaluate the safety, interaction rationality and risk level of the candidate trajectory in the long time series, and feed the evaluation results back to the prediction and planning integrated network to iteratively optimize the trajectory and output the optimal planned trajectory.

[0032] In a specific implementation, as a preferred embodiment of the present invention, step S5 includes: S51. Collect information on human drivers' takeover behaviors, interventions, driving preferences, and system-initiated takeovers during real-world driving processes. Clean and filter the collected information to distinguish between human-led takeover events, system-led takeover events, and normal driving events. For system-led takeover events, retain trajectory segments within a preset time window before and after the system-triggered takeover. For normal driving events, retain reference trajectory segments of equal length. Construct pairs of human-led takeover trajectories containing the system-planned trajectory before takeover and the driver's actual trajectory after takeover, pairs of system-led takeover trajectories containing the driver's trajectory before system-triggered takeover and the system-planned trajectory after takeover, and pairs of normal driving reference trajectories. Perform data augmentation and distribution balancing on the trajectory preference dataset. S52. Based on the trajectory preference dataset, an implicit reward function model is trained using a ranking loss function. The ranking loss function aims to maximize the reward difference between the trajectory after takeover and the trajectory before takeover. It learns the driver's driving preferences, system safety preferences, and social value constraints, and outputs the implicit reward value of the trajectory. S53. Construct a CPEN preference evaluation network based on contrastive learning, including a trajectory feature encoder, a scene context encoder, and a preference score fusion machine. Input the candidate planned trajectory and the current scene features into the trajectory feature encoder and the scene context encoder, respectively, and output the preference score through the preference score fusion machine. S54. Construct a composite optimization objective using the implicit reward value and the preference score. Introduce safety boundary constraints and conservative regularization terms into the composite optimization objective. Perform safety reinforcement learning fine-tuning on the end-to-end autonomous driving model. Introduce safety negative samples, including collision risks and violations, for adversarial training. S55. The elastic weight consolidation method is used to apply secondary penalty constraints to the key network parameters related to social reasoning in the end-to-end autonomous driving model, freeze the basic network parameters that have been trained and converged, and update the relevant parameters to optimize the strategy. S56. Construct an experience replay buffer, prioritize sampling according to scene difficulty and preference intensity, and mix the trajectory preference dataset with the high-value scenes generated by the multi-level dynamic world model in a preset ratio, and use them together to retrain the end-to-end autonomous driving model to output the optimized planning strategy.

[0033] In a specific implementation, as a preferred embodiment of the present invention, step S6 includes: S61. Input the current operating status of the end-to-end autonomous driving model into the risk assessment network, and calculate the scenario complexity risk score, interaction risk score, decision compliance risk score, trajectory safety risk score and takeover urgency risk score respectively. S62. The risk scores for scenario complexity, interaction, decision-making compliance, trajectory safety, and takeover urgency are weighted and integrated to obtain a comprehensive risk score. S63. Map the comprehensive risk score to a risk level according to a preset risk level threshold, and generate safety constraints corresponding to the risk level. S64. Collect the operation data and risk event data of the end-to-end autonomous driving model as the data driving source, collect human driver takeover behavior, intervention operation, driving preference information and system active takeover information, and encode them into cognitive semantic vectors as the cognitive driving source. S65. Input the data-driven source and the cognitive-driven source into the fusion network to generate a human-machine collaborative optimization signal; S66. Monitor the operating status of the end-to-end autonomous driving model and the driver's operation status. When a risk trigger, system anomaly, or driver forced takeover or system active takeover is detected, trigger a full-dimensional state snapshot saving instruction. The full-dimensional state snapshot includes raw sensor data, perception output results, prediction output results, planning output results, control output results, control instruction sequence, vehicle state parameters, driver operation sequence, traffic participant behavior data, and environmental context information. S67. Input the full-dimensional state snapshot into the causal Transformer to extract the state transition relationship and temporal dependency features within the preceding time window of the risk event; locate the key nodes of risk evolution through the temporal backtracking algorithm, and distinguish between system-side causes and driver-side causes; label system-side causes with tags such as perception miss, prediction deviation, unreasonable planning, control delay, or module coordination failure; label driver-side causes with tags such as operational error, lack of concentration, misjudgment of intent, or personal driving style; output standardized cause labels containing cause type, confidence score, and cause elements.

[0034] In a specific implementation, as a preferred embodiment of the present invention, step S7 includes: S71: Real-time data collection of driver takeover timing, driving behavior, driving preferences and attention status is achieved through in-vehicle cameras, steering wheel torque sensors, pedal travel sensors and eye-tracking devices, while monitoring system active takeover trigger signals. S72. The collected driver status signal and system active takeover signal are input into the cooperative signal encoder. The cooperative signal encoder performs sliding window segmentation and feature extraction on the continuous time-series signal, performs one-hot encoding on the discrete event signal, and fuses them to generate a fixed-dimensional human-machine interaction semantic vector. S73. The human-machine interaction semantic vector is input into the policy optimization parameter space of the end-to-end autonomous driving model and the conditional embedding layer of the multi-level dynamic world model through the interface layer, respectively, wherein: In the end-to-end autonomous driving model, the human-machine interaction semantic vector and the scene representation vector are concatenated and then input into the prediction-planning integrated network. In the multi-layered dynamic world model, the human-machine interaction semantic vector is fused with the physical layer, behavioral layer, and rule layer condition vectors to guide the direction of scene generation. S74. Obtain the risk level, the standardized causative label, the future scenario prediction result output by the multi-level dynamic world model, and the current control ownership status, and input them into the human-machine collaborative feedback decision network. The human-machine collaborative feedback decision network matches and outputs content according to a preset hierarchical mapping table. When the risk level is emergency and control is transferred from the system to the human, output a takeover suggestion; when the risk level is warning, output a tiered warning; when the system triggers active takeover, output a system takeover notification; when the cause label points to the system side, output a permission adjustment instruction; when the cause label points to the driver side, output interactive guidance information. S75. The timing of the driver's takeover, driving behavior, driving preferences and attention state are matched one-to-one with the hierarchical warnings, takeover suggestions, permission adjustments and interactive guidance output by the system according to the timestamp, and the data is stored in alignment to establish a human-machine interaction time sequence record chain.

[0035] like Figure 2 As shown, this embodiment of the invention also provides a human-machine co-driving closed-loop optimization system based on the above-mentioned human-machine co-driving closed-loop optimization method based on end-to-end and world model fusion, including: a unified representation module for driving scenarios, a multi-level dynamic world model construction module, a scenario orientation generation module, an end-to-end autonomous driving model construction module, a driver feedback-driven social reasoning optimization module, a real-time risk assessment and causal inversion module, and a human-machine two-way mutual feedback interaction module, wherein: The unified representation module for driving scenarios is used to acquire multimodal sensor data, preprocess and align the multimodal sensor data, extract multimodal features and map them to a shared implicit feature space, project them to the bird's-eye view coordinate system after spatiotemporal information interaction and aggregation, and generate a unified representation vector for driving scenarios based on the BEV feature network. The multi-level dynamic world model construction module is used to construct a multi-level dynamic world model. The generated unified representation vector of the driving scene and the BEV feature grid are used as scene conditions input into the multi-level dynamic world model to generate a visual image of the future driving scene and the movement trajectory of traffic participants. The scene-oriented generation module is used to acquire human-machine collaboration signals and expected risk assessment representations, encode the human-machine collaboration signals into collaboration signal semantic vectors, and interact and fuse the collaboration signal semantic vectors with the expected risk assessment representations through a cross-attention mechanism to obtain scene generation constraints. The scene generation constraints are then introduced into a multi-level dynamic world model to guide the multi-level dynamic world model to generate driving scenarios in an oriented manner. The end-to-end autonomous driving model construction module is used to construct an end-to-end autonomous driving model based on the BEV feature grid. The end-to-end autonomous driving model generates candidate planning trajectories through an integrated prediction and planning network, and iteratively optimizes the candidate planning trajectories through a prediction and planning iterative optimization mechanism coupled with the multi-level dynamic world model, and outputs the optimal planning trajectory. The driver feedback-driven social reasoning optimization module is used to collect human driver takeover behavior, intervention operations, driving preference information, and system active takeover information to construct a trajectory preference pair dataset, including trajectory pairs of human-takeover vehicles, trajectory pairs of system-takeover vehicles, and normal driving trajectory pairs. The implicit reward function model and preference evaluation network are trained using the trajectory preference pair dataset. The preference evaluation network evaluates the social rationality of the optimal planned trajectory and outputs a preference score. Based on the preference score and the implicit reward function model, the end-to-end autonomous driving model is optimized, and the optimized planning strategy is output. The real-time risk assessment and cause inversion module is used to perform real-time multi-dimensional risk assessment on the end-to-end autonomous driving model, and output risk scores, risk levels and safety constraints. When a risk event is detected, a full-dimensional state snapshot is saved, and the causes of the risk are decomposed using causal Transformer and time-series backtracking algorithms. Standardized cause labels are output and confidence and cause elements are marked. The human-machine two-way feedback interaction module is used to collect driver takeover timing, driving behavior, driving preferences, attention state, and system active takeover trigger signals in real time. The collaborative signal encoder generates human-machine feedback semantic vectors, which are then input into the conditional embedding layer of the end-to-end autonomous driving model and the multi-level dynamic world model. Based on risk level, standardized cause labels, future scenario predictions, and control transfer direction, the module outputs graded warnings, takeover suggestions, system takeover notifications, permission adjustments, and interactive guidance to the driver, establishing a human-machine interaction time sequence record chain.

[0036] The embodiments of the present invention are described simply because they correspond to those in the embodiments above. For any similarities, please refer to the descriptions in the embodiments above, which will not be elaborated here.

[0037] Example This embodiment uses the operation scenario of a Level 3 autonomous driving vehicle at a complex urban intersection in rainy weather as an example to illustrate the specific implementation of the human-machine co-driving closed-loop optimization method based on end-to-end and world model fusion described in this invention. The scenario is a two-way six-lane intersection with mixed traffic of motor vehicles, non-motor vehicles, and pedestrians. The road surface is slippery, visibility is moderate, and the vehicle approaches the intersection at a speed of 36 km / h (10 m / s), intending to go straight through. The specific implementation scheme of this invention is as follows: Step 1: Unified representation of driving scenarios The vehicle-mounted sensor array includes a forward-facing camera (1920×1080 resolution, 30 fps), four surround-view cameras (1280×720 resolution, 30 fps), and historical trajectory data from the vehicle's CAN bus (sampling frequency 10 Hz, recording 30 frames of trajectory points over the past 3 seconds). The multimodal data preprocessing unit performs distortion correction, white balance adjustment, and exposure normalization on each image stream, unifies the coordinates of the historical trajectories, removes outliers, and performs hard synchronization based on timestamps to form a temporally consistent multimodal input stream.

[0038] The multimodal feature extraction and mapping unit uses a dedicated backbone network to process each modality of data: multi-scale visual features are extracted from the forward-looking and panoramic images using ResNet-50, and motion features are extracted from the historical trajectory using a one-dimensional convolutional network. Subsequently, a learnable projection layer maps the features of each modality to a unified 256-dimensional shared implicit feature space.

[0039] The spatiotemporal Transformer encoder receives the aligned multimodal feature sequence (16 historical frames, spanning approximately 0.53 seconds). It employs an 8-layer, 16-head attention mechanism to model the dynamic evolution between frames in the temporal dimension and the cross-camera field-of-view correlation in the spatial dimension, outputting a fused feature tensor rich in contextual information (size 16×256×256).

[0040] The microscopic view transformation module projects the fused feature tensor onto a bird's-eye view coordinate system centered on the vehicle, generating a discretized, grid-structured BEV feature grid with a grid size of 200×200. Each grid corresponds to a physical space of 0.5 m×0.5 m, covering a range of 100 m in front of the vehicle and 50 m to the left and right. Each grid encodes the multimodal spatiotemporal semantic information of the corresponding physical region.

[0041] The semantic-aware adaptive pooling unit utilizes a lightweight semantic segmentation head (based on a U-Net encoder-decoder structure) to predict the semantic category of the BEV feature grid grid on a grid-by-grid basis, distinguishing key dynamic traffic participants (vehicles, pedestrians, and cyclists) from the background region. A 3×3 pooling kernel is used for grids containing vehicles to preserve spatial details, while a 7×7 pooling kernel is used for background road regions to compress redundant information. The final output is a unified representation vector of the driving scene with a fixed dimension of 1024.

[0042] Step 2: Construction of a multi-layered dynamic world model The multi-level dynamic world model receives BEV feature grid (200×200×256) and driving scene unified representation vector (1024-dimensional) as scene condition inputs.

[0043] The causal Transformer in the decoupled spatiotemporal diffusion network models the historical 16-frame BEV state sequence in an autoregressive manner. A 12-layer Transformer decoder structure is used to extract the temporal causal dependencies and output the implicit trend code of future state evolution.

[0044] The Diffusion Transformer iteratively generates refined spatial features and behavioral distributions for the next 8 frames (corresponding to the next 4 seconds, with a time interval of 0.5 seconds) through forward diffusion and backward denoising processes. The diffusion step count is set to 100 steps, and the DDPM framework is adopted, with the trend encoding output by the causal Transformer serving as a condition to guide the denoising direction.

[0045] The conditional embedding layer encodes scene conditional information into a 512-dimensional semantic vector, where physical layer information includes the vehicle's current speed of 10. Acceleration 0 Heading angle 0 (North) and road surface adhesion coefficient 0.6 (rainy day); behavioral layer information includes the classification of the behavioral intentions of each traffic participant at the intersection (such as the intention of vehicles on the left to turn left, and the intention of pedestrians on the opposite side to cross); rule layer information includes the current traffic light status (green light, 12 seconds remaining), speed limit sign (60 km / h) and lane guidance arrows.

[0046] The VisDiT Visual Diffusion Transformer accepts semantic vectors, temporal causal dependencies, and future spatial features to generate high-fidelity visual images of future driving scenes (256×256 resolution, 8 frames) for visualization validation and model distillation. The TrajDiT Trajectory Diffusion Transformer generates future motion trajectories for each traffic participant under the same conditions (each frame includes position, speed, and heading) for end-to-end model prediction planning iteration.

[0047] Step 3: Scene orientation generation based on human-machine collaborative signals The collaborative signal encoder collects historical data from the driver's driving over the past 1000 km: the driver performed 12 interventions in TTC (Time To Collision) scenarios with a time to collision (TTC) of less than 2.5 seconds, with braking as the primary intervention (average brake pedal depth of 45%), indicating a conservative driving style. The encoder encodes the intervention timing, intervention type and intensity, and driving preference labels into a 256-dimensional collaborative signal semantic vector.

[0048] The current real-time risk assessment module outputs the expected risk assessment representation: scenario complexity score 0.75, interaction risk score 0.82. The cross-attention mechanism (8 heads, 256 dimensions) interacts and fuses the collaborative signal semantic vector with the expected risk assessment representation to generate scenario generation constraints, specifically "generating a high-risk scenario where a vehicle suddenly cuts into the left lane and the road surface is slippery".

[0049] The scene generation constraints are injected into the diffusion Transformer of the multi-layer dynamic world model through the conditional embedding layer. During the reverse denoising process, the noise elimination direction is constrained, and 10 high-value driving scenarios that meet the constraints are generated in a directional manner for subsequent safety fine-tuning training of the end-to-end model.

[0050] Step 4: End-to-end autonomous driving model construction and predictive planning The end-to-end autonomous driving model is constructed based on a BEV feature grid (200×200×256). The interaction modeling network extracts traffic participants from the BEV feature grid into a graph structure representation: 128-dimensional node features, including the position, speed, and category encoding of each participant; and 64-dimensional edge features, encoding the relative distance and orientation between participants. Interaction features are extracted through a multi-head graph attention mechanism (4 heads) and added element-wise with the BEV features to obtain an interaction-enhanced scene representation.

[0051] The cross-modal contrastive learning unit uses InfoNCE loss as the optimization objective to construct a 64-dimensional intent latent space, encoding the behavioral intents of each traffic participant (such as going straight, turning left, or remaining stationary) as policy latent variables.

[0052] The integrated predictive planning network adopts a Transformer decoder structure, taking interactive enhanced scene representation and policy latent variables as inputs, and generates 5 candidate planning trajectories through autoregression. Each trajectory contains 50 trajectory points for the next 5 seconds (with a time interval of 0.1 seconds), covering five strategies: lane keeping, deceleration and yielding, left-side avoidance, right-side avoidance, and emergency braking.

[0053] A multi-objective reward function scores candidate trajectories based on: safety (based on the minimum distance to the predicted trajectory of other vehicles, weight 0.4), smoothness (based on the rate of change of acceleration, weight 0.2), regularity (based on lane departure and traffic light compliance, weight 0.2), task completion (based on intersection clearance progress, weight 0.1), and human-likeness (based on KL divergence with the driver's historical trajectory distribution, weight 0.1). The optimized candidate trajectory set is obtained by dynamically weighting the trajectories using an attention weight matrix and introducing a conservative regularization term (KL divergence constraint, limiting the offset from the pre-trained policy to less than 0.1).

[0054] The integrated prediction and planning network is first pre-trained using imitation learning (behavior cloning, mean squared error loss) based on 100,000 frames of real driving data. Then, it is trained using the CQL (Conservative Q-Learning) offline reinforcement learning algorithm with a multi-objective reward function as the optimization objective.

[0055] The prediction and planning iterative optimization mechanism inputs five candidate trajectories into a multi-level dynamic world model to predict the scene state over the next eight frames (4 seconds). The multi-level dynamic world model evaluates the safety of each candidate trajectory over long time: Trajectory 1 (lane keeping) has a minimum distance of 1.2 m from the predicted trajectory of the cutting vehicle in the third frame, posing a collision risk; Trajectory 2 (yield) has a minimum distance of 4.5 m, which is safe. The evaluation results are fed back to the integrated prediction and planning network, which eliminates high-risk trajectories, fine-tunes the parameters of the remaining trajectories, and after three iterations of optimization, outputs the optimal planned trajectory as "yield, keep lane".

[0056] Step 5: Driver Feedback-Driven Social Reasoning Optimization The trajectory preference pair dataset construction unit collects driver takeover events during actual driving. Taking a typical event as an example: the system's original planned trajectory was an aggressive left lane change to avoid an obstacle (trajectory before takeover), but the driver actually braked and maintained the lane (trajectory after takeover). Trajectory preference pairs are constructed by capturing 5 seconds before and after the takeover (a total of 10 seconds, 100 trajectory points). The dataset contains 1000 such valid event pairs.

[0057] The implicit reward function model employs a 3-layer MLP (input dimension is the concatenated features of the position, velocity, and heading of the trajectory points, hidden layers 512-256, output scalar reward), with the ranking loss function as the optimization objective (maximizing the reward difference between the trajectory after takeover and the trajectory before takeover), to learn the driver's implicit driving preferences (conservative, courteous).

[0058] The CPEN preference evaluation network consists of a trajectory encoder (LSTM, 128-dimensional hidden state), a scene context encoder (CNN to process BEV features, outputting 256 dimensions), and a preference score fusion (2-layer MLP). It takes the optimal planned trajectory and current scene features as input and outputs a preference score of 0.89 (out of 1.0), indicating that the trajectory conforms to the driver's social preferences.

[0059] The safety reinforcement learning optimization unit constructs a composite optimization objective (weighted summation, each with a weight of 0.5) using implicit reward values ​​and preference scores. It introduces safety boundary constraints (prohibition of intrusion into the oncoming lane) and conservative regularization terms to fine-tune the PPO algorithm on the end-to-end autonomous driving model. At the same time, it introduces 50 sets of safety negative samples (simulated collision scenarios and red-light violation scenarios) for adversarial training to improve the robustness of the model in the intervention scenarios.

[0060] The continuous learning unit employs the Elastic Weight Consolidation (EWC) method, applying a Fisher information matrix-based secondary penalty constraint to the basic feature extraction parameters of the first six layers of the integrated prediction and planning network, freezing their updates; the policy head parameters of the last two layers are then released for fine-tuning. An experience replay buffer (capacity 100,000 records) is constructed, and sampling is prioritized according to scenario difficulty (risk score) and preference strength (reward gap). 700 sets of real data from the trajectory preference dataset are mixed with 300 sets of entry scenarios generated by the multi-level dynamic world model at a 7:3 ratio, and jointly used for retraining the end-to-end autonomous driving model to output an optimized planning policy.

[0061] Step 6: Real-time risk assessment and causal inversion The real-time risk assessment unit evaluates the current operating status of the end-to-end autonomous driving model across five dimensions: scenario complexity 0.75 (multi-vehicle mixed traffic), interaction risk 0.82 (hesitation in the trajectory of the vehicle on the left), decision compliance 0.90 (compliance with green light rules), trajectory safety 0.65 (potential conflict with the predicted trajectory of the vehicle on the left), and takeover urgency 0.40 (no takeover required at present). The weighted fusion results in a comprehensive risk score of 0.74, with a risk level of "warning," generating a safety constraint: maintain a distance of more than 2.5 seconds from the vehicle in front.

[0062] As the vehicle approached the center of the intersection, a vehicle in the left lane suddenly crossed the solid line and cut in front of the vehicle by 2 meters. The system detected a risk trigger (the trajectory safety risk score instantly rose to 0.95) and immediately triggered a full-dimensional state snapshot: saving all data from the previous 5 seconds to the next 2 seconds, including the original image frames (forward and surround view, a total of 210 frames), the BEV feature grid sequence, the other vehicle position sequence output by the perception module, the other vehicle trajectory sequence output by the prediction module, the candidate trajectory of the vehicle output by the planning module, and the control command sequence (accelerator 0%, brake 30%, steering wheel 0%). Vehicle status (speed 10 m / s, longitudinal acceleration -2) ), driver operation sequence (brake pedal depth 50%, T) The system starts at 0.2 seconds, collects traffic participant behavior data (lateral speed of the cutting vehicle is 1.5 m / s), and environmental information (rainy weather, slippery road surface).

[0063] The causal Transformer algorithm and temporal backtracking algorithm decompose the risk causes: Backtracking the previous 3 seconds, the system perception module continuously predicted that the vehicle on the left would maintain its lane (intent classified as "going straight") between T-3 seconds and T-1 seconds, while in reality, the vehicle had already begun to deviate (lateral speed 0.3 m / s, but did not trigger the lane change threshold). The root cause is the perception module's failure to detect the slowly deviating vehicle. The standardized cause label output is: System Cause - Perception Missed Detection, confidence level 0.92, root cause location index is the left rear perception fusion module.

[0064] Step 7: Two-way human-machine interaction The human-machine interaction semantic vector generation unit collects driver status in real time through an in-vehicle camera (eye tracking), steering wheel torque sensor, and pedal travel sensor: eye tracking data shows that the driver's attention is on T. 2 seconds to T During the 0-second period, the focus is on the dashboard (distracted state), the steering wheel grip is relaxed (below the normal threshold by 20%), and the takeover point is T. 0.2 seconds (after the risk occurs). The collaborative signal encoder fuses the above multi-source signals to generate a 256-dimensional human-machine interaction semantic vector.

[0065] Human-machine interaction semantic vectors are injected into the strategy head parameter space of the end-to-end autonomous driving model through the interface layer (adjusting the planning strategy to be more conservative in braking response), and the conditional embedding layer behavior layer information of the multi-level dynamic world model is injected (updating the driver distraction state encoding and guiding subsequent scene generation to focus on the combination of "driver distraction and entry").

[0066] The system's proactive collaborative feedback unit outputs the following to the driver based on the risk level (warning to emergency), standardized cause labels (system cause - perception miss) and the world model's prediction of the next 3 seconds scenario (the vehicle cutting in will decelerate to a stop): graded warning (yellow flashing on the instrument panel, red warning box on the left side of the HUD), takeover suggestion (voice broadcast "Takeover suggestion, vehicle cutting in from the left"), permission adjustment (the system automatically downgrades to L2 level, increases steering wheel torque gain by 50%, and prompts the driver to keep their hands on the steering wheel), and interactive guidance (AR-HUD projects recommended braking areas onto the road surface).

[0067] Human-computer interaction timing record chain establishment unit is stored in timestamp alignment: T The system issues an alert at 0.0 seconds (system feedback), T 0.2 seconds later, the driver begins braking (human input), T The system issues a takeover suggestion (system feedback) after 0.5 seconds. The driver firmly gripped the steering wheel with both hands in 0.8 seconds (human input). The above records were written into the closed-loop evolutionary database in structured format (JSON) to provide data support for subsequent causal inversion, scene-oriented generation, and model retraining.

[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A closed-loop optimization method for human-machine co-driving based on end-to-end fusion with a world model, characterized in that, include: S1. Acquire multimodal sensor data, preprocess and align the multimodal sensor data, extract multimodal features and map them to a shared implicit feature space, project them to the bird's-eye view coordinate system after spatiotemporal information interaction and aggregation, and generate a unified representation vector of driving scene based on BEV feature network. S2. Construct a multi-level dynamic world model, and use the generated unified representation vector of the driving scene and the BEV feature grid as scene conditions input to the multi-level dynamic world model to generate visual images of future driving scenes and movement trajectories of traffic participants. S3. Obtain human-machine collaboration signals and expected risk assessment representations, encode the human-machine collaboration signals into collaboration signal semantic vectors, and use cross-attention mechanism to interactively fuse the collaboration signal semantic vectors with the expected risk assessment representations to obtain scene generation constraints. Introduce the scene generation constraints into a multi-level dynamic world model to guide the multi-level dynamic world model to generate driving scenarios in a targeted manner. S4. Construct an end-to-end autonomous driving model based on BEV feature grid. The end-to-end autonomous driving model generates candidate planning trajectories through an integrated prediction and planning network, and iteratively optimizes the candidate planning trajectories through a prediction and planning iterative optimization mechanism coupled with the multi-level dynamic world model, and outputs the optimal planning trajectory. S5. Collect information on human driver takeover behavior, intervention operations, driving preferences, and system active takeover to construct a trajectory preference pair dataset, including trajectory pairs of human-takeover vehicles, system-takeover vehicles, and normal driving trajectories; use the trajectory preference pair dataset to train an implicit reward function model and a preference evaluation network, respectively; use the preference evaluation network to evaluate the social rationality of the optimal planned trajectory and output a preference score; optimize the end-to-end autonomous driving model based on the preference score and the implicit reward function model, and output the optimized planning strategy. S6. Perform real-time multi-dimensional risk assessment on the end-to-end autonomous driving model and output risk score, risk level and safety constraints. When a risk event is detected, a full-dimensional state snapshot is saved, and the causes of the risk are decomposed using a causal Transformer and a time-series backtracking algorithm. Standardized cause labels are output and confidence and cause elements are marked. S7. Real-time acquisition of driver takeover timing, driving behavior, driving preferences, attention state, and system active takeover trigger signals. Human-machine interaction semantic vectors are generated by the collaborative signal encoder. The human-machine interaction semantic vectors are input into the conditional embedding layer of the end-to-end autonomous driving model and the multi-level dynamic world model. Based on risk level, standardized cause labels, future scenario prediction, and control transfer direction, graded warnings, takeover suggestions, system takeover notifications, permission adjustments, and interactive guidance are output to the driver to establish a human-machine interaction time sequence record chain.

2. The human-machine co-driving closed-loop optimization method based on end-to-end and world model fusion as described in claim 1, characterized in that, Step S1 includes: S11. Select the forward-looking and surround-view image sequences and the vehicle's historical trajectory data as the multimodal sensor data input source, and perform time alignment and spatial registration on the multimodal sensor data to form a time-consistent multimodal input stream; S12. Preliminary features are extracted from the original data of each modality through their respective dedicated backbone networks, and all modal features are uniformly mapped to a shared implicit feature space through a learnable projection layer. S13. Input the aligned multimodal feature sequence into the spatiotemporal Transformer encoder to perform information interaction and aggregation simultaneously in both time and space dimensions, and output the fused feature tensor. S14. The fused feature tensor is uniformly projected onto the bird's-eye view coordinate system centered on the vehicle through a differentiable view transformation module to generate a discretized, grid-structured BEV feature grid, with each grid encoding the multimodal spatiotemporal semantic information of the corresponding physical region. S15. Apply semantically aware adaptive pooling to the BEV feature grid, use a lightweight semantic segmentation head to perform grid-by-grid semantic category prediction, use a smaller pooling kernel to preserve spatial details for regions containing key dynamic traffic participants, use a larger pooling kernel to compress redundant information for background regions, and output a fixed-dimensional unified representation vector of the driving scene.

3. The human-machine co-driving closed-loop optimization method based on end-to-end and world model fusion as described in claim 1, characterized in that, Step S2 includes: S21. A multi-level dynamic world model is constructed using a decoupled spatiotemporal diffusion network, including a causal Transformer, a diffusion Transformer, a temporal dynamic modeling module, and a spatial detail and behavior generation module, wherein: The causal Transformer models the sequence of historical scene states in an autoregressive manner and extracts temporal causal dependencies; The Diffusion Transformer iteratively generates refined spatial features and behavioral distributions of future states through forward diffusion and backward denoising processes. The temporal dynamic modeling module captures the long-term dependencies of driving scenarios as they evolve over time. The spatial detail and behavior generation module generates high-resolution spatial details and traffic participant behavior; S22. The unified representation vector of the driving scenario and the BEV feature grid are used as scene conditions input to the multi-level dynamic world model. The driving scenario condition information is encoded into a semantic vector through the condition embedding layer. The condition embedding layer includes physical layer information, behavioral layer information and rule layer information, which correspond to the physical kinematic state, the behavioral intention of traffic participants and the scene condition information under traffic rule constraints, respectively. S23. Using VisDiT Visual Diffusion Transformer and TrajDiT Trajectory Diffusion Transformer, the semantic vector, temporal causal dependency, future spatial features, and behavioral distribution are accepted as conditional inputs to generate high-fidelity visual images of future driving scenes and motion trajectories of traffic participants, respectively.

4. The human-machine co-driving closed-loop optimization method based on end-to-end and world model fusion as described in claim 1, characterized in that, Step S3 includes: S31. The driver's takeover timing, intervention operation, and driving preference information are extracted and semantically encoded by a cooperative signal encoder to generate a fixed-dimensional cooperative signal semantic vector; wherein, the cooperative signal encoder includes a temporal coding branch and a behavioral coding branch, which encode continuous temporal operation signals and discrete intervention type signals, respectively. S32. Input the collaborative signal semantic vector and the expected risk assessment representation into the cross-attention layer. In the cross-attention layer, use the expected risk assessment representation as the query vector and the collaborative signal semantic vector as the key vector to calculate the attention weight and perform weighted fusion to generate scene generation constraints that include risk orientation and cognition orientation. S33. The scene generation constraints are injected into the diffusion Transformer of the multi-level dynamic world model through the condition embedding layer. During the reverse denoising process, the scene generation constraints guide the gradual elimination of noise, and generate high-risk, high-difficulty or high-value driving scenarios that match the cooperative signal semantic vector and the expected risk assessment representation.

5. The human-machine co-driving closed-loop optimization method based on end-to-end and world model fusion as described in claim 1, characterized in that, Step S4 includes: S41. Input the BEV feature mesh into the interaction modeling network to generate a graph structure of traffic participants, where nodes represent the state of traffic participants and edges represent interaction relationships; extract interaction features through a graph attention mechanism and fuse them with BEV features to obtain an interaction-enhanced scene representation. S42. Construct an intent latent space through cross-modal contrastive learning and encode the behavioral intent of traffic participants as policy latent variables; S43. Input the interaction enhancement scenario representation and the policy latent variables into the prediction planning integrated network to generate multiple candidate planning trajectories; S44. Construct a multi-objective reward function for the candidate planned trajectories, including safety, smoothness, regularity, task completion, and human-like indicators; dynamically weight different objectives through an attention weight matrix, and introduce conservative regularization constraints to limit policy offset, thereby obtaining an optimized set of candidate trajectories. S45. Based on real driving data, the integrated prediction and planning network is pre-trained using imitation learning, and then trained using an offline reinforcement learning method with a multi-objective reward function as the optimization objective. S46. Input the candidate planning trajectory and the unified representation of the driving scene into the multi-level dynamic world model to predict the scene state in future multiple frames; S47. Based on the predicted future multi-frame scene state, evaluate the safety, interaction rationality and risk level of the candidate trajectory in the long time series, and feed the evaluation results back to the prediction and planning integrated network to iteratively optimize the trajectory and output the optimal planned trajectory.

6. The human-machine co-driving closed-loop optimization method based on end-to-end and world model fusion as described in claim 1, characterized in that, Step S5 includes: S51. Collect information on human drivers' takeover behaviors, interventions, driving preferences, and system-initiated takeovers during real-world driving processes. Clean and filter the collected information to distinguish between human-led takeover events, system-led takeover events, and normal driving events. For system-led takeover events, retain trajectory segments within a preset time window before and after the system-triggered takeover. For normal driving events, retain reference trajectory segments of equal length. Construct pairs of human-led takeover trajectories containing the system-planned trajectory before takeover and the driver's actual trajectory after takeover, pairs of system-led takeover trajectories containing the driver's trajectory before system-triggered takeover and the system-planned trajectory after takeover, and pairs of normal driving reference trajectories. Perform data augmentation and distribution balancing on the trajectory preference dataset. S52. Based on the trajectory preference dataset, an implicit reward function model is trained using a ranking loss function. The ranking loss function aims to maximize the reward difference between the trajectory after takeover and the trajectory before takeover. It learns the driver's driving preferences, system safety preferences, and social value constraints, and outputs the implicit reward value of the trajectory. S53. Construct a CPEN preference evaluation network based on contrastive learning, including a trajectory feature encoder, a scene context encoder, and a preference score fusion machine. Input the candidate planned trajectory and the current scene features into the trajectory feature encoder and the scene context encoder, respectively, and output the preference score through the preference score fusion machine. S54. Construct a composite optimization objective using the implicit reward value and the preference score. Introduce safety boundary constraints and conservative regularization terms into the composite optimization objective. Perform safety reinforcement learning fine-tuning on the end-to-end autonomous driving model. Introduce safety negative samples, including collision risks and violations, for adversarial training. S55. The elastic weight consolidation method is used to apply secondary penalty constraints to the key network parameters related to social reasoning in the end-to-end autonomous driving model, freeze the basic network parameters that have been trained and converged, and update the relevant parameters to optimize the strategy. S56. Construct an experience replay buffer, prioritize sampling according to scene difficulty and preference intensity, and mix the trajectory preference dataset with the high-value scenes generated by the multi-level dynamic world model in a preset ratio, and use them together to retrain the end-to-end autonomous driving model to output the optimized planning strategy.

7. The human-machine co-driving closed-loop optimization method based on end-to-end and world model fusion as described in claim 1, characterized in that, Step S6 includes: S61. Input the current operating status of the end-to-end autonomous driving model into the risk assessment network, and calculate the scenario complexity risk score, interaction risk score, decision compliance risk score, trajectory safety risk score and takeover urgency risk score respectively. S62. The risk scores for scenario complexity, interaction, decision-making compliance, trajectory safety, and takeover urgency are weighted and integrated to obtain a comprehensive risk score. S63. Map the comprehensive risk score to a risk level according to the preset risk level threshold, and generate safety constraints corresponding to the risk level. S64. Collect the operation data and risk event data of the end-to-end autonomous driving model as the data driving source, collect human driver takeover behavior, intervention operation, driving preference information and system active takeover information, and encode them into cognitive semantic vectors as the cognitive driving source. S65. Input the data-driven source and the cognitive-driven source into the fusion network to generate a human-machine collaborative optimization signal; S66. Monitor the operating status of the end-to-end autonomous driving model and the driver's operation status. When a risk trigger, system anomaly, or driver forced takeover or system active takeover is detected, trigger a full-dimensional state snapshot saving instruction. The full-dimensional state snapshot includes raw sensor data, perception output results, prediction output results, planning output results, control output results, control instruction sequence, vehicle state parameters, driver operation sequence, traffic participant behavior data, and environmental context information. S67. Input the full-dimensional state snapshot into the causal Transformer to extract the state transition relationship and temporal dependency features within the preceding time window of the risk event; locate the key nodes of risk evolution through the temporal backtracking algorithm, and distinguish between system-side causes and driver-side causes; label system-side causes with tags such as perception miss, prediction deviation, unreasonable planning, control delay, or module coordination failure; label driver-side causes with tags such as operational error, lack of concentration, misjudgment of intent, or personal driving style; output standardized cause labels containing cause type, confidence score, and cause elements.

8. The human-machine co-driving closed-loop optimization method based on end-to-end and world model fusion as described in claim 1, characterized in that, Step S7 includes: S71: Real-time data collection of driver takeover timing, driving behavior, driving preferences and attention status is achieved through in-vehicle cameras, steering wheel torque sensors, pedal travel sensors and eye-tracking devices, while monitoring system active takeover trigger signals. S72. The collected driver status signal and system active takeover signal are input into the cooperative signal encoder. The cooperative signal encoder performs sliding window segmentation and feature extraction on the continuous time-series signal, performs one-hot encoding on the discrete event signal, and fuses them to generate a fixed-dimensional human-machine interaction semantic vector. S73. The human-machine interaction semantic vector is input into the policy optimization parameter space of the end-to-end autonomous driving model and the conditional embedding layer of the multi-level dynamic world model through the interface layer, respectively, wherein: In the end-to-end autonomous driving model, the human-machine interaction semantic vector and the scene representation vector are concatenated and then input into the prediction-planning integrated network. In the multi-layered dynamic world model, the human-machine interaction semantic vector is fused with the physical layer, behavioral layer, and rule layer condition vectors to guide the direction of scene generation. S74. Obtain the risk level, the standardized causative label, the future scenario prediction result output by the multi-level dynamic world model, and the current control ownership status, and input them into the human-machine collaborative feedback decision network. The human-machine collaborative feedback decision network matches and outputs content according to a preset hierarchical mapping table. When the risk level is emergency and control is transferred from the system to the human, output a takeover suggestion; when the risk level is warning, output a tiered warning; when the system triggers active takeover, output a system takeover notification; when the cause label points to the system side, output a permission adjustment instruction; when the cause label points to the driver side, output interactive guidance information. S75. The timing of the driver's takeover, driving behavior, driving preferences and attention state are matched one-to-one with the hierarchical warnings, takeover suggestions, permission adjustments and interactive guidance output by the system according to the timestamp, and the data is stored in alignment to establish a human-machine interaction time sequence record chain.

9. A human-machine co-driving closed-loop optimization system based on end-to-end and world model fusion, implemented according to the human-machine co-driving closed-loop optimization method based on end-to-end and world model fusion as described in any one of claims 1-8, characterized in that, include: The system comprises the following modules: a unified representation module for driving scenarios, a multi-level dynamic world model construction module, a scenario-oriented generation module, an end-to-end autonomous driving model construction module, a driver feedback-driven social reasoning optimization module, a real-time risk assessment and causal inversion module, and a human-machine two-way feedback interaction module. The unified representation module for driving scenarios is used to acquire multimodal sensor data, preprocess and align the multimodal sensor data, extract multimodal features and map them to a shared implicit feature space, project them to the bird's-eye view coordinate system after spatiotemporal information interaction and aggregation, and generate a unified representation vector for driving scenarios based on the BEV feature network. The multi-level dynamic world model construction module is used to construct a multi-level dynamic world model. The generated unified representation vector of the driving scene and the BEV feature grid are used as scene conditions input into the multi-level dynamic world model to generate a visual image of the future driving scene and the movement trajectory of traffic participants. The scene-oriented generation module is used to acquire human-machine collaboration signals and expected risk assessment representations, encode the human-machine collaboration signals into collaboration signal semantic vectors, and interact and fuse the collaboration signal semantic vectors with the expected risk assessment representations through a cross-attention mechanism to obtain scene generation constraints. The scene generation constraints are then introduced into a multi-level dynamic world model to guide the multi-level dynamic world model to generate driving scenarios in an oriented manner. The end-to-end autonomous driving model construction module is used to construct an end-to-end autonomous driving model based on the BEV feature grid. The end-to-end autonomous driving model generates candidate planning trajectories through an integrated prediction and planning network, and iteratively optimizes the candidate planning trajectories through a prediction and planning iterative optimization mechanism coupled with the multi-level dynamic world model, and outputs the optimal planning trajectory. The driver feedback-driven social reasoning optimization module is used to collect human driver takeover behavior, intervention operations, driving preference information, and system active takeover information to construct a trajectory preference pair dataset, including trajectory pairs of human-takeover vehicles, trajectory pairs of system-takeover vehicles, and normal driving trajectory pairs. The implicit reward function model and preference evaluation network are trained using the trajectory preference pair dataset. The preference evaluation network evaluates the social rationality of the optimal planned trajectory and outputs a preference score. Based on the preference score and the implicit reward function model, the end-to-end autonomous driving model is optimized, and the optimized planning strategy is output. The real-time risk assessment and cause inversion module is used to perform real-time multi-dimensional risk assessment on the end-to-end autonomous driving model, and output risk scores, risk levels and safety constraints. When a risk event is detected, a full-dimensional state snapshot is saved, and the causes of the risk are decomposed using causal Transformer and time-series backtracking algorithms. Standardized cause labels are output and confidence and cause elements are marked. The human-machine two-way feedback interaction module is used to collect driver takeover timing, driving behavior, driving preferences, attention state, and system active takeover trigger signals in real time. The collaborative signal encoder generates human-machine feedback semantic vectors, which are then input into the conditional embedding layer of the end-to-end autonomous driving model and the multi-level dynamic world model. Based on risk level, standardized cause labels, future scenario predictions, and control transfer direction, the module outputs graded warnings, takeover suggestions, system takeover notifications, permission adjustments, and interactive guidance to the driver, establishing a human-machine interaction time sequence record chain.

Images