End-to-end autonomous driving method and system based on key frame perception and experience routing

Abstract

The application discloses an end-to-end automatic driving method and system based on key frame perception and experience routing, and the method comprises the following steps: acquiring multi-view images and self-vehicle state information, extracting features and cross-modal fusion to obtain current joint features, and storing the current joint features into a history cache; calculating comprehensive correlation scores of the history joint features, selecting key frames, and time sequence fusion of the current joint features to obtain time sequence fusion representation; based on the time sequence fusion representation, matching long-term experience memory of each expert module to activate part of experts to generate and fuse candidate trajectories, and outputting a future driving trajectory. By explicitly screening key frames and routing experts based on long-term experience memory, the stability and explainability of trajectory planning in complex scenes are improved, and the problem of inconsistent decision-making caused by history information redundancy and lack of long-term constraints in routing is solved.

Description

Technical Field

[0001] This invention relates to the field of motion planning and control technology for autonomous vehicles, and in particular to an end-to-end autonomous driving method and system based on keyframe perception and experience routing. Background Technology

[0002] End-to-end autonomous driving technology directly maps sensor inputs to future trajectories or vehicle control commands through a unified model. Compared to traditional modular solutions, this reduces human interface errors between perception, prediction, planning, and control, and learns implicit driving patterns in complex traffic scenarios driven by large-scale data. With the development of multi-view visual perception, Transformer temporal modeling, trajectory decoders, and hybrid expert networks, end-to-end autonomous driving models have made significant progress in open road scenarios, closed-loop simulation, and real-world road prediction tasks. However, road driving is not a single-frame static recognition problem, but a temporal decision-making problem involving long-term dependencies, phase switching, and multi-agent interactions. Vehicles need to make stable plans by integrating current observations, historical dynamic changes, and long-term driving experience in scenarios such as ramp merging, narrow curves, sudden braking of the vehicle in front, yielding, overtaking, and traffic sign responses.

[0003] Existing end-to-end autonomous driving methods mainly employ the following approaches regarding historical information utilization and decision-making mechanisms. The first approach uses methods such as stacking historical frames, recurrent neural networks, or temporal attention to model historical information. While this approach can capture certain temporal dependencies, it typically assigns approximately the same importance to all historical frames, lacks explicit identification of key driving events, and easily mixes redundant frames, noisy frames, and historical frames that truly influence current driving decisions, leading to attention distraction, reaction delays, or control jitter during long-term inference processes. The second approach uses Transformer or diffusion models for temporal trajectory planning, which can model dependencies over a longer time span, but it has high computational costs, and the importance of frames and the basis for decision-making are not intuitive enough, making it difficult to explain why a particular historical frame affects the current planning. The third type of approach employs a hybrid expert network, using a conditional computation mechanism to assign different driving scenarios or behaviors to different expert modules, thereby improving model capacity and functional division of labor. However, the routing network often relies solely on lightweight gating networks and short-term scenario features for expert selection, lacking long-term empirical constraints and clear temporal basis in the routing process. In phased scenarios such as continuous turns, merging, and braking, expert activation may frequently jump, resulting in uneven planned trajectories or inconsistent decisions. Other approaches introduce supervisory signals to constrain expert routing, relying on additional annotations to distinguish driving behaviors or scenario categories. This can improve expert division of labor to some extent, but in open road environments, it is highly dependent on labeled data, limiting generalization and online adaptability.

[0004] In summary, existing technologies suffer from the following shortcomings: Regarding the utilization of historical information, there is a lack of explicit filtering mechanisms at the keyframe level, resulting in insufficient stability for long-term inference; in terms of temporal perception and expert routing, the two are often disconnected, with expert selection failing to fully utilize historical key events and long-term driving experience, making it difficult to maintain decision consistency across time periods; regarding the stability of expert activation, traditional gating networks have limited ability to distinguish complex driving semantics, expert activation lacks traceable evidence, and is prone to frequent jumps during driving phase transitions; furthermore, some solutions heavily rely on additional annotations, increasing data preparation costs and weakening the model's adaptability in unannotated or long-tailed scenarios. These shortcomings lead to potential delays in braking, oversteering, lane departure, or collision risks in safety-critical scenarios such as ramp merging, narrow turns, and emergency braking, making it difficult to balance planning accuracy, control stability, and inference efficiency. Therefore, there is an urgent need for an end-to-end autonomous driving technology capable of simultaneously characterizing key historical events and long-term expert experience, enabling models to achieve stable, interpretable, and efficient trajectory planning in complex dynamic scenarios. Summary of the Invention

[0005] The purpose of this invention is to provide an end-to-end autonomous driving method and system based on keyframe perception and experience routing. By explicitly selecting the keyframes most relevant to the current driving decision and performing temporal fusion, and then achieving expert routing based on long-term experience memory matching, the stability and interpretability of trajectory planning in complex dynamic scenarios are improved. This solves the problem of decision inconsistency caused by historical information redundancy and the lack of long-term experience constraints in expert routing in existing end-to-end autonomous driving methods.

[0006] To solve the above-mentioned technical problems, the first aspect of the present invention provides an end-to-end autonomous driving method based on keyframe perception and experience routing. Step S1: Obtain multi-view images of the vehicle and vehicle state information at the current moment, extract spatial visual features and vehicle state semantic features and perform cross-modal fusion to obtain current joint features, and store the current joint features in a historical feature cache, wherein the historical feature cache is used to save historical joint features at the most recent sampling moments. Step S2: For each historical joint feature in the historical feature cache, calculate the comprehensive relevance score, and select a preset number L of key frames according to the comprehensive relevance score from high to low. Perform temporal fusion of the key frames and the current joint feature to obtain the temporal fusion representation. Step S3: Based on the temporal fusion representation, activate K expert modules from a preset set of N expert modules using an experience routing mechanism to generate candidate future trajectories, and then fuse and output the vehicle's future driving trajectory; wherein, the experience routing mechanism matches the temporal fusion representation with the long-term experience memory of each expert module to determine the activated expert module, K. <N。

[0007] Further, the calculation of the comprehensive relevance score in step S2 includes: Step S21: Input the historical joint features into the frame quality evaluation network to obtain the frame quality score, wherein the frame quality score characterizes the information reliability of the historical joint features themselves; Step S22: Calculate the scaled dot product attention weights between the historical joint features and the current joint features, and pool the attention heads to obtain a decision relevance score, which represents the degree of decision association between the historical joint features and the current joint features; Step S23: Based on the time interval between the historical moment and the current moment, calculate the time proximity score using exponential decay. The time proximity score characterizes the timeliness of the historical joint features. Step S24: The frame quality score, the decision relevance score, and the time proximity score are weighted and summed to obtain the comprehensive relevance score; The selection of a preset number L of keyframes is described above, where the value of the preset number L is positively correlated with the current vehicle speed.

[0008] Furthermore, the frame quality evaluation network, the calculation of the scaling dot product attention weights, the calculation of the exponential decay, and the weight coefficients of the weighted summation in step S24 are all obtained through end-to-end training. The end-to-end training includes the following steps: Step S201: Obtain driving demonstration samples, which include a multi-frame, multi-view image sequence arranged in chronological order, a vehicle state sequence, a target position sequence, and the corresponding expert driving trajectory ground truth. Step S202: Divide the driving demonstration sample into multiple training segments according to the time window. Each training segment takes the current time frame as input and the ground truth value of the expert driving trajectory at a future time as the supervision signal. Step S203: For each training segment, perform forward propagation according to steps S1 to S3 to obtain the predicted future driving trajectory of the vehicle; Step S204: Calculate the trajectory prediction loss, which includes at least the position error between the predicted future vehicle trajectory and the true value of the expert driving trajectory; Step S205: Based on the trajectory prediction loss, synchronously update the frame quality evaluation network, the calculation parameters of the scaling dot product attention weights, the time decay coefficient in the exponential decay calculation, and the weight coefficients of the weighted summation through backpropagation.

[0009] Furthermore, the empirical routing mechanism described in step S3 includes a routing phase and a trajectory generation phase; The routing phase involves matching and evaluating all N expert modules to obtain the routing weight of each expert module. In the trajectory generation phase, only K expert modules are activated to participate in trajectory generation based on the routing weights, where K <N; The value of the number K of activated expert modules is positively correlated with the scene complexity.

[0010] Furthermore, the routing phase includes: Step S311: Calculate the scaled dot product similarity between the query vector of the temporal fusion representation and the key vector of the long-term experience memory of each expert module to obtain the matching score of each expert module; Step S312: Normalize the matching scores of each expert module to obtain the routing weight of each expert module.

[0011] Furthermore, the trajectory generation stage includes: Step S321: Sort the N expert modules in descending order of their routing weights; Step S322: Select the K expert modules with the highest routing weights as the activated expert set; Step S323: Each expert module in the activated expert set generates a candidate future trajectory.

[0012] Further, in step S323, each expert module in the activated expert set generates candidate future trajectories, including: Step S3231: Input the temporal fusion representation and the expert module's own long-term experience memory into the gating network to obtain gating coefficients, wherein the gating coefficients represent the strength of absorbing experience from long-term experience memory; Step S3232: Based on the gating coefficients, perform gating fusion on the temporal fusion representation and the long-term experience memory to obtain the expert-enhanced representation; Step S3233: Input the expert-enhanced representation into the trajectory decoder and decode it to generate the candidate future trajectory corresponding to the expert module.

[0013] Furthermore, after fusing and outputting the vehicle's future driving trajectory in step S3, it also includes: Step S331: Determine the expert module with the highest matching score in the experience routing mechanism; Step S332: Perform a momentum update on the long-term experience memory of the expert module with the highest matching score, while keeping the long-term experience memory of the other expert modules unchanged. The momentum update is defined by the following formula: ; in, To match the long-term experience memory of the expert module with the highest score. This is the temporal fusion representation. The preset momentum coefficient, This is the projection function.

[0014] Furthermore, before activating K expert modules from a preset N expert modules to generate candidate future trajectories based on the temporal fusion representation in step S3 using an empirical routing mechanism, the method further includes: Step S301: Obtain target location or navigation intent information; Step S302: Based on the target location or navigation intent information, perform feature linear modulation on the temporal fusion representation to obtain channel-level scaling parameters and channel-level offset parameters; Step S303: Using the channel-level scaling parameter and the channel-level offset parameter, the temporal fusion representation is scaled and offset channel by channel to obtain the target perception temporal features; Step S304: Replace the temporal fusion representation with the target-aware temporal features and input them into the empirical routing mechanism for expert module matching.

[0015] Accordingly, a second aspect of the present invention provides an end-to-end autonomous driving system based on keyframe perception and experience-based routing, which plans the future trajectory of the vehicle based on the aforementioned end-to-end autonomous driving method based on keyframe perception and experience-based routing, including: The perceptual coding and feature caching module is used to acquire multi-view images of the vehicle and vehicle status information at the current moment, extract spatial visual features and vehicle status semantic features and perform cross-modal fusion to obtain the current joint features, and store the current joint features in the historical feature cache, which is used to save the historical joint features of the most recent sampling moments. The keyframe selection and temporal fusion module is used to calculate the comprehensive relevance score for each historical joint feature in the historical feature cache, and select a preset number L of keyframes according to the comprehensive relevance score from high to low, and perform temporal fusion of the keyframes with the current joint feature to obtain the temporal fusion representation. An experience-based routing and trajectory generation module is used to activate K expert modules from a preset pool of N expert modules based on the temporal fusion representation, generate candidate future trajectories through an experience-based routing mechanism, and fuse and output the future driving trajectory of the vehicle. The experience-based routing mechanism matches the temporal fusion representation with the long-term experience memory of each expert module to determine the activated expert module. <N。

[0016] Accordingly, a third aspect of the present invention provides an electronic device, including: at least one processor; and a memory connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform the above-described end-to-end autonomous driving method based on keyframe perception and experience routing.

[0017] Accordingly, a fourth aspect of the present invention provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the above-described end-to-end autonomous driving method based on keyframe perception and experience routing.

[0018] The above-described technical solutions of the embodiments of the present invention have the following beneficial technical effects: 1. By constructing a multi-dimensional keyframe screening mechanism that integrates frame quality scoring, decision relevance scoring, and temporal proximity scoring, the model can automatically identify and extract the key moments most relevant to the current driving decision from the historical feature cache, effectively filtering out the interference of redundant and noisy frames. Compared with the existing technology of simply superimposing historical frames or relying on a single attention weight, this multi-dimensional scoring system makes the long-term inference process more focused on key driving events such as lane changing, turning, and emergency braking, significantly improving the accuracy of temporal perception and the foresight of trajectory planning, thereby achieving earlier and more stable decision responses in complex dynamic scenarios. 2. Through a full evaluation and sparse activation experience-driven expert routing architecture, the temporal fusion representation is matched with the long-term experience memory of all expert modules, and only a small number of the most relevant experts are activated to participate in trajectory generation. This mechanism enables expert selection to rely on both the current temporal scenario and long-term accumulated driving experience, overcoming the problem of frequent jumps in expert activation caused by traditional gating networks that rely solely on short-term features for routing. In staged driving scenarios such as continuous turns, ramp merging, and braking, the consistency of expert division of labor can be maintained, and the output future driving trajectory is smoother, more coherent, and in line with driving semantic logic. 3. By employing a strategy of updating the experience memory of only the expert module with the highest matching score using momentum, each expert's long-term experience memory is always defined and updated by the scenario type they are best at handling. This effectively prevents the collapse of expert functions caused by mutual contamination of different expert memories during online learning. This mechanism enables the model to gradually absorb new scenario experience during continuous inference, while maintaining the functional differences and expertise stability between different expert modules in the long term. As a result, it has good online adaptability and continuous evolution capabilities in open road environments and long-tail scenarios, balancing planning safety and system maintainability. Attached Figure Description

[0019] Figure 1This is a flowchart of an end-to-end autonomous driving method based on keyframe perception and experience routing provided in an embodiment of the present invention. Figure 2 This is a block diagram of an end-to-end autonomous driving system based on keyframe perception and experience routing provided in an embodiment of the present invention.

[0020] Figure label: 1. Perceptual encoding and feature caching module; 2. Keyframe selection and temporal fusion module; 3. Experience routing and trajectory generation module. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and the accompanying drawings. It should be understood that these descriptions are merely exemplary and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0022] The end-to-end autonomous driving method based on keyframe perception and experience-based routing proposed in this invention is mainly applied to intelligent vehicles equipped with multi-view cameras and onboard computing platforms, targeting various complex and dynamic driving scenarios such as urban open roads, highway ramps, narrow curves, traffic light intersections, and multi-vehicle merging. In these scenarios, the vehicle's driving decisions are not determined by a single static image, but rather require a comprehensive understanding of the current driving situation by integrating changes in road structure over several past moments, the movement trends of surrounding traffic participants, and the evolution of the vehicle's own motion state. For example, in ramp merging scenarios, the vehicle needs to continuously observe the relative positions and speed changes of vehicles in adjacent lanes to determine the timing of the merging window; in narrow curve scenarios, the vehicle needs to adjust its steering angle and speed according to the gradual changes in the curve's curvature; and at traffic light intersections, the vehicle needs to begin paying attention to the switching of traffic light states and the deceleration behavior of vehicles ahead from a considerable distance. Existing end-to-end autonomous driving methods, when handling such scenarios, typically feed consecutive frames of historical information into the temporal model indiscriminately, or rely solely on short-term features for expert module scheduling. This leads to redundant and noisy frames interfering with the current planning, frequent jumps in expert activation during driving phase transitions, and issues such as jitter, delays, or inconsistencies with navigation intent in the output driving trajectory. To address these problems, this invention proposes a technical solution that combines an explicit keyframe filtering mechanism with long-term experience-driven expert routing. This enables the model to automatically identify the most valuable key moments for the current decision from the historical frame sequence and stably select appropriate expert modules to generate trajectories based on long-term accumulated driving experience, thereby achieving more stable, coherent, and interpretable autonomous driving trajectory planning in complex dynamic scenarios.

[0023] Please refer to Figure 1The first aspect of this invention provides an end-to-end autonomous driving method based on keyframe perception and experience-based routing, comprising the following steps: Step S1: Obtain multi-view images of the vehicle and its status information at the current moment, extract spatial visual features and vehicle status semantic features and perform cross-modal fusion to obtain the current joint features, and store the current joint features in the historical feature cache. The historical feature cache is used to save the historical joint features of the most recent sampling moments.

[0024] In practice, the vehicle can simultaneously acquire images from multiple directions, including front, rear, left front, and right front views, through multiple cameras deployed at different locations on the vehicle body, forming a multi-view image set for the current moment. Simultaneously, the vehicle obtains its own status information via the onboard bus or inertial measurement unit, including current speed, heading angle, steering angle, acceleration, and target point position or navigation instructions obtained from the navigation module.

[0025] The aforementioned multi-view images and vehicle status information can be represented by the following symbolic forms. Specifically, ,in, This is a set of multi-view images at the current time t. These are images captured by cameras 1 through C at time t, where C is the total number of cameras. Let be the vehicle's state vector at the current time t. Current vehicle speed For heading angle, For steering angle, For acceleration, For destination point or navigation intent information.

[0026] Spatial visual feature extraction can employ convolutional neural networks or visual encoding networks such as the Visual Transformer to process multi-view images individually or jointly, obtaining multi-scale spatial feature representations containing scene elements such as road structure, lane lines, traffic signs, surrounding vehicles, and pedestrians. Vehicle state semantic features are encoded into low-dimensional semantic vectors compatible with visual feature dimensions through a state encoder, translating numerical state variables such as vehicle speed, heading, and acceleration into semantic vectors. Cross-modal fusion interactively fuses visual features and state semantic features across feature channels or spatial dimensions. Methods such as linear projection after concatenation, element-wise feature addition, or cross-attention can be used to ensure that the two types of heterogeneous information complement each other in a unified feature space, forming a current joint feature that contains both rich scene appearance information and vehicle motion state context information.

[0027] Specifically, ; ; in, Let be the spatial visual features at time t. For visual encoders, Let be the semantic features of the vehicle's state at time t. For state encoders, Let be the current joint feature at time t. This is a cross-modal fusion function.

[0028] The historical feature cache adopts a first-in-first-out queue structure to store the joint features of the most recent sampling times.

[0029] Specifically, ;in, This is a cache of historical features at time t, i.e., a sequence of historical joint features. The joint features of each sampling time from time t-K_1+1 to time t-1, The capacity of the historical feature cache is the number of most recently sampled frames stored.

[0030] For example, in low-speed urban scenarios, features from the most recent 15 to 20 sampled frames can be saved. In high-speed scenarios, the buffer length can be appropriately increased to cover a longer time window. The sampling interval of the buffer can be consistent with the frame rate of the perception system or downsampled at fixed time intervals to balance storage overhead and time coverage. In ramp merging scenarios, the current joint features not only include the road geometry of the lane where the vehicle is located and information about vehicles ahead, but also introduce the vehicle's speed and heading trend through the vehicle's state semantic features, enabling subsequent steps to perceive the vehicle's motion intention while perceiving the appearance of the scene.

[0031] Step S2: Calculate the comprehensive relevance score for each historical joint feature in the historical feature cache, and select a preset number L of key frames according to the comprehensive relevance score from high to low. Perform temporal fusion of the key frames with the current joint feature to obtain the temporal fusion representation.

[0032] The comprehensive relevance score is calculated to measure the contribution of each historical frame to the current driving decision from multiple dimensions. These dimensions include an assessment of the reliability of the historical frame's information, a measurement of the decision-making correlation between the historical frame and the current frame, and the time decay reflected by the time interval between the historical frame and the current moment. Information reliability assessment focuses on whether the historical frame contains clearly identifiable road elements and traffic participant information. For example, historical frames that are severely occluded or have significant motion blur have lower information reliability and can be scored by mapping these joint features using a lightweight frame quality assessment network. The decision-making correlation measures the degree of matching between the scene configuration, relative positional relationships of traffic participants, and vehicle motion state in the historical frame and the current frame. The closer the semantics of these elements are in the feature space, the higher the reference value of the historical frame for the current decision. The time decay term assigns lower weights to overly old historical frames, and the decay rate is controlled by a time decay coefficient. The scores from the three dimensions are combined according to their respective weight coefficients to form the comprehensive relevance score for each frame. Based on this score from highest to lowest, the system selects the L highest-scoring historical frames from the historical cache as keyframes. In narrow curve scenarios, the keyframe selection module tends to retain frames when the vehicle begins to decelerate and turn before entering the curve, frames at the point of maximum curvature in the curve, and frames showing the gradual unfolding of the road structure as the vehicle exits the curve. These frames contain crucial information about changes in road curvature and vehicle attitude adjustments, while redundant frames during the smooth driving phase in the middle of the curve are naturally filtered out. In high-speed merging scenarios, keyframes may be selected when the vehicle approaches the merging point, when the relative distance between the two vehicles reaches its minimum, and when the vehicle makes a decision to yield or accelerate. The temporal fusion step weights the selected L keyframe features with the current joint features. The fusion weight of each keyframe can be determined by its comprehensive relevance score after normalization, ensuring that keyframes that contribute more to the current decision occupy a more important position in the fused representation. The fusion process can use a hierarchical fusion network to integrate the information of keyframes and the current frame layer by layer, ultimately outputting a temporal fusion representation that simultaneously includes the current observation and key historical context. Since the fusion process uses a small number of filtered key frames instead of all historical frames as input, it effectively reduces the interference of noisy and redundant frames on the current planning, and improves the focus and stability of time series modeling.

[0033] Step S3: Based on the temporal fusion representation, activate K expert modules from a preset N expert modules using an experience routing mechanism to generate candidate future trajectories, and then fuse and output the vehicle's future driving trajectory. The experience routing mechanism matches the temporal fusion representation with the long-term experience memory of each expert module to determine the activated expert module. <N。

[0034] The system pre-configures N expert modules, each corresponding to a long-term experience memory. This memory vector or structure stores the expert's prior knowledge and strategy preferences for the types of driving scenarios they excel at handling. An experience routing mechanism matches the temporal fusion representation with the long-term experience memories of each expert module, calculating a matching score between the temporal fusion representation and each expert memory. The matching score reflects the degree of consistency between the current scenario state and the accumulated experience of each expert; a higher score indicates that the expert is more suitable for handling the current driving situation. After determining the routing weights of each expert based on the matching scores, the system selects the K expert modules with the highest weights as the activated expert set, where K is less than the total number of experts N, ensuring that only a few experts are activated in each inference, maintaining model capacity while controlling computational overhead. Each activated expert module fuses information based on its own long-term experience memory and temporal fusion representation to generate an expert-enhanced representation from that expert's perspective. Then, the trajectory decoder corresponding to that expert autoregressively generates a sequence of candidate trajectory waypoints for several future time points, forming the candidate future trajectory for that expert. In traffic light and sign response scenarios, when a vehicle approaches an intersection, the temporal fusion representation includes key information such as the change in traffic light status from blurry to clear, the stop line entering the field of vision from a distance, and the vehicle's gradual deceleration. The experience-based routing mechanism matches this information with the memories of various experts, allowing experts skilled in handling different navigation intentions such as "going straight at the intersection," "turning left at the intersection," or "turning right at the intersection" to obtain different matching scores. Based on the current navigation command, the corresponding expert module is activated to generate a driving trajectory that conforms to traffic rules and navigation objectives. In ramp merging interaction scenarios, the vehicle needs to determine whether adjacent vehicles are entering its lane. The temporal fusion representation includes key moment information such as the approaching vehicle from the adjacent lane, the formation of the merging window, and the vehicle's speed adjustment. Based on this, the experience-based routing mechanism activates expert modules skilled in handling "yielding" or "following" strategies, generating a safe and reasonable deceleration-yielding or acceleration-following trajectory. The system weights and fuses multiple candidate future trajectories output by the activation experts according to their respective routing weights to obtain a comprehensive trajectory. This trajectory is then smoothed and kinematically feasible corrected by the trajectory head network, ultimately outputting the waypoint sequence for the vehicle over several future time steps. This waypoint sequence can be directly provided to the vehicle's lower-level controller, which then converts it into specific control commands for throttle, braking, and steering.

[0035] This invention constructs a complete decision-making chain from multi-dimensional keyframe screening to long-term experience memory matching, enabling the model to automatically focus on the key moments most relevant to the current driving decision from historical feature caches. This avoids interference from redundant historical frames in planning. At the same time, the activation process of the expert module depends on both the current temporal scenario and long-term accumulated driving experience. In complex dynamic scenarios such as continuous turns, ramp merging, emergency braking, and traffic sign responses, it can maintain the consistency of the expert division of labor and the smoothness of the output trajectory. Thus, while ensuring inference efficiency, it significantly improves the planning stability and decision interpretability of the end-to-end autonomous driving system in safety-critical scenarios.

[0036] Specifically, step S2 calculates the comprehensive relevance score, including: Step S21: Input the historical joint features into the frame quality evaluation network to obtain the frame quality score. The frame quality score represents the information reliability of the historical joint features themselves.

[0037] The frame quality evaluation network is a lightweight discriminative network. In practice, it can be designed as a multi-layer fully connected network or a small convolutional network structure. Its input is the joint features of a single historical time step, and its output is a scalar score. Specifically, its calculation method is as follows: ; in, Rate the frame quality of historical moment i. For frame quality evaluation network, Let i be the joint feature of historical moment i.

[0038] The training objective of this network is to learn to determine whether the joint features of a frame contain clear, complete, and helpful scene information for driving decisions. During actual road driving, factors such as vehicle bumps, camera shake, rain, snow, or strong sunlight can cause motion blur, partial occlusion, or abnormal exposure in some historical frames, leading to a decrease in the quality of the corresponding visual features. The frame quality evaluation network can automatically identify such low-quality frames and assign them lower frame quality scores, thus reducing their probability of being selected as keyframes in subsequent comprehensive scoring. For example, when passing through areas of alternating tree shadows on urban roads, some frames may experience localized overexposure or underexposure due to drastic changes in lighting. The joint features of such frames are significantly weaker than adjacent frames in terms of spatial texture and edge information. The frame quality evaluation network can therefore assign a lower score, enabling the comprehensive scoring system to have the ability to self-distinguish the quality of perceptual input.

[0039] Step S22: Calculate the scaled dot product attention weights between the historical joint features and the current joint features, and pool the attention heads to obtain the decision relevance score. The decision relevance score represents the degree of decision association between the historical joint features and the current joint features.

[0040] In practical implementation, historical joint features and current joint features can be mapped to query vector and key vector respectively through a linear projection layer. The scaled dot product similarity between the two is calculated, and after normalization, the attention weight of the historical frame corresponding to the current frame is obtained. When a multi-head attention mechanism is adopted, multiple attention heads calculate the similarity in different feature subspaces. After pooling the weights of each attention head, a comprehensive decision relevance score is obtained.

[0041] The mathematical form of this process is, ; ; in, Let be the attention weight matrix between the joint features of historical time i and the joint features of current time t. For the linear projection of the current joint features, For the linear projection of the joint features of historical moment i, This represents the dimension of the projection vector, i.e., the denominator in the scaling factor. Scaling factor Score the decision relevance at historical moment i. This is an operation for pooling the attention weights of multiple attention heads.

[0042] This score reflects the degree of matching between the scene configuration, spatial distribution of traffic participants, and motion state in historical frames and the current frame. A higher score indicates a more similar driving situation in the historical frame to the current moment, and thus greater reference value for current decision-making. For example, in a ramp merging scenario, when the current frame shows a vehicle approaching the lane boundary and the vehicle is slowing down to yield, the frames in the historical cache where the vehicle first begins to approach from the adjacent lane and the driver begins to decelerate show a high degree of semantic continuity with the current frame in terms of the relative position change trend of the vehicle and the speed change trend of the vehicle. These frames will have significantly higher decision relevance scores than frames in the cache where the vehicle is still moving at a constant speed or the vehicle is cruising at a constant speed. Similarly, in a traffic light intersection scenario, when the current frame shows the vehicle approaching the stop line and the traffic light is red, the key transition frames in the history where the traffic light changes from green to yellow and then to red have a direct causal relationship with the current frame, resulting in a correspondingly higher decision relevance score. This mechanism enables the model to automatically locate key moments in the historical sequence that are semantically related to the current driving decision, rather than relying solely on temporal proximity.

[0043] Step S23: Based on the time interval between the historical moment and the current moment, calculate the time proximity score using exponential decay. The time proximity score represents the timeliness of the historical joint features.

[0044] In actual driving, even if the scene content of a historical frame is highly relevant to the current frame, if the frame is too old, the traffic conditions it reflects may have changed substantially, and its reference value for current decision-making should be reduced. The decay rate of the exponential decay function is controlled by a time decay coefficient, which can be set according to the actual application scenario or learned automatically during end-to-end training. The calculation method for the time proximity score is as follows: ; in, Score the time proximity of historical moment i. The time decay coefficient, For the current moment, For a historic moment, Let t be the time interval between historical time i and current time t.

[0045] Taking a congested urban driving scenario as an example, a vehicle may only move forward a few meters in a few seconds, and the positional relationships of surrounding vehicles change slowly. In this case, the time decay coefficient can be relatively small, allowing more distant historical frames to retain a certain degree of timeliness. However, in a high-speed cruising scenario, a vehicle may have moved forward tens or even hundreds of meters in a few seconds, and the relative positions of surrounding vehicles and road structure may have changed significantly. In this case, the time proximity score of more distant historical frames decays rapidly, ensuring that the model primarily relies on recent observations for decision-making. By introducing an explicit time decay term, a timeliness constraint that conforms to the physical characteristics of driving scenarios is provided for the comprehensive correlation scoring system, avoiding the model's over-reliance on outdated observation information.

[0046] Step S24: Weight the frame quality score, decision relevance score and time proximity score to obtain the comprehensive relevance score.

[0047] The weight coefficients for the three dimensions correspond to frame quality weight, decision relevance weight, and temporal proximity weight, respectively. These weight coefficients can be automatically learned through end-to-end training, allowing the model to adaptively adjust the importance ratio of each dimension according to different driving scenarios. For example, on highways with clear weather, smooth roads, and good perception conditions, frame quality is generally high, and the quality differences between frames are small. The model may automatically increase the weight ratio of decision relevance and temporal proximity. However, in rainy or snowy weather or under low light conditions at night, some frames may have large fluctuations in perception quality. The model may automatically increase the weight ratio of frame quality to prioritize the elimination of interference from low-quality frames. The scores of the three dimensions are weighted and summed to form a comprehensive relevance score for each frame. This score comprehensively reflects the reliability of historical frame information, the degree of decision relevance, and timeliness. The weighted summation formula for the comprehensive relevance score is as follows: ; in, The overall relevance score for historical moment i, These are the weighting coefficients for frame quality scoring. The weighting coefficients for the decision relevance score. Weighting coefficients for rating the degree of time proximity.

[0048] Based on the aforementioned comprehensive relevance score, step S2 selects a predetermined number L of keyframes from the historical feature cache in descending order of score. Specifically, keyframes are selected according to their comprehensive relevance score from highest to lowest. By combining historical features, a set of keyframes is obtained. The formal representation of the keyframe set is: ; in, Let be the set of keyframes at time t. For the historical joint features selected as keyframes, To select the top scorers from highest to lowest Each operation. To achieve a comprehensive relevance score, The number of keyframes selected is the preset number L.

[0049] The number of keyframes, L, is positively correlated with the current vehicle speed; that is, more keyframes are selected in high-speed driving scenarios, and fewer keyframes are selected in low-speed driving scenarios. This adaptive adjustment mechanism has a clear physical basis: at high speeds, vehicles travel a longer distance per unit time, and the time span of road structure changes and interactions with traffic participants is greater, requiring a longer historical window to capture long-distance interactions and gradual trends; while in low-speed crawling or stationary states, scene changes are limited in a short period of time, and too many historical frames would introduce redundant information, which is detrimental to the focus of temporal fusion. In specific implementation, a baseline number of frames and a speed ratio coefficient can be set, and the number of keyframes to be selected can be calculated in real time based on the current vehicle speed, with truncation processing performed within a preset upper and lower limit range. For example, when a vehicle is traveling at a speed below 20 km / h in congested urban areas, the number of keyframes can be relatively small, focusing on capturing recent events such as the starting and stopping of the vehicle in front and the cutting in by nearby vehicles. When a vehicle is traveling at a speed exceeding 80 km / h on a highway, the number of keyframes can be relatively large, in order to simultaneously cover the approaching trends of distant vehicles, changes in the curvature of the road ahead of the vehicle, and traffic sign information from earlier times. Through this dynamic adaptive mechanism, step S2 can maintain a reasonable balance between temporal perception depth and computational efficiency under different vehicle speed conditions.

[0050] Accordingly, the fusion weight of each keyframe can be determined by normalizing its comprehensive relevance score, so that keyframes that contribute more to the current decision occupy a more important position in the fused representation. The normalization calculation method for the fusion weight is as follows: ; in, The fusion weights for keyframe j The comprehensive relevance score for keyframe j is given, and τ is a temperature coefficient used to control the smoothness of the weights after normalization. for, The sum of the indexed scores for all keyframes.

[0051] Furthermore, the fusion process can employ a hierarchical fusion network to integrate information from keyframes and the current frame layer by layer, ultimately outputting a temporal fusion representation that simultaneously contains the current observation and key historical context. The calculation method for the temporal fusion representation is as follows: ; in, For temporal fusion representation, For hierarchical converged networks, As a current joint feature, The fusion weights for each keyframe, For the joint features of each keyframe, This represents the number of keyframes.

[0052] By independently evaluating and explicitly combining the three dimensions of frame quality scoring, decision relevance scoring, and temporal proximity scoring, this invention constructs a multi-dimensional, decoupled, and interpretable keyframe selection system. Compared to existing technologies that rely solely on a single attention weight or filter historical frames only in chronological order, this comprehensive scoring mechanism enables the model to simultaneously assess the reference value of each historical frame from three perspectives: "whether the historical frame itself is reliable," "whether it is semantically relevant to the current decision," and "whether it is too outdated in time." This effectively filters out the interference of low-quality frames, irrelevant frames, and excessively outdated frames on the current planning, ensuring that the final set of keyframes participating in temporal fusion maintains a high level of information content and decision relevance, providing a high-quality temporal input foundation for subsequent temporal fusion representation generation and empirical route matching.

[0053] Furthermore, the frame quality evaluation network, the calculation of the scaling dot product attention weights, the calculation of exponential decay, and the weight coefficients of the weighted summation in step S24 are all obtained through end-to-end training. End-to-end training includes the following steps: Step S201: Obtain driving demonstration samples, which include multi-frame, multi-view image sequences arranged in chronological order, vehicle state sequences, target position sequences, and corresponding expert driving trajectory ground truth values.

[0054] Multi-frame, multi-view image sequences were acquired through real-vehicle acquisition or simulation platform recording, covering complete image data from multiple cameras at consecutive moments while the vehicle was driving on actual roads or in a high-fidelity simulation environment. These sequences included various road types such as urban roads, highways, ramps, and intersections, as well as various lighting and weather conditions such as sunny days, cloudy days, and nighttime. The vehicle state sequence recorded motion state parameters such as vehicle speed, heading angle, steering angle, and acceleration at each sampling moment. These parameters were acquired in real-time via the vehicle bus or sensors and aligned with the image frame timestamps. The target position sequence recorded the target point coordinates or navigation path information obtained from the navigation module at each moment, providing context for the model's driving intent. The expert driving trajectory ground truth is the source of the supervision signal. It can be obtained by recording the actual driving trajectories of skilled drivers at several future moments in real-world driving, or by high-quality trajectories generated by a rule planner or offline optimization algorithm in a high-precision simulation environment. The ground truth of the expert driving trajectory is represented in the form of a sequence of waypoint coordinates for multiple future time steps. Each waypoint contains the location information that the vehicle should arrive at at the corresponding time. The complete driving demonstration samples cover a variety of driving scenarios, from simple cruising to complex interactive integration, and from high-speed driving to low-speed crawling in urban areas. The diversity and breadth of the samples directly affect the generalization ability of the model after training.

[0055] Step S202: Divide the driving demonstration sample into multiple training segments according to the time window. Each training segment takes the current time frame as input and the ground truth value of the expert driving trajectory at a future time as the supervision signal.

[0056] The time window is segmented along the time axis using a sliding window approach, with adjacent windows overlapping to increase the sample size. Each training segment simulates a real driving decision-making situation faced by the vehicle at a specific moment. The input of each segment includes multi-view images of the current moment, the vehicle's current state information, the target's current location information, and joint features from several historical frames within the historical cache range. The supervision part of the segment is the ground truth of the expert driving trajectory for several future time steps corresponding to that current moment. The number of segmented training segments is much larger than the number of original driving demonstration samples. By fully segmenting the complete driving record, the decision-making situation at each moment can be fully extracted, enabling the model to learn rich driving strategy mapping relationships from limited driving data. During the segmentation process, some segments can be divided into training, validation, and test sets according to a certain ratio. The training set is used for model parameter updates, the validation set is used for hyperparameter adjustment and early stop judgment during training, and the test set is used to finally evaluate the model's generalization performance.

[0057] Step S203: For each training segment, perform forward propagation according to steps S1 to S3 to obtain the predicted future driving trajectory of the vehicle.

[0058] During the forward propagation phase of training, the data flow processing path is completely consistent with that of the inference phase. The current multi-view images and vehicle state information are processed through perceptual encoding and cross-modal fusion to obtain the current joint features. The historical feature cache stores the historical joint feature sequence corresponding to the training segment. The model sequentially performs frame quality assessment, decision relevance calculation, temporal proximity calculation, and weighted summation of comprehensive scores. Keyframes are selected and temporal fusion is performed. Then, an empirical routing mechanism activates some expert modules to generate candidate trajectories, which are then fused and output as predicted trajectories. The only difference between the training and inference phases is that the training phase does not require momentum updates to the long-term experience memory of experts; memory updates are only performed during the online adaptation phase after inference deployment. The predicted trajectory is output as a sequence of waypoint coordinates for several future time steps. Each predicted waypoint corresponds one-to-one with the waypoint in the ground truth of the expert driving trajectory, and the difference between the two forms the basis for subsequent loss calculations.

[0059] Step S204: Calculate the trajectory prediction loss, which includes at least the position error between the predicted future vehicle trajectory and the true value of the expert driving trajectory.

[0060] Position error can be calculated by averaging the Euclidean distance or smoothed L1 distance between the predicted waypoint coordinates and the true waypoint coordinates over time steps. Alternatively, different weights can be assigned to the errors at different time steps, allowing the model to focus more on the prediction accuracy of recent waypoints while also considering the trend accuracy of long-term waypoints. In addition to position error, the trajectory prediction loss can also incorporate auxiliary loss terms, such as curvature smoothing loss, which penalizes drastic changes in steering angle between adjacent time steps, encouraging a smoother and more feasible kinematic trajectory. Heading consistency loss can also be introduced to ensure that the heading angle of the predicted waypoints matches the true heading angle, further improving the overall quality of the trajectory. These loss terms are weighted and combined to form the final trajectory prediction loss, which comprehensively measures the gap between the model's output trajectory and expert driving performance across multiple dimensions, including spatial accuracy, temporal smoothness, and kinematic feasibility.

[0061] Step S205: Based on the trajectory prediction loss, the frame quality evaluation network, the calculation parameters of the scaled dot product attention weights, the time decay coefficient in the exponential decay calculation, and the weight coefficients of the weighted summation are updated synchronously through backpropagation.

[0062] During the backpropagation process, the trajectory prediction loss propagates gradients back layer by layer from the output end along the complete computational graph of forward propagation. All the aforementioned components are located on the path of the computational graph, so their parameters can all receive gradient signals from the final trajectory prediction loss to achieve synchronous optimization. The key advantage of this unified training method is that how the frame quality assessment network judges the information reliability of a frame, how the decision relevance calculation measures the semantic correlation degree between two frames, how the time decay coefficient balances the decay rate of timeliness, and how the three dimension weights assign their respective importance proportions are not artificially preset fixed rules, but are automatically learned and formed by the model on a large amount of driving data with the goal of minimizing the trajectory prediction error. For example, during the training process, if a certain type of historical frame is relatively old in time but its scene configuration is highly related to the current frame and plays an important role in accurately predicting the trajectory, the backpropagated gradient signal will indirectly guide the increase of the decision relevance weight or the decrease of the time decay coefficient through the trajectory prediction loss, so that such frames can obtain higher scores in the comprehensive evaluation and thus be selected as key frames. On the contrary, if some recent frames have a high temporal proximity score but poor frame quality or limited relevance to the current decision, and their contribution to the final trajectory prediction accuracy is proven to be low, the model will suppress the comprehensive scores of such frames by reducing the frame quality weight or the decision relevance weight. Through this end-to-end joint optimization mechanism, the three dimensions of the comprehensive relevance score and their fusion weights can automatically adapt to the differentiated demands of different driving scenarios for the utilization of historical information, making the key frame selection strategy highly consistent with the final trajectory planning target, and avoiding the problems of target inconsistency and information loss that may occur in the phased training or multi-stage tuning process.

[0063] In the present invention, the frame quality assessment network, the calculation parameters of the scaled dot-product attention weight, the time decay coefficient in the exponential decay calculation, and the weight coefficient of the weighted sum in step S24 are not obtained through manual setting or independent training, but are automatically learned from driving demonstration data through a unified end-to-end training process.

[0064] Further, the empirical routing mechanism in step S3 includes a routing phase and a trajectory generation phase. The routing phase performs matching evaluation on all N expert modules to obtain the routing weight of each expert module; the trajectory generation phase activates only K expert modules to participate in trajectory generation based on the routing weights, where K<N. The value of the number K of activated expert modules is positively correlated with the scene complexity.

[0065] In the routing phase, the system performs matching evaluation on all N expert modules to obtain the routing weight of each expert module. The input to this phase is the temporal fusion representation output from step S2, which integrates the joint features at the current moment and the keyframe features selected through multi-dimensional comprehensive scoring, carrying the complete spatiotemporal context information of the current driving situation. The system maps the temporal fusion representation to a query vector through a linear projection layer, and simultaneously maps the long-term experience memories of all N expert modules to key vectors through their respective linear projection layers. The scaling dot product similarity between the query vector and each key vector is calculated to obtain the matching score of each expert module. The matching score reflects the degree of consistency between the driving situation depicted by the current temporal fusion representation and the scenario priors and policy tendencies stored in the long-term experience memories of each expert. The higher the score, the more suitable the expert is to handle the current driving situation. After normalizing the matching scores of each expert module, the routing weight of each expert module is obtained. The sum of the routing weights is 1, which intuitively reflects the relative suitability of each expert in the current scenario. In practical implementation, the total number N of expert modules can be set according to the computational resources and scenario diversity requirements of the application scenario. For example, 8 to 16 expert modules can be set in urban road scenarios, and 16 to 32 expert modules can be set in comprehensive scenarios covering multiple road types such as urban areas, highways, and rural areas. The long-term experience memory of each expert module obtains an initial value through end-to-end learning during the training phase. After inference deployment, it can be continuously optimized through an online update mechanism. During the training process, the memory vectors of different experts naturally differentiate into their respective specialty representations for handling different types of driving scenarios. Some experts gradually become more proficient in handling highway cruising and lane keeping scenarios, some gradually become more proficient in handling intersection turning and traffic sign response scenarios, and some gradually become more proficient in handling interaction scenarios such as ramp merging and yielding. The computational workload of the routing phase is N vector similarity operations. Since vector similarity calculation is a lightweight operation, even if N is large, it will not constitute a significant computational bottleneck. This allows the system to comprehensively match and evaluate the experience of all experts during inference, avoiding the risk of missing the most suitable expert due to rapid screening.

[0066] During the trajectory generation phase, the system activates only K expert modules to participate in trajectory generation based on the routing weights of each expert module output from the routing phase, where K is strictly less than N. Specifically, the system sorts the N expert modules in descending order of routing weight and selects the K expert modules with the highest routing weights as the activated expert set. Each activated expert module receives a temporal fusion representation and combines it with its own long-term experience memory to generate a candidate future trajectory from that expert's perspective. When generating candidate trajectories, each activated expert first adaptively determines the intensity of absorbing experience from its own long-term experience memory through a gating fusion mechanism, fusing the temporal fusion representation with the long-term experience memory into an expert-enhanced representation, which is then autoregressively decoded by the trajectory decoder corresponding to that expert to generate a sequence of waypoint coordinates for several future time steps. Since the scenario priors and driving strategy tendencies implied in each expert's long-term experience memory are different, different experts may output different candidate trajectories for the same temporal fusion representation. For example, in a ramp merging scenario, an expert skilled in yielding strategies may output a trajectory of slowing down and yielding, while an expert skilled in following strategies may output a trajectory of maintaining speed and closely following the vehicle in front. The system weights and fuses the candidate trajectories output by each activated expert based on their routing weights. Experts with higher routing weights contribute more to the fusion result, ultimately forming a comprehensive trajectory which is then smoothed by the trajectory head network before being output. The core advantage of activating only a small number of expert modules in the trajectory generation stage is that candidate trajectory generation requires complete forward computation through the trajectory decoder. The decoding process involves autoregressive generation of waypoint sequences for multiple future time steps, which is computationally much more computationally demanding than vector similarity matching in the routing stage. By using sparse activation, the execution scope of the heavyweight decoding computation is limited to K experts, effectively controlling inference latency while maintaining model capacity and expert diversity, thus meeting the real-time requirements of the onboard computing platform.

[0067] The value of the number of activated experts, K, is positively correlated with the scene complexity; that is, the more complex the scene, the more experts need to be activated. Scene complexity can be determined in actual deployment based on a combination of factors, including the current road type, the number and density of surrounding traffic participants, the presence of traffic lights or signs, the curvature and slope of the road geometry, and the relative motion between the vehicle and surrounding vehicles. When cruising on a straight section of a highway, the number of surrounding vehicles is relatively small and their relative motion is simple, resulting in low scene complexity. In this case, only a small number of experts need to be activated to generate a reasonable cruising trajectory. However, when turning left at a traffic light intersection in the city, the vehicle needs to simultaneously process multiple pieces of information, including traffic light status, oncoming straight-ahead vehicles, non-motorized vehicles and pedestrians traveling in the same direction, and the road geometry constraints at the intersection. Scene complexity increases significantly, requiring the activation of more expert modules to integrate various driving strategies and scene experience. In practice, scenario complexity can be predicted in real time from temporal fusion representations using a lightweight complexity assessment network, or it can be determined hierarchically based on metadata such as the number of traffic participants and road type output by the perception module using preset heuristic rules. The value of K is adjusted linearly or piecewise between preset lower and upper limits based on the complexity assessment results. For example, the lower limit of K can be set to 1 to handle extremely simple scenarios, while the upper limit can be set to 4 to ensure controllable computational overhead. This dynamic adaptive adjustment mechanism enables the system to flexibly allocate computational resources according to the actual needs of the current driving situation. In simple scenarios, trajectory planning is completed with minimal computational overhead, while in complex scenarios, more experts are involved in collaborative decision-making to improve the safety and rationality of the planning, achieving a dynamic balance between inference efficiency and planning quality.

[0068] Furthermore, the routing phase includes: Step S311: Calculate the scaled dot product similarity between the query vector of the temporal fusion representation and the key vector of the long-term experience memory of each expert module to obtain the matching score of each expert module.

[0069] In its implementation, the temporal fusion representation is a unified spatiotemporal representation vector output from step S2, which integrates the current joint features and keyframe features. This vector carries complete contextual information of the current driving situation, including multi-dimensional features such as road structure, distribution of traffic participants, traffic sign status, and vehicle movement trends. The system maps the temporal fusion representation to a query vector through a learnable query projection matrix. During end-to-end training, this query projection matrix automatically learns how to extract the most discriminative information for expert matching from the temporal fusion representation. Simultaneously, the long-term experience memory of each expert module is mapped to a key vector through its respective key projection matrix. The key projection matrix is ​​also learned through end-to-end training, and during training, the key projection matrices of different experts naturally differentiate into feature extractors for their respective preferred driving scenario types. The scaling dot product similarity calculation process between the query vector and each key vector is as follows: calculate the inner product of the query vector and the key vector, and then divide it by the square root of the dimension of the key vector as a scaling factor to prevent the inner product value from being too large when the vector dimension is large, which would lead to gradient instability. The obtained similarity value is the matching score of the expert module in the current driving situation. The higher the matching score, the higher the degree of fit between the scenario prior and strategy tendency stored in the long-term experience memory of the expert and the current driving situation, and the more suitable the expert is to participate in the current trajectory planning.

[0070] Specifically, the system will perceive the temporal features Z of the target. t The query projection matrix is ​​mapped to the query vector. The long-term experience memory of the i-th expert module is mapped to a key vector through a key projection matrix. The scaled dot product similarity between the two is calculated as the matching score of the expert module, and its mathematical form is: ; in, Let be the matching score of the i-th expert module at the current time t. A query vector for perceiving temporal features of the target. Let be the key vector of the long-term experience memory of the i-th expert module. Let be the dimension of the key vector. This is the scaling factor.

[0071] Step S312: Normalize the matching scores of each expert module to obtain the routing weight of each expert module.

[0072] The normalization operation can be implemented using the Softmax function, which converts the matching scores of all N experts into a probability distribution, so that the routing weights of all experts are non-negative and their sum is 1.

[0073] The normalized calculation method for route weights is as follows: ; in, Let be the routing weight of the i-th expert module at the current time t. Let be the matching score of the i-th expert module. The total number of expert modules, The exponential sum of the matching scores for all N expert modules.

[0074] Routing weights intuitively reflect the relative suitability and importance of each expert module in the current driving context. For example, when a vehicle is traveling on a straight section of a highway, expert modules skilled in high-speed cruising and lane keeping have higher matching scores between their long-term experience memories and the current temporal fusion representation, and will receive higher routing weights after normalization. Conversely, expert modules skilled in intersection turning or congestion following will receive routing weights close to zero because of the lower matching degree between the current scenario and their experience memories. Routing weights serve as the core basis for selecting activation experts and fusing candidate trajectories in the subsequent trajectory generation stage, directly determining which experts' driving experience will dominate the final trajectory output and the contribution ratio of each expert's opinion in the final decision. Through these two sub-steps, the routing stage completes a comprehensive matching evaluation of all N expert modules using lightweight vector similarity calculation, providing a reliable basis for expert selection in the sparse activation stage of trajectory generation.

[0075] Furthermore, the trajectory generation stage includes: Step S321: Sort the N expert modules in descending order of routing weight.

[0076] The routing weight is the result of normalizing the matching scores of each expert module in step S312. Each expert's routing weight reflects the relative fit between their long-term experience memory and the driving situation depicted by the current temporal fusion representation. The sorting operation arranges all N expert modules into an ordered list according to their fit, from highest to lowest. The expert with the highest routing weight is placed at the front, indicating that this expert's driving experience best matches the current scenario, and their corresponding candidate trajectory prediction has the highest rationality. The expert with the lowest routing weight is placed at the back, indicating that the scenario prior stored in their long-term experience memory has a lower matching degree with the current scenario. The sorting operation provides a clear selection criterion for subsequent sparse activation, ensuring that the expert module most relevant to the current driving situation is activated first in each inference.

[0077] Step S322: Select the K expert modules with the highest routing weights as the active expert set.

[0078] By selecting only the K expert modules with the highest routing weights, the system strictly limits the execution scope of subsequent heavyweight trajectory decoding calculations to these K experts.

[0079] The formal representation of activating the expert set is as follows: ; in, Let be the set of activation experts at time t. For the i-th expert module, Let i be the routing weight of the i-th expert module. To activate the number of experts, i.e., the preset number K, To select the top routes in descending order of their routing weights Each operation.

[0080] In practical implementation, the value of K can be set according to application requirements. For example, on autonomous driving computing platforms with abundant computing resources, K can be set to 3 or 4, while on embedded platforms with more limited computing resources, K can be set to 1 or 2. The core advantage of this sparse activation mechanism is that the routing stage performs lightweight matching and evaluation on all N experts, enabling the system to have the ability to select the best globally; while the trajectory generation stage only activates K experts, which greatly reduces the computational overhead of trajectory decoding, allowing the system to meet the real-time requirements of the vehicle platform while maintaining model capacity and expert diversity.

[0081] Step S323: Generate candidate future trajectories from each expert module in the activated expert set.

[0082] After receiving the temporal fusion representation, each activated expert module first adaptively determines the intensity of absorbing experience from its own long-term experience memory through a gating fusion mechanism. The temporal fusion representation is then fused with the long-term experience memory to form an expert-enhanced representation. This representation is then decoded by the trajectory decoder corresponding to that expert to generate a sequence of waypoint coordinates for several future time steps, forming a candidate future trajectory from that expert's perspective. Because the long-term experience memories of different experts naturally differentiate into the types of driving scenarios they are good at handling during training, under the same input conditions, different experts may output candidate trajectories with distinct characteristics. For example, when a vehicle faces a complex situation of an approaching vehicle in a ramp merging scenario, an expert module skilled in yielding strategies has a large amount of prior knowledge and strategic tendencies regarding deceleration and yielding in its long-term experience memory. The candidate trajectory generated by this expert may exhibit obvious deceleration characteristics and moderate lateral deviation. Conversely, an expert module skilled in following strategies has a large amount of experience in maintaining a safe distance from the vehicle in front in its long-term experience memory. The candidate trajectory generated by this expert may exhibit maintaining the current speed or slightly accelerating to maintain a safe distance from the vehicle behind. The differentiated candidate trajectories output by different experts provide diverse decision options for subsequent trajectory fusion steps. The system can perform weighted fusion of candidate trajectories based on the routing weights of each expert, so that the final output driving trajectory can comprehensively absorb the reasonable components of multiple experts' experience perspectives, rather than relying solely on the potential strategy biases of a single expert, thereby achieving safer and more robust trajectory planning in complex dynamic scenarios.

[0083] In addition, step S3 also includes: step S324: based on the routing weights of each activation expert, weighted fusion of each candidate future trajectory is performed to obtain the future driving trajectory of the vehicle.

[0084] Step S324 performs weighted fusion of each candidate future trajectory based on the routing weights of each activated expert to obtain the vehicle's future driving trajectory. In step S323, each activated expert module generates its own candidate future trajectory, which is represented as a sequence of waypoint coordinates over multiple future time steps. Since the long-term experience memories of different expert modules contain different scenario priors and driving strategy tendencies, their output candidate trajectories may differ in geometric shape and motion characteristics. Step S324 first multiplies the candidate trajectory of each activated expert by the routing weight corresponding to that expert, sums all the weighted candidate trajectories step-by-step, and then divides by the sum of the routing weights of the activated experts to obtain the weighted fused intermediate trajectory.

[0085] The calculation method for trajectory fusion is as follows: ; ; in, The intermediate trajectory after weighted fusion is a sequence of waypoint coordinates over the next T time steps. Let be the routing weight of the i-th activated expert module at the current time t, with a value between 0 and 1, and the sum of the routing weights of all activated experts is 1. To sum over all experts in the activated expert set E_t, The candidate future trajectory generated for the i-th activated expert module contains a sequence of waypoint coordinates for the next T time steps. The sum of the routing weights of all experts in the activated expert set is used as the normalized denominator.

[0086] This weighted fusion mechanism ensures that experts with higher routing weights contribute more to the final result with their candidate trajectories, guaranteeing that the final output trajectory is dominated by the expert opinions that best fit the current driving situation. Subsequently, the intermediate trajectory after weighted fusion is input into the trajectory head network, which is used to smooth the fused trajectory and correct its kinematic feasibility. This eliminates waypoints that may be discontinuous or not in line with vehicle kinematic constraints due to the weighted superposition of different expert trajectories. Finally, it outputs a sequence of waypoints for the vehicle over several time steps in the future. This sequence of waypoints can be directly provided to the vehicle's lower-level controller for conversion into specific control commands for throttle, braking, and steering.

[0087] Further, in step S323, each expert module in the activated expert set generates candidate future trajectories, including: Step S3231: Input the temporal fusion representation and the long-term experience memory of the expert module into the gating network to obtain the gating coefficients. The gating coefficients represent the strength of absorbing experience from the long-term experience memory.

[0088] In its implementation, the gating network is a lightweight, learnable network module consisting of fully connected layers and a sigmoid activation function. The input is a concatenated vector of the temporal fusion representation and the expert's long-term experience memory, and the output is a scalar gating coefficient with a value between 0 and 1.

[0089] The gating coefficient is calculated as follows: ; in, Let be the gating coefficient for the i-th activated expert. It is the Sigmoid activation function. Here is the weight matrix of the gated network. To perceive temporal features of the target. Let [· || ·] represent the long-term experience memory of the i-th expert module, and let [· || ·] represent the vector concatenation operation. is the bias vector of the gated network.

[0090] The gating coefficient dynamically adjusts the extent to which the trajectory generation in the current scenario relies on the driving priors and strategy preferences stored in the expert's long-term experience memory, and the extent to which it directly relies on the real-time scenario information reflected in the current temporal fusion representation. When the gating coefficient approaches 1, it indicates that the current scenario highly matches the classic driving scenarios stored in the expert's long-term experience memory, and the system tends to absorb more of the strategy priors from the experience memory to guide trajectory generation. When the gating coefficient approaches 0, it indicates that the current scenario differs significantly from the scenarios the expert is proficient in, or that the current scenario belongs to the long-tail scenario outside the expert's experience coverage, and the system tends to rely more on the current real-time perception information for judgment, avoiding being misled by unsuitable past experiences. The value of the gating coefficient is adaptively calculated by the gating network based on the specific content of the temporal fusion representation and long-term experience memory, without the need for manually setting a threshold. In practice, the gating network parameters of different expert modules are independent and optimized together with the long-term experience memory of experts during the end-to-end training process. This enables each expert's gating network to learn in which scenarios to rely more on its own experience and in which scenarios to remain cautious.

[0091] Step S3232: Based on the gating coefficient, perform gating fusion of temporal fusion representation and long-term experience memory to obtain expert-enhanced representation.

[0092] The specific operation of gating fusion is as follows: the long-term experience memory is multiplied by the gating coefficient and then added to the temporal fusion representation element by element, and then processed by layer normalization to obtain the expert-enhanced representation.

[0093] The calculation method for expert-enhanced representation is as follows: ; in, For the expert enhancement representation of the i-th activated expert, For layer normalization operation, To perceive temporal features of the target. Let be the gating coefficient for the i-th activated expert. This represents the long-term experience memory of the i-th expert module.

[0094] The core of this operation lies in controlling the amount of information injected from long-term experience memory into the temporal fusion representation through a gating coefficient, forming an adaptive residual connection. When the gating coefficient is large, the scene priors and policy biases in long-term experience memory are heavily injected into the expert-enhanced representation, allowing the subsequent trajectory decoding process to fully utilize the driving experience accumulated by the expert. When the gating coefficient is small, the injection of long-term experience memory is suppressed, and the expert-enhanced representation is mainly composed of the current temporal fusion representation, with the trajectory decoding process relying more on the current real-time perception information. Layer normalization is used to stabilize the numerical range of feature distribution, ensuring that the expert-enhanced representations output by different experts remain consistent on the numerical scale, facilitating processing by the subsequent trajectory decoder. The key advantage of the gating fusion mechanism lies in its smoothness and adaptability. Compared to directly mixing or splicing the temporal fusion representation with long-term experience memory in a fixed ratio and then linearly projecting it, gating fusion allows each expert to dynamically adjust the degree of utilization of experience knowledge according to the specific characteristics of the current scene, avoiding trajectory deviation caused by over-reliance on past experience when the scene changes abruptly, and also avoiding unnecessary re-exploration of classic scenes with clear experience references.

[0095] Step S3233: Input the expert-enhanced representation into the trajectory decoder and decode it to generate the candidate future trajectory corresponding to the expert module.

[0096] The trajectory decoder can employ an autoregressive decoding network structure, starting with an expert-enhanced representation as the initial state, and progressively generating waypoint coordinates for each future time step. At each decoding time step, the trajectory decoder predicts the waypoint coordinates for that time step based on the current decoding state and the expert-enhanced representation. These waypoint coordinates contain the two-dimensional position information that the vehicle should reach at that moment, and may also include auxiliary motion parameters such as the vehicle's heading angle and velocity at that position.

[0097] The mathematical form of the trajectory decoding process is: ; ; in, Let be the waypoint coordinates predicted by the i-th expert module at the τ-th time step after the current time t. Let be the output weight matrix of the i-th expert module trajectory decoder. Let be the hidden state of the i-th expert module trajectory decoder at the τ-th decoding time step. Let be the output bias vector of the i-th expert module trajectory decoder. The candidate future trajectory generated for the i-th expert module is the set of T waypoints. This represents the total number of time steps for predicting the trajectory.

[0098] In the autoregressive decoding process, the waypoint coordinates output at the current time step are fed back as one of the input conditions for decoding at the next time step, ensuring that the generated trajectory maintains kinematic consistency and smoothness in the time dimension. Different expert modules are equipped with their own independent trajectory decoders. Their network parameters are jointly optimized during end-to-end training with the expert's long-term experience memory and gating network parameters, enabling different experts' trajectory decoders to learn and output trajectory styles that conform to the driving strategies they are proficient in. Finally, the candidate future trajectory output by each activated expert module is a sequence of waypoint coordinates for the next T time steps. The value of T is set according to the planning time-domain requirements, for example, it can be set to a time range of 3 to 8 seconds in the future. The sampling interval is consistent with the frame rate of the perception system or the control cycle of the control system.

[0099] Furthermore, after fusing and outputting the vehicle's future driving trajectory in step S3, it also includes: Step S331: Determine the expert module with the highest matching score in the experience routing mechanism.

[0100] The matching score is the original result of the scaled dot product similarity between the query vector calculated in step S311 (the temporal fusion representation) and the key vectors of each expert module's long-term experience memory. This score is normalized and converted into routing weights in step S312. After the trajectory fusion output is completed, the system backtracks the original matching scores recorded in the routing phase to determine the expert module with the highest score. The highest matching score means that the expert's long-term experience memory matches the current driving situation most prominently among all N experts, and the scene priors and driving strategies accumulated by this expert are closest to the currently encountered driving situation in the feature space. The expert determination process does not depend on whether the routing weight exceeds a specific threshold; it only needs to select the expert corresponding to the maximum matching score among all experts, ensuring that an expert is determined as the update object after each inference, thus guaranteeing the continuity of online learning.

[0101] Step S332: Perform a momentum update on the long-term experience memory of the expert module with the highest matching score, while keeping the long-term experience memory of the other expert modules unchanged.

[0102] The specific operation of momentum-based update is as follows: the temporal fusion representation at the current moment is mapped to the same feature space as the long-term experience memory through a projection function. Then, the projected temporal fusion representation is weighted and mixed with the original long-term experience memory according to a preset momentum coefficient to form the updated long-term experience memory. The momentum coefficient ranges from 0 to 1, and this coefficient controls the rate at which new scene information is injected into the expert memory. The smaller the momentum coefficient, the higher the proportion of the original memory is retained, the more conservative the update, and the stronger the stability and persistence of the expert memory; the larger the momentum coefficient, the higher the proportion of new scene information is injected, and the faster the expert adapts to the new scene. The projection function is used to map the temporal fusion representation from the original feature space to the feature space where the expert's long-term experience memory is located, so that the two are mixed in the same dimension. The projection function is also obtained through end-to-end training. The core feature of this update mechanism is its selectivity and asymmetry, that is, each inference only updates the single expert with the highest matching degree to the current scene. Even if multiple experts are activated and participate in trajectory generation during the routing stage, only the expert with the highest matching score receives the memory update. This design is based on the following technical considerations: Among the K experts activated to participate in trajectory generation, the one with the highest matching score represents the experience type most suitable for the current scenario, and updating their memory can strengthen that expert's ability to handle such scenarios. Other activated experts, whose matching scores are not the highest, although also participating in trajectory planning, have a relatively low degree of fit between their long-term experience memory and the current scenario. Updating them might write information that doesn't perfectly match the features of the current scenario into their memory vectors, leading to blurred boundaries and functional convergence between different experts' memories over time—a phenomenon known as expert collapse. By strictly limiting the update object to the expert with the highest matching score, each expert's long-term experience memory is always defined and updated only by the scenario type they are best at handling, thus maintaining the functional differences and expertise stability among different experts in the long term. For example, in the scenario of a vehicle entering a ramp from a highway exit, the expert skilled in ramp merging will receive the highest matching score and be updated, further strengthening their merging strategy experience. While the expert skilled in high-speed cruising may be activated as an auxiliary reference in this inference, since their matching score is not the highest, their long-term experience memory is unaffected by the current scenario information and remains focused on the high-speed cruising domain. This asymmetric update strategy enables the system to gradually absorb new scenario experience during continuous reasoning, while maintaining clear boundaries of expert division of labor, thus achieving a balance between online adaptability and system stability.

[0103] Specifically, momentum-based updates are defined by the following formula: ; .

[0104] in, To match the index of the expert module with the highest score, This is an index operation to retrieve the maximum route weight. To match the long-term experience memory of the expert module with the highest score. For temporal fusion representation, The preset momentum coefficient, This is the projection function.

[0105] in the formula This represents the long-term experience memory of the expert module with the highest matching score. This memory vector is stored in the form of a fixed-dimensional feature vector and its initial value is obtained through end-to-end learning during the model training phase. It contains the scenario priors and policy preferences of the driving scenario types that the expert is good at handling. The temporal fusion representation output in step S2 carries the complete spatiotemporal context information of the current driving situation. The projection function maps the temporal fusion representation from the original feature space to the feature space containing long-term empirical memory, enabling numerical mixing operations between the two in the same dimension. In its implementation, the projection function can be designed as a linear projection layer, consisting of a learnable projection matrix and a bias vector. Its parameters are optimized synchronously with other network parameters during end-to-end training.

[0106] in the formula The preset momentum coefficient, ranging from 0 to 1, controls the rate at which new scene information is injected into the expert's memory. The updated long-term experience memory is a weighted mixture of two parts: the first part is the original memory multiplied by a retention coefficient (1- The second part is the projected current scene features multiplied by the update coefficient. .when When a smaller value is taken, the updated memory is mainly based on existing experience, and the stability and persistence of expert memory are stronger; when When a larger value is chosen, the proportion of new scene information injected increases, allowing experts to adapt to scene changes more quickly. In practice, the momentum coefficient can be set to a fixed empirical value between 0.01 and 0.1, or it can be dynamically scaled according to the matching score to balance learning efficiency and memory stability.

[0107] The momentum-based update formula applies only to the single expert with the highest matching score, while the long-term experience memories of other experts remain completely unchanged. This ensures that each expert's memory vector is always shaped solely by the scenario type they are best at handling. The exponential moving average hybrid approach guarantees the smoothness and continuity of memory updates. New scenario features are gradually integrated into the existing memory with momentum coefficients as the step size, avoiding memory abrupt changes caused by single abnormal scenarios or perceptual noise. Through this mechanism, the system can gradually absorb and accumulate new driving experience during continuous online operation, while maintaining a clear functional division and expertise boundaries between different expert modules over the long term.

[0108] Furthermore, before activating K expert modules from the preset N expert modules to generate candidate future trajectories based on the temporal fusion representation in step S3 through an empirical routing mechanism, the following steps are also included: Step S301: Obtain target location or navigation intent information.

[0109] Target location information typically originates from the vehicle's navigation module. It can be the absolute or relative coordinates of the next waypoint the vehicle needs to reach in the current driving task, or a sequence of waypoints along the planned path. Navigation intent information encompasses higher-level driving instructions, such as instructions to go straight, turn left, turn right, or make a U-turn at intersections, to merge or exit at ramps, and to maintain or change lanes on multi-lane roads. Target location and navigation intent information together constitute the vehicle's driving target context at the current moment, clarifying the direction and manner in which the vehicle should travel in the current scenario. In traditional end-to-end autonomous driving solutions, this information is usually only used as additional input features and simply concatenated with scene features, making it difficult to have a deep structural impact on the model's decision-making logic.

[0110] Step S302: Based on the target location or navigation intent information, perform feature linear modulation on the temporal fusion representation to obtain channel-level scaling parameters and channel-level offset parameters.

[0111] In its implementation, target location or navigation intent information is first mapped to a target conditional vector through a feature encoding network. This target conditional vector expresses the essential features of the current driving target in a compact semantic encoding form. Subsequently, the target conditional vector is input into a feature linear modulation function, which is typically implemented by a fully connected layer. This function outputs two sets of parameter vectors in parallel, each with the same number of channels as the temporal fusion representation: channel-level scaling parameters and channel-level offset parameters. The scaling parameters control the information intensity of each feature channel in the temporal fusion representation, while the offset parameters adjust the baseline response level of each feature channel.

[0112] Specifically, the target condition modulation module receives target features. Output channel-level scaling parameters and channel-level offset parameters And obtain target perception temporal features. .

[0113] ; ; in, For target features, For the characteristic linear modulation function, For channel-level scaling parameter vectors, Here is the channel-level offset parameter vector, and [· || ·] represents the vector concatenation operation. To perceive temporal features of the target. For element-wise multiplication, This is a temporal fusion representation.

[0114] The characteristic linear modulation function (CMMF) automatically establishes a mapping relationship between navigation target semantics and channel-level modulation parameters through end-to-end training. For example, when the navigation instruction is a left turn at an intersection, the CMMF may assign larger scaling parameters to feature channels related to the left-side road structure, oncoming vehicle movement trends, and left-side pedestrian crossings in the temporal fusion representation to enhance the influence of these channels in subsequent expert matching; at the same time, it assigns smaller scaling parameters or negative offsets to channels related to the right-side road structure or straight lane features to relatively suppress their effects.

[0115] Step S303: Using channel-level scaling parameters and channel-level offset parameters, the temporal fusion representation is scaled and offset channel by channel to obtain the target perception temporal features.

[0116] The specific execution method of the above operation is as follows: each feature channel of the temporal fusion representation is multiplied by the corresponding scaling parameter, and then the corresponding offset parameter is added to form the modulated target perception temporal feature. The channel-by-channel modulation operation enables the navigation target information to reshape the temporal fusion representation at a fine-grained channel level, rather than simply scaling or superimposing it at the global level. After modulation, the target perception temporal feature is injected with a clear driving target bias while retaining the original scene context information. Taking the scene of a traffic light intersection in the city as an example, the current temporal fusion representation contains a variety of scene information such as the status of the traffic light ahead, the position of the stop line, the lane lines in each direction, and oncoming and same-direction vehicles. When the navigation command is to go straight, the feature channels in the modulated target perception temporal feature related to the oncoming lane and the area in front of the current lane are strengthened, while the feature channels related to the intersecting roads on the left and right are relatively weakened; when the navigation command is to turn left, the feature channels related to the entrance area of ​​the intersecting road on the left, the movement trend of oncoming straight vehicles, and the blind spot on the left side of the vehicle are strengthened. This channel-level fine-tuning enables differentiated feature representations of the same scene under different navigation targets, laying the foundation for differentiated activation of subsequent expert routing.

[0117] Step S304: Replace the temporal fusion representation with target-aware temporal features and input the empirical routing mechanism for expert module matching.

[0118] The target-condition modulated temporal features replace the unmodulated temporal fusion representation as input to the experience-based routing mechanism. This ensures that the similarity calculation between the query vector and the long-term experience memory key vectors of each expert in subsequent step S311 not only depends on the spatiotemporal context of the current scene but also deeply couples with the navigation target information. This means that in the same road scenario, different navigation intentions will lead to different expert matching score distributions, thereby activating different expert combinations to generate driving trajectories that conform to their respective navigation goals. For example, at the same intersection, when the navigation instruction is to go straight, the expert module that is good at going straight through the intersection will obtain a higher matching score and be activated; when the navigation instruction is to turn left, the expert module that is good at turning left will replace the straight-going expert and obtain a higher matching score. The target-condition modulation mechanism makes the expert routing process subject to the dual constraints of scene context and driving goal. The output trajectory is guaranteed in both spatiotemporal rationality and navigation consistency, avoiding path selection ambiguity or trajectory deviation caused by the lack of navigation information guidance in end-to-end models at multi-way intersections or complex road topologies.

[0119] Accordingly, please refer to Figure 2 A second aspect of this invention provides an end-to-end autonomous driving system based on keyframe perception and experience-based routing, which plans the future trajectory of a vehicle based on the aforementioned end-to-end autonomous driving method based on keyframe perception and experience-based routing, including: The perceptual coding and feature caching module 1 is used to acquire multi-view images of the vehicle and vehicle status information at the current moment, extract spatial visual features and vehicle status semantic features and perform cross-modal fusion to obtain the current joint features, and store the current joint features in the historical feature cache. The historical feature cache is used to save the historical joint features of the most recent sampling moments.

[0120] The keyframe selection and temporal fusion module 2 is used to calculate the comprehensive relevance score for each historical joint feature in the historical feature cache, and select a preset number L of keyframes according to the comprehensive relevance score from high to low. The keyframes are then temporally fused with the current joint features to obtain the temporal fusion representation.

[0121] The experience routing and trajectory generation module 3 is used to activate K expert modules from a preset set of N expert modules based on temporal fusion representation, generate candidate future trajectories through an experience routing mechanism, and then fuse and output the future driving trajectory of the vehicle. Specifically, the experience routing mechanism matches the temporal fusion representation with the long-term experience memory of each expert module to determine the activated expert module. <N。

[0122] Accordingly, a third aspect of the present invention provides an electronic device, including: at least one processor and a memory connected to the at least one processor. The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, cause the at least one processor to perform the aforementioned end-to-end autonomous driving method based on keyframe perception and experience-based routing.

[0123] Accordingly, a fourth aspect of the present invention provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the above-described end-to-end autonomous driving method based on keyframe perception and experience routing.

[0124] The embodiments of the present invention aim to protect an end-to-end autonomous driving method and system based on keyframe perception and empirical routing, which has the following effects: 1. By constructing a multi-dimensional keyframe screening mechanism that integrates frame quality scoring, decision relevance scoring, and temporal proximity scoring, the model can automatically identify and extract the key moments most relevant to the current driving decision from the historical feature cache, effectively filtering out the interference of redundant and noisy frames. Compared with the existing technology of simply superimposing historical frames or relying on a single attention weight, this multi-dimensional scoring system makes the long-term inference process more focused on key driving events such as lane changing, turning, and emergency braking, significantly improving the accuracy of temporal perception and the foresight of trajectory planning, thereby achieving earlier and more stable decision responses in complex dynamic scenarios. 2. Through a full evaluation and sparse activation experience-driven expert routing architecture, the temporal fusion representation is matched with the long-term experience memory of all expert modules, and only a small number of the most relevant experts are activated to participate in trajectory generation. This mechanism enables expert selection to rely on both the current temporal scenario and long-term accumulated driving experience, overcoming the problem of frequent jumps in expert activation caused by traditional gating networks that rely solely on short-term features for routing. In staged driving scenarios such as continuous turns, ramp merging, and braking, the consistency of expert division of labor can be maintained, and the output future driving trajectory is smoother, more coherent, and in line with driving semantic logic. 3. By employing a strategy of updating the experience memory of only the expert module with the highest matching score using momentum, each expert's long-term experience memory is always defined and updated by the scenario type they are best at handling. This effectively prevents the collapse of expert functions caused by mutual contamination of different expert memories during online learning. This mechanism enables the model to gradually absorb new scenario experience during continuous inference, while maintaining the functional differences and expertise stability between different expert modules in the long term. As a result, it has good online adaptability and continuous evolution capabilities in open road environments and long-tail scenarios, balancing planning safety and system maintainability.

[0125] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0126] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0127] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0128] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0129] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. An end-to-end autonomous driving method based on keyframe perception and experience-based routing, characterized in that, Includes the following steps: Step S1: Obtain multi-view images of the vehicle and its status information at the current moment, extract spatial visual features and vehicle status semantic features and perform cross-modal fusion to obtain the current joint features, and store the current joint features in the historical feature cache, which is used to save the historical joint features of the most recent sampling moments. Step S2: For each historical joint feature in the historical feature cache, calculate the comprehensive relevance score, and select a preset number L of key frames according to the comprehensive relevance score from high to low. Perform temporal fusion of the key frames and the current joint feature to obtain the temporal fusion representation. Step S3: Based on the temporal fusion representation, activate K expert modules from the preset N expert modules through an experience routing mechanism to generate candidate future trajectories, and fuse and output the future driving trajectory of the vehicle; wherein, the experience routing mechanism matches the temporal fusion representation with the long-term experience memory of each expert module to determine the activated expert module, K < N.

2. The end-to-end autonomous driving method based on keyframe perception and experience routing according to claim 1, characterized in that, The calculation of the comprehensive relevance score in step S2 includes: Step S21: Input the historical joint features into the frame quality evaluation network to obtain the frame quality score, wherein the frame quality score characterizes the information reliability of the historical joint features themselves; Step S22: Calculate the scaled dot product attention weights between the historical joint features and the current joint features, and pool the attention heads to obtain a decision relevance score, which represents the degree of decision association between the historical joint features and the current joint features; Step S23: Based on the time interval between the historical moment and the current moment, calculate the time proximity score using exponential decay. The time proximity score characterizes the timeliness of the historical joint features. Step S24: The frame quality score, the decision relevance score, and the time proximity score are weighted and summed to obtain the comprehensive relevance score; The selection of a preset number L of keyframes is described above, where the value of the preset number L is positively correlated with the current vehicle speed.

3. The end-to-end autonomous driving method based on keyframe perception and experience routing according to claim 1, characterized in that, The frame quality evaluation network, the calculation of the scaling dot product attention weights, the calculation of the exponential decay, and the weight coefficients of the weighted summation in step S24 are all obtained through end-to-end training. The end-to-end training includes the following steps: Step S201: Obtain driving demonstration samples, which include a multi-frame, multi-view image sequence arranged in chronological order, a vehicle state sequence, a target position sequence, and the corresponding expert driving trajectory ground truth. Step S202: Divide the driving demonstration sample into multiple training segments according to the time window. Each training segment takes the current time frame as input and the ground truth value of the expert driving trajectory at a future time as the supervision signal. Step S203: For each training segment, perform forward propagation according to steps S1 to S3 to obtain the predicted future driving trajectory of the vehicle; Step S204: Calculate the trajectory prediction loss, which includes at least the position error between the predicted future vehicle trajectory and the true value of the expert driving trajectory; Step S205: Based on the trajectory prediction loss, synchronously update the frame quality evaluation network, the calculation parameters of the scaling dot product attention weights, the time decay coefficient in the exponential decay calculation, and the weight coefficients of the weighted summation through backpropagation.

4. The end-to-end autonomous driving method based on keyframe perception and experience routing according to claim 1, characterized in that, The empirical routing mechanism described in step S3 includes a routing phase and a trajectory generation phase; The routing phase involves matching and evaluating all N expert modules to obtain the routing weight of each expert module. In the trajectory generation stage, only K expert modules are activated to participate in trajectory generation based on the routing weight, where K < N; The value of the number K of activated expert modules is positively correlated with the scene complexity.

5. The end-to-end autonomous driving method based on keyframe perception and experience routing according to claim 4, characterized in that, The routing phase includes: Step S311: Calculate the scaled dot product similarity between the query vector of the temporal fusion representation and the key vector of the long-term experience memory of each expert module to obtain the matching score of each expert module; Step S312: Normalize the matching scores of each expert module to obtain the routing weight of each expert module.

6. The end-to-end autonomous driving method based on keyframe perception and experience routing according to claim 5, characterized in that, The trajectory generation stage includes: Step S321: Sort the N expert modules in descending order of their routing weights; Step S322: Select the K expert modules with the highest routing weights as the activated expert set; Step S323: Each expert module in the activated expert set generates a candidate future trajectory.

7. The end-to-end autonomous driving method based on keyframe perception and experience routing according to claim 6, characterized in that, Step S323, which involves activating each expert module in the expert set to generate candidate future trajectories, includes: Step S3231: Input the temporal fusion representation and the expert module's own long-term experience memory into the gating network to obtain gating coefficients, wherein the gating coefficients represent the strength of absorbing experience from long-term experience memory; Step S3232: Based on the gating coefficients, perform gating fusion on the temporal fusion representation and the long-term experience memory to obtain the expert-enhanced representation; Step S3233: Input the expert-enhanced representation into the trajectory decoder and decode it to generate the candidate future trajectory corresponding to the expert module.

8. The end-to-end autonomous driving method based on keyframe perception and experience routing according to claim 4, characterized in that, After fusing and outputting the vehicle's future driving trajectory in step S3, the following steps are also included: Step S331: Determine the expert module with the highest matching score in the experience routing mechanism; Step S332: Perform a momentum update on the long-term experience memory of the expert module with the highest matching score, while keeping the long-term experience memory of the other expert modules unchanged. The momentum update is defined by the following formula: ； in, To match the long-term experience memory of the expert module with the highest score. This is the temporal fusion representation. The preset momentum coefficient, This is the projection function.

9. The end-to-end autonomous driving method based on keyframe perception and experience routing according to claim 4, characterized in that, Before generating candidate future trajectories by activating K expert modules from a preset N expert modules based on the temporal fusion representation in step S3 using an empirical routing mechanism, the method further includes: Step S301: Obtain target location or navigation intent information; Step S302: Based on the target location or navigation intent information, perform feature linear modulation on the temporal fusion representation to obtain channel-level scaling parameters and channel-level offset parameters; Step S303: Using the channel-level scaling parameter and the channel-level offset parameter, the temporal fusion representation is scaled and offset channel by channel to obtain the target perception temporal features; Step S304: Replace the temporal fusion representation with the target-aware temporal features and input them into the empirical routing mechanism for expert module matching.

10. An end-to-end autonomous driving system based on keyframe perception and experience-based routing, characterized in that, The end-to-end autonomous driving method based on keyframe perception and experience routing as described in any one of claims 1-9 plans the future trajectory of the vehicle, including: The perceptual coding and feature caching module is used to acquire multi-view images of the vehicle and vehicle status information at the current moment, extract spatial visual features and vehicle status semantic features and perform cross-modal fusion to obtain the current joint features, and store the current joint features in the historical feature cache, which is used to save the historical joint features of the most recent sampling moments. The keyframe selection and temporal fusion module is used to calculate the comprehensive relevance score for each historical joint feature in the historical feature cache, and select a preset number L of keyframes according to the comprehensive relevance score from high to low, and perform temporal fusion of the keyframes with the current joint feature to obtain the temporal fusion representation. An experience routing and trajectory generation module is used to activate K expert modules from a preset set of N expert modules based on the temporal fusion representation to generate candidate future trajectories through an experience routing mechanism, and then fuse and output the future driving trajectory of the vehicle; wherein, the experience routing mechanism matches the temporal fusion representation with the long-term experience memory of each expert module to determine the activated expert module, where K < N.

Images