A multi-scene automatic driving decision method based on time series modeling

Abstract

The application provides a multi-scene automatic driving decision method based on time sequence modeling, which constructs a driving scene in a simulation platform, adopts a self-vehicle guiding feature aggregation mechanism, focuses on key traffic participants and retains global information. A plurality of strategy models are trained through a reinforcement learning algorithm, and state, action and reward data are collected. A reward reconstruction mechanism is introduced to realize multi-scene reward alignment, and an avoidance behavior signal conforming to human driving preference is injected to construct a unified high-quality decision dataset. Subsequently, a Markov decision process is converted into a time sequence modeling task, a high-efficiency time sequence strategy model is constructed by combining a GRU and a Transformer module, and high-level behavior instructions are output. On the basis of not sacrificing single-scene decision capability, multi-scene unified strategy modeling is realized. The model can embed a trajectory through guiding speed, and is convenient for deployment to a real automatic driving system. Compared with a traditional method, the application realizes multi-scene decision integration, and has stronger decision capability and deployment adaptability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent transportation and automatic driving, in particular to a multi-scene automatic driving decision method based on time series modeling. BACKGROUND

[0002] In recent years, with the rapid development of artificial intelligence and automatic driving technology, the decision planning module as the key link connecting perception and control directly determines the behavior performance of the automatic driving system in complex environments. Among them, the reinforcement learning method has become an important research direction in the field of automatic driving decision because it does not rely on artificial annotation data, has good strategy generalization ability, and has the potential to surpass human drivers in specific tasks. Currently, the mainstream research is usually based on high-fidelity simulation platforms such as CARLA, SUMO, etc., to realize interactive training and strategy learning of agents by constructing virtual traffic environment, and the related achievements are widely used in typical scenes such as automatic overtaking, obstacle avoidance, lane changing, and intersection passing.

[0003] However, the existing reinforcement learning method still faces many challenges in practical application. On the one hand, how to effectively construct the state representation of surrounding traffic participants based on semantic vectors and reasonably splice the input reinforcement learning strategy network. On the other hand, the current reinforcement learning training process usually needs to design a reward function for specific driving scenes and train independent models, which is difficult to unify into a single model. Some solutions attempt to switch between multiple trained models through a scene recognition module, but in the case of ambiguous scene boundaries or drastic state changes, it is easy to cause strategy jumping and discontinuity, thereby affecting the decision stability of the system. SUMMARY

[0004] The present application aims to propose a multi-scene automatic driving decision method based on time series modeling, which combines the feature aggregation mechanism of the ego vehicle to train reinforcement learning algorithms and collect behavior data in multiple driving scenes, and constructs a unified time series strategy model to realize the integration of decision-making in multiple scenes.

[0005] To achieve the above purpose, the present application provides a multi-scene automatic driving decision method based on time series modeling, comprising the following steps:

[0006] S1, constructing multiple automatic driving scenes in a simulation environment, screening the information of traffic participants within the sensing range of the sensor, and constructing their state vectors based on the ego vehicle coordinate system;

[0007] S2, introducing a feature aggregation mechanism of the ego vehicle to perform weighted summation on the state vectors, dynamically focusing on key traffic participants while preserving global information;

[0008] S3, train multiple reinforcement learning strategy models using proximal policy optimization algorithm, and collect state, action and reward data; introduce a reward reconstruction mechanism, and inject a reward signal that conforms to human driving preferences to avoid behavior, realize multi-scenario reward alignment, and construct a unified high-quality decision dataset;

[0009] S4, convert the collected data from Markov decision process form to time series modeling form, combine GRU network and Transformer module to build time series strategy model, and obtain unified multi-scenario automatic driving decision model;

[0010] S5, export and deploy the trained unified strategy model, the model outputs high-level behavior instructions, and dynamically adjusts the expected speed based on the guidance speed during deployment, and fuses it into the trajectory for execution by the control module.

[0011] Further, the feature aggregation mechanism guided by the ego vehicle in step S2 specifically includes:

[0012] Project the state of the ego vehicle and the state of the traffic participants within the perception range into the same feature space to obtain the hidden state representation of the ego vehicle and the first traffic participant;

[0013] Then, take the average of the features of all traffic participants to extract global information, and add it to the individual features to enhance the modeling of individual features;

[0014] Finally, guided by the hidden state representation of the ego vehicle, different weights are assigned to each traffic participant by scaling the dot product, and the original state vector is weighted and aggregated to form the traffic participant feature vector.

[0015] Further, in step S3, the reward reconstruction mechanism specifically includes:

[0016] For different driving scenarios, multiple reward indicators are constructed by considering multiple factors such as traffic efficiency, safety and traffic regulation; the original reward function is defined as , , is the environment state at time , is the action at time ;

[0017] The original reward function includes: safety reward, which punishes collision behavior; rule reward, which regulates traffic behavior such as traffic lights; task completion reward, which rewards successful completion of driving; timeout penalty term, which limits the number of execution steps; per-step basic penalty term, which encourages completing the task as soon as possible; speed reward term, which is calculated based on the ego vehicle speed and target speed, which encourages reasonable speed;

[0018] Then, the reinforcement learning strategy model in each scenario is trained by using the proximal policy optimization algorithm, and the behavior data of the state, action and reward in each scenario is collected after the training is completed;

[0019] To realize unified modeling across scenarios, the reward scales of different scenarios need to be aligned, so a reward reconstruction mechanism is introduced; the reward reconstruction mechanism uniformly uses a standardized reward function instead of the reward function of each scenario so that each reward in the reward function is the same in different scenarios; based on this, all collected samples are revalued according to the unified reward function;

[0020] During the revaluation process, the reward signal of the evasive behavior conforming to the human driving preference is injected to realize reward reconstruction; for the traffic light passing preference, the speed reward item at each time step is calculated according to the distance between the vehicle and the stop line, and the corresponding signal is injected; for pedestrian avoidance, the speed reward item is calculated based on the distance between the vehicle and the pedestrian, and an additional reward signal is injected.

[0021] Further, different rewards are set in different scenarios.

[0022] Further, in the step S4, it includes:

[0023] The collected trajectory data is organized into a number of reward-state-action triplet sequences in chronological order, in the form of:

[0024]

[0025] wherein is the total number of time steps of an episode, is the reward value at time is the environment state at time is the action at time

[0026] To adapt to the needs of time series modeling, the original data is format-converted, and each step reward is replaced by the cumulative reward so as to have more intensive information, and is reconstructed in the following form:

[0027]

[0028] wherein, is the cumulative reward from time to the end of the episode;

[0029] ​​​The cumulative reward sequence is obtained by performing reverse-order accumulation calculation on the reward sequence in each episode.

[0030] Further, the step S4 further includes:

[0031] A time sequence strategy model combining a GRU and a Transformer module is constructed, which is used for modeling a state-reward sequence and outputting a corresponding high-level decision action; the model structure includes the following functional modules:

[0032] An input embedding module:

[0033] The state vector of each time step and the cumulative reward are inputted and mapped to a unified hidden space dimension through two fully connected layers respectively , then the two are spliced and sent to a layer normalization module to obtain input features :

[0034] A sequence modeling module:

[0035] The above input features are spliced into an input sequence in a time sequence, which is inputted into a multi-layer GRU network to model local time dependence and update into a hidden state sequence ;

[0036] A Transformer module:

[0037] To enhance the modeling capability of long-term behavior, a layer of Transformer module is introduced on the GRU output features, which models global time dependence through multi-head self-attention and feedforward network to obtain a final representation ;

[0038] A high-level policy generation module:

[0039] The final representation is inputted into an action output layer to generate a policy action corresponding to the current time ;

[0040] The output is a high-level action decision instruction {accelerate, uniform speed, decelerate}, which is convenient for deployment to a trajectory planning module.

[0041] Further, the GRU network adopts a 5-layer stack, and the Transformer module is set to 1 layer and placed after the GRU module.

[0042] Further, the step S5 includes:

[0043] The unified decision model obtained by training is exported in an intermediate representation format and deployed in a platform system with automatic driving control capability, and the model output is a high-level decision instruction, including discrete behavior signals such as acceleration, constant speed and deceleration, which is used to guide the update of the target speed; during deployment, the following execution process is adopted in combination with the trajectory input structure required by the lower layer motion control module:

[0044] The initialization phase sets the initial target speed In each decision cycle, the behavior instruction output by the current model is used to The speed is dynamically adjusted according to the following update rule:

[0045]

[0046] Wherein is a preset speed adjustment step, is the target speed at the time t;

[0047] Subsequently, the updated target speed is fused into the current reference trajectory point sequence to form an expected trajectory containing position and speed information, and is tracked by the lower layer controller, so that the high-level strategy output is effectively mapped to the actual control command of the vehicle.

[0048] The beneficial effects of the present application are:

[0049] The present application introduces three key improvements: a feature aggregation mechanism guided by the ego vehicle, a reward reconstruction mechanism and a time sequence strategy model with a "GRU+Transformer" structure, so that in a complex traffic environment with ambiguous scene boundaries, multiple scene decisions can be completed by relying on a unified time sequence strategy without switching models. Among them, the feature aggregation mechanism guided by the ego vehicle can dynamically focus on key traffic participants while retaining global information perception ability, improving the effectiveness and robustness of state representation; the reward reconstruction mechanism realizes the alignment of rewards in multiple scenes, and injects avoidance behavior rewards consistent with human driving preferences, enhancing the consistency of the strategy; the structure design combining GRU and Transformer further improves the time sequence modeling capability on the basis of ensuring computational efficiency. The proposed method can be deployed in a real automatic driving system with only a small amount of modification, and has the advantages of simple implementation and strong migration. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 is a flowchart of a multi-scene automatic driving decision method based on time sequence modeling according to an embodiment of the present application;

[0051] Figure 2 ​​is a training process schematic diagram of a multi-scenario automatic driving decision method based on time sequence modeling provided by an embodiment of the present application;

[0052] Figure 3 is a first experimental result schematic diagram of a multi-scenario automatic driving decision method based on time sequence modeling provided by an embodiment of the present application;

[0053] Figure 4 is a second experimental result schematic diagram of a multi-scenario automatic driving decision method based on time sequence modeling provided by an embodiment of the present application;

[0054] Figure 5 is a third experimental result schematic diagram of a multi-scenario automatic driving decision method based on time sequence modeling provided by an embodiment of the present application. DETAILED DESCRIPTION

[0055] The present application will be described in detail below with reference to the specific embodiments shown in the drawings, but these embodiments do not limit the present application, and the structural, method, or functional changes made by those of ordinary skill in the art based on these embodiments are also included in the protection scope of the present application.

[0056] Please refer to Figure 1 The embodiment of the present application provides a multi-scenario automatic driving decision method based on time sequence modeling, which comprises the following steps:

[0057] S1, a plurality of automatic driving scenes are constructed in a CARLA simulation environment, traffic participant information within the sensing range of a sensor is screened, and a state vector thereof is constructed based on a vehicle coordinate system;

[0058] S2, a feature aggregation mechanism of vehicle guidance is introduced, a state semantic vector is weighted and summed, key traffic participants are dynamically focused on, and global information is retained;

[0059] S3, a proximal policy optimization algorithm (PPO) is used to train a plurality of reinforcement learning strategy models respectively, and state, action, and reward data are collected; a reward reconstruction mechanism is introduced, and an avoidance behavior reward signal conforming to human driving preference is injected, multi-scenario reward alignment is realized, a unified high-quality decision dataset is constructed;

[0060] S4, the collected data is converted from a Markov decision process form to a time sequence modeling form, a GRU network and a Transformer module are combined to construct a time sequence strategy model, and a unified multi-scenario automatic driving decision model is obtained;

[0061] S5, the trained unified strategy model is exported and deployed, the model outputs high-level behavior instructions (acceleration, constant speed, or deceleration), and in the deployment process, the expected speed is dynamically adjusted based on the guidance speed and fused into the trajectory for execution by the control module.

[0062] Current autonomous driving systems typically require decision-making modules to train multiple reinforcement learning models for specific scenarios, relying on scene recognition models to switch between models during operation. However, when multiple scenarios are difficult to differentiate, such as those at intersections (e.g., scenario 1 involves avoiding pedestrians on a crosswalk, scenario 2 involves dealing with a pedestrian suddenly crossing the intersection, and scenario 3 involves turning left at a traffic light to avoid oncoming traffic), the boundaries of these scenarios are blurred and their features are similar, making it difficult for scene recognition models to accurately distinguish them. Therefore, this application constructs a temporal policy model with a "five-layer GRU + one-layer Transformer" structure, trained on data collected from reinforcement learning models trained in multiple scenarios, enabling unified decision-making. Furthermore, a feature aggregation mechanism for vehicle guidance is introduced during reinforcement learning model training, and a reward reconstruction mechanism is introduced during data collection. Figure 2 The training process diagram shown is shown below.

[0063] like Figure 2 As shown, three scenarios were first set up: Scenario 1 is to avoid pedestrians crossing the crosswalk normally while making a left turn; Scenario 2 is to deal with pedestrians suddenly running into the crosswalk ahead; and Scenario 3 is to avoid oncoming traffic while making a left turn under special traffic lights.

[0064] against Figure 2 The state vector construction shown below is a unified construction of the vehicle's state vector and the state vectors of other traffic participants, using the following format:

[0065]

[0066] in For the first The state vector of each traffic participant. For the perceived coordinates of traffic participants in the vehicle coordinate system, To predict speed, The heading angle of the traffic participant relative to the vehicle. These are the predicted length and width for traffic participants, respectively. One-hot encoding is used for each traffic participant category (pedestrian or vehicle). As for the vehicle state vector... and All are 0.

[0067] In scenarios involving multiple traffic participants, it is necessary to determine how to organize their state information before inputting it into the reinforcement learning policy model. If a simple concatenation method is used, and the number of participants is uniformly processed by truncating or padding virtual vectors, the result would look like this:

[0068] in, Fix the number of received traffic participants for the model, The number of participants actually observed at the current moment. When using the ranking strategy based on Euclidean distance, if the state at a certain moment is , and in the next moment, due to the traffic participant Close to the ego vehicle, the ranking changes to , although the environment changes slightly, but the state input encoding has a dramatic jump.

[0069] Therefore, the present application introduces a feature aggregation mechanism guided by the ego vehicle, as shown in Figure 2 , the weighted sum of all traffic participants, not only pays attention to the key traffic participants, but also retains the global perception, while avoiding the problem of vector mutation caused by ranking splicing, the specific process is as follows:

[0070] First, project the ego vehicle state vector and the traffic participant state vector within the perception range into the same feature space, as follows:

[0071]

[0072] Where is the ego vehicle state vector, is the state vector of the th traffic participant, is the dimension of the feature embedding. is a multi-layer perception machine used to extract hidden features. and are the hidden state representations of the ego vehicle and the th traffic participant, respectively.

[0073] Secondly, take the average of the hidden features of all traffic participants to extract global information, and add it to the individual feature to enhance the modeling of the individual feature, as follows:

[0074]

[0075]

[0076] Where n is the number of traffic participants, is the average feature of the traffic participant, is the enhanced feature of the th traffic participant after fusing global information. is an adjustable parameter that controls the influence strength of global information.

[0077] Finally, use the properties of vector inner product to represent the ego vehicle hidden state To guide, different weights are given to each traffic participant by scaling the dot product, and the original state vector is weighted and aggregated to form the final environment feature vector, as follows:

[0078]

[0079]

[0080] where represents the exponential function, is the vector inner product, is the weight of the th traffic participant, is the aggregated traffic participant state vector, which is subsequently spliced with other feature vectors to form the environment feature vector.

[0081] As shown in Figure 3 , in the second scenario, when faced with complex situations such as pedestrians suddenly crossing the red light, the introduction of the self-vehicle guided feature aggregation mechanism can eventually converge to a higher reward level, despite a slight decrease in reward value at the beginning of training (due to the more complex encoding structure), demonstrating better policy learning results.

[0082] Figure 4 The decision-making process at three key moments in scenario three is shown. After adopting the self-vehicle guided feature aggregation mechanism, the self-vehicle (white long car in the figure) always allocates the highest attention to the oncoming straight vehicle (whose roof is marked with a red square), indicating that the mechanism can effectively identify and focus on key traffic participants.

[0083] Next, for different driving scenarios, multiple reward indicators are constructed by considering multiple factors such as traffic efficiency, safety, and traffic regulation. The original reward function is defined in the first scenario as follows:

[0084] where is the reward function of the th scenario, is the environment state at time , and is the action at time ; represents the safety reward, which is used to punish collision behavior; represents the rule reward, which is used to regulate traffic behavior such as traffic lights; represents the task completion reward; is the timeout penalty term, which is used to limit the number of execution steps; is the per-step basic penalty term, which is used to encourage the task to be completed as soon as possible; ​​Adjustable coefficients for different scenarios; Speed reward term for encouraging reasonable speed:

[0085]

[0086] where , is the speed of ego vehicle, is the target speed. The values of each term in the reward function need to be further adjusted according to the driving scenario.

[0087] Then use the proximal policy optimization algorithm (PPO) to train the reinforcement learning policy model under each scenario. After training, collect the behavior data of state, action and reward under each scenario:

[0088]

[0089] where, represents the PPO policy trained in the th scenario. and

[0090] To achieve unified modeling across scenarios, the reward scales of different scenarios need to be aligned, so the reward reconstruction mechanism is introduced. This mechanism uniformly uses the standardized reward function:

[0091]

[0092] where all parameters are uniformly configured across scenarios. Based on this, all collected samples are revalued according to the unified reward function:

[0093]

[0094] where, , and are the reconstructed reward, state and action of the th scenario. During the revaluation process, the reward signal of the avoidance behavior consistent with human driving preference can be injected to achieve the effect of reward reconstruction:

[0095] For the traffic light passing preference, the reward signal injection is as follows:

[0096]

[0097] where is the basic speed, is the distance between the vehicle and the stop line, is the red light safe stopping distance, is the maximum safe distance. According to this, update , to encourage the pre-deceleration behavior at red light.

[0098] For pedestrian avoidance, the reward signal is injected as follows:

[0099]

[0100] where is the distance between the vehicle and the pedestrian. Again, by updating , the encouragement of active avoidance of pedestrians is embodied, and the safety of the strategy is improved.

[0101] The original Markov decision process is then converted into a time series modeling task. The Markov decision process assumes that the current action only depends on the current state, and its decision trajectory can be regarded as a state, action, reward sequence evolving in time order, so it can be converted into time series modeling to solve.

[0102] The collected trajectory data is organized in time order into a number of reward-state-action triplet sequences, in the form of:

[0103]

[0104] where is the total number of time steps of an episode, is the reward value at time , is the environment state vector at time , is the action at time

[0105] To adapt to the needs of time series modeling, the original data is format-converted, and each step reward is replaced by the cumulative reward , so as to have more intensive information, and is reconstructed in the following form:

[0106]

[0107] where, is the cumulative reward from time to the end of the episode, defined as:

[0108]

[0109] The cumulative reward sequence is obtained by calculating the reward sequence in each episode in reverse order. To balance the calculation efficiency and model performance, the length of the time series modeling is set to 20.

[0110] ​​​After converting the decision problem into a time series modeling task, considering the need to control the computational overhead while ensuring a high decision frequency, the application adopts a five-layer structure of a lighter GRU model to model local time series characteristics, and introduces a Transformer module after it to enhance the modeling ability of global dependence, improve the strategy performance under the premise of considering the computational efficiency. The model structure includes the following functional modules:

[0111] Input embedding module:

[0112] The state vector of each time step and the cumulative reward are inputted through two fully connected layers to map to the same hidden space dimension , denoted as:

[0113]

[0114] where and are the weight matrices of the two fully connected layers, and are the bias vectors, and are the obtained hidden state vectors. Then the two are concatenated and sent to the layer normalization module to obtain the input feature :

[0115]

[0116] This structure explicitly introduces the cumulative reward target into the input sequence, guiding the model to consider long-term goals and effectively alleviate the short-sightedness of traditional methods.

[0117] Sequence modeling module:

[0118] The above input features are concatenated in time sequence to form an input sequence , which is inputted into a multi-layer gated recurrent unit (GRU) network to model local time dependence and update the hidden state sequence :

[0119]

[0120] The GRU structure can effectively model short-term behavior.

[0121] Transformer module (self-attention + feedforward sublayer):

[0122] To enhance the modeling ability of long-term behavior, a Transformer module is introduced on the GRU output feature, which specifically includes:

[0123] Multi-head self-attention sublayer :

[0124]

[0125] For each time step , the residual connection and layer normalization operation are adopted

[0126]

[0127] The feedforward sublayer:

[0128]

[0129]

[0130] where , , , are the weight matrix and bias of the fully connected layer, and GELU is the activation function. The outputs of all time steps are spliced to obtain the final representation sequence .

[0131] The Transformer module effectively compensates for the memory decay problem of the GRU network in long sequence modeling. In the specific implementation, the GRU network adopts 5-layer stacking, and the Transformer module is set to 1 layer and placed after the GRU module, which improves the long-term strategy modeling capability while ensuring the computing efficiency.

[0132] High-level strategy generation module:

[0133] The final representation is input into the action output layer to generate the strategy action corresponding to the current time step :

[0134]

[0135] where and are the weight matrix and bias of the fully connected layer, is the activation function.

[0136] The output is a high-level action decision instruction {accelerate, uniform speed, decelerate}, which is convenient for deployment to the trajectory planning module.

[0137] During training, since it is a discrete action, the cross-entropy loss is adopted:

[0138] where is the time sequence length, is the number of action categories (accelerate, uniform speed, decelerate, here ), Indicates the first Real-time action labels (one-hot encoding) This indicates that the model is in the model's... Time for the first Predicted probability of action class The cross-entropy loss is calculated over the entire sequence. The goal during training is to minimize the cross-entropy difference between the model's predicted actions and the actual actions.

[0139] It is important to note that in the initial stage of system startup, since the length of the historical sequence is less than the preset 20 steps, the missing parts need to be manually filled in to meet the model input requirements. When the sequence length exceeds 20, a truncation strategy can be adopted, retaining only the most recent 20 steps as model input. Additionally, the cumulative reward in the initial stage can be set to 1-2 (after normalization) to guide the model to make high-quality decisions.

[0140] like Figure 5 As shown, the temporal policy model with the "five-layer GRU + one-layer Transformer" structure exhibits good training performance on the driving dataset, with training loss decreasing rapidly, action prediction accuracy steadily improving, and convergence being relatively fast.

[0141] As shown in the table below, the trained model was tested 50 times in each of the three scenarios: scenario 1, scenario 2, and scenario 3.

[0142] Success times (50 times) Scenario one Scenario two Scenario three Reinforcement learning model 48 42 45 Time series policy model 49 45 47

[0143] Since scenarios one and two primarily involve interaction with pedestrians, while scenario three focuses on interaction with oncoming vehicles, this test can be used to evaluate whether the model possesses multimodal decision-making capabilities. The results show that the model exhibits good decision-making performance across different scenario types.

[0144] Furthermore, the trained unified decision model is exported as an intermediate representation format (ONNX) and deployed on a platform system with autonomous driving control capabilities. The model outputs high-level decision commands, including discrete behavioral signals such as acceleration, constant speed, and deceleration, to guide the update of the target speed. During deployment, the following execution flow is adopted, taking into account the trajectory input structure required by the lower-level motion control module (such as an MPC-based controller):

[0145] Initialization phase: Set initial target velocity Each decision cycle is based on the behavioral instructions output by the current model. ,Right now The speed will be dynamically adjusted according to the following update rules:

[0146]

[0147] wherein is a preset speed adjustment step size, preferably to achieve a smooth and controllable speed transition process, is the target speed at time t.

[0148] Subsequently, the updated target speed is fused into the current reference trajectory point sequence to form a desired trajectory containing position and speed information:

[0149]

[0150] wherein are the horizontal and vertical coordinates of the th trajectory point, respectively. The desired trajectory is tracked by a lower-level controller (e.g. MPC) to effectively map the high-level strategy output to actual control commands of the vehicle.

[0151] While the preferred embodiments of the application have been disclosed in connection with example purposes, one of ordinary skill in the art will readily recognize that various modifications, additions and substitutions are possible without departing from the scope and spirit of the application as disclosed by the appended claims.

Claims

1. A multi-scenario automatic driving decision-making method based on timing modeling, characterized in that The method comprises the following steps: S1, constructing a plurality of automatic driving scenes in a simulation environment, screening traffic participant information within the sensing range of the sensor, and constructing a state vector based on the vehicle coordinate system; S2, introducing a feature aggregation mechanism for vehicle guidance, performing weighted summation on the state vector, dynamically focusing on key traffic participants, while retaining global information; S3, training a plurality of reinforcement learning strategy models respectively using the proximal policy optimization algorithm, and collecting state, action and reward data; introducing a reward reconstruction mechanism and injecting an avoidance behavior reward signal consistent with human driving preferences to achieve multi-scenario reward alignment and build a unified high-quality decision dataset; S4, converting the collected data from a Markov decision process form to a time series modeling form, combining a GRU network and a Transformer module to build a time series strategy model, and obtaining a unified multi-scenario automatic driving decision model; S5, exporting and deploying the trained unified strategy model, the model outputs high-level behavior instructions, and in the deployment process, the expected speed is dynamically adjusted based on the guidance speed and fused into the trajectory for execution by the control module; The feature aggregation mechanism for vehicle guidance in step S2 specifically includes: projecting the ego vehicle state and the state of the traffic participant within the perception range into the same feature space, respectively, to obtain a latent state representation of the ego vehicle and the traffic participant , respectively; Then, the average value of the features of all traffic participants is calculated to extract global information, which is added to the individual features for enhanced modeling; Finally, the vehicle hidden state is taken as the guidance, different weights are given to each traffic participant by scaling the dot product, and the original state vector is weighted and aggregated to form the traffic participant feature vector.

2. The multi-scenario autonomous driving decision-making method based on timing modeling of claim 1, wherein, In step S3, the reward reconstruction mechanism specifically includes: For different driving scenarios, multiple reward indicators are constructed by comprehensively considering factors such as traffic efficiency, safety, and traffic regulations; the original reward function is in the first... Defined in each scenario , for The environmental state at any given time, for Actions at any given moment; The original reward function includes: safety item reward for punishing collision behavior; rule reward for regulating traffic behavior at traffic lights; task completion reward for rewarding smooth driving; timeout penalty item for limiting the number of execution steps; per-step basic penalty item for encouraging task completion as soon as possible; and speed reward item calculated according to the vehicle speed and target speed for encouraging reasonable speed; Then, the proximal policy optimization algorithm is used to train the reinforcement learning strategy model in each scene, and after training, the behavior data of state, action and reward in each scene is collected; To realize the unified modeling across scenes, the reward scales of different scenes need to be aligned, so the reward reconstruction mechanism is introduced; the reward reconstruction mechanism uniformly adopts a standardized reward function Instead of the reward function of each scene So that each reward in the reward function is the same in different scenes; based on this, all the collected samples are revalued according to the unified reward function; During the reassignment process, the avoidance behavior reward signal consistent with human driving preferences is injected to achieve reward reconstruction; for traffic light passing preference, the distance between the vehicle and the stop line is calculated to calculate the speed reward item at each time step and inject the corresponding signal; for pedestrian avoidance, the distance between the vehicle and the pedestrian is calculated to calculate the speed reward item and inject an additional reward signal.

3. The multi-scenario autonomous driving decision-making method based on timing modeling of claim 2, wherein, Each reward is set differently in different scenarios.

4. The multi-scenario autonomous driving decision-making method based on timing modeling of claim 1, wherein, In step S4, the following steps are included: The collected trajectory data is organized in chronological order into a plurality of reward-state-action triplet sequences, in the form of: wherein is the total number of time steps for an episode, is the time step reward value, is the time step environment state, is the time step action; To accommodate the timing modeling requirement, the original data is format converted and each step reward is replaced by the cumulative reward so that there is more dense information, reconstructed in the following form: wherein, is the cumulative reward from time to the end of the episode; The cumulative reward sequence is obtained by calculating the reward sequence in each episode in reverse order.

5. The method of claim 1, wherein, In step S4, the following steps are also included: A time series strategy model combining a GRU and a Transformer module is constructed to model the state-reward sequence and output corresponding high-level decision actions; the model structure includes the following functional modules: Input embedding module: The state vector of each time step is inputted into the network and the cumulative reward The input is mapped to a unified hidden space dimension through two fully connected layers respectively , and then the two are spliced and sent to the layer normalization module to obtain the input feature : Sequence modeling module: The input features are sequentially spliced as an input sequence , input a multi-layer GRU network to model local time dependence, and update the hidden state sequence ; Transformer module: To enhance the modeling ability of long-term behavior, a Transformer module is introduced on the GRU output features to model the global temporal dependencies through multi-head self-attention and feedforward networks, resulting in the final representation ; High-level strategy generation module: final representation input action output layer, generate the policy action corresponding to the current moment ; The output is a high-level action decision instruction (acceleration, uniform speed, deceleration), which is convenient for deployment to the trajectory planning module.

6. The multi-scenario autonomous driving decision-making method based on timing modeling of claim 5, wherein, The GRU network adopts 5-layer stacking, and the Transformer module is set to 1 layer and placed after the GRU module.

7. The method of claim 1, wherein, In step S5, it includes: The unified decision model obtained by training is exported in intermediate representation format and deployed in a platform system with automatic driving control capability. The model output is a high-level decision instruction, including acceleration, uniform speed, and deceleration discrete behavior signals, which are used to guide the update of the target speed. During deployment, the following execution process is adopted in combination with the required trajectory input structure of the lower motion control module: Initialization phase sets initial target speed Behavioral instructions from current model output at each decision cycle Dynamic adjustment of speed according to the following update rule:​ wherein is a preset speed adjustment step size, is a target speed at the instant Subsequently, the updated target speed is fused into the current reference trajectory point sequence, constituting a desired trajectory containing position and velocity information, and tracked by the lower-level controller, effectively mapping the high-level strategy output to actual control commands of the vehicle.

Images