End-to-end automatic driving system in strong interaction scene considering styles of other vehicles

Abstract

The invention belongs to an automobile automatic driving system, and particularly relates to an end-to-end automatic driving system under a strong interaction scene considering other automobile styles. The traffic participant style sensing module extracts driving style characteristics of traffic participants; the features are spliced and then input to the value estimation module and the strategy module, the value output by the value estimation module is used for guiding automatic driving strategy training of the strategy module, and the action output by the strategy module is used for interaction between the automatic driving vehicle and the traffic environment. According to the method, the style implicit features are used as branch parts of model input, and priori cognition can be expressed, so that the training efficiency of a reinforcement learning agent is improved, and the learning processing capability of the agent on a workshop interaction relationship is improved; and meanwhile, the reinforcement learning reward is dynamically corrected according to the driving styles of the traffic participants, so that the intelligent agent can adopt different coping strategies according to different styles of the traffic participants in a strong interaction scene.

Description

Technical Field

[0001] This invention pertains to autonomous driving systems for automobiles, and particularly relates to an end-to-end autonomous driving system for automobiles in highly interactive scenarios that take into account the styles of traffic participants. Background Technology

[0002] With the continuous advancement of automotive intelligence technology, autonomous driving systems have achieved commercial deployment in simple traffic scenarios with minimal vehicle-to-vehicle interaction. However, the performance of existing autonomous driving systems in scenarios with strong interaction remains unsatisfactory. Strong interaction scenarios refer to traffic scenarios where the vehicle interacts with surrounding road users frequently, complexly, and dynamically over time. These scenarios are characterized by highly random and uncertain behavior of road users, and the decision-making and control window of the autonomous driving system is typically short due to constraints such as road changes and traffic rules. For example, ramp merging is a typical strong interaction scenario. In recent years, several autonomous driving strategies for strong interaction scenarios have been proposed, mainly categorized into three types: rule-driven, outcome-imitating, and autonomous exploration. Currently, these three types of autonomous driving strategies for strong interaction scenarios still face the following problems:

[0003] (1) Rule-driven methods usually require the environmental perception system to provide pre-specified explicit scene features. In simple scenarios, these features can provide the information needed by the autonomous driving system. However, in highly interactive scenarios, due to the complexity and time-varying nature of the interaction relationships, these explicit features are often difficult to fully and accurately describe and express the scene information, and cannot extract the deep implicit semantics in the scene. In addition, the generalization ability of rules formulated based on prior knowledge such as expert experience is poor, and there are great safety hazards in highly interactive scenarios not covered by the rules.

[0004] (2) Result-based imitation methods learn from driver data to enable autonomous driving systems to produce driving behaviors consistent with those of drivers. However, in the real world, data samples for highly interactive scenarios are scarce and the interaction patterns are simple. This easily leads to incomplete and insufficient extraction of interaction features by result-based imitation methods that heavily rely on the quality and quantity of training samples. Furthermore, the quality of driver data varies greatly, and indiscriminate learning of driver behavior may result in poor performance of the autonomous driving system. In addition, the fixed imitation learning paradigm makes it difficult to incorporate the uncertainty of traffic participant motion into the loss function as a constraint term. Therefore, result-based imitation methods have significant limitations in real-world applications.

[0005] (3) Autonomous exploration methods use reinforcement learning as the basic paradigm. They train autonomous driving strategies through the interaction between the vehicle and the environment. With proper design of the reward function, autonomous driving strategies can achieve performance that surpasses that of human drivers. However, due to the inherent limitations of reinforcement learning itself, these methods are usually less efficient in training under complex and highly interactive scenarios, and the strategies are difficult to converge.

[0006] Furthermore, current rule-driven, result-imitating, and autonomous exploration methods generally treat traffic participants who interact with the vehicle in a uniform manner, without fully considering the heterogeneous feedback generated by traffic participants with different styles to the vehicle's behavior. This results in the vehicle being unable to take targeted driving behaviors for traffic participants with different styles, which seriously affects the performance of autonomous driving in highly interactive scenarios.

[0007] Therefore, how to develop an autonomous driving system that can comprehensively extract and process the interaction information between the vehicle and traffic participants, fully consider the motion uncertainty of traffic participants, and has high policy model training efficiency and convergence speed, while effectively dealing with traffic participants of different styles, has become a core technical challenge that urgently needs to be overcome in the field of intelligent vehicle technology. Summary of the Invention

[0008] To address the aforementioned technical problems, this invention provides an end-to-end autonomous driving system for strong interactive scenarios that consider the style of other vehicles, including an image feature extraction module, a traffic participant style perception module, a value estimation module, and a strategy module.

[0009] The image feature extraction module receives the original image data, performs convolution processing on the image, stitches the image features together, and then performs downsampling and dimensionality reduction to obtain the final image features.

[0010] The traffic participant style perception module establishes the spatial features of a three-dimensional class graph. The three dimensions correspond to the vertical arrangement position, horizontal arrangement position, and basic composition of the state of the traffic participants. The convolutional network is used to extract the feature encoding of each spatial state within the historical time window. The temporal convolutional network combined with the attention mechanism is used to extract the implicit features of driving style. After processing by linear layers and the softmax function, the final driving style features of the traffic participants are obtained.

[0011] Image features and driving style features of traffic participants are concatenated and then input into the value estimation module and the policy module. The value output by the value estimation module is used to guide the training of the autonomous driving policy by the policy module, and the actions output by the policy module are used for the interaction between the autonomous vehicle and the traffic environment.

[0012] Furthermore, in the image feature extraction module, image information from different angles is input, which consists of single-frame images from three perspectives—front, left rear, and right rear—at the same moment. composition, , , These represent the number of channels, height, and width of the image, respectively; a two-dimensional convolutional network is used to process the image input, and the number of output channels of the two-dimensional convolutional network is set to... The kernel size is The sliding step size is The outputs from the three perspectives after convolution are as follows: , , The height and width of the image convolution output, where , After flattening, the output shows the one-dimensional image features from each of the three perspectives. The two-dimensional convolutional network outputs dimensions. After stitching, a fully connected layer is used for downsampling to finally obtain the image features at time step t. , This represents the dimension of the image features output by the linear fully connected layer.

[0013] Furthermore, in the traffic participant style perception module, the temporal convolutional network, combined with an attention mechanism, uses a multi-layer temporal convolutional network to extract causal temporal features and a one-dimensional convolution to extract direct temporal features; it uses a linear fully connected layer to convert the expanded direct temporal features into value vectors and key vectors, and uses a linear fully connected layer to convert the expanded causal temporal features into query vectors, thereby enhancing the temporal features by utilizing the causal relationships within the temporal features.

[0014] Furthermore, the traffic participant style perception module collects spatiotemporal information of the vehicle and surrounding traffic participants, constructs historical information of the vehicle and surrounding traffic participants based on the vehicle's basic state, builds a three-dimensional spatial state of a class graph, and selects eight traffic participants around the vehicle EV as: Front Vehicle (FV), Rear Vehicle (ReV), Left Vehicle (LV), Right Vehicle (RV), Left Front Vehicle (LFV), Right Front Vehicle (RFV), Left Rear Vehicle (LRV), and Right Rear Vehicle (RRV). The motion state of surrounding traffic participants Composition, described as:

[0015]

[0016] in, It represents the length of the time window, t is the time step, and t0 is the current time; it also represents the motion state of traffic participants. including the longitudinal speed of traffic participants Lateral speed of traffic participants longitudinal acceleration of traffic participants Lateral acceleration of traffic participants longitudinal agility of traffic participants Longitudinal distance of traffic participants relative to the main vehicle Lateral distance of traffic participants relative to the main vehicle There are a total of 7 dimensions, namely the motion state dimension of traffic participants. The motion state of traffic participants writing:

[0017]

[0018] A two-dimensional convolutional network is used to process spatial states and extract spatial interaction relationships between adjacent traffic participants; the number of output channels of the two-dimensional convolution is... The kernel size is K CS =2, sliding step size is S CS =1, and the output 3D tensor is subjected to layer normalization, linear correction and flattening to obtain a one-dimensional spatial feature encoding. Among them, the spatial feature encoding dimension ;

[0019] Forming spatial feature encoding historical time series ,in Then, a temporal convolutional network combined with an attention mechanism is used for processing. The temporal convolutional network combined with the attention mechanism combines causal convolution and dilated convolution in the temporal convolutional network. The dilated convolution expands the sampling interval of the convolution kernel on the basis of causal convolution, and flattens the encoding results of traditional one-dimensional convolution and multi-layer dilated causal convolution to obtain direct temporal features. and causal temporal characteristics ,in , These are the direct temporal feature dimension and the causal temporal feature dimension, respectively; in the attention mechanism, the linearly changing direct temporal feature is a value vector. and key vector Linearly changing causal time-series features are used as query vectors To enhance the temporal characteristics from a causal perspective, among which The dimension of the attention ternary vector; the value vector Key vector and query vector It is expressed as follows:

[0020]

[0021]

[0022]

[0023] in, , These are the linear projection weight matrices for the value vector, key vector, and query vector, respectively. and These correspond to the dimensions of the feature encoding after one-dimensional convolution and dilated causal processing, respectively.

[0024] Furthermore, in the attention mechanism, regarding time steps... First, calculate the query vector. With key vector The dot product between them; then, based on the dimension of the attention triple vector. The dot product result is scaled; the scaled score is then normalized using the Softmax function to obtain the time step. The importance weight vector of time-series feature encoding based on causal mechanisms :

[0025]

[0026] Based on importance weight vector , value vector By weighting by importance, the implicit features of driving style are obtained. :

[0027]

[0028] Ultimately, the implicit characteristics of driving style will be... The input is fed into a linear fully connected layer, and the result is... DS's result-oriented driving style for all traffic participants at all times t .

[0029] Furthermore, the traffic participant style perception module is trained using deep supervised learning. To handle situations where there are no traffic participants in a specific direction, the absence of traffic participants is included in the output of the traffic participant style perception module, classifying driving styles into four categories: conservative, general, aggressive, and no traffic participants. The goal of driving style recognition is to identify the driving styles of adjacent traffic participants, assuming that the number of adjacent traffic participants for the vehicle is 8. Eight sets of Softmax activation functions are set at the end of the temporal convolutional network combined with the attention mechanism module, each with 4 branches, meaning the output dimension of the fully connected output layer is 32. Finally, the model's final driving style recognition is the one with the highest probability value in the output dimension. During training, the loss function L... TA The cross-entropy loss function is used and defined as follows:

[0030]

[0031] in, For the sample size, The total number of categories, For the sample In category The unique and genuine label on the top. Samples output by the model In category The predicted probability.

[0032] Furthermore, the value estimation module and policy module are built based on the MetaDrive simulation interactive environment, and reinforcement learning is used to optimize the autonomous driving policy; a first-order Markov decision process is established. ,in, For state space, For the action space, Let be the state transition probability. For cumulative rewards; the specific composition of the tuple is as follows:

[0033] (1) Time step status From the original information of a single frame image and implicit characteristics of the driving styles of adjacent traffic participants. Composition; Time step status Writing in key-value pair form:

[0034] Image information: Implicit stylistic features:

[0035] (2) Time step Action output This refers to the direct lateral and longitudinal control quantities of intelligent vehicles, namely the accelerator pedal opening. Brake pedal opening and steering wheel angle Time step Action output writing:

[0036]

[0037] (3) The state transition probability is determined by the environment settings. In the MetaDrive simulation interaction environment, the intelligent vehicle performs actions based on the current state. Then, the onboard sensors collect information and the behavior of surrounding traffic participants causes a state transition. The state transition probability is defined as follows: , The state at the next moment;

[0038] (4) Time step Rewards It consists of several sub-reward items, represented as follows:

[0039]

[0040] in, Let be the weight matrix of the sub-reward vector. This is the sub-reward vector at the current moment;

[0041] The rewards in reinforcement learning consist of goal-oriented rewards, evaluative rewards, and penalties. Goal-oriented rewards aim to help the vehicle reach its destination quickly; evaluative rewards aim to improve the safety and comfort of the vehicle and other road users; penalties serve as the baseline for environmental exploration, ensuring vehicle safety and adherence to traffic rules. Upon being penalized, the intelligence is reset. The sub-reward vector at the current time step... Specifically defined as:

[0042]

[0043] in, As a displacement reward, To maintain the reward for speed, A reward will be given for successfully reaching the destination. As a security reward, As an aggressive reward, As a comfort reward, This is the penalty vector.

[0044] Furthermore, the specific definitions of each reward in the sub-reward vector at the current moment are as follows:

[0045] a. Displacement bonus The system is designed to provide a positive reward when a vehicle is centered in the lane and moving towards its destination. This is a goal-oriented reward and is defined as follows:

[0046]

[0047] in, This represents the longitudinal displacement of the vehicle within a single step. The lateral keeping coefficient is used to penalize a vehicle after it deviates from the center of the lane. This is the directional coefficient, which is penalized when a vehicle travels in the opposite direction.

[0048] b. Speed ​​Maintenance Bonus The system is designed to reward a vehicle for maintaining a relatively high speed in the correct direction; this is a targeted reward and is defined as follows:

[0049]

[0050] in, The current speed of the vehicle. This is the maximum speed allowed in the scenario.

[0051] c. Success Reward The reward for a vehicle successfully reaching its navigation destination is sparse but high, constituting a goal-oriented reward, and is defined as follows:

[0052]

[0053] in, This is the constant reward received by the vehicle after successfully reaching its destination.

[0054] d. Security rewards This invention characterizes the impact of traffic participant behavior on vehicle safety; it utilizes a risk field to assess vehicle driving safety; and it measures the risk field strength generated by surrounding traffic participants relative to the vehicle. Defined as:

[0055]

[0056] in, This refers to traffic participants surrounding the vehicle, specifically those in front of, behind, to the left, to the right, to the left front, to the left rear, to the right front, and to the right rear. ; For the vehicle and other traffic participants in the Frenet coordinate system of traffic participant j Distance vector between them; This is the distance scaling factor. and For traffic participants Acceleration along its direction of travel and normal acceleration; The peak field strength of the risk field occurs among traffic participants. The location of the center of mass; and They are respectively With traffic participants The angle between the direction of travel and the normal direction; and These are the acceleration coefficients in the forward direction and normal direction; For traffic participants The quality;

[0057] The driving safety reward function is constructed as follows:

[0058]

[0059] in, The threshold field strength at which a driver will not respond to a risk.

[0060] e. Aggressive Rewards Corresponding to safety rewards, it represents the impact of the vehicle's behavior on surrounding traffic participants; the concept of constructing safety rewards is used to construct aggression rewards:

[0061]

[0062] in, The risk level posed by a vehicle to other road users.

[0063] f. Comfort reward This represents the driver's need for driving comfort, and the comfort reward is defined as:

[0064]

[0065] in, The longitudinal abruptness of the vehicle.

[0066] g. Penalty vector This corresponds to various situations that will reset the environment, including driving off the road, vehicle collision, object collision, and pedestrian collision. The penalty vector is represented as:

[0067]

[0068] in, , , , The penalties are for driving off the road, vehicle collision, object collision, and pedestrian collision, respectively, and are all constant terms.

[0069] Furthermore, the policy module and the value estimation module use a dual-delay deep deterministic policy gradient reinforcement learning algorithm. The policy module and the value estimation module are respectively a policy network and two action value estimation networks. The policy network and the action value estimation network share a feature extraction head. The network objects trained include the image feature extraction module, the policy module and the value estimation module. The traffic participant style perception module is pre-trained by deep supervised learning.

[0070] In terms of network design, there is one real-policy network and two real-action value estimation networks, each with a corresponding target network, totaling six networks. During training, the parameters of each neural network are first initialized, with the parameters of the real-policy network and the target policy network corresponding to... , The parameters of the two sets of real-action value estimation networks and target action value estimation networks correspond to respectively Real-world strategy network Based on the time state Output Action Environmental reception After state transition to And obtain time rewards. This forms a state transition pair. Store it in the experience pool Medium; Experience Pool Once a certain scale is reached, [the experience pool will be used]. Extracting mini-batch state transition pairs for real-world action value estimation networks Based on the time state and actions Output action value estimate Target action value estimation network Based on the state at the next moment And the next action estimated by the target policy network. Output the estimated value of the action at the next moment. .

[0071] Furthermore, in calculating actions Normally distributed noise is added and truncated according to the actual range of motion to enhance the exploration capabilities of intelligent vehicles. The formula is as follows:

[0072]

[0073] in, For real-world policy networks in state The next action to re-output, To conform to the standard deviation Gaussian exploration noise with a normal distribution.

[0074] Estimate the next action of the target policy network The surrounding region is subjected to fuzzy fitting by adding normally distributed noise and truncating the noise in advance, as shown in the following formula:

[0075]

[0076] in, For the target policy network in state The next action to re-output, The noise amplitude clipping threshold. It is the raw exploration noise sampled from a normal distribution. This is for effective exploration of noise after trimming.

[0077] Calculate the value of the target action The formula is as follows:

[0078] ;

[0079] The update process of the real-world action value estimation network is a process of minimizing the temporal difference error, which represents the difference between the Q-value calculated by the real-world network and the target Q-value. The parameter optimization method is gradient descent. The final loss function of the action value estimation network is... The calculation formula is as follows:

[0080]

[0081] in, For time steps The reward The number of unlabeled samples used for fine-tuning in reinforcement learning; This is the estimated value of the current action.

[0082] The purpose of establishing the policy network is to select the action with the highest value. The Q-value estimated by the action value estimation network guides the policy network update. The parameter optimization method uses gradient ascent, and the objective function is... as follows:

[0083]

[0084] in, For real-world policy networks, based on state The action of re-outputting.

[0085] For updates to the target network, a soft update approach is adopted, incorporating a learning rate. During each update, a mixture of the old and new parameters is used, and the learning rate is... The magnitude of the soft update is determined by the following formula:

[0086]

[0087] The update of the real-world policy network is delayed compared to the real-world action value estimation network, which improves the stability of the training process.

[0088] The beneficial effects of this invention are:

[0089] Compared with rule-driven methods, this invention adopts an end-to-end control approach, using original images from multi-angle cameras as the main input to the autonomous driving strategy and implicit features of the driving styles of surrounding traffic participants as branch inputs. This helps the agent extract deep implicit semantics in the scene and improves the agent's cognitive and adaptive abilities in highly interactive environments.

[0090] Compared with result-imitating methods, the agent in this invention explores a set of high-fidelity, highly interactive virtual scenarios and deploys traffic participant agents with random driving styles within the scenarios. This can better mimic highly interactive environments with drivers of various styles, greatly improving the comprehensiveness of the samples and helping to establish intelligent driving strategies for highly interactive scenarios with stronger generalization ability and wider coverage.

[0091] Compared with other autonomous exploration methods, this invention utilizes style implicit features as a branch of the model input. These style implicit features can express prior cognition, thereby accelerating the training efficiency of the reinforcement learning agent and improving the agent's ability to learn and process vehicle-to-vehicle interactions. At the same time, the reinforcement learning reward is dynamically adjusted according to the driving style of traffic participants, enabling the agent to adopt different coping strategies according to the different styles of traffic participants in highly interactive scenarios. Attached Figure Description

[0092] Figure 1 This is a schematic diagram of the overall logical framework of the present invention;

[0093] Figure 2 This is a schematic diagram of the image feature extraction and fusion logic framework for multi-angle original images according to the present invention;

[0094] Figure 3 This is a schematic diagram of the logical framework of the traffic participant style perception module of the present invention;

[0095] Figure 4 This is a logical schematic diagram of the temporal convolutional network combined with the attention mechanism of the present invention;

[0096] Figure 5 This is a schematic diagram illustrating the enhanced interaction between the value estimation module and the strategy module of the present invention.

[0097] Figure 6 This is a logical schematic diagram of the dual-delay deep deterministic strategy gradient algorithm of the present invention. Detailed Implementation

[0098] This invention provides an end-to-end autonomous driving system for scenarios with strong interaction that consider the style of other vehicles, such as... Figure 1 As shown, it includes an image feature extraction module, a traffic participant style perception module, a value estimation module, and a strategy module;

[0099] The image feature extraction module receives the original image data, performs convolution processing on the image, stitches the image features together, and then performs downsampling and dimensionality reduction to obtain the final image features.

[0100] The traffic participant style perception module establishes the spatial features of a three-dimensional class graph. The three dimensions correspond to the vertical arrangement position, horizontal arrangement position, and basic composition of the state of the traffic participants, respectively. The convolutional network is used to extract the feature encoding of each spatial state within the historical time window. The temporal convolutional network combined with the attention mechanism module is used to extract the implicit features of driving style. After processing by linear layers and the softmax function, the final driving style features of the traffic participants are obtained.

[0101] Image features and driving style features of traffic participants are concatenated and then input into the value estimation module and the policy module. The value output by the value estimation module is used to guide the training of the autonomous driving policy by the policy module, and the actions output by the policy module are used for the interaction between the autonomous vehicle and the traffic environment.

[0102] Furthermore, such as Figure 2 As shown, in the image feature extraction module, image information from different angles is input. This image information consists of single-frame images from three perspectives—front, left rear, and right rear—at the same moment. composition, , , These represent the number of channels, height, and width of the image, respectively; a two-dimensional convolutional network is used to process the image input, and the number of output channels of the two-dimensional convolutional network is set to... The kernel size is The sliding step size is The outputs from the three perspectives after convolution are as follows: , , The height and width of the convolution output are given by , where , After flattening, the output shows the one-dimensional image features from each of the three perspectives. The two-dimensional convolutional network outputs dimensions. After stitching, the integrated one-dimensional image features are obtained. Then, a fully connected layer is used for downsampling to finally obtain the image features at time step t. , This represents the dimension of the image features output by the linear fully connected layer.

[0103] Furthermore, in the traffic participant style perception module, the temporal convolutional network combined with the attention mechanism module uses a multi-layer temporal convolutional network to extract causal temporal features and a one-dimensional convolution to extract direct temporal features; it uses a linear fully connected layer to convert the expanded direct temporal features into value vectors and key vectors, and uses a linear fully connected layer to convert the expanded causal temporal features into query vectors, and uses the causal relationships within the temporal features to enhance the temporal features.

[0104] Furthermore, such as Figure 3 As shown, the traffic participant style perception module collects spatiotemporal information about the vehicle and surrounding traffic participants, constructs historical information about the vehicle and surrounding traffic participants based on the vehicle's basic state, and builds a three-dimensional spatial state of the class graph. The spatial relationship between the vehicle and surrounding traffic participants is as follows: Figure 3Eight traffic participants surrounding the ego vehicle (EV) are selected and named: front vehicle (FV), rear vehicle (ReV), left vehicle (LV), right vehicle (RV), left-front vehicle (LFV), right-front vehicle (RFV), left-rear vehicle (LRV), and right-rear vehicle (RRV). The three-dimensional spatial state is as follows. The motion state of surrounding traffic participants Composition, described as:

[0105]

[0106] in, It represents the length of the time window, t is the time step, and t0 is the current time; it also represents the motion state of traffic participants. including the longitudinal speed of traffic participants Lateral speed of traffic participants longitudinal acceleration of traffic participants Lateral acceleration of traffic participants longitudinal agility of traffic participants Longitudinal distance of traffic participants relative to the main vehicle Lateral distance of traffic participants relative to the main vehicle There are a total of 7 dimensions, namely the motion state dimension of traffic participants. The motion state of traffic participants writing:

[0107] A two-dimensional convolutional network is used to process spatial states and extract spatial interaction relationships between adjacent traffic participants; the number of output channels of the two-dimensional convolution is... The kernel size is K CS =2, sliding step size is S CS =1, and the output 3D tensor is subjected to layer normalization, linear correction and flattening to obtain a one-dimensional spatial feature encoding. Among them, the spatial feature encoding dimension ;

[0108] Forming spatial feature encoding historical time series ,in Then, a Temporal Convolutional Network Combined with Attention Mechanism (TCN-A) module is used for processing. This TCN-A module combines causal convolution and dilated convolution within the temporal convolutional network. The temporal convolutional network is essentially a one-dimensional convolutional network, and its structure is as follows: Figure 4 As shown. Traditional convolutional methods for extracting features from sequential data are bidirectional, while real-world temporal features only depend on the time series preceding the current time step. To address this issue, temporal convolutional networks introduce causal convolution, which ensures that the output at the current moment depends only on current and past information. Furthermore, traditional convolutional methods extract features based on the convolution kernel; when processing time series data, multiple convolutional layers are needed to capture dependencies over longer time spans, consuming computational resources. Therefore, temporal convolutional networks utilize dilated convolution to solve this problem. In dilated convolution, each layer extracts information from the previous layer in a leapfrog manner. Dilated convolution expands the sampling interval of the convolution kernel based on causal convolution, resulting in a larger receptive field for the output feature map and richer information. Flattening the encoding results of traditional one-dimensional convolution and multi-layer dilated causal convolution yields direct temporal features. and causal temporal characteristics ,in , These are the dimensions of direct temporal features and causal temporal features, respectively; in the attention mechanism, the linearly changing direct temporal features are value vectors. and key vector Linearly changing causal time-series features are used as query vectors Further enhancing the temporal characteristics from a causal perspective, among which The dimension of the attention ternary vector; the value vector Key vector and query vector It is expressed as follows:

[0109]

[0110]

[0111]

[0112] in, , These are the linear projection weight matrices for the value vector, key vector, and query vector, respectively. and These correspond to the dimensions of the feature encoding after one-dimensional convolution and dilated causal processing, respectively.

[0113] In attention mechanisms, the focus is on time steps. First, calculate the query vector. With key vector The dot product between them; then, based on the dimension of the attention triple vector. The dot product result is scaled; the scaled score is then normalized using the Softmax function to obtain the time step. The importance weight vector of time-series feature encoding based on causal mechanisms :

[0114]

[0115] Based on importance weight vector , value vector By weighting by importance, the implicit features of driving style are obtained. :

[0116]

[0117] Ultimately, the implicit characteristics of driving style will be... The input is fed into a linear fully connected layer, and the result is... DS's result-oriented driving style for all traffic participants at all times t .

[0118] Furthermore, the traffic participant style perception module is trained using deep supervised learning. To address situations where there are no traffic participants in a specific direction of the scene, the absence of traffic participants is added to the output of the traffic participant style perception module. Thus, this invention categorizes driving styles into four types: conservative, general, aggressive, and no traffic participants. The goal of driving style recognition is to identify the driving styles of adjacent traffic participants. Assuming the number of adjacent traffic participants for the vehicle is 8, 8 sets of Softmax activation functions are set at the end of the temporal convolutional network combined with the attention mechanism module, and each Softmax activation function corresponds to 4 branches, meaning the output dimension of the fully connected output layer is 32. Finally, the model's final driving style recognition is the one with the highest probability value in the output dimension. During training, the loss function L... TA The cross-entropy loss function is used and defined as follows:

[0119]

[0120] in, For the sample size, The total number of categories, For the sample In category The unique and genuine label on the top. Samples output by the model In category The predicted probability.

[0121] Furthermore, such as Figure 5 As shown, the value estimation module and policy module are built based on the MetaDrive simulation interactive environment, and use reinforcement learning to optimize the autonomous driving policy; a first-order Markov decision process is established. ,in, For state space, For the action space, Let be the state transition probability. For cumulative rewards; the specific composition of the tuple is as follows:

[0122] (1) Time step status From the original information of a single frame image and implicit characteristics of the driving styles of adjacent traffic participants. The reason for selecting implicit features of driving style instead of directly using the historical states of traffic participants and the vehicle is to ensure the Markov property of the process, that is, the future state depends only on the current state and is independent of the historical state. The reason for not directly selecting traffic participant styles as states is that implicit features of driving style contain more spatiotemporal interaction information between vehicles, further enhancing the judgment ability of the reinforcement learning backbone network; time step. status Writing in key-value pair form:

[0123] Image information: Implicit stylistic features:

[0124] (2) This invention is an end-to-end autonomous driving strategy, therefore the time step Action output This refers to the direct lateral and longitudinal control quantities of intelligent vehicles, namely the accelerator pedal opening. Brake pedal opening and steering wheel angle Time step Action output writing:

[0125]

[0126] (3) The state transition probability is determined by the environment settings. In the MetaDrive simulation interaction environment, the intelligent vehicle performs actions based on the current state. Then, the onboard sensors collect information and the behavior of surrounding traffic participants causes a state transition. The state transition probability is defined as follows: , The state at the next moment;

[0127] In the specific virtual environment settings, the driver models of traffic participants are Intelligent Driver Models (IDMs). To mimic the different styles of traffic participant vehicles, random parameters are set for the IDMs. To reproduce a highly interactive environment, a high traffic density is set for the environment, and a high lane-changing frequency is set for some driver models.

[0128] (4) Time step Rewards It consists of several sub-reward items, represented as follows:

[0129]

[0130] in, Let be the weight matrix of the sub-reward vector. This represents the sub-reward vector at the current moment. The intelligent driving task can be described as enabling the vehicle to successfully reach its destination while ensuring the safety and comfort of the vehicle and surrounding vehicles. Therefore, the reward in this invention's reinforcement learning consists of a goal-oriented reward, an evaluative reward, and a penalty. The goal-oriented reward aims to enable the vehicle to reach its destination quickly; the evaluative reward aims to improve the safety and comfort of the vehicle and other traffic participants; the penalty is the bottom line for environmental exploration, ensuring vehicle safety and compliance with traffic rules. After being penalized, the intelligent system is reset. The sub-reward vector at the current moment... Specifically defined as:

[0131]

[0132] in, As a displacement reward, To maintain the reward for speed, A reward will be given for successfully reaching the destination. As a security reward, As an aggressive reward, As a comfort reward, This is the penalty vector.

[0133] Furthermore, the specific definitions of each reward in the sub-reward vector at the current moment are as follows:

[0134] a. Displacement bonus The system is designed to provide a positive reward when a vehicle is centered in the lane and moving towards its destination. This is a goal-oriented reward and is defined as follows:

[0135]

[0136] in, This represents the longitudinal displacement of the vehicle within a single step. The lateral keeping coefficient is used to penalize a vehicle after it deviates from the center of the lane. This is the directional coefficient, which is penalized when a vehicle travels in the opposite direction.

[0137] b. Speed ​​Maintenance Bonus The system is designed to reward a vehicle for maintaining a relatively high speed in the correct direction; this is a targeted reward and is defined as follows:

[0138]

[0139] in, The current speed of the vehicle. This is the maximum speed allowed in the scenario.

[0140] c. Success Reward The reward for a vehicle successfully reaching its navigation destination is sparse but high, constituting a goal-oriented reward, and is defined as follows:

[0141]

[0142] in, This is the constant reward received by the vehicle after successfully reaching its destination.

[0143] d. Security rewards This invention characterizes the impact of traffic participant behavior on vehicle safety; it utilizes a risk field to assess vehicle driving safety; and it measures the risk field strength generated by surrounding traffic participants relative to the vehicle. Defined as:

[0144]

[0145] in, This refers to traffic participants surrounding the vehicle, specifically those in front of, behind, to the left, to the right, to the left front, to the left rear, to the right front, and to the right rear. ; For the vehicle and other traffic participants in the Frenet coordinate system of traffic participant j Distance vector between them; This is the distance scaling factor. and For traffic participants Acceleration along its direction of travel and normal acceleration; The peak field strength of the risk field occurs among traffic participants. The location of the center of mass; and They are respectively With traffic participants The angle between the direction of travel and the normal direction; and These are the acceleration coefficients in the forward direction and normal direction; For traffic participants The quality;

[0146] Human drivers' risk perception is truncated. If the electric field strength exerted on their vehicle by a traffic participant is below a certain threshold, the driver will perceive that participant as posing no safety threat. For aggressive traffic participants, the risk threshold for the vehicle is lower; for conservative traffic participants, the risk threshold is higher. Therefore, the driving safety reward function is constructed as follows:

[0147]

[0148] in, The threshold field strength at which a driver will not respond to a risk.

[0149] e. Aggressive Rewards Corresponding to safety rewards, it represents the impact of the vehicle's behavior on surrounding traffic participants; the concept of constructing safety rewards is used to construct aggression rewards:

[0150]

[0151] in, The risk threshold posed by a vehicle to traffic participants varies. For aggressive traffic participants, the risk threshold is higher, while for conservative traffic participants, the risk threshold is lower.

[0152] f. Comfort reward This represents the driver's need for driving comfort, and the comfort reward is defined as:

[0153]

[0154] in, The longitudinal abruptness of the vehicle.

[0155] g. Penalty vector This corresponds to various situations that will reset the environment, including driving off the road, vehicle collision, object collision, and pedestrian collision. The penalty vector is represented as:

[0156]

[0157] in, , , , The penalties are for driving off the road, vehicle collision, object collision, and pedestrian collision, respectively, and are all constant terms.

[0158] Furthermore, the policy module and value estimation module employ the Twin Delayed Deep Deterministic Policy Gradient (TD3) reinforcement learning algorithm. The policy module and value estimation module each consist of one policy network and two action value estimation networks, respectively. The policy network and action value estimation networks share a common feature extraction head. The trained network objects include the image feature extraction module, the policy module, and the value estimation module. The traffic participant style perception module is pre-trained using deep supervised learning. In this reinforcement learning algorithm, the Q-values ​​estimated by the two value estimation networks are established, and the temporal difference (TD) error is calculated separately to update the network. This reduces the algorithm's sensitivity to hyperparameters and is beneficial for solving optimization problems in high-dimensional continuous spaces.

[0159] In terms of network design, there is one real-policy network and two real-action value estimation networks, each with a corresponding target network, totaling six networks. During training, the parameters of each neural network are first initialized; the parameters of the real-policy network and the target policy network correspond to... , The parameters of the two sets of real-action value estimation networks and target action value estimation networks correspond to respectively Real-world strategy network Based on the time state Output Action Environmental reception After state transition to And obtain time rewards. This forms a state transition pair. Store it in the experience pool Medium; Experience Pool Once a certain scale is reached, [the experience pool will be used]. Extracting mini-batch state transition pairs for real-world action value estimation networks Based on the time state and actions Output action value estimate Target action value estimation network Based on the state at the next moment And the next action estimated by the target policy network. Output the estimated value of the action at the next moment. .

[0160] Furthermore, in order to enhance the exploratory capabilities of intelligent vehicles' intelligent agents, computational actions will be performed. Normally distributed noise is added, and the range is truncated according to the actual operating range, as shown in the following formula:

[0161]

[0162] in, For real-world policy networks in state The next action to re-output, To conform to the standard deviation Gaussian exploration noise with a normal distribution.

[0163] To ensure that similar actions have similar values, it is necessary to estimate the next-time action of the target policy network. The surrounding area is then subjected to fuzzy fitting, also by adding normally distributed noise, but the noise needs to be truncated beforehand, as shown in the following formula:

[0164]

[0165] in, For the target policy network in state The next action to re-output, The noise amplitude clipping threshold. It is the raw exploration noise sampled from a normal distribution. This is for effective exploration of noise after trimming.

[0166] Calculate the value of the target action When calculating the target action value, a target network action value below a certain preset threshold is selected. This alleviates the problem of overestimation of action value. The formula is as follows:

[0167]

[0168] The update process of the real-world action value estimation network is a process of minimizing the temporal difference error. The temporal difference error represents the gap between the action value calculated by the network and the target action value. The parameter optimization method is gradient descent. Finally, the loss function of the action value estimation network... The calculation formula is as follows:

[0169]

[0170] in, The number of unlabeled samples used for fine-tuning in reinforcement learning; This is the estimated value of the current action.

[0171] The purpose of establishing the policy network is to select the highest-value action. The action value estimated by the action value estimation network guides the policy network update. The parameter optimization method uses gradient ascent, and the objective function is as follows:

[0172] To reduce fluctuations in the target action value output during training, it is necessary to limit the update magnitude and frequency of the target network. For the update of the target network, a soft update approach is adopted, incorporating a learning rate. During each update, instead of using all the new parameters, a mixture of old and new values ​​is used for the hyperparameters. The magnitude of the soft update is determined by the following formula:

[0173] in, For real-world policy networks, based on state The action of re-outputting.

[0174] For updates to the target network, a soft update approach is adopted, incorporating a learning rate. During each update, a mixture of the old and new parameters is used, and the learning rate is... The magnitude of the soft update is determined by the following formula:

[0175]

[0176] In addition, the updates of the real policy network are usually delayed compared to the real action value estimation network, which helps to improve the stability of the training process.

[0177] This invention directly uses raw images from multi-angle cameras as the main input for autonomous driving strategies and directly outputs longitudinal and lateral control quantities such as steering wheel angle, acceleration, and brake pedal opening. This end-to-end architecture avoids the problem of insufficient scene information extraction caused by manually specifying explicit input features.

[0178] This invention establishes a set of high-fidelity, highly interactive virtual scenarios and deploys traffic participant agents with random driving styles within these scenarios. Based on a reinforcement learning framework, the autonomous vehicle agent completes policy training in different environments through autonomous exploration, and the performance of autonomous driving is ensured through the rational design of the reward function.

[0179] This invention designs a driving style recognition system based on a temporal convolutional neural network to extract implicit features and style types of different traffic participants' driving styles. Utilizing these implicit style features as a branch of the model input, which expresses prior cognition, accelerates the training efficiency of the reinforcement learning agent and improves its ability to learn and process vehicle-to-vehicle interactions. Simultaneously, the reinforcement learning reward is dynamically adjusted based on the driving style of each traffic participant, setting a lower safety threshold and a higher aggression threshold for aggressive traffic participants, and a higher safety threshold and a lower aggression threshold for conservative traffic participants. This enables intelligent vehicles to adopt different response strategies based on the different styles of traffic participants in highly interactive scenarios. This design allows intelligent vehicles to adopt different styles for different highly interactive processes, contributing to a comprehensive improvement in intelligent driving performance in highly interactive scenarios.

Claims

1. An end-to-end autonomous driving system considering strong interaction scenarios with other vehicles, characterized in that: It includes an image feature extraction module, a traffic participant style perception module, a value estimation module, and a strategy module; The image feature extraction module receives the original image data, performs convolution processing on the image, stitches the image features together, and then performs downsampling and dimensionality reduction to obtain the final image features. The traffic participant style perception module establishes the spatial features of a three-dimensional class graph. The three dimensions correspond to the vertical arrangement position, horizontal arrangement position, and basic composition of the state of the traffic participants. The convolutional network is used to extract the feature encoding of each spatial state within the historical time window. The temporal convolutional network combined with the attention mechanism is used to extract the implicit features of driving style. After processing by linear layers and the softmax function, the final driving style features of the traffic participants are obtained. Image features and driving style features of traffic participants are concatenated and then input into the value estimation module and the policy module. The value output by the value estimation module is used to guide the training of the autonomous driving policy by the policy module, and the actions output by the policy module are used for the interaction between the autonomous vehicle and the traffic environment.

2. The end-to-end autonomous driving system for a strong interaction scenario considering the style of other vehicles, as described in claim 1, is characterized in that: In the image feature extraction module, image information from different angles is input. This image information consists of single-frame images from three perspectives—front, left rear, and right rear—at the same moment. composition, , , These represent the number of channels, height, and width of the image, respectively; a two-dimensional convolutional network is used to process the image input, and the number of output channels of the two-dimensional convolutional network is set to... The kernel size is The sliding step size is The outputs from the three perspectives after convolution are as follows: , , The height and width of the image convolution output, where , After flattening, the output shows the one-dimensional image features from each of the three perspectives. The two-dimensional convolutional network outputs dimensions. After stitching, a fully connected layer is used for downsampling to finally obtain the image features at time step t. , This represents the dimension of the image features output by the linear fully connected layer.

3. The end-to-end autonomous driving system for a strong interaction scenario considering the style of other vehicles, as described in claim 1, is characterized in that: In the traffic participant style perception module, the temporal convolutional network combines an attention mechanism, uses a multi-layer temporal convolutional network to extract causal temporal features, and uses one-dimensional convolution to extract direct temporal features; it uses a linear fully connected layer to convert the expanded direct temporal features into value vectors and key vectors, and uses a linear fully connected layer to convert the expanded causal temporal features into query vectors, and uses the causal relationships within the temporal features to enhance the temporal features.

4. The end-to-end autonomous driving system for a strong interaction scenario considering the style of other vehicles, as described in claim 1, is characterized in that: The traffic participant style perception module collects spatiotemporal information about the vehicle and surrounding traffic participants. Based on the vehicle's basic state, it constructs historical information about the vehicle and surrounding traffic participants, builds a three-dimensional spatial state of a class graph, and selects eight traffic participants around the vehicle EV as: Front Vehicle (FV), Rear Vehicle (ReV), Left Vehicle (LV), Right Vehicle (RV), Left Front Vehicle (LFV), Right Front Vehicle (RFV), Left Rear Vehicle (LRV), and Right Rear Vehicle (RRV). The motion state of surrounding traffic participants Composition, described as: ； in, It represents the length of the time window, t is the time step, and t0 is the current time; it also represents the motion state of traffic participants. including the longitudinal speed of traffic participants Lateral speed of traffic participants longitudinal acceleration of traffic participants Lateral acceleration of traffic participants longitudinal agility of traffic participants Longitudinal distance of traffic participants relative to the main vehicle Lateral distance of traffic participants relative to the main vehicle There are a total of 7 dimensions, namely the motion state dimension of traffic participants. The motion state of traffic participants writing: ； A two-dimensional convolutional network is used to process spatial states and extract spatial interaction relationships between adjacent traffic participants; the number of output channels of the two-dimensional convolution is... The kernel size is K CS =2, sliding step size is S CS =1, and the output 3D tensor is subjected to layer normalization, linear correction and flattening to obtain a one-dimensional spatial feature encoding. Among them, the spatial feature encoding dimension ; Forming spatial feature encoding historical time series ,in Then, a temporal convolutional network combined with an attention mechanism is used for processing. The temporal convolutional network combined with the attention mechanism combines causal convolution and dilated convolution in the temporal convolutional network. The dilated convolution expands the sampling interval of the convolution kernel on the basis of causal convolution, and flattens the encoding results of traditional one-dimensional convolution and multi-layer dilated causal convolution to obtain direct temporal features. and causal temporal characteristics ,in , These are the direct temporal feature dimension and the causal temporal feature dimension, respectively; in the attention mechanism, the linearly changing direct temporal feature is a value vector. and key vector Linearly changing causal time-series features are used as query vectors To enhance the temporal characteristics from a causal perspective, among which The dimension of the attention ternary vector; the value vector Key vector and query vector It is expressed as follows: ； ； ； in, , These are the linear projection weight matrices for the value vector, key vector, and query vector, respectively. and These correspond to the dimensions of the feature encoding after one-dimensional convolution and dilated causal processing, respectively.

5. The end-to-end autonomous driving system for a strong interaction scenario considering the style of other vehicles, as described in claim 4, is characterized in that: In attention mechanisms, the focus is on time steps. First, calculate the query vector. With key vector The dot product between them; then, based on the dimension of the attention triple vector. The dot product result is scaled; the scaled score is then normalized using the Softmax function to obtain the time step. The importance weight vector of time-series feature encoding based on causal mechanisms : ； Based on importance weight vector , value vector By weighting by importance, the implicit features of driving style are obtained. : ； Ultimately, the implicit characteristics of driving style will be... The input is fed into a linear fully connected layer to obtain the resulting driving style of each traffic participant at time t. .

6. An end-to-end autonomous driving system for a strong interaction scenario considering the style of other vehicles, as described in claim 4 or 5, characterized in that: The traffic participant style perception module is trained using deep supervised learning. To handle situations where there are no traffic participants in a specific direction, the absence of traffic participants is included in the output of the module, classifying driving styles into four categories: conservative, general, aggressive, and no traffic participants. The goal of driving style recognition is to identify the driving styles of adjacent traffic participants. Assuming there are 8 adjacent traffic participants for the vehicle, 8 Softmax activation functions are set at the end of the temporal convolutional network combined with the attention mechanism module. Each Softmax activation function has 4 branches, meaning the output dimension of the fully connected output layer is 32. Finally, the model's final driving style recognition is the one with the highest probability value in the output dimension. The loss function during training is L. TA The cross-entropy loss function is used and defined as follows: ； in, For the sample size, The total number of categories, For the sample In category The unique and genuine label on the top. Samples output by the model In category The predicted probability.

7. An end-to-end autonomous driving system for a strong interaction scenario considering the style of other vehicles, as described in claim 1, is characterized in that: The value estimation module and policy module are built based on the MetaDrive simulation interactive environment, and use reinforcement learning to optimize the autonomous driving policy; a first-order Markov decision process is established. ,in, For state space, For the action space, Let be the state transition probability. As a reward; the specific composition of the tuple is as follows: Time step status From the original information of a single frame image and implicit characteristics of the driving styles of adjacent traffic participants. Composition; Time step status Writing in key-value pair form: Image information: Style characteristics: ; (2) Time step Action output This refers to the direct lateral and longitudinal control quantities of intelligent vehicles, namely the accelerator pedal opening. Brake pedal opening and steering wheel angle Time step Action output writing: ； (3) The state transition probability is determined by the environment settings. In the MetaDrive simulation interaction environment, the intelligent vehicle performs actions based on the current state. Then, the onboard sensors collect information and the behavior of surrounding traffic participants causes a state transition. The state transition probability is defined as follows: , The state at the next moment; (4) Time step Rewards It consists of several sub-reward items, represented as follows: ； in, Let be the weight matrix of the sub-reward vector. This is the sub-reward vector at the current moment; The rewards in reinforcement learning consist of goal-oriented rewards, evaluative rewards, and penalties. Goal-oriented rewards aim to help the vehicle reach its destination quickly; evaluative rewards aim to improve the safety and comfort of the vehicle and other road users; penalties serve as the baseline for environmental exploration, ensuring vehicle safety and adherence to traffic rules. Upon being penalized, the intelligence is reset. The sub-reward vector at the current time step... Specifically defined as: ； in, As a displacement reward, To maintain the reward for speed, A reward will be given for successfully reaching the destination. As a security reward, As an aggressive reward, As a comfort reward, This is the penalty vector.

8. An end-to-end autonomous driving system for a strong interaction scenario considering the style of other vehicles, as described in claim 7, is characterized in that: The specific definitions of each reward in the sub-reward vector at the current moment are as follows: a. Displacement bonus The system is designed to provide a positive reward when a vehicle is centered in the lane and moving towards its destination. This is a goal-oriented reward and is defined as follows: ； in, This represents the longitudinal displacement of the vehicle within a single step. The lateral keeping coefficient is used to penalize a vehicle after it deviates from the center of the lane. This is the directional coefficient, which is penalized when a vehicle travels in the opposite direction. b. Speed ​​Maintenance Bonus The system is designed to reward a vehicle for maintaining a relatively high speed in the correct direction; this is a targeted reward and is defined as follows: ； in, The current speed of the vehicle. The maximum speed allowed in the scenario; c. Success Reward The reward for a vehicle successfully reaching its navigation destination is sparse but high, constituting a goal-oriented reward, and is defined as follows: ； in, The constant reward received by the vehicle upon successfully reaching its destination; d. Security rewards This invention characterizes the impact of traffic participant behavior on vehicle safety; it utilizes a risk field to assess vehicle driving safety; and it measures the risk field strength generated by surrounding traffic participants relative to the vehicle. Defined as: ； in, This refers to traffic participants surrounding the vehicle, specifically those in front of, behind, to the left, to the right, to the left front, to the left rear, to the right front, and to the right rear. ; For the vehicle and other traffic participants in the Frenet coordinate system of traffic participant j Distance vector between them; This is the distance scaling factor. and For traffic participants Acceleration along its direction of travel and normal acceleration; The peak field strength of the risk field occurs among traffic participants. The location of the center of mass; and They are respectively With traffic participants The angle between the direction of travel and the normal direction; and These are the acceleration coefficients in the forward direction and normal direction; For traffic participants The quality; The driving safety reward function is constructed as follows: ； in, The threshold field strength at which a driver will not respond to a risk; e. Aggressive Rewards This characterizes the impact of autonomous vehicle behavior on surrounding traffic participants; and constructs aggressive rewards: ； in, The risk intensity posed by vehicles to traffic participants in each scenario; f. Comfort reward This represents the driver's need for driving comfort, and the comfort reward is defined as: ； in, The longitudinal abruptness of the vehicle; g. Penalty vector This corresponds to various situations that will reset the environment, including driving off the road, vehicle collision, object collision, and pedestrian collision. The penalty vector is represented as: ； in, , , , The penalties are for driving off the road, vehicle collision, object collision, and pedestrian collision, respectively, and are all constant terms.

9. An end-to-end autonomous driving system for a strong interaction scenario considering the style of other vehicles, as described in claim 1, is characterized in that: The policy module and value estimation module use a dual-delay deep deterministic policy gradient reinforcement learning algorithm. The policy module and value estimation module are respectively a policy network and two action value estimation networks. The policy network and action value estimation network share a feature extraction head. The network objects trained include the image feature extraction module, the policy module and the value estimation module. The traffic participant style perception module is pre-trained by deep supervised learning. In terms of network design, there is one real-policy network and two real-action value estimation networks, each with a corresponding target network, totaling six networks. During training, the parameters of each neural network are first initialized, with the parameters of the real-policy network and the target policy network corresponding to... , The parameters of the two sets of real-action value estimation networks and target action value estimation networks correspond to respectively Real-world strategy network Based on the time state Output Action Environmental reception Transition from the last state to the next state And obtain time rewards. This forms a state transition pair. Store it in the experience pool Medium; Experience Pool Once a certain scale is reached, [the experience pool will be used]. Extracting mini-batch state transition pairs for real-world action value estimation networks Based on the time state and actions Output action value estimate Target action value estimation network Based on the state at the next moment And the next action estimated by the target policy network. Output the estimated value of the action at the next moment. .

10. An end-to-end autonomous driving system for a strong interaction scenario considering the style of other vehicles, as described in claim 9, characterized in that: In calculating actions Normally distributed noise is added and truncated according to the actual range of motion to enhance the exploration capabilities of intelligent vehicles. The formula is as follows: ； in, For real-world policy networks in state The next action to re-output, To conform to the standard deviation Gaussian exploration noise with a normal distribution; Estimate the next action of the target policy network The surrounding region is subjected to fuzzy fitting by adding normally distributed noise and truncating the noise in advance, as shown in the following formula: ； in, For the target policy network in state The next action to re-output, The noise amplitude clipping threshold. It is the raw exploration noise sampled from a normal distribution. To effectively explore noise after trimming; Calculate the value of the target action The formula is as follows: ； Loss function of action value estimation network The calculation formula is as follows: ； in, For time steps The reward The number of unlabeled samples used for fine-tuning in reinforcement learning; This is the estimated value of the current action. The Q-value estimated by the action value estimation network guides the policy network update. The parameter optimization method uses gradient ascent, and the objective function is... as follows: ； in, For real-world policy networks, based on state The action of re-outputting; For updates to the target network, a soft update approach is adopted, incorporating a learning rate. During each update, a mixture of the old and new parameters is used, and the learning rate is... The magnitude of the soft update is determined by the following formula: ； The update of the real-world policy network is delayed compared to the real-world action value estimation network, which improves the stability of the training process.

Images