Training method, device and system for target network for automatic driving

Abstract

The application provides a target network training method, device and system for automatic driving. The method comprises the following steps: generating first state information and environment identification information corresponding to the first state information by using a simulation environment processing module, wherein the environment identification information is used to represent driving information within a preset range of a target vehicle; inputting the first state information into a hierarchical graph neural network to obtain first encoding features and first decoding features corresponding to the first state information; inputting the first encoding features into a multi-layer perception machine to obtain decoding information corresponding to the first encoding features; determining a first loss value of the hierarchical graph neural network according to the environment identification information and the decoding information; determining a second loss value of a first actor network; and updating parameters of the hierarchical graph neural network and the first actor network according to the second loss value. According to the technology, the learning ability of the hierarchical graph neural network for key information is improved, and thus the decision performance of the target network is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of automatic driving, and in particular to a training method, device and system for a target network for automatic driving. BACKGROUND

[0002] For the training scene of the target network for automatic driving, more and more deep reinforcement learning methods are used to train the target network. In the related art, in the process of training the target network based on the PPO (Proximal Policy Optimization) algorithm, a deep learning network such as a convolutional neural network or a multi-layer perceptron is usually used as a pre-network of an actor network to learn environment information. However, the deep learning network such as the convolutional neural network or the multi-layer perceptron has poor effect in capturing important environment information affecting decision-making behavior when learning environment information, which leads to that the target network obtained finally cannot derive an accurate decision-making result, thereby affecting the decision-making performance of the target network. SUMMARY

[0003] Embodiments of the present application provide a training method, device and system for a target network for automatic driving to solve the problems in the related art, and the technical solutions are as follows:

[0004] In a first aspect, the embodiments of the present application provide a training method for a target network for automatic driving, comprising:

[0005] generating first state information and environment identification information corresponding to the first state information by using a simulation environment processing module, the environment identification information being used to represent driving information within a preset range of a target vehicle;

[0006] inputting the first state information into a hierarchical graph neural network to obtain first encoding features and first decoding features corresponding to the first state information; inputting the first encoding features into a first actor network to obtain a first decision-making result, and obtaining second state information after the first decision-making result is executed, a reward value corresponding to the first decision-making result, and a judgment result of whether a decision-making process is terminated by using the simulation environment processing module, and constructing training data;

[0007] inputting the first encoding features into a multi-layer perceptron to obtain decoding information corresponding to the first encoding features; and obtaining a second decision-making result corresponding to the first state information by using a second actor network based on the training data, and obtaining an advantage value corresponding to the first decision-making result by using a critic network and an advantage function;

[0008] determining a first loss value of the hierarchical graph neural network according to the environment identification information and the decoding information; and determining a second loss value of the first actor network according to a comparison result of the first loss value, the first decision-making result and the second decision-making result, the first decoding features and the advantage value;

[0009] The parameters of the hierarchical graph neural network and the first actor network are updated based on the second loss value until the target hierarchical graph neural network and the target network that meet the preset conditions are obtained.

[0010] In one implementation, the driving information within a preset range of the target vehicle includes the lane information currently occupied by the target vehicle, the determination information on whether there are other vehicles within the preset range of the target vehicle, and the speed information of other vehicles within the preset range.

[0011] In one implementation, determining a first loss value for the hierarchical graph neural network based on environmental identification information and decoding information includes:

[0012] Based on environmental identification information and decoding information, the first loss value of the hierarchical graph neural network is determined using the mean square error formula.

[0013] In one implementation, determining a second loss value for the first actor network based on a first loss value, a comparison of a first decision result and a second decision result, a first decoding feature, and an advantage value includes:

[0014] The first loss value, the comparison result of the first decision result and the second decision result, the first decoding feature and the advantage value are input into the actor network loss function to obtain the second loss value of the first actor network.

[0015] In one implementation, based on training data, a second decision result corresponding to the first state information is obtained through a second actor network, and an advantage value corresponding to the first decision result is obtained through a critic network and an advantage function, including:

[0016] The first state information in the training data is input into a hierarchical graph neural network to obtain the second encoded feature and the second decoded feature corresponding to the first state information.

[0017] The second encoded feature is input into the second actor network to obtain the second decision result corresponding to the first state information;

[0018] The second encoded feature is input into the critic network to obtain the state value corresponding to the first state information, and the advantage value corresponding to the first decision result is determined by using the advantage function.

[0019] In one implementation, after inputting the second encoded features into the critic network to obtain the state value corresponding to the first state information, and determining the advantage value corresponding to the first decision result using the advantage function, the method further includes:

[0020] Use the value loss function to determine the third loss value corresponding to the advantage value;

[0021] The parameters of the critic network are updated based on the third loss value.

[0022] In one implementation, the parameters of the second actor network are obtained by hard updating the parameters of the first actor network after a preset number of updates, and the initialization parameters of the first actor network and the second actor network are the same.

[0023] In one implementation, the server communicates with the client via remote procedure call technology.

[0024] Secondly, embodiments of this application provide an electronic device, including:

[0025] At least one processor; and

[0026] A memory that is communicatively connected to at least one processor; wherein,

[0027] The memory stores instructions that can be executed by at least one processor, which enables the at least one processor to perform the training method for a target network for autonomous driving according to the embodiments of this application.

[0028] Thirdly, embodiments of this application provide a training system for a target network for autonomous driving, used to execute the training method for a target network for autonomous driving according to embodiments of this application. The system includes a client and a server. The client is equipped with a simulation environment processing module, a hierarchical graph neural network, and a first actor network. The client obtains training data based on first state information, a first decision result, second state information, a reward value, and a judgment result indicating whether the decision process has terminated, and sends the training data to the server. The server is equipped with a first actor network, a second actor network, a critic network, and a parameter update module. The parameter update module is used to update the parameters of the hierarchical graph neural network, the first actor network, the second actor network, and the critic network according to the training data, and synchronizes the updated network parameters to the client.

[0029] According to the technology of this application, by using a simulation environment processing module to generate environmental identification information corresponding to the first state information, and using a multilayer perceptron to obtain decoding information corresponding to the first encoded features output by the hierarchical graph neural network, and determining a first loss value based on the environmental identification information and the decoding information, the hierarchical graph neural network can be guided to focus on key information in the state information, namely, the driving information within a preset range of the target vehicle. This can include, for example, the driving information within the preset range of the target vehicle, including the lane information currently occupied by the target vehicle, the judgment information regarding the presence of other vehicles within the preset range, and the speed information of other vehicles within the preset range. This makes the first decision result obtained by the first actor network based on the first encoded features output by the hierarchical graph neural network more accurate. Secondly, a second loss value for the first actor network is determined based on the first loss value, the comparison result between the first and second decision results, the first decoded features, and the advantage value. The parameters of the hierarchical graph neural network, the first actor network, and the hierarchical graph neural network are updated based on the second loss value, enabling the hierarchical graph neural network and the first actor network to more stably capture key information and thus make more reasonable decision results. In summary, the method according to the embodiments of this application can realize the synchronous updating of parameters of the hierarchical graph neural network and the first actor network, and significantly improve the learning ability of the hierarchical graph neural network for key information, thereby improving the decision-making performance of the target network.

[0030] The above overview is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features of this application will become readily apparent from the accompanying drawings and the following detailed description. Attached Figure Description

[0031] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments disclosed in this application and should not be construed as limiting the scope of this application.

[0032] Figure 1 A flowchart of a method for training a target network for autonomous driving according to an embodiment of this application;

[0033] Figure 2 This is an architecture diagram of a training method for a target network for autonomous driving according to an embodiment of this application;

[0034] Figure 3 This is a schematic diagram of a training method for a target network for autonomous driving according to an embodiment of this application;

[0035] Figure 4 This is a schematic diagram of a training system for a target network for autonomous driving according to an embodiment of this application;

[0036] Figure 5 This is a block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0037] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0038] Figure 1 A flowchart illustrating a method for training a target network for autonomous driving according to an embodiment of this application is shown. Figure 1 As shown, the method may include the following steps:

[0039] Step S101: Use the simulation environment processing module to generate first state information and environmental identification information corresponding to the first state information. The environmental identification information is used to characterize the driving information of the target vehicle within a preset range.

[0040] Step S102: Input the first state information into the hierarchical graph neural network to obtain the first encoded feature and the first decoded feature corresponding to the first state information; input the first encoded feature into the first actor network to obtain the first decision result, and use the simulation environment processing module to obtain the second state information after executing the first decision result, the reward value corresponding to the first decision result, and the judgment result of whether the decision process has terminated, and construct training data;

[0041] Step S103: Input the first encoded feature into the multilayer perceptron to obtain the decoding information corresponding to the first encoded feature; and, based on the training data, obtain the second decision result corresponding to the first state information through the second actor network, and obtain the advantage value corresponding to the first decision result through the critic network and the advantage function.

[0042] Step S104: Determine the first loss value of the hierarchical graph neural network based on the environmental identification information and the decoding information; and determine the second loss value of the first actor network based on the first loss value, the comparison result of the first decision result and the second decision result, the first decoding feature and the advantage value.

[0043] Step S105: Update the parameters of the hierarchical graph neural network and the first actor network according to the second loss value until the target hierarchical graph neural network and the target network that meet the preset conditions are obtained.

[0044] In the embodiments of this application, such as Figure 2As shown, the application of reinforcement learning algorithms to downstream tasks involves two main parts: the training environment and reinforcement learning itself. In the training environment, the simulation environment processing module constructs observation information (i.e., the current state information) based on the basic state data in the current environment using an observation generation algorithm, and then passes this observation information to the reinforcement learning model. In the reinforcement learning part, the model generates action information based on the observation information and feeds this action information back to the simulation environment processing module. Upon receiving the action information, the simulation environment processing module generates reward information according to the reward generation definition and feeds it back to the reinforcement learning model so that the model can update the parameters of the actor network. The reward information may include the reward value, the state information for the next time step, and the judgment result regarding whether the decision process should terminate.

[0045] The reinforcement learning model can employ any of the following reinforcement learning algorithms: Q-Learning, DQN, Actor-Critic, DDPG, and PPO. In the following description of the embodiments of this application, the PPO algorithm is specifically used as an example. It can be understood that the PPO algorithm is a policy-based reinforcement learning algorithm that uses three neural networks (i.e., a first actor network, a second actor network, and a critic network). By inputting the current "state" of the "agent" into the neural network, a corresponding "action" and "reward" are obtained. The state of the "agent" is then updated based on the "action." Gradient ascent is used to update the weight parameters in the actor network based on the objective function containing the "reward" and "action," thereby obtaining an "action" judgment that maximizes the overall reward value. In the following description of this application, the first state information can be understood as the state information at the current moment, and the second state information can be understood as the state information at the next moment after executing the first decision result.

[0046] In this embodiment, the target network outputs a target decision result based on the target vehicle's state information. The state information includes the vehicle's current road information and driving information within a preset range. Specifically, the driving information may include whether other vehicles exist within the preset range and the historical trajectory information of other vehicles within a preset time period. The target decision result output by the target network is used to control the target vehicle to perform corresponding driving behaviors. For example, the target decision result may include acceleration, deceleration, changing lanes to the left or right, or maintaining the current driving behavior. Based on the target decision result generated by the target network, autonomous driving in high-speed driving scenarios can be achieved.

[0047] For example, in step S101, the simulation environment processing module simulates a driving environment and generates state information based on that environment. This module can be a joint simulation environment for reinforcement learning built using Gym (a toolkit for developing and comparing reinforcement learning algorithms) and Carla (an autonomous driving simulator). Gym provides a simulation environment and data protocols for reinforcement learning. Carla is an open-source autonomous driving simulator that uses a modular and flexible API (Application Programming Interface) to solve a range of tasks related to autonomous driving, helping to achieve autonomy in autonomous driving research and development. It is a tool that can be easily accessed and customized.

[0048] Environmental identification information can be key information from the marked first state information, used to indicate the driving information of the target vehicle within a preset range. More specifically, in one embodiment, the driving information of the target vehicle within the preset range includes the lane information currently occupied by the target vehicle, information on whether there are other vehicles within the preset range, and the speed information of other vehicles within the preset range.

[0049] For example, environmental identification information may include driving information of the target vehicle within multiple preset ranges, specifically including: whether there are other vehicles 100 meters and 50 meters ahead of the target vehicle, and if so, the speed of other vehicles; the lane information of the target vehicle; and the speed information of other vehicles within 100 meters ahead and behind the target vehicle in the lane it is currently in and the adjacent lane.

[0050] For example, in step S102, the hierarchical graph neural network serves as a pre-network of the first actor network to extract key information from the first state information. The extracted first state information is then input into the first actor network so that the first actor network can generate a first decision result based on the key information. Specifically, the key information may be driving information of the target vehicle within multiple preset ranges.

[0051] In one example, the hierarchical graph neural network could specifically be VectorNet. VectorNet includes an encoding layer and a decoding layer. The encoding layer vectorizes all graph elements (e.g., map features and dynamic traffic participants) in the first state information, constructs a polyline subgraph based on the vectorized representation, and obtains the encoded features. The decoding layer uses a multi-layer perceptron (MLP) to aggregate the local features of different polyline subgraphs, and then integrates all trajectory and map features globally to obtain the decoded features.

[0052] After obtaining the first decision result output by the first actor network, the first decision result can be input into the simulation environment processing module to obtain the second state information of the target vehicle after executing the first decision result, the reward value corresponding to the first decision result, and the judgment result of whether the current driving scenario has terminated. Among them, the reward value is used to evaluate the merits of the first decision result, and the judgment result of whether the decision process has terminated is obtained based on the second state information.

[0053] In addition, training data is obtained based on the first state information, the first decision result, the reward value, the second state information, and the judgment result of whether the decision process has terminated, and the training data is added to the experience pool.

[0054] For example, after obtaining the second state information by executing the first decision result, if the judgment result of whether the decision process terminates is negative, the second state information is input as the state information of the current moment into the first actor network to obtain the first decision result corresponding to the current moment. This first decision result is then input into the simulation environment processing module to obtain the reward value of the first decision result corresponding to the current moment, the state information of the next moment, and the judgment result of whether the decision process terminates. Based on this, the training data can be multiple data points corresponding to consecutive moments to form a training dataset. The training data can be expressed in the form of a sequence, for example: {S t A t R t S t+1 , is_done}, where S t A is used to represent the state information at time t. t R is used to represent the decision result obtained based on the state information at time t. t The reward value is used to represent the decision outcome at time t, and is_done is used to represent the judgment result of whether to terminate the decision process at time t+1. After obtaining the training data, it can be added to the experience pool as the data foundation for subsequent reinforcement learning of the first actor network and the critic network.

[0055] For example, in step S103, a multilayer perceptron is a feedforward artificial neural network that maps a set of input vectors to a set of output vectors. A multilayer perceptron includes a first input layer, intermediate hidden layers, and a final output layer. The product of the input elements and weights is fed back to a summing node with neuron biases. Its main advantage lies in its ability to quickly solve complex problems.

[0056] For example, in step S103, training data can be extracted from the experience pool, and the first state information in the training data can be input into a hierarchical graph neural network to obtain the second encoded feature and the second decoded feature corresponding to the first state information. The second encoded feature is then input into a second actor network to obtain a second decision result. Furthermore, the second encoded feature is input into a critic network, and the advantage function is used to obtain the advantage value corresponding to the first decision result.

[0057] It should be noted that the initialization parameters of the first actor network and the second actor network are the same, and the parameters of the second actor network are updated based on the parameters of the first actor network. For example, after updating the parameters of the first actor network a preset number of times, the parameters of the second actor network can be updated based on the parameters of the first actor network after the preset number of updates. Therefore, the update frequencies of the first and second actor networks are different. In subsequent reinforcement learning processes, the parameters of the first and second actor networks may differ, and the first and second decision results obtained based on the first state information may also differ.

[0058] For example, in step S104, the first loss value can be calculated using the mean squared error formula. The second loss value can be calculated using a preset actor network loss function, wherein the actor network loss function is designed based on the first loss value, the comparison result of the first decision result and the second decision result, the first decoding feature, and the advantage value.

[0059] For example, in step S105, synchronous parameter updates can be performed on the hierarchical graph neural network and the first actor network. A preset condition can be that the average reward value of the first actor network in a preset number of consecutive rounds is greater than or equal to the expected preset reward value. A round represents the period from the start to the end of a decision. Specifically, at the start of a decision, after the first actor network makes a decision based on initial first state information, if the judgment result indicates that the current driving scenario will not terminate, the first actor network continues to make decisions based on second state information, and so on, until the judgment result obtained after making a decision indicates that the current driving scenario will terminate, at which point the decision-making stops, i.e., the round ends. In a round, there are corresponding first decision results and corresponding reward values ​​made by the first actor network at multiple consecutive moments, based on which the average reward value for that round can be obtained. Based on the average reward values ​​corresponding to a preset number of consecutive rounds, if the average reward value corresponding to each round is greater than or equal to the expected preset reward value, it can be determined that the first actor network meets the preset condition.

[0060] Furthermore, if the updated first actor network does not meet the preset conditions, steps S101 to S105 are repeated until the first actor network meets the preset conditions, and the current hierarchical graph neural network and the first actor network are determined as the target hierarchical graph neural network and the target network.

[0061] According to the method of this application embodiment, by using a simulation environment processing module to generate environmental identification information corresponding to the first state information, and using a multilayer perceptron to obtain decoding information corresponding to the first encoded features output by the hierarchical graph neural network, a first loss value is determined based on the environmental identification information and the decoding information. This guides the hierarchical graph neural network to focus on key information in the state information, namely, driving information within a preset range of the target vehicle. For example, this may include the target vehicle's current lane information, a judgment on the presence of other vehicles within the preset range, and the speed information of other vehicles within the preset range. This makes the first decision result obtained by the first actor network based on the first encoded features output by the hierarchical graph neural network more accurate. Secondly, a second loss value for the first actor network is determined based on the first loss value, the comparison result between the first and second decision results, the first decoded features, and the advantage value. The parameters of the hierarchical graph neural network and the first actor network are updated based on the second loss value, enabling the hierarchical graph neural network and the first actor network to more stably capture key information and thus make more reasonable decision results. In summary, the method according to the embodiments of this application can realize the synchronous updating of parameters of the hierarchical graph neural network and the first actor network, and significantly improve the learning ability of the hierarchical graph neural network for key information, thereby improving the decision-making performance of the target network.

[0062] In one implementation, step S104 may include:

[0063] Step S1041: Based on the environmental identification information and decoding information, determine the first loss value of the hierarchical graph neural network using the mean square error formula.

[0064] For example, the formula for mean squared error is as follows:

[0065]

[0066] in, θ is used to represent decoding information, and θ is used to represent environmental identification information.

[0067] In one implementation, step S104 may include:

[0068] Step S1042: Input the first loss value, the comparison result of the first decision result and the second decision result, the first decoding feature and the advantage value into the actor network loss function to obtain the second loss value of the first actor network.

[0069] For example, the actor network loss function used to calculate the second loss value can be as follows:

[0070] Actor_loss=mean(-min(surr1,surr2))+aux_loss+node_decoder_loss,

[0071] surr1 = ratio * advantage,

[0072] surr2=max(min(ratio,1-ε),1+ε)*ratio,

[0073] node_decoder_loss=Smooth_L1_Loss(aux_pred,aux_true),

[0074]

[0075] Where ratio represents the ratio of the first decision result to the second decision result, advantage represents the advantage value, hyperparameter ε represents the cutoff ratio, and ε∈(0,1); aux_pred represents the first decoded feature, aux_true represents the true value corresponding to the first decoded feature, and β represents the numerical difference between the first decoded feature and the true value corresponding to the first decoded feature.

[0076] In one implementation, step S103 may include:

[0077] Step S1031: Input the first state information in the training data into the hierarchical graph neural network to obtain the second encoded feature and the second decoded feature corresponding to the first state information;

[0078] Step S1032: Input the second encoded feature into the second actor network to obtain the second decision result corresponding to the first state information;

[0079] Step S1033: Input the second encoded feature into the critic network to obtain the state value corresponding to the first state information, and use the advantage function to determine the advantage value corresponding to the first decision result.

[0080] In this embodiment, the value output by the critic network is used to characterize the value corresponding to the first state information; the advantage function is used to express the superiority or inferiority of the first decision output by the first actor network relative to the average strategy under the first state information, and can reflect the deviation of the random variable relative to the mean.

[0081] For example, the advantage function A(s,a) can be specifically as follows:

[0082] A(s,a)=Q(s,a)-V(s)

[0083] Where Q(s,a) is the state-behavior value function, used to calculate the expected reward value obtained after performing action a in state s; V(s) is the state value function, used to calculate the expected reward value in state s.

[0084] In one embodiment, after inputting the second encoded features into the second actor network to obtain the second decision result corresponding to the first state information, the method of this application embodiment further includes:

[0085] Use the value loss function to determine the third loss value corresponding to the advantage value;

[0086] The parameters of the critic network are updated based on the third loss value.

[0087] For example, the value loss function can be as follows:

[0088] Critic_loss=mean(mse_loss(states,td_target)),

[0089] td_target=rewards+γ*next_states*(1-J),

[0090] Here, states represents the first state information, next_states represents the second state information, mse_loss is used to calculate the mean squared error loss, rewards represents the set of reward values ​​in the experience pool, hyperparameter γ represents the decay rate, and J represents the judgment result of whether the decision process terminates after executing the first decision result.

[0091] In one implementation, the parameters of the second actor network are obtained by hard updating the parameters of the first actor network after a preset number of updates, and the initialization parameters of the first actor network and the second actor network are the same.

[0092] For example, after performing a preset number of updates on the parameters of the first actor network, a hard update can be performed on the parameters of the second actor network based on the parameters of the first actor network after the preset number of updates. It can be understood that a hard update means assigning all the parameters of the first actor network after the preset number of updates to the second actor network; that is, the parameters of the second actor network after the update should be the same as the parameters of the first actor network after the preset number of updates.

[0093] It should be noted that the embodiments of this application do not specifically limit the preset quantity, which can be set according to the actual situation. For example, the preset quantity can be 100 or 200 or any other arbitrary value.

[0094] In this embodiment, the updates to the parameters of the hierarchical graph neural network, the first actor network, the second actor network, and the critic network are performed by the server. Specifically, the server deploys the hierarchical graph neural network, the first actor network, the second actor network, and the critic network, and can use the PPO algorithm to train the first actor network, the second actor network, and the critic network. The client deploys an agent for running the hierarchical graph neural network and the first actor network, as well as a simulation environment processing module. The simulation environment processing module generates first state information and corresponding environment identification information, and inputs the first state information into the agent so that the hierarchical graph neural network can obtain first encoded features and first decoded features based on the first state information. The first actor network obtains a first decision result based on the first encoded features and inputs the first decision result into the simulation environment processing module to obtain the reward value of the first decision result, the second state information after executing the first decision result, and a judgment result indicating whether the decision process has terminated. The client sends the first state information, the first decision result, the reward value, and the second state information as training data to the server's data buffer (i.e., the experience pool). The server updates the parameters of the hierarchical graph neural network, the first actor network, the second actor network, and the critic network based on the training data, and feeds the updated parameters back to the client so that the client can adjust the parameters of the hierarchical graph neural network and the first actor network deployed on the agent.

[0095] The following reference Figure 3 A specific example is used to describe a method for training a target network for autonomous driving according to embodiments of this application. For instance... Figure 3 As shown, this method can use the PPO algorithm to train the actor network and the critic network.

[0096] Specifically, the first actor network processes the first state information s output by the simulation environment processing module. t And environmental identification information info, based on the first encoded feature corresponding to the first state information output by the hierarchical graph neural network, outputs the corresponding first decision result a. t Then, the first decision result a t The simulation environment processing module is input again to obtain the first decision result a. t The second state information s t+1 Reward value r t And the result of determining whether the decision-making process has ended. Based on the first state information s t First decision result a t Second state information s t+1 Reward value r t The training data is obtained by determining whether the decision-making process has ended, and the training data is stored in the experience pool.

[0097] In the reinforcement learning process, training data is extracted from the experience pool, and the first state information is input into a hierarchical graph neural network to obtain the second encoded features. Then, the second encoded features are input into a second actor network to obtain the second decision result, and into a critic network to obtain the state value. Finally, the advantage function is used to obtain the advantage value.

[0098] Based on the first encoded feature output by the hierarchical graph neural network, encoded information is obtained using a multilayer perceptron. Then, based on the encoded information and environmental labeling information, the first loss value corresponding to the hierarchical graph neural network is determined using the mean squared error formula. Furthermore, based on the first loss value, the comparison between the first and second decision results, the first decoded feature, and the advantage value, the second loss value of the first actor network is determined using the actor network loss function.

[0099] Finally, the parameters of the hierarchical graph neural network and the first actor network are updated synchronously based on the first loss value and the second loss value until the target hierarchical graph neural network and the target network that meet the preset conditions are obtained.

[0100] According to another aspect of the embodiments of this application, a training system for a target network for autonomous driving is also provided, used to execute the training method for a target network for autonomous driving according to the embodiments of this application. The system includes a client and a server. Specifically, the client deploys a simulation environment processing module, a hierarchical graph neural network, and a first actor network. The client obtains training data based on first state information, a first decision result, second state information, a reward value, and a judgment result indicating whether the decision process has terminated, and sends the training data to the server. The server deploys a first actor network, a second actor network, a critic network, and a parameter update module. The parameter update module is used to update the parameters of the hierarchical graph neural network, the first actor network, the second actor network, and the critic network according to the training data, and synchronizes the updated network parameters to the client.

[0101] For example, such as Figure 4As shown, the system can include a client and a server. The server includes a data cache, a reinforcement learning model training module, and a storage module. The reinforcement learning model training module updates the parameters of the hierarchical graph neural network, the first actor network, the second actor network, and the critic network deployed on the server. The client deploys an agent for running the hierarchical graph neural network and the first actor network, as well as a simulation environment processing module. The simulation environment processing module generates first state information and inputs it into the agent, allowing the hierarchical graph neural network to output first encoded features based on the first state information, and allowing the first actor network to generate a first decision result based on the first encoded features. The agent inputs the first decision result into the simulation environment processing module to obtain the reward value of the first decision result, as well as the second state information, reward value, and a judgment result indicating whether the decision process has ended after executing the first decision result. The client sends the first state information, the first decision result, the reward value, and the second state information as training data to the server's data cache (i.e., experience pool). The server updates the parameters of the hierarchical graph neural network, the first actor network, the second actor network, and the critic network based on the training data, and feeds the updated parameters back to the client so that the client can adjust the parameters of the hierarchical graph neural network and the first actor network deployed on the agent.

[0102] In one implementation, the server communicates with the client via remote procedure call technology.

[0103] For example, remote procedure call technology can specifically employ gRPC. gRPC is a language-neutral, platform-neutral, and open-source remote procedure call technology. By utilizing gRPC, transparent communication between the client and server can be achieved, simplifying the construction of communication systems.

[0104] According to another aspect of the embodiments of this application, a vehicle is also provided, the vehicle including an autonomous driving device, the autonomous driving device being configured with an actor network for autonomous driving, the actor network being used to generate automatic decision results based on driving environment information, wherein the actor network is generated using the training method for the target network for autonomous driving of the above embodiments of this application.

[0105] Figure 5 A structural block diagram of an electronic device according to an embodiment of this application is shown. Figure 5As shown, the electronic device includes a memory 510 and a processor 520. The memory 510 stores instructions that can be executed on the processor 520. When the processor 520 executes the instructions, it implements the training method for the target network for autonomous driving described in the above embodiments. The number of memories 510 and processors 520 can be one or more. This electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0106] The electronic device may also include a communication interface 530 for communicating with external devices and exchanging data. The devices are interconnected using different buses and can be mounted on a common motherboard or otherwise as needed. The processor 520 can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). The bus can be divided into address buses, data buses, control buses, etc. For ease of illustration, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0107] Optionally, in a specific implementation, if the memory 510, processor 520, and communication interface 530 are integrated on a single chip, then the memory 510, processor 520, and communication interface 530 can communicate with each other through an internal interface.

[0108] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0109] The aforementioned computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the aforementioned computer instructions may be transmitted from one website, computer, training equipment, or data center to another website, computer, training equipment, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0110] Computer-readable storage media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other physical classes of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape or other magnetic storage media, which can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.

[0111] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0112] It should be understood that the various processes shown above can be used, with steps rearranged, added, or deleted. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved, and this is not limited herein. The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope disclosed in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for training a target network for autonomous driving, characterized in that, include: The simulation environment processing module generates first state information and corresponding environmental identification information, wherein the environmental identification information is used to characterize the driving information of the target vehicle within a preset range. The first state information is input into a hierarchical graph neural network to obtain the first encoded feature and the first decoded feature corresponding to the first state information; The first encoded feature is input into the first actor network to obtain the first decision result, and the simulation environment processing module is used to obtain the second state information after executing the first decision result, the reward value corresponding to the first decision result, and the judgment result of whether the decision process has terminated, and to construct training data. The first encoded feature is input into a multilayer perceptron to obtain the decoding information corresponding to the first encoded feature; Furthermore, based on the training data, a second decision result corresponding to the first state information is obtained through a second actor network, and an advantage value corresponding to the first decision result is obtained through a critic network and an advantage function; Based on the environmental identification information and the decoding information, a first loss value of the hierarchical graph neural network is determined; and based on the first loss value, the comparison result of the first decision result and the second decision result, the first decoding feature and the advantage value, a second loss value of the first actor network is determined. The parameters of the hierarchical graph neural network and the first actor network are updated according to the second loss value until a target hierarchical graph neural network and a target network that meet the preset conditions are obtained.

2. The method according to claim 1, characterized in that, The driving information within the preset range of the target vehicle includes the lane information where the target vehicle is currently located, the judgment information on whether there are other vehicles within the preset range of the target vehicle, and the speed information of other vehicles within the preset range.

3. The method according to claim 1, characterized in that, Based on the environmental identification information and the decoding information, the first loss value of the hierarchical graph neural network is determined, including: Based on the environmental identification information and the decoding information, the first loss value of the hierarchical graph neural network is determined using the mean square error formula.

4. The method according to claim 1, characterized in that, Based on the comparison results of the first loss value, the first decision result, and the second decision result, the first decoding feature, and the advantage value, the second loss value of the first actor network is determined, including: The first loss value, the comparison result of the first decision result and the second decision result, the first decoding feature and the advantage value are input into the actor network loss function to obtain the second loss value of the first actor network.

5. The method according to claim 1, characterized in that, Based on the training data, a second decision result corresponding to the first state information is obtained through a second actor network, and an advantage value corresponding to the first decision result is obtained through a critic network and an advantage function, including: The first state information in the training data is input into a hierarchical graph neural network to obtain the second encoded feature and the second decoded feature corresponding to the first state information. The second encoded feature is input into the second actor network to obtain the second decision result corresponding to the first state information; The second encoded feature is input into the critic network to obtain the state value corresponding to the first state information, and the advantage value corresponding to the first decision result is determined using the advantage function.

6. The method according to claim 5, characterized in that, After inputting the second encoded features into the critic network to obtain the state value corresponding to the first state information, and determining the advantage value corresponding to the first decision result using the advantage function, the method further includes: The third loss value corresponding to the advantage value is determined using the value loss function; The parameters of the critic network are updated based on the third loss value.

7. The method according to any one of claims 1 to 6, characterized in that, The parameters of the second actor network are obtained by hard updating the parameters of the first actor network after a preset number of updates, and the initialization parameters of the first actor network and the second actor network are the same.

8. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method according to any one of claims 1 to 7.

9. A training system for a target network for autonomous driving, characterized in that, The system for performing the method according to any one of claims 1 to 7 includes a client and a server; The client is equipped with a simulation environment processing module, a hierarchical graph neural network, and a first actor network. The client obtains training data based on first state information, a first decision result, second state information, reward value, and a judgment result indicating whether the decision process has terminated, and sends the training data to the server. The server is equipped with a first actor network, a second actor network, a critic network, and a parameter update module. The parameter update module is used to update the parameters of the hierarchical graph neural network, the first actor network, the second actor network, and the critic network according to the training data, and synchronizes the updated network parameters to the client.

10. The system according to claim 9, characterized in that, The server communicates with the client via remote procedure call technology.

Images