Vehicle control strategy generation model training method and device, medium and program product

Abstract

The invention provides a vehicle control strategy generation model training method and device, a medium and a program product. The vehicle control strategy generation model comprises an Actor network and a Critic network. The method comprises the following steps: acquiring a sample driving environment state corresponding to a sample time step, and respectively inputting the sample driving environment state into a muscle memory model and a vehicle control strategy generation model; the Actor network is used for outputting predicted vehicle control action probability distribution; the Critic network is used for outputting predicted value estimation; the muscle memory model is obtained based on priori knowledge training of the vehicle control action and is used for outputting a target vehicle control sequence corresponding to the sample driving environment state, and the target vehicle control sequence comprises a target vehicle control action and target value estimation. Updating network parameters of the Actor network based on the target vehicle control action and the predicted vehicle control action probability distribution; and updating network parameters of the Critic network based on the target value estimation and the predicted value estimation.

Description

Technical Field

[0001] The present application relates to the field of reinforcement learning technology, and in particular to a vehicle control strategy generation model training method, device, medium and program product. Background Art

[0002] At present, reinforcement learning is an effective way to make dynamic decisions and plans for multiple application scenarios. It focuses on how the subject should take the best behavior in an environment to maximize long-term benefits. The application scenarios of reinforcement learning generally include transportation, business management, energy and other fields. For example, vehicle control - using reinforcement learning to generate vehicle control strategies corresponding to each driving environment through the vehicle's driving environment to improve the accuracy and safety of vehicle control.

[0003] However, in the related art, when the reinforcement learning network is trained, its initial training speed is slow, and the Actor network and Critic network contained in the reinforcement learning network are difficult to converge, which seriously affects the training efficiency and training effect of the reinforcement learning network. Summary of the invention

[0004] The present application provides a vehicle control strategy generation model training method, device, medium and program product to address the deficiencies in the related art.

[0005] According to a first aspect of one or more embodiments of the present application, a method for training a vehicle control strategy generation model is provided, wherein the vehicle control strategy generation model includes an Actor network and a Critic network, and the method includes:

[0006] Obtaining a sample driving environment state corresponding to a sample time step, and inputting the sample driving environment state into a muscle memory model and a vehicle control strategy generation model respectively, wherein the Actor network is used to output a predicted vehicle control action probability distribution corresponding to the sample driving environment state, and the Critic network is used to output a predicted value estimate corresponding to the sample driving environment state, and the muscle memory model is obtained by training based on prior knowledge of vehicle control actions, and is used to output a target vehicle control sequence corresponding to the sample driving environment state, wherein the target vehicle control sequence includes a target vehicle control action and a target value estimate;

[0007] Update the network parameters of the Actor network based on the target vehicle control action and the predicted vehicle control action probability distribution;

[0008] The network parameters of the critic network are updated based on the target value estimate and the predicted value estimate.

[0009] Optionally, the muscle memory model also outputs a target vehicle control action probability distribution, and the probability corresponding to the target vehicle control action in the target vehicle control action probability distribution is the largest, and the updating of the network parameters of the Actor network based on the target vehicle control action and the predicted vehicle control action probability distribution includes: when the predicted vehicle control action probability distribution contains the target vehicle control action, determining the predicted probability corresponding to the target vehicle control action in the predicted vehicle control action probability distribution, and calculating the action loss value based on the difference between the predicted probability and the probability corresponding to the target vehicle control action in the target vehicle control action probability distribution; and updating the network parameters of the Actor network based on the action loss value.

[0010] Optionally, updating the network parameters of the Critic network based on the target value estimate and the predicted value estimate includes: determining a target value corresponding to the Critic network based on the predicted value estimate; calculating a statistic based on the target value and the target value estimate, and using the statistic as a new target value estimate for the Critic network; calculating a value loss value based on the difference between the new target value estimate and the predicted value estimate, and updating the network parameters of the Critic network based on the value loss value.

[0011] Optionally, the target vehicle control sequence also includes the sample driving environment state and the sample backward driving environment state corresponding to the next time step of the sample time step; the method also includes: storing the target vehicle control sequence in an experience replay buffer, the experience replay buffer storing at least one candidate vehicle control sequence, each candidate vehicle control sequence including a candidate driving environment state, a candidate vehicle control action, a candidate backward driving environment state and a candidate value estimate; selecting a candidate vehicle control sequence from the experience replay buffer, and updating the network parameters of the Critic network based on the selected candidate vehicle control sequence.

[0012] Optionally, any candidate vehicle control sequence also includes a candidate state value weight; the updating of the network parameters of the Critic network based on the selected candidate vehicle control sequence includes: adjusting the candidate value estimate contained in the selected candidate vehicle control sequence based on a first state value weight to obtain an adjusted target value, wherein the first state value weight is the candidate state value weight contained in the selected candidate vehicle control sequence; calculating a loss value based on the adjusted target value and a predicted value estimate output by the Critic network for the selected candidate vehicle control sequence, and updating the network parameters of the Critic network based on the loss value.

[0013] Optionally, the method further includes: combining the target value estimate with the predicted value estimate to obtain a combined value estimate; and updating the network parameters of the Critic network according to the difference between the combined value estimate and the predicted value estimate.

[0014] Optionally, the vehicle control strategy generation model also includes a reward model, and the target vehicle control sequence also includes a target immediate reward; the method also includes: calculating a prediction error of the reward model based on the target immediate reward and a predicted immediate reward output by the reward model, wherein the predicted immediate reward corresponds to the sample driving environment state; and updating model parameters of the reward model based on the prediction error.

[0015] According to a second aspect of one or more embodiments of the present application, a vehicle control strategy generation method is provided, comprising:

[0016] Acquiring a driving environment state of a vehicle, wherein the driving environment state includes driving state information of the vehicle and environment information outside the vehicle;

[0017] The driving environment state is input into a vehicle control strategy generation model to obtain a target vehicle control strategy corresponding to the driving environment state, wherein the vehicle control strategy generation model is trained by the method described in any embodiment of the first aspect, and the target vehicle control strategy includes a target vehicle control action.

[0018] According to a third aspect of one or more embodiments of the present application, an electronic device is provided, including:

[0019] processor;

[0020] a memory for storing processor-executable instructions;

[0021] The processor implements the method described in any embodiment of the first aspect or the second aspect by running the executable instructions.

[0022] According to a fourth aspect of one or more embodiments of the present application, a computer-readable storage medium is provided, on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in any embodiment of the first aspect or the second aspect are implemented.

[0023] According to a fifth aspect of one or more embodiments of the present application, a computer program product is provided, comprising a computer program and / or instructions, which, when executed by a processor, implement the steps of the method described in any embodiment of the first or second aspect above.

[0024] It can be seen from the above technical solutions that in one or more embodiments of the present application, the sample driving environment state is input into the muscle memory model and the vehicle control strategy generation model respectively. Among them, the vehicle control strategy generation model is constructed based on reinforcement learning, including an Actor network and a Critic network. The Actor network can output the predicted vehicle control action probability distribution corresponding to the sample driving environment state, and the Critic network can output the predicted value estimate corresponding to the sample driving environment. The muscle memory model is trained based on the prior knowledge of the vehicle control action. Therefore, the muscle memory model can output the target vehicle control sequence (including the target vehicle control action and the target value estimate) corresponding to the sample driving environment state, and the target vehicle control sequence can be used as the true value label for training. Furthermore, the network parameters of the Actor network are updated according to the target vehicle control action and the predicted vehicle control action probability distribution, and the network parameters of the Critic network are updated according to the target value estimate and the predicted value estimate.

[0025] Since the muscle memory model is trained based on the prior knowledge of vehicle control actions, the target vehicle control sequence it outputs can be understood as the optimal vehicle control sequence under the sample driving environment state determined based on the prior knowledge. By using the optimal vehicle control sequence as the training truth label to update the parameters of the Actor network and the Critic network, it is possible to use the optimal vehicle control sequence to guide the Actor network to select actions, rather than blindly exploring, thereby accelerating the update and convergence of the Actor network. In addition, using the optimal vehicle control sequence to guide the Critic network to adjust the value estimates corresponding to different driving environment states can effectively alleviate the problem of the Critic network's difficulty in convergence. In summary, the technical solution of the present application can significantly improve the training efficiency and training effect of the reinforcement learning network.

[0026] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0028] Figure 1 It is a schematic diagram of the architecture of a vehicle control strategy generation system provided by an exemplary embodiment.

[0029] Figure 2 It is a flowchart of a vehicle control strategy generation model training method provided by an exemplary embodiment.

[0030] Figure 3It is a schematic diagram of the architecture of a vehicle control strategy generation model provided by an exemplary embodiment.

[0031] Figure 4 It is a flowchart of a vehicle control strategy generation method provided by an exemplary embodiment.

[0032] Figure 5 It is a schematic structural diagram of an electronic device provided by an exemplary embodiment.

[0033] Figure 6 It is a block diagram of a training device for a vehicle control strategy generation model provided by an exemplary embodiment.

[0034] Figure 7 It is a block diagram of a vehicle control strategy generating device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0035] Next, one or more embodiments of the present application are described in detail.

[0036] Figure 1 FIG. 1 is a schematic diagram of the architecture of a vehicle control strategy generation system provided by an exemplary embodiment. Figure 1 As shown, the vehicle control strategy generation system may include a server 11, a network 12, and several vehicles, such as vehicles 13-14.

[0037] The server 11 may be a physical server including an independent host, or the server 11 may be a virtual server carried by a host cluster. During operation, the server 11 may run a server-side program of the vehicle control strategy generation system to realize the server end of the vehicle control strategy generation system, which may also be referred to as a corresponding vehicle control strategy generation platform.

[0038] The vehicles 13-14 may be any vehicles. The vehicles 13-14 and the server 11 may be directly or indirectly connected via wired communication or wireless communication, and this application does not impose any special restrictions.

[0039] based on Figure 1 In the vehicle control strategy generation system shown, any vehicle among the vehicles 13 to 14 can obtain its own driving environment state and submit the obtained driving environment state to the server 11. The server 11 inputs the received driving environment state into the vehicle control strategy generation model, obtains the target vehicle control strategy output by the vehicle control strategy generation model, and returns the target vehicle control strategy to the vehicle that submitted the driving environment state. The vehicle can implement the target vehicle control strategy to control itself.

[0040] In another embodiment, the vehicle control strategy generation system may only include the vehicle. In this case, the vehicle control strategy generation model is deployed locally in the vehicle. Figure 1 Taking vehicle 13 in the example, vehicle 13 can obtain its own driving environment state, and input the driving environment state into the vehicle control strategy generation model to obtain a target vehicle control strategy corresponding to the driving environment state. Then, vehicle 13 can implement the target vehicle control strategy to control itself. In the actual implementation process of the present application scheme, the vehicle control strategy generation system can be integrated into the vehicle's smart cockpit, so that the smart cockpit can implement the vehicle control strategy generation method described in this application.

[0041] The vehicle control strategy generation model described in the above embodiment can be obtained through certain training. Figure 2 , Figure 2 A flowchart of a training method for a vehicle control strategy generation model provided for an exemplary embodiment. First of all, it should be noted that the vehicle control strategy generation model in the present application can be constructed based on the principle of reinforcement learning. In reinforcement learning, the Actor-Critic architecture is a commonly used policy gradient method. Therefore, the vehicle control strategy generation model can include an Actor network and a Critic network. Generally, there is another basic element in reinforcement learning-an agent. The goal of reinforcement learning is to learn a strategy that tells the agent what actions to take in any given state to maximize the cumulative reward. The Actor network is usually responsible for outputting the action probability distribution, which directly determines how the agent chooses actions based on the current state. The Critic network is responsible for evaluating the quality of the strategy (which is mainly composed of the output of the Actor network). In the embodiment of the present application, the vehicle can be regarded as an agent.

[0042] like Figure 2 As shown, the training method of the vehicle control strategy generation model may include the following steps:

[0043] S201. Obtain a sample driving environment state corresponding to a sample time step, and input the sample driving environment state into a muscle memory model and a vehicle control strategy generation model respectively, wherein the Actor network is used to output a predicted vehicle control action probability distribution corresponding to the sample driving environment state, and the Critic network is used to output a predicted value estimate corresponding to the sample driving environment state. The muscle memory model is trained based on prior knowledge of vehicle control actions, and is used to output a target vehicle control sequence corresponding to the sample driving environment state, wherein the target vehicle control sequence includes a target vehicle control action and a target value estimate.

[0044] During vehicle driving, the driving environment state corresponding to each time step is not exactly the same. Therefore, during the training process, it is necessary to obtain the sample driving environment state corresponding to the sample time step, so as to generate a predicted vehicle control strategy based on the sample driving environment state, and then update the parameters of the vehicle control strategy generation model based on the predicted vehicle control strategy and the target vehicle control strategy corresponding to the sample driving environment state.

[0045] Exemplarily, the sample driving environment state may include sample driving state information of the vehicle (such as vehicle speed, acceleration, wheel steering, etc.) and sample environment information outside the vehicle (such as obstacle conditions, traffic congestion conditions, etc.). Then, the sample driving environment state is input into the muscle memory model and the vehicle control strategy generation model respectively. Among them, the Actor network included in the vehicle control strategy generation model will output the predicted vehicle control action probability distribution corresponding to the sample driving environment state, and the predicted vehicle control action probability distribution will include: at least one predicted vehicle control action and the probability of each predicted vehicle control action being selected in the sample driving environment state. The Actor network included in the vehicle control strategy generation model will output the predicted value estimate corresponding to the sample driving environment state, and the predicted value estimate can be understood as: the expected value of the future cumulative reward after the vehicle selects the predicted vehicle control action with the highest probability in the predicted vehicle control action probability distribution. It should be noted that the predicted vehicle control action probability output by the Actor network constitutes the aforementioned "predicted vehicle control strategy", and the predicted value estimate output by the Critic network reflects the quality of the "predicted vehicle control strategy".

[0046] The muscle memory model is trained based on the prior knowledge of vehicle control actions. Prior knowledge may include, but is not limited to: control data of the vehicle by professional-level drivers (such as turning radius control, braking habit control, following vehicle safety distance control, etc.), high-quality historical vehicle control data (such as collecting and mining vehicle control data without safety accidents to statistically obtain high-quality vehicle control data). There are many ways to train the muscle memory model, and this application does not limit this. For example, the muscle memory model can be trained through Transformer. The trained muscle memory model can take the vehicle driving environment state as input and output the optimal vehicle control sequence suitable for the vehicle driving environment state. The optimal vehicle control sequence can be used as the true value label of the training stage. Therefore, the optimal vehicle control sequence output by the muscle memory model can be called the "target vehicle control sequence", and the target vehicle control sequence includes the target vehicle control action (i.e., the optimal vehicle control action under the sample driving environment state) and the target value estimate (i.e., taking the optimal vehicle control action under the sample driving environment state can obtain the maximum future cumulative reward). It should be noted that the target vehicle control action constitutes the "target vehicle control strategy" mentioned above.

[0047] S202. Update the network parameters of the Actor network based on the target vehicle control action and the predicted vehicle control action probability distribution.

[0048] In one embodiment, before starting training, the network parameters of the Actor network can be obtained by random initialization. The muscle memory model can output a target vehicle control action probability distribution, which will include: at least one vehicle control action and the probability of each vehicle control action being selected in the sample driving environment state. The "target vehicle control action" refers to the vehicle control action with the highest probability in the target vehicle control action probability distribution. When the predicted vehicle control action probability distribution output by the Actor network includes the target vehicle control action, the predicted probability corresponding to the target vehicle control action can be determined from the predicted vehicle control action probability distribution. Then, the action loss value is calculated based on the difference between the predicted probability and the probability corresponding to the target vehicle control action in the target vehicle control action probability distribution, and then the network parameters of the Actor network are updated based on the action loss value. The parameter update process is an iterative training process.

[0049] For example, in any iteration process, it is assumed that the original network parameter of the Actor network is parameter θ, and the target vehicle control action is action a. Then, the principle of calculating the action loss value can refer to the following formula (1):

[0050]

[0051] In formula (1), a represents the target vehicle control action; s represents the sample driving environment state; p(a|s; θ) represents the predicted probability corresponding to the target vehicle control action, that is, the Actor network predicts the probability of taking the target vehicle control action under the sample driving environment state based on the original network parameter θ; p(a|s) represents the probability corresponding to the target vehicle control action in the target vehicle control action probability distribution; Loss(θ) represents the action loss value.

[0052] Next, use gradient descent or other optimization algorithms to calculate the gradient of the loss function (Loss(θ)) with respect to the parameter (θ) Update the network parameters of the Actor network according to the gradient. The update principle can be referred to formula (2):

[0053]

[0054] In formula (2), θ on the left side of the arrow is the updated network parameter, θ on the right side of the arrow is the network parameter before the update, and α is the preset learning rate of the Actor network. is the gradient. The network parameters of the Actor network are updated from θ on the right side of the arrow to θ on the left side of the arrow, and iterative training is performed based on the updated network parameters until the iterative training stops. It should be noted that the stopping conditions of the iterative training may include but are not limited to: reaching the number of iterations, minimizing the action loss value, and the similarity between the predicted vehicle control action probability distribution and the target vehicle control action probability distribution reaches a preset threshold. Those skilled in the art can set the stopping conditions of the iterative training according to actual needs, and this application is not limited to this.

[0055] In the above content, the target vehicle control action probability distribution output by the muscle memory model is used as the supervision signal of the Actor network. By comparing the output of the muscle memory model and the Actor network, the Actor network is guided to choose actions purposefully instead of blindly exploring. This can effectively accelerate the update and convergence of the Actor network and improve the training effect of the Actor network.

[0056] In addition, the target vehicle control sequence output by the muscle memory model can also include immediate rewards, state value weights, and the sample backward driving environment state corresponding to the next time step of the sample time step. Furthermore, the positive / negative feedback of the immediate reward can be used to guide the agent (i.e., the vehicle) to accelerate learning. The network parameters of the Actor network are initialized or adjusted through the state value weights. Helping the Actor network understand the state transfer relationship based on the sample driving environment state and the sample backward driving environment state helps the Actor network better plan future vehicle control sequences.

[0057] S203: Update the network parameters of the Critic network based on the target value estimate and the predicted value estimate.

[0058] It should be noted that steps S202 and S203 may be implemented in parallel or in series, and the present application does not limit the order in which the two steps are implemented.

[0059] In one embodiment, the process of updating the network parameters of the Critic network may include the following operations: first, determine the target value (i.e., "TD target") corresponding to the Critic network based on the predicted value estimate output by the Critic network. Exemplarily, the target value y is calculated based on the conventional Actor-Critic algorithm, which is usually obtained by estimating the predicted value corresponding to the current predicted instant reward and the sample backward driving environment state, as shown in formula (3):

[0060] y=r t +γV(s t+1 θ - ) (3)

[0061] In formula (3), y represents the target value; t represents the sample time step; r t Represents the immediate reward of the current prediction; t+1 Characterizes the state of the sample's backward driving environment; θ - Characterize the network parameters of the target network of the Critic network; V(s t+1 θ - ) represents the predicted value estimate corresponding to the sample's backward driving environment state; γ represents the discount factor.

[0062] Then, the statistic between the target value y and the target value estimate output by the muscle memory model is calculated, and the statistic is used as the new target value estimate of the Critic network. The statistic can be calculated by weighted average, standard deviation, etc., which is not limited in this application. Taking weighted average as an example, the calculation method of the statistic can refer to formula (4):

[0063] y combined =β*V m (s;θ m )+(1-β)*y (4)

[0064] In formula (4), y combined represents a statistic, i.e., a new target value estimate; β represents a weight between 0 and 1, which is used to control the influence of the target value estimate output by the muscle memory model in the update of the Critic network; s represents the state of the sample driving environment; θ m Model parameters that characterize the muscle memory model; V m (s;θ m ) represents the target value estimate output by the muscle memory model; y represents the target value.

[0065] Then, the value loss value is calculated based on the difference between the new target value estimate and the predicted value estimate output by the Critic network, and the network parameters of the Critic network are updated based on the value loss value. The calculation principle of the value loss value can be referred to formula (5):

[0066] L=(V c (s;θ c )-y combined ) 2 (5)

[0067] In formula (5), L represents the value loss; s represents the sample driving environment state; θ c Characterize the network parameters of the Critic network; V c (s;θ c ) represents the prediction value estimate of the output of the Critic network; combined Characterize new target value estimates.

[0068] The principle of updating the network parameters of the Critic network can be referred to formula (6):

[0069]

[0070] In formula (6), Δθ c Represents the decrease value of the network parameters of the Critic network; α represents the preset learning rate; Characterizes the gradient of the loss function with respect to the Critic network parameters. When the drop value Δθ c When the minimization is achieved, it can be determined that the iterative training of the Critic network is completed.

[0071] In this embodiment, a new target value estimate is created by combining the target value estimate output by the muscle memory model. Since the new target value estimate combines the short-term value estimate (i.e., the target value y) and the long-term value estimate (i.e., the target value estimate output by the muscle memory model), the Critic network is iteratively trained based on the new target value estimate. This helps the Critic network to converge more stably during the training process, while significantly improving the Critic network's evaluation accuracy of the value estimate.

[0072] In the above embodiment, since the muscle memory model is trained based on the prior knowledge of vehicle control actions, the target vehicle control sequence it outputs can be understood as the optimal vehicle control sequence under the sample driving environment state determined based on the prior knowledge. By using the optimal vehicle control sequence as the training truth label to update the parameters of the Actor network and the Critic network, it is possible to use the optimal vehicle control sequence to guide the Actor network to select actions, rather than blindly exploring, thereby accelerating the update and convergence of the Actor network. In addition, using the optimal vehicle control sequence to guide the Critic network to adjust the value estimates corresponding to different driving environment states can effectively alleviate the problem of the difficulty of the Critic network to converge. In summary, the technical solution of the present application can significantly improve the training efficiency and training effect of the reinforcement learning network.

[0073] In one embodiment, the target vehicle control sequence output by the muscle memory model may also include a sample driving environment state and a sample backward driving environment state corresponding to the next time step of the sample time step. In other words, the target vehicle control sequence may be expressed as (s t ,a i ,s t+1 ,V m (s;θ m )). Among them, s t represents the sample driving environment state corresponding to the sample time step, a i represents the target vehicle control action corresponding to the sample driving environment state, st+1 Indicates the sample backward driving environment state corresponding to the next time step of the sample time step, V m (s;θ m ) represents the target value estimate corresponding to the sample driving environment state. The target vehicle control sequence can be stored in the experience replay buffer. At least one candidate vehicle control sequence is stored in the experience replay buffer, and each vehicle control sequence includes a candidate driving environment state, a candidate vehicle control action, a candidate backward driving environment state, and a candidate value estimate. Any candidate vehicle control sequence cached in the experience replay buffer can be regarded as an experience, and experience is selected from the experience replay buffer as a training sample to iteratively train the Critic network. In other words, a candidate vehicle control sequence is selected from the experience replay buffer to update the network parameters of the Critic network based on the selected candidate vehicle control sequence.

[0074] Since the candidate vehicle control sequences stored in the experience replay buffer are all successful sequences output by the muscle memory model under different driving environment states, storing these successful sequences and directly selecting sequences from the experience replay buffer as training samples for the vehicle control strategy generation model can avoid collecting training samples for each training, thereby improving the training effect and training efficiency of the vehicle control strategy generation model.

[0075] In one embodiment, the output of the muscle memory model may also include a state value weight, the length of which may be related to the number of experiences in the experience replay buffer, or a more abstract representation. Thus, each experience stored in the experience replay buffer may also include a candidate state value weight. The process of selecting a candidate vehicle control sequence from the experience replay buffer and updating the network parameters of the Critic network using the selected candidate vehicle control sequence (hereinafter referred to as the "first candidate vehicle control sequence") may include:

[0076] First, the candidate value estimates included in the first candidate vehicle control sequence are adjusted according to the first state value weight to obtain the adjusted target value. The first state value weight refers to the candidate state value weight included in the first candidate vehicle control sequence. The above adjustment process can refer to formula (7):

[0077]

[0078] In formula (7), y represents the adjusted target value. r represents the immediate reward obtained by the agent from the environment after taking action a in the sample driving environment state. γ represents the discount factor, which is used to balance the importance of immediate rewards and future rewards. When γ is close to 0, the agent pays more attention to immediate rewards; when γ is close to 1, the agent pays more attention to future long-term rewards. s ′ Represents the new state that the agent transfers to after taking action a.′ Representation in the new state s ′ One of all possible actions to be taken under this state is used to calculate ′ The maximum Q value under θ - Characterizes the network parameters of the target network, where the target network is introduced to improve the stability of the learning process; the parameters of the target network are usually copied from the Critic network and remain unchanged for a period of time, or are updated at a slower rate, which can reduce the fluctuation of the target during the learning process and make the training more stable. ′ ,a ′ θ - ) characterizes that this is the Critic network (using the parameters θ- of the target network) in the new state s ′ Take action a ′ An estimate of the expected return that can be obtained. Representation in the new state s ′ Next, by traversing all possible actions a ′ , and find the value that makes Q(s ′ ,a ′ θ - )The biggest action a ′ , and then calculate the Q value corresponding to the action. muscle (e) represents the first state value weight, and e represents the first candidate vehicle control sequence. It should be emphasized that w muscle (e) is the state value weight output by the muscle memory model based on the first candidate vehicle control sequence.

[0079] It should be noted that the first candidate vehicle control sequence includes: a first candidate driving environment state, a first candidate vehicle control action, a first candidate backward driving environment state and a first candidate value estimate. Then, determine the predicted value estimate output by the Critic network for the first candidate vehicle control sequence. That is, determine the future cumulative reward predicted by the Critic network after taking the first candidate vehicle control action under the first candidate driving environment state. Then, calculate the loss value based on the difference between the adjusted target value and the predicted value estimate output by the Critic network for the first candidate vehicle control sequence, and update the network parameters of the Critic network based on the loss value. The update of the network parameters is based on gradient descent, please refer to formula (8):

[0080]

[0081] In formula (8), θ on the left side of the arrow is the updated network parameter, θ on the right side of the arrow is the network parameter before the update, α is the preset learning rate of the Actor network, is the gradient of the loss function with respect to the parameters θ before updating.

[0082] In this embodiment, the state value weight output by the muscle memory model is used to correct the experience stored in the experience playback buffer, thereby controlling the parameter update of the Critic network. Specifically, the loss function and the muscle memory model are used to balance the new experience and the old experience, thereby adjusting the parameter update strategy of the Critic network, thereby preventing the Critic network from over-relying on historical experience and giving up exploration, and avoiding the problem that the strategy generated by the vehicle control strategy generation model falls into the local optimum.

[0083] In one embodiment, in the related art, the parameter update of the critic network is usually based on the mean square error loss function, that is, minimizing the difference between the predicted value estimate and the actual value estimate. The update principle can refer to formula (9):

[0084]

[0085] In formula (9), Δw represents the decrease value of the critic network parameter. α represents the preset learning rate. s represents the sample driving environment state. w represents the network parameter of the critic network. V(s;w) represents the predicted value estimate of the critic network output based on the network parameter w. Characterizes the gradient of the loss function with respect to the network parameter w. TD target The actual value estimation can be obtained by the immediate reward and the predicted value estimation corresponding to the backward driving environment of the sample by the Critic network.

[0086] In the case where rewards are sparse or difficult to obtain, the above-mentioned update method will make the agent unable to explore the best strategy in the case of sparse rewards, or give up exploration because it is difficult to obtain rewards. Based on this, the present application proposes: using the target value estimate output by the muscle memory model to guide the Critic network. Exemplarily, the target value estimate output by the muscle memory model is combined with the predicted value estimate output by the Critic network to obtain a combined value estimate. Among them, there are many ways to combine, and the present application does not limit the specific combination of the target value estimate and the predicted value estimate. Taking weighted average as an example of a combination method, the principle of the combination process can refer to formula (10):

[0087] combined value =β*V muscle (s)+(1-β)*V(s;w) (10)

[0088] In formula (9), V muscle(s) represents the target value estimate output by the muscle memory model. β represents a weight between 0 and 1, which is used to indicate the degree of trust in the target value estimate. V(s;w) represents the predicted value estimate output by the Critic network based on the network parameter w. combined value Characterize the combined value estimate. Then, the network parameters of the Critic network are updated according to the difference between the combined value estimate and the predicted value estimate output by the Critic network. The updating principle can be referred to in formula (11):

[0089]

[0090] In formula (11), combined value Characterize the combined value estimate. α, V(s; w) and The meaning of is the same as that in formula (9).

[0091] In this embodiment, the target value estimate is combined with the predicted value estimate. Since the target value estimate reflects the long-term reward trend, the combined value estimate obtained by the combination can provide a stable learning target for the Critic network, thereby achieving stable guidance of the intelligent agent to explore in a reward-sparse scenario. At the same time, capturing the long-term reward trend helps to increase the generalization ability of the Critic network. In addition, during the combination process, the Critic network's attention to immediate rewards (reflected by TD targets) and long-term values ​​(reflected by muscle memory networks) can be balanced by adjusting the weight β. This balance helps the Critic network find an optimal solution that takes both short-term and long-term benefits into account during training, thereby improving overall performance.

[0092] In one embodiment, the vehicle control strategy generation model also includes a reward model, which is used to output an immediate reward. This immediate reward can be understood as a reward obtained from the environment for taking any action in any state. In this embodiment, the target vehicle control sequence output by the muscle memory model also includes a target immediate reward. It should be noted that the target vehicle control sequence can be represented as a four-tuple (s t ,a i ,r i ,s t+1 ,)). Among them, s t represents the sample driving environment state corresponding to the sample time step, a i represents the target vehicle control action corresponding to the sample driving environment state, r i represents the target immediate reward corresponding to the sample driving environment state, s t+1Indicates the sample backward driving environment state corresponding to the next time step of the sample time step. In this case, when storing, the four-tuple can be associated with the corresponding state value weight and stored in the experience playback buffer area.

[0093] In this embodiment, the predicted vehicle control action probability distribution output on the Actor network is input into the reward model, and the reward model will output a predicted instant reward, which corresponds to the sample driving environment state. The predicted instant reward refers to the instant reward that can be obtained from the environment after taking the predicted vehicle control action probability distribution under the sample driving environment state. Then, the prediction error of the reward model is calculated based on the target instant reward and the predicted instant reward output by the reward model, and the model parameters of the reward model are updated based on the prediction error. It should be noted that this application does not limit the calculation method of the prediction error. For example, the prediction error can be calculated using MSE (Mean Squared Error), and the calculation principle can refer to formula (12):

[0094]

[0095] In formula (12), MSE represents the prediction error, n represents the number of samples, and y i Represents the target immediate reward corresponding to the i-th sample, Characterizes the predicted immediate reward corresponding to the i-th sample. It should be noted that each sample may be a sample driving environment state or a sample driving environment state-action pair. The smaller the MSE value, the closer the predicted immediate reward of the reward model is to the target immediate reward, that is, the reward model and the muscle memory model are consistent and unbiased. A low MSE value (for example, close to 0) indicates that the prediction of the reward model is very accurate.

[0096] In this embodiment, the output of the muscle memory model is used to guide the parameter update of the reward model, which can prevent the situation where the intelligent agent training cannot achieve the expected training goals due to unreasonable parameter settings of the reward model.

[0097] See also Figure 3 , Figure 3 FIG. 1 is a schematic diagram of the architecture of a vehicle control strategy generation model shown in an exemplary embodiment. Figure 3In the model, there are Actor network, Critic network, intelligent agent, reward model and muscle memory model. During the training process, the sample driving environment state is input into the Actor network and the muscle memory model respectively to obtain the predicted vehicle control action probability distribution output by the Actor network and the target vehicle control sequence output by the muscle memory model. The network parameters of the Actor network are updated based on the target vehicle control action and the predicted vehicle control action probability distribution. The sample driving environment state and the predicted vehicle control action probability are input into the Critic network to obtain the predicted value estimate, and then the network parameters of the Critic network are updated according to the predicted value estimate and the target value estimate contained in the target vehicle control action.

[0098] In one embodiment, exemplarily, the target vehicle control sequence output by the muscle memory network includes the target vehicle control action, the sample driving environment state, the sample backward driving environment state corresponding to the next time step of the sample time step, and the target immediate reward. In addition, the muscle memory network also outputs the state value weight. Therefore, the target vehicle control sequence and the state value weight can be associated and stored in the experience replay buffer area, so that the vehicle control sequence stored in the experience replay buffer area can be used as a training sample to update the parameters of the Critic network. The update of the reward model can refer to the above content and will not be repeated here.

[0099] The vehicle control strategy generation model trained by the method shown in any of the above embodiments can generate the corresponding optimal vehicle control strategy under various driving environment conditions of the vehicle, so that the long-term value can be maximized by controlling the vehicle according to the optimal vehicle control strategy. Figure 4 , Figure 4 FIG. 1 is a flow chart of a method for generating a vehicle control strategy according to an exemplary embodiment. The flow chart may include the following steps:

[0100] S401. Acquire a driving environment state of a vehicle, where the driving environment state includes driving state information of the vehicle and environment information outside the vehicle.

[0101] The driving environment status of the vehicle may include: the driving status information of the vehicle (such as vehicle speed, acceleration, wheel steering, etc.) and the external environment information of the vehicle (such as obstacle conditions, traffic congestion conditions, etc.).

[0102] S402. Input the driving environment state into a vehicle control strategy generation model to obtain a target vehicle control strategy corresponding to the driving environment state, wherein the vehicle control strategy generation model is trained by the method shown in any of the aforementioned embodiments, and the target vehicle control strategy includes a target vehicle control action.

[0103] Exemplarily, the vehicle control strategy generation model can use the Bellman optimal equation to obtain the optimal state value function and the optimal action value function, so that the Actor network outputs the optimal vehicle control sequence based on the optimal action value function, and the Critic network generates the maximum value estimate based on the optimal state value function. The principle of this process can be referred to the following formulas (13) to (14):

[0104]

[0105] In formulas (13) to (14), V * (s) represents the optimal state value function. Q * (s,a) represents the optimal action value function. p(s ′ ,r|s,a) represents the probability of transitioning from state s to state s' and obtaining reward r after taking action a. γ is the discount factor.

[0106] The existence and uniqueness of Bellman's optimal solution can be proved by using the law of compression mapping: if there is a compression mapping T, that is, there is a constant γ (0≤γ<1), such that for all v1, v2: ||T(v1)-T(v2)||≤γ||v1-v2||, then the iteration of the compression mapping T will converge to a unique limit point v*. This v* is the solution to the Bellman optimal equation, that is, the optimal value function. Then by maximizing q*(s,a)=r(s,a)+γ∑p(s ′ |s,a)v*(s ′ ) to solve the optimal strategy π*. v*(s) is the optimal value function in state s.

[0107] The generated target vehicle control strategy includes the target vehicle control action. The vehicle can call the corresponding functional components to implement the target vehicle control action to achieve the best vehicle control management. For example, the target vehicle control action is: control the turning radius to 15-25 meters. Then the vehicle's steering gear or steering mechanism can be called to implement the target vehicle control action.

[0108] In this embodiment, a trained vehicle control strategy generation model is used (through reinforcement learning) to generate the optimal vehicle control strategy corresponding to each driving environment state, and then the vehicle is controlled according to the optimal vehicle control strategy, which helps to improve the performance and safety of the vehicle under different driving conditions.

[0109] Figure 5 is a schematic diagram of the structure of an electronic device according to an exemplary embodiment of the present application. Figure 5At the hardware level, the electronic device includes a processor 502, an internal bus 504, a network interface 506, a memory 508, and a non-volatile memory 510, and may also include hardware required for other services. The processor 502 reads the corresponding computer program from the non-volatile memory 510 into the memory 508 and then runs it. Of course, in addition to the software implementation, this application does not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0110] With the above Figure 2 Corresponding to the illustrated method embodiment, the present application also provides an embodiment of a training device for a vehicle control strategy generation model.

[0111] Figure 6 is a block diagram of a training device for a vehicle control strategy generation model according to an exemplary embodiment of the present application. Figure 6 , the device comprises an acquisition unit 601, a first updating unit 602 and a second updating unit 603, wherein:

[0112] The acquisition unit 601 is configured to acquire a sample driving environment state corresponding to a sample time step, and input the sample driving environment state into a muscle memory model and a vehicle control strategy generation model respectively, wherein the Actor network is used to output a predicted vehicle control action probability distribution corresponding to the sample driving environment state, and the Critic network is used to output a predicted value estimate corresponding to the sample driving environment state, and the muscle memory model is obtained by training based on prior knowledge of vehicle control actions, and is used to output a target vehicle control sequence corresponding to the sample driving environment state, and the target vehicle control sequence includes a target vehicle control action and a target value estimate;

[0113] A first updating unit 602 is configured to update a network parameter of the Actor network based on the target vehicle control action and the predicted vehicle control action probability distribution;

[0114] The second updating unit 603 is configured to update the network parameters of the Critic network based on the target value estimate and the predicted value estimate.

[0115] Optionally, the muscle memory model also outputs a probability distribution of a target vehicle control action, and the probability corresponding to the target vehicle control action in the target vehicle control action probability distribution is the largest, and the first update unit 602 is specifically used to: when the predicted vehicle control action probability distribution includes the target vehicle control action, determine the predicted probability corresponding to the target vehicle control action in the predicted vehicle control action probability distribution, and calculate the action loss value based on the difference between the predicted probability and the probability corresponding to the target vehicle control action in the target vehicle control action probability distribution; and update the network parameters of the Actor network based on the action loss value.

[0116] Optionally, the second updating unit 603 is specifically used to: determine a target value corresponding to the Critic network based on the predicted value estimate; calculate a statistic based on the target value and the target value estimate, and use the statistic as a new target value estimate for the Critic network; calculate a value loss value based on the difference between the new target value estimate and the predicted value estimate, and update the network parameters of the Critic network based on the value loss value.

[0117] Optionally, the target vehicle control sequence further includes the sample driving environment state and a sample backward driving environment state corresponding to the next time step of the sample time step; the device further includes:

[0118] The cache unit 604 is configured to store the target vehicle control sequence in an experience replay cache area, wherein the experience replay cache area stores at least one candidate vehicle control sequence, each candidate vehicle control sequence includes a candidate driving environment state, a candidate vehicle control action, a candidate backward driving environment state and a candidate value estimate; select a candidate vehicle control sequence from the experience replay cache area, and update the network parameters of the Critic network based on the selected candidate vehicle control sequence.

[0119] Optionally, any candidate vehicle control sequence also includes a candidate state value weight; the updating of the network parameters of the Critic network based on the selected candidate vehicle control sequence includes: adjusting the candidate value estimate contained in the selected candidate vehicle control sequence based on a first state value weight to obtain an adjusted target value, wherein the first state value weight is the candidate state value weight contained in the selected candidate vehicle control sequence; calculating a loss value based on the adjusted target value and a predicted value estimate output by the Critic network for the selected candidate vehicle control sequence, and updating the network parameters of the Critic network based on the loss value.

[0120] Optionally, the device further comprises:

[0121] The combination updating unit 605 is configured to combine the target value estimate with the predicted value estimate to obtain a combined value estimate; and update the network parameters of the Critic network according to the difference between the combined value estimate and the predicted value estimate.

[0122] Optionally, the vehicle control strategy generation model further includes a reward model, and the target vehicle control sequence further includes a target immediate reward; and the device further includes:

[0123] The reward model updating unit 606 is configured to calculate the prediction error of the reward model based on the target instant reward and the predicted instant reward output by the reward model, wherein the predicted instant reward corresponds to the sample driving environment state; and update the model parameters of the reward model according to the prediction error.

[0124] With the above Figure 4 Corresponding to the method embodiment shown, the present application also provides an embodiment of a vehicle control strategy generating device.

[0125] Figure 7 is a block diagram of a vehicle control strategy generation device according to an exemplary embodiment of the present application. Figure 7 , the device comprises an acquisition unit 701 and an input unit 702, wherein:

[0126] An acquisition unit 701 is configured to acquire a driving environment state of a vehicle, wherein the driving environment state includes driving state information of the vehicle and environment information outside the vehicle;

[0127] The input unit 702 is configured to input the driving environment state into a vehicle control strategy generation model to obtain a target vehicle control strategy corresponding to the driving environment state, wherein the vehicle control strategy generation model is trained by the method shown in any of the aforementioned embodiments, and the target vehicle control strategy includes a target vehicle control action.

[0128] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0129] The devices or modules described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0130] In a typical configuration, a computer includes one or more processors, including a central processing unit (CPU) and a graphics processing unit (GPU), an input / output interface, a network interface, and a memory. The CPU is used for computing simulations, and the GPU is used for outputting high-quality 3D images.

[0131] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0132] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0133] Corresponding to the embodiments of the aforementioned method, the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of any embodiment of the aforementioned method are implemented.

[0134] Corresponding to the embodiments of the aforementioned method, the present application also provides a computer program product, including a computer program and / or instructions, which implement the steps of any embodiment of the aforementioned method when executed by a processor.

[0135] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A training method for a vehicle control strategy generation model, characterized in that: The vehicle control strategy generation model includes an Actor network and a Critic network, and the method includes: Obtaining a sample driving environment state corresponding to a sample time step, and inputting the sample driving environment state into a muscle memory model and a vehicle control strategy generation model respectively, wherein the Actor network is used to output a predicted vehicle control action probability distribution corresponding to the sample driving environment state, and the Critic network is used to output a predicted value estimate corresponding to the sample driving environment state, and the muscle memory model is obtained by training based on prior knowledge of vehicle control actions, and is used to output a target vehicle control sequence corresponding to the sample driving environment state, wherein the target vehicle control sequence includes a target vehicle control action and a target value estimate; Update the network parameters of the Actor network based on the target vehicle control action and the predicted vehicle control action probability distribution; The network parameters of the critic network are updated based on the target value estimate and the predicted value estimate.

2. The method according to claim 1, characterized in that: The muscle memory model further outputs a target vehicle control action probability distribution, and the probability corresponding to the target vehicle control action in the target vehicle control action probability distribution is the largest, and updating the network parameters of the Actor network based on the target vehicle control action and the predicted vehicle control action probability distribution includes: In a case where the predicted vehicle control action probability distribution includes the target vehicle control action, determining a predicted probability corresponding to the target vehicle control action in the predicted vehicle control action probability distribution, and calculating an action loss value according to a difference between the predicted probability and a probability corresponding to the target vehicle control action in the target vehicle control action probability distribution; Update the network parameters of the Actor network according to the action loss value.

3. The method according to claim 1, characterized in that The updating of the network parameters of the Critic network based on the target value estimate and the predicted value estimate includes: Determining a target value corresponding to the Critic network based on the predicted value estimate; Calculating a statistic based on the target value and the target value estimate, and using the statistic as a new target value estimate of the Critic network; A value loss value is calculated according to the difference between the new target value estimate and the predicted value estimate, and a network parameter of the Critic network is updated according to the value loss value.

4. The method according to claim 1, characterized in that The target vehicle control sequence also includes the sample driving environment state and the sample backward driving environment state corresponding to the next time step of the sample time step; the method also includes: Storing the target vehicle control sequence in an experience replay buffer, wherein the experience replay buffer stores at least one candidate vehicle control sequence, each candidate vehicle control sequence including a candidate driving environment state, a candidate vehicle control action, a candidate backward driving environment state, and a candidate value estimate; A candidate vehicle control sequence is selected from the experience replay buffer, and a network parameter of the critic network is updated based on the selected candidate vehicle control sequence.

5. The method according to claim 4, characterized in that Any candidate vehicle control sequence also includes a candidate state value weight; the updating of the network parameters of the Critic network based on the selected candidate vehicle control sequence includes: adjusting the candidate value estimates included in the selected candidate vehicle control sequence based on a first state value weight to obtain an adjusted target value, wherein the first state value weight is a candidate state value weight included in the selected candidate vehicle control sequence; A loss value is calculated based on the adjusted target value and the predicted value output by the Critic network for the selected candidate vehicle control sequence, and a network parameter of the Critic network is updated based on the loss value.

6. The method according to claim 1, characterized in that The method further comprises: Combining the target value estimate with the predicted value estimate to obtain a combined value estimate; The network parameters of the Critic network are updated according to the difference between the combined value estimate and the predicted value estimate.

7. The method according to claim 1, characterized in that The vehicle control strategy generation model further includes a reward model, and the target vehicle control sequence further includes a target immediate reward; the method further includes: Calculating a prediction error of the reward model according to the target immediate reward and a predicted immediate reward output by the reward model, wherein the predicted immediate reward corresponds to the sample driving environment state; A model parameter of the reward model is updated according to the prediction error.

8. A vehicle control strategy generation method, characterized in that: include: Acquiring a driving environment state of a vehicle, wherein the driving environment state includes driving state information of the vehicle and environment information outside the vehicle; The driving environment state is input into a vehicle control strategy generation model to obtain a target vehicle control strategy corresponding to the driving environment state, wherein the vehicle control strategy generation model is trained by the method described in any one of claims 1-7, and the target vehicle control strategy includes a target vehicle control action.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 8 by running the executable instructions.

10. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer program product, characterized in that The method comprises a computer program and / or instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 8.

Images