Strategy network training method and device, computer equipment and storage medium

Abstract

The invention discloses a strategy network training method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining a first state observed by an intelligent agent, and determining a plurality of executable actions of the intelligent agent and an action value vector corresponding to each executable action; determining a target action from the executable actions by utilizing the action value vector, and obtaining a first reward obtained by executing the target action by the intelligent agent in the first state; constructing a current decision data point based on the first state and the target action, and calculating a second reward by using the current decision data point and a preset demonstration sample; obtaining a second state of the intelligent agent after executing the target action, and constructing interaction data based on the first state, the target action, the first reward, the second reward and the second state; and training the target policy network by using the interaction data to obtain a trained target policy network. According to the method, the problems of invalid strategy learning and training interference caused by environment over-fitting and poor demonstration data quality in reinforcement learning reality are solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a training method, device, computer equipment and storage medium for a strategy network. Background Art

[0002] In the field of machine learning and reinforcement learning technology, reinforcement learning is an algorithmic framework for solving sequential decision-making problems. It learns strategies through the interaction between the agent and the environment with the goal of maximizing cumulative rewards. After decades of theoretical research and development, reinforcement learning methods have been applied in many fields such as simulation, robot control, game theory, and unmanned driving. However, in real-world scenarios, the application of reinforcement learning faces many challenges.

[0003] At present, there are the following main problems in the application of reinforcement learning in real-world scenarios: First, the reward function is difficult to design. Researchers can usually only judge how to reward the agent based on the final result. For tasks with longer sequences, strategy learning is difficult to carry out effectively; second, there is an overfitting problem in the environment, which makes it difficult for strategies trained in virtual environments to make optimal decisions in real-world scenarios. The existing two-stage imitation learning method can efficiently learn strategies when high-quality demonstration data is obtained, but if the quality of the demonstration data is poor, effective strategies cannot be learned and may even interfere with subsequent training. Summary of the invention

[0004] In view of this, embodiments of the present invention provide a training method, apparatus, computer device and storage medium for a policy network to solve the problems of overfitting of the environment in real-world reinforcement learning, poor quality of demonstration data leading to invalid policy learning and interference with training.

[0005] In a first aspect, an embodiment of the present invention provides a method for training a policy network, the method comprising:

[0006] Acquire a first state observed by the agent, and use the first state to determine a plurality of executable actions of the agent and an action value vector corresponding to each executable action;

[0007] Determine a target action from the executable actions using the action value vector, and obtain a first reward obtained by the agent performing the target action in a first state;

[0008] Constructing a current decision data point based on the first state and the target action, and calculating a second reward using the current decision data point and a preset demonstration sample;

[0009] Acquire a second state of the agent after executing the target action, and construct interaction data based on the first state, the target action, the first reward, the second reward, and the second state;

[0010] The target policy network is trained using the interaction data to obtain a trained target policy network.

[0011] In an optional implementation of the present application, the determining of a plurality of executable actions of the agent and an action value vector corresponding to each executable action by using the first state includes:

[0012] Acquire a target policy network, wherein the target policy network includes an action network and a supervision network;

[0013] The first state is input into the action network in the target strategy network to obtain a plurality of executable actions output by the action network and an action value vector corresponding to each of the executable actions.

[0014] In an optional implementation manner of the present application, the calculating the second reward using the current decision data point and a preset demonstration sample includes:

[0015] Setting a first weight and an initial value of weight transfer consumption for the current decision data point;

[0016] Querying a preset demonstration sample for a target data point that matches the current decision data point;

[0017] Obtaining a second weight corresponding to the target data point and a target distance between the current decision data point and the target data point;

[0018] Comparing the first weight with the second weight to obtain a comparison result;

[0019] updating the weight transfer consumption, the first weight, and the second weight according to the comparison result and the target distance until the first weight reaches a preset value, thereby obtaining a target weight transfer consumption;

[0020] A second reward is calculated based on the target weight transfer cost.

[0021] In an optional implementation manner of the present application, the step of searching for a target data point matching the current decision data point in a preset demonstration sample includes:

[0022] Traversing the demonstration data points in the preset demonstration sample;

[0023] Calculating the Euclidean distance between the current decision data point and each of the demonstration data points;

[0024] The demonstration data points are screened based on the Euclidean distance to obtain target data points that match the current decision data points.

[0025] In an optional implementation manner of the present application, updating the weight transfer consumption, the first weight, and the second weight according to the comparison result and the target distance includes:

[0026] If the comparison result is that the first weight is greater than the second weight, then calculating a first product of the second weight and the target distance, updating the weight transfer consumption by using a sum of the initial value and the first product, updating the second weight to a preset value, and updating the first weight by using a difference between the first weight and the second weight;

[0027] Or, if the comparison result is that the first weight is less than or equal to the second weight, then calculate the second product of the first weight and the target distance, and use the sum of the initial value and the second product to update the weight transfer consumption, update the first weight to a preset value, and use the difference between the second weight and the first weight to update the second weight.

[0028] In an optional implementation manner of the present application, the step of training the target policy network using the interaction data to obtain the trained target policy network includes:

[0029] Storing the interaction data in a replay buffer;

[0030] Sampling samples in the replay buffer to obtain a plurality of training samples;

[0031] Using the training samples, the network parameters of the action network in the target strategy network are updated to obtain an updated action network;

[0032] Determining whether the updated motion network meets preset training conditions;

[0033] If the preset training conditions are met, the network parameters of the action network are copied to the supervision network of the target policy network to obtain a trained target policy network.

[0034] In an optional implementation of the present application, the updating of network parameters of the action network in the target strategy network by using the training sample to obtain an updated action network includes:

[0035] Inputting the first state in the training sample into the action network in the target policy network to obtain a prediction item corresponding to the target action;

[0036] Inputting the second state in the training sample into the supervision network in the target strategy network to obtain the supervision item corresponding to the target action;

[0037] Calculate a corresponding loss function based on the prediction item and the supervision item;

[0038] The network parameters of the action network in the target strategy network are updated according to the loss function to obtain an updated action network, wherein the updated action network is used to output multiple executable actions and an action value vector corresponding to each executable action according to the state observed by the agent.

[0039] In a second aspect, an embodiment of the present invention provides a training device for a policy network, the device comprising:

[0040] An acquisition module, used to acquire a first state observed by the agent, and use the first state to determine a plurality of executable actions of the agent and an action value vector corresponding to each executable action;

[0041] A determination module, configured to determine a target action from the executable actions using the action value vector, and obtain a first reward obtained by the agent when executing the target action in a first state;

[0042] A calculation module, configured to construct a current decision data point based on the first state and the target action, and calculate a second reward using the current decision data point and a preset demonstration sample;

[0043] A construction module, configured to obtain a second state of the agent after executing the target action, and construct interaction data based on the first state, the target action, the first reward, the second reward, and the second state;

[0044] The training module is used to train the target policy network using the interaction data to obtain a trained target policy network.

[0045] In a third aspect, an embodiment of the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0046] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method of the first aspect or any corresponding embodiment thereof.

[0047] The method provided in the embodiment of the present application has the following beneficial effects:

[0048] The method provided in the embodiment of the present application determines the executable action and the action value vector by obtaining the first state observed by the intelligent agent, thereby providing a basic basis for the decision-making of the intelligent agent. The target action is determined by using the action value vector and the first reward is obtained. The second reward is calculated in combination with the current decision data point and the preset demonstration sample, taking into account the guidance of the intelligent agent's own exploration and the demonstration sample. The target policy network is trained by constructing interactive data based on multi-faceted information, which effectively solves the problems of overfitting of the environment in reality of reinforcement learning, poor quality of demonstration data leading to invalid policy learning and interference with training, and improves the adaptability and effectiveness of the policy network in different scenarios.

[0049] The method provided in the embodiment of the present application divides the target policy network into an action network and a supervision network, and clarifies the functional division of labor. The action network outputs executable actions and action value vectors based on the first state, provides data for policy learning, and improves the accuracy and efficiency of intelligent agent decision-making. At the same time, weights and initial consumption values ​​are set for the current decision data points, and matching target data points are found in the preset demonstration samples. The consumption and weights are updated according to the weight comparison results and the target distance, and the second reward is calculated based on the target weight transfer consumption. The similarities and differences between the current decision and the demonstration samples are fully considered, and the demonstration sample information is reasonably used to adjust the reward, improve the effect of policy learning, and avoid low-quality demonstration data interfering with training.

[0050] The method provided in the embodiment of the present application traverses the demonstration data points of the preset demonstration samples, and screens the target data points by calculating the Euclidean distance. It is simple and intuitive and can quickly find the points most similar to the current decision data points, providing an effective reference for calculating the second reward. According to the relationship between the first and second weights, different products are calculated to update the weight transfer consumption and weights. Dynamic adjustment of weights can reasonably allocate weights, make reward calculations more accurate, and improve the quality of strategy learning. The interaction data is stored in the replay buffer for sampling, and the action network parameters are updated with training samples. When the conditions are met, it is copied to the supervision network to obtain the trained target strategy network, effectively utilizing historical data, improving training efficiency and stability, and ensuring the overall performance of the strategy network.

[0051] The method provided in the embodiment of the present application is to input the first state in the training sample into the action network to obtain the prediction item, and the second state into the supervision network to obtain the supervision item, and the loss function is calculated based on the two and the action network parameters are updated. This method can accurately evaluate the performance of the action network by comparing the prediction item and the supervision item, and make targeted parameter adjustments to improve the accuracy and reliability of the action network output, thereby improving the performance of the entire strategy network. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 is a flowchart of a training method for a policy network according to some embodiments of the present invention;

[0054] Figure 2 is a schematic diagram of a flow chart of weight transfer consumption calculation according to an embodiment of the present invention;

[0055] Figure 3 is a schematic diagram of a process of interaction between an intelligent agent and an environment according to an embodiment of the present invention;

[0056] Figure 4 is a schematic diagram of a process of training an action network based on interactive data according to an embodiment of the present invention;

[0057] Figure 5 is a flowchart of another policy network training method according to some embodiments of the present invention;

[0058] Figure 6 is a structural block diagram of a training device for a policy network according to an embodiment of the present invention;

[0059] Figure 7 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0061] According to an embodiment of the present invention, a training method, apparatus, computer device and storage medium for a policy network are provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0062] In this embodiment, a training method for a policy network is provided. Figure 1is a flow chart of a training method of a policy network according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0063] Step S11, obtaining a first state observed by the agent, and using the first state to determine a plurality of executable actions of the agent and an action value vector corresponding to each executable action.

[0064] It should be noted that in the field of vehicles, the intelligent agent can be an autonomous vehicle. The intelligent agent is used to realize the autonomous driving function by sensing the surrounding environment, making decisions and performing corresponding control actions. First, the current observed image data is preprocessed, including operations such as cropping, downsampling, erasing the background, converting to grayscale, and then the current first state is generated based on the processed image information. Then, the first state will be input into the decision-making system of the autonomous vehicle (equivalent to the action network in the target policy network). The action network has been pre-trained and can output multiple executable actions based on the input state information, and provide a corresponding action value vector for each executable action.

[0065] In the embodiment of the present application, the first state is used to determine multiple executable actions of the agent and the action value vector corresponding to each executable action, including the following steps A1-A2:

[0066] Step A1, obtaining a target policy network, wherein the target policy network includes an action network and a supervision network.

[0067] Specifically, the target policy network includes an action network (behaviour) and a supervision network (target), both of which have the same structure, consisting of 3 convolutional layers and 2 fully connected layers. The action network gives an action value vector based on the state information observed by the agent, which is used to evaluate the value of executable actions and make decision choices. The supervision network uses the state information as input to calculate the expected return value (i.e., the supervision term) based on the Bellman equation, providing a reference for the action network. The two networks use state information as input and output the value vector of executable actions, providing more accurate guidance for the agent's decision-making and improving the effect of strategy learning.

[0068] Step A2: input the first state into the action network in the target strategy network to obtain multiple executable actions output by the action network and an action value vector corresponding to each executable action.

[0069] Specifically, in the vehicle field, when an autonomous vehicle acts as an intelligent agent, the first state (current environmental state) observed by it is input into the action network in the target strategy network. The action network is pre-trained and can output multiple executable actions, such as acceleration, deceleration, steering, etc., based on the input state information, and provide a corresponding action value vector for each action, that is, the expected benefit evaluation value of executing each action. For example, in a specific first state, considering the position, speed, driving direction of surrounding vehicles, road conditions, traffic signs and signal light status, etc., the action network evaluates the value of each action. The vector composed of these values ​​is the action value vector, which provides a basis for the autonomous vehicle to select the optimal action, which helps to achieve safe and efficient autonomous driving functions.

[0070] Step S12, using the action value vector to determine the target action from the executable actions, and obtain the first reward obtained by the agent when executing the target action in the first state.

[0071] In an embodiment of the present application, the action selection strategy is defined in combination with the idea of ​​maximum entropy, and the target action is determined from the executable actions considering the action value and the importance coefficient of entropy. The target action is the action that is considered to be most likely to bring good results in the current state. After the autonomous driving car executes the target action, it obtains the first reward r from the environment. The first reward r can come from multiple aspects: in terms of safety, if the target action avoids collisions or dangerous situations, a corresponding safety reward will be obtained; in terms of efficiency, if the target action can reach the destination faster or save more energy, a corresponding efficiency reward will be obtained. The first reward is fed back to the decision-making system of the autonomous driving vehicle for subsequent strategy learning and optimization to continuously improve the performance and safety of autonomous driving.

[0072] Specifically, the calculation formula of the action selection strategy is as follows:

[0073]

[0074] Where a represents an executable action; s represents the first state; π(a|s) represents the target action; represents the algorithm for the action selection strategy, where is defined as follows:

[0075]

[0076] Among them, q i represents the action value vector (Q value) corresponding to the i-th action; α represents the importance coefficient of entropy.

[0077] Step S13, constructing a current decision data point based on the first state and the target action, and calculating a second reward using the current decision data point and a preset demonstration sample.

[0078] In the embodiments of the present application, Figure 2 As shown, firstly, the current decision data point (s, a) is constructed based on the first state s and the target action a. Then, the first weight w is set for the current decision data point (s, a) π and weight transfer consumes initial value C 0 Then traverse the data points in the preset demonstration sample and filter out the target data points by Euclidean distance. Get its second weight w D Compare the first and second weights. If the first weight is greater than the second weight, calculate the product of the second weight and the target distance, update the weight transfer consumption, the second weight, and the first weight; if the first weight is less than or equal to the second weight, calculate the product of the first weight and the target distance, update the weight transfer consumption C, the first weight w π and the second weight w D , until the first weight reaches the preset value to obtain the target weight transfer consumption. Finally, the second reward r' is calculated based on the target weight transfer consumption.

[0079] In the embodiment of the present application, the second reward is calculated using the current decision data point and the preset demonstration sample, including the following steps B1-B6:

[0080] Step B1, setting the first weight and the initial value of the weight transfer consumption for the current decision data point.

[0081] Specifically, in the intelligent decision-making process, an initial importance value (first weight) is assigned to the current state of the agent and the corresponding target action (i.e., the current decision data point (s, a)). ), and set an initial value for measuring the degree of match between the decision behavior and the demonstration sample (the initial value C consumed by the weight transfer 0 =0).

[0082] Step B2, querying the preset demonstration samples for target data points that match the current decision data point.

[0083] In the embodiment of the present application, step B2 includes the following steps B21-B23:

[0084] Step B21, traverse the demonstration data points in the preset demonstration sample.

[0085] It should be noted that the preset demonstration sample D consists of a series of ordered demonstration data points Each data point represents a state-action pair. Each data point has a common second weight w D , used to measure its importance when matching and calculating the second reward r′. The data points are numbered in the order in which they are generated for subsequent processing.

[0086] Specifically, in the process of decision-making learning, in order to find the demonstration data points that are most similar to the current decision-making situation, the agent first needs to traverse the preset demonstration samples. The demonstration samples contain a series of state-action pairs generated by experts or excellent strategies, which are instructive for the agent's decision-making. Traversing the demonstration samples means that the agent will check each data point in the sample one by one for subsequent matching and calculation.

[0087] Step B22, calculating the Euclidean distance between the current decision data point and each demonstration data point.

[0088] Specifically, in the process of traversing the demonstration samples, the agent needs to calculate the similarity or distance between the current decision data point and each demonstration data point. Here, Euclidean distance is used as the metric, which is a commonly used distance measurement method used to measure the distance between two points in multidimensional space. By calculating the Euclidean distance, the agent can quantify the difference between the current decision data point and each demonstration data point, thereby providing a basis for subsequent matching.

[0089] Step B23, screening the demonstration data points based on the Euclidean distance to obtain the target data points that match the current decision data points.

[0090] Specifically, after calculating the Euclidean distance between the current decision data point and all demonstration data points, the agent will obtain a distance list or array. Each element in this list or array represents the distance between the current decision data point and a demonstration data point. The agent will filter the demonstration data points according to the size of the distance. Generally, the smaller the distance, the more similar the two data points are, and the agent will select the demonstration data point with the smallest distance as the target data point. Based on the screening criteria, the agent will find a data point from the demonstration sample that best matches the current decision data point. This screened data point is the target data point, which will be used for subsequent decision learning or strategy updates. During the screening process, the agent will also consider the sequence number of the demonstration data point to ensure the consistency of the decision path, and may skip some data points to avoid interfering with subsequent decisions.

[0091] Step B3, obtaining the second weight corresponding to the target data point and the target distance between the current decision data point and the target data point.

[0092] Specifically, the agent determines the target data point that best matches the current decision data point and obtains two key pieces of information: the second weight of the target data point in the demonstration sample (this weight is a preset value shared by all demonstration data points) and the target distance between the current decision data point and the target data point. The target distance quantifies the difference between the current decision data point and the target data point. The smaller the distance, the more similar they are, and the greater the second reward obtained. These two pieces of information will be used for subsequent calculations.

[0093] Step B4, comparing the first weight and the second weight to obtain a comparison result.

[0094] Specifically, the agent needs to compare the weight of the current decision data point (first weight) with the weight of the demonstration data point (second weight). Since the demonstration data points all have the same second weight, the comparison is mainly to determine the weight status of the current decision data point. π =0 or w D =0), it means that the data point has been matched and the agent will stop further processing it. If both weights are not 0, the comparison result will guide the subsequent matching strategy, but in this specific situation, since the second weight is fixed, the comparison mainly serves to determine whether there is any remaining weight of the current decision data point.

[0095] Step B5, updating the weight transfer consumption, the first weight and the second weight according to the comparison result and the target distance, until the first weight reaches a preset value, and obtaining the target weight transfer consumption.

[0096] It should be noted that the demonstration data points all have the same weight (i.e., the second weight). The second weight is set to fit the inherent form of EMD distribution matching, so as to calculate the distribution difference between the agent's behavior trajectory and the demonstration sample, and finally obtain the corresponding auxiliary reward. After each match, the second weight corresponding to the target data point will be reduced accordingly, and the data point with exhausted weight will no longer participate in subsequent matches. The agent must also pay attention to the update of the minimum sequence number to be matched to ensure the consistency of the decision path, and may skip some data points to avoid interference with subsequent decisions (i.e., demonstration sample jumping). Taking into account the distance, weight, and sequence number factors, the agent will select a data point that best matches the current state, has a non-zero weight, and a sequence number that meets the requirements as the current execution target.

[0097] In the embodiment of the present application, updating the weight transfer consumption, the first weight, and the second weight according to the comparison result and the target distance includes the following steps B51-B52:

[0098] Step B51, if the comparison result is that the first weight is greater than the second weight, calculate the first product of the second weight and the target distance, use the sum of the initial value and the first product to update the weight transfer consumption, update the second weight to the preset value, and use the difference between the first weight and the second weight to update the first weight.

[0099] Specifically, if the comparison result is that the first weight is greater than the second weight (i.e., w π >w D ), which is the first weight w of the current decision data point π The second weight W is greater than the demonstration data pointD , the calculation formulas for weight transfer consumption, first weight and second weight are as follows:

[0100]

[0101] W π =w π -w D

[0102] w D =0

[0103] Among them, C represents the weight transfer consumption, C 0 Represents the initial value of weight transfer consumption, w π Represents the first weight of the current decision data point, w D represents the second weight of the target data point, (s, a) represents the current decision data point, represents the target data point, Represents the target distance between the current decision data point and the target data point.

[0104] It should be noted that the update weight transfer consumption indicates that during the matching process, a certain amount of consumption is generated due to the difference and weight distribution between the current decision data point and the target data point. In addition, since the weight of the current decision data point is relatively large, it is believed that the target data point has been fully matched, so the second weight of the target data point is updated to a preset value (usually 0, indicating that the demonstration data point will no longer participate in subsequent matching). The difference between the first weight and the second weight is used to update the first weight, indicating that part of the weight of the current decision data point has been matched with the target data point and is therefore consumed.

[0105] Step B52, if the comparison result is that the first weight is less than or equal to the second weight, then calculate the second product of the first weight and the target distance, and use the sum of the initial value and the second product to update the weight transfer consumption, update the first weight to the preset value, and use the difference between the second weight and the first weight to update the second weight.

[0106] Specifically, if the comparison result is that the first weight is less than or equal to the second weight (i.e., w π ≤w D ), which is the first weight w of the current decision data point π Less than or equal to the second weight w of the demonstration data point D , the calculation formulas for weight transfer consumption, first weight and second weight are as follows:

[0107]

[0108] w π =0

[0109] W D =W D -w π

[0110] Among them, C represents the weight transfer consumption, C 0 Represents the initial value of weight transfer consumption, w π Represents the first weight of the current decision data point, w D represents the second weight of the target data point, (s, a) represents the current decision data point, represents the target data point, Represents the target distance between the current decision data point and the target data point.

[0111] It should be noted that the updated weight transfer consumption indicates that during the matching process, due to the difference between the weight distribution and the data points, a certain amount of additional consumption is generated. In addition, since the weight of the demonstration data point is large, the agent may think that the current decision data point has been fully matched or cannot match more weights, so the first weight of the current decision data point is updated to a preset value (usually 0, indicating that the decision data point will not be matched subsequently). The second weight is updated using the difference between the second weight and the first weight, indicating that part of the weight of the demonstration data point has been matched with the current decision data point and is therefore consumed.

[0112] Step B6, calculating the second reward based on the target weight transfer consumption.

[0113] Specifically, the calculation formula for the second reward is as follows:

[0114]

[0115] Where r′ represents the second reward, f(C) is a decreasing function, and decision data points that are relatively close to the demonstration sample trajectory will receive a larger second reward. β represents the importance of the second reward, C represents the target weight transfer consumption, |s| represents the size of the state dimension, and |a| represents the size of the action dimension.

[0116] Step S14, obtaining the second state of the agent after executing the target action, and constructing interaction data based on the first state, the target action, the first reward, the second reward and the second state.

[0117] In the embodiment of the present application, the agent learns how to take the best action by interacting with the environment. This interaction follows a basic cycle, such as Figure 3As shown, first, the agent observes the current image information, preprocesses the image information, and generates the first state. Secondly, the first state is input into the action network to obtain multiple executable actions and corresponding action value vectors; according to the maximum entropy strategy, the target action is selected in combination with the action value vector; the target action is executed, a first reward is given for the action, and an auxiliary reward, namely the second reward, is calculated. Finally, the agent enters a new second state and generates interaction data. Among them, the interaction data can be a tuple, represented as (s, a, r+r′, s′), where s is the first state; a is the target action taken; r is the first reward given by the environment; r′ is the second reward obtained by matching calculation; s′ is the new second state after executing the action.

[0118] Step S15, using the interaction data to train the target policy network to obtain a trained target policy network.

[0119] In the embodiments of the present application, Figure 4 As shown in the figure, first, some interaction data is selected from the replay buffer as training sample minibatch{s i ,a i ,r i ,s i+1}. Secondly, the first state s in the training sample i Input the action network to obtain the prediction item corresponding to the target action; the second state s in the training sample i+1 Input the supervision network to obtain the supervision item corresponding to the target action. Finally, calculate the loss function based on the prediction item and the supervision item, and update the network parameters of the action network in the target policy network according to the loss function to obtain the updated action network.

[0120] In the embodiment of the present application, step S15 includes the following steps C1-C5:

[0121] Step C1, storing the interaction data into the replay buffer.

[0122] Specifically, storing interaction data allows the agent to reuse it in multiple training cycles, improving data utilization and training efficiency. In reinforcement learning, there is a strong correlation between consecutive data points, which may affect the training process. Storing data in the replay buffer and randomly sampling can break this correlation and make training more stable. The replay buffer also allows the agent to learn from past experience, which helps to learn complex tasks and long-term dependencies.

[0123] Step C2: sampling in the replay buffer to obtain a plurality of training samples.

[0124] Specifically, a sampling technique is used to select part of the data from the replay buffer as training samples, i.e., minibatch{si ,a i ,r i ,s i+1 The purpose of sampling is to break the temporal correlation between data while ensuring data diversity, thereby improving training efficiency and stability. Sampling can be random or based on a certain strategy or rule to ensure that the selected data is representative and diverse.

[0125] Step C3, using the training samples to update the network parameters of the action network in the target strategy network to obtain an updated action network.

[0126] In the embodiment of the present application, step C3 includes the following steps C31-C34:

[0127] Step C31, input the first state in the training sample into the action network in the target strategy network to obtain the prediction item corresponding to the target action.

[0128] Specifically, the training sample minibatch {s i ,a i ,r i ,s i+1 The first state s in} i As input, according to the action network (behaviour) in the target policy network, the prediction item (i.e. predicted action value) of the executed action a is obtained. The specific calculation formula is as follows:

[0129] predict = Q soft (s i ,a;θ)

[0130] Among them, predict represents the prediction item, s i represents the first state, a represents the target action, θ represents the network parameters of the action network in the target policy network, Q soft Represents the predicted item, that is, the predicted action value (predicted Q value).

[0131] Step C32, inputting the second state in the training sample into the supervision network in the target strategy network to obtain the supervision item corresponding to the target action.

[0132] Specifically, the training sample minibatch {s i , a i , r i ,s i+1 The second state s in} i+1 As input, according to the supervision network (target) in the target strategy network, the supervision item (i.e. the actual action value) of the executed action a is obtained. The specific calculation formula is as follows:

[0133]

[0134] Among them, Y i represents the supervision item, s i+1 represents the second state, a represents the target action, θ - represents the network parameters of the supervisory network in the target policy network, γ represents the discount factor of the reward, and Y i The calculation formula is derived from the soft-Bellman equation which introduces the maximum entropy idea.

[0135] Step C33, calculating the corresponding loss function based on the prediction item and the supervision item.

[0136] Specifically, the calculation formula of the loss function is as follows:

[0137]

[0138] in, represents the loss function, Y i represents the supervision item, and predict represents the prediction item.

[0139] Step C34, updating the network parameters of the action network in the target strategy network according to the loss function to obtain an updated action network, wherein the updated action network is used to output multiple executable actions and an action value vector corresponding to each executable action according to the state observed by the agent.

[0140] Specifically, the gradient descent method is used to update the parameters of the action network according to the gradient of the loss function. The core of this method is to adjust the parameters in the opposite direction of the gradient to reduce the loss and improve the prediction accuracy of the action network. The updated action network can more accurately output the value vectors of each action based on the state observed by the agent. These vectors reflect the expected cumulative reward after executing the action. When making decisions, the agent will choose the action with the highest value to execute. Through continuous learning and optimization, the decision-making performance of the agent will gradually improve.

[0141] Step C4, determining whether the updated motion network meets the preset training conditions.

[0142] Specifically, after updating the parameters of the action network using the gradient descent method, it is necessary to evaluate whether the updated network has reached the preset training conditions. These conditions may include the prediction accuracy of the network, the convergence of the loss function, the number of training rounds reaching the preset value, etc. By judging these conditions, it is decided whether to continue network training or terminate the training and proceed to the next step.

[0143] Step C5: If the preset training conditions are met, the network parameters of the action network are copied to the supervision network of the target policy network to obtain the trained target policy network.

[0144] Specifically, when the action network meets the preset training conditions, it is considered that the network has reached a good performance level. At this time, the parameters of the action network are copied to the supervision network of the target policy network. The purpose of this step is to pass the knowledge and experience learned by the action network to the target policy network, so that the target policy network can also have better decision-making ability. By copying the parameters, a trained target policy network is obtained, which can be used to guide the decision-making of the intelligent agent in subsequent tasks.

[0145] As an example, the decision-making process of the Smart Driving-Obstacle Avoidance (SD-OA) system is as follows: the intelligent driving system observes the road images captured by the latest 3 frames of the on-board camera and obtains the corresponding image information; the image information data is preprocessed, where the preprocessing process mainly includes image cropping to focus on key areas, downsampling to reduce computational complexity, removing irrelevant background interference, and converting to grayscale to simplify processing, and the preprocessed image is regarded as the current road state; the current road state is input into the trained driving model to obtain the action value vectors corresponding to all executable driving actions (such as steering, acceleration, deceleration, etc.), which reflect the expected effect of safe driving of the vehicle under different actions; the intelligent driving system selects and executes the driving action with the largest action value vector, controls the vehicle to perform corresponding operations to avoid obstacles, and observes new road image information; repeats the above decision-making process until the vehicle arrives safely at the destination and the intelligent decision is completed.

[0146] Figure 5 is a flowchart of another policy network training method according to an embodiment of the present invention. Figure 5 As shown, the process includes: initializing the target strategy network and network parameters; obtaining the first state observed by the intelligent agent, using the first state to input the action network, and determining multiple executable actions and corresponding action value vectors; using the action value vector to determine the target action from the executable actions, executing the action and obtaining the first reward, and obtaining the second state after execution; constructing the current decision data point, and calculating the second reward using the preset demonstration sample; constructing the first state, target action, first reward, second reward and second state as interaction data, and storing them in the replay buffer; sampling training data from the replay buffer; using the training samples to update the network parameters of the action network and calculate the loss function; updating the parameters of the action network according to the loss function; judging whether the updated action network meets the preset training conditions, if so, synchronizing the parameters of the action network to the supervision network, if not, continuing the iterative training.

[0147] In this embodiment, a training device for a policy network is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0148] This embodiment provides a training device for a policy network, such as Figure 6 As shown, including:

[0149] An acquisition module 61 is used to acquire a first state observed by the agent, and determine multiple executable actions of the agent and an action value vector corresponding to each executable action using the first state;

[0150] A determination module 62, configured to determine a target action from executable actions using the action value vector, and obtain a first reward obtained by the agent when executing the target action in the first state;

[0151] A calculation module 63, configured to construct a current decision data point based on the first state and the target action, and calculate a second reward using the current decision data point and a preset demonstration sample;

[0152] A construction module 64 is used to obtain a second state of the agent after performing the target action, and construct interaction data based on the first state, the target action, the first reward, the second reward and the second state;

[0153] The training module 65 is used to train the target policy network using the interaction data to obtain a trained target policy network.

[0154] In an optional embodiment of the present application, an acquisition module 61 is used to obtain a target policy network, wherein the target policy network includes an action network and a supervision network; the first state is input into the action network in the target policy network to obtain multiple executable actions output by the action network and an action value vector corresponding to each executable action.

[0155] In an optional embodiment of the present application, the calculation module 63 is used to set the first weight and the initial value of the weight transfer consumption for the current decision data point; query the target data point matching the current decision data point in the preset demonstration sample; obtain the second weight corresponding to the target data point and the target distance between the current decision data point and the target data point; compare the first weight with the second weight to obtain a comparison result; update the weight transfer consumption, the first weight and the second weight according to the comparison result and the target distance until the first weight reaches the preset value, and the target weight transfer consumption is obtained; calculate the second reward based on the target weight transfer consumption.

[0156] In an optional embodiment of the present application, the calculation module 63 is used to traverse the demonstration data points in the preset demonstration sample; calculate the Euclidean distance between the current decision data point and each demonstration data point; and filter the demonstration data points based on the Euclidean distance to obtain the target data point that matches the current decision data point.

[0157] In an optional embodiment of the present application, the calculation module 63 is used to calculate the first product of the second weight and the target distance if the comparison result is that the first weight is greater than the second weight, update the weight transfer consumption using the sum of the initial value and the first product, update the second weight to a preset value, and update the first weight using the difference between the first weight and the second weight; or, if the comparison result is that the first weight is less than or equal to the second weight, calculate the second product of the first weight and the target distance, update the weight transfer consumption using the sum of the initial value and the second product, update the first weight to a preset value, and update the second weight using the difference between the second weight and the first weight.

[0158] In an optional implementation of the present application, the training module 65 is used to store the interaction data in a replay buffer; perform sample sampling in the replay buffer to obtain multiple training samples; use the training samples to update the network parameters of the action network in the target policy network to obtain an updated action network; determine whether the updated action network meets the preset training conditions; if the preset training conditions are met, copy the network parameters of the action network to the supervision network of the target policy network to obtain a trained target policy network.

[0159] In an optional embodiment of the present application, the training module 65 is used to input the first state in the training sample into the action network in the target policy network to obtain a prediction item corresponding to the target action; input the second state in the training sample into the supervision network in the target policy network to obtain a supervision item corresponding to the target action; calculate the loss function based on the prediction item and the supervision item; update the network parameters of the action network in the target policy network according to the loss function to obtain an updated action network, wherein the updated action network is used to output multiple executable actions and an action value vector corresponding to each executable action according to the state observed by the agent.

[0160] See also Figure 7 , Figure 7 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 7As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system).

[0161] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0162] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0163] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the use of a computer device based on the presentation of a small program landing page, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0164] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0165] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0166] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0167] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A training method for a policy network, characterized in that: The method comprises: Acquire a first state observed by the agent, and use the first state to determine a plurality of executable actions of the agent and an action value vector corresponding to each executable action; Determine a target action from the executable actions using the action value vector, and obtain a first reward obtained by the agent performing the target action in a first state; Constructing a current decision data point based on the first state and the target action, and calculating a second reward using the current decision data point and a preset demonstration sample; Acquire a second state of the agent after executing the target action, and construct interaction data based on the first state, the target action, the first reward, the second reward, and the second state; The target policy network is trained using the interaction data to obtain a trained target policy network.

2. The method according to claim 1, characterized in that: The method of using the first state to determine a plurality of executable actions of the agent and an action value vector corresponding to each executable action includes: Acquire a target policy network, wherein the target policy network includes an action network and a supervision network; The first state is input into the action network in the target strategy network to obtain a plurality of executable actions output by the action network and an action value vector corresponding to each of the executable actions.

3. The method according to claim 1, characterized in that The calculating the second reward by using the current decision data point and the preset demonstration sample includes: Setting a first weight and an initial value of weight transfer consumption for the current decision data point; Querying a preset demonstration sample for a target data point that matches the current decision data point; Obtaining a second weight corresponding to the target data point and a target distance between the current decision data point and the target data point; Comparing the first weight with the second weight to obtain a comparison result; updating the weight transfer consumption, the first weight, and the second weight according to the comparison result and the target distance until the first weight reaches a preset value, thereby obtaining a target weight transfer consumption; A second reward is calculated based on the target weight transfer cost.

4. The method according to claim 3, characterized in that: The step of searching the preset demonstration sample for a target data point that matches the current decision data point includes: Traversing the demonstration data points in the preset demonstration sample; Calculating the Euclidean distance between the current decision data point and each of the demonstration data points; The demonstration data points are screened based on the Euclidean distance to obtain target data points that match the current decision data points.

5. The method according to claim 3, characterized in that: The updating of the weight transfer consumption, the first weight, and the second weight according to the comparison result and the target distance includes: If the comparison result is that the first weight is greater than the second weight, then calculating a first product of the second weight and the target distance, updating the weight transfer consumption by using a sum of the initial value and the first product, updating the second weight to a preset value, and updating the first weight by using a difference between the first weight and the second weight; Or, if the comparison result is that the first weight is less than or equal to the second weight, then calculate the second product of the first weight and the target distance, and use the sum of the initial value and the second product to update the weight transfer consumption, update the first weight to a preset value, and use the difference between the second weight and the first weight to update the second weight.

6. The method according to claim 1, characterized in that The step of training the target policy network using the interaction data to obtain the trained target policy network includes: Storing the interaction data in a replay buffer; Sampling samples in the replay buffer to obtain a plurality of training samples; Using the training samples, the network parameters of the action network in the target strategy network are updated to obtain an updated action network; Determining whether the updated motion network meets preset training conditions; If the preset training conditions are met, the network parameters of the action network are copied to the supervision network of the target policy network to obtain a trained target policy network.

7. The method according to claim 6, characterized in that The updating of the network parameters of the action network in the target strategy network by using the training sample to obtain an updated action network includes: Inputting the first state in the training sample into the action network in the target policy network to obtain a prediction item corresponding to the target action; Inputting the second state in the training sample into the supervision network in the target strategy network to obtain the supervision item corresponding to the target action; Calculate a corresponding loss function based on the prediction item and the supervision item; The network parameters of the action network in the target strategy network are updated according to the loss function to obtain an updated action network, wherein the updated action network is used to output multiple executable actions and an action value vector corresponding to each executable action according to the state observed by the agent.

8. A training device for a policy network, characterized in that: The device comprises: An acquisition module, used to acquire a first state observed by the agent, and use the first state to determine a plurality of executable actions of the agent and an action value vector corresponding to each executable action; A determination module, configured to determine a target action from the executable actions using the action value vector, and obtain a first reward obtained by the agent when executing the target action in a first state; A calculation module, configured to construct a current decision data point based on the first state and the target action, and calculate a second reward using the current decision data point and a preset demonstration sample; A construction module, configured to obtain a second state of the agent after executing the target action, and construct interaction data based on the first state, the target action, the first reward, the second reward, and the second state; The training module is used to train the target policy network using the interaction data to obtain a trained target policy network.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 7.

Images