An Imitation Learning Method Based on Decision Confidence under Non-Optimal Demonstrations

Abstract

The present invention relates to an imitation learning method based on decision confidence under non-optimal demonstration, which belongs to the field of autonomous driving reinforcement learning technology. c and unlabeled example dataset D u Sampling is performed respectively, and classifier g is estimated; using the D c The confidence score of the classifier g is used to form a regular term, and the loss function L(g) of the classifier g is modified. The classifier g is optimized by gradient descent. The D u The confidence scores of the state-action pairs are obtained, and the data with the labeled confidence scores are merged to obtain the merged D′ c ; D′ c The confidence scores of the state-action pairs in [ ] are converted into weights φ for the discriminator and policy networks. This is combined with a generative adversarial imitation learning method to learn the agent's expert policy. This method can learn a classifier that assigns confidence scores to state-action pairs, assign appropriate weights to datasets obtained from non-optimal expert demonstrations for imitation learning, and ultimately produce a policy network for the agent's autonomous driving task.

Description

Technical Field

[0001] The present invention relates to an imitation learning method, and in particular to an imitation learning method based on decision confidence under non-optimal demonstration, belonging to the technical field of autonomous driving reinforcement learning. Background Art

[0002] With the rapid development of artificial intelligence, machine learning has found applications in all areas of society. Reinforcement learning, a learning method within machine learning, has achieved remarkable success in recent years. During reinforcement learning training, feedback from the environment and corresponding specific feedback values are required. Reinforcement learning primarily guides the training subject in making decisions at each step and determining what actions to take to achieve a specific goal or maximize benefits. Reinforcement learning consists of the following five components:

[0003] 1. Agent: The subject of reinforcement learning training.

[0004] 2. Environment: The general background in which the intelligent agent operates.

[0005] 3. State: This refers to the current state of the environment and the agent. Since both the agent and the environment are constantly changing, the state is also changing.

[0006] 4. Action: Based on the current state, the actions that the agent can take are actions.

[0007] 5. Reward: When the agent takes a specific action in the current state, it will receive certain feedback from the environment.

[0008] Reinforcement learning has made significant progress in industry and gaming. In real-world environments, reinforcement learning has numerous training methods. In autonomous driving tasks, to ensure that intelligent agents' driving performance more closely resembles human driving style, a common approach to training agent models is to have them learn from expert demonstrations. This approach effectively replicates the experience of experts manually designing, modeling, and compiling applications for the agent to handle the task. This approach is known as learning from demonstration. Imitation learning is an existing learning from demonstration approach that directly learns a predictive model—a mapping from states to desired actions. However, imitation learning in autonomous driving tasks requires high-quality expert human decision-making demonstrations. For example, in complex urban scenarios, intelligent agents should drive in a straight line at a consistent speed and respond quickly to changing traffic lights. However, collecting optimal demonstrations in these scenarios is often costly, and the quality of the collected demonstration data cannot be guaranteed due to the varying driving styles of different human drivers. In real-world tasks, it is often impossible to determine the optimality of human-demonstrated driving decisions. For example, considering the vehicle avoidance scenario, the collected driving observation scenarios and their corresponding avoidance actions may be the safest decision for vehicle control, but this action may cause discomfort to passengers. Therefore, it is often difficult to judge the optimality of driving decisions.

[0009] Current imitation learning research often relies heavily on high-quality expert examples. However, for a given driving scenario decision data, there is often no prior information on its quality. Therefore, previous methods have often been forced to consider all driving decisions as part of the optimal decision, or to manually select a small number of labeled, high-confidence expert driving decisions through further screening and labeling. However, such methods can either result in the agent's trained policy being far inferior to the optimal policy, or require significant manpower and resources to complete the confidence labeling of driving scene image observations and control actions, making them difficult to implement in practice. Summary of the Invention

[0010] In response to some or all of the above-mentioned deficiencies in the prior art, the present invention aims to address the technical problem that existing solutions for imitation learning training of autonomous driving tasks by intelligent agents have excessive requirements for optimal expert demonstrations and insufficient robustness. The present invention provides an imitation learning method based on decision confidence under non-optimal demonstrations, including:

[0011] From the labeled example dataset D c and unlabeled example dataset D u Sampling is performed respectively, and the classifier g is estimated. The D c Each state-action pair in is labeled with a confidence score, which is the degree of similarity between the state-action pair and the optimal expert demonstration;

[0012] Using the D c The confidence score of the classifier g constitutes a regularization term, and the accuracy of the classifier prediction is regulated by comparing the error between the prediction score of the classifier g and the confidence score, and the loss function L(g) of the classifier g is corrected. The classifier g is parameterized by a neural network, and the classifier g is optimized by gradient descent according to the corrected loss function L(g) of the classifier g. Among them, α is the proportion of the state-action pairs of the optimal strategy in all state-action pairs;

[0013] Calculate the D using the optimized classifier g u The confidence score of the state action pair in the dataset is obtained, and the data with the labeled confidence score is compared with D c Merge to get the merged data set D′ c ;

[0014] The combined D′ c The confidence scores of the state-action pairs in are converted into the weights φ of the discriminator and the policy network. Combined with the imitation learning method based on generative adversarial learning, the target formulas of the discriminator and the policy network are generated to learn the agent expert strategy.

[0015] The present invention also provides a device for applying the above-mentioned imitation learning method based on decision confidence under non-optimal demonstration, comprising:

[0016] A data interaction module collects and preprocesses image data of the driving environment, and obtains environmental feedback information through the interaction between the intelligent agent and the environment;

[0017] The agent deployment module uses the labeled example dataset D c and unlabeled example dataset D u Construct the classifier g and optimize it, expand the number of samples that can be used, and use the classifier g to u After the confidence scores are performed, the policy network parameters are updated using the idea of adversarial imitation learning. The confidence scores of the expert demonstrations are used to adaptively select good demonstrations for training, and the policy network of the agent is constructed. The image data is used as input to obtain the decision output of the agent.

[0018] The testing module is used to test the policy network, debug the parameters of the policy network, construct a reward function using the current discriminator output, and match the reward function with the simulated physical environment in the driving scenario to observe the feasibility of decision-making in the traffic environment.

[0019] The present invention further provides an electronic device, comprising:

[0020] at least one processor; and,

[0021] A memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the aforementioned automatic driving control method.

[0022] The present invention also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the aforementioned automatic driving control method.

[0023] Beneficial effects:

[0024] This paper proposes an imitation learning method based on decision confidence under non-optimal demonstrations. It designs a method for imitation learning in non-optimal expert demonstrations. After learning a classifier that assigns confidence scores to state-action pairs, it then assigns appropriate weights to the dataset obtained from the non-optimal expert demonstrations for imitation learning. This ultimately yields a policy network that can be used in autonomous driving tasks. This policy network can control the steering, acceleration, and braking of an autonomous vehicle via a PID controller. This paper effectively optimizes the training process for imitation learning in autonomous driving tasks, achieving good performance, robustness, and interpretability. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 Schematic diagram of the overall framework of the model of the embodiment of the present invention;

[0026] Figure 2 This is a schematic diagram of an embodiment of the present invention;

[0027] Figure 3 This is an example diagram of an intelligent agent autonomous driving task simulation environment used in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0029] First, we introduce the variables and their mathematical representations needed in autonomous driving tasks:

[0030] In reinforcement learning for autonomous driving tasks, the agent is in a Markov decision process framework M, where the elements are represented by a 5-tuple M = {S, A, P, γ, r}, where S represents the parsed finite state space captured by the multi-camera system, A represents the agent's action space within the autonomous driving framework, primarily consisting of three control operations: steering, throttle, and braking, P represents the state transition matrix, and γ represents the decay factor. Considering the problem of imitation learning from imperfect expert demonstrations, this paper uses r: S × A → R to represent the immediate feedback (reward function) that the autonomous driving agent receives for taking an action in a given state. Furthermore, this paper uses p(s, a) to represent the frequency of visits to state-action pairs experienced by the agent under policy π, where (s, a) represents a state-action pair, s represents the state, and a represents the action.

[0031] In the imperfect expert demonstration imitation learning problem, it is assumed that the imperfect expert demonstration in the autonomous driving task is composed of the optimal policy and non-optimal strategies The sampled state-action pairs are composed of the corresponding state-action pair access frequencies under the optimal strategy and non-optimal strategy respectively. opt and p non , then the access frequency p(s, a) of the state-action pair in the demonstration set in the autonomous driving task is equal to the weighted sum of the access frequency under the optimal strategy and the access frequency under the non-optimal strategy:

[0032] p(s, a) = αp opt (s, a)+(1-α)p non (s, a),

[0033] Where α is the proportion of state-action pairs demonstrated by the best expert to all state-action pairs. To measure the optimality of the expert demonstration data, a confidence score c∈[0,1] is defined. The closer it is to 1, the more likely the state-action pair is from the best expert strategy, and the closer it is to 0, the worse the state-action pair is. Consider the expert demonstration dataset: They represent the labeled example dataset and the unlabeled example dataset, respectively, where the labeled example dataset D c For each state-action pair (S i , a i ) has its corresponding confidence score c i The goal of learning is to use a small amount of labeled example data set D c With a large number of unlabeled example datasets D u Learn a good expert policy. In addition, define a classifier g with parameters to evaluate the confidence score of the state-action pair and define the discriminator network D ω Used to identify whether the given data is expert data and define the strategy network πθ Used to get the agent's decision action in a given state.

[0034] Based on the existing definition, in order to achieve the purpose of imitation learning in the above-mentioned autonomous driving task, the technical solution adopted by the present invention is as follows:

[0035] From the labeled example dataset D c and unlabeled example dataset D u Sampling is performed respectively, and the classifier g is estimated. The D c Each state-action pair in is labeled with a confidence score, which is the degree of similarity between the state-action pair and the optimal expert demonstration;

[0036] Using the D c The confidence score of the classifier g constitutes a regularization term, and the accuracy of the classifier prediction is standardized by comparing the error between the prediction score of the classifier g and the confidence score, and the loss function L(g) of the classifier g is corrected. The classifier g is parameterized by a neural network, and the classifier g is optimized by gradient descent according to the corrected loss function L(g) of the classifier g. Among them, α is the proportion of the state-action pairs of the optimal strategy in all state-action pairs;

[0037] Calculate the D using the optimized classifier g u The confidence score of the state action pair in the dataset is obtained, and the data with the labeled confidence score is compared with D c Merge to get the merged data set D′ c ;

[0038] The combined D′ c The confidence scores of the state-action pairs in are converted into the weights φ of the discriminator and the policy network. Combined with the imitation learning method based on generative adversarial learning, the target formulas of the discriminator and the policy network are generated to learn the agent expert strategy.

[0039] In the embodiment of the present invention, the classifier g is composed of a multi-layer neural network, each layer of the neural network is activated by a nonlinear activation function ReLU, and its input is a state-action pair (s i , a i ), the output is the confidence score c i The network structure of the discriminator is a multi-layer neural network, each layer of the neural network is activated by a nonlinear activation function ReLU, and its input is a state-action pair (s i , a i), the output is the probability that the state action pair is the expert data. The policy network is also a multi-layer neural network. Different from the classifier g and the discriminator, the last layer of the neural network is nonlinearly activated by the tanh activation function, which converts the state s i As input, the output is the specific action corresponding to the state.

[0040] The method is used in the imitation learning training of autonomous driving tasks to label example data D based on small samples. c A classifier h is constructed to evaluate the confidence scores of a large number of expert human driving decision demonstrations, expanding the number of available samples and allowing the agent to train on a wider range of driving scenario decision examples. After using classifier g to assign confidence scores to driving scenario demonstrations, adversarial imitation learning based on the Wasserstein distance is employed. Finally, reinforcement learning based on the state representation distance is introduced to update the parameters of the policy network. The confidence scores of expert demonstrations are used to adaptively select good demonstrations for training, while discarding suboptimal demonstrations for the autonomous driving task (e.g., driving that causes passenger dizziness, sudden stops, sharp turns, and other dangerous driving). This allows the agent to better imitate the well-performing expert demonstrations.

[0041] The technical solution of the embodiment of the present invention learns decision confidence from an imperfect expert demonstration dataset, and learns a good strategy by weighted joint training of the discriminator and the policy network. Figure 1 The overall framework of the model is presented. The model first learns a classifier g to improve the utilization of expert demonstration samples for autonomous driving tasks. This classifier g automatically annotates confidence scores for unlabeled example datasets and converts these confidence scores into weights for the discriminator and policy networks through a nonlinear function. This classifier then learns the agent's policy using a generative adversarial imitation learning method.

[0042] Figure 3 This is an example diagram of the intelligent autonomous driving task simulation environment used in the present invention. The specific implementation process is as follows: Figure 2 As shown, in a preferred embodiment of the present invention, the following steps are included:

[0043] Step 1: Train the classifier g to automatically label the confidence score for the state-action pair, and define the degree of similarity between the state-action pair and the optimal expert demonstration. Considering that the state-action pairs in the driving strategy demonstrations collected in reality are often collected from the behavioral trajectories of different human users, the existing technology often manually labels all strategy demonstrations and divides the demonstration data into optimal demonstrations and non-optimal demonstrations for training. However, considering the labeling cost, the application of Bayesian theorem in the embodiment of the present invention can change the data sampling process from sampling on optimal demonstrations and non-optimal demonstrations to sampling from labeled and unlabeled data distributions, that is, by u and the Dc Sampling is performed on the classifier g to estimate the classifier g. Based on this, the loss function L(g) of the classifier g is defined as follows:

[0044]

[0045] The loss function is mainly divided into two parts, c i is the number of labeled datasets (s i , a i ) confidence score, the first item is the D c Sampling is performed on the D u Upsampling, β∈[0,1] is a hyperparameter in the loss function, which is used to balance the weight coefficient between different loss terms. The l function is the logistic loss function l(x)=log(1+exp(-x)).

[0046] In order to increase the accuracy of model prediction, the D c The confidence score of constitutes a regularization term, which regulates the accuracy of the classifier prediction by comparing the error between the prediction score of classifier g and the original confidence score. The loss function of the corrected classifier g is as follows:

[0047]

[0048] The last term in this formula is the regularization term, which means (s i , a i ) The final score of the data pair obtained by the classifier g and its true confidence score label c i The square loss, where λ is the weight coefficient of the regularization term, is a hyperparameter. The classifier g is parameterized by a neural network, and the classifier g is optimized by gradient descent according to the modified loss function L(g):

[0049] Step 2: Obtain the optimized classifier g through the above gradient descent training, and use the optimized classifier g to calculate the D u Medium (s i , a i ) corresponding to the confidence score c i =g(s i , a i ), and merge all the labeled data into D c Merge to form the merged annotation dataset D′ c .

[0050] Step 3: Introduce a weight φ(s, a) for each state-action pair in the dataset, which represents the probability that the demonstration is the optimal demonstration. The weight φ(s, a) is calculated using the confidence score c, and its calculation formula is: Where η is a hyperparameter, represents the expected return of the state-action pair demonstrated by the current expert, N(s, a) is the number of state-action pairs demonstrated by the current expert, and r(s, a) is the reward for the state-action pair (s, a).

[0051] Step 4: Since the merged labeled dataset D′ c All state-action pairs in have corresponding confidence scores and weight information. The weight φ(s, a) of the obtained state-action pairs is introduced into the imitation learning problem, and the target formula is defined as:

[0052]

[0053] Where ω is the discriminator D ω Parameters of the policy network π θ Parameters, is the merged driving demonstration dataset D′ with confidence score labels c The access frequency of the state-action pair on represents the access frequency of state-action pairs under the parameterized policy network, ||·||2 represents the two-norm, represents the gradient and E represents the expectation.

[0054] In the target formula, the first term is the c After sampling (s, a) pairs, the discriminator network D with φ weights is used. ω The result obtained is that the second term is obtained through the strategy network π θ The samples are sampled and passed through the discriminator network D ω The result obtained is that the third term is the gradient penalty term, which means that first from the D′ c Sample a batch of data (such as n pieces of data) and simultaneously extract data from the policy network π θ Upsample a batch of data of the same size (n pieces of data), mix the two, and then randomly extract data of the same batch size (still n pieces). After this sampling process, the sampled samples are passed to the discriminator network D ω , calculate the (s, a) pair through the discriminator network D ω The obtained result is the two norm of the derivative, so that the sample (s, a) satisfies the discriminator D ω The Lipsitz constraint of (i.e., the Lipsitz value of the discriminator network should be as close to 1 as possible), is a hyperparameter used to balance the weight of the gradient penalty. During each training, the model calculates the result according to the target formula, and automatically derives the discriminator parameter ω and the policy network parameter θ according to the target formula, and updates the discriminator D through back propagation. ω and policy network πθ .

[0055] Step 5: Basis Figure 2 , it can be concluded that the entire training process is as follows: first, initialize the expert trajectory, network parameters and discriminator parameters, and set the three hyperparameters related to the embodiment of the present invention; after setting the hyperparameters and data set, start the training step. Since the embodiment of the present invention involves the training of confidence scores, the first step should consider using the existing labeled data set D c Supervised training is performed on the classifier g, and the loss function of the classifier g is L(g); after the classifier is trained to its maximum number of training steps, the classifier g is used to calculate the loss function of the entire unlabeled example dataset D u The confidence score, with the D c Merge to obtain the merged data set D′ c ; Calculate the corresponding weight coefficient φ based on the confidence score; Set the maximum number of interaction rounds between the agent and the environment, and set the current interaction to start from the first round. Each time, use the initialization strategy to generate the corresponding strategy network trajectory, and then update the iterative parameters of the discriminator network and the strategy network according to the target formula in the above step 4. When the preset maximum number of interaction rounds is finally reached, the training process is completed, the training is stopped, the model parameters are saved, and the imitation of the expert strategy is completed.

[0056] The construction process of the simulation environment in the embodiment of the present invention is as follows:

[0057] Step 1: Determine the vehicle kinematic model. Specifically, use the dynamic equations of vehicle movement and collision to model the vehicle's operation process and obtain its kinematic model. This simulation environment considers the test scenario as an urban traffic environment under complex road conditions.

[0058] Step 2: The reward function is defined as a formula related to speed and collision degree, which is:

[0059]

[0060] Among them, v ego To control the vehicle's velocity vector, map it to the city unit vector The speed of the agent in the city is evaluated by multiplying the discretized time Δt=0.05. The impulse, or collision index, is used to measure the degree of collision in Newton-seconds. i is the coefficient corresponding to the collision index, which is set to 0.1 in the embodiment of the present invention.

[0061] Step 3: Construct an autonomous driving simulation environment based on the vehicle kinematic model and the reward function. During testing, the agent is placed in the autonomous driving engine platform for policy evaluation. The agent's observation images in the driving engine are input into the deep imitation learning framework, and the agent's decision actions are obtained as the agent's output. The agent interacts with the decision environment according to the aforementioned steps. The agent's effectiveness is determined based on the final reward function value, and the agent's driving ability and performance are evaluated.

[0062] In a preferred embodiment of the present invention, an application deployment scenario under the autonomous driving engine is first constructed. It mainly includes three modules:

[0063] Data interaction modules, such as Figure 3 As shown in the figure, three sensors are deployed in the autonomous vehicle. Each sensor has a 60-degree horizontal field of view and can observe image data from the front, left front, and right front. In addition, the data interaction module preprocesses the input image into an image tuple consisting of color images and obtains environmental feedback information through the interaction between the intelligent agent and the environment.

[0064] Agent deployment module, using labeled example data D c and unlabeled example dataset D u Construct the classifier g and optimize it, expand the number of samples that can be used, and use the classifier g to u After the confidence score is assigned, the policy network parameters are updated using adversarial imitation learning. The confidence scores of the expert demonstrations are used to adaptively select good demonstrations for training. This constructs the policy network of the intelligent agent, which uses the image data as input to obtain the decision output of the intelligent agent, thereby mimicking the driving decisions of human experts and achieving autonomous driving.

[0065] Testing module, for the agent's policy network π θ The test fixes all the parameters of the entire agent network and conducts autonomous driving simulation deployment in complex urban traffic scenarios in the Carla engine platform. ω Output construction reward function logD ω (s, a), and correspond it to the simulated physical environment in the driving scenario to observe the feasibility of its decision-making in complex urban traffic.

[0066] This specification provides a specific embodiment to illustrate the present invention and its implementation. The details included in this embodiment are not intended to limit the scope of the claims, but rather to facilitate understanding of the present invention. Those skilled in the art will appreciate that various modifications, variations, and substitutions of the steps of the preferred embodiment are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the disclosure of the preferred embodiment and the accompanying drawings.

Claims

1. An imitation learning method based on decision confidence under non-optimal demonstration, characterized in that include: From the labeled example dataset and unlabeled example datasets Sampling is performed separately and classifier is performed It is estimated that Each state-action pair in is labeled with a confidence score, which is the degree of similarity between the state-action pair and the optimal expert demonstration; Using the The confidence score of The error between the prediction score and the confidence score is used to standardize the accuracy of the classifier prediction and correct the classifier The loss function , the classifier is trained by neural network Parameterize and use the modified classifier The loss function For the classifier Perform gradient descent optimization for ,in, is the proportion of state-action pairs of the optimal strategy in all state-action pairs; Using the optimized classifier Calculate the The confidence score corresponding to the state action pair in the data is merged into the data after the confidence score is marked. , forming a merged annotation dataset ; The The confidence scores of the state-action pairs in are converted into weights of the discriminator and the policy network , combined with the imitation learning method based on generative adversarial, the target formula of the discriminator and the policy network is generated to learn the agent expert strategy.

2. The imitation learning method based on decision confidence under non-optimal demonstration according to claim 1, characterized in that The classifier The loss function Expressed as: ； in, For the The loss obtained by sampling on For the The loss obtained by upsampling, For the in The confidence score of is a hyperparameter in the loss function, which is used to balance the weight coefficient between different loss terms. Represents state-action pairs The confidence score of The function is the logistic loss function: 。 3. The imitation learning method based on decision confidence under non-optimal demonstration as claimed in claim 2, characterized in that The modified loss function Expressed as: ； in, For the The loss obtained by sampling on For the The loss obtained by upsampling, Represents state-action pairs The confidence score of The regular term obtained by sampling on Data pairs are passed through the classifier The final score obtained and its true confidence score label The squared loss, where is the weight coefficient of the regularization term, which is a hyperparameter.

4. The imitation learning method based on decision confidence under non-optimal demonstration according to claim 1, characterized in that The weight Expressed as: ； in is a hyperparameter, represents the expected return of the state-action pair demonstrated by the current expert, is a state-action pair, s represents the state, a represents the action, For the For the number, For the For the corresponding confidence score, State-action pair rewards.

5. The imitation learning method based on decision confidence under non-optimal demonstration according to claim 1, characterized in that: The objective formulas of the discriminator and policy network are: ； in, , For the The frequency of access of state-action pairs in the dataset, Represents the access frequency of state-action pairs under the parameterized strategy network. In the target formula, For the Sampling on After the Weighted Discriminator The result obtained is, To pass the policy network The sample is taken and passed through the discriminator The result obtained is, is the gradient penalty term, indicating that And the policy network Let's do it together The right sample, and let the sample For the discriminator The Lipsitz constraint, is a hyperparameter used to balance the weight of the gradient penalty, Represents state-action pairs The confidence score of Represents the gradient.

6. A device using the imitation learning method based on decision confidence under non-optimal demonstration as claimed in claim 1, characterized in that: include: A data interaction module collects and preprocesses image data of the driving environment, and obtains environmental feedback information through the interaction between the intelligent agent and the environment; Agent deployment module, using labeled example data and unlabeled example datasets Constructing a classifier And optimize, expand the number of samples that can be used, use the classifier Give the After the confidence score is performed, the data with the marked confidence score is merged into the , forming a merged annotation dataset , the The confidence scores of the state-action pairs in are converted into weights of the discriminator and the policy network , using the idea of adversarial imitation learning, updating the parameters of the policy network, using the confidence scores of expert demonstrations to adaptively select good demonstrations for training, constructing the policy network of the agent, taking the image data as input, and obtaining the decision output of the agent; The testing module is used to test the policy network, debug the parameters of the policy network, construct a reward function using the current discriminator output, and match the reward function with the simulated physical environment in the driving scenario to observe the feasibility of decision-making in the traffic environment.

7. The device according to claim 6, characterized in that In the agent deployment module, the classifier The loss function Expressed as: ； in, For the The loss obtained by sampling on For the The loss obtained by upsampling, For the in The confidence score of is a hyperparameter in the loss function, which is used to balance the weight coefficient between different loss terms. Represents state-action pairs The confidence score of The function is the logistic loss function .

8. The device according to claim 6, wherein In the agent deployment module, the weight Expressed as: ； in is a hyperparameter, represents the expected return of the state-action pair demonstrated by the current expert, is a state-action pair, s represents the state, a represents the action, For the For the number, For the For the corresponding confidence score, State-action pair rewards.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, A memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor so as to enable the at least one processor to execute the method of any one of claims 1 to 5.

10. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method according to any one of the preceding claims 1 to 5.

Images