Imitation learning mixed sample processing method and device based on visual pre-training model
By using visual pre-training models in adversarial generative imitation learning, adding noise to suboptimal expert samples and optimizing sample sets, the problems of lack of high-quality data and poor generalization ability of high-dimensional state space are solved, and the imitation learning intelligent body is improved generalization ability.
Patent Information
- Application Number
- CN202311868899.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-29
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2043-12-29
AI Technical Summary
The existing adversarial generative imitation learning algorithm relies on high-quality data. When high-quality data is lacking, the imitation learning agent cannot be fully trained, and the expert data set has poor generalization ability in the high-dimensional state space, resulting in generalization of the agent. Low ability.
Using a simulation learning mixed sample processing method based on visual pre-training model, by adding target noise to suboptimal expert samples, optimizing mixed sample sets using adversarial generation network, adjusting sample weight coefficients, updating the strategy and reward function network, optimizing expert samples, and Improve data set quality and agent generalization capabilities.
The data distribution and feature distribution of suboptimal expert samples have been improved, the generalization ability of imitation learning agents has been improved, and differentiated learning is possible for mixed expert samples with different qualities, improving the quality of sample distribution in the data set.
Smart Images

Figure CN117975190B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of deep learning technology, and in particular to a method and device for processing mixed samples of imitation learning based on a visual pre-training model. Background Art
[0002] With the continuous development of high-tech technologies such as robots, self-driving cars, and game agents, how to enable them to complete complex decision-making tasks and quickly adapt to environmental changes has become an important research issue.
[0003] Adversarial generative imitation learning is one of the representative algorithms of imitation learning. It draws on the idea of Generative Adversarial Network (GAN) and makes the agent close to the decision-making ability of expert agents through adversarial training of agents and reward functions. Adversarial generative imitation learning aims to imitate the behavior of experts by training a generative model to achieve automatic task solving. In this method, the generative model is trained to generate samples similar to expert behavior, while the discriminative model is used to distinguish between samples generated by the generative model and real samples of experts, so as to enable the generative model to gradually approach expert behavior.
[0004] In the related technology, adversarial generative imitation learning relies on high-quality data. In many task scenarios, due to the limitation of labor costs, there is a lack of sufficient optimal expert samples for the imitation learning agent to be fully trained. In addition, common expert data sets usually contain suboptimal expert samples. Imitation of suboptimal expert samples will reduce the performance of the imitation learning algorithm, resulting in low efficiency in processing related characters. In addition, the data in the expert data set has poor generalization ability in high-dimensional state space. With the increase of the dimension of the state space, for example, in the visual observation task scenario (the state is represented by a high-dimensional image), the redundant information about the environment increases significantly, making it difficult to capture the changes in the main body of the agent. In this case, the converged agent will be significantly affected by the environmental changes when it is placed in a new test environment, resulting in a decrease in the generalization ability and learning ability of the agent. Summary of the invention
[0005] The present invention provides a sample processing method and device based on an adversarial generative imitation learning algorithm, which is used to solve the problems that the adversarial generative imitation learning algorithm in the prior art relies on high-quality data, but lacks high-quality data, and is affected by low-quality data, resulting in the inability to fully train the imitation learning agent, resulting in poor training effect. In addition, the generalization ability of the existing expert data set in high-dimensional state space is poor, resulting in low generalization ability of the trained agent. The method improves the quality of the expert sample set, thereby improving the generalization ability of the imitation learning agent.
[0006] The present invention provides a mixed sample processing method for imitation learning based on a visual pre-training model, comprising:
[0007] Acquire an expert sample set, wherein the expert sample set includes an optimal expert sample and a suboptimal expert sample;
[0008] Adding target noise to the suboptimal expert samples to obtain noise expert samples, and processing the noise expert samples and the optimal expert samples according to a generative adversarial network to obtain a mixed sample set;
[0009] Calibrate the weight coefficient of each sample in the mixed sample set to obtain a redistributed mixed sample set, predict the redistributed mixed sample set according to the policy network to obtain an action prediction result; score the action prediction result according to the reward function network to obtain a scoring result; train the policy network and the reward function network according to the discriminant loss function and the scoring result to obtain a target policy network and a target reward function network;
[0010] According to the target reward function network, each sample in the evaluation data set is scored to obtain the predicted ranking corresponding to the evaluation data set, the ranking error loss is calculated according to the predicted ranking, the weight coefficient corresponding to each sample in the redistributed mixed sample set is updated by gradient optimization to obtain the redistributed weight coefficient, and the redistributed weight coefficient is imitated and learned according to the target strategy network to obtain the optimized expert sample; the evaluation data set belongs to the mixed sample set.
[0011] The present invention also provides a mixed sample processing method for imitation learning based on a visual pre-training model, wherein obtaining an expert sample set comprises:
[0012] Based on the visual pre-training model, forward graph reasoning is performed on each sample in the expert sample set, and the feature graph corresponding to the middle layer of the network is determined to be the effective feature of the expert sample set.
[0013] The present invention also provides an imitation learning mixed sample processing method based on a visual pre-training model, wherein the adversarial generative network includes a generative network and a discriminative network;
[0014] The processing of the noise expert samples and the optimal expert samples according to the generative adversarial network to obtain a mixed sample set includes:
[0015] Extracting features from the noise expert sample and the optimal expert sample according to the generating network to obtain a state representation vector;
[0016] Performing feature discrimination on the source of the state representation vector according to the discrimination network to obtain a discrimination result;
[0017] Calculating the loss value of the generating network according to the discrimination result and the first loss function, and optimizing the parameters of the generating network according to the loss value of the generating network to obtain the optimized generating network; calculating the loss value of the discriminant network according to the discrimination result and the second loss function, and optimizing the parameters of the discriminant network according to the loss value of the discriminant network to obtain the optimized discriminant network to output the mixed sample set;
[0018] Among them, the first loss function is determined based on the occupancy measurement of the optimal expert sample state-action pair, the occupancy measurement after normal distribution noise, the state characterization vector, the conditional probability distribution corresponding to the state characterization vector, and the discriminant network parameters; the second loss function is determined based on the occupancy measurement of the optimal expert sample state-action pair, the occupancy measurement after normal distribution noise, the state characterization vector, the conditional probability corresponding to the state characterization vector and the generation network parameters.
[0019] The present invention also provides a mixed sample processing method for imitation learning based on a visual pre-training model, wherein the first loss function includes:
[0020]
[0021] The second loss function includes:
[0022]
[0023] Among them, E is the expectation, D(z) is the discriminant network, ρ * (s, a) is the occupancy measure of the optimal expert sample state-action pair, ρ′(s, a) is the occupancy measure after adding normal distribution noise, z is the state representation vector, p(z|x) is the conditional probability distribution corresponding to the state representation vector, θ D is the discriminant network parameter, θ E To generate network parameters.
[0024] The present invention also provides a mixed sample processing method for imitation learning based on a visual pre-training model, wherein the first loss function further includes:
[0025] A target regularization term, wherein the target regularization term is determined according to the KL divergence corresponding to the optimal expert sample and the state representation vector;
[0026] The second loss function also includes: the target regularization term.
[0027] The present invention also provides a mixed sample processing method for imitation learning based on a visual pre-training model, wherein the evaluation data set is obtained by sorting the expert sample set according to a priori;
[0028] The calculating of the sorting error loss according to the predicted sorting and updating the weight coefficient corresponding to each sample in the redistributed mixed sample set by gradient optimization to obtain the weight coefficient after redistribution include:
[0029] Calculate the ranking difference loss through the predicted ranking, the prior ranking and the ranking loss function;
[0030] Determining the weight coefficient after redistribution according to the sorting difference loss;
[0031] The ranking loss function includes:
[0032]
[0033] Among them, i, j are sample numbers, η ζi is the true cumulative expected return of the i-th sample, η′ ζi is the predicted cumulative expected return of sample No. i; η ζj is the true cumulative expected return of the jth sample, η′ ζj is the predicted cumulative expected return of the j-th sample.
[0034] The present invention also provides a mixed sample processing device for imitation learning based on a visual pre-training model, comprising:
[0035] A sample acquisition module, used to acquire an expert sample set, wherein the expert sample set includes an optimal expert sample and a suboptimal expert sample, and each sample in the expert sample set corresponds to a different weight coefficient;
[0036] A first processing module is used to add target noise to the suboptimal expert samples to obtain noise expert samples, and process the noise expert samples and the optimal expert samples according to a generative adversarial network to obtain a mixed sample set;
[0037] The second processing module is used to calibrate the weight coefficient of each sample in the mixed sample set to obtain a redistributed mixed sample set, predict the redistributed mixed sample set according to the policy network to obtain an action prediction result; score the action prediction result according to the reward function network to obtain a scoring result; train the policy network and the reward function network according to the discriminant loss function and the scoring result to obtain a target policy network and a target reward function network;
[0038] The third processing module is used to score each sample in the evaluation data set according to the target reward function network, obtain the predicted ranking corresponding to the evaluation data set, calculate the ranking error loss according to the predicted ranking, update the weight coefficient corresponding to each sample in the redistributed mixed sample set through gradient optimization, obtain the redistributed weight coefficient, and perform imitation learning on the redistributed weight coefficient according to the target strategy network to obtain the optimized expert sample; the evaluation data set belongs to the mixed sample set.
[0039] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the mixed sample processing method for imitation learning based on a visual pre-training model as described in any one of the above is implemented.
[0040] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described methods for processing mixed samples by imitation learning based on a visual pre-training model.
[0041] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for processing mixed samples by imitation learning based on a visual pre-training model.
[0042] The present invention provides an imitation learning mixed sample processing method and device based on a visual pre-training model, which improves the data distribution of suboptimal expert samples by adding target noise to suboptimal expert samples, optimizes the parameters of the adversarial generative network according to the noise expert samples and the optimal expert samples, obtains a mixed sample set, improves the feature distribution of the suboptimal expert samples, and then calibrates the weight coefficient of each sample in the mixed sample set, predicts and scores the calibrated mixed sample set according to the policy network to update the policy network and the reward function network, scores each sample in the evaluation data set according to the target reward function network to obtain the corresponding prediction ranking, and then updates the weight coefficient corresponding to each sample in the redistributed mixed sample set according to the prediction ranking to obtain the redistributed weight coefficient, and imitates the redistributed weight coefficient according to the target policy network to obtain optimized expert samples, which can perform differentiated learning for mixed expert samples of different qualities, improve the sample distribution of the data set, and enhance the generalization ability of the imitation learning agent. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0044] Figure 1 It is one of the flow charts of the mixed sample processing method of imitation learning based on the visual pre-training model provided by the present invention;
[0045] Figure 2 This is the second flow chart of the mixed sample processing method for imitation learning based on the visual pre-training model provided by the present invention;
[0046] Figure 3 It is a schematic diagram of a process for extracting an intermediate layer feature map based on a visual pre-training model provided by the present invention;
[0047] Figure 4 This is the third flow chart of the mixed sample processing method for imitation learning based on the visual pre-training model provided by the present invention;
[0048] Figure 5 It is a structural schematic diagram of a mixed sample processing device for imitation learning based on a visual pre-training model provided by the present invention;
[0049] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0050] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0051] Combine the following Figure 1-Figure 5 The present invention describes a mixed sample processing method and device for imitation learning based on a visual pre-training model.
[0052] Figure 1 is one of the flow charts of the mixed sample processing method for imitation learning based on the visual pre-training model provided by the present invention, such as Figure 1 As shown, the mixed sample processing method of imitation learning based on the visual pre-training model includes the following steps:
[0053] Step 110: Obtain an expert sample set, where the expert sample set includes optimal expert samples and suboptimal expert samples.
[0054] In this step, the expert sample set may include at least one required training sample or test sample in the following scenarios: image processing tasks such as image classification, target detection and image segmentation.
[0055] For example, for image classification tasks, the expert sample set includes image data to be classified and its sample image data; for image segmentation tasks, the expert sample set includes image data to be segmented and its sample image data.
[0056] In this embodiment, the optimal expert sample is of higher quality than the suboptimal expert sample. For example, the sample contains richer data content and more effective information.
[0057] Step 120: Add target noise to the suboptimal expert samples to obtain noisy expert samples, and process the noisy expert samples and the optimal expert samples according to the generative adversarial network to obtain a mixed sample set.
[0058] In this step, the target noise includes normally distributed noise, such as Gaussian noise or Gaussian white noise.
[0059] In this embodiment, the generative adversarial network includes a generative network and a discriminative network; wherein the generative network uses noise expert samples and optimal expert samples as features to extract new state representations, and discriminates the source of the new state representations through the discriminative network, and calculates the corresponding loss function according to the discrimination results, thereby optimizing the parameters of the generative adversarial network and the discriminative network, and through the supervision and guidance of the feature extractor learning, the hidden layer state representation of the suboptimal expert sample approaches the representation of the optimal expert sample.
[0060] In this embodiment, the behavioral characteristics of the suboptimal expert sample are compared with the behavioral characteristics of the optimal expert sample, and a noise signal is used to simulate the difference in distribution between the two, while being compatible with the deep generative imitation learning algorithm architecture; specifically, by minimizing the distribution distance between the two distributions, such as the KL (Kullback-Leibler) divergence, the state representation of the suboptimal expert sample after feature extraction is made close to the state representation of the optimal expert sample.
[0061] Step 130: calibrate the weight coefficient of each sample in the mixed sample set to obtain a redistributed mixed sample set, predict the redistributed mixed sample set according to the policy network to obtain an action prediction result; score the action prediction result according to the reward function network to obtain a scoring result; train the policy network and the reward function network according to the discriminant loss function and the scoring result to obtain a target policy network and a target reward function network.
[0062] In this step, each expert sample corresponds to a different weight coefficient. The higher the weight coefficient corresponding to the best expert sample in the expert sample set, the higher the quality of the expert sample set, and the more suitable it is as input data for the above-mentioned image processing task.
[0063] For example, the weight coefficient β∈(0,1), if the weight coefficient of a sample is closer to 1, the quality of the sample is higher.
[0064] In this step, the imitation learning strategy is used. The closer the weight coefficient corresponding to the sample is to 1 (the larger the sample weight is), the more important it is to imitate the sample.
[0065] Figure 2 This is the second flow chart of the mixed sample processing method for imitation learning based on the visual pre-training model provided by the present invention. Figure 2 In the illustrated embodiment, in the weight learning stage, a weight coefficient is calibrated for each sample in the mixed sample set to indicate the quality of the sample (initialized to 1, indicating that each sample is considered to have the same quality a priori); after calibrating the weight coefficient, the expert sample redistributed by the weight coefficient is obtained, that is, the redistributed mixed sample set.
[0066] In this embodiment, the redistributed mixed sample set is taken as input into the policy network to predict the corresponding action output.
[0067] For example, the scenario for imitation learning is a sequential decision-making task. Taking autonomous driving as an example, the expert sample is the road image obtained by the camera, and the predicted action is the steering wheel angle and throttle force.
[0068] exist Figure 2 In the illustrated embodiment, in the imitation learning stage, the action output of the policy network, i.e., the action prediction result, is input into the reward function network to score different actions (for example, high scores are given to expert samples and low scores are given to samples generated by the policy network). The policy network and the reward function network are then updated by calculating the loss function to obtain the target policy network and the target reward function network, so as to improve the ability of the policy network to generate expert-like actions and improve the ability of the reward function to distinguish samples generated by the policy network, i.e., adversarial training.
[0069] Step 140: score each sample in the evaluation data set according to the target reward function network to obtain the predicted ranking corresponding to the evaluation data set, calculate the ranking error loss according to the predicted ranking, update the weight coefficient corresponding to each sample in the redistributed mixed sample set through gradient optimization to obtain the redistributed weight coefficient, perform imitation learning on the redistributed weight coefficient according to the target strategy network to obtain the optimized expert sample; the evaluation data set belongs to the mixed sample set.
[0070] In this step, the corresponding evaluation data set is extracted from the mixed sample set by sorting according to preset weight coefficients (ie, prior sorting).
[0071] For example, after obtaining the target reward function network, a small number of samples are extracted from the original mixed expert samples, and the relative quality of these samples is calibrated as prior knowledge to form an evaluation data set.
[0072] In this embodiment, the evaluation data set is obtained by prior sorting the expert sample set; the sorting error loss is calculated according to the predicted sorting update, and the weight coefficient corresponding to each sample in the mixed sample set is redistributed by gradient optimization to obtain the weight coefficient after redistribution, including: calculating the sorting difference loss through the predicted sorting, the prior sorting and the sorting loss function; determining the weight coefficient after redistribution according to the sorting difference loss; the sorting loss function includes:
[0073]
[0074] Among them, i, j are sample numbers, η ζi is the true cumulative expected return of the i-th sample, η′ ζi is the predicted cumulative expected return of sample No. i; η ζj is the true cumulative expected return of the jth sample, η′ ζj is the predicted cumulative expected return of the j-th sample.
[0075] exist Figure 2 In the illustrated embodiment, a reward function network is used to score the data in the evaluation data set to obtain a predicted ranking. The predicted ranking and the calibrated prior ranking information can be compared to calculate the ranking difference loss, and the weight coefficient is updated with this loss function. The smaller the loss, the better the reward function is learned in the imitation learning stage, which indirectly means that the weight coefficient is set reasonably.
[0076] In this embodiment, after obtaining the redistributed weight coefficients, the mixed samples corresponding to the redistributed weight coefficients are sent to the strategy network for imitation learning. Imitation learning and weight learning are performed alternately until a predetermined number of iterations are reached and then the optimized expert samples are output.
[0077] In this embodiment, in a specific training process, the weight coefficients are adaptively optimized for differentiated learning of each sample in the expert sample set, and the current imitation learning strategy is redistributed by redistributing the weight coefficients of each sample. In each differentiated learning process, the optimized generation network outputs a reward value, and the weight coefficients corresponding to each sample are reordered according to the reward value and the sorting loss function by the optimized discriminant network; in multiple training processes, the adaptive optimization of the sorting of the weight coefficients corresponding to each sample is achieved by maximizing the cumulative expected reward value to improve the distribution of each sample in the expert sample set and improve the quality of the expert sample set; wherein, the maximized cumulative expected reward is the cumulative value of the expected reward predicted by the reward function after training with the adversarial inverse reinforcement learning algorithm.
[0078] Specifically, assume that the current hybrid expert strategy is represented by π d , the occupancy rate of the current strategy in the process of interaction with the environment is measured as Through π d After redistributing the state-action pairs, we can get the new strategy π new ; Its occupancy rate can be expressed as The imitation learning loss function after weight coefficient redistribution can be expressed as:
[0079] Among them, L 模仿 It can be the loss function of any traditional imitation learning algorithm such as behavior cloning or inverse reinforcement learning.
[0080] In this embodiment, by performing differentiated learning on different expert samples through weight coefficients, the performance of the imitation learning algorithm can be maximized, that is, the purpose of maximizing the cumulative expected return can be achieved; the specific optimization problem can be expressed as:
[0081]
[0082] Among them, β * represents the optimal weight coefficient distribution, Represents the redistributed strategy π new The cumulative expected return is expressed as:
[0083]
[0084] Among them, R is the reward function of the environment, γ is the attenuation coefficient; the purpose of weight learning is to maximize the performance of the imitation learning algorithm, so the performance of the imitation learning algorithm can be evaluated to reflect the effect of the current weight coefficient learning; let the current time be t, s t represents the expert sample state information at time t, a t Represents the expert sample action information at time t.
[0085] The imitation learning mixed sample processing method based on the visual pre-training model provided by the embodiment of the present invention improves the data distribution of the suboptimal expert samples by adding target noise to the suboptimal expert samples, optimizes the parameters of the adversarial generative network according to the noise expert samples and the optimal expert samples, obtains a mixed sample set, improves the feature distribution of the suboptimal expert samples, and then calibrates the weight coefficient of each sample in the mixed sample set, predicts and scores the calibrated mixed sample set according to the policy network to update the policy network and the reward function network, scores each sample in the evaluation data set according to the target reward function network to obtain the corresponding prediction ranking, and then updates the weight coefficient corresponding to each sample in the redistributed mixed sample set according to the prediction ranking to obtain the redistributed weight coefficient, and imitates the redistributed weight coefficient according to the target policy network to obtain the optimized expert sample, which can perform differentiated learning for mixed expert samples of different qualities, improve the sample distribution of the data set, and enhance the generalization ability of the imitation learning agent.
[0086] In some embodiments, obtaining the expert sample set includes: performing forward graph reasoning on each sample in the expert sample set based on a visual pre-training model, and determining that a feature map corresponding to a middle layer of the network is a valid feature of the expert sample set.
[0087] In this embodiment, the visual pre-trained model is a deep learning model pre-trained using a large-scale dataset, and can generally be used to initialize network parameters for downstream tasks such as image classification, target detection, and image segmentation.
[0088] In this embodiment, the general visual reinforcement learning method uses high-dimensional image information as observation of the environment, and has higher requirements on the state abstraction of the feature extractor; because in the adversarial generative imitation learning scenario, factors such as the expressiveness, trainability, and capacity of the image feature extractor are key links in the generalization of the algorithm, generative imitation learning implemented by features extracted from a pre-trained model based on a large-scale visual dataset can improve the generalization ability of the intelligent agent in new tasks or new environments.
[0089] Figure 3 is a schematic diagram of a process for extracting an intermediate layer feature map based on a visual pre-training model provided by the present invention. Figure 3 In the illustrated embodiment, the original feature map is input into a visual pre-trained model (corresponding to a convolutional neural network pre-trained on a large-scale visual data set). By fixing the network parameters, only forward reasoning is performed on the input image data, and the feature map of the middle layer of the network is extracted as a visual observation input and input into a subsequent imitation learning algorithm.
[0090] In this embodiment, during the inference process, only the normalization layer (BatchNorm) of the pre-trained model can be updated by updating the parameters, that is, by keeping the mean and standard deviation parameters corresponding to the normalization layer of the pre-trained model during training, because the statistics updated by BatchNorm help to better adapt to the shift in visual observations, thereby improving the generalization ability of the agent.
[0091] The embodiment of the present invention provides an imitation learning mixed sample processing method based on a visual pre-training model. It performs forward graph reasoning on each sample in the expert sample set through the visual pre-training model, and determines that the feature map corresponding to the middle layer of the network is the effective feature of the expert sample set, thereby reducing R&D costs and providing data input for subsequent generative imitation learning algorithms to improve the generalization ability of the intelligent agent in new tasks or new environments.
[0092] In some embodiments, the adversarial generative network includes a generative network and a discriminative network; the noise expert samples and the optimal expert samples are processed according to the adversarial generative network to obtain a mixed sample set, including: extracting features from the noise expert samples and the optimal expert samples according to the generative network to obtain a state characterization vector; performing feature discrimination on the source of the state characterization vector according to the discriminative network to obtain a discrimination result; calculating the loss value of the generative network according to the discrimination result and the first loss function, and optimizing the parameters of the generative network according to the loss value of the generative network to obtain an optimized generative network; calculating the loss value of the discriminative network according to the discrimination result and the second loss function, and optimizing the parameters of the discriminative network according to the loss value of the discriminative network to obtain an optimized discriminative network to output a mixed sample set.
[0093] In this embodiment, the feature extraction network E (corresponding to the generation network) and the discriminant network D are initialized; the optimal expert sample ξ in the expert sample set is collected * and suboptimal expert samples ξ′, inject normally distributed noise ∈ into the suboptimal expert samples to obtain noisy expert samples, and then input the optimal expert samples and the noisy expert samples into the feature extraction network E to obtain a new state representation z, that is, the state representation vector; input the state representation vector into the discriminant network, and discriminate the source of the features, calculate the loss function according to the discrimination results, and optimize the parameters of the feature extraction network and the discriminant network.
[0094] Among them, the first loss function is determined based on the occupancy measurement of the state-action pair of the optimal expert sample, the occupancy measurement after normal distribution noise, the state representation vector, the conditional probability distribution corresponding to the state representation vector, and the discriminant network parameters; the second loss function is determined based on the occupancy measurement of the state-action pair of the optimal expert sample, the occupancy measurement after normal distribution noise, the state representation vector, the conditional probability corresponding to the state representation vector, and the generation network parameters.
[0095] Specifically, the first loss function includes:
[0096]
[0097] The second loss function includes:
[0098]
[0099] Among them, E is the expectation, D(z) is the discriminant network, ρ * (s, a) is the occupancy measure of the optimal expert sample state-action pair, ρ′(s, a) is the occupancy measure after adding normal distribution noise, z is the state representation vector, p(z|x) is the conditional probability distribution corresponding to the state representation vector, θ D is the discriminant network parameter, θ E To generate network parameters.
[0100] Figure 4 This is a flow chart of the third method for processing mixed samples of imitation learning based on a visual pre-training model provided by the present invention. Figure 4 In the illustrated embodiment, noise is injected into suboptimal expert samples to obtain noise expert samples, and the optimal expert samples and noise expert samples are input into the generation network (corresponding to the feature extraction network) to obtain a state characterization vector (corresponding to the state feature), which is used to provide mutual information constraints for the optimal expert samples; the state characterization vector is input into the discriminant network to perform feature discrimination on the feature source to obtain a discrimination result, and then the loss value of the generation network is calculated according to the discrimination result, and the network gradient of the feature extraction network is updated according to the loss value of the generation network to obtain an optimized generation network; at the same time, the loss value of the discriminant network is calculated according to the discrimination result, and the gradient of the discriminant network is updated according to the loss value of the discriminant network to obtain an optimized discriminant network, thereby outputting a mixed sample set.
[0101] In this embodiment, the first loss function also includes: a target regularization term, which is determined according to the KL divergence corresponding to the optimal expert sample and the state representation vector; the second loss function also includes: a target regularization term.
[0102] In this embodiment, during the training process of the generative network and the discriminative network, the KL divergence of the optimal expert sample and the extracted features can be used as a regularization term and added to the loss function of the feature extraction network to prevent the extracted features from being too different from the optimal expert sample features, thereby improving the algorithm performance.
[0103] The embodiment of the present invention provides an imitation learning mixed sample processing method based on a visual pre-training model, which processes noisy expert samples and optimal expert samples through an adversarial generative network to obtain a mixed sample set, then uses a generative network to extract features from the noisy expert samples and the optimal expert samples to obtain a state representation vector, and uses a discriminant network to perform feature discrimination on the source of the state representation vector to obtain a discrimination result, and finally optimizes the generative network and the discriminant network respectively according to the discrimination result, the first loss function and the second loss function, and outputs a mixed sample set; by introducing noise contrast estimation into adversarial generative imitation learning to improve the data distribution of suboptimal expert samples, simulating the feature difference between the optimal expert samples and the suboptimal expert samples through normal noise, and using an adversarial generation method to train a feature extractor to improve the feature distribution of suboptimal expert samples, thereby improving the performance of the imitation learning algorithm under mixed expert samples.
[0104] The following is a description of the mixed sample processing device for imitation learning based on a visual pre-training model provided by the present invention. The mixed sample processing device for imitation learning based on a visual pre-training model described below and the mixed sample processing method for imitation learning based on a visual pre-training model described above can be referenced to each other.
[0105] Figure 5 is a schematic diagram of the structure of the mixed sample processing device for imitation learning based on the visual pre-training model provided by the present invention, such as Figure 5 As shown, the hybrid sample processing device for imitation learning based on the visual pre-training model includes: a sample acquisition module 510, a first processing module 520, a second processing module 530 and a third processing module 540.
[0106] The sample acquisition module 510 is used to acquire an expert sample set, where the expert sample set includes an optimal expert sample and a suboptimal expert sample, and each sample in the expert sample set corresponds to a different weight coefficient;
[0107] A first processing module 520 is used to add target noise to the suboptimal expert samples to obtain noise expert samples, and process the noise expert samples and the optimal expert samples according to the generative adversarial network to obtain a mixed sample set;
[0108] The second processing module 530 is used to calibrate the weight coefficient of each sample in the mixed sample set to obtain a redistributed mixed sample set, predict the redistributed mixed sample set according to the policy network to obtain an action prediction result; score the action prediction result according to the reward function network to obtain a scoring result; train the policy network and the reward function network according to the discriminant loss function and the scoring result to obtain a target policy network and a target reward function network;
[0109] The third processing module 540 is used to score each sample in the evaluation data set according to the target reward function network, obtain the predicted ranking corresponding to the evaluation data set, calculate the ranking error loss according to the predicted ranking, update the weight coefficient corresponding to each sample in the redistributed mixed sample set through gradient optimization, obtain the redistributed weight coefficient, and perform imitation learning on the redistributed weight coefficient according to the target strategy network to obtain the optimized expert sample; the evaluation data set belongs to the mixed sample set.
[0110] The imitation learning mixed sample processing device based on the visual pre-training model provided by the embodiment of the present invention improves the data distribution of the suboptimal expert samples by adding target noise to the suboptimal expert samples, optimizes the parameters of the adversarial generative network according to the noise expert samples and the optimal expert samples, obtains a mixed sample set, improves the feature distribution of the suboptimal expert samples, and then calibrates the weight coefficient of each sample in the mixed sample set, predicts and scores the calibrated mixed sample set according to the policy network to update the policy network and the reward function network, scores each sample in the evaluation data set according to the target reward function network to obtain the corresponding prediction ranking, and then updates the weight coefficient corresponding to each sample in the redistributed mixed sample set according to the prediction ranking to obtain the redistributed weight coefficient, and imitates the redistributed weight coefficient according to the target policy network to obtain the optimized expert samples, which can perform differentiated learning for mixed expert samples of different qualities, improve the sample distribution of the data set, and enhance the generalization ability of the imitation learning agent.
[0111] Figure 6 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 6As shown, the electronic device may include: a processor (processor) 610, a communication interface (Communications Interface) 620, a memory (memory) 630 and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call the logic instructions in the memory 630 to execute the imitation learning mixed sample processing method based on the visual pre-training model, the method comprising: obtaining an expert sample set, the expert sample set comprising an optimal expert sample and a suboptimal expert sample; adding target noise to the suboptimal expert sample to obtain a noise expert sample, processing the noise expert sample and the optimal expert sample according to the adversarial generation network to obtain a mixed sample set; calibrating the weight coefficient of each sample in the mixed sample set to obtain a redistributed mixed sample set, predicting the redistributed mixed sample set according to the policy network to obtain an action prediction result; scoring the action prediction result according to the reward function network to obtain a scoring result; training the policy network and the reward function network according to the discriminant loss function and the scoring result to obtain a target policy network and a target reward function network; scoring each sample in the evaluation data set according to the target reward function network to obtain a predicted ranking corresponding to the evaluation data set, calculating the ranking error loss according to the predicted ranking, updating the weight coefficient corresponding to each sample in the redistributed mixed sample set through gradient optimization to obtain the redistributed weight coefficient, and imitating learning the redistributed weight coefficient according to the target policy network to obtain an optimized expert sample; the evaluation data set belongs to the mixed sample set.
[0112] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0113] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the imitation learning mixed sample processing method based on the visual pre-training model provided by the above methods, and the method includes: obtaining an expert sample set, the expert sample set includes an optimal expert sample and a suboptimal expert sample; adding target noise to the suboptimal expert sample to obtain a noise expert sample, processing the noise expert sample and the optimal expert sample according to a generative adversarial network to obtain a mixed sample set; calibrating the weight coefficient of each sample in the mixed sample set to obtain a redistributed mixed sample set, and Predict the redistributed mixed sample set to obtain the action prediction result; score the action prediction result according to the reward function network to obtain the scoring result; train the policy network and the reward function network according to the discriminant loss function and the scoring result to obtain the target policy network and the target reward function network; score each sample in the evaluation data set according to the target reward function network to obtain the predicted ranking corresponding to the evaluation data set, calculate the ranking error loss according to the predicted ranking, update the weight coefficient corresponding to each sample in the redistributed mixed sample set through gradient optimization to obtain the redistributed weight coefficient, and imitate the redistributed weight coefficient according to the target policy network to obtain the optimized expert sample; the evaluation data set belongs to the mixed sample set.
[0114] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the mixed sample processing method for imitation learning based on the visual pre-training model provided by the above methods, the method comprising: obtaining an expert sample set, the expert sample set comprising an optimal expert sample and a suboptimal expert sample; adding target noise to the suboptimal expert sample to obtain a noise expert sample, processing the noise expert sample and the optimal expert sample according to a generative adversarial network to obtain a mixed sample set; calibrating a weight coefficient for each sample in the mixed sample set to obtain a redistributed mixed sample set, predicting the redistributed mixed sample set according to a strategy network, Obtain action prediction results; score the action prediction results according to the reward function network to obtain scoring results; train the policy network and the reward function network according to the discriminant loss function and the scoring results to obtain the target policy network and the target reward function network; score each sample in the evaluation data set according to the target reward function network to obtain the predicted ranking corresponding to the evaluation data set, calculate the ranking error loss according to the predicted ranking, update the weight coefficient corresponding to each sample in the redistributed mixed sample set through gradient optimization, obtain the redistributed weight coefficient, perform imitation learning on the redistributed weight coefficient according to the target policy network to obtain the optimized expert sample; the evaluation data set belongs to a mixed sample set.
[0115] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0116] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A mixed sample processing method for imitation learning based on a visual pre-training model, characterized in that: include: Acquire an expert sample set, wherein the expert sample set includes an optimal expert sample and a suboptimal expert sample; Adding target noise to the suboptimal expert samples to obtain noise expert samples, and processing the noise expert samples and the optimal expert samples according to a generative adversarial network to obtain a mixed sample set; Calibrate the weight coefficient of each sample in the mixed sample set to obtain a redistributed mixed sample set, predict the redistributed mixed sample set according to the policy network to obtain an action prediction result; score the action prediction result according to the reward function network to obtain a scoring result; train the policy network and the reward function network according to the discriminant loss function and the scoring result to obtain a target policy network and a target reward function network; Scoring each sample in the evaluation data set according to the target reward function network to obtain a predicted ranking corresponding to the evaluation data set, calculating a ranking error loss according to the predicted ranking, updating a weight coefficient corresponding to each sample in the redistributed mixed sample set by gradient optimization to obtain a redistributed weight coefficient, and performing imitation learning on the redistributed weight coefficient according to the target strategy network to obtain an optimized expert sample; The evaluation data set belongs to the mixed sample set; The obtaining of the expert sample set comprises: Perform forward graph reasoning on each sample in the expert sample set based on the visual pre-training model, and determine that the feature graph corresponding to the middle layer of the network is the effective feature of the expert sample set; The performing forward graph reasoning on each sample in the expert sample set based on the visual pre-training model comprises: The original feature maps corresponding to each sample in the expert sample set are input into the visual pre-training model. By fixing the network parameters, only forward reasoning is performed on the input image data. During the reasoning process, only the mean and standard deviation parameters corresponding to the normalization layer of the visual pre-training model are updated. The expert sample set includes training samples or test samples required by at least one of the following scenarios: image processing tasks of image classification, target detection and image segmentation.
2. The method for processing mixed samples based on imitation learning of a visual pre-training model according to claim 1, characterized in that: The adversarial generation network includes a generation network and a discrimination network; The processing of the noise expert samples and the optimal expert samples according to the generative adversarial network to obtain a mixed sample set includes: Extracting features from the noise expert sample and the optimal expert sample according to the generating network to obtain a state representation vector; Performing feature discrimination on the source of the state representation vector according to the discrimination network to obtain a discrimination result; Calculating the loss value of the generating network according to the discrimination result and the first loss function, and optimizing the parameters of the generating network according to the loss value of the generating network to obtain the optimized generating network; calculating the loss value of the discriminant network according to the discrimination result and the second loss function, and optimizing the parameters of the discriminant network according to the loss value of the discriminant network to obtain the optimized discriminant network to output the mixed sample set; Among them, the first loss function is determined based on the occupancy measurement of the optimal expert sample state-action pair, the occupancy measurement after normal distribution noise, the state characterization vector, the conditional probability distribution corresponding to the state characterization vector, and the discriminant network parameters; the second loss function is determined based on the occupancy measurement of the optimal expert sample state-action pair, the occupancy measurement after normal distribution noise, the state characterization vector, the conditional probability corresponding to the state characterization vector and the generation network parameters.
3. The method for processing mixed samples based on imitation learning of a visual pre-training model according to claim 2, characterized in that: The first loss function includes: The second loss function includes: Among them, E is the expectation, D(z) is the discriminant network, ρ * (s, a) is the occupancy measure of the optimal expert sample state-action pair, ρ′(s, a) is the occupancy measure after adding normal distribution noise, z is the state representation vector, p(z|x) is the conditional probability distribution corresponding to the state representation vector, θ D is the discriminant network parameter, θ E To generate network parameters.
4. The method for processing mixed samples based on imitation learning of a visual pre-training model according to claim 2, characterized in that: The first loss function also includes: A target regularization term, wherein the target regularization term is determined according to the KL divergence corresponding to the optimal expert sample and the state representation vector; The second loss function also includes: the target regularization term.
5. The method for processing mixed samples by imitation learning based on a visual pre-training model according to claim 1, characterized in that: The evaluation data set is obtained by sorting the expert sample set according to a priori; The calculating of the sorting error loss according to the predicted sorting and updating the weight coefficient corresponding to each sample in the redistributed mixed sample set by gradient optimization to obtain the weight coefficient after redistribution include: Calculate the ranking difference loss through the predicted ranking, the prior ranking and the ranking loss function; Determining the weight coefficient after redistribution according to the sorting difference loss; The ranking loss function includes: Among them, i and j are sample numbers. is the true cumulative expected return of the i-th sample, is the predicted cumulative expected return of sample No. i; is the true cumulative expected return of the j-th sample, is the predicted cumulative expected return of the j-th sample.
6. A mixed sample processing device for imitation learning based on a visual pre-training model, characterized in that: include: A sample acquisition module, used to acquire an expert sample set, wherein the expert sample set includes an optimal expert sample and a suboptimal expert sample, and each sample in the expert sample set corresponds to a different weight coefficient; A first processing module is used to add target noise to the suboptimal expert samples to obtain noise expert samples, and process the noise expert samples and the optimal expert samples according to a generative adversarial network to obtain a mixed sample set; The second processing module is used to calibrate the weight coefficient of each sample in the mixed sample set to obtain a redistributed mixed sample set, predict the redistributed mixed sample set according to the policy network to obtain an action prediction result; score the action prediction result according to the reward function network to obtain a scoring result; train the policy network and the reward function network according to the discriminant loss function and the scoring result to obtain a target policy network and a target reward function network; A third processing module is used to score each sample in the evaluation data set according to the target reward function network, obtain the predicted ranking corresponding to the evaluation data set, calculate the ranking error loss according to the predicted ranking, update the weight coefficient corresponding to each sample in the redistributed mixed sample set through gradient optimization, obtain the redistributed weight coefficient, and perform imitation learning on the redistributed weight coefficient according to the target strategy network to obtain an optimized expert sample; the evaluation data set belongs to the mixed sample set; The sample acquisition module is specifically used for: Perform forward graph reasoning on each sample in the expert sample set based on the visual pre-training model, and determine that the feature graph corresponding to the middle layer of the network is the effective feature of the expert sample set; The performing forward graph reasoning on each sample in the expert sample set based on the visual pre-training model comprises: The original feature maps corresponding to each sample in the expert sample set are input into the visual pre-training model. By fixing the network parameters, only forward reasoning is performed on the input image data. During the reasoning process, only the mean and standard deviation parameters corresponding to the normalization layer of the visual pre-training model are updated. The expert sample set includes training samples or test samples required by at least one of the following scenarios: image processing tasks of image classification, target detection and image segmentation.
7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the mixed sample processing method for imitation learning based on the visual pre-training model as described in any one of claims 1 to 5 is implemented.
8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the mixed sample processing method for imitation learning based on a visual pre-training model as described in any one of claims 1 to 5 is implemented.
9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the mixed sample processing method for imitation learning based on a visual pre-training model as described in any one of claims 1 to 5 is implemented.
