A dual biased imitation learning method and system supported by a dynamics model

Abstract

The present invention provides a dual biased imitation learning method and system supported by a dynamic model, comprising: obtaining a training sample set required for autonomous driving training, wherein the training sample includes an offline expert data set, an offline unrewarded label data set, and a dynamic model data set; constructing a loss function by the proximity between the preset first discriminator output data and the expert data and the preset second discriminator output data and the real data, and training a driving imitation strategy model; and performing imitation training on an autonomous driving vehicle through the trained driving imitation strategy model. The present invention solves the defects of the small amount of available expert data and low training efficiency in existing imitation training, so as to realize fast and efficient biased imitation learning.

Description

Technical Field

[0001] The present invention relates to the technical field of machine imitation learning, and in particular to a dual biased imitation learning method and system supported by a dynamics model. Background Art

[0002] Through imitation learning, the characteristics of the target learning object can be obtained, and a model with basically the same functions as the target object can be generated, playing the same role. Existing imitation learning can adopt online imitation training or offline training. In the online training method, the commonly used GAN structure is difficult to train and the training results are difficult to converge; in the offline training method, the BCND structure uses the output of the previous moment to weight the next moment, which has high requirements on the quality of expert data set samples and is difficult to achieve ideal results; the ORIL structure trains the discriminator as a reward, which takes a long time, and the discriminator and strategy training are separated, resulting in low training efficiency.

[0003] And when using offline imitation learning, it is necessary to obtain learning data from expert datasets, but expert datasets have a small amount of data, high acquisition costs, difficult to guarantee quality, and limited coverage of scenarios. Therefore, it is necessary to expand learning data, but there is also a small amount of expert data in the existing unrewarded label datasets, and a large amount of interfering and useless data. It is difficult to quickly determine which data is expert data and can be used directly, resulting in poor overall training results and low training efficiency. Summary of the invention

[0004] The present invention provides a dual biased imitation learning method and system supported by a dynamics model, which are used to solve the defects of small amount of available expert data and low training efficiency in existing imitation training, so as to realize fast and efficient biased imitation learning.

[0005] The present invention provides a dual biased imitation learning method supported by a dynamics model, comprising:

[0006] Obtaining a training sample set required for autonomous driving training, wherein the training sample includes an offline expert data set, an offline unrewarded label data set, and a dynamic model data set;

[0007] A loss function is constructed by determining the proximity between the output data of the preset first discriminator and the expert data and the proximity between the output data of the preset second discriminator and the real data to train the driving imitation strategy model;

[0008] The trained driving imitation strategy model is used to perform imitation training on the autonomous driving vehicle.

[0009] According to a dual biased imitation learning method supported by a dynamics model provided by the present invention, the loss function is constructed by the proximity between the preset first discriminator output data and the expert data and the proximity between the preset second discriminator output data and the real data to train the driving imitation strategy model, specifically including:

[0010] The offline expert data set, the offline unrewarded label data set and the dynamic model data set are input into a first discriminator, the expert data is discriminated, and the degree of closeness between the output data and the expert data is output;

[0011] Inputting the offline expert data set, the offline unrewarded label data set and the dynamic model data set into a second discriminator, discriminating the real data, and outputting the degree of closeness between the data and the real data;

[0012] The loss function is constructed by the proximity between the output data and the expert data and the proximity between the output data and the real data to train the driving imitation strategy model and learn the expert data;

[0013] The dynamics model dataset is generated by learning an offline expert dataset and an offline unrewarded label dataset through a dynamics model.

[0014] According to a dual biased imitation learning method supported by a dynamics model provided by the present invention, the dynamics model dataset is generated by learning an offline expert dataset and an offline unrewarded label dataset through a dynamics model, specifically comprising:

[0015] The offline expert dataset and the offline unrewarded label dataset are jointly input into the dynamics model;

[0016] The dynamic model is learned based on the expert data in the offline expert data set and the unrewarded label data in the offline unrewarded label data set, and data amplification is performed to generate a dynamic model data set;

[0017] The dynamics model data set includes real expert data and pseudo expert data.

[0018] According to a dual biased imitation learning method supported by a dynamics model provided by the present invention, the offline expert data set, the offline unrewarded label data set and the dynamics model data set are input into a first discriminator, the expert data is discriminated, and the degree of closeness between the output data and the expert data is specifically including:

[0019] The first discriminator judges the input offline expert data set, offline unrewarded label data set and dynamic model data set to identify the expert data suboptimal data;

[0020] Increase the weight of expert data and reduce the weight of suboptimal data;

[0021] The closeness of the output data to the expert data is determined based on the weight information of the expert data and the weight information of the suboptimal dataset.

[0022] According to a dual biased imitation learning method supported by a dynamics model provided by the present invention, the offline expert data set, the offline unrewarded label data set and the dynamics model data set are input into a second discriminator, the real data is discriminated, and the closeness of the output data to the real data is specifically including:

[0023] The second discriminator judges the input offline expert data set, offline unrewarded label data set and dynamic model data set to distinguish real data from false data;

[0024] Increase the weight of the identified real data and reduce the weight of the identified false data;

[0025] The degree of closeness between the output data and the real data is determined based on the weight information of the real data and the weight information of the false data set.

[0026] According to a dual biased imitation learning method supported by a dynamics model provided by the present invention, adversarial coupling training is performed between the second discriminator and the dynamics model;

[0027] The result of the second discriminator is fed back to the dynamics model, so that the dynamics model adjusts its learning strategy and learns more real data.

[0028] According to a dual biased imitation learning method supported by a dynamic model provided by the present invention, a loss function is constructed by the proximity between the output data and the expert data and the proximity between the output data and the real data to train the driving imitation strategy model and learn the expert data, specifically including:

[0029] The degree of closeness between the output data and the expert data, the degree of closeness between the output data and the real data, and the loss function of the imitation strategy itself are added to obtain a total loss function;

[0030] The driving imitation strategy model is trained through the total loss function, the imitation learning strategy is corrected, and a corrected driving imitation strategy model is generated;

[0031] The modified driving imitation strategy model performs imitation learning on real expert data to imitate the functions corresponding to the expert data set.

[0032] The present invention also provides a dual biased imitation learning system supported by a dynamics model, the system comprising:

[0033] A data acquisition module, which acquires a training sample set required for autonomous driving training, wherein the training sample includes an offline expert data set, an offline unrewarded label data set, and a dynamic model data set;

[0034] An imitation learning module constructs a loss function based on the proximity between the preset first discriminator output data and the expert data and the preset second discriminator output data and the real data to train the driving imitation strategy model;

[0035] The training module is used to perform imitation training on the autonomous driving vehicle through the trained driving imitation strategy model.

[0036] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements a dual biased imitation learning method supported by a dynamic model as described above.

[0037] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a dual biased imitation learning method supported by a dynamic model as described in any one of the above.

[0038] The present invention provides a dual biased imitation learning method and system supported by a dynamics model. The dynamics model is introduced and the data generated by the dynamics model is used to train the imitation strategy, thereby making up for the shortage of the sample size of the expert data set and, to a certain extent, alleviating the compound error that may be caused by traditional imitation learning when the limited expert data set cannot cover the entire state space. Biased imitation learning is achieved by introducing two discriminators, and different weights are calculated for samples from different data sets, so that the imitation strategy is more sensitive to the unrewarded label samples or dynamics model generated samples that are close to the expert data set, thereby making the imitation strategy more robust and having better performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 This is one of the flow charts of a dual biased imitation learning method supported by a dynamics model provided by the present invention;

[0041] Figure 2 This is the second flow chart of a dual biased imitation learning method supported by a dynamics model provided by the present invention;

[0042] Figure 3 This is the third flow chart of a dual biased imitation learning method supported by a dynamics model provided by the present invention;

[0043] Figure 4 This is the fourth flow chart of a dual biased imitation learning method supported by a dynamics model provided by the present invention;

[0044] Figure 5 This is the fifth flow chart of a dual biased imitation learning method supported by a dynamics model provided by the present invention;

[0045] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention.

[0046] Figure 7 1 is a schematic diagram of module connection of a dual biased imitation learning system supported by a dynamics model provided by the present invention;

[0047] Figure 8 An architectural diagram of a dual biased imitation learning system supported by a dynamics model provided by the present invention;

[0048] Fig. 9 A learning curve diagram of the original imitation learning method of dual biased imitation learning supported by a dynamic model provided by the present invention in a Hopper-expert environment;

[0049] Fig.10 It is a learning curve diagram of a dual biased imitation learning method supported by a dynamic model in a Hopper-expert environment provided by the present invention.

[0050] Reference numerals:

[0051] 110: data acquisition module; 120: imitation learning module; 130: training module; 610: processor; 620: communication interface; 630: memory; 640: communication bus. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] Combine the following Figure 1-Figure 5A dual biased imitation learning method supported by a dynamics model of the present invention is described, comprising:

[0054] S100, obtaining a training sample set required for autonomous driving training, wherein the training sample includes an offline expert data set, an offline unrewarded label data set, and a dynamic model data set;

[0055] S200, constructing a loss function based on the proximity between the preset first discriminator output data and the expert data and the proximity between the preset second discriminator output data and the real data, and training the driving imitation strategy model;

[0056] S300: Performing simulation training on the autonomous driving vehicle through the trained driving simulation strategy model.

[0057] The loss function is constructed by the proximity between the preset first discriminator output data and the expert data and the preset second discriminator output data and the real data to train the driving imitation strategy model, specifically including:

[0058] S201, the offline expert data set, the offline unrewarded label data set and the dynamic model data set are input into a first discriminator, the expert data is discriminated, and the closeness between the output data and the expert data is output;

[0059] S202, inputting the offline expert data set, the offline unrewarded label data set and the dynamic model data set into a second discriminator, discriminating the real data, and outputting the degree of closeness between the data and the real data;

[0060] S203, constructing a loss function based on the proximity between the output data and the expert data and the proximity between the output data and the real data to train the driving imitation strategy model and learn the expert data;

[0061] The dynamics model dataset is generated by learning an offline expert dataset and an offline unrewarded label dataset through a dynamics model.

[0062] Since a large amount of expert data is required for imitation learning, learning training can ensure that the learning effect reaches an ideal state, but the amount of expert data is small and the acquisition cost is high, and there are some expert data and a large amount of interfering useless data in the offline unrewarded label data set. Therefore, the present invention learns the offline expert data set and the offline unrewarded label data set through a dynamic model to achieve data amplification, and discriminates the data in multiple offline expert data sets, offline unrewarded label data sets and dynamic model data sets through a first discriminator and a second discriminator, selects the real expert data, and assigns a higher weight, thereby achieving the amplification of expert data, covering the entire state space, and making the imitation strategy more sensitive to the unrewarded label samples or dynamic model generated samples close to the expert data set, so that the imitation strategy is more robust and has better performance.

[0063] The dynamics model dataset is generated by learning the offline expert dataset and the offline unrewarded label dataset through the dynamics model, including:

[0064] The offline expert dataset and the offline unrewarded label dataset are jointly input into the dynamics model;

[0065] The dynamic model is learned based on the expert data in the offline expert data set and the unrewarded label data in the offline unrewarded label data set, and data amplification is performed to generate a dynamic model data set;

[0066] The dynamics model data set includes real expert data and pseudo expert data.

[0067] The offline expert dataset includes expert data, and the offline unrewarded label dataset contains a small amount of expert data and a large amount of useless interference data. The offline expert dataset and the offline unrewarded label dataset are learned through the dynamics model, and the data in the offline expert dataset and the offline unrewarded label dataset are learned and amplified, so there are expert data, useless interference data and some false data in the dynamics model dataset. Learning through the dynamics model increases the amount of expert data, reduces the cost of data acquisition, and helps to make the subsequent training of the driving imitation strategy model more accurate. The offline expert data and the offline unrewarded label dataset are called the real dataset, and the offline unrewarded label dataset and the dynamics model dataset are collectively referred to as the suboptimal dataset.

[0068] The offline expert data set, offline unrewarded label data set and dynamic model data set are input into the first discriminator to discriminate the expert data and output the degree of closeness between the data and the expert data, including:

[0069] S2011, the first discriminator judges the input offline expert data set, offline unrewarded label data set and dynamic model data set, and discriminates the expert data suboptimal data;

[0070] S2012: Increase the weight of expert data and reduce the weight of suboptimal data;

[0071] S2013. Output the degree of closeness between the data and the expert data based on the weight information of the expert data and the weight information of the suboptimal data set.

[0072] refer to Figure 8 In the present invention, the first discriminator uses an expert / suboptimal data discriminator to judge the data in the offline expert data set, the offline reward-free label data set and the dynamic model data set, determine which data belongs to the expert data and which belongs to the suboptimal data, give a higher weight to the expert data, and give a lower weight to the suboptimal data. The degree of closeness between the output data of the expert / suboptimal data discriminator and the expert data helps to select more useful expert data.

[0073] The expert / suboptimal data discriminator and the driving imitation strategy model are trained in adversarial coupling. The expert / suboptimal data discriminator selects expert data with higher weights, so that the driving imitation strategy model learns more accurate expert data. The driving imitation strategy model also provides feedback to the expert / suboptimal data discriminator, so that the subsequent expert / suboptimal data discriminator can more accurately discriminate the expert data.

[0074] The offline expert dataset, offline unrewarded label dataset and dynamic model dataset are input into the second discriminator to discriminate the real data and output the degree of closeness between the data and the real data, specifically including:

[0075] S2021, the second discriminator judges the input offline expert data set, offline unrewarded label data set and dynamic model data set to distinguish real data from false data;

[0076] S2022. Increase the weight of the identified real data and reduce the weight of the identified false data;

[0077] S2023. Output the degree of closeness between the real data and the weight information of the false data set and the real data.

[0078] In the present invention, the second discriminator adopts the Real-Sim data discriminator to judge the data in the offline expert data set, the offline unrewarded label data set and the dynamic model data set, determine which data are real data and which are false data, give higher weights to real data and lower weights to false data, and the degree of closeness between the output data of the Real-Sim data discriminator and the real data helps to select more useful real data.

[0079] The Real-Sim data discriminator and the dynamics model are trained in adversarial coupling. The Real-Sim data discriminator selects real data with higher weights, allowing the dynamics model to learn more real data. The dynamics model also provides feedback to the Real-Sim data discriminator, allowing the subsequent Real-Sim data discriminator to more accurately discriminate real data.

[0080] In the present invention, an expert / suboptimal data discriminator and a Real-Sim data discriminator are set to perform comprehensive judgment on the data in the offline expert data set, the offline unrewarded label data set and the dynamic model data set, and the real expert data is selected based on the degree of proximity between the output data and the expert data and the degree of proximity between the output data and the real data, thereby increasing the amount of expert data.

[0081] The loss function is constructed by the proximity between the output data and the expert data and the proximity between the output data and the real data to train the driving imitation strategy model and learn the expert data, specifically including:

[0082] S2031, adding the degree of proximity between the output data and the expert data, the degree of proximity between the output data and the real data, and the loss function of the imitation strategy itself to obtain a total loss function;

[0083] S2032, training the driving imitation strategy model through a total loss function, correcting the imitation learning strategy, and generating a corrected driving imitation strategy model;

[0084] S2033. The modified driving imitation strategy model performs imitation learning on real expert data to imitate the functions corresponding to the expert data set.

[0085] The expert data is expanded, the cost of acquiring expert data is reduced, and the driving imitation strategy model imitates the expert data more closely.

[0086] The imitation strategy training process is derived in the form of min max formula:

[0087]

[0088]

[0089]

[0090]

[0091]

[0092]

[0093] in:

[0094] α1: imitation strategy loss weight;

[0095] β1: loss correction coefficient introduced by the first discriminator;

[0096] β2: loss correction coefficient introduced by the second discriminator;

[0097] D: full dataset; De: expert dataset; Do: offline dataset;

[0098] Df: Dataset generated by the dynamics model; d1: Discriminator for determining whether it is a real dataset; d2: Discriminator for determining whether it is an expert dataset;

[0099] d1 loss function; d2 loss function; π: imitation strategy

[0100] F: Kinetic model; L f : dynamics model loss; η: coefficient of the unlabeled learning method;

[0101] g: weight function; s: state; a: action.

[0102] Through specific experiments, reference Fig. 9 and Fig.10 ,In the Hopper-expert environment, Figure 9 shows the learning curve of the original imitation learning method, Fig.10 This is the learning curve of the dual biased imitation learning method disclosed in the present invention.

[0103] The present invention provides a dual biased imitation learning method supported by a dynamics model. The dynamics model is introduced and the data generated by the dynamics model is used to train the imitation strategy, thereby making up for the shortage of sample size of the expert data set and, to a certain extent, alleviating the compound error that may be caused by traditional imitation learning when the limited expert data set cannot cover the entire state space. Biased imitation learning is achieved by introducing two discriminators, and different weights are calculated for samples from different data sets, so that the imitation strategy is more sensitive to samples without reward labels or samples generated by the dynamics model that are close to the expert data set, thereby making the imitation strategy more robust and having better performance.

[0104] refer to Figure 7 The present application also discloses a dual biased imitation learning system supported by a dynamics model, the system comprising:

[0105] A data acquisition module, which acquires a training sample set required for autonomous driving training, wherein the training sample includes an offline expert data set, an offline unrewarded label data set, and a dynamic model data set;

[0106] An imitation learning module constructs a loss function based on the proximity between the preset first discriminator output data and the expert data and the preset second discriminator output data and the real data to train the driving imitation strategy model;

[0107] The training module is used to perform imitation training on the autonomous driving vehicle through the trained driving imitation strategy model.

[0108] The offline expert data set acquired by the data acquisition module 110 includes expert data, and the offline non-reward label data set contains a small amount of expert data and a large amount of useless interference data. The offline expert data set and the offline non-reward label data set are learned through the dynamic model, and the data in the offline expert data set and the offline non-reward label data set are learned and amplified, so there are expert data, useless interference data and some false data in the dynamic model data set. Learning through the dynamic model increases the amount of expert data, reduces the cost of data acquisition, and helps to make the subsequent training of the driving imitation strategy model more accurate. The offline expert data and the offline non-reward label data set are called real data sets, and the offline non-reward label data set and the dynamic model data set are collectively referred to as suboptimal data sets.

[0109] The imitation learning module judges the input offline expert data set, offline unrewarded label data set and dynamic model data set to identify the suboptimal data of the expert data;

[0110] Increase the weight of expert data and reduce the weight of suboptimal data;

[0111] The closeness of the output data to the expert data is determined based on the weight information of the expert data and the weight information of the suboptimal dataset.

[0112] In the present invention, the first discriminator adopts an expert / suboptimal data discriminator to judge the data in the offline expert data set, the offline reward-free label data set and the dynamic model data set, determine which data belong to expert data and which belong to suboptimal data, assign a higher weight to the expert data and a lower weight to the suboptimal data, and the degree of closeness between the output data of the expert / suboptimal data discriminator and the expert data helps to select more useful expert data.

[0113] The expert / suboptimal data discriminator and the driving imitation strategy model are trained in adversarial coupling. The expert / suboptimal data discriminator selects expert data with higher weights, so that the driving imitation strategy model learns more accurate expert data. The driving imitation strategy model also provides feedback to the expert / suboptimal data discriminator, so that the subsequent expert / suboptimal data discriminator can more accurately discriminate the expert data.

[0114] The imitation learning module judges the input offline expert dataset, offline unrewarded label dataset and dynamic model dataset to distinguish real data from false data;

[0115] Increase the weight of the identified real data and reduce the weight of the identified false data;

[0116] The degree of closeness between the output data and the real data is determined based on the weight information of the real data and the weight information of the false data set.

[0117] In the present invention, the second discriminator adopts the Real-Sim data discriminator to judge the data in the offline expert data set, the offline unrewarded label data set and the dynamic model data set, determine which data are real data and which are false data, give higher weights to real data and lower weights to false data, and the degree of closeness between the output data of the Real-Sim data discriminator and the real data helps to select more useful real data.

[0118] The Real-Sim data discriminator and the dynamics model are trained in adversarial coupling. The Real-Sim data discriminator selects real data with higher weights, allowing the dynamics model to learn more real data. The dynamics model also provides feedback to the Real-Sim data discriminator, allowing the subsequent Real-Sim data discriminator to more accurately discriminate real data.

[0119] The training module obtains the total loss function by adding the closeness between the output data and the expert data, the closeness between the output data and the real data, and the loss function of the imitation strategy itself;

[0120] The driving imitation strategy model is trained through the total loss function, the imitation learning strategy is corrected, and a corrected driving imitation strategy model is generated;

[0121] The modified driving imitation strategy model performs imitation learning on real expert data to imitate the functions corresponding to the expert data set.

[0122] The present invention provides a dual biased imitation learning system supported by a dynamics model. The dynamics model is introduced and the data generated by the dynamics model is used to train the imitation strategy, thereby making up for the shortage of sample size of the expert data set and, to a certain extent, alleviating the compound error that may be caused by traditional imitation learning when the limited expert data set cannot cover the entire state space. Biased imitation learning is achieved by introducing two discriminators, and different weights are calculated for samples from different data sets, so that the imitation strategy is more sensitive to samples without reward labels or samples generated by the dynamics model that are close to the expert data set, thereby making the imitation strategy more robust and having better performance.

[0123] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute a dual biased imitation learning method supported by a dynamic model, the method comprising: obtaining a training sample set required for autonomous driving training, wherein the training sample comprises an offline expert data set, an offline unrewarded label data set and a dynamic model data set;

[0124] A loss function is constructed by determining the proximity between the output data of the preset first discriminator and the expert data and the proximity between the output data of the preset second discriminator and the real data to train the driving imitation strategy model;

[0125] The trained driving imitation strategy model is used to perform imitation training on the autonomous driving vehicle.

[0126] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0127] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute a dual biased imitation learning method supported by a dynamics model provided by the above methods, the method comprising: obtaining a training sample set required for autonomous driving training, wherein the training sample includes an offline expert data set, an offline unrewarded label data set and a dynamics model data set;

[0128] A loss function is constructed by determining the proximity between the output data of the preset first discriminator and the expert data and the proximity between the output data of the preset second discriminator and the real data to train the driving imitation strategy model;

[0129] The trained driving imitation strategy model is used to perform imitation training on the autonomous driving vehicle.

[0130] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is implemented when executed by a processor to execute a dual biased imitation learning method supported by a dynamics model provided by the above methods, the method comprising: obtaining a training sample set required for autonomous driving training, wherein the training sample comprises an offline expert data set, an offline unrewarded label data set, and a dynamics model data set;

[0131] A loss function is constructed by determining the proximity between the output data of the preset first discriminator and the expert data and the proximity between the output data of the preset second discriminator and the real data to train the driving imitation strategy model;

[0132] The trained driving imitation strategy model is used to perform imitation training on the autonomous driving vehicle.

[0133] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0134] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A dual biased imitation learning method supported by a dynamics model, characterized in that: include: Obtaining a training sample set required for autonomous driving training, wherein the training sample includes an offline expert data set, an offline unrewarded label data set, and a dynamic model data set; A loss function is constructed by determining the proximity between the preset first discriminator output data and the expert data and the preset second discriminator output data and the real data to train the driving imitation strategy model, wherein the offline expert data and the offline unrewarded label data set are referred to as the real data set; Through the trained driving imitation strategy model, the autonomous driving vehicle is trained by imitation; The dynamic model dataset is generated by learning the offline expert dataset and the offline unrewarded label dataset through the dynamic model, and specifically includes: The offline expert dataset and the offline unrewarded label dataset are jointly input into the dynamics model; The dynamic model is learned based on the expert data in the offline expert data set and the unrewarded label data in the offline unrewarded label data set, and data amplification is performed to generate a dynamic model data set; The dynamics model data set includes real expert data and pseudo expert data.

2. The dual biased imitation learning method supported by a dynamic model according to claim 1, characterized in that: The loss function is constructed by determining the proximity between the preset first discriminator output data and the expert data and the preset second discriminator output data and the real data to train the driving imitation strategy model, specifically including: The offline expert data set, the offline unrewarded label data set and the dynamic model data set are input into a first discriminator, the expert data is discriminated, and the degree of closeness between the output data and the expert data is output; Inputting the offline expert data set, the offline unrewarded label data set and the dynamic model data set into a second discriminator, discriminating the real data, and outputting the degree of closeness between the data and the real data; The driving imitation strategy model is trained by constructing a loss function based on the proximity between the output data and the expert data and the proximity between the output data and the real data to learn the expert data.

3. The dual biased imitation learning method supported by a dynamic model according to claim 2, characterized in that: The offline expert data set, the offline unrewarded label data set and the dynamic model data set are input into the first discriminator, the expert data is discriminated, and the degree of closeness between the output data and the expert data is specifically including: The first discriminator judges the input offline expert data set, offline unrewarded label data set and dynamic model data set to identify the expert data suboptimal data; Increase the weight of expert data and reduce the weight of suboptimal data; The closeness between the output data and the expert data is calculated based on the weight information of the expert data and the weight information of the suboptimal data set. Among them, the offline unrewarded label dataset and the dynamics model dataset are collectively referred to as suboptimal datasets.

4. The dual biased imitation learning method supported by a dynamic model according to claim 2, characterized in that: The offline expert data set, the offline unrewarded label data set and the dynamic model data set are input into the second discriminator to discriminate the real data and output the degree of closeness between the data and the real data, specifically including: The second discriminator judges the input offline expert data set, offline unrewarded label data set and dynamic model data set to distinguish real data from false data; Increase the weight of the identified real data and reduce the weight of the identified false data; The degree of closeness between the output data and the real data is determined based on the weight information of the real data and the weight information of the false data set.

5. The dual biased imitation learning method supported by a dynamic model according to claim 4, characterized in that: The second discriminator and the dynamics model are trained in adversarial coupling; The result of the second discriminator is fed back to the dynamics model, so that the dynamics model adjusts its learning strategy and learns more real data.

6. The dual biased imitation learning method supported by a dynamic model according to claim 2, characterized in that: The loss function is constructed by the proximity between the output data and the expert data and the proximity between the output data and the real data to train the driving imitation strategy model and learn the expert data, specifically including: The degree of closeness between the output data and the expert data, the degree of closeness between the output data and the real data, and the loss function of the imitation strategy itself are added to obtain a total loss function; The driving imitation strategy model is trained through the total loss function, the imitation learning strategy is corrected, and a corrected driving imitation strategy model is generated; The modified driving imitation strategy model performs imitation learning on real expert data to imitate the functions corresponding to the expert data set.

7. A dual biased imitation learning system supported by a dynamics model, characterized in that: The system comprises: A data acquisition module, which acquires a training sample set required for autonomous driving training, wherein the training sample includes an offline expert data set, an offline unrewarded label data set, and a dynamic model data set; The imitation learning module constructs a loss function by determining the proximity between the output data of the preset first discriminator and the expert data and the output data of the preset second discriminator and the real data, and trains the driving imitation strategy model, wherein the offline expert data and the offline unrewarded label data set are called the real data set; A training module is used to perform imitation training on the autonomous driving vehicle through the trained driving imitation strategy model; The dynamic model dataset is generated by learning the offline expert dataset and the offline unrewarded label dataset through the dynamic model, and specifically includes: The offline expert dataset and the offline unrewarded label dataset are jointly input into the dynamics model; The dynamic model is learned based on the expert data in the offline expert data set and the unrewarded label data in the offline unrewarded label data set, and data amplification is performed to generate a dynamic model data set; The dynamics model data set includes real expert data and pseudo expert data.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, a dual biased imitation learning method supported by a dynamics model as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements a dual biased imitation learning method supported by a dynamics model as described in any one of claims 1 to 6.

Images