Simulation learning method and device based on mixed quality data and state search, equipment and medium
By using a hybrid quality data and state search method to select high-quality trajectory segments, the problem of high data collection costs and inconsistent data quality in imitation learning is solved, achieving efficient and low-cost imitation learning results.
Patent Information
- Application Number
- CN202510783425.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-12
- Publication Date
- 2025-11-04
AI Technical Summary
In existing imitation learning techniques, the collection of expert data is costly and time-consuming, especially for novel or diverse tasks. Furthermore, the quality of manually collected human operation data is inconsistent, which affects the learning effect.
A hybrid quality data and state search method is used to obtain expert datasets and suboptimal datasets. High-quality trajectory segments are selected by state similarity to construct a training dataset and perform imitation learning.
It reduces the labeling cost of the training dataset, improves the robustness of imitation learning, obtains a high-quality training dataset, and implements a more robust imitation learning strategy.
Smart Images
Figure CN120892809A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of robots and artificial intelligence, and particularly relates to a mimic learning method and device based on mixed quality data and state search, equipment and medium. BACKGROUND
[0002] Mimic learning is a core concept in the fields of robotics, artificial intelligence and cognitive science, which learns complex tasks by imitating the behavior of humans or other intelligent agents, aiming to understand and build intelligent systems with autonomous behavior. This learning method is widely used in industrial automation, household robots and other scenarios. However, the effectiveness of mimic learning is closely related to the quality and diversity of expert data, and the collection of these data is often costly and time-consuming, especially for novel or diverse tasks.
[0003] Therefore, the prior art still needs to be improved and improved. SUMMARY
[0004] The technical problem to be solved by the present application is to provide a mimic learning method and device based on mixed quality data and state search, equipment and medium in view of the deficiencies of the prior art.
[0005] In order to solve the above technical problems, the first aspect of the present application provides a mimic learning method based on mixed quality data and state search, wherein the mimic learning method based on mixed quality data and state search specifically includes:
[0006] Obtaining mixed quality data, wherein the mixed quality data includes an expert data set and a suboptimal data set, the expert data set includes a plurality of expert state-action data pairs, and the suboptimal data set includes a plurality of suboptimal state-action data pairs;
[0007] Obtaining the state similarity of each suboptimal state in the suboptimal data set and the state of the expert data set, searching for a target suboptimal state from the suboptimal data set according to the state similarity corresponding to the suboptimal state, and determining a screening data set according to the target suboptimal state;
[0008] Constructing a training data set according to the screening data set and the expert data set, and performing mimic learning on an initial mimic learning module based on the training data set to obtain a trained mimic learning module.
[0009] The mimic learning method based on mixed quality data and state search, wherein the searching for a target suboptimal state from the suboptimal data set according to the state similarity corresponding to the suboptimal state specifically includes:
[0010] Searching for a candidate suboptimal state with a state similarity greater than a preset similarity threshold in the suboptimal data set;
[0011] selecting a suboptimal state at a previous time point of the candidate suboptimal state in the suboptimal data set, and taking the selected suboptimal state at the previous time point as a target suboptimal state.
[0012] The imitation learning method based on mixed quality data and state search, wherein the determining a screening data set according to the target suboptimal state specifically comprises:
[0013] constructing a set of suboptimal state-action data pairs corresponding to the target suboptimal state as a candidate screening data set;
[0014] obtaining a state similarity between each target suboptimal state in the candidate screening data set and a state in the expert data set;
[0015] removing, from the candidate screening data set, a suboptimal state-action data pair corresponding to a target suboptimal state with a state similarity greater than a preset similarity threshold, to obtain a screening data set.
[0016] The imitation learning method based on mixed quality data and state search, wherein the obtaining a state similarity between each suboptimal state in the suboptimal data set and a state in the expert data set specifically comprises:
[0017] obtaining a suboptimal state code of each suboptimal state in the suboptimal data set and an expert state code of each expert state in the expert data set;
[0018] for each suboptimal state, calculating a code similarity between the suboptimal state code of the suboptimal state and each expert state code, and selecting a maximum code similarity from all the calculated code similarities to obtain a state similarity between the suboptimal state and the state in the expert data set.
[0019] The imitation learning method based on mixed quality data and state search, wherein after the determining a screening data set according to the target suboptimal state, the method further comprises:
[0020] removing the screening data set from the suboptimal data set, and continuing the step of obtaining a state similarity between each suboptimal state in the suboptimal data set and a state in the expert data set until a preset number of screening data sets are obtained.
[0021] The imitation learning method based on mixed quality data and state search, wherein before the step of continuing the step of obtaining a state similarity between each suboptimal state in the suboptimal data set and a state in the expert data set, the method further comprises:
[0022] adding the screening data set to the expert data set.
[0023] The imitation learning method based on mixed quality data and state search, wherein the constructing a training data set according to the screening data set and the expert data set specifically comprises:
[0024] obtaining a union of all screening data sets to obtain a target screening data set;
[0025] obtaining a union of the target screening data set and the expert data set to obtain a training data set.
[0026] The second aspect of the present application provides an imitation learning device based on mixed quality data and state search, wherein the imitation learning device based on mixed quality data and state search specifically comprises:
[0027] an acquisition module configured to acquire mixed quality data, wherein the mixed quality data comprises an expert data set and a suboptimal data set, the expert data set comprises a plurality of expert state-action data pairs, and the suboptimal data set comprises a plurality of suboptimal state-action data pairs;
[0028] a search module configured to acquire a state similarity between each suboptimal state in the suboptimal data set and a state in the expert data set, search a target suboptimal state from the suboptimal data set according to the state similarity corresponding to the suboptimal state, and determine a screening data set according to the target suboptimal state;
[0029] a training module configured to construct a training data set according to the screening data set and the expert data set, and perform imitation learning on an initial imitation learning module based on the training data set to obtain a trained imitation learning module.
[0030] The third aspect of the present application provides a computer readable storage medium storing one or more programs, which can be executed by one or more processors to implement the steps in the imitation learning method based on mixed quality data and state search according to any one of the above.
[0031] The fourth aspect of the present application provides a terminal device comprising a processor and a memory.
[0032] The memory stores a computer readable program executable by the processor;
[0033] The processor implements the steps in the imitation learning method based on mixed quality data and state search according to any one of the above when executing the computer readable program.
[0034] Beneficial effects: Compared with the prior art, the application provides a imitation learning method, device, equipment and medium based on mixed quality data and state search, the method comprises the following steps: acquiring mixed quality data comprising an expert data set and a suboptimal data set, then acquiring the state similarity of each suboptimal state in the suboptimal data set and the expert data set, searching for a target suboptimal state from the suboptimal data set according to the state similarity corresponding to the suboptimal state, and determining a screening data set according to the target suboptimal state; constructing a training data set according to the screening data set and the expert data set, and performing imitation learning on an initial imitation learning module based on the training data set to obtain a trained imitation learning module. According to the similarity of the suboptimal state and the expert state set, the embodiment of the application screens high-quality trajectory segments from the suboptimal data set, and combines the screened high-quality trajectory segments with the expert state set, so that a high-quality training data set can be obtained only by labeling a small amount of expert states, which not only reduces the labeling cost of the training data set, but also learns a more robust imitation learning strategy. BRIEF DESCRIPTION OF DRAWINGS
[0035] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.
[0036] Figure 1 The flow chart of the imitation learning method based on mixed quality data and state search provided by the embodiment of the application.
[0037] Figure 2 The architecture schematic diagram of the imitation learning method based on mixed quality data and state search provided by the embodiment of the application.
[0038] Figure 3 The schematic diagram of the state search process.
[0039] Figure 4 The principle block diagram of the imitation learning device based on mixed quality data and state search provided by the embodiment of the application.
[0040] Figure 5 The principle block diagram of the terminal device provided by the embodiment of the application. DETAILED DESCRIPTION
[0041] The embodiment of the present application provides a kind of based on mixed quality data and state search's imitative learning method, device, equipment and medium, for the purpose, technical scheme and effect of the present application are more clear and explicit, the following is further detailed with reference to the drawings and the embodiment of the present application.It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.
[0042] Those skilled in the art of the present technology can understand that, unless specifically stated, the singular form "a", "an" and "said" used herein can also include the plural form. It should be further understood that the phrase "comprising" used in the specification of the present application means that the features, integers, steps, operations, elements and / or components are present, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or their groups. It should be understood that when we say that an element is "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or there can be intermediate elements. In addition, "connected" or "coupled" used herein can include wireless connection or wireless coupling. The phrase "and / or" used herein includes all or any single unit and all combinations of the associated listed items.
[0043] Those skilled in the art of the present technology can understand that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as that generally understood by those skilled in the art to which the present application belongs. It should also be understood that terms such as those defined in a general dictionary should be understood to have meanings consistent with those in the prior art, and unless specifically defined as here, should not be interpreted in an idealized or overly formal sense.
[0044] It should be understood that the sequence numbers and sizes of the steps in the embodiments do not mean the order of execution, and the execution order of the processes is determined by their functions and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0045] It is found through research that imitation learning is a core concept in the fields of robotics, artificial intelligence, and cognitive science, which learns complex tasks by imitating the behavior of humans or other intelligent agents, aiming to understand and build intelligent systems with autonomous behavior. This learning method is widely used in industrial automation, household robots, and other scenarios. However, the effectiveness of imitation learning is closely related to the quality and diversity of expert data, and the collection of such data is often costly and time-consuming, especially for novel or diverse tasks. Moreover, due to the different operating skills of different operators, manually collected human operation data often exhibits inconsistent behavior, making directly collected human operation data include suboptimal operations such as retrying failed grasping, which affects the data quality of the collected operation data and further affects the learning effect of imitation learning.
[0046] To solve the above problems, in the embodiments of the present application, mixed quality data including an expert data set and a suboptimal data set is obtained, then the similarity of each suboptimal state in the suboptimal data set to the state of the expert data set is obtained, the target suboptimal state is searched from the suboptimal data set according to the state similarity corresponding to the suboptimal state, and the filtered data set is determined according to the target suboptimal state; the training data set is constructed according to the filtered data set and the expert data set, and the initial imitation learning module is subjected to imitation learning based on the training data set to obtain the trained imitation learning module. In the embodiments of the present application, high-quality trajectory segments are filtered from the suboptimal data set according to the similarity of the suboptimal state to the expert state set, and the filtered high-quality trajectory segments are combined with the expert state set, so that a high-quality training data set can be obtained only by labeling a small amount of expert states, which not only reduces the labeling cost of the training data set, but also learns a more robust imitation learning strategy.
[0047] The application content will be further described through the description of the embodiments in combination with the drawings.
[0048] The embodiments provide a kind of imitation learning method based on mixed quality data and state search, as shown in Figure 1 And Figure 2 The imitation learning method based on mixed quality data and state search specifically includes:
[0049] S10, obtaining mixed quality data.
[0050] Specifically, the mixed quality data is an initial data set collected for imitation learning, the mixed quality data includes an expert data set and a suboptimal data set, the expert data set is an expert demonstration and a data set detected for correctness, and the suboptimal data set is a manually collected data set, the suboptimal data set can include a trajectory segment of suboptimal operation, that is, the expert data set includes a plurality of expert state-action data pairs, the suboptimal data set includes a plurality of suboptimal state-action data pairs, and the data quality of the expert data set is higher than that of the suboptimal data set.
[0051] S20, obtaining a state similarity between each suboptimal state in the suboptimal data set and the expert data set, searching for a target suboptimal state from the suboptimal data set according to the state similarity corresponding to the suboptimal state, and determining a screening data set according to the target suboptimal state.
[0052] Specifically, the state similarity is determined based on the similarity between the suboptimal state and each expert state in the expert data set, and the similarity is used to measure the data quality similarity between two state data, that is, the similarity between the suboptimal state and the expert state is used to measure the data quality similarity between the suboptimal state and the expert state, and the higher the similarity, the more similar the data quality between the state data, and vice versa. The lower the similarity, the less similar the data quality between the state data. Since the expert data set is a high-quality data set, the higher the similarity between the suboptimal state and the expert state, the higher the data quality of the suboptimal state, and vice versa. The lower the similarity between the suboptimal state and the expert state, the lower the data quality of the suboptimal state.
[0053] Based on this, obtaining a state similarity between each suboptimal state in the suboptimal data set and the expert data set specifically includes:
[0054] Obtaining a suboptimal state code of each suboptimal state in the suboptimal data set and an expert state code of each expert state in the expert data set;
[0055] For each suboptimal state, calculating a code similarity between the suboptimal state code of the suboptimal state and each expert state code, and selecting the maximum code similarity from all calculated code similarities to obtain the state similarity between the suboptimal state and the expert data set.
[0056] In particular, the suboptimal state encoding is obtained by encoding the suboptimal state through a pre-trained encoder, and the expert state encoding is obtained by encoding the expert state through the encoder. The training process of the encoder can include first obtaining an initial world model, the initial world model including an encoder, a decoder and a dynamics prediction module, the encoder being used to encode the state to obtain state encoding, the decoder being used to reconstruct the state data based on the state encoding, and the dynamics prediction module being used to predict the current state based on the state encoding of the previous moment and the previous action of the previous moment. Wherein, the encoder, the decoder and the dynamics prediction module can be represented as:
[0057] z t =q φ (s t ),
[0058]
[0059] Wherein, q φ represents the encoder, p φ represents the decoder, d φ represents the dynamics prediction module, z t represents the state encoding at t moment, s t represents the state at t moment, represents the reconstructed state at t moment, represents the predicted state encoding at t moment, z t-1 represents the state at t-1 moment, a t-1 represents the action corresponding to the state at t-1 moment, and φ represents the model parameters of the initial world model.
[0060] Further, the loss function used by the initial world model can be:
[0061]
[0062] L dyn (φ)=mse(s t ,p φ (z t )),
[0063] Wherein, L wm (φ) represents the loss function, L recon (φ) represents the reconstruction loss term, and L dyn (φ) represents the prediction loss term.
[0064] Further, after obtaining the suboptimal state encoding of each suboptimal state and the expert state encoding of each expert state, as Figure 3As shown, the encoding similarity of each suboptimal state code and each expert state code can be calculated respectively, where the encoding similarity can be the L2 distance between the suboptimal state code and the expert state code, or the cosine similarity between the suboptimal state code and the expert state code, etc. In the embodiment of the present application, the calculation formula of the encoding similarity can be:
[0065] S(s o ,s e )=-||z o -z e ||2=||q φ (s o )-q φ (s e )||2。
[0066] Where S(s o ,s e ) represents the encoding similarity, s o represents the suboptimal state, s e represents the expert state, z o represents the suboptimal state code, and z e represents the expert state code.
[0067] After obtaining the encoding similarity of each suboptimal state code and each expert state code, all the obtained encoding similarities are divided according to the suboptimal state to obtain the encoding similarity group corresponding to each suboptimal state, which includes the encoding similarity of the suboptimal state and each expert state in the expert data set. Then, the maximum encoding similarity in the encoding similarity group can be selected, and then the state similarity of the suboptimal state and the expert data set is determined according to the maximum encoding similarity. Specifically, the maximum encoding similarity can be directly used as the state similarity of the suboptimal state and the expert data set, or the maximum encoding similarity can be normalized first, and the normalized maximum encoding similarity can be used as the state similarity of the suboptimal state and the expert data set, etc.
[0068] In one embodiment, after obtaining the maximum encoding similarity, the maximum value in the maximum encoding similarity corresponding to each suboptimal state can be selected as the upper limit encoding similarity, the minimum value in the maximum encoding similarity corresponding to each suboptimal state can be selected as the lower limit encoding similarity, and finally the state similarity corresponding to each suboptimal state is calculated based on the upper limit encoding similarity, the maximum encoding similarity corresponding to each suboptimal state, and the lower limit encoding similarity, so that the difference between the state similarities of the suboptimal states can be reduced, and the accuracy of the state data screening can be improved. Wherein, the calculation formula of the state similarity can be:
[0069]
[0070] wherein, denotes a state similarity, denotes an expert dataset, denotes a lower bound encoding similarity of
[0071] denotes an upper bound encoding similarity of denotes a maximum encoding similarity corresponding to a suboptimal state, denotes a suboptimal dataset.
[0072] In one embodiment, the searching for a target suboptimal state from the suboptimal dataset according to the state similarity corresponding to the suboptimal state specifically comprises:
[0073] searching for a candidate suboptimal state in the suboptimal dataset, the candidate suboptimal state having a state similarity greater than a preset similarity threshold;
[0074] selecting a previous time point suboptimal state of the candidate suboptimal state in the suboptimal dataset, and taking the selected previous time point suboptimal state as the target suboptimal state.
[0075] Specifically, after obtaining the state similarity corresponding to each suboptimal state, the state similarity corresponding to the suboptimal state can be compared with a preset similarity threshold to search for a candidate suboptimal state in the suboptimal dataset, the candidate suboptimal state having a state similarity greater than the preset similarity threshold, wherein the preset similarity threshold is preset. Of course, in actual application, other ways can also be used to obtain the candidate suboptimal state, such as selecting a preset number of candidate suboptimal states in order of state similarity from large to small.
[0076] After obtaining the candidate suboptimal state, a trajectory segment capable of reaching the candidate suboptimal state can be iteratively searched from the candidate suboptimal state as a starting point, that is, a trajectory segment (i.e., a suboptimal state-action data pair) in the expert trajectory that reaches the candidate suboptimal state is searched in the suboptimal dataset. Among them, the trajectory segment that reaches the candidate suboptimal state is generally the trajectory segment of the previous time point of the candidate suboptimal state, so that the previous time point suboptimal state of the candidate suboptimal state in the suboptimal dataset can be selected, and the selected target suboptimal state is obtained.
[0077] The present application takes the suboptimal state in the trajectory segment reaching the candidate suboptimal state as the target suboptimal state, so that the trajectory segment reaching the candidate suboptimal state can be added to the screening dataset, improving the data quality of the training dataset. Of course, in actual application, the trajectory segment corresponding to the candidate suboptimal state, or the trajectory segments of the previous two time points of the candidate suboptimal state, etc. can also be selected according to requirements.
[0078] In one embodiment, the determining the screening data set according to the target suboptimal state specifically comprises:
[0079] forming a set of suboptimal state-action data pairs corresponding to the target suboptimal state as a candidate screening data set;
[0080] obtaining a state similarity between each target suboptimal state in the candidate screening data set and the state of the expert data set;
[0081] removing the suboptimal state-action data pair corresponding to the target suboptimal state with a state similarity greater than a preset similarity threshold from the candidate screening data set to obtain the screening data set.
[0082] Specifically, after obtaining the target suboptimal state, the suboptimal state-action data pair (i.e., trajectory segment) corresponding to each target suboptimal state can be read from the suboptimal data set, and then all the read state-action data pairs form a set as a candidate screening data set, that is, the candidate screening data set includes part of the suboptimal state-action data pairs in the suboptimal data set. Wherein, the candidate screening data set can be represented as:
[0083]
[0084] Wherein, represents the candidate screening data set, represents the suboptimal state of t-1, represents the suboptimal state of t, and δ represents the preset similarity threshold.
[0085] The calculation method of the state similarity between each target suboptimal state in the candidate screening data set and the state of the expert data set is the same as the calculation method of the state similarity between the suboptimal state and the state of the expert data set, which is not specifically described here. Only the calculation formula of the state similarity between each target suboptimal state in the candidate screening data set and the state of the expert data set is given, which is:
[0086]
[0087] Wherein, represents the upper limit encoding similarity of represents the lower limit encoding similarity of represents the candidate screening data set, s u represents the target suboptimal state in the candidate screening data set.
[0088] Further, after obtaining the state similarity between each target suboptimal state and the state of the expert dataset, the suboptimal state-action data pair corresponding to the target suboptimal state with a state similarity greater than a preset similarity threshold is removed from the candidate screening dataset to obtain a screening dataset. The screening dataset can be represented as:
[0089]
[0090] In an embodiment, after obtaining the screening dataset, it can be detected whether sufficient screening datasets are searched. When sufficient screening datasets are searched, a training dataset is constructed according to the screening dataset and the expert dataset. When insufficient screening datasets are searched, the next round of search is performed until sufficient screening datasets are searched. Based on this, after the screening dataset is determined according to the target suboptimal state, the method further includes:
[0091] The screening dataset is removed from the suboptimal dataset, and the step of obtaining the state similarity between each suboptimal state in the suboptimal dataset and the state of the expert dataset is continued until a preset number of screening datasets are obtained.
[0092] Specifically, removing the screening dataset from the suboptimal dataset can avoid the influence of the screened suboptimal state-action data pair on the current screening process, so that the current screening can screen the suboptimal state-action data pair that has not been screened. The preset number is a basis for determining whether sufficient screening datasets are searched, which can be pre-set, for example, 5, 10, etc. In addition, in actual application, whether sufficient screening datasets are searched can also be determined according to the required training data amount. Specifically, after obtaining the screening dataset, the data amount of the union of the screening dataset and the expert dataset can be read. If the data amount is greater than or equal to the required training data amount, it means that sufficient screening datasets are searched. Otherwise, if the data amount is less than the required training data amount, it means that insufficient screening datasets are searched.
[0093] In an implementation, in order to better search the suboptimal dataset, while the screening dataset is removed from the suboptimal dataset, the suboptimal state-action data pair in the screening dataset can be added to the expert dataset to expand the expert dataset. In this way, the next round of search (i.e., the step of obtaining the state similarity between each suboptimal state in the suboptimal dataset and the state of the expert dataset) can search more high-quality trajectory segments, improve the search speed of the trajectory segments, reduce the time cost and resource cost required for searching the trajectory segments, and thus the learning cost of the imitation learning can be reduced.
[0094] S30, constructing a training dataset according to the screening dataset and the expert dataset, and performing imitation learning on the initial imitation learning module based on the training dataset to obtain a trained imitation learning module.
[0095] Specifically, the training dataset is constructed according to the screening dataset and the expert dataset, for example, data in the screening dataset can be added to the expert dataset, or the union of the screening dataset and the expert dataset is taken as the training dataset, etc.
[0096] Specifically, the training dataset is constructed according to the screening dataset and the expert dataset, for example, data in the screening dataset can be added to the expert dataset, or the union of the screening dataset and the expert dataset is taken as the training dataset, etc.
[0097] Obtaining the union of all screening datasets to obtain a target screening dataset;
[0098] Obtaining the union of the target screening dataset and the expert dataset to obtain a training dataset.
[0099] Specifically, all screening datasets include screening datasets searched in each search round, and each screening dataset searched in each search round includes a suboptimal state-action data pair corresponding to a target suboptimal state searched in the search round. To this end, when constructing the training dataset, all screening datasets are first combined to obtain a target screening dataset, wherein the target screening dataset is the union of all screening datasets, so as to reduce the repeated data in the target screening dataset. Then, the target screening dataset and the expert dataset are combined to generate the training dataset.
[0100] After obtaining the training dataset, the initial imitation learning module is trained using the training dataset to obtain a trained imitation learning module, wherein the imitation learning module is obtained based on the state-action pairs in the training dataset, and can be used to control a robot or a mechanical arm, etc. For example, it can control a biped robot to walk, control a mechanical arm to complete a preset placement task or perform a grabbing operation, etc.
[0101] The optimization target used in the training process of the imitation learning module can be:
[0102]
[0103] Wherein, s represents the state, a represents the action, θ represents the model parameter of the imitation learning module, and π θ represents the imitation strategy of the imitation learning module.
[0104] To sum up, the embodiment provides a imitation learning method based on mixed quality data and state search, which comprises: acquiring mixed quality data comprising an expert data set and a suboptimal data set; acquiring state similarity of each suboptimal state in the suboptimal data set to the expert data set; searching a target suboptimal state from the suboptimal data set according to the state similarity corresponding to the suboptimal state; and determining a screening data set according to the target suboptimal state; constructing a training data set according to the screening data set and the expert data set; and performing imitation learning on an initial imitation learning module based on the training data set to obtain a trained imitation learning module. According to the embodiment, high-quality trajectory segments are screened from the suboptimal data set according to the similarity of the suboptimal state to the expert state set, and the screened high-quality trajectory segments are combined with the expert state set, so that a high-quality training data set can be acquired only by labeling a small amount of expert states, which not only reduces the labeling cost of the training data set, but also learns a more robust imitation learning strategy.
[0105] Based on the above imitation learning method based on mixed quality data and state search, the embodiment provides an imitation learning device based on mixed quality data and state search, as shown in Figure 4 The imitation learning device based on mixed quality data and state search specifically comprises:
[0106] An acquisition module 100 is configured to acquire mixed quality data, wherein the mixed quality data comprises an expert data set and a suboptimal data set, the expert data set comprises a plurality of expert state-action data pairs, and the suboptimal data set comprises a plurality of suboptimal state-action data pairs.
[0107] A search module 200 is configured to acquire state similarity of each suboptimal state in the suboptimal data set to the expert data set, search a target suboptimal state from the suboptimal data set according to the state similarity corresponding to the suboptimal state, and determine a screening data set according to the target suboptimal state.
[0108] A training module 300 is configured to construct a training data set according to the screening data set and the expert data set, and perform imitation learning on an initial imitation learning module based on the training data set to obtain a trained imitation learning module.
[0109] Based on the above imitation learning method based on mixed quality data and state search, the embodiment provides a computer readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the imitation learning method based on mixed quality data and state search.
[0110] Based on the above-mentioned imitation learning method based on mixed quality data and state search, the application further provides a terminal device, as shown in Figure 5 The terminal device includes at least one processor 20, a display screen 21, and a memory 22, and can further include a communications interface 23 and a bus 24. The processor 20, the display screen 21, the memory 22, and the communications interface 23 can communicate with each other through the bus 24. The display screen 21 is configured to display a preset user guide interface in an initial setting mode. The communications interface 23 can transmit information. The processor 20 can call logical instructions in the memory 22 to execute the method in the above-mentioned embodiments.
[0111] In addition, when the logical instructions in the above-mentioned memory 22 are implemented in the form of a software function unit and sold or used as an independent product, the logical instructions can be stored in a computer-readable storage medium.
[0112] The memory 22, as a computer-readable storage medium, can be configured to store software programs, computer executable programs, such as program instructions or modules corresponding to the method in the embodiments of the present disclosure. The processor 20 executes the functions of the application and data processing by running the software programs, instructions, or modules stored in the memory 22, that is, implements the method in the above-mentioned embodiments.
[0113] The memory 22 can include a program storage area and a data storage area. The program storage area can store an operating system and at least one application required by a function. The data storage area can store data created during use of the terminal device. In addition, the memory 22 can include a high-speed random access memory and can further include a non-volatile memory. For example, the memory can be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and the like, which can store program codes, and can also be a transitory storage medium.
[0114] In addition, the specific processes of the above-mentioned storage medium and the terminal device in which the processor loads and executes a plurality of instructions have been described in detail in the above-mentioned method, and will not be repeated here.
[0115] Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the same; although the present application has been described in detail with reference to the foregoing examples, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. An imitation learning method based on hybrid quality data and state search, characterized in that, The imitation learning method based on mixed quality data and state search specifically includes: Acquire mixed-quality data, wherein the mixed-quality data includes an expert dataset and a suboptimal dataset, the expert dataset includes several expert state-action data pairs, and the suboptimal dataset includes several suboptimal state-action data pairs; Obtain the similarity between each suboptimal state in the suboptimal dataset and the state in the expert dataset, search for a target suboptimal state from the suboptimal dataset based on the state similarity corresponding to the suboptimal state, and determine the filtered dataset based on the target suboptimal state. A training dataset is constructed based on the selected dataset and the expert dataset, and the initial imitation learning module is subjected to imitation learning based on the training dataset to obtain a trained imitation learning module.
2. The imitation learning method based on hybrid quality data and state search according to claim 1, characterized in that, The step of searching for the target suboptimal state from the suboptimal dataset based on the state similarity corresponding to the suboptimal state specifically includes: Search for candidate suboptimal states in the suboptimal dataset whose state similarity is greater than a preset similarity threshold; Select the suboptimal state of the previous moment from the candidate suboptimal state in the suboptimal dataset, and take the selected suboptimal state of the previous moment as the target suboptimal state.
3. The imitation learning method based on hybrid quality data and state search according to claim 1, characterized in that, The step of determining the selected dataset based on the target suboptimal state specifically includes: The set of suboptimal state-action data pairs corresponding to the target suboptimal state is used as the candidate screening dataset; Obtain the similarity between each suboptimal state of a target in the candidate screening dataset and the state in the expert dataset; The suboptimal state-action data pairs corresponding to the target suboptimal state with a state similarity greater than a preset similarity threshold are removed from the candidate filtering dataset to obtain the filtering dataset.
4. The imitation learning method based on hybrid quality data and state search according to claim 1, characterized in that, The step of obtaining the similarity between each suboptimal state in the suboptimal dataset and the state in the expert dataset specifically includes: Obtain the suboptimal state code for each suboptimal state in the suboptimal dataset and the expert state code for each expert state in the expert dataset. For each suboptimal state, calculate the coding similarity between the suboptimal state code and the coding of each expert state. Select the maximum coding similarity among all the calculated coding similarities to obtain the state similarity between the suboptimal state and the expert dataset.
5. The imitation learning method based on hybrid quality data and state search according to claim 1, characterized in that, After determining the selected dataset based on the target suboptimal state, the method further includes: The process involves removing the filtered dataset from the suboptimal dataset and continuing to obtain the similarity between each suboptimal state in the suboptimal dataset and the state in the expert dataset, until a preset number of filtered datasets are obtained.
6. The imitation learning method based on hybrid quality data and state search according to claim 5, characterized in that, Before the step of obtaining the similarity between each suboptimal state in the suboptimal dataset and the state in the expert dataset, the method further includes: Add the filtered dataset to the expert dataset.
7. The imitation learning method based on hybrid quality data and state search according to claim 1, characterized in that, The step of constructing the training dataset based on the selected dataset and the expert dataset specifically includes: Obtain the union of all filtered datasets to get the target filtered dataset; Obtain the union of the target screening dataset and the expert dataset to obtain the training dataset.
8. An imitation learning device based on hybrid quality data and state search, characterized in that, The imitation learning device based on hybrid quality data and state search specifically includes: An acquisition module is used to acquire mixed-quality data, wherein the mixed-quality data includes an expert dataset and a suboptimal dataset, the expert dataset includes several expert state-action data pairs, and the suboptimal dataset includes several suboptimal state-action data pairs; The search module is used to obtain the similarity between each suboptimal state in the suboptimal dataset and the state in the expert dataset, search for a target suboptimal state from the suboptimal dataset based on the state similarity corresponding to the suboptimal state, and determine the filter dataset based on the target suboptimal state. The training module is used to construct a training dataset based on the selected dataset and the expert dataset, and to perform imitation learning on the initial imitation learning module based on the training dataset to obtain a trained imitation learning module.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs that can be executed by one or more processors to implement the steps in the imitation learning method based on mixed quality data and state search as described in any one of claims 1-7.
10. A terminal device, characterized in that, include: Processor and memory; The memory stores a computer-readable program that can be executed by the processor; When the processor executes the computer-readable program, it implements the steps of the imitation learning method based on mixed quality data and state search as described in any one of claims 1-7.