Simulation learning method and device based on synthetic trajectory enhancement, equipment and medium

Abstract

The invention discloses an imitation learning method and device based on synthetic trajectory enhancement, equipment and a medium, and the method comprises the steps: obtaining an original data set, training a discriminator and a world model through the original data set, selecting a third number of state data from the original data set, generating a synthetic data set according to the selected state data by utilizing the world model, wherein synthetic data in the synthetic data set comprises state data, actions and next state data; constructing a training data set according to the original data set and the synthetic data set, and adding an award for each piece of training data in the training data set by using the discriminator; and performing imitation learning on an initial training strategy based on the training data set to obtain a trained strategy network. According to the method, the discriminator and the world model are trained by using the original data, then the synthetic data are constructed by using the world model, and finally the initial strategy network model is subjected to imitation learning by using the original data and the synthetic data, so that the suboptimal data can be utilized, and the learning effect of imitation learning is improved; and the task execution effect of the strategy network obtained based on imitation learning can be improved.

Description

Technical Field

[0001] This application relates to the fields of robotics and artificial intelligence, and in particular to an imitation learning method, apparatus, device, and medium based on synthetic trajectory enhancement. Background Technology

[0002] Imitation learning is an important concept in robotics, artificial intelligence, and cognitive science. It aims to learn complex tasks by mimicking the behavior of humans or other intelligent agents, enabling the understanding and construction of intelligent systems with autonomous capabilities. Imitation learning can be applied to a wide range of scenarios, such as industrial automation and home robots. However, the effectiveness of imitation learning is closely related to the quality and diversity of expert data. Due to varying operational skills, manually collected data often exhibits inconsistent behavior, making it difficult to obtain reliable expert demonstrations. Furthermore, demonstrations may include suboptimal actions, such as retrying failed grasping maneuvers. Therefore, directly applying behavioral clones to this data may result in the replication of undesirable failures, affecting the learning outcome of imitation learning and consequently impacting the agent's task performance.

[0003] Therefore, existing technologies still need to be improved and enhanced. Summary of the Invention

[0004] The technical problem to be solved by this application is to provide an imitation learning method, apparatus, device and medium based on synthetic trajectory enhancement, which addresses the shortcomings of the existing technology.

[0005] To address the aforementioned technical problems, the first aspect of this application provides an imitation learning method based on synthetic trajectory enhancement, wherein the imitation learning method based on synthetic trajectory enhancement specifically includes: Obtain the original dataset and train the discriminator and world model using the original dataset, wherein the original dataset includes a first number of expert data and a second number of suboptimal data, and both the expert data and the suboptimal data include state data, actions and next state data; A third number of state data are selected from the original dataset, and a synthetic dataset is generated based on the selected state data using the world model. The synthetic data in the synthetic dataset includes state data, actions, and next state data. A training dataset is constructed based on the original dataset and the synthetic dataset, and a reward is added to each training data point in the training dataset using the discriminator; Based on the training dataset, the initial training policy is subjected to imitation learning to obtain a trained policy network.

[0006] The aforementioned imitation learning method based on synthetic trajectory enhancement, wherein the training process of the discriminator specifically includes: The state data in the original dataset is input into the initial discriminator, and the initial discriminator outputs the expert state discrimination value corresponding to the state data. The initial discriminator is trained based on the expert state discriminant value to obtain a trained discriminator.

[0007] The imitation learning method based on synthetic trajectory enhancement, wherein the training process of the world model specifically includes: Select an original trajectory from the original dataset, the original trajectory including several original data; The state data from each raw data point in the original trajectory is input into the world model, and the state code is determined by the world model. Based on the state encoding, the reconstructed state data is determined using the world model; Based on the state code and the action, the predicted state code for the next state data is determined using the world model; The world model is trained based on all state data, all predicted state codes, and all reconstructed state data.

[0008] The aforementioned imitation learning method based on synthetic trajectory enhancement, wherein generating a synthetic dataset using the world model based on selected state data specifically includes: For each selected state data, the selected state data is used as the initial state data; The initial state data is input into the world model, and the initial state data is encoded by the world model to obtain the state code; The state code is input into the exploration noise action model, and the exploration noise action model generates actions for the initial state data. The world model determines the next state code based on the initial state data and the action, and determines the next state data based on the next state code. Using the next state data as the initial state, the step of inputting the initial state data into the world model is repeated until a synthetic trajectory of a preset length is generated for the selected state data; The synthetic dataset is determined based on the synthetic trajectory corresponding to each selected state data.

[0009] The aforementioned imitation learning method based on synthetic trajectory enhancement, wherein adding a reward to each training data point in the training dataset using the discriminator specifically includes: The next state data from the training data in the training dataset is input into the discriminator, and the discriminator outputs the expert state discrimination value corresponding to the next state data. If the expert state judgment value is greater than a preset threshold, the expert state judgment value will be used as the reward for the training data. If the expert state judgment value is less than or equal to a preset threshold, the preset value is used as the reward for the training data.

[0010] The aforementioned imitation learning method based on synthetic trajectory enhancement, wherein the step of performing imitation learning on the initial training policy based on the training dataset to obtain a trained policy network specifically includes: A first training batch is randomly selected from the training dataset, and a second training batch is randomly selected from the expert data in the training dataset. Implicit Q-learning is performed on the initial training policy based on the first training batch to obtain the loss function of the state value function, the loss function of the state-action value function, and the policy network loss function. Implicit Q-learning is performed on the initial training strategy based on the second training batch to obtain the cloning loss function; An imitation learning loss function is constructed based on the policy network loss function and the cloning loss function. The initial training policy is then subjected to imitation learning using the imitation learning loss function, the loss function of the state value function, and the loss function of the state-action value function to obtain a trained policy network.

[0011] The aforementioned imitation learning method based on synthetic trajectory enhancement, wherein after performing imitation learning on the initial training policy based on the training dataset to obtain a trained policy network, the method further includes: The policy network is deployed on intelligent mechanical devices so that the intelligent mechanical devices perform tasks based on the policy network.

[0012] A second aspect of this application provides an imitation learning device based on synthetic trajectory enhancement, wherein the imitation learning device based on synthetic trajectory enhancement specifically includes: An acquisition module is used to acquire a raw dataset and train a discriminator and a world model using the raw dataset. The raw dataset includes a first number of expert data and a second number of suboptimal data. Both the expert data and the suboptimal data include state data, actions, and next state data. The synthesis module is used to select a third number of state data from the original dataset and generate a synthetic dataset based on the selected state data using the world model. The synthetic data in the synthetic dataset includes state data, actions, and next state data. A construction module is used to construct a training dataset based on the original dataset and the synthetic dataset, and to add a reward to each training data point in the training dataset using the discriminator; The training module is used to perform imitation learning on the initial training policy based on the training dataset to obtain a trained policy network.

[0013] A third aspect of this application provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps in the synthetic trajectory-enhanced imitation learning method as described above.

[0014] A fourth aspect of this application provides a terminal device, which includes: a processor and a memory; The memory stores a computer-readable program that can be executed by the processor; When the processor executes the computer-readable program, it implements the steps in any of the above-described synthetic trajectory-enhanced imitation learning methods.

[0015] Beneficial Effects: Compared with existing technologies, this application provides a method, apparatus, device, and medium for imitation learning based on synthetic trajectory enhancement. The method includes acquiring an original dataset and training a discriminator and a world model using the original dataset; selecting a third number of state data points from the original dataset; generating a synthetic dataset using the world model based on the selected state data; the synthetic data in the synthetic dataset includes state data, actions, and next state data; constructing a training dataset based on the original dataset and the synthetic dataset; adding a reward to each training data point in the training dataset using the discriminator; and performing imitation learning on an initial training policy based on the training dataset to obtain a trained policy network. This application utilizes the original data to train the discriminator and the world model, then uses the world model to construct synthetic data, and finally uses the original data and synthetic data to perform imitation learning on the initial policy network model. This allows for the utilization of suboptimal data, improves the learning effect of imitation learning, and consequently improves the task performance of the policy network obtained based on imitation learning. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating the imitation learning method based on synthetic trajectory enhancement provided in this application embodiment.

[0018] Figure 2A flowchart illustrating the principle of the imitation learning method based on synthetic trajectory enhancement provided in this application embodiment.

[0019] Figure 3 This is a schematic diagram of the synthesized trajectory.

[0020] Figure 4 This is a schematic diagram of the imitation learning device based on synthetic trajectory enhancement provided in the embodiments of this application.

[0021] Figure 5 A schematic block diagram of the terminal device provided in the embodiments of this application. Detailed Implementation

[0022] This application provides a method, apparatus, device, and medium for imitation learning based on synthetic trajectory enhancement. To make the objectives, technical solutions, and effects of this application clearer and more explicit, the following detailed description, with reference to the accompanying drawings and embodiments, further illustrates this application. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit this application.

[0023] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0024] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0025] It should be understood that the sequence number and size of each step in this embodiment do not imply the order of execution. The execution order of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application embodiment.

[0026] The application content will be further explained below with reference to the accompanying drawings and the description of the embodiments.

[0027] This embodiment provides an imitation learning method based on synthetic trajectory enhancement, such as... Figure 1 and Figure 2 As shown, the method includes: S10. Obtain the original dataset and use the original dataset to train the discriminator and the world model.

[0028] Specifically, the original dataset is the initial dataset collected for imitation learning. The original dataset includes a first amount of expert data and a second amount of suboptimal data. The expert data consists of data demonstrated by experts and tested for correctness. The suboptimal dataset is a manually collected dataset, which may include trajectory segments of suboptimal operations; that is, the quality of the expert data is higher than that of the suboptimal data. Both the expert data and the suboptimal data include state data, actions, and next state data.

[0029] In one embodiment, obtaining the original dataset specifically includes: Collect the expert trajectory of the fourth preset length and the suboptimal trajectory of the fifth preset length; The first number of expert data is determined based on the expert trajectory of the fourth number of lines with a preset length, and the second number of suboptimal data is determined based on the suboptimal trajectory of the fifth number of lines with a preset length. The original data is determined based on the first set of expert data and the second set of suboptimal data.

[0030] Specifically, each trajectory point in the expert trajectory and the suboptimal trajectory includes a state-action data pair. For example, a human operator performs a set task on an intelligent mechanical device, and then acquires the operation trajectory formed by the robotic arm in performing the set task. A portion of the resulting operation trajectory is then manually inspected to obtain the expert trajectory, while the operation trajectory not manually inspected is designated as the suboptimal trajectory. After acquiring the expert trajectory and the suboptimal trajectory, a data tuple is constructed for each time step in both trajectories. This data tuple includes state data, action data, and next state data. Finally, the data tuple corresponding to each time step of the expert trajectory is considered as expert data, and the data tuple corresponding to each time step of the suboptimal trajectory is considered as suboptimal data.

[0031] For example, a robotic arm is manually operated, acquiring image observation data through cameras at several viewpoints, and obtaining state observation data of the robotic arm itself. Then, based on this image and state observation data, the state data for each time step is determined. Finally, the actions performed by the robotic arm at each time step are obtained, resulting in state-action data pairs for each time step, thus yielding the operational trajectory. Of course, in practical applications, the original dataset can also be obtained through interaction between an intelligent agent (such as a robot or drone) equipped with an expert policy network and the environment.

[0032] It should be noted that the amount of expert data is greater than the amount of second-best data, that is, the first quantity is less than the second quantity. This means that only a small number of operation trajectories need to be manually checked, which reduces the manual cost required to obtain the original dataset and improves the speed of obtaining the original dataset.

[0033] Furthermore, the discriminator is used to distinguish between expert states and non-expert states, that is, to determine whether the state data is state data within the expert data. The discriminator's input is the state data, and its output is the discrimination value indicating whether the state data is state data within the expert data. In other words, the discriminator's output is the expert state discrimination value, which represents the probability that the state data is an expert state (i.e., state data within the expert data). The discriminator's output is a value between 0 and 1.

[0034] In one embodiment, the training process of the discriminator specifically includes: The state data in the original dataset is input into the initial discriminator, and the initial discriminator outputs the expert state discrimination value corresponding to the state data. The initial discriminator is trained based on the expert state discriminant value to obtain a trained discriminator.

[0035] Specifically, positive sample learning (PU-learning) is used to train the discriminator to distinguish between expert and non-expert states. The training objective of the discriminator is: , in, Indicates the discriminator, Indicates an optimized discriminator This will make the following expression take the maximum value. Represented as taking the logarithm, Representing expert data, This represents suboptimal data. Represents state data. This indicates that state data is sampled from expert data. This indicates sampling state data from suboptimal data. Expressing expectations, This represents a constant coefficient.

[0036] Furthermore, the world model's role is to imagine future state trajectories at each time step, thereby obtaining a predicted state sequence. The world model includes an encoder, a dynamic prediction model, and a decoder. The encoder encodes the state data to obtain encoded states, the decoder reconstructs the state data based on the encoded states, and the dynamic prediction model predicts the state encoding of the next state based on the state encoding of the previous state and the previous action. That is, the dynamic prediction model can predict the state encoding of the current time step's state data based on the previous state encoding and the previous action. Therefore, the encoder, decoder, and dynamic prediction model can be represented as: , , , in, Indicates time step Status data, Indicates time step Status coding, Indicates time step Status data, Indicates time step The action, Indicates encoder, Indicates decoder, Represents a dynamic prediction model. Indicates time step Predictive state coding.

[0037] In one embodiment, the training process of the world model specifically includes: Select an original trajectory from the original dataset, the original trajectory including several original data; The state data from each raw data point in the original trajectory is input into the world model, and the state code is determined by the world model. Based on the state encoding, the reconstructed state data is determined using the world model; Based on the state code and the action, the predicted state code for the next state data is determined using the world model; The world model is trained based on all state data, all predicted state codes, and all reconstructed state data.

[0038] Specifically, the original trajectory can be either an expert trajectory or a suboptimal trajectory. Correspondingly, when the original trajectory is an expert trajectory, the original data is expert data; when the original trajectory is a suboptimal trajectory, the original data is suboptimal data. After obtaining the original trajectory, the state data of the first time step in the original data is input into the world model. The encoder in the world model encodes the state data to obtain the state code. Then, the decoder in the world model determines the reconstructed state data based on the state code, and determines the predicted state code of the next state data based on the state code and action using the dynamic prediction model in the world model.

[0039] After obtaining the predicted state code and reconstructed state data corresponding to each state data point, a world model loss function is constructed using all state data, all predicted state codes, and all reconstructed state data. This world model loss function is then used to train the world model. The world model loss function can be expressed as: , , , in, Represents the world model loss function. The model parameters represent the world model. This represents the length of the original trajectory. This represents the encoder in the world model. Expressing expectations, Indicates the reconstruction loss item. This represents the predicted loss term. Indicates time step Status data, Indicates time step Status coding, Indicates time step Predicted state coding, Indicates time step Status data, This indicates that the gradient has stopped. This represents the decoder in the world model.

[0040] S20. Select a third number of state data from the original dataset, and use the world model to generate a synthetic dataset based on the selected state data.

[0041] Specifically, the third quantity of status data can include status data from expert data or electrical system day data from suboptimal data. For example, a third quantity of raw data can be randomly selected from the raw data, and then the status data from the selected raw data can be read to obtain the third quantity of status data. Alternatively, the number of expert data and the number of suboptimal data in the third quantity of status data can be determined according to the ratio of the first quantity and the second quantity. Then, a certain number of expert data points are randomly selected from the expert data and a certain number of suboptimal data points are randomly selected from the suboptimal data, and the status data from the selected expert data and suboptimal data can be read to obtain the third quantity of status data, etc.

[0042] The synthetic dataset comprises multiple synthetic datasets, each determined using a world model based on selected state data. Each synthetic dataset includes state data, action data, and next state data. Specifically, generating the synthetic dataset using the world model based on the selected state data includes: For each selected state data, the selected state data is used as the initial state data; The initial state data is input into the world model, and the initial state data is encoded by the world model to obtain the state code; The state code is input into the exploration noise action model, and the exploration noise action model generates actions for the initial state data. The world model determines the next state code based on the initial state data and the action, and determines the next state data based on the next state code. Using the next state data as the initial state, the step of inputting the initial state data into the world model is repeated until a synthetic trajectory of a preset length is generated for the selected state data; The synthetic dataset is determined based on the synthetic trajectory corresponding to each selected state data.

[0043] Specifically, the preset length is pre-set, meaning the expert trajectory and the suboptimal trajectory acquired when obtaining the original dataset have the same length. In other words, the lengths of the synthesized trajectory, the expert trajectory, and the suboptimal trajectory are all the same. Each time step in the synthesized trajectory corresponds to a state-action data pair, and the generation process for the state-action data pairs is identical for all time steps except the first time step, where the state data is one of the selected third number of state data. Therefore, the generation process for a state-action data pair corresponding to one time step will be explained here as an example.

[0044] The world model is used to generate predicted state sequences, so that after state data is input into the world model, the world model generates the next state data for that state data. Here, the state data input into the world model is used as the initial state data. The world model will use an encoder Encode the initial state data to obtain the state code. = Then the status is encoded. Input an exploration noise action model (such as a policy model with exploration noise), and determine the action to be taken for the current interaction based on the exploration noise action model. Finally, the state is encoded. and actions The dynamic prediction model in the world model is input, the next state code is input through the dynamic prediction model, and the next state code is input into the decoder of the world model. The decoder outputs the next state data. , as Figure 3 The generated synthetic trajectory is shown.

[0045] After obtaining the synthetic trajectory, for each time step in the synthetic trajectory, the state data and action of that time step are obtained, and the state data of the next time step is obtained to obtain synthetic data including state data, action and next state data; finally, the dataset composed of all synthetic data is used as the synthetic dataset.

[0046] S30. Construct a training dataset based on the original dataset and the synthetic dataset, and use the discriminator to add a reward to each training data in the training dataset.

[0047] Specifically, the training dataset can be obtained by merging the original dataset and the synthetic dataset, or by merging the original time set and the synthetic dataset and then performing deduplication. Each training data point in the training dataset includes state data, action data, and next state data. In the process of applying reinforcement learning for imitation learning, a reward mechanism is also needed to incentivize the policy network to continuously optimize its behavioral strategy. Specifically, after constructing the training dataset, a discriminator evaluates each training data point and adds a corresponding reward value. This reward value reflects the quality of the action in that training data point. The policy network can adjust its parameters based on these reward values, enabling it to select better actions in subsequent decision-making processes, thereby gradually improving the effectiveness of imitation learning and better completing the specific task.

[0048] In one embodiment, adding a reward to each training data point in the training dataset using the discriminator specifically includes: The next state data from the training data in the training dataset is input into the discriminator, and the discriminator outputs the expert state discrimination value corresponding to the next state data. If the expert state judgment value is greater than a preset threshold, the expert state judgment value will be used as the reward for the training data. If the expert state judgment value is less than or equal to a preset threshold, the preset value is used as the reward for the training data.

[0049] Specifically, the expert state discriminant value is used to reflect the similarity between the state data and the expert state, and an action reward signal (i.e., reward) is assigned based on the similarity between the next state data and the expert state. In other words, the reward is assigned to the action reward signal at the current time step based on the probability that the state data at the next time step is expert state data.

[0050] The preset threshold is pre-configured and serves as the basis for determining whether to assign a reward signal to the action at the current time step. Specifically, if the expert state discrimination value is greater than the preset threshold, the next state data has a high similarity to the expert state, indicating that the current action is moving in a direction consistent with the expert policy. Therefore, the expert state discrimination value is used as a reward for the training data to encourage this beneficial action behavior and strengthen the model to learn a policy closer to the expert level. Conversely, if the expert state discrimination value is less than or equal to the preset threshold, the next state data has a low similarity to the expert state, indicating that the current action may deviate from the direction consistent with the expert policy. In this case, the preset value is used as a reward for the training data. The preset value is usually set to a relatively small value or zero, with the aim of discouraging actions that may deviate from the expert policy, avoiding the model learning poor policies, and thus guiding the model to learn and optimize in a direction closer to the expert level.

[0051] Based on this, in one embodiment, the reward function can be: , in, Indicates a reward. Indicates the preset threshold. Indicates time step Status data The expert state discriminant value, Indicates time step Status data, Indicates time step The action, Represents a conditional function, when hour, The value of is 1, when hour, The value of is 0.

[0052] S40. Based on the training dataset, the initial training policy is subjected to imitation learning to obtain a trained policy network.

[0053] Specifically, the trained policy network is obtained through imitation learning based on state-action data pairs in the training dataset, and it can be used to control intelligent mechanical devices (such as robots, robotic arms, etc.). The imitation learning of the initial training policy can employ an implicit Q-learning strategy, that is, training the policy network on the training dataset through implicit Q-learning.

[0054] In one embodiment, the imitation learning of the initial training policy based on the training dataset to obtain the trained policy network specifically includes: A first training batch is randomly selected from the training dataset, and a second training batch is randomly selected from the expert data in the training dataset. Implicit Q-learning is performed on the initial training policy based on the first training batch to obtain the loss function of the state value function, the loss function of the state-action value function, and the policy network loss function. Implicit Q-learning is performed on the initial training strategy based on the second training batch to obtain the cloning loss function; An imitation learning loss function is constructed based on the policy network loss function and the cloning loss function. The initial training policy is then subjected to imitation learning using the imitation learning loss function, the loss function of the state value function, and the loss function of the state-action value function to obtain a trained policy network.

[0055] Specifically, the first training batch includes a preset number of training data points from the training dataset, and the second training batch includes a preset number of expert data points from the training dataset. The training process of performing implicit Q-learning on the initial training strategy based on the first training batch and performing implicit Q-learning on the initial training strategy based on the second training batch is the same.

[0056] The loss functions for the state-value function, the state-action value function, and the imitation learning loss function are expressed as follows: , , , , , in, Represents the state value function. Represents the state-action value function. The loss function represents the state-value function. The loss function represents the state-action value function. Represents the training dataset. Represents the expert dataset, This indicates Expectile Regression. It's a hyperparameter. Expressing expectations, , and Both represent network parameters. The discount factor is a hyperparameter. This represents the imitation learning loss function. The policy network loss function is represented as follows. Represents the cloning loss function. This represents the weighting coefficient.

[0057] In one embodiment, after performing imitation learning on the initial training policy based on the training dataset to obtain a trained policy network, the method further includes: The policy network is deployed on intelligent mechanical devices so that the intelligent mechanical devices perform tasks based on the policy network. Specifically, intelligent mechanical devices can be robots, robotic arms, etc., and the task is the target task obtained from the training dataset. For example, a task that can be carried by a robotic arm.

[0058] In summary, this embodiment provides a method, apparatus, device, and medium for imitation learning based on synthetic trajectory enhancement. The method includes acquiring an original dataset and training a discriminator and a world model using the original dataset; selecting a third number of state data points from the original dataset; generating a synthetic dataset using the world model based on the selected state data; the synthetic data in the synthetic dataset includes state data, actions, and next state data; constructing a training dataset based on the original dataset and the synthetic dataset; adding a reward to each training data point in the training dataset using the discriminator; and performing imitation learning on an initial training policy based on the training dataset to obtain a trained policy network. This application utilizes the original data to train the discriminator and the world model, then uses the world model to construct synthetic data, and finally uses the original data and synthetic data to perform imitation learning on the initial policy network model. This allows for the utilization of suboptimal data, improves the learning effect of imitation learning, and consequently enhances the task execution performance of the policy network obtained based on imitation learning.

[0059] Based on the above-described imitation learning method based on synthetic trajectory enhancement, this embodiment provides a second aspect of this application, which provides an imitation learning apparatus based on synthetic trajectory enhancement, such as... Figure 4As shown, the imitation learning device based on synthetic trajectory enhancement specifically includes: The acquisition module 100 is used to acquire the original dataset and train the discriminator and the world model using the original dataset. The original dataset includes a first number of expert data and a second number of suboptimal data. Both the expert data and the suboptimal data include state data, actions, and next state data. The synthesis module 200 is used to select a third number of state data from the original dataset and generate a synthetic dataset based on the selected state data using the world model. The synthetic data in the synthetic dataset includes state data, actions, and next state data. The construction module 300 is used to construct a training dataset based on the original dataset and the synthetic dataset, and to add a reward to each training data in the training dataset using the discriminator; The training module 400 is used to perform imitation learning on the initial training policy based on the training dataset to obtain a trained policy network.

[0060] Based on the above-described imitation learning method based on synthetic trajectory enhancement, this embodiment provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps in the imitation learning method based on synthetic trajectory enhancement as described in the above embodiment.

[0061] Based on the aforementioned imitation learning method based on synthetic trajectory enhancement, this application also provides a terminal device, such as... Figure 5 As shown, it includes at least one processor 20; a display screen 21; and a memory 22, and may also include a communications interface 23 and a bus 24. The processor 20, display screen 21, memory 22, and communications interface 23 can communicate with each other via the bus 24. The display screen 21 is configured to display a preset user guide interface in the initial setup mode. The communications interface 23 can transmit information. The processor 20 can invoke logical instructions in the memory 22 to execute the methods described in the above embodiments.

[0062] Furthermore, the logical instructions in the aforementioned memory 22 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0063] The memory 22, as a computer-readable storage medium, can be configured to store software programs, computer-executable programs, such as program instructions or modules corresponding to the methods in the embodiments of this disclosure. The processor 20 executes functional applications and data processing by running the software programs, instructions, or modules stored in the memory 22, thereby implementing the methods in the above embodiments.

[0064] The memory 22 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 22 may include high-speed random access memory (RAM) and non-volatile memory. Examples include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, as well as transient storage media.

[0065] Furthermore, the specific process of loading and executing multiple instruction processors in the aforementioned storage medium and terminal device has been described in detail in the above method, and will not be repeated here.

[0066] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An imitation learning method based on synthetic trajectory enhancement, characterized in that, The aforementioned imitation learning method based on synthetic trajectory enhancement specifically includes: Obtain the original dataset and train the discriminator and world model using the original dataset, wherein the original dataset includes a first number of expert data and a second number of suboptimal data, and both the expert data and the suboptimal data include state data, actions and next state data; A third number of state data are selected from the original dataset, and a synthetic dataset is generated based on the selected state data using the world model. The synthetic data in the synthetic dataset includes state data, actions, and next state data. A training dataset is constructed based on the original dataset and the synthetic dataset, and a reward is added to each training data point in the training dataset using the discriminator; Based on the training dataset, the initial training policy is subjected to imitation learning to obtain a trained policy network.

2. The imitation learning method based on synthetic trajectory enhancement according to claim 1, characterized in that, The training process of the discriminator specifically includes: The state data in the original dataset is input into the initial discriminator, and the initial discriminator outputs the expert state discrimination value corresponding to the state data. The initial discriminator is trained based on the expert state discriminant value to obtain a trained discriminator.

3. The imitation learning method based on synthetic trajectory enhancement according to claim 1, characterized in that, The training process of the world model specifically includes: Select an original trajectory from the original dataset, the original trajectory including several original data; The state data from each raw data point in the original trajectory is input into the world model, and the state code is determined by the world model. Based on the state encoding, the reconstructed state data is determined using the world model; Based on the state code and the action, the predicted state code for the next state data is determined using the world model; The world model is trained based on all state data, all predicted state codes, and all reconstructed state data.

4. The imitation learning method based on synthetic trajectory enhancement according to claim 1, characterized in that, The specific steps of generating a synthetic dataset using the world model based on the selected state data include: For each selected state data, the selected state data is used as the initial state data; The initial state data is input into the world model, and the initial state data is encoded by the world model to obtain the state code; The state code is input into the exploration noise action model, and the exploration noise action model generates actions for the initial state data. The world model determines the next state code based on the initial state data and the action, and determines the next state data based on the next state code. Using the next state data as the initial state, the step of inputting the initial state data into the world model is repeated until a synthetic trajectory of a preset length is generated for the selected state data; The synthetic dataset is determined based on the synthetic trajectory corresponding to each selected state data.

5. The imitation learning method based on synthetic trajectory enhancement according to claim 1, characterized in that, The step of adding a reward to each training data point in the training dataset using the discriminator specifically includes: The next state data from the training data in the training dataset is input into the discriminator, and the discriminator outputs the expert state discrimination value corresponding to the next state data. If the expert state judgment value is greater than a preset threshold, the expert state judgment value will be used as the reward for the training data. If the expert state judgment value is less than or equal to a preset threshold, the preset value is used as the reward for the training data.

6. The imitation learning method based on synthetic trajectory enhancement according to claim 1, characterized in that, The step of performing imitation learning on the initial training policy based on the training dataset to obtain the trained policy network specifically includes: A first training batch is randomly selected from the training dataset, and a second training batch is randomly selected from the expert data in the training dataset. Implicit Q-learning is performed on the initial training policy based on the first training batch to obtain the loss function of the state value function, the loss function of the state-action value function, and the policy network loss function. Implicit Q-learning is performed on the initial training strategy based on the second training batch to obtain the cloning loss function; An imitation learning loss function is constructed based on the policy network loss function and the cloning loss function. The initial training policy is then subjected to imitation learning using the imitation learning loss function, the loss function of the state value function, and the loss function of the state-action value function to obtain a trained policy network.

7. The imitation learning method based on synthetic trajectory enhancement according to claim 1, characterized in that, After performing imitation learning on the initial training policy based on the training dataset to obtain a trained policy network, the method further includes: The policy network is deployed on intelligent mechanical devices so that the intelligent mechanical devices perform tasks based on the policy network.

8. A synthetic trajectory enhancement-based imitation learning device, characterized in that, The aforementioned imitation learning device based on synthetic trajectory enhancement specifically includes: An acquisition module is used to acquire a raw dataset and train a discriminator and a world model using the raw dataset. The raw dataset includes a first number of expert data and a second number of suboptimal data. Both the expert data and the suboptimal data include state data, actions, and next state data. The synthesis module is used to select a third number of state data from the original dataset and generate a synthetic dataset based on the selected state data using the world model. The synthetic data in the synthetic dataset includes state data, actions, and next state data. A construction module is used to construct a training dataset based on the original dataset and the synthetic dataset, and to add a reward to each training data point in the training dataset using the discriminator; The training module is used to perform imitation learning on the initial training policy based on the training dataset to obtain a trained policy network.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the imitation learning method based on synthetic trajectory enhancement as described in any one of claims 1-7.

10. A terminal device, characterized in that, include: Processor and memory; The memory stores a computer-readable program that can be executed by the processor; When the processor executes the computer-readable program, it implements the steps of the imitation learning method based on synthetic trajectory enhancement as described in any one of claims 1-7.

Images