Offline reinforcement learning method and device based on time reversal symmetry

Abstract

The invention provides an off-line reinforcement learning method and device based on time reversal symmetry, and the method comprises the steps: obtaining a current moment state, and coding the current moment state into a low-dimensional representation of the current moment state; on the basis of a pre-trained strategy network, according to the low-dimensional representation of the state at the current moment, predicting the low-dimensional representation of the state at the next moment; on the basis of a pre-trained inverse dynamic model, according to the low-dimensional representation of the state at the current moment and the low-dimensional representation of the state at the next moment, predicting the low-dimensional representation of the action corresponding to the state at the current moment; acquiring a target action corresponding to the current moment state according to the low-dimensional representation of the action corresponding to the current moment state; wherein the strategy network and the inverse dynamic model are obtained by training and optimizing based on time reversal symmetry constraints, and the time reversal symmetry constraints are obtained by setting based on the forward dynamic model and the inverse dynamic model. According to the method, the characteristic of time reversal symmetry of the data is extracted in the hidden space, and the strategy network and the inverse dynamic model are trained and optimized based on the time reversal symmetry constraint, so that the prediction of low-dimensional representation in the hidden space is more reliable and stable, the performance of the reinforcement learning algorithm in a small sample scene is effectively improved, and the prediction efficiency is improved. And the sample utilization rate is improved.

Description

Technical Field

[0001] The present invention relates to the field of machine learning, and in particular to an offline reinforcement learning method and device based on time reversal symmetry. Background Art

[0002] Existing offline reinforcement learning (Offline RL) algorithms face the following challenges (1)-(3).

[0003] (1) Insufficient performance. Existing offline RL algorithms still perform worse than online RL algorithms on many tasks, especially for complex environments with long-term dependencies. This is mainly because offline RL algorithms cannot directly collect samples from the environment to update the model and can only rely on pre-collected historical data, which limits their learning ability.

[0004] (2) Large sample requirements. Offline RL algorithm training usually requires a large amount of historical data to learn effectively. The huge sample requirements limit the application of the algorithm in the real world because collecting high-quality offline data is usually costly and time-consuming.

[0005] (3) Low sample utilization. Existing offline RL algorithms are insufficient in utilizing historical data. Often, the training data contains a large amount of noisy data and irrelevant information, making it difficult for the algorithm to effectively learn useful decision-making strategies from it.

[0006] The above technical challenges have seriously hindered the promotion and application of offline reinforcement learning technology, and there is an urgent need to propose more effective and robust solutions.

[0007] Therefore, it is necessary to solve the problems of poor performance of existing offline reinforcement learning algorithms, large sample size required and low sample utilization. Summary of the invention

[0008] The present invention provides an offline reinforcement learning method and device based on time reversal symmetry, which are used to overcome the defects of existing offline reinforcement learning algorithms, such as poor performance, large sample size and low sample utilization, effectively improve the performance of reinforcement learning algorithms in small sample scenarios, and improve sample utilization.

[0009] On the one hand, the present invention provides an offline reinforcement learning method based on time reversal symmetry, including: obtaining the current state and encoding the current state into a low-dimensional representation of the current state; based on a pre-trained policy network, predicting the low-dimensional representation of the next state according to the low-dimensional representation of the current state; based on a pre-trained inverse dynamics model, predicting the low-dimensional representation of the action corresponding to the current state according to the low-dimensional representation of the current state and the low-dimensional representation of the next state; according to the low-dimensional representation of the action corresponding to the current state, obtaining the target action corresponding to the current state; wherein the policy network and the inverse dynamics model are obtained by training and optimization based on time reversal symmetry constraints, and the time reversal symmetry constraints are set based on a forward dynamics model and an inverse dynamics model.

[0010] Furthermore, encoding the current state into a low-dimensional representation of the current state specifically includes: encoding the current state based on a pre-trained state encoder to obtain a low-dimensional representation of the current state.

[0011] Furthermore, training and optimizing the inverse dynamics model includes: determining an offline data set, wherein the offline data set includes states, actions and corresponding rewards at consecutive historical moments; based on the offline data set, iteratively optimizing the inverse dynamics model through a forward dynamics model loss function, a reverse dynamics loss function, an inverse dynamics model loss function and a time reversal symmetry constraint to obtain a pre-trained inverse dynamics model.

[0012] Furthermore, the inverse dynamics model is trained and optimized, specifically including: inputting the historical current moment state in the offline data set into the state encoder to obtain a low-dimensional representation of the historical current moment state; and, inputting the historical next moment state in the offline data set into the state encoder to obtain a low-dimensional representation of the historical next moment state; inputting the low-dimensional representation of the historical current moment state and the low-dimensional representation of the historical next moment state into the inverse dynamics model to obtain an implicit representation of the corresponding action under the historical current moment state; inputting the low-dimensional representation of the historical current moment state and the implicit representation of the corresponding action under the historical current moment state into the forward dynamics model together to obtain a forward change corresponding to the low-dimensional representation of the historical current moment state; and, The low-dimensional representation of the historical state at the current moment and the latent representation of the corresponding action at the historical state at the current moment are input into the inverse dynamics model together to obtain the reverse change corresponding to the low-dimensional representation of the historical state at the current moment; the forward change corresponding to the low-dimensional representation of the historical state at the current moment is decoded to obtain the forward change of the original state space; and the reverse change corresponding to the low-dimensional representation of the historical state at the current moment is decoded to obtain the reverse change of the original state space; based on the offline data set, the forward change and reverse change of the decoded original state space, the low-dimensional representation of the historical state at the current moment and the low-dimensional representation of the historical state at the next moment, and the latent representation of the corresponding action at the historical state at the current moment, the inverse dynamics model is iteratively optimized to obtain a pre-trained inverse dynamics model.

[0013] Furthermore, the forward dynamics loss function is expressed as follows:

[0014] The inverse dynamics loss function is expressed as follows:

[0015] The inverse dynamics model loss function is expressed as follows:

[0016] The time reversal symmetry constraint is expressed as follows:

[0017] in, Represents the historical current state in the offline dataset. Represents the historical state of the next moment in the offline dataset. Represents the partial differential of the hidden state at the current moment of history to the original dimension state, It represents the partial differential of the hidden state at the next moment in history to the original dimension state. Represents the forward change obtained by decoding, Indicates the reverse change obtained by decoding, It represents the change in the hidden state at the current moment of the history output by the forward dynamics model. A low-dimensional representation of the state of the current moment in history, represents the implicit representation of the corresponding action at the current state of the history output of the inverse dynamics model, Indicates the change in the hidden state at the next moment of the history output of the inverse dynamics model, Indicates the action corresponding to the current state of the history in the offline dataset. A low-dimensional representation of the state at the next moment in history, , represents the reward in the offline dataset, represents the discount factor, The value prediction of the low-dimensional representation of the state at the next moment in history, The value prediction of the low-dimensional representation of the current state of history, Represents an action decoder.

[0018] Furthermore, the training and optimization of the policy network specifically includes: based on the low-dimensional representation of the state at the current historical moment, the low-dimensional representation of the state at the next historical moment, and the implicit representation of the corresponding action under the state at the current historical moment, the policy network is iteratively optimized through a preset loss function to obtain a pre-trained policy network.

[0019] Furthermore, the preset loss function is expressed as follows:

[0020] in, In latent space The value prediction of represents the policy network, represents the offline dataset, Indicates the current state of history. Indicates the state of the next moment in history. Represents the low-dimensional representation of the policy network based on the current state of history The low-dimensional representation of the predicted state at the next historical moment, represents the weight of the control strategy constraint, A low-dimensional representation of the state at the next moment in history, represents the inversion symmetry constraint, It represents the implicit representation of the corresponding action at the current state of the history output by the inverse dynamics model.

[0021] In a second aspect, the present invention also provides an offline reinforcement learning device based on time reversal symmetry, comprising: an encoding module, used to obtain the current state and encode the current state into a low-dimensional representation of the current state; a first prediction module, used to predict the low-dimensional representation of the next state based on a pre-trained policy network according to the low-dimensional representation of the current state; a second prediction module, used to predict the low-dimensional representation of the action corresponding to the current state based on a pre-trained inverse dynamics model according to the low-dimensional representation of the current state and the low-dimensional representation of the next state; a target action acquisition module, used to obtain the target action corresponding to the current state according to the low-dimensional representation of the action corresponding to the current state; wherein the policy network and the inverse dynamics model are trained and optimized based on time reversal symmetry constraints, and the time reversal symmetry constraints are set based on the forward dynamics model and the inverse dynamics model.

[0022] In a third aspect, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements any one of the above-described offline reinforcement learning methods based on time reversal symmetry.

[0023] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described offline reinforcement learning methods based on time reversal symmetry.

[0024] The offline reinforcement learning method based on time reversal symmetry provided by the present invention obtains the state at the current moment, encodes the state at the current moment into a low-dimensional representation of the state at the current moment, and predicts the low-dimensional representation of the state at the next moment based on the low-dimensional representation of the state at the current moment based on the pre-trained policy network, and then predicts the low-dimensional representation of the action corresponding to the state at the current moment based on the pre-trained inverse dynamics model, according to the low-dimensional representation of the state at the current moment and the low-dimensional representation of the state at the next moment, so that the target action corresponding to the state at the current moment is obtained according to the low-dimensional representation of the action corresponding to the state at the current moment; wherein, the policy network and the inverse dynamics model are trained and optimized based on the time reversal symmetry constraint, and the time reversal symmetry constraint is set based on the forward dynamics model and the inverse dynamics model. The method extracts the time reversal symmetry of data in the latent space, and optimizes the policy network and the inverse dynamics model based on the time reversal symmetry constraint training, so that the prediction of the low-dimensional representation in the final latent space is more reliable and stable, effectively improving the performance of the reinforcement learning algorithm in a small sample scenario, and improving the sample utilization rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 It is a flowchart of an offline reinforcement learning method based on time reversal symmetry provided by an embodiment of the present invention.

[0027] Figure 2 It is a schematic diagram of training optimization of the inverse dynamics model provided by an embodiment of the present invention.

[0028] Figure 3 It is a schematic diagram of the joint training of the inverse dynamics model and the strategy network provided by the implementation of the present invention.

[0029] Figure 4 It is a schematic diagram of the structure of an offline reinforcement learning device based on time reversal symmetry provided by an embodiment of the present invention.

[0030] Figure 5 It is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] It should be noted that the existing offline reinforcement learning algorithms have the disadvantages of poor performance, large sample requirements and low sample utilization. Taking this into consideration, the present invention proposes an offline reinforcement learning method based on time reversal symmetry.

[0033] Figure 1 A schematic flow chart of an offline reinforcement learning method based on time reversal symmetry provided by an embodiment of the present invention is shown.

[0034] like Figure 1 As shown, the method includes steps S110-S140, and steps S110-S140 and related steps will be described in detail below.

[0035] S110, obtaining the current state, and encoding the current state into a low-dimensional representation of the current state.

[0036] The current state refers to the specific situation or configuration of the environment in which the agent is located at the current moment. It contains all relevant information required for the agent to execute decisions, such as current position, speed, number of resources, etc. It can be determined based on the actual application scenario and the characteristics of the environment, and is not specifically limited here.

[0037] For example, for a chess game, the current state may be the position of all chess pieces on the board; in the case of an autonomous vehicle, the current state may include the vehicle's speed, position, direction, distance to surrounding objects, and other sensor data; in a recommendation system, the current state may include the user's historical behavior data, preference settings, etc.

[0038] In offline reinforcement learning, the role of the current state is to provide the agent with information about the current environmental conditions so that the agent can learn the optimal decision-making strategy from it.

[0039] It is easy to understand that, in this embodiment, the current state is first obtained, and then the obtained current state is encoded to obtain a low-dimensional representation of the current state.

[0040] The current state can be in the form of a series of numerical features or in the form of an image. When encoding the acquired current state, this step can adopt feature extraction, or use an embedding layer to reduce the dimension, or use a convolutional neural network to process the current state.

[0041] In a specific embodiment, a pre-trained state encoder is used to encode the current state to obtain a low-dimensional representation of the current state as an output.

[0042] It should be noted that the subsequent use of the low-dimensional representation of the current state to make decisions can reduce computational complexity, improve learning efficiency, and maintain the key features of the state information.

[0043] Based on obtaining and encoding the current state in step S110, further, step S120 is executed.

[0044] S120, based on the pre-trained policy network, predict the low-dimensional representation of the state at the next moment according to the low-dimensional representation of the state at the current moment.

[0045] It can be understood that by inputting the low-dimensional representation of the current state into the pre-trained strategy network, the low-dimensional representation of the state at the next moment can be predicted.

[0046] It is worth mentioning that the policy network is pre-trained. During training, not only is the value function of the next-moment state in the latent space generated by the policy network required to be as large as possible, but the next-moment state in the generated latent space is also required to be as close as possible to the corresponding state in the offline data set. At the same time, the next-time state in the generated latent space must also satisfy the time reversal symmetry constraint.

[0047] Among them, the time reversal symmetry constraint is set based on the forward dynamics model and the reverse dynamics model.

[0048] Time reversal symmetry is a fundamental concept in physics that describes whether the laws of physics remain unchanged under time reversal operations. In classical mechanics, a system has time reversal symmetry if its dynamical equations remain unchanged under time reversal transformations. This means that if time is reversed, the state evolution of the system will occur in the opposite order, but the laws of physics themselves will not change.

[0049] In this embodiment, when training the optimization policy network, the time reversal symmetry constraint can help the algorithm better understand the dynamic characteristics of the environment. For example, if an action sequence Lead to state To status , then the reverse sequence of actions Should be able to make the state To status , provided that the environment has time reversal symmetry.

[0050] When predicting the low-dimensional representation of the next state, a more robust state representation can be obtained by using time reversal symmetry. Specifically, if a state representation can remain unchanged or change in a predetermined way under time reversal, then such a state representation will be more stable.

[0051] Based on the low-dimensional representation of the state at the next moment predicted according to the low-dimensional representation of the state at the current moment in step S120 based on the pre-trained policy network, step S130 is further performed.

[0052] S130, based on a pre-trained inverse dynamics model, predict the low-dimensional representation of the action corresponding to the current state according to the low-dimensional representation of the current state and the low-dimensional representation of the next state; wherein the policy network and the inverse dynamics model are trained and optimized based on time reversal symmetry constraints, and the time reversal symmetry constraints are set based on the forward dynamics model and the inverse dynamics model.

[0053] In this embodiment, the role of the inverse dynamics model is to predict the action that causes the state transition from two given continuous states (the state at the current moment and the state at the next moment).

[0054] Specifically, the low-dimensional representation of the current state and the low-dimensional representation of the next state are input into a pre-trained inverse dynamics model together, and the low-dimensional representation of the action corresponding to the current state can be predicted.

[0055] It is worth mentioning that all operations in this embodiment are performed in the latent space. Therefore, step S110 needs to first encode the current state into a low-dimensional representation of the current state. The policy network predicts the low-dimensional representation of the next state, and the inverse dynamics model predicts the low-dimensional representation of the action corresponding to the current state.

[0056] It should be noted that the inverse dynamics model is pre-trained, and during its training process, its prediction output is required to satisfy the time reversal symmetry constraint.

[0057] In step S130, based on the pre-trained inverse dynamics model, based on the low-dimensional representation of the state at the current moment and the low-dimensional representation of the state at the next moment, predicting the low-dimensional representation of the action corresponding to the current moment state, further, step S140 is executed.

[0058] S140, obtaining a target action corresponding to the current state according to a low-dimensional representation of the action corresponding to the current state.

[0059] It is easy to understand that the low-dimensional representation of the action corresponding to the current state predicted in step S130 is a representation in the latent space. Therefore, this step needs to decode the low-dimensional representation of the action corresponding to the current state back to the original action space. The specific decoding process can be achieved by an automatic encoder (the inverse operation of the encoding process of step S110).

[0060] It is worth mentioning that this embodiment introduces time reversal symmetry constraints in the training process of the policy network and the inverse dynamics model, which can help improve the robustness and generalization ability of the model. The time reversal symmetry constraints can ensure that the performance of the model on the forward and reverse time axes is consistent, thereby better reflecting the physical characteristics of the system.

[0061] In order to realize the time reversal symmetry constraint, this embodiment introduces the forward dynamics model f and inverse dynamics models g , these two models together form the basis of the time reversal symmetry constraint.

[0062] Among them, the forward dynamics model is used to predict the state at the next moment based on the current state and the current action, and the reverse dynamics model is used to predict the current state based on the next state and the current action.

[0063] In this implementation, the current state is obtained and encoded into a low-dimensional representation of the current state, and based on the pre-trained policy network, the low-dimensional representation of the next state is predicted according to the low-dimensional representation of the current state, and then based on the pre-trained inverse dynamics model, the low-dimensional representation of the action corresponding to the current state is predicted according to the low-dimensional representation of the current state and the low-dimensional representation of the next state, so that the target action corresponding to the current state is obtained according to the low-dimensional representation of the action corresponding to the current state; wherein, the policy network and the inverse dynamics model are trained and optimized based on the time reversal symmetry constraint, and the time reversal symmetry constraint is set based on the forward dynamics model and the inverse dynamics model. This method extracts the time reversal symmetry of the data in the latent space, and optimizes the policy network and the inverse dynamics model based on the time reversal symmetry constraint training, so that the prediction of the low-dimensional representation in the final latent space is more reliable and stable, which effectively improves the performance of the reinforcement learning algorithm in small sample scenarios and improves the sample utilization rate.

[0064] On the basis of the above embodiments, the training process of the inverse dynamics model and the strategy network will be described in detail below.

[0065] Training and optimizing the inverse dynamics network includes: determining an offline data set, the offline data set includes states, actions and corresponding rewards at consecutive historical moments; based on the offline data set, iteratively optimizing the inverse dynamics model through a forward dynamics model loss function, a reverse dynamics loss function, an inverse dynamics model loss function and a time reversal symmetry constraint to obtain a pre-trained inverse dynamics model.

[0066] Figure 2 A schematic diagram of training optimization of the inverse dynamics model provided by an embodiment of the present invention is shown.

[0067] like Figure 2 As shown, first, the current state of history Input into the state encoder to obtain a low-dimensional representation of the current state of history ; and, the state of the next moment in history Input into the state encoder to obtain a low-dimensional representation of the state at the next moment in history .

[0068] Then, the low-dimensional representation of the current state of history And the low-dimensional representation of the state at the next moment in history Input to Inverse Dynamics Model In the current state of history, the implicit representation of the corresponding action is obtained .

[0069] Next, the low-dimensional representation of the current state of history and the implicit representation of the corresponding action at the current state of history Input to the forward dynamics model In the example above, we can get the forward change corresponding to the low-dimensional representation of the current state of history ; and, the low-dimensional representation of the state at the next moment in history and the implicit representation of the corresponding action at the current state of history Import into inverse dynamics model In the example above, we can get the reverse change corresponding to the low-dimensional representation of the current state of history - .

[0070] Then, the forward change corresponding to the low-dimensional representation of the current state of history is Decode and get the forward change of the original state space ; and the reverse change corresponding to the low-dimensional representation of the current state of history - Decode and get the reverse change of the original state space .

[0071] Finally, based on the offline dataset , the forward change of the original state space obtained by decoding and reverse change , low-dimensional representation of the current state of history And the low-dimensional representation of the state at the next moment in history , the implicit representation of the corresponding action in the current state of history , the inverse dynamics model is iteratively optimized to obtain a pre-trained inverse dynamics model.

[0072] It should be noted that in Figure 2 The iterative optimization process shown can simultaneously optimize the state encoder, inverse dynamics model, forward dynamics model, reverse dynamics model, state decoder and motion decoder.

[0073] Specifically, this iterative optimization process can be achieved through the forward dynamics model loss function, the reverse dynamics loss function, the inverse dynamics model loss function and the time reversal symmetry constraint.

[0074] The forward dynamics loss function is expressed as follows:

[0075] (1).

[0076] The inverse dynamics loss function is expressed as follows (2).

[0077] (2).

[0078] The loss function of the inverse dynamics model is expressed as follows (3).

[0079] (3).

[0080] The time reversal symmetry constraint is expressed as follows:

[0081] (4).

[0082] In formulas (1)-(4), Represents the historical current state in the offline dataset. Represents the historical state of the next moment in the offline dataset. Represents the partial differential of the hidden state at the current moment of history to the original dimension state, It represents the partial differential of the hidden state at the next moment in history to the original dimension state. Represents the forward change obtained by decoding, Indicates the reverse change obtained by decoding, It represents the change in the hidden state at the current moment of the history output by the forward dynamics model. A low-dimensional representation of the state of the current moment in history, Represents the implicit representation of the corresponding action at the current state of the history output of the inverse dynamics model (same as above ), Indicates the change in the hidden state at the next moment of the history output of the inverse dynamics model, Indicates the action corresponding to the current state of the history in the offline dataset. A low-dimensional representation of the state at the next moment in history, , represents the reward in the offline dataset, represents the discount factor, The value prediction of the low-dimensional representation of the state at the next moment in history, The value prediction of the low-dimensional representation of the current state of history, Represents an action decoder.

[0083] Figure 3 A schematic diagram of the joint training of the inverse dynamics model and the strategy network provided by an embodiment of the present invention is shown.

[0084] like Figure 3 As shown, Figure 3 Left is Figure 2 The description content will not be repeated here.

[0085] for Figure 3Right, describes the low-dimensional representation of the current state based on history , low-dimensional representation of the state at the next moment in history And the implicit representation of the corresponding action at the current state of history , the process of iteratively optimizing the policy network.

[0086] The process of iteratively optimizing the policy network can be achieved by using a preset loss function. The preset loss function can be specifically described in the following formula (5).

[0087] (5).

[0088] In formula (5), In latent space The value prediction of represents the policy network, represents the offline dataset, Indicates the current state of history. Indicates the state of the next moment in history. Represents the low-dimensional representation of the policy network based on the current state of history The low-dimensional representation of the predicted state at the next historical moment, represents the weight of the control strategy constraint, A low-dimensional representation of the state at the next moment in history, represents the inversion symmetry constraint, Represents the implicit representation of the corresponding action at the current state of the history output of the inverse dynamics model (same as above ).

[0089] The meaning of formula (5) is that when the policy network generates a state, it must satisfy the requirement that the value function of the next state in the generated latent space is as large as possible (corresponding to ), and also to ensure that the state of the next moment in the generated latent space is as close as possible to the corresponding state in the offline dataset (corresponding ). At the same time, the next time state in the generated latent space must also satisfy the time reversal symmetry constraint (corresponding to ).

[0090] Formula (5) involves the value function ,according to Figure 3 It can be seen that the learning process of the value function can be expressed by the following equations (6)-(9).

[0091] (6).

[0092] (7).

[0093] (8).

[0094] (9).

[0095] In equations (6)-(9), represents the expectation (of the cumulative reward that may be obtained), Represents the historical current state in the offline dataset. Represents the historical state of the next moment in the offline dataset. represents the reward in the offline dataset, represents the offline dataset, represents the expected regression loss function, which is used to learn the expected regression value of the current value function. Its expansion is shown in equations (8)-(9). represents the discount factor, Represents the value prediction of the state at the next historical moment in the latent space, Represents the value prediction of the current state of history in the latent space, represents the time step, represents the reward obtained at the k+1th step, It is a hyperparameter that controls the expected regression score.

[0096] It should be noted that the state encoder, inverse dynamics model, forward dynamics model, reverse dynamics model, state decoder and action decoder involved in this embodiment can be constructed based on a simple fully connected network, or based on a more complex convolutional neural network or recurrent neural network, depending on actual needs, and no specific limitation is made here.

[0097] After completing the training and optimization of the inverse dynamics model and policy network, they can be deployed in actual application scenarios.

[0098] In practical applications, the overall decision-making process can be expressed as follows:

[0099] (10).

[0100] In formula (10), represents the target action, represents the action decoder, represents the inverse dynamics model, A low-dimensional representation of the current state, represents the policy network.

[0101] According to formula (10), after obtaining the current state, the current state is first encoded into the corresponding low-dimensional representation ; Then, the low-dimensional representation of the current state is input into the trained policy network In the model, the low-dimensional representation of the state at the next moment can be predicted; then, the low-dimensional representation of the current state and the predicted low-dimensional representation of the next state are input into the inverse dynamics model together, and the low-dimensional representation of the action corresponding to the current state can be predicted; finally, the low-dimensional representation of the action corresponding to the current state is decoded back to the original action space, and the target action corresponding to the current state can be obtained.

[0102] The target action here depends on the application field and specific tasks, for example, a robot grasps an object, walks, a vehicle turns, accelerates, changes lanes, turns lights on and off, adjusts the temperature, plays music, etc., and is not specifically limited here.

[0103] In this embodiment, the inverse dynamics model and the policy network are optimized by training based on the time reversal symmetry constraint, and then, based on the pre-trained policy network, the low-dimensional representation of the state at the next moment is predicted according to the low-dimensional representation of the state at the current moment, and then based on the pre-trained inverse dynamics model, the low-dimensional representation of the action corresponding to the current moment state is predicted according to the low-dimensional representation of the state at the current moment and the low-dimensional representation of the state at the next moment, so that the target action corresponding to the current moment state is obtained according to the low-dimensional representation of the action corresponding to the current moment state; wherein, the policy network and the inverse dynamics model are trained and optimized based on the time reversal symmetry constraint, and the time reversal symmetry constraint is set based on the forward dynamics model and the reverse dynamics model. This method extracts the time reversal symmetry of data in the latent space, and optimizes the policy network and the inverse dynamics model based on the time reversal symmetry constraint training, so that the prediction of the low-dimensional representation in the final latent space is more reliable and stable, which effectively improves the performance of the reinforcement learning algorithm in small sample scenarios and improves the sample utilization rate.

[0104] It is worth mentioning that the offline reinforcement learning method based on time reversal symmetry provided by the embodiment of the present invention can not only improve the performance of the reinforcement learning algorithm in small sample scenarios (not limited to small sample scenarios, but the advantages in small sample scenarios are more obvious), but also has the characteristics of high sample utilization and easy deployment, and is more in line with the actual algorithm implementation needs.

[0105] Corresponding to the offline reinforcement learning method based on time reversal symmetry described in the above embodiments, the present invention also proposes an offline reinforcement learning device based on time reversal symmetry. Specifically, Figure 4 A schematic structural diagram of an offline reinforcement learning device based on time reversal symmetry provided by an embodiment of the present invention is shown.

[0106] like Figure 4As shown, the device includes: an encoding module 410, which is used to obtain the current state and encode the current state into a low-dimensional representation of the current state; a first prediction module 420, which is used to predict the low-dimensional representation of the next state based on a pre-trained policy network according to the low-dimensional representation of the current state; a second prediction module 430, which is used to predict the low-dimensional representation of the action corresponding to the current state based on a pre-trained inverse dynamics model according to the low-dimensional representation of the current state and the low-dimensional representation of the next state; a target action acquisition module 440, which is used to obtain the target action corresponding to the current state according to the low-dimensional representation of the action corresponding to the current state; wherein the policy network and the inverse dynamics model are obtained by training and optimization based on time reversal symmetry constraints, and the time reversal symmetry constraints are set based on the forward dynamics model and the inverse dynamics model.

[0107] In this embodiment, the current state is obtained by the encoding module 410, and the current state is encoded as a low-dimensional representation of the current state. The first prediction module 420 predicts the low-dimensional representation of the next state based on the pre-trained policy network according to the low-dimensional representation of the current state, and then the second prediction module 430 predicts the low-dimensional representation of the action corresponding to the current state based on the pre-trained inverse dynamics model according to the low-dimensional representation of the current state and the low-dimensional representation of the next state. Thus, the target action acquisition module 440 obtains the target action corresponding to the current state according to the low-dimensional representation of the action corresponding to the current state; wherein, the policy network and the inverse dynamics model are trained and optimized based on the time reversal symmetry constraint, and the time reversal symmetry constraint is set based on the forward dynamics model and the inverse dynamics model. The device extracts the time reversal symmetry of data in the latent space, and optimizes the policy network and the inverse dynamics model based on the time reversal symmetry constraint, so that the prediction of the low-dimensional representation in the final latent space is more reliable and stable, which effectively improves the performance of the reinforcement learning algorithm in a small sample scenario and improves the sample utilization rate.

[0108] It should be noted that the offline reinforcement learning device based on time reversal symmetry provided in this embodiment can correspond to the offline reinforcement learning method based on time reversal symmetry described in the above embodiments, and will not be described in detail here.

[0109] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute an offline reinforcement learning method based on time reversal symmetry, the method comprising: obtaining the current state and encoding the current state into a low-dimensional representation of the current state; based on a pre-trained policy network, predicting the low-dimensional representation of the next state according to the low-dimensional representation of the current state; based on a pre-trained inverse dynamics model, predicting the low-dimensional representation of the action corresponding to the current state according to the low-dimensional representation of the current state and the low-dimensional representation of the next state; obtaining the target action corresponding to the current state according to the low-dimensional representation of the action corresponding to the current state; wherein the policy network and the inverse dynamics model are obtained by training and optimization based on the time reversal symmetry constraint, and the time reversal symmetry constraint is set based on the forward dynamics model and the inverse dynamics model.

[0110] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0111] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the offline reinforcement learning method based on time reversal symmetry provided by the above-mentioned methods, the method comprising: obtaining the state at a current moment, and encoding the state at a current moment into a low-dimensional representation of the state at a current moment; based on a pre-trained policy network, predicting the low-dimensional representation of the state at a next moment according to the low-dimensional representation of the state at a current moment; based on a pre-trained inverse dynamics model, predicting the low-dimensional representation of the action corresponding to the state at a current moment according to the low-dimensional representation of the state at a current moment and the low-dimensional representation of the state at a next moment; obtaining the target action corresponding to the state at a current moment according to the low-dimensional representation of the action corresponding to the state at a current moment; wherein, the policy network and the inverse dynamics model are obtained by training and optimization based on time reversal symmetry constraints, and the time reversal symmetry constraints are set based on a forward dynamics model and an inverse dynamics model.

[0112] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0113] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An offline reinforcement learning method based on time reversal symmetry, characterized in that: include: Obtain the current state and encode the current state into a low-dimensional representation of the current state; Based on the pre-trained policy network, the low-dimensional representation of the state at the next moment is predicted according to the low-dimensional representation of the state at the current moment; Based on the pre-trained inverse dynamics model, the low-dimensional representation of the action corresponding to the current state is predicted according to the low-dimensional representation of the current state and the low-dimensional representation of the next state. According to the low-dimensional representation of the action corresponding to the current state, obtain the target action corresponding to the current state; The strategy network and the inverse dynamics model are obtained by training and optimization based on time reversal symmetry constraints, and the time reversal symmetry constraints are set based on the forward dynamics model and the inverse dynamics model.

2. The offline reinforcement learning method based on time reversal symmetry according to claim 1, characterized in that: The encoding of the current state into a low-dimensional representation of the current state specifically includes: Based on a pre-trained state encoder, the current state is encoded to obtain a low-dimensional representation of the current state.

3. The offline reinforcement learning method based on time reversal symmetry according to claim 1, characterized in that: Training and optimizing inverse dynamics models, including: Determine an offline data set, wherein the offline data set includes states, actions, and corresponding rewards at consecutive historical moments; Based on the offline data set, the inverse dynamics model is iteratively optimized through the forward dynamics model loss function, the reverse dynamics loss function, the inverse dynamics model loss function and the time reversal symmetry constraint to obtain a pre-trained inverse dynamics model.

4. The offline reinforcement learning method based on time reversal symmetry according to claim 3, characterized in that: Training and optimizing the inverse dynamics model, including: Input the historical current moment state in the offline data set into the state encoder to obtain a low-dimensional representation of the historical current moment state; and input the historical next moment state in the offline data set into the state encoder to obtain a low-dimensional representation of the historical next moment state; The low-dimensional representation of the state at the current historical moment and the low-dimensional representation of the state at the next historical moment are input into the inverse dynamics model to obtain the implicit representation of the corresponding action under the state at the current historical moment; The low-dimensional representation of the state at the current moment of history and the implicit representation of the corresponding action at the current moment of history are input into the forward dynamics model together, and the forward change corresponding to the low-dimensional representation of the state at the current moment of history is obtained; and the low-dimensional representation of the state at the next moment of history and the implicit representation of the corresponding action at the current moment of history are input into the reverse dynamics model together, and the reverse change corresponding to the low-dimensional representation of the state at the current moment of history is obtained; Decoding the forward change amount corresponding to the low-dimensional representation of the state at the current moment of history to obtain the forward change amount of the original state space; and decoding the reverse change amount corresponding to the low-dimensional representation of the state at the current moment of history to obtain the reverse change amount of the original state space; Based on the offline data set, the forward change and reverse change of the original state space obtained by decoding, the low-dimensional representation of the state at the current historical moment and the low-dimensional representation of the state at the next historical moment, and the implicit representation of the corresponding action under the state at the current historical moment, the inverse dynamics model is iteratively optimized to obtain a pre-trained inverse dynamics model.

5. The offline reinforcement learning method based on time reversal symmetry according to claim 3, characterized in that: The forward dynamics loss function is expressed as follows: The inverse dynamics loss function is expressed as follows: The inverse dynamics model loss function is expressed as follows: The time reversal symmetry constraint is expressed as follows: in, Represents the historical current state in the offline dataset. Represents the historical state of the next moment in the offline dataset. Represents the partial differential of the hidden state at the current moment of history to the original dimension state, It represents the partial differential of the hidden state at the next moment in history to the original dimension state. Represents the forward change obtained by decoding, Indicates the reverse change obtained by decoding, It represents the change in the hidden state at the current moment of the history output by the forward dynamics model. A low-dimensional representation of the state of the current moment in history, represents the implicit representation of the corresponding action at the current state of the history output of the inverse dynamics model, Indicates the change in the hidden state at the next moment of the history output of the inverse dynamics model, Indicates the action corresponding to the current state of the history in the offline dataset. A low-dimensional representation of the state at the next moment in history, , represents the reward in the offline dataset, represents the discount factor, The value prediction of the low-dimensional representation of the state at the next moment in history, The value prediction of the low-dimensional representation of the current state of history, Represents an action decoder.

6. The offline reinforcement learning method based on time reversal symmetry according to claim 4, characterized in that: Training optimization strategy network, including: Based on the low-dimensional representation of the state at the current historical moment, the low-dimensional representation of the state at the next historical moment, and the implicit representation of the corresponding action under the state at the current historical moment, the policy network is iteratively optimized through a preset loss function to obtain a pre-trained policy network.

7. The offline reinforcement learning method based on time reversal symmetry according to claim 6, characterized in that: The preset loss function is expressed as follows: in, In latent space The value prediction of represents the policy network, represents the offline dataset, Indicates the current state of history. Indicates the state of the next moment in history. Represents the low-dimensional representation of the policy network based on the current state of history The low-dimensional representation of the predicted state at the next historical moment, represents the weight of the control strategy constraint, A low-dimensional representation of the state at the next moment in history, represents the inversion symmetry constraint, It represents the implicit representation of the corresponding action at the current state of the history output by the inverse dynamics model.

8. An offline reinforcement learning device based on time reversal symmetry, characterized in that: include: An encoding module, used for obtaining the current state and encoding the current state into a low-dimensional representation of the current state; The first prediction module is used to predict the low-dimensional representation of the state at the next moment based on the low-dimensional representation of the state at the current moment based on the pre-trained policy network; A second prediction module is used to predict the low-dimensional representation of the action corresponding to the current state based on the pre-trained inverse dynamics model according to the low-dimensional representation of the current state and the low-dimensional representation of the next state; A target action acquisition module is used to acquire the target action corresponding to the current state according to the low-dimensional representation of the action corresponding to the current state; The strategy network and the inverse dynamics model are obtained by training and optimization based on time reversal symmetry constraints, and the time reversal symmetry constraints are set based on the forward dynamics model and the inverse dynamics model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the offline reinforcement learning method based on time reversal symmetry according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the offline reinforcement learning method based on time reversal symmetry according to any one of claims 1 to 7 is implemented.

Images