Offline reinforcement learning method and apparatus based on time reversal symmetry

Abstract

The present application provides an offline reinforcement learning method and apparatus based on time reversal symmetry. The method comprises: acquiring the state at a current moment, and encoding the state at the current moment into a low-dimensional representation of the state at the current moment; on the basis of a pre-trained policy network and according to the low-dimensional representation of the state at the current moment, predicting a low-dimensional representation of the state at the next moment; on the basis of a pre-trained inverse dynamics model and according to the low-dimensional representation of the state at the current moment and the low-dimensional representation of the state at the next moment, predicting a low-dimensional representation of an action corresponding to the state at the current moment; and on the basis of the low-dimensional representation of the action corresponding to the state at the current moment, acquiring a target action corresponding to the state at the current moment, wherein the policy network and the inverse dynamics model are obtained by training and optimizing on the basis of a time reversal symmetry constraint, and the time reversal symmetry constraint is obtained by performing configurations on a forward dynamics model and an inverse kinematics model.

Description

Offline reinforcement learning method and device based on time reversal symmetry

[0001] Cross-reference to Related Applications

[0002] This application claims priority to Chinese Patent Application No. 202411363121.2, filed on September 27, 2024, entitled “Offline reinforcement learning method and device based on time reversal symmetry”, which is incorporated by reference herein in its entirety. TECHNICAL FIELD

[0003] The present application relates to the field of machine learning, and in particular, to an offline reinforcement learning method and device based on time reversal symmetry. BACKGROUND

[0004] Existing offline reinforcement learning (Offline RL) algorithms face the following challenges (1)-(3).

[0005] (1) Poor performance. Existing offline RL algorithms still perform worse than online RL algorithms on many tasks, especially for complex and long-term dependent environments. This is mainly because offline RL algorithms cannot directly collect samples from the environment to update the model, but only rely on pre-collected historical data, which limits their learning ability.

[0006] (2) Large sample requirement. Offline RL algorithms usually require a large amount of historical data to effectively learn, and the huge sample requirement limits the application of the algorithm in the real world, because collecting high-quality offline data is usually costly and time-consuming.

[0007] (3) Low sample utilization. Existing offline RL algorithms have deficiencies in utilizing historical data. Many times, the training data contains a large amount of noise data and irrelevant information, and the algorithm is difficult to effectively learn useful decision-making strategies from it.

[0008] The above technical challenges seriously hinder the popularization and application of offline reinforcement learning technology, and it is urgent to propose more effective and robust solutions.

[0009] Therefore, it is necessary to solve the problems of poor performance, large sample requirement, and low sample utilization of existing offline reinforcement learning algorithms. SUMMARY

[0010] The present application provides an offline reinforcement learning method and device based on time reversal symmetry to overcome the defects of poor performance, large sample requirement, and low sample utilization of existing offline reinforcement learning algorithms, effectively improve the performance of reinforcement learning algorithms in small sample scenarios, and improve sample utilization.

[0011] In a first aspect, the application provides an offline reinforcement learning method based on time reversal symmetry, comprising: obtaining a current time state and encoding the current time state into a low-dimensional representation of the current time state; predicting a low-dimensional representation of a next time state based on the low-dimensional representation of the current time state according to a pre-trained policy network; predicting a low-dimensional representation of an action corresponding to the current time state based on the low-dimensional representation of the current time state and the low-dimensional representation of the next time state according to a pre-trained inverse dynamics model; and obtaining a target action corresponding to the current time state according to the low-dimensional representation of the action corresponding to the current time state; wherein the policy network and the inverse dynamics model are obtained by training and optimizing based on a time reversal symmetry constraint, and the time reversal symmetry constraint is obtained by setting a forward dynamics model and a reverse dynamics model.

[0012] Further, the encoding of the current time state into a low-dimensional representation of the current time state specifically comprises: encoding the current time state based on a pre-trained state encoder to obtain a low-dimensional representation of the current time state.

[0013] Further, the training and optimization of the inverse dynamics model comprises: determining an offline data set comprising states, actions and corresponding rewards of consecutive historical time states; and iteratively optimizing the inverse dynamics model based on the offline data set through a forward dynamics model loss function, a reverse dynamics loss function, an inverse dynamics model loss function and a time reversal symmetry constraint to obtain the pre-trained inverse dynamics model.

[0014] Further, the inverse dynamics model is trained and optimized, specifically including: inputting the historical current time state in the offline data set into the state encoder to obtain a low-dimensional representation of the historical current time state; and inputting the historical next time state in the offline data set into the state encoder to obtain a low-dimensional representation of the historical next time state; inputting the low-dimensional representation of the historical current time state and the low-dimensional representation of the historical next time state into the inverse dynamics model to obtain an implicit representation of the corresponding action under the historical current time state; inputting the low-dimensional representation of the historical current time state and the implicit representation of the corresponding action under the historical current time state into the forward dynamics model to obtain a forward change corresponding to the low-dimensional representation of the historical current time state; and inputting the low-dimensional representation of the historical next time state and the implicit representation of the corresponding action under the historical current time state into the reverse dynamics model to obtain a reverse change corresponding to the low-dimensional representation of the historical current time state; decoding the forward change corresponding to the low-dimensional representation of the historical current time state to obtain a forward change in the original state space; and decoding the reverse change corresponding to the low-dimensional representation of the historical current time state to obtain a reverse change in the original state space; based on the offline data set, the decoded forward change and reverse change in the original state space, the low-dimensional representation of the historical current time state and the low-dimensional representation of the historical next time state, the implicit representation of the corresponding action under the historical current time state, the inverse dynamics model is iteratively optimized to obtain a pre-trained inverse dynamics model.

[0015] Further, the forward dynamics loss function is represented as follows:

[0016] The reverse dynamics loss function is represented as follows:

[0017] The inverse dynamics model loss function is represented as follows:

[0018] The time reversal symmetry constraint is represented as follows:

[0019] where s represents the historical current time state in the offline data set, s' represents the historical next time state in the offline data set, represents the partial derivative of the historical current time implicit state to the original dimension state, represents the partial derivative of the historical next time implicit state to the original dimension state, represents the decoded forward change, represents the decoded reverse change, represents the change corresponding to the historical current time implicit state output by the forward dynamics model, z s represents the low-dimensional representation of the historical current time state, an action corresponding to the history current time state output by the inverse dynamics model, a change corresponding to the history next time hidden state output by the inverse dynamics model, a represents an action corresponding to the history current time state in the offline data set, and z s′ represents a low-dimensional representation of the history next time state, μ(z s ,z s′ ) represents an inverse dynamics model, ψ a represents an action decoder.

[0020] Further, the policy network is trained, and specifically, the policy network is iteratively optimized based on the low-dimensional representation of the history current time state, the low-dimensional representation of the history next time state, and the hidden representation of the action corresponding to the history current time state through a preset loss function, to obtain a pre-trained policy network.

[0021] Further, the preset loss function is represented as follows:

[0022] wherein V(π g (z s )) represents a value prediction of π g (z s ) in the hidden space, π g represents a policy network, represents an offline data set, s represents a history current time state, s' represents a history next time state, π g (z s ) represents a low-dimensional representation of the history next time state predicted by the policy network according to the low-dimensional representation z s of the history current time state, and α represents a weight of a control policy constraint term, z s′ represents a low-dimensional representation of the history next time state, represents an inversion symmetry constraint, represents a hidden representation of an action corresponding to the history current time state output by the inverse dynamics model.

[0023] In a second aspect, the present application also provides an offline reinforcement learning device based on time reversal symmetry, comprising: an encoding module configured to obtain a current state and encode the current state into a low-dimensional representation of the current state; a first prediction module configured to predict, based on a pre-trained policy network, a low-dimensional representation of a next state according to the low-dimensional representation of the current state; a second prediction module configured to predict, based on a pre-trained inverse dynamics model, a low-dimensional representation of an action corresponding to the current state according to the low-dimensional representation of the current state and the low-dimensional representation of the next state; and a target action obtaining module configured to obtain a target action corresponding to the current state according to the low-dimensional representation of the action corresponding to the current state; wherein the policy network and the inverse dynamics model are obtained by training and optimization based on a time reversal symmetry constraint, and the time reversal symmetry constraint is obtained by setting based on a forward dynamics model and a backward dynamics model.

[0024] In a third aspect, the present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the offline reinforcement learning method based on time reversal symmetry according to any one of the above aspects when executing the computer program.

[0025] In a fourth aspect, the present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the offline reinforcement learning method based on time reversal symmetry according to any one of the above aspects.

[0026] The offline reinforcement learning method based on time reversal symmetry provided by the present application comprises the following steps: obtaining a current state and encoding the current state into a low-dimensional representation of the current state; predicting, based on a pre-trained policy network, a low-dimensional representation of a next state according to the low-dimensional representation of the current state; predicting, based on a pre-trained inverse dynamics model, a low-dimensional representation of an action corresponding to the current state according to the low-dimensional representation of the current state and the predicted low-dimensional representation of the next state; and obtaining a target action corresponding to the current state according to the low-dimensional representation of the action corresponding to the current state through an action decoder; wherein the policy network and the inverse dynamics model are obtained by training and optimization based on a time reversal symmetry constraint, and the time reversal symmetry constraint is obtained by setting based on a forward dynamics model and a backward dynamics model. This method extracts the time reversal symmetry of data in the hidden space, and trains and optimizes the policy network and the inverse dynamics model based on the time reversal symmetry constraint, so that the prediction of the low-dimensional representation in the final hidden space is more reliable and stable, effectively improves the performance of the reinforcement learning algorithm in the small sample scenario, and improves the sample utilization rate. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to make the technical solutions in the application or the prior art clearer, the accompanying drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description are some embodiments of the application, and other drawings can be obtained by a person of ordinary skill in the art without creative effort.

[0028] FIG. 1 is a flowchart of an offline reinforcement learning method based on time reversal symmetry according to an embodiment of the application.

[0029] FIG. 2 is a schematic diagram of training and optimization of an inverse dynamics model according to an embodiment of the application.

[0030] FIG. 3 is a schematic diagram of joint training of an inverse dynamics model and a policy network according to an embodiment of the application.

[0031] FIG. 4 is a schematic diagram of a structure of an offline reinforcement learning device based on time reversal symmetry according to an embodiment of the application.

[0032] FIG. 5 is a schematic diagram of a physical structure of an electronic device according to an embodiment of the application. DETAILED DESCRIPTION

[0033] To make the purposes, technical solutions, and advantages of the application clearer, the technical solutions in the application will be described clearly and completely below with reference to the accompanying drawings in the application. Obviously, the described embodiments are some embodiments of the application, rather than all the embodiments. Based on the embodiments in the application, all other embodiments obtained by a person of ordinary skill in the art without creative effort fall within the protection scope of the application.

[0034] It should be noted that the existing offline reinforcement learning algorithm has the disadvantages of poor performance, large sample demand, and low sample utilization. Therefore, the application provides an offline reinforcement learning method based on time reversal symmetry.

[0035] FIG. 1 shows a flowchart of an offline reinforcement learning method based on time reversal symmetry according to an embodiment of the application.

[0036] As shown in FIG. 1, the method includes steps S110-S140, which will be described in detail below.

[0037] S110, obtaining a current state and encoding the current state into a low-dimensional representation of the current state.

[0038] The current time state refers to the specific situation or configuration of the environment in which the agent is located at the current time, which contains all relevant information required by the agent to make decisions, such as current position, speed, resource quantity, etc., which can be determined according to the actual application scenario and the characteristics of the environment, and is not specifically limited here.

[0039] For example, for a chess game, the current time state may be the positions of all chess pieces on the board; in the case of an autonomous vehicle, the current time state may include the speed, position, direction of the vehicle, the distance of surrounding objects, and other sensor data; in a recommendation system, the current time state may include historical behavior data, preference settings, etc.

[0040] In offline reinforcement learning, the role of the current time state is to provide information about the current environment to the agent so that the agent can learn the optimal decision-making strategy from it.

[0041] It is easy to understand that in this embodiment, the current time state is first obtained, and then the obtained current time state is encoded to obtain a low-dimensional representation of the current time state.

[0042] The current time state can be in the form of a series of numerical features or in the form of an image. When encoding the obtained current time state, this step can use feature extraction, use an embedding layer to reduce dimensionality, or use a convolutional neural network to process the current time state.

[0043] In a specific embodiment, the current time state is encoded using a pre-trained state encoder to obtain a low-dimensional representation of the output current time state.

[0044] It should be noted that using the low-dimensional representation of the current time state for decision-making can reduce computational complexity, improve learning efficiency, and at the same time maintain the key features of the state information.

[0045] On the basis of obtaining and encoding the current time state in step S110, further, step S120 is performed.

[0046] S120, based on a pre-trained policy network, predicting a low-dimensional representation of the next time state according to the low-dimensional representation of the current time state.

[0047] It can be understood that inputting the low-dimensional representation of the current time state into the pre-trained policy network can predict the low-dimensional representation of the next time state.

[0048] It is worth mentioning that the policy network is pre-trained, and during training, the policy network is required not only to generate a value function of the next time state in the latent space as large as possible, but also to generate a next time state in the latent space as close as possible to the corresponding state in the offline data set, while the generated next time state in the latent space also needs to satisfy the time reversal symmetry constraint.

[0049] The time reversal symmetry constraint is set based on the forward dynamics model and the inverse dynamics model.

[0050] Time reversal symmetry is a basic concept in physics, which describes whether the physical law remains unchanged under time reversal operation. In classical mechanics, if the dynamics equation of a system remains unchanged under time reversal transformation, the system has time reversal symmetry. This means that if the time is reversed, the state evolution of the system will occur in reverse order, but the physical law itself will not change.

[0051] In this embodiment, when training the optimized policy network, the time reversal symmetry constraint can help the algorithm better understand the dynamic characteristics of the environment. For example, if a sequence of actions a1, a2, …, aT causes the state s0 to the state sT, then the inverse sequence of actions aT, aT-1, …, a1 should be able to cause the state sT to the state s0, provided that the environment has time reversal symmetry. T T T T-1 T

[0052] When predicting the low-dimensional representation of the next time state, using time reversal symmetry can obtain a more robust state representation. Specifically, if a state representation remains unchanged or changes in a certain predetermined way under time reversal, such a state representation will be more stable.

[0053] At step S120, based on the pre-trained policy network, the low-dimensional representation of the next time state is predicted according to the low-dimensional representation of the current time state, and further, step S130 is performed.

[0054] S130, based on the pre-trained inverse dynamics model, the low-dimensional representation of the current time state corresponding action is predicted according to the low-dimensional representation of the current time state and the low-dimensional representation of the next time state; wherein the policy network and the inverse dynamics model are obtained by training and optimizing based on the time reversal symmetry constraint, and the time reversal symmetry constraint is obtained by setting based on the forward dynamics model and the inverse dynamics model.

[0055] ​​​​​In this embodiment, the role of the inverse dynamics model is to predict the action that leads to the state transition from the given two consecutive states (the current time state and the next time state).

[0056] Specifically, the low-dimensional representation of the current time state and the low-dimensional representation of the next time state are input into the pre-trained inverse dynamics model, and the low-dimensional representation of the action corresponding to the current time state is predicted.

[0057] It is worth mentioning that all operations in this embodiment are carried out in the latent space, therefore, step S110 needs to first encode the current time state into the low-dimensional representation of the current time state, the policy network predicts the low-dimensional representation of the next time state, and the inverse dynamics model also predicts the low-dimensional representation of the action corresponding to the current time state.

[0058] It should be noted that the inverse dynamics model is pre-trained, and during its training process, the prediction output is required to satisfy the time reversal symmetry constraint.

[0059] On the basis of step S130 of predicting the low-dimensional representation of the action corresponding to the current time state according to the low-dimensional representation of the current time state and the low-dimensional representation of the next time state based on the pre-trained inverse dynamics model, further, step S140 is performed.

[0060] S140, according to the low-dimensional representation of the action corresponding to the current time state, obtaining the target action corresponding to the current time state.

[0061] It is easy to understand that the low-dimensional representation of the action corresponding to the current time state predicted by step S130 is the representation in the latent space, therefore, this step needs to decode the low-dimensional representation of the action corresponding to the current time state back to the original action space, and the specific decoding process can be realized by an autoencoder (the inverse operation of the encoding process of step S110).

[0062] It is worth mentioning that the time reversal symmetry constraint is introduced in the training process of the policy network and the inverse dynamics model in this embodiment, which can help to improve the robustness and generalization ability of the model, and the time reversal symmetry constraint can ensure that the performance of the model on the forward and reverse time axes is consistent, thereby better reflecting the physical characteristics of the system.

[0063] In order to realize the time reversal symmetry constraint, this embodiment introduces a forward dynamics model f and a reverse dynamics model g, and these two models jointly constitute the basis of the time reversal symmetry constraint.

[0064] Among them, the forward dynamics model is used to predict the next time state according to the current time state and the current time action, and the reverse dynamics model is used to predict the current time state according to the next time state and the current time action.

[0065] In the embodiment, the target action corresponding to the current time state is obtained by obtaining the current time state, encoding the current time state into a low-dimensional representation of the current time state, predicting a low-dimensional representation of a next time state from the low-dimensional representation of the current time state based on a pre-trained policy network, and predicting a low-dimensional representation of an action corresponding to the current time state from the low-dimensional representation of the current time state and the low-dimensional representation of the next time state based on a pre-trained inverse dynamics model. The policy network and the inverse dynamics model are trained and optimized based on a time reversal symmetry constraint, and the time reversal symmetry constraint is set based on a forward dynamics model and a reverse dynamics model. The method extracts the time reversal symmetry of the data in the hidden space, and trains and optimizes the policy network and the inverse dynamics model based on the time reversal symmetry constraint, so that the prediction of the low-dimensional representation in the final hidden space is more reliable and stable, effectively improving the performance of the reinforcement learning algorithm in the small sample scene, and improving the sample utilization rate.

[0066] On the basis of the above embodiment, further, the training process of the inverse dynamics model and the policy network will be described in detail.

[0067] The inverse dynamics network is trained and optimized, including: determining an offline data set, the offline data set including states, actions and corresponding rewards of consecutive historical time; based on the offline data set, the inverse dynamics model is iteratively optimized through a forward dynamics model loss function, a reverse dynamics loss function, an inverse dynamics model loss function and a time reversal symmetry constraint, to obtain a pre-trained inverse dynamics model.

[0068] FIG. 2 shows a training and optimization diagram of the inverse dynamics model provided by the embodiment of the application.

[0069] As shown in FIG. 2, first, the historical current time state s is input into the state encoder to obtain the low-dimensional representation z s of the historical current time state s, and the historical next time state s' is input into the state encoder to obtain the low-dimensional representation z s′ of the historical next time state s'.

[0070] Then, the low-dimensional representation z s of the historical current time state and the low-dimensional representation z s′ of the historical next time state are input into the inverse dynamics model π inv to obtain the hidden representation z a of the action corresponding to the historical current time state.

[0071] Next, the low-dimensional representation zs and the latent representation z of the corresponding action at the historical current time state a together input into the forward dynamics model f to obtain the low-dimensional representation of the historical current time state corresponding to the forward change and the low-dimensional representation z of the historical next time state s′ and the latent representation z of the corresponding action at the historical current time state a together input into the reverse dynamics model g to obtain the low-dimensional representation of the historical current time state corresponding to the reverse change

[0072] and the low-dimensional representation of the historical current time state corresponding to the forward change is decoded to obtain the forward change in the original state space and the low-dimensional representation of the historical current time state corresponding to the reverse change is decoded to obtain the reverse change in the original state space

[0073] Finally, based on the offline data set decoded to obtain the forward change in the original state space and the reverse change the low-dimensional representation z of the historical current time state s and the low-dimensional representation z of the historical next time state s′ , the latent representation z of the corresponding action at the historical current time state a , the inverse dynamics model is iteratively optimized to obtain the pre-trained inverse dynamics model.

[0074] It should be noted that the iterative optimization process shown in FIG. 2 can simultaneously optimize the state encoder, the inverse dynamics model, the forward dynamics model, the reverse dynamics model, the state decoder, and the action decoder.

[0075] Specifically, this process of iterative optimization can be realized through the forward dynamics model loss function, the reverse dynamics loss function, the inverse dynamics model loss function, and the time reversal symmetry constraint.

[0076] The forward dynamics loss function is represented as follows (1).

[0077] The reverse dynamics loss function is represented as follows (2).

[0078] The inverse dynamics model loss function is represented as follows (3).

[0079] The time reversal symmetry constraint is expressed as formula (4).

[0080] In formula (1)-(4), s represents a historical current time state in an offline data set, s' represents a historical next time state in the offline data set, represents a partial differential of a historical current time hidden state with respect to an original dimension state, represents a partial differential of a historical next time hidden state with respect to an original dimension state, represents a forward change quantity obtained by decoding, represents a reverse change quantity obtained by decoding, represents a change quantity corresponding to a historical current time hidden state output by a forward dynamics model, and z s represents a low-dimensional representation of a historical current time state, represents a hidden representation of a corresponding action under a historical current time state output by an inverse dynamics model (z a ), represents a change quantity corresponding to a historical next time hidden state output by a reverse dynamics model, a represents a corresponding action of a historical current time state in an offline data set, and z s′ represents a low-dimensional representation of a historical next time state, and z s represents a low-dimensional representation of a historical current time state, s′ represents an inverse dynamics model, and ψ a represents an action decoder.

[0081] FIG. 3 shows a schematic diagram of joint training of an inverse dynamics model and a policy network provided by an embodiment of the present application.

[0082] As shown in FIG. 3, the left part of FIG. 3 is the content described in FIG. 2, which is not repeated here.

[0083] For the right part of FIG. 3, a process of iteratively optimizing a policy network is described based on a low-dimensional representation z s of a historical current time state, a low-dimensional representation z s′ of a historical next time state, and a hidden representation z a of a corresponding action under a historical current time state.

[0084] The process of iteratively optimizing the policy network can be realized by a preset loss function, and the preset loss function can refer to formula (5) as follows.

[0085] In formula (5), V(π g (z s )) represents a value prediction of π g (z s ) in a hidden space, π g represents a policy network, denotes the offline dataset, s denotes the historical current time state, s' denotes the historical next time state, π g (z s ) denotes the low-dimensional representation of the historical next time state predicted by the policy network from the low-dimensional representation of the historical current time state z s , and a denotes the weight of the control policy constraint term, z s′ denotes the low-dimensional representation of the historical next time state, denotes the time reversal symmetry constraint, denotes the hidden representation of the corresponding action of the historical current time state output by the inverse dynamics model (z a ).

[0086] The meaning of formula (5) is that when the policy network generates a state, it not only needs to satisfy that the value function of the next time state in the generated hidden space is as large as possible (corresponding to V(π g (z s )), but also needs to satisfy that the next time state in the generated hidden space is as close as possible to the corresponding state in the offline dataset (corresponding to a(π g (z s )-z s′ ). At the same time, the next time state in the generated hidden space also needs to satisfy the time reversal symmetry constraint (corresponding to ).

[0087] Formula (5) involves the value function V, and according to FIG. 3, the learning process of the value function can be represented by the following formulas (6)-(9). μ = r + γV(z s′ )-V(z s ) (9).

[0088] In formulas (6)-(9), denotes the expectation (of the cumulative reward that can be obtained), s denotes the historical current time state in the offline dataset, s' denotes the historical next time state in the offline dataset, r denotes the reward in the offline dataset, denotes the offline dataset, denotes the expected regression loss function, which is used to learn the expected regression value of the current value function, and its expansion is as shown in formulas (8)-(9), γ denotes a discount factor, V(z s′ ) denotes the value prediction of the historical next time state in the hidden space, V(z s ) denotes the value prediction of the historical current time state in the hidden space, k denotes a time step, R k+1 denotes the reward obtained at the k+1 step, and τ is a hyperparameter that controls the height of the expected regression score line; μ(zs z s′ )=r+γV(z s′ )-V(z s ), r represents the reward in the offline dataset, γ represents the discount factor, V(z s′ ) represents the value prediction of the low-dimensional representation of the next time state, and V(z s ) represents the value prediction of the low-dimensional representation of the current time state.

[0089] It should be noted that the state encoder, inverse dynamics model, forward dynamics model, reverse dynamics model, state decoder and action decoder involved in the embodiment can be constructed based on a simple fully connected network, or can be constructed based on a more complex convolutional neural network or recurrent neural network, depending on actual needs, which are not specifically limited here.

[0090] After the training and optimization of the inverse dynamics model and the policy network are completed, they can be deployed to actual application scenarios.

[0091] In actual application, the overall decision-making process can be represented by the following formula (10). a=ψ a (π inv (z s ,π g (z s ))) (10).

[0092] In formula (10), a represents the target action, ψ a represents the action decoder, π inv represents the inverse dynamics model, z s represents the low-dimensional representation of the current time state, and π g represents the policy network.

[0093] According to formula (10), after obtaining the current time state, the current time state is first encoded into the corresponding low-dimensional representation z s ; then, the low-dimensional representation of the current time state is input into the trained policy network π g , and the low-dimensional representation of the next time state can be predicted; then, the low-dimensional representation of the current time state and the predicted low-dimensional representation of the next time state are input into the inverse dynamics model, and the low-dimensional representation of the action corresponding to the current time state can be predicted; finally, the low-dimensional representation of the action corresponding to the current time state is decoded back to the original action space, and the target action corresponding to the current time state can be obtained.

[0094] The target action here depends on the application field and specific task, for example, robot grasping objects, walking, vehicle steering, acceleration, lane changing, turning on and off lights, temperature regulation, playing music, etc., which are not specifically limited here.

[0095] In the embodiment, the inverse dynamics model and the policy network are trained and optimized based on the time reversal symmetry constraint, then, based on the pre-trained policy network, the low-dimensional representation of the next time state is predicted according to the low-dimensional representation of the current time state, and then based on the pre-trained inverse dynamics model, the low-dimensional representation of the action corresponding to the current time state is predicted according to the low-dimensional representation of the current time state and the low-dimensional representation of the next time state, so as to obtain the target action corresponding to the current time state according to the low-dimensional representation of the action corresponding to the current time state; wherein the policy network and the inverse dynamics model are obtained by training and optimizing based on the time reversal symmetry constraint, and the time reversal symmetry constraint is obtained by setting based on the forward dynamics model and the reverse dynamics model. The method extracts the time reversal symmetry of the data in the hidden space, and trains and optimizes the policy network and the inverse dynamics model based on the time reversal symmetry constraint, so that the prediction of the low-dimensional representation in the final hidden space is more reliable and stable, effectively improving the performance of the reinforcement learning algorithm in the small sample scene, and improving the sample utilization rate.

[0096] It is worth mentioning that the offline reinforcement learning method based on time reversal symmetry provided in the embodiments of the application not only can improve the performance of the reinforcement learning algorithm in the small sample scene (not limited to the small sample scene, but the advantage is more obvious in the small sample scene), but also has the characteristics of high sample utilization rate and easy deployment, and is more in line with the actual algorithm landing demand.

[0097] Corresponding to the offline reinforcement learning method based on time reversal symmetry described in the above embodiments, the application further provides an offline reinforcement learning device based on time reversal symmetry. Specifically, FIG. 4 shows a structural schematic diagram of the offline reinforcement learning device based on time reversal symmetry provided by the embodiments of the application.

[0098] As shown in FIG. 4, the device comprises: an encoding module 410 configured to obtain a current time state and encode the current time state into a low-dimensional representation of the current time state; a first prediction module 420 configured to predict a low-dimensional representation of a next time state based on a pre-trained policy network according to the low-dimensional representation of the current time state; a second prediction module 430 configured to predict a low-dimensional representation of an action corresponding to the current time state based on a pre-trained inverse dynamics model according to the low-dimensional representation of the current time state and the low-dimensional representation of the next time state; and a target action obtaining module 440 configured to obtain a target action corresponding to the current time state according to the low-dimensional representation of the action corresponding to the current time state; wherein the policy network and the inverse dynamics model are obtained by training and optimizing based on the time reversal symmetry constraint, and the time reversal symmetry constraint is obtained by setting based on the forward dynamics model and the reverse dynamics model.

[0099] In the embodiment, the current time state is acquired by the encoding module 410, and the current time state is encoded into a low-dimensional representation of the current time state. The first prediction module 420 predicts a low-dimensional representation of a next time state based on a pre-trained policy network according to the low-dimensional representation of the current time state. Then, the second prediction module 430 predicts a low-dimensional representation of an action corresponding to the current time state based on a pre-trained inverse dynamics model according to the low-dimensional representation of the current time state and the low-dimensional representation of the next time state. Thus, the target action acquisition module 440 acquires a target action corresponding to the current time state according to the low-dimensional representation of the action corresponding to the current time state. The policy network and the inverse dynamics model are obtained by training and optimization based on a time reversal symmetry constraint, and the time reversal symmetry constraint is obtained by setting based on a forward dynamics model and a reverse dynamics model. The device extracts the time reversal symmetry of data in the hidden space, and trains and optimizes the policy network and the inverse dynamics model based on the time reversal symmetry constraint, so that the prediction of the low-dimensional representation in the final hidden space is more reliable and stable, effectively improves the performance of the reinforcement learning algorithm in the small sample scene, and improves the sample utilization rate.

[0100] It should be noted that the offline reinforcement learning device based on the time reversal symmetry provided in the embodiment can be correspondingly referred to the offline reinforcement learning method based on the time reversal symmetry described in the above embodiments, which will not be described here.

[0101] FIG. 5 illustrates an entity structure diagram of an electronic device. As shown in FIG. 5, the electronic device can include a processor 510, a communications interface 520, a memory 530, and a communications bus 540. The processor 510, the communications interface 520, and the memory 530 can communicate with each other through the communications bus 540. The processor 510 can invoke a logical instruction in the memory 530 to execute the offline reinforcement learning method based on the time reversal symmetry, which includes: acquiring a current time state and encoding the current time state into a low-dimensional representation of the current time state; predicting a low-dimensional representation of a next time state based on a pre-trained policy network according to the low-dimensional representation of the current time state; predicting a low-dimensional representation of an action corresponding to the current time state based on a pre-trained inverse dynamics model according to the low-dimensional representation of the current time state and the low-dimensional representation of the next time state; and acquiring a target action corresponding to the current time state according to the low-dimensional representation of the action corresponding to the current time state. The policy network and the inverse dynamics model are obtained by training and optimization based on a time reversal symmetry constraint, and the time reversal symmetry constraint is obtained by setting based on a forward dynamics model and a reverse dynamics model.

[0102] In addition, the logic instructions in the memory 530 described above can be implemented in the form of a software functional unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0103] In another aspect, the present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement an offline reinforcement learning method based on time reversal symmetry provided by the above method, the method comprising: obtaining a current time state and encoding the current time state into a low-dimensional representation of the current time state; predicting a low-dimensional representation of a next time state based on the low-dimensional representation of the current time state according to a pre-trained policy network; predicting a low-dimensional representation of an action corresponding to the current time state based on the low-dimensional representation of the current time state and the low-dimensional representation of the next time state according to a pre-trained inverse dynamics model; obtaining a target action corresponding to the current time state according to the low-dimensional representation of the action corresponding to the current time state; wherein the policy network and the inverse dynamics model are obtained by training and optimizing based on a time reversal symmetry constraint, and the time reversal symmetry constraint is obtained by setting based on a forward dynamics model and a reverse dynamics model.

[0104] The device embodiments described above are only schematic, wherein the units illustrated as separate components can or can not be physically separate, and the components illustrated as units can or can not be physical units, i.e., they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment. Those skilled in the art can understand and implement without creative labor.

[0105] Those skilled in the art can clearly understand the implementation of the various embodiments by means of software and necessary general hardware platforms through the description of the above embodiments, and of course, the various embodiments can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that contributes to the technical solutions can be embodied in the form of a software product. The computer software product can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0106] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An offline reinforcement learning method based on time reversal symmetry, comprising: obtaining a current time state and encoding the current time state into a low-dimensional representation of the current time state; based on a pre-trained policy network, predicting a low-dimensional representation of a next time state according to the low-dimensional representation of the current time state; based on a pre-trained inverse dynamics model, predicting a low-dimensional representation of an action corresponding to the current time state according to the low-dimensional representation of the current time state and the low-dimensional representation of the next time state; obtaining a target action corresponding to the current time state according to the low-dimensional representation of the action corresponding to the current time state; wherein the policy network and the inverse dynamics model are trained and optimized based on a time reversal symmetry constraint, and the time reversal symmetry constraint is set based on a forward dynamics model and a reverse dynamics model.

2. The off-line reinforcement learning method based on time reversal symmetry of claim 1, wherein, The encoding of the current time state into a low-dimensional representation of the current time state specifically includes: based on a pre-trained state encoder, encoding the current time state to obtain a low-dimensional representation of the current time state.

3. The off-line reinforcement learning method based on time reversal symmetry of claim 1, wherein, Training and optimizing the inverse dynamics model includes: determining an offline data set, the offline data set including states, actions and corresponding rewards of consecutive historical time states; based on the offline data set, iteratively optimizing the inverse dynamics model through a forward dynamics model loss function, a reverse dynamics loss function, an inverse dynamics model loss function and a time reversal symmetry constraint to obtain a pre-trained inverse dynamics model.

4. The off-line reinforcement learning method based on time reversal symmetry of claim 3, wherein, Training and optimizing the inverse dynamics model specifically includes: inputting the historical current time state in the offline data set into the state encoder to obtain a low-dimensional representation of the historical current time state; and inputting the historical next time state in the offline data set into the state encoder to obtain a low-dimensional representation of the historical next time state; inputting the low-dimensional representation of the historical current time state and the low-dimensional representation of the historical next time state into the inverse dynamics model to obtain a hidden representation of the action corresponding to the historical current time state; inputting the low-dimensional representation of the historical current time state and the hidden representation of the action corresponding to the historical current time state into the forward dynamics model to obtain a forward change amount corresponding to the low-dimensional representation of the historical current time state; and inputting the low-dimensional representation of the historical next time state and the hidden representation of the action corresponding to the historical current time state into the reverse dynamics model to obtain a reverse change amount corresponding to the low-dimensional representation of the historical current time state; decoding the forward change amount corresponding to the low-dimensional representation of the historical current time state to obtain a forward change amount in the original state space; and decoding the reverse change amount corresponding to the low-dimensional representation of the historical current time state to obtain a reverse change amount in the original state space; based on the offline data set, the decoded forward change amount and reverse change amount in the original state space, the low-dimensional representation of the historical current time state and the low-dimensional representation of the historical next time state, and the hidden representation of the action corresponding to the historical current time state, iteratively optimizing the inverse dynamics model to obtain the pre-trained inverse dynamics model.

5. The off-line reinforcement learning method based on time reversal symmetry of claim 3, wherein, The forward dynamics loss function is represented as follows: The reverse kinetic loss function is represented as follows: The inverse dynamics model loss function is represented as follows: The time reversal symmetry constraint is expressed as follows: wherein s represents a historical current time state in the offline dataset, s' represents a historical next time state in the offline dataset, denotes the partial derivative of the history current time hidden state with respect to the original dimension state, denotes the partial derivative of the hidden state at the next time instance with respect to the original dimension state at the previous time instance, represents a decoded forward change amount, represents the decoded reverse change amount, a change in the historical current time hidden state output by the forward dynamic model, z s a low-dimensional representation of the historical current time state, a history of the current time state of the inverse dynamics model output, denotes the change in the hidden state at the next time step corresponding to the history, a denotes the action corresponding to the state at the current time step in the offline dataset, z s′ denotes the low-dimensional representation of the state at the next time step in the history, μ(z s ,z s′ ) denotes the inverse dynamics model, ψ a denotes the action decoder.

6. The off-line reinforcement learning method based on time reversal symmetry of claim 4, wherein, Training and optimizing the policy network specifically includes: The policy network is pre-trained by iteratively optimizing the policy network based on the low-dimensional representation of the historical current time state, the low-dimensional representation of the historical next time state, and the latent representation of the corresponding action under the historical current time state through a preset loss function.

7. The off-line reinforcement learning method based on time reversal symmetry of claim 6, wherein, The preset loss function is represented as follows: where V(π g (z s )) represents the value prediction of π g (z s ) in the latent space, and π g represents the policy network, denotes the offline dataset, s denotes the historical current time step state, s' denotes the historical next time step state, π g (z s ) denotes the low-dimensional representation of the historical next time step state predicted by the policy network from the low-dimensional representation of the historical current time step state z s , and a denotes the weight of the control policy constraint term. s′ denotes the low-dimensional representation of the historical next time step state, denotes an inversion symmetry constraint, The latent representation of the corresponding action under the historical current time state representing the output of the inverse dynamics model.

8. An offline reinforcement learning device based on time reversal symmetry, comprising: An encoding module configured to obtain a current time state and encode the current time state into a low-dimensional representation of the current time state; A first prediction module configured to predict a low-dimensional representation of a next time state based on the low-dimensional representation of the current time state according to a pre-trained policy network; A second prediction module configured to predict a low-dimensional representation of a corresponding action of the current time state based on the low-dimensional representation of the current time state and the low-dimensional representation of the next time state according to a pre-trained inverse dynamics model; A target action obtaining module configured to obtain a target action corresponding to the current time state according to the low-dimensional representation of the corresponding action of the current time state. The policy network and the inverse dynamics model are trained and optimized based on a time reversal symmetry constraint, and the time reversal symmetry constraint is obtained based on a forward dynamics model and a reverse dynamics model.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, The processor executes the computer program to implement the offline reinforcement learning method based on time reversal symmetry according to any one of claims 1 to 7.

10. A non-transitory computer readable storage medium having stored thereon a computer program, wherein, The computer program is executed by the processor to implement the offline reinforcement learning method based on time reversal symmetry according to any one of claims 1 to 7.

Images