Offline reinforcement learning method and device based on dynamic symmetry modeling

Abstract

The application provides an offline reinforcement learning method and device based on kinetic symmetry modeling, wherein the method comprises the following steps: obtaining an offline data set; processing original state-action pairs in the offline data set through an encoder and a bidirectional kinetic model to obtain low-dimensional state-action pairs and a current time-reversal state change amount; and inputting the low-dimensional state-action pairs into a value network to obtain a state-action value function value corresponding to the original state-action pairs. The method processes offline data by using the bidirectional kinetic model, extracts effective information, i.e., the time-reversal state change amount, from the offline data, and the information has high generalization and interpretability. The state-action value function value obtained by taking the effective information as the input of the value network can effectively assist reinforcement learning to obtain a policy that maximizes long-term rewards, and thus the optimization of the behavior policy is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning, and particularly relates to an offline reinforcement learning method and device based on dynamic symmetry modeling. BACKGROUND

[0002] At present, most reinforcement learning algorithms can only learn a good strategy after experiencing a large number of trials and errors in a simulated environment, such as in the game field and the robot field. However, in real-world scenarios, such as autonomous driving cars and complex industrial system control, there is no perfect simulation environment, but only a batch of pre-collected environment interaction data, namely offline data.

[0003] In offline reinforcement learning, since the agent cannot interact with the environment, the learning difficulty is greatly increased, and a series of problems are also derived therefrom. The existing representative offline reinforcement learning methods include SAC (Soft Actor-Critic) algorithm, CQL (Conservative Q-Learning) algorithm and BAIL (Best-Action Imitation Learning) algorithm.

[0004] Among them, the SAC algorithm introduces the idea of entropy in the traditional reinforcement learning actor-critic framework, and uses a soft update method to make the training process of the agent more stable, and the final performance is also better. Although this algorithm can be directly migrated to the offline reinforcement learning scene, it has poor adaptability and often has fatal problems such as divergence of Q function value in experiments, and is not suitable for direct application.

[0005] The CQL algorithm imposes an additional restriction on the Q function, strictly controls the update of the Q function outside the data set, and avoids the problem of overestimation of the Q function by limiting the Q function. However, correspondingly, the Q function corresponding to the CQL algorithm is often too pessimistic, and the strategy obtained is too conservative, and there is a certain distance from the optimal strategy, and the CQL algorithm is sensitive to hyperparameters. Very fine adjustment of the hyperparameters is required to achieve good results.

[0006] The BAIL algorithm uses the idea of imitation learning, completely discards the actor-critic framework commonly used in reinforcement learning, and does not contain the Q function, thereby fundamentally avoiding the error problem of the Q function. In addition, the algorithm selects good data points by using the upper envelope of all data points in the training data set, and uses these data points for imitation learning, thereby obtaining a good strategy.

[0007] However, although the BAIL algorithm is not subject to the Q function, the envelope method of the calculation thereof needs to estimate the future expected return of each point in the data set, and such estimation has certain errors, especially for an infinite time sequence scenario, the errors will be greater.

[0008] Therefore, the problem of extracting effective information from offline data to assist reinforcement learning to obtain a policy that maximizes long-term rewards in the prior art is an important problem to be solved in the field of machine learning technology. SUMMARY

[0009] The present application provides an offline reinforcement learning method and device based on dynamic symmetry modeling, to overcome the defects in the prior art that effective information cannot be extracted from offline data to assist reinforcement learning to obtain a policy that maximizes long-term rewards, and to realize optimization of behavior policy.

[0010] In one aspect, the present application provides an offline reinforcement learning method based on dynamic symmetry modeling, for policy optimization, comprising: obtaining an offline data set; processing original state-action pairs in the offline data set through an encoder and a bidirectional dynamic model to obtain low-dimensional state-action pairs and current time reversal state change amounts; inputting the low-dimensional state-action pairs into a value network to obtain a state-action value function value corresponding to the original state-action pairs; wherein the current time reversal state change amount comprises a forward time reversal state change amount and a reverse time reversal state change amount.

[0011] Further, the processing of the original state-action pairs in the offline data set through the encoder and the bidirectional dynamic model to obtain the low-dimensional state-action pairs and the current time reversal state change amount comprises: processing the original state-action pairs through the encoder to obtain the low-dimensional state-action pairs; processing the current time low-dimensional state-action pairs and the next time low-dimensional state through the bidirectional dynamic model respectively, and obtaining the corresponding forward reversal state change amount and reverse reversal state change amount through a preset loss function. Forward time reversal state change amount

[0012] Further, the preset loss function comprises a dynamic loss function, a dynamic loss function in a low-dimensional space, and a reconstruction loss function; wherein the formulas of the dynamic loss function, the dynamic loss function in the low-dimensional space, and the reconstruction loss function are as follows in turn:

[0013]

[0014]

[0015]

[0016] wherein s is a state in the offline dataset, a is an action under the state s in the offline dataset, ΔX(s, s′) is the state change amount at the next time, s′ is the state at the next time in the offline dataset, θ and is a hyperparameter, is the forward time reversal state change amount obtained by processing the bidirectional dynamic model, is the next time state-action change amount obtained by decoding the forward time reversal state change amount, is the forward time reversal state change amount obtained by direct encoding by the encoder, and X(s, a) is the original state-action pair in the offline dataset, is the original state-action pair obtained by decoding the low-dimensional state-action pair.

[0017] Further, the inputting of the preset input data into the value network to obtain the state-action value function value corresponding to the original state-action pair comprises: calculating a next time low-dimensional policy state change amount according to a low-dimensional state in the low-dimensional state-action pair and a low-dimensional policy output action; calculating a forward time reversal state change amount according to a low-dimensional state in the low-dimensional state-action pair and a low-dimensional representation of the original state at the next time; and obtaining the state-action value function value by taking a difference between the next time low-dimensional policy state change amount and the forward time reversal state change amount as a preset constraint term.

[0018] Further, the behavior policy corresponding to the original state in the original state-action pair is obtained by taking the state-action value function value and the difference value between the low-dimensional policy action and the low-dimensional action in the low-dimensional state-action pair as constraint terms, or the behavior policy corresponding to the original state in the original state-action pair is obtained by taking the state-action value function value and the difference value between the forward time reversal state change quantity obtained based on the low-dimensional state-action pair and the forward time reversal state change quantity obtained based on the low-dimensional state in the low-dimensional state-action pair and the low-dimensional policy action as constraint terms, or the behavior policy corresponding to the original state in the original state-action pair is obtained by taking the state-action value function value, the difference value between the forward time reversal state change quantity obtained based on the low-dimensional state-action pair and the forward time reversal state change quantity obtained based on the low-dimensional state in the low-dimensional state-action pair and the low-dimensional policy action, and the difference value between the low-dimensional policy action and the low-dimensional action in the low-dimensional state-action pair as constraint terms, or the behavior policy corresponding to the original state in the original state-action pair is obtained by taking the state-action value function value, the difference value between the forward time reversal state change quantity obtained based on the low-dimensional state-action pair and the forward time reversal state change quantity obtained based on the low-dimensional state in the low-dimensional state-action pair and the low-dimensional policy action, the difference value between the low-dimensional policy action and the low-dimensional action in the low-dimensional state-action pair, and the difference value between the low-dimensional policy action and the original action in the original state-action pair as constraint terms.

[0019] Further, the bidirectional dynamics model is trained by a time reversal loss function.

[0020] In a second aspect, the present application further provides an offline reinforcement learning device based on modeling of dynamics symmetry, for policy optimization, comprising: a data acquisition module for acquiring an offline data set; a data processing module for processing an original state-action pair in the offline data set through an encoder and a bidirectional dynamics model to obtain a low-dimensional state-action pair and a current time reversal state change quantity; a value acquisition module for inputting the low-dimensional state-action pair into a value network to obtain a state-action value function value corresponding to the original state-action pair; wherein the current time reversal state change quantity comprises a forward time reversal state change quantity and a reverse time reversal state change quantity.

[0021] In a third aspect, the present application further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the offline reinforcement learning method based on modeling of dynamics symmetry according to any one of the above aspects when executing the program.

[0022] In a fourth aspect, the present application also provides a non-transitory computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of any of the above-mentioned off-policy reinforcement learning method based on modeling of dynamical symmetries.

[0023] In a fifth aspect, the present application also provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of any of the above-mentioned off-policy reinforcement learning method based on modeling of dynamical symmetries.

[0024] The off-policy reinforcement learning method based on modeling of dynamical symmetries provided by the present application processes the original state-action pairs in the obtained offline dataset by using an encoder and a bidirectional dynamical model to obtain a low-dimensional representation of the original state-action pairs and a current time-reversal state change amount, and inputs the low-dimensional representation of the original state-action pairs into a value network to obtain a state-action value function value corresponding to the original state-action pairs. The method uses the bidirectional dynamical model to process the offline data, extracts effective information, i.e., the forward time-reversal state change amount and the reverse time-reversal state change amount, from the offline data, and these information have high generalization and interpretability. The state-action value function value obtained by using the effective information as the input of the value network can effectively assist the reinforcement learning to obtain a policy that maximizes the long-term reward and achieve the optimization of the behavior policy. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0026] Figure 1 A flowchart of the off-policy reinforcement learning method based on modeling of dynamical symmetries provided by the present application is shown in the figure.

[0027] Figure 2 A prediction schematic diagram of the bidirectional dynamical model provided by the present application is shown in the figure.

[0028] Figure 3 An acquisition schematic diagram of the state-action value function value provided by the present application is shown in the figure.

[0029] Figure 4 A schematic diagram of the overall flow of the off-policy reinforcement learning method based on modeling of dynamical symmetries provided by the present application is shown in the figure.

[0030] Figure 5A structural schematic diagram of the off-line reinforcement learning device based on the kinetic symmetry modeling provided by the present application is shown in the figure.

[0031] Figure 6 A structural schematic diagram of the electronic device provided by the present application is shown in the figure. DETAILED DESCRIPTION

[0032] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0033] Figure 1 A flowchart of the off-line reinforcement learning method based on the kinetic symmetry modeling provided by the present application is shown in the figure. As shown in the figure, the off-line reinforcement learning method comprises the following steps. Figure 1

[0034] S101, obtaining an off-line data set;

[0035] It can be understood that the off-line data set is composed of original state-action pairs at different time points. The original state-action pair comprises an original state and an original action, and there is a one-to-one correspondence between the original state and the original action. In addition, the off-line data set can also include rewards corresponding to the original state-action pair.

[0036] It should be noted that the off-line data set can be simulation data obtained by simulation, or interaction data based on a real environment, or both simulation data and real data.

[0037] It should also be noted that all data in the off-line data set can be directly obtained from existing large databases, such as the Robomimic database and the D4RL database. The Robomimic database is an off-line database related to a robot arm; and the D4RL database collects large data sets, including records of agents in interactive environments, such as the interactive records of autonomous driving Carla, AntMaze and Mujoco, and the database also classifies the data sets simply and complexly, and the categories are also very rich.

[0038] Taking autonomous driving as an example, when the off-line data set is derived from the data records of autonomous driving, the vehicle driving environment information is the original state in the off-line data set, and the vehicle operation behavior information is the original action in the off-line data set. Of course, the off-line data set can also include reward information corresponding to the vehicle driving environment information and the vehicle operation behavior information. ​

[0039] S102, processing the original state-action pairs in the offline data set through the encoder and the bidirectional dynamic model to obtain low-dimensional state-action pairs and current time reversal state change amounts, the current time reversal state change amounts including forward time reversal state change amounts and reverse time reversal state change amounts.

[0040] The encoder can be a convolution-based encoding network for compressing the original state-action pairs in the offline data set and related data extended from the original state-action pairs, so as to compress the high-dimensional original data into low-dimensional data. The related data extended from the original state-action pairs includes a next time state-action change amount, that is, a change amount of the original state and the original action at the next time.

[0041] It should be noted that, unlike general deep learning prediction models, the bidirectional dynamic model is a bidirectional dynamic model that approximates the total differential equation of the current system. The total differential equation describes how the state changes over time in the current dynamic system. Therefore, the bidirectional dynamic model focuses on the change amount of the state in the time dimension. The general deep learning prediction model is based on fitting the transition probability between data according to the data distribution, and does not focus on the feature that data changes in the time dimension. The bidirectional dynamic model is a deep learning network model with a preset number of network layers, wherein the preset number can be set according to actual needs, and is not specifically set here.

[0042] In this embodiment, the reliability of the state can be determined by using the bidirectional dynamic model, thereby increasing the generalization and robustness of the algorithm. At the same time, the model is also used for a certain degree of reliability data enhancement, which expands the data coverage of the original data, thereby enhancing the effect of the proposed reinforcement learning algorithm.

[0043] Specifically, the bidirectional dynamic model can perform bidirectional prediction, that is, the model can not only predict the change amount of the next time state relative to the current time state according to the action made at the current time state, but also predict the change amount of the current time state relative to the next time state according to the action corresponding to the current time state made at the next time state, thereby realizing the prediction of the forward and reverse state change amounts on the time axis.

[0044] It should be noted that the forward time reversal state change amount obtained by processing the bidirectional dynamic model has more generalization and interpretability than the original state-action pairs of the original dimension.

[0045] The generalization refers to that the low-dimensional data obtained through the processing of the bidirectional dynamic model can be applied to various scenes, and accurate predictions can be made based on the low-dimensional data, for example, the accurate prediction of the forward time reversal state change amount in the embodiment, and the corresponding state-action value function value can be obtained based on the low-dimensional data. The explainability refers to that each parameter in the low-dimensional data obtained through the processing of the bidirectional dynamic model is explainable.

[0046] It can be understood that, on the basis of obtaining the offline data set in the above step S101, the original state-action pair in the offline data set is extracted, and the original state-action pair is processed by using the encoder and the bidirectional dynamic model, so as to obtain the low-dimensional state-action pair, the forward time reversal state change amount and the reverse time reversal state change amount. The low-dimensional state-action pair is a low-dimensional representation of the original state-action pair, the forward time reversal state change amount is a state change amount of a corresponding action under a state at a current time, and the reverse time reversal state change amount is a state change amount of a current action under a state at a next time.

[0047] It should be noted that the forward time reversal state change amount and the reverse time reversal state change amount are both low-dimensional representations.

[0048] It should be further noted that the processing of the original state-action pair by using the encoder and the bidirectional dynamic model also includes processing of data obtained by extending the original state-action pair, and the data obtained by extending the original state-action pair can be a next time state-action change amount.

[0049] S103, inputting the low-dimensional state-action pair into a value network to obtain a state-action value function value corresponding to the original state-action pair.

[0050] The value network in the present step refers to a state-action value network, which is used to obtain the value of performing a certain action under a certain state, that is, to obtain the state-action value function value of each state-action pair.

[0051] It can be understood that, on the basis of obtaining the low-dimensional state-action pair, the forward time reversal state change amount and the reverse time reversal state change amount in the above step S102, the low-dimensional state-action pair is input into the value network, so that the state-action value function value corresponding to the original state-action pair can be obtained.

[0052] It should be noted that the forward time reversal state change amount and the reverse time reversal state change amount obtained in step S103 are used to set corresponding policy constraints to obtain an optimized behavior policy.

[0053] Taking autonomous driving as an example, after obtaining the state-action value function value corresponding to the vehicle driving environment information and the vehicle operation behavior information, the policy network can be optimized based on the state-action value function value to obtain a behavior strategy that maximizes the reward, that is, the operation behavior of the vehicle, so that the vehicle can be more efficiently and accurately controlled according to the optimized behavior strategy.

[0054] In this embodiment, the original state-action pairs in the obtained offline data set are processed by the encoder and the bidirectional dynamics model to obtain low-dimensional representations of the original state-action pairs, and the forward time reversal state change amount and the reverse time reversal state change amount. By inputting the preset input data including the low-dimensional representations of the original state-action pairs and / or the time reversal state change amounts into the value network, the state-action value function value corresponding to the original state-action pairs can be obtained. This method uses the bidirectional dynamics model to process the offline data, extracts effective information, i.e., the forward time reversal state change amount and the reverse time reversal state change amount, from the offline data. These information have high generalization and interpretability. The state-action value function value obtained by using the effective information as the input of the value network can effectively assist the reinforcement learning to obtain a policy that maximizes the long-term reward, and realize the optimization of the behavior policy.

[0055] On the basis of the above-mentioned embodiments, further, the original state-action pairs in the offline data set are processed by the encoder and the bidirectional dynamics model to obtain low-dimensional state-action pairs, forward time reversal state change amounts and reverse time reversal state change amounts, including: the original state-action pairs are processed by the encoder to obtain low-dimensional state-action pairs; the low-dimensional state-action pairs and the next time low-dimensional state are processed by the bidirectional dynamics model respectively, and the corresponding forward time reversal state change amounts and reverse time reversal state change amounts are obtained by a preset loss function.

[0056] It can be understood that the original state-action pairs in the offline data set are processed by the encoder and the bidirectional dynamics model. Specifically, the original state-action pairs are compressed by the encoder to obtain low-dimensional representations of the original state-action pairs, i.e., low-dimensional state-action pairs. The low-dimensional representations of the original state-action pairs include low-dimensional representations of the original state-action pairs at different times, for example, the current time and the next time.

[0057] After obtaining the low-dimensional state-action pairs, the current time low-dimensional state-action pairs are processed by the bidirectional dynamics model, so that the forward time reversal state change amount can be predicted; the next time low-dimensional state is processed by the bidirectional dynamics model, so that the reverse time reversal state change amount can be obtained.

[0058] The low-dimensional state-action pair and the next time low-dimensional state are processed by using a bidirectional dynamics model, and specifically, a preset loss function is obtained.

[0059] The formula of the preset loss function is as follows:

[0060] L total =λ1L ldl +λ2L dl +λ3L rl

[0061] L ldl is a dynamic loss function, L dl is a dynamic loss function in a low-dimensional space, L rl is a reconstruction loss function, λ1, λ2, and λ3 are weight coefficients of the three loss functions respectively.

[0062] According to the above, the preset loss function includes the dynamic loss function, the dynamic loss function in the low-dimensional space, and the reconstruction loss function.

[0063] The dynamic loss function is used to ensure that the forward time reversal state change quantity can decode the original next time state-action change quantity. The formula of the dynamic loss function is as follows:

[0064]

[0065] In the formula, s is a state in an offline data set, a is an action under the state s in the offline data set, ΔX(s, s') is a next time state change quantity, s' is a next time state in the offline data set, θ and are hyperparameters, is a forward time reversal state change quantity obtained by processing by the bidirectional dynamics model, is a next time state-action change quantity obtained by decoding the forward time reversal state change quantity.

[0066] The dynamic loss function in the low-dimensional space is used to ensure the accuracy of the forward time reversal state change quantity predicted by the bidirectional dynamics model. The formula of the dynamic loss function in the low-dimensional space is as follows:

[0067]

[0068] In the formula, is a forward time reversal state change quantity obtained by directly encoding by an encoder. The loss function takes the forward time reversal state change quantity obtained by directly encoding by the encoder as a positive label, and the closer, the more accurate the predicted More accurate.

[0069] The reconstruction loss function is used to ensure that the original state-action pair obtained by decoding the low-dimensional state-action pair can be reconstructed back to the original state-action pair in the offline dataset. The formula of the reconstruction loss function is as follows:

[0070]

[0071] In the formula, X(s, a) is the original state-action pair in the offline dataset, is the original state-action pair obtained by decoding the low-dimensional state-action pair.

[0072] According to the above, the preset loss function can be expanded as:

[0073]

[0074] Figure 2 The bidirectional dynamics model prediction schematic diagram provided by the application is shown, which further explains Figure 1 the processing process of step S102. As Figure 2 shown, first, based on the offline dataset, the original state-action pair X(s, a) is extracted from the offline dataset, and the next time state s' is added, that is, X(s, a, s') is extracted from the offline dataset, and according to the original state-action pair at different times, the next time state change amount ΔX(s, s') is obtained by subtracting the current time original state s from the next time original state s'.

[0075] The original state-action pair is processed by the encoding layer (i.e. the encoder mentioned above) to obtain the low-dimensional state-action pair Z(X), that is, Z(s, a), and the next time original state s' and the current time original action a are processed by the encoder, that is, Z(s', a) can be obtained.

[0076] Then, the low-dimensional state-action pair Z(s, a) is processed by the bidirectional dynamics model, that is, the low-dimensional state-action pair Z(s, a) is input into the bidirectional dynamics model, and the forward time reversal state change amount

[0077] Similarly, the low-dimensional representation of the next time low-dimensional state and the current time action is processed by the bidirectional dynamics model, that is, the low-dimensional representation of the next time low-dimensional state and the current time action, that is, Z(s', a) in the bidirectional dynamics model is input into the bidirectional dynamics model, and the inverse time reversal state change amount Figure 2 ​

[0078] Specifically, in the process of the bidirectional dynamics model processing the low-dimensional state at the current time, the low-dimensional state at the next time, and the low-dimensional action at the current time, the predicted corresponding forward time reversal state change and reverse time reversal state change can be obtained through the dynamic loss function and the dynamic loss function in the low-dimensional space.

[0079] Specifically, in the process of the bidirectional dynamics model processing the low-dimensional state-action pair and the low-dimensional state at the next time, the dynamic loss function and the dynamic loss function in the low-dimensional space are involved to make the predicted forward time reversal state change and reverse time reversal state change have extremely high generalization and interpretability.

[0080] Finally, the encoding layer and the decoding layer in the bidirectional dynamics model are constantly optimized and updated through the preset loss function including the dynamic loss function, the dynamic loss function in the low-dimensional space, and the reconstruction loss function, so as to improve the prediction accuracy of the processing result of the bidirectional dynamics model. Figure 2

[0081] According to the bidirectional dynamics model, Figure 2 It can also be seen that the bidirectional dynamics model utilizes the time reversal loss function in the learning process, wherein the expression of the time reversal loss function is as follows:

[0082]

[0083] Wherein, λ1, λ2, λ3, λ4, λ5 are weights of corresponding loss functions, λ6 is a neural network parameter regularization weight, θ is a forward dynamics model parameter, β is a reverse dynamics model parameter, is an encoder parameter, and ψ is a decoder parameter, is a change of the current low-dimensional state in the time dimension obtained by the full differential equation, and Δs is a change of the state at the next time and the state at the current time, is a forward time reversal state change, is a change of the next low-dimensional state in the time dimension obtained by the full differential equation, is a reverse time reversal state change, is a decoding of the forward low-dimensional state change to the original dimension forward state change, is a decoding of the reverse low-dimensional state change to the original dimension reverse state change, D ψ (z(s,a,s′)) decodes the current state, low-dimensional action, and next time state to the original dimension current state, action, and next time state, ||θ1 is L1 regularization of the forward dynamics model network parameter, and ||β1 is L1 regularization of the reverse dynamics model network parameter, ​L1 regularization for encoder network parameters, ||ψ||1 for decoder network parameters.

[0084] In this embodiment, a bidirectional dynamics model is used to process the low-dimensional state-action pair and the low-dimensional state at the next time step. The corresponding forward time-reversed state change and backward time-reversed state change are obtained through a preset loss function. The obtained forward time-reversed state change and backward time-reversed state change have extremely high generalization and interpretability. The state-action value function obtained by using it as input to the value network can effectively assist reinforcement learning in obtaining a strategy that maximizes long-term reward and achieves optimization of the behavior strategy.

[0085] Based on the above embodiments, further, preset input data is input into the value network to obtain the state-action value function value corresponding to the original state-action pair, including: calculating the change in the low-dimensional policy state at the next time step based on the low-dimensional state and the low-dimensional policy output action in the low-dimensional state-action pair; calculating the forward time-inversion state change based on the low-dimensional state and its corresponding low-dimensional action in the low-dimensional state-action pair; and obtaining the state-action value function value by using the difference between the change in the low-dimensional policy state at the next time step and the forward time-inversion state change as a preset constraint term.

[0086] Understandably, pre-set input data is fed into the value network to obtain the state-action value function value corresponding to the original state-action pair. Specifically, based on the low-dimensional state and low-dimensional policy output action in the low-dimensional state-action pair, the change in the low-dimensional policy state at the next time step is calculated. Here, the low-dimensional policy output action is the low-dimensional representation of the policy output action.

[0087] It should be noted that the policy output action refers to the output action of the policy network when the original state is used as input. The low-dimensional policy output action is obtained by compressing the policy output action through an encoder.

[0088] The formula for obtaining the change in the low-dimensional policy state at the next moment is as follows:

[0089]

[0090] in, For state s t+1 Below, a low-dimensional representation of the state at the next time step, obtained by performing an action chosen by the policy. For the original state s t The low-dimensional representation of is the low-dimensional state in the low-dimensional state-action pair.

[0091] Based on the low-dimensional state in the low-dimensional state-action pair and the low-dimensional representation of the original state at the next moment, the forward time-reversal state change is calculated.

[0092] The formula for obtaining the state change in the forward time-reversal is as follows:

[0093]

[0094] in, The initial state s at the next moment t+1 The low-dimensional representation, For the original state s t The low-dimensional representation of .

[0095] Based on the changes in the low-dimensional policy state at the next time step and the changes in the forward time inversion state, the difference between the two is used as a preset constraint term to obtain the state-action value function value. This preset constraint term can be understood as a penalty term, the purpose of which is to ensure that the obtained state-action value corresponds to the actual value of the original state and the original action in the original state-action pair.

[0096] The optimization process of the original value network can be represented by the following formula:

[0097]

[0098] In this embodiment, the process of optimizing the value network using the aforementioned preset constraints as constraints can be expressed by the following formula:

[0099] Where r(s,a) is the immediate reward for performing action a in state s, and z s′ For a low-dimensional representation of the state at the next time step, a π z' is a low-dimensional representation of the action generated by the policy in the next time step. s z is a low-dimensional representation of the current state. a Here, γ is a low-dimensional representation of the action at the current moment, and γ is a hyperparameter. In this embodiment, the value network is optimized and updated by using the difference between the low-dimensional policy state change at the next moment and the forward time-reversed state change as a preset constraint. This makes the state-value function output by the value network more accurate, thereby effectively assisting reinforcement learning in obtaining a policy that maximizes long-term rewards and achieving optimization of the behavioral policy.

[0100] Figure 3 The diagram illustrates the acquisition of the state-action value function provided by this invention, further explaining the case where the value network uses low-dimensional state-action pairs as input.

[0101] like Figure 3As shown, by changing the input of the value network in the existing reinforcement learning method, the corresponding state-action value function value is obtained. Specifically, based on the original state-action pair X(s, a) in the offline data set as the input of the encoding layer, a low-dimensional representation of the original state-action pair is obtained, and the low-dimensional representation Z(X) of the original state-action pair is input into the value network, so that the corresponding state-action value function value Q(Z(X)) is obtained.

[0102] It should be noted that in the process of optimizing and updating the value network, the related network parameters involved in the encoding layer in the figure are not updated. Moreover, the reinforcement learning algorithm provided in the embodiment can be applied to any existing algorithm including a state-action value function.

[0103] On the basis of the above embodiment, further, the offline reinforcement learning method further comprises:

[0104] Based on the obtained state-action value function value, a constraint term on the policy is set to obtain a behavior policy corresponding to the original state.

[0105] It should be noted that in the prior art, the constraint term set on the policy is the difference of the original action, which can be specifically expressed by the following formula:

[0106]

[0107] Wherein, The meaning of is the behavior policy corresponding to the maximum expected value of the original state-action, and is the obtained behavior policy, represents the expectation based on the state-action pair in the data set, D represents the offline data set, Q(s, a π ) is the state-action value function value, and λ is a hyperparameter, (a π -a) is the constraint term set, which specifically represents the difference of the original action, that is, the difference between the policy output action a π and the original action a.

[0108] In the embodiment, different constraint terms on the policy are set based on the original state-action pair and the obtained state-action value function value, and the low-dimensional data obtained by processing through the encoder and the bidirectional dynamics model.

[0109] Specifically, the constraint term 1 on the policy: by taking the state-action value function value and the difference value between the low-dimensional policy action and the low-dimensional action in the low-dimensional state-action pair as the constraint term, a behavior policy corresponding to the original state in the original state-action pair is obtained. Specifically, it can be expressed by the following formula:

[0110]

[0111] wherein, is a low-dimensional state-action pair, a is a low-dimensional action in the low-dimensional state-action pair,

[0112] Alternatively, constraint term 2 on the policy: obtain the behavior policy corresponding to the original state in the original state-action pair by taking the difference value between the state-action value function value and the forward time reversal state change quantity based on the low-dimensional state-action pair and the forward time reversal state change quantity based on the low-dimensional state in the low-dimensional state-action pair and the low-dimensional policy action as the constraint term. Specifically, it can be expressed by the following formula:

[0113]

[0114] wherein, is the forward time reversal state change quantity based on the low-dimensional state in the low-dimensional state-action pair and the low-dimensional policy action, is the forward time reversal state change quantity based on the low-dimensional state-action pair.

[0115] Alternatively, constraint term 3 on the policy: obtain the behavior policy corresponding to the original state in the original state-action pair by taking the difference value between the state-action value function value, the forward time reversal state change quantity based on the low-dimensional state-action pair and the forward time reversal state change quantity based on the low-dimensional state in the low-dimensional state-action pair and the low-dimensional policy action, and the difference value between the low-dimensional policy action and the low-dimensional action in the low-dimensional state-action pair as the constraint term. Specifically, it can be expressed by the following formula:

[0116]

[0117] Alternatively, constraint term 4 on the policy: obtain the behavior policy corresponding to the original state in the original state-action pair by taking the difference value between the state-action value function value, the forward time reversal state change quantity based on the low-dimensional state-action pair and the forward time reversal state change quantity based on the low-dimensional state in the low-dimensional state-action pair and the low-dimensional policy action, the difference value between the low-dimensional policy action and the low-dimensional action in the low-dimensional state-action pair, and the difference value between the low-dimensional policy action and the original action in the original state-action pair as the constraint term. Specifically, it can be expressed by the following formula:

[0118]

[0119] It should be noted that the four different constraint terms on the policy set in the embodiment are or relationship, that is, when obtaining the behavior policy corresponding to the original state in the original state-action pair, any one of the above constraint terms 1-4 on the policy can be selected.

[0120] Among them, the behavioral strategy obtained based on constraint 3 of the above strategy has the best effect.

[0121] In this embodiment, different policy constraints are set based on the original state-action pair, the obtained state-action value function value, and the low-dimensional data processed by the encoder and bidirectional dynamics model, thereby obtaining the behavior policy corresponding to the original state and achieving more comprehensive and efficient optimization of the behavior policy.

[0122] in addition, Figure 4 The diagram illustrates the overall process of the offline reinforcement learning method based on dynamic symmetry modeling provided by this invention.

[0123] like Figure 4 As shown, it is mainly divided into two stages. First, the original state-action pairs are processed through a bidirectional dynamic model to obtain corresponding low-dimensional data with extremely high generalization and interpretability. Then, the corresponding constraints are set using this low-dimensional data to learn the state-action value function (i.e., the value network mentioned above) and obtain the state-action value function value Q(Z(X)).

[0124] Then, based on the obtained state-action value function, and combined with the obtained low-dimensional data, policy constraints are set to optimize and update the policy network, thereby obtaining a behavior policy π that maximizes the reward value.

[0125] In summary, the overall flowchart of the offline reinforcement learning method based on dynamic symmetry modeling provided in this embodiment can be summarized as a process of "original state - action pair processing - state - action value function - behavior policy".

[0126] Figure 5 A schematic diagram of the offline reinforcement learning device based on dynamic symmetry modeling provided by this invention is shown. Figure 5 As shown, the offline reinforcement learning device includes: a data acquisition module 501 for acquiring an offline dataset; a data processing module 502 for processing the original state-action pairs in the offline dataset through an encoder and a bidirectional dynamics model to obtain low-dimensional state-action pairs and current time-reversed state changes; and a value acquisition module for inputting preset input data into a value network to obtain the state-action value function value corresponding to the original state-action pairs; wherein, the current time-reversed state changes include forward time-reversed state changes and backward time-reversed state changes.

[0127] The device for offline reinforcement learning based on modeling of dynamic symmetry provided by the present application can correspond to the method for offline reinforcement learning based on modeling of dynamic symmetry described above, and will not be described here again.

[0128] In the present embodiment, the original state-action pairs in the obtained offline data set are processed by the encoder and the bidirectional dynamic model to obtain low-dimensional representations of the original state-action pairs, and the forward time-reversal state change amount and the reverse time-reversal state change amount, and the low-dimensional representations of the original state-action pairs are input into the value network to obtain the state-action value function value corresponding to the original state-action pairs. The device uses the bidirectional dynamic model to process the offline data, extracts effective information, i.e., the forward time-reversal state change amount and the reverse time-reversal state change amount, from the offline data, and these information have high generalization and interpretability. The state-action value function value obtained by taking the effective information as the input of the value network can effectively assist the reinforcement learning to obtain a policy that maximizes the long-term reward, and realize the optimization of the behavior policy.

[0129] Figure 6 An example of a schematic diagram of a physical structure of an electronic device is shown in FIG. 6. Figure 6 As shown in FIG. 6, the electronic device can include a processor 610, a communications interface 620, a memory 630, and a communications bus 640, wherein the processor 610, the communications interface 620, and the memory 630 can communicate with each other through the communications bus 640. The processor 610 can invoke the logic instructions in the memory 630 to execute the method for offline reinforcement learning based on modeling of dynamic symmetry, which includes: obtaining an offline data set; processing original state-action pairs in the offline data set by an encoder and a bidirectional dynamic model to obtain low-dimensional state-action pairs and a current time-reversal state change amount; inputting the low-dimensional state-action pairs into a value network to obtain a state-action value function value corresponding to the original state-action pairs; wherein the current time-reversal state change amount includes a forward time-reversal state change amount and a reverse time-reversal state change amount.

[0130] In addition, the logic instructions in the memory 630 described above can be implemented in the form of software functional units and sold or used as independent products, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0131] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to execute the off-line reinforcement learning method based on the modeling of dynamic symmetry provided by the above-mentioned methods, the method comprising: obtaining an off-line data set; processing original state-action pairs in the off-line data set through an encoder and a bidirectional dynamic model to obtain low-dimensional state-action pairs and a current time-reversal state change amount; inputting the low-dimensional state-action pairs into a value network to obtain a state-action value function value corresponding to the original state-action pairs; wherein the current time-reversal state change amount comprises a forward time-reversal state change amount and a reverse time-reversal state change amount.

[0132] In another aspect, the present application also provides a non-transitory computer readable storage medium, which stores a computer program, the computer program is executed by a processor to implement the off-line reinforcement learning method based on the modeling of dynamic symmetry provided by the above-mentioned methods, the method comprising: obtaining an off-line data set; processing original state-action pairs in the off-line data set through an encoder and a bidirectional dynamic model to obtain low-dimensional state-action pairs and a current time-reversal state change amount; inputting the low-dimensional state-action pairs into a value network to obtain a state-action value function value corresponding to the original state-action pairs; wherein the current time-reversal state change amount comprises a forward time-reversal state change amount and a reverse time-reversal state change amount.

[0133] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0134] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0135] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An off-line reinforcement learning method based on modeling of dynamical symmetries, characterized in that, For policy optimization, comprising: Obtaining an offline dataset; Processing original state-action pairs in the offline dataset through an encoder and a bidirectional dynamics model to obtain low-dimensional state-action pairs and a current time reversal state change amount; Inputting the low-dimensional state-action pairs into a value network to obtain a state-action value function value corresponding to the original state-action pairs; Wherein, the current time reversal state change amount includes a forward time reversal state change amount and a reverse time reversal state change amount, and the forward time reversal state change amount and the reverse time reversal state change amount are used to set a corresponding policy constraint term to obtain an optimized behavior policy.

2. The off-line reinforcement learning method based on modeling of dynamical symmetries according to claim 1, characterized in that, The processing of the original state-action pairs in the offline dataset through the encoder and the bidirectional dynamics model to obtain the low-dimensional state-action pairs and the current time reversal state change amount comprises: Processing the original state-action pairs through the encoder to obtain the low-dimensional state-action pairs; Using the bidirectional dynamics model to process the current time low-dimensional state-action pairs and the next time low-dimensional state respectively, and obtaining the corresponding forward time reversal state change amount and reverse time reversal state change amount through a preset loss function.

3. The off-line reinforcement learning method based on modeling of dynamical symmetries according to claim 2, characterized in that, The preset loss function includes a dynamic loss function, a low-dimensional space dynamic loss function, and a reconstruction loss function; wherein the formulas of the dynamic loss function, the low-dimensional space dynamic loss function, and the reconstruction loss function are as follows: where s is a state in the offline dataset, a is an action under state s in the offline dataset, ΔX(s, s′) is a state change amount at the next time, s′ is a state at the next time in the offline dataset, θ and is a hyperparameter, is the forward-time inverse state change amount obtained by processing the bidirectional dynamics model, is a next-time state-action change amount obtained by decoding the forward-time inverse state change amount, is a forward-time inverse state change amount obtained by direct encoding by the encoder, X(s, a) is an original state-action pair in the offline dataset, is an original state-action pair obtained by decoding the low-dimensional state-action pair.

4. The off-line reinforcement learning method based on modeling of dynamical symmetries according to claim 1, characterized in that, The inputting of the low-dimensional state-action pairs into the value network to obtain the state-action value function value corresponding to the original state-action pairs comprises: According to the low-dimensional state in the low-dimensional state-action pairs and the low-dimensional policy output action, a next time low-dimensional policy state change amount is calculated; According to the low-dimensional state in the low-dimensional state-action pairs and the low-dimensional representation of the next time original state, a forward time reversal state change amount is calculated; The difference between the next time low-dimensional policy state change amount and the forward time reversal state change amount is taken as a preset constraint term to obtain the state-action value function value.

5. The off-line reinforcement learning method based on modeling of dynamical symmetries according to claim 4, characterized in that, Further comprising: Taking the state-action value function value, and the difference between the low-dimensional policy action and the low-dimensional action in the low-dimensional state-action pairs as a constraint term, an action policy corresponding to the original state in the original state-action pairs is obtained; Or, Taking the state-action value function value, and the difference between the forward time reversal state change amount based on the low-dimensional state-action pairs and the forward time reversal state change amount based on the low-dimensional state in the low-dimensional state-action pairs and the low-dimensional policy action as a constraint term, an action policy corresponding to the original state in the original state-action pairs is obtained; Or, The behavior policy corresponding to the original state in the original state-action pair is obtained by taking the state-action value function value, the difference value between the forward time reversal state change quantity obtained based on the low-dimensional state-action pair and the forward time reversal state change quantity obtained based on the low-dimensional state in the low-dimensional state-action pair and the low-dimensional policy action, and the difference value between the low-dimensional policy action and the low-dimensional action in the low-dimensional state-action pair as a constraint term. Alternatively, The behavior policy corresponding to the original state in the original state-action pair is obtained by taking the state-action value function value, the difference value between the forward time reversal state change quantity obtained based on the low-dimensional state-action pair and the forward time reversal state change quantity obtained based on the low-dimensional state in the low-dimensional state-action pair and the low-dimensional policy action, the difference value between the low-dimensional policy action and the low-dimensional action in the low-dimensional state-action pair, and the difference value between the low-dimensional policy action and the original action in the original state-action pair as a constraint term.

6. The off-line reinforcement learning method based on modeling of dynamical symmetries according to claim 1, characterized in that, Further comprising: The bidirectional dynamics model is trained by a time reversal loss function.

7. An off-line reinforcement learning apparatus based on modeling of dynamical symmetries, characterized by, For policy optimization, comprising: A data acquisition module is configured to acquire an offline data set; A data processing module is configured to process an original state-action pair in the offline data set by an encoder and a bidirectional dynamics model to obtain a low-dimensional state-action pair and a current time reversal state change quantity; A value acquisition module is configured to input the low-dimensional state-action pair into a value network to obtain a state-action value function value corresponding to the original state-action pair; The current time reversal state change quantity includes a forward time reversal state change quantity and a reverse time reversal state change quantity, and the forward time reversal state change quantity and the reverse time reversal state change quantity are used to set corresponding policy constraint terms to obtain an optimized behavior policy.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to realize the steps of the offline reinforcement learning method based on the modeling of the symmetry of dynamics according to any one of claims 1 to 6. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the offline reinforcement learning method based on the modeling of the symmetry of dynamics according to any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the offline reinforcement learning method based on the modeling of the symmetry of dynamics according to any one of claims 1 to 6.

Images