A cross-domain offline reinforcement learning method and device based on contrast representation

Abstract

This invention discloses a cross-domain offline reinforcement learning method and apparatus based on contrastive representation, relating to the field of reinforcement learning. It addresses the shortcomings of existing cross-domain offline reinforcement learning methods, such as errors in explicit estimation of the dynamic model when given finite target domain data, and the inability of the domain classifier to smoothly estimate dynamic bias, potentially leading to unboundedness problems. The method includes: obtaining a contrastive learning objective based on positive samples sampled from the target domain dataset, negative samples sampled from the source domain dataset, and a first mutual information; optimizing the state-action encoder and subsequent state encoder based on the simplified learning objective to obtain an information density based on a fractional function and the dot product representation of the two encoders; obtaining a data filtering function based on the information density; filtering the source domain dataset using the data filtering function to obtain extracted samples; and inputting the extracted samples and the target domain dataset data into an offline reinforcement learning model to optimize the value function.

Description

Technical Field

[0001] This invention relates to the field of reinforcement learning, and more specifically to a cross-domain offline reinforcement learning method and apparatus based on contrastive representation. Background Technology

[0002] In offline reinforcement learning, agents learn policies from a fixed dataset without additional online interaction with the environment, effectively saving time and money in the data collection process. However, in specific real-world scenarios such as autonomous driving and healthcare, the testing and training environments are often different, making the collection of sufficient offline data with good state transition coverage extremely time-consuming and expensive. A promising solution is to employ cross-domain offline reinforcement learning, which utilizes additional source domain datasets with different state transition dynamics to compensate for the lack of target domain data, thereby improving data effectiveness.

[0003] Cross-domain offline reinforcement learning is a method to improve the performance of offline reinforcement learning by combining limited target domain data with some source domain data that contains dynamic biases. Its main purpose is to alleviate the problem of target domain data shortage by sharing source domain data, thereby increasing the data effectiveness of the target domain. However, due to the dynamic bias between the source and target domains, simply merging the offline datasets of the two domains may lead to performance degradation and the inability to achieve stable policy improvements. Therefore, cross-domain offline reinforcement learning faces two key challenges: how to effectively quantify dynamic biases and how to utilize cross-domain data.

[0004] To address the first problem, existing methods either directly use offline datasets to estimate the dynamic model or train domain discriminators to approximate the dynamic differences. However, given the limited data in the target domain, directly estimating the dynamic model results in significant extrapolation errors, while domain discriminators cannot provide a smooth measurement of dynamic differences, potentially leading to an unbounded problem (i.e., when two domains are significantly mismatched, the dynamic differences are not fully understood). (This could potentially increase indefinitely), making it impossible to obtain an effective estimate. To address the second problem, existing methods either use dynamic bias as a compensation term to modify the reward function, apply pessimistic support constraints to the source domain data (i.e., select source domain samples with high transition probabilities for training and perform pessimistic estimation of the action-value function of the source domain), or employ data sharing to filter source domain data with smaller dynamic biases. Despite these advances, these methods typically experience rapid performance degradation when faced with larger dynamic differences.

[0005] In summary, existing cross-domain offline reinforcement learning methods suffer from several drawbacks. When given a finite target domain of data, explicit estimation of the dynamic model introduces errors, and the domain classifier cannot smoothly estimate dynamic biases, potentially leading to unboundedness issues. Summary of the Invention

[0006] This invention provides a cross-domain offline reinforcement learning method and apparatus based on contrastive representation, which addresses the problems of existing cross-domain offline reinforcement learning methods, such as errors in explicit estimation of dynamic models when given finite target domain data, and the inability of domain classifiers to smoothly estimate dynamic biases, which may lead to unbounded problems.

[0007] This invention provides a cross-domain offline reinforcement learning method based on contrastive representation, comprising:

[0008] The state-action pairs and subsequent states sampled from the offline dataset are input into the state-action encoder and the subsequent state encoder, respectively, to obtain the first mutual information and the difference between the two-domain mutual information based on the state-action pairs and subsequent states; the contrastive learning target is obtained based on the positive samples sampled from the target domain dataset, the negative samples sampled from the source domain dataset, and the first mutual information.

[0009] Maximizing the contrastive learning objective yields an approximate representation of the difference between the mutual information of the two domains and an equivalent simplified learning objective. Based on the simplified learning objective, the state-action encoder and the subsequent state encoder are optimized to obtain the information density based on the fractional function and the dot product representation of the two encoders.

[0010] A data filtering function is obtained based on the information density. The source domain dataset is filtered using the data filtering function to obtain extracted samples. The extracted samples and the target domain dataset are then input into an offline reinforcement learning model to optimize the value function.

[0011] Preferably, obtaining the difference between the first mutual information and the two-domain mutual information based on the state-action pair and the subsequent state specifically includes:

[0012] Based on the first mutual information, target domain mutual information and source domain mutual information are obtained, and the difference between the two domain mutual information is obtained based on the target domain mutual information and the source domain mutual information.

[0013] The first mutual information is as follows:

[0014]

[0015] The difference in mutual information between the two domains is shown below:

[0016]

[0017] in, Represents the state-action pairs in the offline dataset. Joint distribution and subsequent states The first mutual information between them This represents the expectation of all sample values ​​in the offline dataset D. This indicates the current state action pair. Indicates the subsequent state. Indicates the current state action pair and subsequent states The joint probability distribution function, Indicates the current state action pair The marginal probability distribution function, Indicates subsequent state The marginal probability distribution function, This represents the mutual information between the target domain dataset and its subsequent states regarding the state and actions within the target domain dataset. This represents the mutual information between the state-action pairs in the source domain dataset and their subsequent states. This represents the difference in mutual information between the two domains.

[0018] Preferably, the subsequent states in the offline dataset include a first subsequent state from the target domain dataset and a second subsequent state from the source domain dataset;

[0019] The step of obtaining the contrastive learning target based on positive samples sampled from the target domain dataset, negative samples sampled from the source domain dataset, and the first mutual information specifically includes:

[0020] A state transition sample is formed by sampling state-action pairs and a first subsequent state from the target domain dataset, and the state transition sample is determined as a positive sample; a state-action pair sampled from the target domain dataset and a second subsequent state sampled from the source domain dataset are determined as negative samples;

[0021] The contrastive learning objective is as follows:

[0022]

[0023]

[0024]

[0025] in, A fractional function representing the ratio of information density in the quantized target domain. A fractional function representing the ratio of the information density of the quantized source domain. This represents the empirical dynamic transition function in the target domain dataset. This represents the empirical dynamic transition function in the source domain dataset. This represents the normalized state distribution of the target domain dataset. This represents the normalized state distribution of the source domain dataset. This represents the expectation of the positive sample values ​​within the target domain dataset. This represents the expectation of the negative sample values ​​within the source domain dataset. , , .

[0026] Preferably, maximizing the contrastive learning objective to obtain an approximate representation of the difference between the mutual information of the two domains and a simplified learning objective equivalent to the contrastive learning objective specifically includes:

[0027] Maximizing the contrastive learning objective yields an approximate estimate of the difference in mutual information between the two domains, as shown below:

[0028]

[0029] The simplified learning objective is as follows:

[0030]

[0031] in, This indicates the number of negative samples. Defined as the pessimistic lower bound , With dynamic ratio In comparison, it is a relatively tighter lower bound for the mutual information difference between two domains, and it becomes tighter as K increases. , , This represents the expectation of all sample values ​​within the source domain dataset. This represents the empirical dynamic transition function in the target domain dataset. This represents the empirical dynamic transition function in the source domain dataset. This indicates a simplified learning objective. This represents the expectation of the positive sample values ​​within the target domain dataset. This represents the expectation of the negative sample values ​​within the source domain dataset. This represents the score function corresponding to positive samples from the target domain. This indicates that all negative and positive samples from the source domain are selected. This represents the score function corresponding to all samples.

[0032] Preferably, the information density based on the fractional function and the dot product of the two encoders is as follows:

[0033]

[0034] The data filtering function is as follows:

[0035]

[0036] in, Represents approximate information density. This indicates a state-action encoder. Indicates the subsequent state encoder, This represents the defined data filtering function. This indicates an indicator function that returns 1 if the condition within the parentheses is true, and 0 otherwise. The threshold of the fractional function representing the top percentage.

[0037] Preferably, the value function is as follows:

[0038]

[0039]

[0040] in, This represents the expectation of all sample values ​​within the target domain dataset. This represents the expectation of all sample values ​​in the source domain dataset. This represents the Bellman loss function used for value function learning. This represents the importance coefficient of the TD-error weighted using a fractional function. This represents the Bellman operator used for offline reinforcement learning. This represents the action-value function for offline reinforcement learning. Representing approximate information density, This indicates the data filtering function.

[0041] Preferably, after obtaining the two-domain mutual information difference based on the state-action pair and the subsequent state, the method further includes:

[0042] When the extracted sample obtained from the source domain dataset is added to the target domain dataset, if there is a large deviation between the source domain dataset and the target domain dataset, then the probability of the state in which the extracted sample appears in the target domain dataset tends to zero, and the dynamic deviation between the source domain dataset and the target domain dataset tends to be infinitesimal.

[0043] The dynamic deviation between the source domain dataset and the target domain dataset is shown below:

[0044]

[0045] in, This represents the dynamic deviation between the source domain dataset and the target domain dataset. This indicates calculating the expected value for all samples within the source domain dataset. For the empirical dynamic transition function of the target domain dataset, This is the empirical dynamic transfer function for the source domain dataset.

[0046] This invention provides a cross-domain offline reinforcement learning device based on contrastive representation, comprising:

[0047] The first obtaining unit is used to input the state-action pair and subsequent state sampled from the offline dataset into the state-action encoder and the subsequent state encoder, respectively, to obtain the first mutual information and the difference between the two-domain mutual information based on the state-action pair and the subsequent state; and to obtain the contrastive learning target based on the positive samples sampled from the target domain dataset, the negative samples sampled from the source domain dataset, and the first mutual information.

[0048] The second obtaining unit is used to maximize the contrastive learning objective to obtain an approximate representation of the difference between the mutual information of the two domains and an equivalent simplified learning objective. Based on the simplified learning objective, the state action encoder and the subsequent state encoder are optimized to obtain the information density based on the fractional function and the dot product representation of the two encoders.

[0049] An optimization unit is used to obtain a data filtering function based on the information density, filter the source domain dataset according to the data filtering function to obtain extracted samples, and input the extracted samples and the target domain dataset into an offline reinforcement learning model to optimize the value function.

[0050] This invention provides a computer device, which includes a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs any of the above-described cross-domain offline reinforcement learning methods based on contrastive representation.

[0051] This invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform any of the above-described cross-domain offline reinforcement learning methods based on contrastive representation.

[0052] This invention provides a cross-domain offline reinforcement learning method and apparatus based on contrastive representation. The method includes: inputting state-action pairs and subsequent states sampled from an offline dataset into a state-action encoder and a subsequent state encoder, respectively, to obtain a first mutual information and the difference between the mutual information of the two domains based on the state-action pairs and subsequent states; obtaining a contrastive learning objective based on positive samples sampled from a target domain dataset, negative samples sampled from a source domain dataset, and the first mutual information; maximizing the contrastive learning objective to obtain an approximate representation of the difference between the mutual information of the two domains and an equivalent simplified learning objective; optimizing the state-action encoder and the subsequent state encoder based on the simplified learning objective to obtain an information density based on a fractional function and the dot product representation of the two encoders; obtaining a data filtering function based on the information density; filtering the source domain dataset based on the data filtering function to obtain extracted samples; and inputting the extracted samples and the target domain dataset data into an offline reinforcement learning model to optimize the value function. This method calculates mutual information between two domains by utilizing the joint empirical distribution between state-action pairs and the next state in offline datasets, and establishes a quantitative relationship between the difference in mutual information between the two domains and dynamic bias. This measurement method is more robust than previous methods that directly calculate dynamic ratios, avoiding the unbounded problem that exists when dynamic bias is large. By introducing a variational estimator based on a neural network, the method can effectively estimate mutual information in high-dimensional state space, and obtain a tighter lower bound for pessimistic estimation by approximating information density. The filtered state transition pairs are sorted by the dot product of the two encoders to extract samples for data sharing, selectively sharing source domain data with smaller dynamic bias for training. By using a fractional function to weight the temporal difference error of the filtered data, it is more convenient than modifying the value function for pessimistic estimation, and further improves the performance of the algorithm. This method solves the problems of existing cross-domain offline reinforcement learning methods, such as errors in explicit estimation of dynamic models when given finite target domain data, and the inability of domain classifiers to smoothly estimate dynamic bias, which may lead to unbounded problems. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 A flowchart of a cross-domain offline reinforcement learning method based on contrastive representation provided in an embodiment of the present invention;

[0055] Figure 2A flowchart of a cross-domain offline reinforcement learning method based on contrastive representation provided in an embodiment of the present invention;

[0056] Figure 3 A schematic diagram illustrating the scenario of mutual information difference between two domains of shared data provided in an embodiment of the present invention;

[0057] Figure 4 This is a schematic diagram of a cross-domain offline reinforcement learning device based on contrastive representation, provided as an embodiment of the present invention. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] In the embodiments of the present invention, the technical terms involved are as follows:

[0060] Mutual information: Mutual information (MI) between two random variables measures the degree of interdependence between them. Specifically, for two random variables, mutual information is the amount of "information" (usually measured in bits) that a random variable loses due to knowledge of the other random variable. Let random variables... It is space Given a pair of random variables. If their joint distribution is... The edge distributions are respectively and Then the mutual information between them It can be defined as:

[0061] (1)

[0062] in, This is the KL divergence (Kullback–Leibler divergence). According to the properties of the KL divergence, if the joint distribution... equal to marginal distribution and The product of , then That is, when and When they are independent, their mutual information is 0. This can be expressed in expected form as follows:

[0063] (2)

[0064] The solution process provided in the embodiments of the present invention is as follows: Figure 1 As shown, it is mainly divided into two stages: contrastive representation learning and data filtering.

[0065] Contrastive representation learning: First, a batch of training samples is randomly sampled from the target domain and source domain datasets respectively. Each sample contains the current state. ,action and subsequent states We will sample the target domain data. Input state-action encoder, and simultaneously select subsequent states from the target domain data. As a positive sample, the subsequent state of the source domain data The negative sample is input into the subsequent state encoder; the state action encoder and the subsequent state encoder are optimized by minimizing the contrastive learning objective, so that the dot product of the two encoders can effectively distinguish positive and negative samples in the encoding space and quantize the dynamic deviation between the two domain data.

[0066] Data filtering: Using the encoder dot product obtained above as a fractional function, the source domain data is filtered to select samples with smaller dynamic biases as extraction samples. These extraction samples (shared source domain data) are then added as additional input to the target domain dataset for offline reinforcement learning training, thereby reducing the impact of inter-domain data bias on model learning.

[0067] Figure 2 This invention provides a cross-domain offline reinforcement learning method based on contrastive representation; the following is combined with... Figure 1 and Figure 2 As shown, this invention provides a detailed description of a cross-domain offline reinforcement learning method based on contrastive representation, as illustrated in the embodiments below. Figure 2 As shown, the method mainly includes the following steps:

[0068] Step 101: Input the state-action pairs and subsequent states sampled from the offline dataset into the state-action encoder and the subsequent state encoder, respectively, to obtain the first mutual information and the difference between the two domain mutual information based on the state-action pairs and subsequent states; obtain the contrastive learning target based on the positive samples sampled from the target domain dataset, the negative samples sampled from the source domain dataset, and the first mutual information.

[0069] Step 102: Minimize the contrastive learning objective to obtain an approximate representation of the difference between the mutual information of the two domains and a simplified learning objective equivalent to the contrastive learning objective. Optimize the state-action encoder and the subsequent state encoder based on the simplified learning objective to obtain the information density based on the fractional function and the dot product representation of the two encoders.

[0070] Step 103: Obtain a data filtering function based on the information density, filter the source domain dataset according to the data filtering function to obtain extracted samples, and input the extracted samples and the target domain dataset into an offline reinforcement learning model to optimize the value function.

[0071] In the comparative representation phase, specifically step 101, mutual information is first attempted to capture the dynamic transition correlations between different domains using an offline dataset. The experience stored in the middle is the state transition tuple A set, where each tuple contains a state. ,action ,award and the state at the next moment The state-action pair can be centralized through two offline datasets. and subsequent states The joint empirical distribution is used to estimate the mutual information, as shown below:

[0072] (3)

[0073] in, Represents offline datasets Actions in the middle state Joint distribution and subsequent states Mutual information between them This represents the expectation of all sample values ​​in the offline dataset. This indicates the current state action pair. Indicates the subsequent state. Indicates the current state action pair and subsequent states The joint probability distribution function, Indicates the current state action pair The marginal probability distribution function, Indicates subsequent state The marginal probability distribution function.

[0074] It should be noted that since the data collected in the offline dataset comes from the target domain dataset and the source domain dataset, the following explanation may estimate the mutual information based on the two source domain datasets or the two target domain datasets. In order to avoid confusion caused by multiple mutual information, the mutual information estimated by the offline dataset is referred to as the first mutual information. Correspondingly, the mutual information estimated by the source domain dataset is referred to as the source domain mutual information, and the mutual information estimated by the target domain dataset is referred to as the target domain mutual information.

[0075] Once the first mutual information is obtained, the difference in mutual information between the two domains can be obtained based on the method used to determine the first mutual information. Specifically, the difference in mutual information between the two domains is as follows:

[0076] (4)

[0077] in, The mutual information between state-action pairs and subsequent states in the target domain dataset is called target domain mutual information. The mutual information between state-action pairs in the source domain dataset and their subsequent states is called source domain mutual information. This represents the difference in mutual information between the two domains.

[0078] Furthermore, the mutual information of the target domain and the mutual information of the source domain can be expressed by the following formulas:

[0079] (5-1)

[0080] (5-2)

[0081] in, Indicates the current state action pair and subsequent states The joint probability distribution function, Indicates the current state action pair The marginal probability distribution function, Indicates subsequent state The marginal probability distribution function; For the empirical dynamic transition function of the target domain dataset, This represents the normalized state distribution of the target domain dataset. This represents the expectation of all sample values ​​within the target domain dataset. For the empirical dynamic transfer function of the source domain dataset, This represents the normalized state distribution of the source domain dataset. This represents the expectation of all sample values ​​within the source domain dataset.

[0082] It should be noted that the subsequent states in the offline dataset include the subsequent states from the target domain dataset and the subsequent states from the source domain dataset. In this embodiment of the invention, in order to avoid confusion between the subsequent states from different datasets, it is preferable to refer to the subsequent states from the target domain dataset as the first subsequent state and the subsequent states from the source domain dataset as the second subsequent state.

[0083] In this embodiment of the invention, to better estimate the difference in mutual information between the two domains in the high-dimensional state space, a variational estimator based on a neural network is introduced, and contrastive learning is used to estimate the mutual information. One of the most direct approaches is to quantize the source domain information density and the target domain information density using two variational estimators respectively. Specifically, a state-action pair and a first subsequent state are sampled from the target domain dataset as state transition samples. The state transition pair is then identified as a positive sample; correspondingly, a state-action pair is sampled from the target domain dataset. Sample the second subsequent state from the source domain dataset. Then it was determined to be a negative sample.

[0084] In this embodiment of the invention, the comparative learning target can be expressed by the following formula:

[0085] (6)

[0086] in, This represents the contrastive representation learning objective (the loss function for contrastive learning). This represents the expectation of the positive sample values ​​in the offline dataset. This represents the expectation of positive and negative sample values ​​in an offline dataset. This represents a fractional function used to quantify the information density ratio of the target domain dataset. This represents a fractional function used to quantify the information density ratio of the source domain dataset.

[0087] Furthermore, the fractional functions for quantifying the information density ratio of the target domain dataset and the fractional functions for quantifying the information density ratio of the source domain dataset provided in this embodiment of the invention can be expressed by the following formulas:

[0088] (7-1)

[0089] (7-2)

[0090] in, This represents the empirical dynamic transition function for the target domain dataset. This represents the empirical dynamic transfer function of the source domain dataset. This represents the normalized state distribution of the target domain dataset. These represent the normalized state distributions of the source domain dataset, This indicates the ratio of information density used to quantify the target domain dataset. This is used to quantify the information density ratio of the source domain dataset.

[0091] In step 102, the above two fractional functions and Provides a way to measure the corresponding domain state-action pair and subsequent states The correlation that emerges. When the contrastive learning target has a sufficient number of negative samples, it can serve as an approximate estimate of the mutual information difference, while the difference in mutual information between the two domains... It can be just enough to be lower bounded by the pessimistic target. Constrained, therefore minimizing the contrastive learning objective The difference in mutual information between the two domains can be obtained. The approximate estimate, and the approximate representation of the difference in mutual information between the two domains, are as follows:

[0092] (8)

[0093] in, Indicates the number of negative samples. Defined as the pessimistic lower bound Compared to dynamic ratio , It is a relatively tighter lower bound on the difference in mutual information between two domains, and as... As it increases in size, it becomes tighter, which can be expressed as:

[0094] (9)

[0095] It should be noted that the dynamic ratio here can be expressed by the following formula:

[0096] (10)

[0097] in, This represents the expectation of all sample values ​​in the source domain dataset. Represented as the empirical dynamic transition function in the target domain dataset, It is represented as the empirical dynamic transfer function in the source domain dataset.

[0098] Furthermore, in the comparative learning objectives, two fractional functions (i.e., ),in Only state transition samples from the target domain dataset are used as input, and each sample corresponds to a higher score. Only state transition samples from the source domain dataset are used as input and assigned a lower score. However, it's worth noting that negative samples are not used when training two independent contrastive targets. Positive samples were not used Based on this, in this embodiment of the invention, the difference in mutual information between two domains is estimated by applying a single objective, that is, by using only one fractional function. To represent the simplified learning objectives for equivalent and contrastive learning objectives, the simplified learning objectives are as follows:

[0099] (11)

[0100] in, This indicates a simplified learning objective. This represents the score function result corresponding to the positive samples from the target domain dataset. This indicates that all negative and positive samples from the source domain dataset are selected. This represents the score function corresponding to all samples.

[0101] for A simplified version, Using a single fractional function The estimation is performed because the source domain dataset is only shared with the target domain dataset; there is no relative data sharing process. Therefore, a separate scoring function is not needed to determine whether a state transition comes from the source domain dataset. It is only necessary to determine whether the shared state transitions are similar in distribution to the state transitions in the target domain dataset.

[0102] In its specific implementation, this embodiment of the invention employs two neural networks. and To learn the representation between state-action pairs and subsequent states, we map these pairs and subsequent states to a low-order vector space, and then perform an inner product operation on these two representation vectors to obtain a real value. Specifically, the fractional function uses a linear parameter to approximate the information density, as shown below:

[0103] (9)

[0104] in, This represents the dot product of two encoders. This represents the approximate information density, and can also be referred to as the score function value of the training sample. This indicates a state-action encoder. This indicates the subsequent state encoder.

[0105] Furthermore, When normalized to 1, it can make In contrastive representation learning, normalization operations help simplify model training and improve its stability.

[0106] In the data filtering stage, specifically step 103, a fractional function is learned based on the contrastive representation stage. Filter the source domain data.

[0107] The data filtering function provided in this embodiment of the invention can be obtained based on the information density provided by formula (9), as shown below:

[0108]

[0109] in, This represents the defined data filtering function. This indicates an indicator function that returns 1 if the condition within the parentheses is true, and 0 otherwise. The threshold of the fractional function representing the top percentage.

[0110] Specifically, This indicates that if the parentheses contain " "If the condition is met, then the indicator function..." Returns 1 otherwise returns 0.

[0111] Furthermore, the source domain dataset can be filtered based on the obtained data filtering function, which means sampling a batch of state transition samples from the source domain dataset and then filtering them based on the dot product of the two encoders. The collected state transition samples are sorted by their values, and the highest values ​​are extracted from them. The source domain data samples of the scores, also known as the source domain data of data sharing, are referred to as extracted samples in this embodiment of the invention. The extracted samples are added to the target domain dataset and then input into the offline learning model to optimize the value function.

[0112] It should be noted that when samples from the source domain dataset are shared to the target domain dataset, that is, when the extracted samples included in the source domain dataset are added to the target domain dataset, if there is a significant deviation between the source domain dataset and the target domain dataset, the probability of the extracted samples appearing in the target domain dataset tends to zero. 0), at which point the dynamic deviation between the source domain dataset and the target domain dataset tends to be infinitesimal ( At this point, the difference in mutual information between the two domains... It will be restricted to a certain range, therefore: , specifically Figure 3 As shown.

[0113] In this embodiment of the invention, the data sharing algorithm is more convenient than modifying the value function for pessimistic estimation because it eliminates the need for fine-tuning the pruning range and reward ratio. To further improve the algorithm's performance, in this embodiment of the invention, the time difference (TD) error of the filtered data is weighted using a fractional function, and the formally trained value function is as follows:

[0114] (10)

[0115] in, This represents the expectation of all sample values ​​within the target domain dataset. This represents the expectation of all sample values ​​in the source domain dataset. This represents the Bellman loss function used for value function learning. This represents the importance coefficient of the TD-error weighted using a fractional function. This represents the Bellman operator used for offline reinforcement learning. This represents the action-value function for offline reinforcement learning, since its result is determined by the parameters. The neural network structure is thus represented by the subscript . , Describing approximate information density, This indicates the data filtering function.

[0116] This invention provides a technical solution for training on the gym-Mujoco simulator of D4RL, including cross-domain related tasks. Results show that the method provided by this invention outperforms other related current techniques.

[0117] Table 1 Comparison of Experimental Results between Single Domain and Cross Domain

[0118]

[0119] Table 2. Cross-domain experimental results of agent body mass and joint noise deviation.

[0120]

[0121] Table 3. Results of cross-domain experiments on damaged joints and morphological changes in intelligent agents.

[0122]

[0123] Table 4 Comparison of Additional Performance Results

[0124]

[0125] As observed in the table above, when utilizing 10% D4RL data, the algorithm IGDF (Info-Gap Data Filtering) provided in this embodiment of the invention achieved the highest total score across 18 tasks, demonstrating a significant advantage over baseline methods. Compared to the best performance of the baseline methods on 100% D4RL data, IGDF's performance on 10% D4RL data only decreased by -11.89% and -10.81%, indicating the algorithm's excellent performance in data utilization. Because the algorithm provided in this embodiment of the invention can fully utilize a large number of negative samples, it can accurately estimate the mutual information difference, thus achieving significant progress in determining whether sampled source domain data is helpful for training target domain data. Meanwhile, other baseline methods are limited by the target data, which exacerbates the underfitting problem of the domain discriminator, leading to a decline in performance. Therefore, IGDF outperforms other baseline methods; experimental results show that it achieved the best results in 11 out of 18 tasks.

[0126] Meanwhile, to evaluate the performance of IGDF under conditions of significant dynamic changes, tests were conducted on damaged joints and morphological tasks based on the method provided in this embodiment of the invention. The results show that DARA exhibits poor performance in these situations, and its failure can be attributed to the potential unbounded problem in its likelihood-based dynamic difference estimation. Similarly, SRPO showed almost no significant improvement on these two tasks compared to the results trained using a mixed dataset for IQL. In contrast, IGDF provides more robust performance, even achieving state-of-the-art results on 17 out of 18 tasks.

[0127] In summary, the method provided by this invention calculates mutual information between two domains by utilizing the joint empirical distribution between the state-action pairs and the next state in offline datasets, and establishes a quantitative relationship between the difference in mutual information between the two domains and dynamic bias. This measurement method is more robust than the previous method of directly calculating the dynamic ratio, avoiding the unbounded problem that exists when the dynamic bias is large. By introducing a variational estimator based on a neural network, the method can effectively estimate mutual information in a high-dimensional state space, and obtain a tighter lower bound for pessimistic estimation by approximating the information density. By sorting the filtered state transition pairs by the dot product of the two encoders, extraction samples for data sharing are extracted, and source domain data with smaller dynamic bias is selectively shared for training. By using a fractional function to weight the temporal difference error of the filtered data, it is more convenient than modifying the value function for pessimistic estimation, and further improves the performance of the algorithm. This solves the problems of existing cross-domain offline reinforcement learning methods, such as errors in explicit estimation of the dynamic model when given finite target domain data, and the inability of the domain classifier to smoothly estimate the dynamic bias, which may lead to the unbounded problem.

[0128] Based on the same inventive concept, this invention provides a cross-domain offline reinforcement learning device based on contrastive representation. Since the principle of this device in solving the technical problem is similar to that of a cross-domain offline reinforcement learning device method based on contrastive representation, the implementation of this device can refer to the implementation of the method, and the repeated parts will not be described again.

[0129] like Figure 4 As shown, the device includes: a first obtaining unit 401, a second obtaining unit 402, and an optimization unit 403.

[0130] The first obtaining unit 401 is used to input the state-action pair and subsequent state sampled from the offline dataset into the state-action encoder and the subsequent state encoder, respectively, to obtain the first mutual information and the difference between the two-domain mutual information based on the state-action pair and the subsequent state; and to obtain the contrastive learning target based on the positive samples sampled from the target domain dataset, the negative samples sampled from the source domain dataset, and the first mutual information.

[0131] The second obtaining unit 402 is used to maximize the contrastive learning objective to obtain an approximate representation of the difference between the mutual information of the two domains and an equivalent simplified learning objective. Based on the simplified learning objective, the state action encoder and the subsequent state encoder are optimized to obtain the information density based on the fractional function and the dot product representation of the two encoders.

[0132] The optimization unit 403 is used to obtain a data filtering function based on the information density, filter the source domain dataset according to the data filtering function to obtain extracted samples, and input the extracted samples and the target domain dataset data into an offline reinforcement learning model to optimize the value function.

[0133] It should be understood that the units included in the above-described cross-domain offline reinforcement learning device based on contrastive representation are merely a logical division based on the functions implemented by the device. In practical applications, the units can be superimposed or split. Furthermore, the functions implemented by the cross-domain offline reinforcement learning device based on contrastive representation provided in this embodiment correspond one-to-one with the cross-domain offline reinforcement learning method based on contrastive representation provided in the above embodiment. The more detailed processing flow implemented by this device has been described in detail in the first embodiment of the method above, and will not be described in detail here.

[0134] Another embodiment of the present invention also provides a computer device, the computer device including: a processor and a memory; the memory is used to store computer program code, the computer program code including computer instructions; when the processor executes the computer instructions, the electronic device executes each step of the cross-domain offline reinforcement learning method based on contrastive representation in the method flow shown in the above method embodiment.

[0135] Another embodiment of the present invention provides a computer-readable storage medium storing computer instructions that, when executed on a computer device, cause the computer device to perform each step of the cross-domain offline reinforcement learning method based on contrastive representation in the method flow shown in the above method embodiment.

[0136] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0137] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A cross-domain offline reinforcement learning method based on contrastive representation, characterized in that, include: The state-action pairs and subsequent states sampled from the offline dataset are input into the state-action encoder and the subsequent state encoder, respectively, to obtain the first mutual information and the difference between the two-domain mutual information based on the state-action pairs and subsequent states; the contrastive learning target is obtained based on the positive samples sampled from the target domain dataset, the negative samples sampled from the source domain dataset, and the first mutual information. Maximizing the contrastive learning objective yields an approximate representation of the difference between the mutual information of the two domains and an equivalent simplified learning objective. Based on the simplified learning objective, the state-action encoder and the subsequent state encoder are optimized to obtain the information density based on the fractional function and the dot product representation of the two encoders. A data filtering function is obtained based on the information density. The source domain dataset is filtered according to the data filtering function to obtain extracted samples. The extracted samples and the target domain dataset are input into an offline reinforcement learning model to optimize the value function. The process of obtaining the difference between the first mutual information and the two-domain mutual information based on the state-action pair and the subsequent state specifically includes: Based on the first mutual information, target domain mutual information and source domain mutual information are obtained, and the difference between the two domain mutual information is obtained based on the target domain mutual information and the source domain mutual information. The step of obtaining the contrastive learning target based on positive samples sampled from the target domain dataset, negative samples sampled from the source domain dataset, and the first mutual information specifically includes: The subsequent states in the offline dataset include a first subsequent state from the target domain dataset and a second subsequent state from the source domain dataset; A state transition sample is formed by sampling state-action pairs and a first subsequent state from the target domain dataset, and the state transition sample is determined as a positive sample; a state-action pair sampled from the target domain dataset and a second subsequent state sampled from the source domain dataset are determined as negative samples; The step of maximizing the contrastive learning objective to obtain an approximate representation of the difference between the mutual information of the two domains and an equivalent simplified learning objective specifically includes: Maximizing the contrastive learning objective yields an approximate estimate of the difference in mutual information between the two domains, as shown below: The simplified learning objective is as follows: The information density based on the fractional function and the dot product of the two encoders is as follows: The data filtering function is as follows: in, This indicates the number of negative samples. Defined as the pessimistic lower bound , With dynamic ratio In comparison, it is a relatively tighter lower bound for the mutual information difference between two domains, and it becomes tighter as K increases. , , This represents the expectation of all sample values ​​within the source domain dataset. This represents the empirical dynamic transition function in the target domain dataset. This represents the empirical dynamic transition function in the source domain dataset. This indicates a simplified learning objective. This represents the expectation of the positive sample values ​​within the target domain dataset. This represents the expectation of the negative sample values ​​within the source domain dataset. This represents the score function corresponding to positive samples from the target domain. This indicates that all negative and positive samples from the source domain are selected. This represents the score function corresponding to all samples. Represents approximate information density. This represents a state-action encoder. Indicates the subsequent state encoder, This represents the defined data filtering function. This indicates an indicator function that returns 1 if the condition within the parentheses is true, and 0 otherwise. The threshold of the fractional function representing the top percentage. This represents the next state of a single negative sample in the source domain dataset. This represents the next state of a single positive sample in the target domain dataset. This represents the set of the next states of all negative samples in the source domain dataset.

2. The method as described in claim 1, characterized in that, The first mutual information is as follows: The difference in mutual information between the two domains is shown below: in, Represents the state-action pairs in the offline dataset. Joint distribution and subsequent states The first mutual information between them This represents the expectation of all sample values ​​in the offline dataset D. This indicates the current state action pair. Indicates the subsequent state. Indicates the current state action pair and subsequent states The joint probability distribution function, Indicates the current state action pair The marginal probability distribution function, Indicates subsequent state The marginal probability distribution function, This represents the mutual information between the target domain dataset and its subsequent states regarding the state and actions within the target domain dataset. This represents the mutual information between the state-action pairs in the source domain dataset and their subsequent states. This represents the difference in mutual information between the two domains. Indicates the current state. Indicates the action at the current moment. Represents the current state space. Indicates the current action space.

3. The method as described in claim 1, characterized in that, The contrastive learning objective is as follows: in, A fractional function representing the ratio of information density in the quantized target domain. A fractional function representing the ratio of the information density of the quantized source domain. This represents the empirical dynamic transition function in the target domain dataset. This represents the empirical dynamic transition function in the source domain dataset. This represents the normalized state distribution of the target domain dataset. This represents the normalized state distribution of the source domain dataset. This represents the expectation of the positive sample values ​​within the target domain dataset. This represents the expectation of the negative sample values ​​within the source domain dataset. This represents the next state of a single negative sample in the source domain dataset. This represents the next state of a single positive sample in the target domain dataset. This represents the set of the next states of all negative samples in the source domain dataset. Represents the source domain dataset. This represents the target domain dataset.

4. The method as described in claim 1, characterized in that, The value function is as follows: in, This represents the expectation of all sample values ​​within the target domain dataset. This represents the expectation of all sample values ​​in the source domain dataset. This represents the Bellman loss function used for value function learning. This represents the importance coefficient of the TD-error weighted using a fractional function. This represents the Bellman operator used for offline reinforcement learning. This represents the action-value function for offline reinforcement learning. Representing approximate information density, This indicates the data filtering function.

5. The method as described in claim 1, characterized in that, After obtaining the two-domain mutual information difference based on the state-action pair and the subsequent state, the method further includes: When the extracted sample obtained from the source domain dataset is added to the target domain dataset, if there is a large deviation between the source domain dataset and the target domain dataset, then the probability of the state in which the extracted sample appears in the target domain dataset tends to zero, and the dynamic deviation between the source domain dataset and the target domain dataset tends to be infinitesimal. The dynamic deviation between the source domain dataset and the target domain dataset is shown below: in, This represents the dynamic deviation between the source domain dataset and the target domain dataset. This indicates calculating the expected value for all samples within the source domain dataset. For the empirical dynamic transition function of the target domain dataset, This is the empirical dynamic transfer function for the source domain dataset.

6. A cross-domain offline reinforcement learning device based on contrastive representation, characterized in that, include: The first obtaining unit is used to input the state-action pair and subsequent state sampled in the offline dataset into the state-action encoder and the subsequent state encoder, respectively, to obtain the first mutual information and the difference between the two-domain mutual information based on the state-action pair and the subsequent state. The contrastive learning target is obtained based on positive samples sampled from the target domain dataset, negative samples sampled from the source domain dataset, and the first mutual information; The second obtaining unit is used to maximize the contrastive learning objective to obtain an approximate representation of the difference between the mutual information of the two domains and an equivalent simplified learning objective. Based on the simplified learning objective, the state action encoder and the subsequent state encoder are optimized to obtain the information density based on the fractional function and the dot product representation of the two encoders. An optimization unit is configured to obtain a data filtering function based on the information density, filter the source domain dataset according to the data filtering function to obtain extracted samples, and input the extracted samples and the target domain dataset into an offline reinforcement learning model to optimize the value function. The first obtaining unit is specifically used for: Based on the first mutual information, target domain mutual information and source domain mutual information are obtained, and the difference between the two domain mutual information is obtained based on the target domain mutual information and the source domain mutual information. The subsequent states in the offline dataset include a first subsequent state from the target domain dataset and a second subsequent state from the source domain dataset; A state transition sample is formed by sampling state-action pairs and a first subsequent state from the target domain dataset, and the state transition sample is determined as a positive sample; a state-action pair sampled from the target domain dataset and a second subsequent state sampled from the source domain dataset are determined as negative samples; The second obtaining unit is specifically used for: Maximizing the contrastive learning objective yields an approximate estimate of the difference in mutual information between the two domains, as shown below: The simplified learning objective is as follows: The information density based on the fractional function and the dot product of the two encoders is as follows: The data filtering function is as follows: in, This indicates the number of negative samples. Defined as the pessimistic lower bound , With dynamic ratio In comparison, it is a relatively tighter lower bound for the mutual information difference between two domains, and it becomes tighter as K increases. , , This represents the expectation of all sample values ​​within the source domain dataset. This represents the empirical dynamic transition function in the target domain dataset. This represents the empirical dynamic transition function in the source domain dataset. This indicates a simplified learning objective. This represents the expectation of the positive sample values ​​within the target domain dataset. This represents the expectation of the negative sample values ​​within the source domain dataset. This represents the score function corresponding to positive samples from the target domain. This indicates that all negative and positive samples from the source domain are selected. This represents the score function corresponding to all samples. Represents approximate information density. This represents a state-action encoder. Indicates the subsequent state encoder, This represents the defined data filtering function. This indicates an indicator function that returns 1 if the condition within the parentheses is true, and 0 otherwise. The threshold of the fractional function representing the top percentage. This represents the next state of a single negative sample in the source domain dataset. This represents the next state of a single positive sample in the target domain dataset. This represents the set of the next states of all negative samples in the source domain dataset.

7. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the cross-domain offline reinforcement learning method based on contrastive representation as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The system contains a computer program that, when executed by a processor, causes the processor to perform the cross-domain offline reinforcement learning method based on contrastive representation as described in any one of claims 1-5.

Images