Continuous offline reinforcement learning method for spatial alignment and selective weight activation
Through spatial alignment and selective weight activation among different tasks, the problems of changes in different task states and actions and multi-mode distribution data processing are solved, and effective memory of previous knowledge and efficient learning of new tasks are achieved.
Patent Information
- Application Number
- CN202510248451.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-04
- Publication Date
- 2025-04-04
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing technology has difficulty handling the changes in state and action space of different tasks in the real world, as well as handling offline data of multimodal distributions, resulting in catastrophic forgetting and ineffective learning of new tasks.
The spatial alignment method is used to align states and action spaces in different tasks, and assign different weight mask matrices to different tasks through selective weight activation to preserve previous knowledge and alleviate catastrophic forgetting.
Effectively performing reinforcement learning for different tasks in the same space greatly retains the memory of previous knowledge, reduces catastrophic forgetting, and improves the learning efficiency of new tasks.
Smart Images
Figure CN119761454A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the technical field of reinforcement learning, and in particular, relates to a continuous offline reinforcement learning method with spatial alignment and selective weight activation. Background Art
[0002] With the rapid development of large model technology, real data, artificial data, and synthetic data have shown explosive growth, especially in the field of reinforcement learning. Researchers no longer rely too much on the interaction between agents and the environment, which greatly avoids the computing resource overhead and potential safety hazards required for online interaction between agents and the environment. Continuous offline reinforcement learning (CORL) aims to build an agent that can continuously learn on offline data (that is, only data sets, no corresponding environment for interaction). The agent can continuously learn new tasks and master new skills while maintaining a certain memory of the skills already mastered. Under the conditions of growing task scenarios, learning high-performance agent strategies from a large amount of offline data collected from various sources poses a challenge to most continuous reinforcement learning methods based on static data training, that is, existing technologies are more likely to show catastrophic forgetting of previous knowledge and ineffective learning of new tasks. In the face of these problems, existing technologies are mainly divided into three categories: sample replay-based methods, regularization-based methods, and structure-based methods. However, existing technologies are limited to continuous reinforcement learning environments with consistent state and action spaces, ignoring more general needs in the real world. For example, the state and action spaces of different tasks may change due to changes in demand or different sensors. In addition, the sources of offline data sets may also be diverse, from children, experts, or large models, so offline data can present a multimodal distribution. Existing technologies cannot effectively model the multimodal distribution of offline data, resulting in unsatisfactory results and difficulty in achieving significant performance improvements. Summary of the invention
[0003] In view of this, the purpose of this application is to provide a continuous offline reinforcement learning method with spatial alignment and selective weight activation. The state and action spaces in different tasks are aligned through the spatial alignment method to ensure that the reinforcement learning process of each task is carried out in the same space. Different weight mask matrices are assigned to different tasks through selective weight activation, thereby greatly retaining the memory of previous knowledge and effectively alleviating catastrophic forgetting.
[0004] The present application provides a continuous offline reinforcement learning method for spatial alignment and selective weight activation, including: For each task in the continuous learning task sequence, the corresponding data set of the task is used, and the state space and action space of the task are aligned using the quantitative space alignment method to obtain the state sequence and action sequence of the task in the same space; Obtain a single time step reward sequence from the data set corresponding to the task, and based on the single time step reward sequence, use the reward function to calculate the cumulative discounted reward and sequence corresponding to the task; Based on the state sequence of the task, the cumulative discounted reward and sequence corresponding to the task are used as the condition information of the diffusion model, the activated diffusion model of the task is used to generate the given state sequence of the task, and then the inverse dynamics model is used to generate the given action sequence of the task; Taking the maximization of the accumulated discounted reward of each task in the continuous learning task sequence as the goal, continuous reinforcement learning is performed on each task to update the parameters of the diffusion model and the parameters of the inverse dynamics model corresponding to each task.
[0005] Furthermore, the aligning the state space and action space of the task by using the quantized space alignment method includes: Collect state data and action data from the data set corresponding to the task; Encode the collected state data and action data using a pre-trained vector quantization encoder, and obtain state vector data and action vector data with the same length under the task using a pre-trained quantization codebook; Based on the state vector data and action vector data of the task, the state sequence and action sequence of the task are constructed respectively.
[0006] Furthermore, the vectorized encoder and the quantization codebook are pre-trained in the following manner: Using the vector quantization encoder and quantization codebook, as well as the vector quantization decoder, a neural network required for aligning the state space and the action space is constructed; De-mapping the state vector data and the motion vector data having the same length using the vector quantization decoder to obtain reconstructed state data and reconstructed motion data; The loss of the neural network is determined based on the reconstructed state data and the reconstructed action data, as well as the state data and the action data, so as to update the parameters of the vector quantization encoder, the quantization codebook, and the vector quantization decoder.
[0007] Furthermore, the diffusion model includes: a forward diffusion process and a reverse generation process: The forward diffusion process is used to construct a transformation relationship between complex data distribution and simple Gaussian distribution; The reverse generation process is used to gradually restore the conversion relationship from a simple Gaussian distribution to a complex data distribution.
[0008] Furthermore, after updating the parameters of the diffusion model and the parameters of the inverse dynamics model corresponding to each task, the method further includes: For each task, a mask matrix related to the task is generated based on the parameters of the diffusion model of the task; Based on the mask matrix related to each task, weights are assembled through bitwise product and sum operations to generate a weight matrix; The weight matrix and the mask matrix are used to activate the diffusion model corresponding to each task.
[0009] The present application provides a continuous offline reinforcement learning method based on spatial alignment and selective weight activation. The state and action spaces in different tasks are aligned through the spatial alignment method to ensure that the reinforcement learning process of each task is carried out in the same space. Different weight mask matrices are assigned to different tasks through selective weight activation, thereby greatly retaining the memory of previous knowledge and effectively alleviating catastrophic forgetting. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] Figure 1 A flowchart of a continuous offline reinforcement learning method for spatial alignment and selective weight activation provided by an embodiment of the present application is shown. DETAILED DESCRIPTION
[0011] In order to make the purpose, technical solution and advantages of the technical solution more clear, the technical solution is further described in detail below in conjunction with specific implementation methods. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the technical solution.
[0012] See also Figure 1 , Figure 1 The flowchart of the continuous offline reinforcement learning method of spatial alignment and selective weight activation provided by the embodiment of the present application is shown. Figure 1 As shown, the method includes: S101. For each task in the continuous learning task sequence, use the data set corresponding to the task and align the state space and action space of the task using a quantized space alignment method to obtain a state sequence and an action sequence of the task in the same space.
[0013] In this step, the continuous learning task sequence consists of multiple tasks. The corresponding data set of the task is used for reinforcement learning, and each task can be represented by a Markov quintuple, that is, each task is represented by the state space, action space, transfer function, reward function and discount coefficient. The value range of the discount coefficient is .
[0014] In order to adapt to continuous reinforcement learning in any task scenario, for tasks with different state spaces and action spaces, the embodiment of the present application uses the data set corresponding to the task, and uses a quantized space alignment method to align the state space and action space of the task to obtain the state sequence and action sequence of the task in the same space, so that the reinforcement learning process of the task can be carried out in the same space.
[0015] In specific implementation, the state space and action space of the task can be aligned using the quantized space alignment method in the following way: Step 1011: Collect state data and action data from the data set corresponding to the task.
[0016] As an example, from the dataset D corresponding to this task i Collecting status data and action data , where i represents any task in the continuous learning task sequence.
[0017] Step 1012: Encode the collected state data and action data using a pre-trained vector quantization encoder, and obtain state vector data and action vector data with the same length under the task using a pre-trained quantization codebook.
[0018] As an example, the collected state data is quantized using a pre-trained vector quantized encoder (VQE). and action data Encode and use the pre-trained quantization codebook to obtain the state vector data with the same length under this task and motion vector data .
[0019] Here, the operations of the vector quantization encoder VQE and the quantization codebook codebook can be expressed as formulas (1) and (2): ; (1) ; (2) In the formula, are the parameters of the vector quantization encoder, is the parameter of the quantization codebook, is the quantized representation, i.e., a vector composed of the embedded representations in a fixed number of quantized codebooks, Indicates the status data of the collection Perform encoding operations. Indicates the collected action data Perform encoding operations. Indicates that the quantization operation is performed on the encoded state data according to the quantization codebook. Indicates that the encoded motion data is quantized according to the quantization codebook.
[0020] In a specific implementation, the vectorized encoder and the quantization codebook may be pre-trained in the following manner: Step 201: Use the vector quantization encoder and quantization codebook, as well as the vector quantization decoder to construct a neural network required for aligning the state space and the action space.
[0021] Step 202: De-map the state vector data and the motion vector data having the same length using the vector quantization decoder to obtain reconstructed state data and reconstructed motion data.
[0022] Step 203: determine the loss of the neural network based on the reconstructed state data and the reconstructed action data, as well as the state data and the action data, so as to update the parameters of the vector quantization encoder, the quantization codebook, and the vector quantization decoder.
[0023] Here, the above steps 201-203 can be represented by mathematical functions. Specifically, a neural network is constructed based on a vector quantization encoder VQE, a quantization codebook codebook, and a vector quantization decoder (VQD). Then, formula (3) is used as the objective function and formula (4) is used as the constraint condition of the objective function to train the neural network.
[0024] ; (3) ; (4) In the formula, is the loss function of the above objective function, x represents the state vector and action vector in any task, are the parameters of the vector quantization decoder, is a restriction parameter used to limit the range of the embedding vector in the quantization codebook. is the magnitude of the vector.
[0025] Among them, the loss function is solved by the following formula (5): : (5) In the formula, represents the gradient truncation operation, Indicates the desired operation. Denotes a decoding operation, ,in, is the output of the vector quantization encoder, .
[0026] Here, in order to always keep the objective function (3) satisfying the constraint condition (4) during the training process of the neural network, the embodiment of the present application updates the parameters of the quantization codebook by using the stochastic gradient descent method each time. Afterwards, the quantization Truncation is performed to ensure that the constraints are always met.
[0027] In addition, it is worth noting that, considering that the number of continuous learning tasks is increasing, the embodiment of the present application integrates the vector quantization encoder VQE, the quantization codebook codebook and the vector quantization decoder VQD to achieve separate training of the vector quantization encoder VQE, the quantization codebook codebook and the vector quantization decoder VQD, and then obtains the corresponding parameters respectively. , and the parameters obtained by separate training can be merged.
[0028] Step 1013: construct a state sequence and an action sequence of the task based on the state vector data and the action vector data of the task.
[0029] As an example, based on the state data and action data The time sequence of the obtained state vector data and motion vector data Constructed into state sequences and action sequences .
[0030] S102. Obtain a single time step reward sequence from the data set corresponding to the task, and based on the single time step reward sequence, calculate the cumulative discounted reward and sequence corresponding to the task using a reward function.
[0031] In this step, a single time step reward sequence is obtained from the dataset of this task. , and the reward function can be used to obtain the cumulative discounted reward and sequence of the task .
[0032] S103. Based on the state sequence of the task, the cumulative discounted reward and sequence corresponding to the task are used as conditional information of the diffusion model, the activated diffusion model of the task is used to generate a given state sequence of the task, and then the inverse dynamics model is used to generate a given action sequence of the task.
[0033] In this step, each task in the continuous learning task sequence corresponds to a different diffusion model, which is specifically manifested in that the parameters of the diffusion model are different. The present application uses a selective weight activation method to activate the diffusion model of the task by activating the parameters of the diffusion model of the task, and then uses the activated diffusion model to learn the data. Specifically, the diffusion model is used to first construct a conversion relationship between complex data distribution and simple Gaussian distribution, and then the conversion relationship from simple Gaussian distribution to complex data distribution is gradually restored to realize data generation. The diffusion model can generate data through the learned denoising neural network.
[0034] As an example, the diffusion model consists of the forward diffusion process function shown in formula (6) and the reverse generation process function shown in formula (7): ; (6) ; (7) In the formula, ,in, is standard Gaussian noise, For the data set, are the parameters of the diffusion model, is a predefined decreasing sequence, , is the identity matrix.
[0035] By transforming the above formulas (6) and (7), we can construct the generating function (8) of a given state sequence with K steps: ; (8) In the formula, is the diffusion time step, represents the state sequence generated by the diffusion model, ,in, is the length of the sequence, Represents the state sequence of the task, represents a noise vector sampled from a standard Gaussian distribution, which has the same dimension as the state vector data, is the conditional information added by the diffusion model (i.e., the cumulative discounted reward and sequence corresponding to the task), is the conditional guidance coefficient of the diffusion model. In general, =1.2.
[0036] Furthermore, based on the generated given state sequence of the task, the inverse dynamics model is used to make a decision to generate a given action sequence of the task. Current status in and subsequent status , using the inverse dynamics model Produce a given action .
[0037] S104, taking the maximization of the accumulated discounted reward of each task in the continuous learning task sequence as the goal, performing continuous reinforcement learning on each task to update the parameters of the diffusion model and the parameters of the inverse dynamics model corresponding to each task.
[0038] In this step, the goal of continuous reinforcement learning is to maximize the cumulative discounted reward of each task in the continuous learning task sequence, which can be expressed using the following formula (9): ; (9) Where i is the task in the continuous learning task sequence, i=1, 2, 3..., is the discount factor, is the reward function, is the current status data, is the current action data, where t is the reinforcement learning time step, t=0, 1, 2…
[0039] In this step, the following mathematical functions (10) and (11) are used to represent the loss functions of the diffusion model and the inverse dynamics model respectively: ; (10) ; (11) In the formula, The uniform distribution of represents the Bernoulli distribution, represents the parameters of the Bernoulli distribution, is a sequence of 0s and 1s sampled from a Bernoulli distribution. represents the diffusion model, represents the parameters of the diffusion model, Indicates the desired operation, specifically, Indicates that by sampling a batch of data ,calculate , and then take the average, Indicates that by sampling a batch of data ,calculate , and then take the average, where Indicates the module length, Represents the decision-making process of the inverse dynamics model.
[0040] With the goal of maximizing the cumulative discounted reward of each task, the parameters of the diffusion model corresponding to each task and the parameters of the inverse dynamics model are updated through the above loss function.
[0041] After updating the parameters of the diffusion model and the parameters of the inverse dynamics model corresponding to each task, the method further includes: Step 301: For each task, a mask matrix related to the task is generated based on the parameters of the diffusion model of the task.
[0042] The mask matrix information is used to characterize the update parameters of the forward diffusion process.
[0043] In this step, after the task completes reinforcement learning, the parameters of the diffusion model of the task are saved, wherein the update parameters of the forward diffusion process are represented as mask matrix information related to the task, and the parameters are updated in the reverse generation process using a common optimizer.
[0044] Step 302: Based on the mask matrices related to each task, weight assembly is performed through bitwise product and sum operations to generate a weight matrix.
[0045] The weight matrix and the mask matrix are used to activate the diffusion model corresponding to each task.
[0046] In this step, after each task completes reinforcement learning, based on the mask matrix information corresponding to each task, the task scenario-related parameters are extracted and assembled into a weight matrix of the diffusion model containing all task-related parameters through bitwise multiplication and summation operations.
[0047] Furthermore, for each task, the diffusion model of the task can be activated in the following manner: using the weight matrix to activate the mask matrix related to the task, using the mask matrix to extract the parameters of the diffusion model of the task, and using the extracted parameters of the diffusion model of the task to activate the diffusion model of the task.
[0048] As an example, suppose the diffusion model is a Layer block model, each block The weight matrix included is , the present application embodiment uses Representation Block function; for each block All weights , the embodiment of the present application adopts a random position selection method to assign each task Allocation Mask , when the block All mask matrices of Add them together, and you will get a matrix with all 1s. , The shape and It is the same. In addition, in the embodiment of the present application, the position where the element in the mask matrix is 1 indicates that the weight of the position is activated.
[0049] As an example, the weight matrix is obtained by the following formula (12): ; (12) In the formula, is the mask matrix related to the task, is the weight sum of each layer block of the diffusion model for this task, represents the Hadamard product, that is, the product of corresponding positions, The shape and size of of the same shape and size, .
[0050] The above contents are only preferred embodiments of the present invention. For ordinary technicians in this field, many changes can be made in the specific implementation methods and application scopes based on the ideas of the technical content. As long as these changes do not deviate from the concept of the present invention, they all belong to the protection scope of the present invention.
Claims
1. A continuous offline reinforcement learning method with spatial alignment and selective weight activation, characterized in that: The method comprises: For each task in the continuous learning task sequence, the corresponding data set of the task is used, and the state space and action space of the task are aligned using the quantitative space alignment method to obtain the state sequence and action sequence of the task in the same space; Obtain a single time step reward sequence from the data set corresponding to the task, and based on the single time step reward sequence, use the reward function to calculate the cumulative discounted reward and sequence corresponding to the task; Based on the state sequence of the task, the cumulative discounted reward and sequence corresponding to the task are used as the condition information of the diffusion model, the activated diffusion model of the task is used to generate the given state sequence of the task, and then the inverse dynamics model is used to generate the given action sequence of the task; Taking the maximization of the accumulated discounted reward of each task in the continuous learning task sequence as the goal, continuous reinforcement learning is performed on each task to update the parameters of the diffusion model and the parameters of the inverse dynamics model corresponding to each task.
2. The method according to claim 1, characterized in that The step of aligning the state space and action space of the task by using the quantized space alignment method includes: Collect state data and action data from the data set corresponding to the task; Encode the collected state data and action data using a pre-trained vector quantization encoder, and obtain state vector data and action vector data with the same length under the task using a pre-trained quantization codebook; Based on the state vector data and action vector data of the task, the state sequence and action sequence of the task are constructed respectively.
3. The method according to claim 2, characterized in that The vectorized encoder and quantized codebook are pre-trained by: Using the vector quantization encoder and quantization codebook, as well as the vector quantization decoder, a neural network required for aligning the state space and the action space is constructed; De-mapping the state vector data and the motion vector data having the same length using the vector quantization decoder to obtain reconstructed state data and reconstructed motion data; The loss of the neural network is determined based on the reconstructed state data and the reconstructed action data, as well as the state data and the action data, so as to update the parameters of the vector quantization encoder, the quantization codebook, and the vector quantization decoder.
4. The method according to claim 1, characterized in that The diffusion model includes: a forward diffusion process and a reverse generation process: The forward diffusion process is used to construct a transformation relationship between complex data distribution and simple Gaussian distribution; The reverse generation process is used to gradually restore the conversion relationship from a simple Gaussian distribution to a complex data distribution.
5. The method according to claim 4, characterized in that After updating the parameters of the diffusion model and the parameters of the inverse dynamics model corresponding to each task, the method further includes: For each task, a mask matrix related to the task is generated based on the parameters of the diffusion model of the task; Based on the mask matrix related to each task, weights are assembled through bitwise product and sum operations to generate a weight matrix; The weight matrix and the mask matrix are used to activate the diffusion model corresponding to each task.