A method of packaging a reinforcement learning task as a learnable task

Abstract

The present application relates to the technical field of reinforcement learning and machine learning, and discloses a method for packaging a reinforcement learning task as a learnable task. The method comprises obtaining an original reinforcement learning task comprising a reset function reset() and a step function step(). The task is packaged, and the environment state observation value obs and the action action of the original task are mapped into the observation value and the action of a learnable model, respectively. A feedback state input is added to the learnable model, and the feedback state is generated by transforming the cumulative reward rewards of a complete round of the original task. The model is trained using the packaged learnable task comprising a plurality of reinforcement learning task round data. The method realizes the packaging and reuse of offline or high-cost interactive tasks through interface mapping, and provides a global feedback signal for the model by introducing a feedback state based on the cumulative reward of a round, thereby improving the training efficiency and policy optimization effect.

Description

Technical Field

[0001] This invention relates to the fields of reinforcement learning and machine learning technology, specifically a method for packaging reinforcement learning tasks into learnable tasks. Background Technology

[0002] In reinforcement learning, algorithmic models typically need to interact directly with the task environment, obtaining an initial state through a reset function and receiving rewards and the next state through an execution function. This direct interaction mode requires the model to be able to run tasks in real-time in online or simulated environments to collect data. In many real-world scenarios, real-time task interaction is costly, risky, or only offline datasets of historical interactions are available. Existing approaches treat these offline data as static datasets for imitation learning or offline reinforcement learning, but this approach severs the structural definition of the task itself. The model cannot perceive the continuity of the task as an interactive environment, nor can it flexibly adapt to different initial conditions or dynamic changes within the same task.

[0003] Conventional task wrapping techniques aim to enable tasks from different sources to be processed by a unified learning framework, but they primarily focus on interface standardization, mapping the state and action spaces of different tasks to a unified dimension. This wrapping fails to address the effective utilization of dynamic information within the task itself; the model learning process still relies on feedback from single-step reward signals, lacking holistic guidance on the overall performance of the entire task execution sequence. This limits learning efficiency, especially in tasks with sparse or delayed rewards, where the model struggles to establish a correlation between current actions and long-term gains. Summary of the Invention

[0004] The purpose of this invention is to provide a method for packaging reinforcement learning tasks into learnable tasks, so as to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides a method for packaging reinforcement learning tasks into learnable tasks, the method comprising: Obtain a reinforcement learning task, which includes a reset function reset() and an execution function step(), wherein the environmental state observations obs of the reinforcement learning task are used as input, the action is used as output, and a reward is returned; The reinforcement learning task is packaged such that the environmental state observations obs of the reinforcement learning task are mapped to the environmental state observations obs of the learnable model, and the action of the reinforcement learning task is mapped to the action of the learnable model. Add a feedback state input to the learnable model, the feedback state being generated after transforming the cumulative rewards of a full epoch of the reinforcement learning task; The learnable model is trained using a packaged learnable task, wherein the learnable task contains rounds of data from multiple reinforcement learning tasks.

[0006] Preferably, the reinforcement learning task is a task based on the Gym framework, wherein the reset function reset() is used to initialize the environment state, and the execution function step() is used to update the environment state according to the input action and return the reward.

[0007] Preferably, the step of packaging the reinforcement learning task includes: The environmental state observations obs of the reinforcement learning task are directly used as the input of the environmental state observations obs of the learnable model; The action of the reinforcement learning task is directly used as the output of the action of the learnable model; The cumulative rewards of a complete round of the reinforcement learning task are used as the basis for calculating the feedback state.

[0008] Preferably, the calculation of the feedback state is based on a transformation of the cumulative rewards, which is a normalized transformation, specifically in the form of: The feedback signal s is calculated using the following formula: s = (rewards_max - reward) / (rewards_max - reward_min), where reward_max represents the maximum possible value of the cumulative reward of the reinforcement learning task in a complete round, reward_min represents the minimum possible value of the cumulative reward of the reinforcement learning task in a complete round, and reward represents the actual cumulative reward of the previous round.

[0009] Preferably, the values ​​of rewards_max and rewards_min are predefined based on the historical round data or task specifications of the reinforcement learning task and are input as fixed parameters during the packaging process.

[0010] Preferably, the packaged learnable task comprises multiple rounds of the reinforcement learning task, with each round generating a corresponding cumulative reward. The method further includes: A weighted average of the cumulative rewards over multiple rounds is calculated to generate a fitness value, which is used to evaluate the performance of the learnable model.

[0011] Preferably, in the weighted average calculation, the cumulative reward for each round is assigned a weight, and the weight value increases as the round number increases, making the score of subsequent rounds more important than the score of previous rounds.

[0012] Preferably, the fitness value is calculated dynamically during training, and the fitness value is used as a feedback signal to adjust the parameters of the learnable model.

[0013] Preferably, the transformations used in calculating the above-mentioned feedback states also include logarithmic scaling or piecewise function transformations, used to map the cumulative rewards to a specific numerical range to accommodate reward signals of different magnitudes.

[0014] Preferably, the method further includes the step of: Store environmental state observations (obs), actions, and corresponding cumulative rewards from multiple historical rounds; When training the learnable model, a batch of data is randomly sampled from the stored historical round data for updating the model parameters.

[0015] Compared with the prior art, the beneficial effects of the present invention are: By mapping the environmental state observations and actions of the original reinforcement learning task to the corresponding interface of the learnable model, an abstract learnable task layer is constructed. This encapsulates and preserves the interaction logic of the original task, while the model's learning process is based on this encapsulated interface. The original task no longer needs to interact with the model in real-time physical or simulation; a set of pre-recorded or generated complete interaction trajectory data can be directly input into the model as a task instance for learning. This improves the reusability of task data, making reinforcement learning training possible in safety-critical environments, high-cost simulations, or purely offline data scenarios. The model can learn from a large number of such encapsulated task instances without directly contacting the original environment, thereby mastering the general strategy of the task.

[0016] By inputting a feedback state generated based on the cumulative reward transformation over a full round into the model, this design provides the model with global performance feedback that goes beyond single-step rewards. It is a condensed encoding of the final result of a complete decision sequence, inputting as additional information along with current environmental observations at each decision step. When making decisions, the model not only focuses on the immediate state and local reward but also perceives the overall reward level achieved by the current policy in the history of similar tasks. This guides the model to optimize long-term rewards, alleviating the inefficiency of traditional reinforcement learning in sparse reward tasks due to delayed and scarce feedback signals. By learning the correlation between state-action and final reward results, the model can more quickly evaluate the long-term value of different policies, thereby accelerating the convergence and optimization process. Attached Figure Description

[0017] Figure 1 This is a schematic diagram illustrating the working principle of the method for making the packaging reinforcement learning task a learnable task according to the present invention. Figure 2 A flowchart for packaging reinforcement learning tasks; Figure 3 A flowchart for generating fitness values ​​based on cumulative rewards over multiple rounds. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Please see Figure 1This invention provides a method for packaging a reinforcement learning task into a learnable task. The method includes: constructing a task form suitable for direct model learning by adapting the interface and enhancing the feedback information of the reinforcement learning task. Reinforcement learning tasks are typically provided in the form of an environment, whose core interaction interface includes a reset function and an execution function. The reset function initializes the environment and returns the initial environment state observations. The execution function receives an action as input, updates the internal state of the environment based on this action, and returns new state observations, immediate rewards, and other information. The method of this invention obtains such a reinforcement learning task, first packaging it so that the environment state observations of the task can be directly mapped to the environment state observation inputs received by the learnable model, and simultaneously mapping the actions output by the learnable model directly to the action inputs required by the reinforcement learning task. Based on this, the method adds an additional feedback state input channel to the learnable model. This feedback state is not from the immediate reward of a single-step interaction, but is generated after a specific transformation of the cumulative reward generated after a complete round of interaction in the reinforcement learning task. Once packaged, a complete learnable task is formed, including state observation, action, and feedback state. When training a learnable model using this task, it is necessary to collect round-by-round interaction data from multiple reinforcement learning tasks. The model learns how to output effective actions given a state and feedback information about the previous round's performance through this data.

[0020] In one embodiment of the present invention, see [reference] Figure 2 The reinforcement learning task is implemented using the Gym framework. Tasks within the Gym framework explicitly define reset and execution functions. The reset function resets the environment to its initial state and returns the initial environment state observations. The execution function receives an action parameter; the environment evolves to the next state based on this action. The function returns the new environment state observations, the immediate reward obtained for this action, a flag indicating whether the round has ended, and other information. During the wrapper process, the environment state observations of the reinforcement learning task are directly used as input to the environment state observations of the learnable model, and the actions output by the learnable model are directly passed to the execution function as action parameters for the reinforcement learning task. The cumulative reward obtained by the reinforcement learning task at the end of a complete round is recorded as the basis for calculating the feedback state information.

[0021] In practice, reinforcement learning tasks can be concretized into a publicly available standard benchmark task, such as the classic control task CartPole-v1 in the OpenAIGym framework. In this specific scenario, the environment of the reinforcement learning task fully conforms to the interface specification of the Gym framework. When the reset function `reset()` is called, the physical states of the car and poles within the CartPole-v1 environment are initialized, and an array containing four floating-point numbers: car position, car velocity, pole angle, and pole tip velocity. This array represents the environment state observations for the reinforcement learning task. The execution function `step()` receives an action as input. In the CartPole-v1 task, the action is a discrete integer with a value of 0 or 1, representing applying a force to the left or right to the car, respectively. The execution function `step()` updates the dynamic states of the car and poles within the environment based on the input action and returns a tuple containing the new environment state observations, the immediate reward obtained for this action, a Boolean value indicating whether the round has ended, and other diagnostic information. At the data comparison level, an unpackaged raw CartPole-v1 task interacts directly with the agent algorithm. The agent algorithm receives environmental state observations and outputs actions, which are then sent to the execution function step(), and the process repeats until the round ends.

[0022] In some embodiments, the reinforcement learning task described above is packaged, and the packaging process establishes a direct mapping relationship between the original task and the learnable model. The wrapper first calls the reset function `reset()` of the reinforcement learning task to obtain initial environmental state observations, such as a four-dimensional array of the form [0.012, -0.043, 0.005, 0.031]. The wrapper does not modify this array and directly uses it as the input of the environmental state observations for the learnable model. The learnable model performs calculations based on this input and outputs an action, such as the integer 1. The wrapper does not process this integer value 1 and directly passes it as a parameter to the execution function `step(1)` of the reinforcement learning task. After the execution function `step(1)` performs calculations within the environment, it returns new environmental state observations, an immediate reward, and termination flag information. The wrapper then directly provides the new environmental state observations to the learnable model again, and this process is repeated. When a round terminates due to excessive lever tilting or reaching the maximum number of steps, the wrapper records the sum of all immediate reward values ​​returned by the `step()` function for that round, i.e., the cumulative reward. For example, if a round lasts 50 steps, the cumulative reward is 50.0. This cumulative reward value is saved as the basis for subsequent calculations of the feedback state. In terms of data flow comparison, the key to the wrapping process is the construction of a transparent channel between the original task and the model. The environmental state observations of the original task and the actions output by the model are directly transmitted through this channel, with the values ​​and format remaining unchanged, while the cumulative reward of the round is additionally captured.

[0023] Optionally, in another specific implementation, the reinforcement learning task is also the MuJoCo continuous control task in the Gym framework, such as HalfCheetah-v4. In this task, the reset function `reset()` returns a 17-dimensional floating-point vector containing information such as the position and velocity of each joint of the agent. The execution function `step()` receives a 6-dimensional continuous action vector, with each dimension taking values ​​between [-1, 1], representing the torque applied to different joints. The packaging process also follows the direct mapping principle: the 17-dimensional state observation vector is directly input into the learnable model, and the 6-dimensional action vector output by the learnable model is directly passed to the execution function `step()`. At the end of a round, the cumulative reward is the sum of the immediate rewards for all steps, for example, -125.6.

[0024] In some embodiments, the implementation of the wrapper process involves encapsulating a Gym environment object. A wrapper class is created that receives a raw Gym environment instance during initialization. The wrapper class provides an interface similar to the raw Gym environment, but its internal logic implements the direct mapping described above. When the wrapper class's reset method is called, it calls the reset function reset() of the internal Gym environment instance and stores and outputs the returned environment state observations as is. When the wrapper class's execute method is called, it receives action parameters from the learnable model, passes these action parameters as is to the execute function step() of the internal Gym environment instance, and extracts the environment state observation portion from the new environment state observations, rewards, and other information returned by the execute function step(), returning it as is. Simultaneously, the wrapper class maintains a cumulative reward variable, accumulating the immediate reward returned by the execute function step() at each step, and recording the final cumulative reward value in the storage structure when a round termination flag is detected. Optionally, the packaging process does not change the data structure and physical meaning of the original reinforcement learning task environment state observations and actions. For example, the state observations of CartPole-v1 are always a four-dimensional physical quantity array, and the actions of HalfCheetah-v4 are always a six-dimensional torque vector. The mapping process does not involve dimensional transformation or type conversion.

[0025] In one embodiment of the present invention, see [reference] Figure 3The calculation of the feedback state is based on a normalized transformation of the cumulative reward. The maximum and minimum possible values ​​of the cumulative reward for a complete round of the reinforcement learning task are defined, and these values ​​can be predetermined based on task specifications or historical round data. During the packaging process, these values ​​are used as fixed parameter inputs. For a completed previous round, its actual cumulative reward is obtained. Based on the maximum and minimum possible values, a feedback signal is calculated through a normalized transformation. The formula for this feedback signal is (maximum possible value minus actual cumulative reward) divided by (maximum possible value minus minimum possible value). The calculated feedback signal serves as the feedback state input.

[0026] In practice, the calculation of the feedback state is based on a normalized transformation of the cumulative reward generated after a complete epoch of the reinforcement learning task. After recording the actual cumulative reward of the previous epoch, the wrapper process needs to convert the cumulative reward value into a standardized feedback signal for the learnable model as additional input. This transformation process relies on predefined maximum and minimum possible values ​​of the cumulative reward. For example, in the CartPole-v1 task, the maximum number of steps in an epoch is fixed at 500, and the immediate reward for each step is fixed at +1.0. Therefore, the maximum possible value of the cumulative reward is set to 500.0, and the minimum possible value of the cumulative reward is set to 0.0, representing that the epoch terminates immediately at the first step. For the actual cumulative reward generated in the previous epoch, such as 350.0, the wrapper will use these parameters for normalization calculation.

[0027] In some embodiments, the normalization transformation is performed according to a specific mathematical form to generate a feedback signal. The formula for calculating the feedback signal is expressed as follows: Where: symbol The calculated feedback signal is represented by the symbol. Represents the maximum possible cumulative reward for a reinforcement learning task over a full round, denoted by [symbol]. The symbol represents the minimum possible cumulative reward of a reinforcement learning task over a full round. This represents the actual cumulative reward from the previous round. Based on the specific value, [the reward will be calculated / adjusted]. , and Substituting into the formula, the feedback signal is calculated. This value, between 0 and 1, is used as a feedback state input to the learnable model.

[0028] Optionally, the maximum and minimum possible cumulative rewards can be determined based on historical round data of the reinforcement learning task. In specific implementations, for continuous control tasks like HalfCheetah-v4, the single-step reward includes both forward reward and control cost penalty, and the cumulative reward has no theoretically absolute upper or lower bounds. Before the packaging process starts, multiple exploratory rounds of the reinforcement learning task can be pre-run, for example, 100 rounds with a random policy. From the historical cumulative reward data of these 100 rounds, the observed maximum and minimum cumulative rewards are statistically analyzed and used as predefined values ​​for the maximum and minimum possible cumulative rewards, respectively. Assuming the maximum cumulative reward recorded in the historical runs is 1025.6 and the minimum cumulative reward recorded is -189.4, then the maximum possible cumulative reward is fixed at 1025.6, and the minimum possible cumulative reward is fixed at -189.4.

[0029] In some embodiments, the maximum and minimum possible cumulative rewards are directly derived from the specifications or design documents of the reinforcement learning task itself. In specific implementations, some reinforcement learning tasks explicitly define theoretical limits to task performance. For example, some tasks rewarded by achieving a specific objective have a maximum possible cumulative reward corresponding to the total reward for perfectly achieving all sub-objectives, and a minimum possible cumulative reward corresponding to the total penalty for complete failure. The packaging process directly follows the task specification text, using the theoretical limits stated in the document as fixed parameters for the maximum and minimum possible cumulative rewards. It can be understood that, whether based on historical data statistics or the task specification definition, the maximum and minimum possible cumulative rewards are treated as known, unchanging values ​​in the calculation of the feedback signal during the packaging process.

[0030] Optionally, the calculation of the feedback state is implemented as a separate function within the wrapper. In practice, the wrapper calls this calculation function each time a reinforcement learning task round ends. The calculation function receives three input parameters: a predefined maximum possible cumulative reward, a predefined minimum possible cumulative reward, and the actual cumulative reward recorded in the previous round. The calculation function performs arithmetic operations strictly according to the formula and returns the result. It can be understood that the normalization transformation ensures that cumulative rewards of different magnitudes and ranges can be mapped to a relatively uniform numerical range, thus adapting to the requirements of the learnable model for the scale of input data.

[0031] In one embodiment of the invention, the packaged learnable task comprises round-based interaction data from multiple reinforcement learning tasks. After each independent round-based interaction, a corresponding cumulative reward is generated. The cumulative rewards from multiple rounds are used to generate a comprehensive fitness value, which is used to evaluate the overall performance of the learnable model on a task. The fitness value is calculated by weighted averaging of these cumulative rewards. In the weighted averaging calculation, each round's cumulative reward is assigned a weight, with the weight set such that an increasing round number results in an increasing weight. This means that when calculating fitness, the cumulative rewards generated in subsequent rounds have higher importance than those in earlier rounds.

[0032] In practice, the packaged learnable task contains round-based interaction data from multiple reinforcement learning tasks. Each independent round generates a corresponding cumulative reward upon termination, and these cumulative rewards serve as the foundational data for evaluating model performance. For example, in training a learnable model to solve the CartPole-v1 task, the model is executed sequentially for five complete rounds, and a cumulative reward value is recorded for each round. Let's assume the cumulative rewards for these five rounds are 25.0, 80.0, 150.0, 300.0, and 450.0, respectively.

[0033] In some embodiments, the cumulative rewards from multiple rounds are used to generate a single, comprehensive fitness value, which quantifies the overall performance of the learnable model on a specific task. The fitness value is not calculated by simply taking the arithmetic mean of all cumulative rewards, but rather by a weighted average. This weighted average means that the cumulative reward from each round contributes a share to the final summed fitness value, but with different weights for each share.

[0034] Optionally, the weights in the weighted average calculation follow a clear rule: the weights increase with the round number. In practice, the round number refers to the order in which the model executes rounds during training; the first round is numbered 1, the second 2, and so on. The weight assigned to each round is positively correlated with its round number; for example, the round number can be directly used as the weight. Assuming there are five rounds with round numbers from 1 to 5, the corresponding weight sequence could be 1, 2, 3, 4, 5.

[0035] In some embodiments, the specific formula for calculating the fitness value can be expressed as follows: Where: symbol Represents the calculated fitness value, symbol Represents the total number of rounds used for calculation, symbol The sequence number representing a specific round, symbol The representative serial number is The weight value corresponding to the round, symbol The representative serial number is The actual cumulative reward obtained in each round. Taking the data from the aforementioned five rounds as an example, the cumulative reward... If weights are set Then the numerator is calculated as The denominator is calculated as follows: fitness value It's understandable that if the arithmetic mean is calculated directly without weighting, the result would be... The weighted average result of 318.67 is significantly higher than the arithmetic mean, reflecting the greater influence of higher cumulative returns in the later stages.

[0036] Optionally, the specific function for the weights to increase with the round number can be linear or non-linear, such as using the square of the round number as the weight. In a practical implementation, assuming there are five rounds, a squared weight is used. Then the fitness value calculation becomes .

[0037] In one embodiment of the present invention, the fitness value is calculated dynamically during the training of the learnable model, and the model parameters are adjusted based on the calculated fitness value as a feedback signal. The transformation method used in calculating the feedback state is not limited to normalization transformation, but also includes logarithmic scaling transformation or piecewise function transformation. Logarithmic scaling transformation performs numerical scaling by taking the logarithm of the cumulative reward, while piecewise function transformation maps the cumulative reward to different function intervals according to the numerical range of the cumulative reward. These transformation methods are used to map cumulative reward signals of different magnitudes to numerical intervals suitable for model learning.

[0038] In practice, the calculation of fitness values ​​is synchronized with the training of the learnable model and dynamically completed within the training loop. The update of model parameters directly depends on the fitness value generated in each dynamic calculation. For example, when training a neural network model to solve the CartPole-v1 task using an evolutionary strategy, each generation of training involves evaluating multiple individual models in the population. Assume a population contains 5 different individual neural network models, each deployed to the packaged learnable task and running independently for 3 rounds, receiving 3 cumulative rewards. For one individual model, its cumulative rewards over 3 rounds are [50.0, 180.0, 400.0]. Based on a weighted average or other aggregation rule, the fitness value of this model in the current generation can be dynamically calculated immediately, for example, 280.5. This fitness value, just calculated in this training iteration, is then used as a feedback signal. This feedback signal is passed to the optimizer, which adjusts the parameter weights of the neural network model based on the differences in fitness values ​​among all individual models, for example, assigning larger update weights to the parameters of individual models with higher fitness values.

[0039] In some embodiments, the transformation used to calculate the feedback state is not limited to normalization; logarithmic scaling is an alternative method to map the cumulative reward to a specific numerical range. In practice, for reinforcement learning tasks with a large range of cumulative rewards or a highly skewed distribution, logarithmic scaling can compress the numerical range. The mathematical expression of logarithmic scaling is: Where: symbol This represents the feedback signal obtained after logarithmic scaling transformation, with the symbol... Represents the actual cumulative reward from the previous round that needs to be transformed, symbol Represents the base of the logarithm, sign Represents a constant offset. Sets the base of the logarithm. constant offset If the cumulative reward from the previous round Then calculate the feedback signal. If the cumulative return Then the feedback signal By logarithmic scaling, the cumulative returns, which originally varied greatly, are mapped to feedback signals with smaller differences. This transformed feedback signal... As a feedback state input model.

[0040] Optionally, piecewise function transformation is another way to map cumulative returns to specific numerical intervals. Piecewise function transformation works by defining different numerical intervals of the cumulative return and their corresponding mapping rules. In a specific implementation, a piecewise function with three intervals is defined. Assume the cumulative return... The theoretical range is [-1000, 1000]. The piecewise function transformation rule can be defined as follows: when the cumulative return... When the value is less than 0, change the output value. Fixed at 0; when the cumulative return When the value is greater than or equal to 0 and less than 500, change the output value. Equal to cumulative return Divide by 500; when the cumulative return When the value is greater than or equal to 500, change the output value. It is fixed at 1. According to this rule, different cumulative returns are mapped to the interval [0,1].

[0041] In some embodiments, the specific mapping relationship of the piecewise function transformation can be implemented through a lookup table or logical conditional statements. The table below shows an example of a piecewise function transformation, illustrating how different cumulative reward inputs are mapped to output values. In a specific implementation, the transformation function inside the wrapper receives the cumulative reward. As input, by sequentially judging Within the specified interval, perform the corresponding calculations or directly assign a preset value to generate the final transformed feedback signal. See Table 1.

[0042] Table 1: Piecewise Function Transformation Mapping Table Optionally, the specific use of the fitness value as a feedback signal during model training depends on the optimization algorithm employed. In practice, if a population-based evolutionary algorithm is used, the dynamic fitness value obtained by each individual model in each generation of training is used to calculate its probability of being selected for the next generation; individuals with higher fitness values ​​have a greater probability of being selected. If a gradient-based policy gradient method is used, the dynamically calculated fitness value can be used as part of the baseline or advantage function to scale the gradient of policy updates.

[0043] In one embodiment of the present invention, data from multiple historical rounds are stored during the implementation of the method. The stored data includes the sequence of environmental state observations generated in each round, the sequence of actions output by the model, and the cumulative reward obtained at the end of that round. When it is necessary to use these data to train a learnable model, a certain number of samples are randomly selected from all the stored historical round data to form a batch, and the model parameters are updated using the data from this batch.

[0044] In its implementation, the method includes storing interaction data from multiple historical rounds. This stored data covers the sequence of environmental state observations, action sequences, and corresponding cumulative rewards for each round. For example, during the training of a learnable model to solve the HalfCheetah-v4 task, the wrapper executes 100 complete rounds consecutively. Each round begins with a call to the reset function `reset()` and ends when the execution function `step()` returns a termination flag of `True`. In a specific implementation of training a learnable model to solve the HalfCheetah-v4 task, after initialization, the wrapper calls the reset function `reset()` to obtain the initial state observations of the environment. These observations are 17-dimensional floating-point vectors containing information such as the positions and velocities of the agent's joints. The learnable model outputs an action based on the current state observation. This action is a 6-dimensional continuous action vector, with each dimension ranging from -1 to 1, representing the torque applied to different joints. The wrapper passes this action to the execution function `step()`, which updates the internal state of the environment based on the input action and returns the new state observations, immediate reward, and termination flag. The interaction process repeats until the termination flag returned by the execution function `step()` is True, marking the end of the round. At this point, the wrapper calculates the cumulative reward for this round, which is the sum of the immediate rewards for all steps, and stores the entire round's environment state observation sequence, action sequence, and cumulative reward in a historical data storage structure. For a specific round, the interaction process lasts for 1000 time steps. Therefore, the environment state observation sequence for this round contains 1000 seventeen-dimensional floating-point vectors, the action sequence contains 1000 six-dimensional floating-point action vectors, and the cumulative reward calculated at the end of the round is a specific floating-point value, such as -45.3. The wrapper completely saves the trajectory data for these 100 rounds. Each trajectory data consists of a triple, which contains the environment state observation sequence, action sequence, and cumulative reward value. In terms of data comparison, traditional online reinforcement learning algorithms usually only use the most recently generated interaction data of the current policy and discard it immediately. However, this implementation establishes a historical data repository, retaining the complete interaction trajectories under different policy performances in multiple past rounds.

[0045] In some embodiments, historical round data is stored through a data structure. Specifically, a list or a circular buffer can be used to store data from multiple historical rounds. Each storage entry corresponds to a complete round and contains three parts: an observation list, storing the environmental state observations for each step in chronological order; an action list, storing the actions output by the model for each step in the same chronological order; and a floating-point scalar, storing the cumulative reward for that round. For example, in the CartPole-v1 task, the 15th stored entry contains data from 200 time steps, with 200 four-dimensional arrays in the observation list, 200 integers (0 or 1) in the action list, and a cumulative reward scalar of 200.0. After completing each new round, the wrapper adds the observation list, action list, and cumulative reward scalar of that new round as a new entry to the historical data storage structure.

[0046] Optionally, the storage structure sets an upper limit on the total number of historical rounds stored. In practice, a fixed capacity can be set, such as 5000 rounds. When the number of historical rounds stored reaches this limit, if new round data needs to be stored, the oldest stored round data is overwritten according to the first-in, first-out (FIFO) principle. In this way, the historical data storage structure always retains the interaction data of the most recently generated rounds, ensuring that the data used for training is relatively new and reflects the recent policy performance of the model. When training the learnable model, updating the model parameters requires randomly sampling a batch of data from the stored historical round data. The random sampling process first randomly selects several rounds from all the stored historical rounds, and then randomly selects one or more time steps of data points from each selected round.

[0047] In some embodiments, the process of randomly sampling a batch of data specifically involves: setting a batch size, for example, setting the batch size to 64. Each time model parameters need to be updated, N rounds are randomly selected from a storage structure containing M historical rounds, with N less than M. Assuming M is 1000, N can be set to 16. For each randomly selected round, such as the 247th round, this round contains data from L time steps. Then, K time step indices are randomly selected from these L time steps with a uniform distribution, where K can be set to 4. Finally, the environmental state observations and actions corresponding to each selected time step in each selected round are extracted to form a dataset containing N... A batch of K data samples. These data samples come from completely different rounds and time points, and are fed together into a learnable model to calculate the loss or gradient, thereby updating the model parameters.

[0048] Optionally, the random sampling process can use a probability distribution to guide the selection of historical rounds, rather than a simple uniform distribution. In practice, the sampling probability can be allocated based on the cumulative reward value of each historical round, with rounds having a higher cumulative reward having a greater probability of being sampled. A formula for calculating the sampling probability is defined as follows: Where: symbol Represents the probability of being selected in the e-th historical round, symbol [symbol missing]. This represents the weight assigned to the e-th historical round, which can be some function of its cumulative reward, denoted by [symbol]. This represents the total number of historical rounds stored. Through this non-uniform sampling, model parameter updates can focus more on learning the decision patterns from those historical rounds that performed better.

[0049] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for packaging reinforcement learning tasks into learnable tasks, characterized in that, The method includes the following steps: Obtain a reinforcement learning task, which includes a reset function reset() and an execution function step(), wherein the environmental state observations obs of the reinforcement learning task are used as input, the action is used as output, and a reward is returned; The reinforcement learning task is packaged such that the environmental state observations obs of the reinforcement learning task are mapped to the environmental state observations obs of the learnable model, and the action of the reinforcement learning task is mapped to the action of the learnable model. Add a feedback state input to the learnable model, the feedback state being generated after transforming the cumulative rewards of a full epoch of the reinforcement learning task; The learnable model is trained using a packaged learnable task, wherein the learnable task contains rounds of data from multiple reinforcement learning tasks.

2. The method for packaging reinforcement learning tasks as a learnable task as described in claim 1, characterized in that, The reinforcement learning task is a task based on the Gym framework, wherein the reset function reset() is used to initialize the environment state, and the execution function step() is used to update the environment state according to the input action and return the reward.

3. The method for packaging reinforcement learning tasks as a learnable task as described in claim 1, characterized in that, The steps of packaging the reinforcement learning task include: The environmental state observations obs of the reinforcement learning task are directly used as the input of the environmental state observations obs of the learnable model; The action of the reinforcement learning task is directly used as the output of the action of the learnable model; The cumulative rewards of a complete round of the reinforcement learning task are used as the basis for calculating the feedback state.

4. The method for packaging reinforcement learning tasks as a learnable task as described in claim 3, characterized in that, The calculation of the feedback state is based on a transformation of the cumulative rewards, which is a normalized transformation, specifically in the form of: The feedback signal s is calculated using the following formula: s = (rewards_max - reward) / (rewards_max - reward_min), where reward_max represents the maximum possible value of the cumulative reward of the reinforcement learning task in a complete round, reward_min represents the minimum possible value of the cumulative reward of the reinforcement learning task in a complete round, and reward represents the actual cumulative reward of the previous round.

5. The method for packaging reinforcement learning tasks as a learnable task as described in claim 4, characterized in that, The values ​​of rewards_max and rewards_min are predefined based on the historical round data or task specifications of the reinforcement learning task and are input as fixed parameters during the packaging process.

6. The method for packaging reinforcement learning tasks as a learnable task as described in claim 1, characterized in that, The packaged learnable task comprises multiple rounds of the reinforcement learning task, with each round generating a corresponding cumulative reward. The method further includes: A weighted average of the cumulative rewards over multiple rounds is calculated to generate a fitness value, which is used to evaluate the performance of the learnable model.

7. The method for packaging reinforcement learning tasks as a learnable task as described in claim 6, characterized in that, In the weighted average calculation, the cumulative rewards for each round are assigned a weight, which increases with the round number, making the scores of subsequent rounds more important than the scores of previous rounds.

8. The method for packaging reinforcement learning tasks as a learnable task as described in claim 6, characterized in that, The fitness value is calculated dynamically during training, and is used as a feedback signal to adjust the parameters of the learnable model.

9. The method for packaging reinforcement learning tasks as a learnable task as described in claim 1, characterized in that, The transformations used in calculating the feedback state also include logarithmic scaling or piecewise function transformation, which are used to map the cumulative rewards to a specific numerical range to accommodate reward signals of different magnitudes.

10. The method for packaging a reinforcement learning task as a learnable task as described in claim 1, characterized in that, The method further includes the following steps: Store environmental state observations (obs), actions, and corresponding cumulative rewards from multiple historical rounds; When training the learnable model, a batch of data is randomly sampled from the stored historical round data for updating the model parameters.

Images