Data processing method and system based on continuous reinforcement learning

Abstract

The invention discloses a data processing method and system based on continuous reinforcement learning, and belongs to the field of machine learning and artificial intelligence, and the method specifically comprises the steps: sampling conversion data from a scene memory module and a playback buffer module, calculating the strategy network gradient of a previous task and the strategy network gradient of a current task, and calculating a gradient included angle according to the strategy network gradient; the current task gradient direction is adjusted according to the gradient included angle, and a countermeasure sample is generated; updating a strategy network parameter and a criticality network parameter by using an adversarial sample, and dynamically adjusting a learning rate at the same time; in the iterative training process, the memory module and the cache module are continuously updated, and the model performance is regularly evaluated; the method can effectively improve the training efficiency and generalization ability of the reinforcement learning model, and prevents the model from forgetting old knowledge.

Description

Technical Field

[0001] The present invention belongs to the field of machine learning and artificial intelligence, and specifically relates to a data processing method and system based on continuous reinforcement learning. Background Art

[0002] Continuous reinforcement learning is a technique that allows machine learning algorithms to learn and adapt in a series of continuously changing tasks. In the process of continuous learning, the model faces a major challenge: the learning of new knowledge may lead to the loss of old knowledge, a phenomenon called catastrophic forgetting. Although there are some strategies such as experience replay and elastic weight combination to alleviate this problem to a certain extent, these methods usually cannot fully meet the requirement of maintaining the performance of old tasks while learning new tasks.

[0003] For example, a Chinese patent with publication number CN117034770A discloses a data processing method, device, electronic device and storage medium, including: constructing a simulation scene according to a scene configuration file, the simulation scene includes an intelligent agent with a continuous action space and an intelligent agent with a discrete action space; determining the action instructions of each intelligent agent according to the output of the reinforcement learning model, and in the same timeline, whenever the control time step of the intelligent agent is reached, the intelligent agent is controlled to execute the corresponding action instruction, thereby determining the scene observation information and the reward value, and the reinforcement learning model can be trained based on the scene observation information and the reward value. By constructing a simulation scene with a mixture of multiple action spaces, a test environment with higher authenticity can be provided, thereby improving the training effect of reinforcement learning. Therefore, the simulation scene can provide a test environment with higher authenticity, thereby improving the effect of reinforcement learning, and can be widely used in cloud technology, artificial intelligence, smart transportation, assisted driving and other scenarios.

[0004] The above existing technologies all have the following problems: lack of specific training steps and lack of flexibility; the description is relatively broad and lacks specific implementation details, making it difficult to implement and reproduce. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention proposes a data processing method and system based on continuous reinforcement learning, which samples and converts data from the context memory module and the playback cache module, calculates the policy network gradients of the previous and current tasks, and calculates the gradient angle accordingly; adjusts the current task gradient direction according to the gradient angle, and generates adversarial samples; uses adversarial samples to update the policy network and criticism network parameters, and dynamically adjusts the learning rate; during the iterative training process, continuously updates the memory module and the cache module, and regularly evaluates the model performance; this method can effectively improve the training efficiency and generalization ability of the reinforcement learning model, and prevent the model from forgetting old knowledge.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] Data processing methods based on continuous reinforcement learning include:

[0008] Step S1: Initialize the situational memory module and the playback cache module. When completing the past task, store the conversion samples in the situational memory module. When processing the current task, collect the conversion data in the task in real time and store it in the playback cache module.

[0009] Step S2: sampling conversion data from the episodic memory module and the playback cache module, using the sampled conversion data to calculate the policy network gradients of the previous task and the current task respectively, and calculating the gradient angle between the previous task and the current task based on the calculated policy network gradient results;

[0010] Step S3: adjust the gradient direction of the current task according to the calculated gradient angle, add perturbations to the state of the current task, and generate adversarial samples according to the adjusted gradient direction and perturbation size;

[0011] Step S4: Use the generated adversarial samples to update the policy network and critic network parameters, and dynamically adjust the learning rate according to the current learning state to optimize the policy network and critic network parameters;

[0012] Step S5: Repeat steps S1 to S4 to perform iterative training. In each iteration, new conversion data is used to update the context memory module and the replay cache module. At the same time, the performance of the reinforcement learning model is regularly evaluated.

[0013] Specifically, the conversion data in the episodic memory module and the replay cache module are stored in the form of a four-tuple of state-action-reward-subsequent state, and the sampling data of the episodic memory module is in the form of The playback cache module sampling data is in the form of in, represents the state of the environment at the i-th moment in task k, Indicates in status The following actions are taken, Indicates taking action Back from state The reward for moving to the next state, Indicates taking action After transferring to the new state, represents the state of the environment at the i-th moment in the current task, Indicates in status The following actions are taken, Indicates taking action Back from state The reward for moving to the next state, Indicates taking action Then transfer to the new state.

[0014] Specifically, the specific steps of step S2 include:

[0015] S2.1: Sample the transition data of the previous task from the episodic memory module and sample the transition data of the current task from the replay buffer module;

[0016] S2.2: The sampled conversion data and Arrange it into the input format of the neural network;

[0017] S2.3: Use and As input, the policy network gradient g of the previous task is calculated through the policy network M and the policy network gradient g of the current task D , the formula is:

[0018]

[0019] in, represents the loss function of the policy network, π θ represents the policy function about the policy network parameter θ, S i represents the state of the previous task, S i ' represents the state of the current task, M represents the episodic memory library of the previous task, D represents the playback buffer library of the current task, and ▽ represents the gradient symbol;

[0020] S2.4: Calculate the angle between the policy network gradients for the previous task and the current task Among them, cos -1 (·) represents the arc cosine function, and ||·|| represents the modulus of the vector.

[0021] Specifically, the specific steps of step S3 include:

[0022] S3.1: Get the gradient angle between the previous task and the current task And according to Determine the degree to which the current task gradient direction needs to be adjusted;

[0023] S3.2: Add perturbation information ε to the state of the current task according to the degree to which the gradient direction of the current task needs to be adjusted;

[0024] S3.3: Generate adversarial samples based on the adjusted gradient direction and the added perturbation size The formula is:

[0025]

[0026] Among them, ε represents the disturbance information, sign(·) represents the sign function, H(·) represents the step function, ⊙ represents the element-by-element product, Indicates g D The transpose of , T stands for transpose.

[0027] Specifically, the specific steps of step S4 include:

[0028] S4.1: Obtaining Generated Adversarial Examples

[0029] For the policy network, use adversarial examples Calculate the strategy loss function Lπ, the formula is: L π =-∑logπ(a|s;θ π )×A(s,a), where, π(a|s; θ π ) means that in a given state s and parameter θ π The probability of taking action a under s is the output of the policy network, and A(s,a) represents the advantage function;

[0030] For the critic network, use adversarial examples Calculate the temporal difference loss function L Q , the formula is: L Q =-∑(Q(s,a;θ Q )-y) 2 , where Q(s,a;θ Q ) means that given the state s and action a, and the parameter θ Q The expected future reward is the output of the critic network, θ Q represents the critical network parameters, and y represents the target of the temporal difference function.

[0031] Specifically, the specific steps of step S4 also include:

[0032] S4.2: According to the policy loss function L π And the temporal difference loss function L Q , calculate the gradient of the policy network and the gradient of the critical network parameters The formula is:

[0033]

[0034] Among them, E π [·] represents the expected value under the strategy π, R represents the reward function, and N represents the number of gradient samples;

[0035] S4.3: Based on the calculated gradient and The parameters of the policy network and the critic network are updated through the optimization algorithm. The formula is:

[0036]

[0037] in, represents the updated policy network parameters, represents the updated critic network parameters, and β represents the learning rate;

[0038] S4.4: Based on the current learning state, adjust the learning rate according to the change of loss value;

[0039] S4.5: Repeat S4.1 to S4.5 until the preset number of training rounds is reached.

[0040] The data processing system based on continuous reinforcement learning includes: a situational memory module, a playback cache module, an adversarial sample generation module, and a model update module;

[0041] The context memory module is used to store conversion samples generated when completing past tasks;

[0042] The playback cache module is used to collect and store the conversion data in the current task in real time;

[0043] The adversarial sample generation module is used to sample conversion data from the episodic memory module and the playback module, calculate the policy network gradient information of the previous task and the current task, use the projected gradient descent method and the step function to adjust the gradient direction of the current task according to the gradient angle, and add perturbations to the state of the current task, and generate adversarial samples according to the adjusted gradient direction and perturbation size;

[0044] The model updating module is used to automatically apply the generated adversarial samples to update the policy network and the critic network, and dynamically adjust the learning rate according to the current learning state.

[0045] Specifically, the adversarial sample generation module includes: a gradient calculation unit, an angle evaluation unit, a gradient adjustment unit, and a disturbance adding unit;

[0046] The gradient calculation unit is used to sample data from the context memory and the replay buffer to calculate the gradient of the policy network;

[0047] The angle evaluation unit is used to evaluate the angle between the gradient of the previous task and the current task, and determine the similarity and conflict between the tasks;

[0048] The gradient adjustment unit is used to adjust the gradient direction of the current task according to the angle;

[0049] The disturbance adding unit is used to add disturbance to the state of the current task and generate adversarial samples according to the adjusted gradient direction and disturbance size.

[0050] Specifically, the model updating module includes: a parameter updating unit, a learning rate adjusting unit, and a performance evaluation unit;

[0051] The parameter updating unit is used to update the parameters of the policy network and the critic network using adversarial samples;

[0052] The learning rate adjustment unit is used to dynamically adjust the learning rate according to the current learning state;

[0053] The performance evaluation unit is used to periodically evaluate the performance of the reinforcement learning model.

[0054] Specifically, the context memory module, playback cache module, adversarial sample generation module, and model update module are seamlessly integrated with the continuous reinforcement learning framework.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] 1. The present invention proposes a data processing system based on continuous reinforcement learning, and optimizes and improves the architecture, operation steps and processes. The system has the advantages of simple process, low investment and operation costs, and low production work costs.

[0057] 2. The present invention proposes a data processing method based on continuous reinforcement learning. By combining historical samples in episodic memory and current task samples, adversarial samples are effectively generated to improve the robustness and generalization ability of the model. In the data collection stage, the system ensures that historical information is completely preserved by saving transition samples of state-action-reward-subsequent state of past tasks. When generating adversarial samples, the algorithm uses a projected gradient descent attack to generate the most challenging adversarial samples by calculating the gradients of the current task and the previous task, maximizing the loss of the current task and the past task, and avoiding the model from forgetting old knowledge. This not only enhances the performance of the model in new tasks, but also prevents catastrophic forgetting problems, ensuring that the model always remains efficient and stable when processing continuous tasks.

[0058] 3. The present invention proposes a data processing method based on continuous reinforcement learning, and also proposes a framework based on data enhancement for continuous reinforcement learning, which collects data through interaction with the environment, and optimizes the policy network and the critic network by combining data enhancement technology and original conversion samples. The framework optimizes the training process of the policy network by averaging the policy value, Q value and target Q value, enhances the generalization ability of the model, and reduces the risk of overfitting; through the generation and use of these enhanced data samples, the present invention significantly improves the learning efficiency and performance of the model in multi-task scenarios, and effectively responds to possible distribution changes in future tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a schematic diagram of the data processing method based on continuous reinforcement learning of the present invention;

[0060] Figure 2 This is a data flow diagram of the data processing method based on continuous reinforcement learning of the present invention;

[0061] Figure 3 This is a framework diagram of data enhancement of the data processing method based on continuous reinforcement learning of the present invention;

[0062] Figure 4 This is an architecture diagram of the data processing system based on continuous reinforcement learning of the present invention. DETAILED DESCRIPTION

[0063] Example 1

[0064] See also Figure 1-Figure 3 The present invention provides an embodiment: a data processing method based on continuous reinforcement learning, which is mainly applied to robotics and autonomous driving systems to solve the catastrophic forgetting problem faced by traditional continuous reinforcement learning algorithms when processing continuous tasks, thereby improving the adaptability and efficiency of the system in a dynamic and continuously changing environment, comprising the following steps:

[0065] Step S1: Initialize the situational memory module and the playback cache module. When completing the past task, store the conversion samples in the situational memory module. When processing the current task, collect the conversion data in the task in real time and store it in the playback cache module.

[0066] The conversion data in the episodic memory module and the replay cache module are stored in the form of a four-tuple of state-action-reward-subsequent state. The sampling data of the episodic memory module is in the form of The playback cache module sampling data is in the form of in, represents the state of the environment at the i-th moment in task k, Indicates in status The following actions are taken, Indicates taking action Back from state The reward for moving to the next state, Indicates taking action After transferring to the new state, represents the state of the environment at the i-th moment in the current task, Indicates in status The following actions are taken, Indicates taking action Back from state The reward for moving to the next state, Indicates taking action Then transfer to the new state.

[0067] Step S2: sampling conversion data from the episodic memory module and the playback cache module, using the sampled conversion data to calculate the policy network gradients of the previous task and the current task respectively, and calculating the gradient angle between the previous task and the current task based on the calculated policy network gradient results;

[0068] Step S3: adjust the gradient direction of the current task according to the calculated gradient angle, add perturbations to the state of the current task, and generate adversarial samples according to the adjusted gradient direction and perturbation size;

[0069] Step S4: Use the generated adversarial samples to update the policy network and critic network parameters, and dynamically adjust the learning rate according to the current learning state to optimize the policy network and critic network parameters;

[0070] Step S5: Repeat steps S1 to S4 to perform iterative training. In each iteration, new conversion data is used to update the context memory module and the replay cache module. At the same time, the performance of the reinforcement learning model is regularly evaluated.

[0071] The specific steps of step S2 include:

[0072] S2.1: Sample the transition data of the previous task from the episodic memory module and sample the transition data of the current task from the replay buffer module;

[0073] S2.2: The sampled conversion data and Arrange it into the input format of the neural network;

[0074] Among them, when organizing into the input format of the neural network, it is necessary to ensure that the input layer of the neural network matches the constructed sampled conversion data, including adjusting the number of neurons, activation function and other parameters of the input layer to adapt to the dimension and characteristics of the sampled conversion data.

[0075] S2.3: Use and As input, the policy network gradient g of the previous task is calculated through the policy network M and the policy network gradient g of the current task D , the formula is:

[0076]

[0077] in, represents the loss function of the policy network, π θ represents the policy function about the policy network parameter θ, S i represents the state of the previous task, S i ' represents the state of the current task, M represents the episodic memory library of the previous task, D represents the playback buffer library of the current task, and ▽ represents the gradient symbol;

[0078] S2.4: Calculate the angle between the policy network gradients for the previous task and the current task Among them, cos -1 (·) represents the arc cosine function, and ||·|| represents the modulus of the vector.

[0079] The specific steps of step S3 include:

[0080] S3.1: Get the gradient angle between the previous task and the current task And according to Determine the degree to which the current task gradient direction needs to be adjusted;

[0081] Among them, according to the calculated gradient angle, the degree to which the gradient direction of the current task needs to be adjusted is determined, which can be achieved by setting a threshold or ratio. For example, if the gradient angle is greater than or equal to the preset threshold, it means that the gradient directions of the two tasks are quite different and require a greater adjustment; conversely, if the gradient angle is less than the preset threshold, the degree of adjustment may be smaller.

[0082] S3.2: Add perturbation information ε to the state of the current task according to the degree to which the gradient direction of the current task needs to be adjusted;

[0083] Furthermore, the specific steps of S3.2 include:

[0084] (1) Determine a perturbation coefficient based on the degree to which the gradient direction of the current task needs to be adjusted. This coefficient can be a scalar used to control the size of the perturbation.

[0085] (2) Generate a disturbance information that matches the current task state based on the determined disturbance coefficient. This disturbance information can be a random vector or noise, and its dimension should be the same as the dimension of the current task state.

[0086] (3) Add the generated disturbance information to the state of the current task to obtain a new state with disturbance information;

[0087] (4) Use the new state with perturbation information for subsequent reinforcement learning training process.

[0088] S3.3: Generate adversarial samples based on the adjusted gradient direction and the added perturbation size The formula is:

[0089]

[0090] Among them, ε represents the disturbance information, sign(·) represents the sign function, H(·) represents the step function, ⊙ represents the element-by-element product, Indicates g D The transpose of , T stands for transpose.

[0091] The specific steps of step S4 include:

[0092] S4.1: Obtaining Generated Adversarial Examples

[0093] For the policy network, use adversarial examples Calculate the strategy loss function Lπ, the formula is: L π =-∑logπ(a|s;θ π )×A(s,a), where, π(a|s; θ π ) means that in a given state s and parameter θ π The probability of taking action a under s is the output of the policy network, and A(s,a) represents the advantage function;

[0094] For the critic network, use adversarial examples Calculate the temporal difference loss function L Q , the formula is: L Q =-∑(Q(s,a;θ Q )-y) 2 , where Q(s,a;θ Q ) means that given the state s and action a, and the parameter θ Q The expected future reward is the output of the critic network, θ Q represents the critical network parameters, and y represents the target of the temporal difference function;

[0095] S4.2: According to the policy loss function L π And the temporal difference loss function L Q , calculate the gradient of the policy network and the gradient of the critical network parameters The formula is:

[0096]

[0097] Among them, E π [·] represents the expected value under the strategy π, R represents the reward function, and N represents the number of gradient samples;

[0098] S4.3: Based on the calculated gradient and The parameters of the policy network and the critic network are updated through the optimization algorithm. The formula is:

[0099]

[0100] in, represents the updated policy network parameters, represents the updated critic network parameters, and β represents the learning rate;

[0101] S4.4: Based on the current learning state, adjust the learning rate according to the change of loss value;

[0102] Furthermore, the specific steps of S4.4 include:

[0103] (1) Before training begins, set an initial learning rate;

[0104] (2) For each training iteration:

[0105] a. Calculate the loss value using the current model parameters and training data;

[0106] b. Monitor the changing trend of loss value;

[0107] c. Adjust the learning rate according to the change of loss value. If the loss value decreases slowly or stagnates, you need to reduce the learning rate. If the loss value fluctuates too much, you need to increase the learning rate.

[0108] d. Update model parameters using adjusted learning rate and gradient;

[0109] (3) Check whether the predetermined number of training rounds has been reached. If the stopping condition is met, output the trained model; otherwise, continue to the next training iteration.

[0110] S4.5: Repeat S4.1 to S4.5 until the preset number of training rounds is reached.

[0111] Example 2

[0112] See also Figure 4 Another embodiment provided by the present invention is a data processing system based on continuous reinforcement learning, comprising:

[0113] Episodic memory module, replay cache module, adversarial sample generation module, model update module;

[0114] An episodic memory module that stores transition samples from past tasks to aid in long-term memory retention and provide historical context;

[0115] The replay cache module is used to collect and store the transformation data of the current task in real time to support immediate policy updates and training needs;

[0116] It should be understood that the replay cache module has real-time data collection capabilities, which is used to record and store sample data in real time during the execution of the current task. The module is specially configured to work synchronously with the task execution environment, capturing state changes and decision results at each moment, and transmitting these data to the adversarial sample generation module in real time. This function ensures that the generated adversarial samples can reflect the latest task dynamics, thereby making the policy network more adaptable and robust to current and future task environments.

[0117] The adversarial sample generation module is used to sample and transform data from the episodic memory module and the playback module, calculate the policy network gradient information of the previous task and the current task, use the projected gradient descent method and the step function to adjust the gradient direction of the current task according to the gradient angle, and add perturbations to the state of the current task, and generate adversarial samples according to the adjusted gradient direction and perturbation size;

[0118] It should be noted that the configuration of the adversarial sample generation module requires the following steps:

[0119] Step 1: Collect data from the episodic memory module and the playback buffer module;

[0120] D〗) gradient;

[0121] Step 3: Apply the Heaviside step function to compare the angle between g_M and g_D. If the angle between the two is greater than 90 degrees, adjust the gradient g_D of the current task to zero, otherwise it remains unchanged to ensure that the loss in past context memory will not decrease, thereby generating adversarial samples.

[0122] A model update module that automatically applies the generated adversarial examples to update the policy network and the critic network, and dynamically adjusts the learning rate based on the current learning state.

[0123] It should be noted that the model update module uses the generated adversarial samples to train the model to adapt to the new task environment and optimize the model performance while retaining the old task knowledge. This module adopts dynamic learning rate adjustment to ensure that in the continuous task learning process, the network parameter update can quickly adapt to the new task without losing the memory of the previous task.

[0124] It should be noted that the data enhancement system in the present invention adopts a modular plug-in design, which allows seamless integration into various standard continuous reinforcement learning frameworks, such as mainstream algorithms such as soft actor-critic. This module does not need to modify the existing reinforcement learning algorithm architecture, ensuring compatibility with multiple learning frameworks. In addition, this plug-in design increases the flexibility of the present invention in multiple application scenarios such as robotics and autonomous driving, and also supports customization of data enhancement strategies according to specific application requirements. While improving the versatility and adaptability of the present invention, it also significantly improves the practicality and efficiency of the continuous reinforcement learning system based on the data enhancement method.

[0125] The episodic memory module includes: a data storage unit and a data retrieval unit;

[0126] Data storage unit, used to store transition samples of past tasks, including states, actions, and rewards;

[0127] A data retrieval unit is used to retrieve transformation samples from episodic memory as needed for subsequent processing.

[0128] The playback cache module includes: a real-time acquisition unit, a cache storage unit, and a data update unit;

[0129] The real-time acquisition unit is used to collect the conversion data in the current task in real time to ensure that the model can use the latest data for learning;

[0130] A cache storage unit is used to store the collected conversion data in the playback cache, maintain the timeliness of the data in the playback cache, and improve the generalization ability of the model;

[0131] The data updating unit is used to update the playback cache according to the new conversion data to ensure that the data in the cache is up to date.

[0132] The adversarial sample generation module includes: gradient calculation unit, angle evaluation unit, gradient adjustment unit, and disturbance addition unit;

[0133] The gradient calculation unit is used to sample data from the context memory and replay cache, calculate the gradient of the policy network, and provide directional information for the optimization of the policy network;

[0134] Angle evaluation unit, used to evaluate the angle between the previous task and the current task gradient, understand the similarities and conflicts between tasks, help the model identify potential conflicts between tasks, and thus adjust the learning strategy;

[0135] The gradient adjustment unit is used to adjust the gradient direction of the current task according to the angle to reduce the conflict between tasks, ensure that the model can take into account the influence of previous tasks when processing the current task, and improve the stability of the model;

[0136] The perturbation adding unit is used to add perturbations to the state of the current task and generate adversarial samples according to the adjusted gradient direction and perturbation size. By generating adversarial samples, the model's robustness to perturbations is improved and the generalization ability is enhanced.

[0137] The model update module includes: parameter update unit, learning rate adjustment unit, and performance evaluation unit;

[0138] A parameter updating unit for updating the parameters of the policy network and the critic network using adversarial samples;

[0139] The learning rate adjustment unit is used to dynamically adjust the learning rate according to the current learning state to optimize network performance, accelerate the training process and improve the convergence of the model;

[0140] The performance evaluation unit is used to regularly evaluate the performance of the reinforcement learning model and provide feedback to adjust the training strategy.

[0141] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in the field may also change, modify, replace and modify the above-mentioned embodiments without departing from the purpose and scope of protection of the present invention, and all of these are within the protection of the present invention.

Claims

1. A data processing method based on continuous reinforcement learning, characterized in that: include: Step S1: Initialize the situational memory module and the playback cache module. When completing the past task, store the conversion samples in the situational memory module. When processing the current task, collect the conversion data in the task in real time and store it in the playback cache module. Step S2: sampling conversion data from the episodic memory module and the playback cache module, using the sampled conversion data to calculate the policy network gradients of the previous task and the current task respectively, and calculating the gradient angle between the previous task and the current task based on the calculated policy network gradient results; Step S3: adjust the gradient direction of the current task according to the calculated gradient angle, add perturbations to the state of the current task, and generate adversarial samples according to the adjusted gradient direction and perturbation size; Step S4: Use the generated adversarial samples to update the policy network and critic network parameters, and dynamically adjust the learning rate according to the current learning state to optimize the policy network and critic network parameters; Step S5: Repeat steps S1 to S4 to perform iterative training. In each iteration, new conversion data is used to update the context memory module and the replay cache module. At the same time, the performance of the reinforcement learning model is regularly evaluated.

2. The data processing method based on continuous reinforcement learning according to claim 1, characterized in that: The conversion data in the episodic memory module and the replay cache module are stored in the form of a four-tuple of state-action-reward-subsequent state. The sampling data of the episodic memory module is in the form of The playback cache module sampling data is in the form of in, represents the state of the environment at the i-th moment in task k, Indicates in status The following actions are taken, Indicates taking action Back from state The reward for moving to the next state, Indicates taking action After transferring to the new state, represents the state of the environment at the i-th moment in the current task, Indicates in status The following actions are taken, Indicates taking action Back from state The reward for moving to the next state, Indicates taking action Then transfer to the new state.

3. The data processing method based on continuous reinforcement learning according to claim 2, characterized in that: The specific steps of step S2 include: S2.1: Sample the transition data of the previous task from the episodic memory module and sample the transition data of the current task from the replay buffer module; S2.2: The sampled conversion data and Arrange it into the input format of the neural network; S2.3: Use and As input, the policy network gradient g of the previous task is calculated through the policy network M and the policy network gradient g of the current task D , the formula is: in, represents the loss function of the policy network, π θ represents the policy function about the policy network parameter θ, S i represents the state of the previous task, S i ' represents the state of the current task, M represents the episodic memory library of the previous task, D represents the playback buffer library of the current task, and ▽ represents the gradient symbol; S2.4: Calculate the angle between the policy network gradients for the previous task and the current task Among them, cos -1 (·) represents the arc cosine function, and ||·|| represents the modulus of the vector.

4. The data processing method based on continuous reinforcement learning according to claim 3, characterized in that: The specific steps of step S3 include: S3.1: Get the gradient angle between the previous task and the current task And according to Determine the degree to which the current task gradient direction needs to be adjusted; S3.2: Add perturbation information ε to the state of the current task according to the degree to which the gradient direction of the current task needs to be adjusted; S3.3: Generate adversarial samples based on the adjusted gradient direction and the added perturbation size The formula is: Among them, ε represents the disturbance information, sign(·) represents the sign function, H(·) represents the step function, ⊙ represents the element-by-element product, Indicates g D The transpose of , T stands for transpose.

5. The data processing method based on continuous reinforcement learning according to claim 4, characterized in that: The specific steps of step S4 include: S4.1: Obtaining Generated Adversarial Examples For the policy network, use adversarial examples Calculate the strategy loss function Lπ, the formula is: L π = -∑logπ(a|s;θ π )×A(s,a), where, π(a|s; θ π ) means that in a given state s and parameter θ π The probability of taking action a under s is the output of the policy network, and A(s,a) represents the advantage function; For the critic network, use adversarial examples Calculate the temporal difference loss function L Q , the formula is: L Q =-∑(Q(s,a;θ Q )-y) 2 , where Q(s,a;θ Q ) means that given the state s and action a, and the parameter θ Q The expected future reward is the output of the critic network, θ Q represents the critical network parameters, and y represents the target of the temporal difference function.

6. The data processing method based on continuous reinforcement learning according to claim 5, characterized in that: The specific steps of step S4 also include: S4.2: According to the policy loss function L π And the temporal difference loss function L Q , calculate the gradient of the policy network and the gradient of the critical network parameters The formula is: Among them, E π [·] represents the expected value under the strategy π, R represents the reward function, and N represents the number of gradient samples; S4.3: Based on the calculated gradient and The parameters of the policy network and the critic network are updated through the optimization algorithm. The formula is: in, represents the updated policy network parameters, represents the updated critic network parameters, and β represents the learning rate; S4.4: Based on the current learning state, adjust the learning rate according to the change of loss value; S4.5: Repeat S4.1 to S4.5 until the preset number of training rounds is reached.

7. A data processing system based on continuous reinforcement learning, which is used to implement the data processing method based on continuous reinforcement learning according to any one of claims 1 to 6, characterized in that: include: Episodic memory module, replay cache module, adversarial sample generation module, model update module; The context memory module is used to store conversion samples generated when completing past tasks; The playback cache module is used to collect and store the conversion data in the current task in real time; The adversarial sample generation module is used to sample conversion data from the episodic memory module and the playback module, calculate the policy network gradient information of the previous task and the current task, use the projected gradient descent method and the step function to adjust the gradient direction of the current task according to the gradient angle, and add perturbations to the state of the current task, and generate adversarial samples according to the adjusted gradient direction and perturbation size; The model updating module is used to automatically apply the generated adversarial samples to update the policy network and the critic network, and dynamically adjust the learning rate according to the current learning state.

8. The data processing system based on continuous reinforcement learning according to claim 7, characterized in that: The adversarial sample generation module includes: a gradient calculation unit, an angle evaluation unit, a gradient adjustment unit, and a disturbance adding unit; The gradient calculation unit is used to sample data from the context memory and the replay buffer to calculate the gradient of the policy network; The angle evaluation unit is used to evaluate the angle between the gradient of the previous task and the current task, and determine the similarity and conflict between the tasks; The gradient adjustment unit is used to adjust the gradient direction of the current task according to the angle; The disturbance adding unit is used to add disturbance to the state of the current task and generate adversarial samples according to the adjusted gradient direction and disturbance size.

9. The data processing system based on continuous reinforcement learning according to claim 8, characterized in that: The model updating module includes: a parameter updating unit, a learning rate adjusting unit, and a performance evaluation unit; The parameter updating unit is used to update the parameters of the policy network and the critic network using adversarial samples; The learning rate adjustment unit is used to dynamically adjust the learning rate according to the current learning state; The performance evaluation unit is used to periodically evaluate the performance of the reinforcement learning model.

10. The data processing system based on continuous reinforcement learning according to claim 9, characterized in that: The context memory module, playback cache module, adversarial sample generation module, and model update module are seamlessly integrated with the continuous reinforcement learning framework.

Images