Calibration lifelong reinforcement learning method and system for dynamic action space

Abstract

The application relates to the technical field of artificial intelligence, and particularly discloses a calibration lifelong reinforcement learning method and system for a dynamic action space. The method comprises the following steps: when detecting that the action space changes, automatically archiving historical experience data to an offline buffer and retaining network parameters; acquiring training data through a mixed sampling strategy of online and offline buffers; introducing reference value based on a historical strategy as a calibration constraint in value function updating, constructing a lower limit of value to prevent systematic underestimation; and stably updating a policy network according to the calibrated value function. Through the value calibration and double-buffer cooperative mechanism, the application effectively suppresses performance shock in the learning process, realizes smooth and rapid adaptation of an intelligent agent in a dynamic environment, and significantly improves the stability and sample efficiency of lifelong learning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a calibration lifelong reinforcement learning method and system for dynamic action space. BACKGROUND

[0002] Reinforcement learning, as a core branch of artificial intelligence, has achieved remarkable success in games, robot control and other fields by trial and error and interaction in the environment. However, these successful applications are usually based on a key assumption that the environment in which the agent is located is static, especially the set of executable actions of the agent is fixed. This assumption greatly limits the application of traditional reinforcement learning in real-world scenarios, because in reality, the ability of the agent is often growing. For example, a household robot may acquire new skills through software updates, and a recommendation system needs to constantly handle newly listed goods. The action space in these scenarios is dynamically expanding, making it difficult for traditional models that rely on fixed output dimensions to be directly applied.

[0003] To cope with environmental dynamics, the lifelong reinforcement learning paradigm emerged, aiming to enable agents to continuously accumulate and transfer knowledge in a sequence of continuous tasks. However, when this paradigm faces the specific challenge of dynamic action space, existing methods have serious flaws. The mainstream method (such as LAICA based on action representation learning) generally suffers a catastrophic performance drop after the expansion of the action set. Research shows that the root cause lies in the introduction of new actions leading to systematic pessimistic underestimation of the value function, and the wrong value signal misleads the policy update direction, making the agent show the problem of forgetting old knowledge and sharply low learning efficiency.

[0004] In addition, another seemingly related technology, such as offline-to-online reinforcement learning methods (such as Cal-QL, CQL), although good at starting learning with a fixed data set, its design paradigm is single and non-continuous. They cannot cope with the continuous change of the action set, and each change requires interrupting learning, reconstructing the model and re-collecting data, fundamentally losing the continuity and knowledge accumulation ability required by lifelong learning. Therefore, existing technologies cannot guarantee the stability, efficiency and continuity of learning at the same time under dynamic action space, which constitutes a technical bottleneck that needs to be broken through in this field. SUMMARY

[0005] The technical problem to be solved by the present application is how to avoid the catastrophic decline in the performance of the agent when the action space of the agent is dynamically expanded, and to achieve efficient and stable continuous learning.

[0006] To solve the above technical problems, the technical scheme adopted by the present application is: a calibration lifelong reinforcement learning method for dynamic action space, comprising the following steps:

[0007] S1: the agent performs initial reinforcement learning training in an initial action space environment, and stores experience data generated by interaction into an online replay buffer;

[0008] S2: in response to detecting that the action space changes, transferring historical experience data in the online replay buffer to an offline replay buffer for archival storage, and retaining the current trained policy network and value network parameters;

[0009] S3: the agent interacts in a new environment containing new actions, and stores newly generated experience data into the online replay buffer;

[0010] S4: in the model updating phase, mixed sampling is performed from the online replay buffer and the offline replay buffer according to a preset ratio to form a mixed training data batch;

[0011] S5: based on the mixed training data batch, and using the reference value determined by the historical policy to constrain the current value function estimate, updating the value network;

[0012] S6: based on the calibrated value function, updating the policy network to adapt the agent to the new action space.

[0013] Further, in step S2, the historical experience data in the online replay buffer is transferred to the offline replay buffer for archival storage in response to detecting that the action space changes, and the current trained policy network and value network parameters are retained. Specifically, it includes:

[0014] monitoring the expansion event of the set of executable actions of the agent, and automatically triggering the transfer and archival operation of the historical experience data when a new action is identified to be added, while retaining the learning parameters of the policy network and the value network as initial parameters for subsequent learning.

[0015] Further, in step S4, the mixed sampling from the online replay buffer and the offline replay buffer according to a preset ratio to form a mixed training data batch specifically includes:

[0016] sampling a part of data from the offline replay buffer, and sampling another part of data from the online replay buffer, which together constitute the mixed training data batch, to ensure that the model can review historical knowledge while adapting to the new environment.

[0017] Further, in step S5, based on the mixed training data batch, and using the reference value determined by the historical policy to constrain the current value function estimate, updating the value network, a regularization term is introduced, and the mathematical expression of the calibration constraint is:

[0018] ;

[0019] wherein, is the learned value, Vμ is a reference value calculated from historical policy s , the calibration constraint forces the current value estimate to be no lower than the reference value. μ

[0020] Further, the calculation process of the reference value is:

[0021] Based on the historical experience data stored in the offline replay buffer, the historical policy μ is used to calculate the state value as a reliable lower bound reference of the value function estimate.

[0022] Further, in step S5, when updating the value network, the total loss function is composed of the time difference loss and the calibration constraint term.

[0023] Further, in step S6, updating the policy network based on the calibrated value function specifically includes:

[0024] Using the value function updated in step S5, the parameters of the policy network are updated by a policy gradient method or a deterministic policy optimization method to optimize the decision-making performance of the agent in the new action space.

[0025] Further, after step S2 and before step S3, there is also a step S2a:

[0026] Emptying the online replay buffer to store new experience data generated in the new action space.

[0027] Further, after step S6, there is also:

[0028] Continuously monitoring whether the action space changes again;

[0029] If a new change is detected, steps S2 to S6 are repeated to realize lifelong learning of the agent in the continuously and dynamically changing environment of the action space.

[0030] The present application also provides a calibration lifelong reinforcement learning system for dynamic action space, comprising:

[0031] A first environment interaction module is used for initial reinforcement learning training of the agent in the initial action space environment, and experience data generated by interaction is stored in the online replay buffer.

[0032] ​An action space change detection module is configured to, in response to detecting a change in the action space, transfer historical experience data in the online replay buffer to an offline replay buffer for archival storage, and retain the currently trained policy network and value network parameters;

[0033] A second environment interaction module is configured to cause the agent to interact in a new environment containing a new action, and store newly generated experience data to the online replay buffer;

[0034] A hybrid sampling module is configured to, in a model update phase, sample from the online replay buffer and the offline replay buffer in a preset ratio to form a hybrid training data batch;

[0035] A value calibration update module is configured to, based on the hybrid training data batch, and using a reference value determined by a historical policy, calibrate and constrain a current value function estimate, and update the value network;

[0036] A policy update module is configured to, based on the calibrated value function, update the policy network to adapt the agent to the new action space.

[0037] The present application has the beneficial effects that, by using a calibration constraint mechanism based on a historical policy reference value, the problem of systematic underestimation of the value function caused by the introduction of a new action is fundamentally solved, the agent maintains a stable learning curve after the expansion of the action space, and the performance is prevented from declining catastrophically; the hybrid sampling strategy of the online and offline double buffers guarantees the ability to quickly adapt to the new environment, significantly improves the sample utilization efficiency through historical data reuse, and effectively suppresses catastrophic forgetting; the entire technical chain forms a complete lifelong learning closed loop, enabling the agent to continuously accumulate knowledge in a changing environment, and providing a reinforcement learning solution with continuous evolution capability for real scenarios such as autonomous driving and flexible robots. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, brief descriptions will be given below of the drawings needed to be used in the embodiments or prior art descriptions. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the drawings shown.

[0039] Figure 1 A flowchart of the calibration lifelong reinforcement learning method for a dynamic action space according to an embodiment of the present application;

[0040] Figure 2 A technical effect comparison chart of the calibration regularization term according to an embodiment of the present application;

[0041] Figure 3 A calibration lifelong reinforcement learning system block diagram for dynamic action space oriented embodiments of the present invention. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by a person of ordinary skill in the art without creative work are within the protection scope of the present invention.

[0043] It should be noted that the descriptions involving "first", "second", etc. in the present invention are only for the purpose of description, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In addition, the technical solutions of various embodiments can be combined with each other, but it must be based on the fact that a person of ordinary skill in the art can realize it, and when the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist, nor within the protection scope required by the present invention.

[0044] Term explanation:

[0045] Lifelong Reinforcement Learning (LRL):

[0046] It refers to an advanced machine learning paradigm in which an agent does not learn in a single, isolated task, but continuously acquires, accumulates and transfers knowledge in a continuous sequence of tasks. Unlike traditional reinforcement learning, which aims to find the optimal strategy for a single static task, the core goal of lifelong reinforcement learning is to achieve continuous accumulation of knowledge, using the experience, representation or strategy learned from past tasks to significantly accelerate learning of new tasks (forward transfer), while avoiding forgetting the knowledge of old tasks in the learning process of new tasks (overcoming catastrophic forgetting). In the scenario in which the present invention is located, it specifically refers to the ability of an agent to continuously learn in an environment with a constantly changing action set.

[0047] Dynamic Action Space (DAS):

[0048] The set of actions A available to the agent is not static but changes over time or task phase k, denoted as A_k. This change often manifests as an increase (e.g., a robot gains a new arm functionality, a software agent discovers a new API interface) or a change in the set of actions. This is an important modeling of real-world complexity, as the architecture of traditional reinforcement learning algorithms (e.g., the output layer dimension of a policy network) is often strongly coupled with the number of actions, and any change in the action space renders the model inapplicable, thus interrupting the learning process. The invention aims to address the challenges posed by this dynamism.

[0049] Value Function Underestimation:

[0050] This is the core technical problem addressed by the invention. The value function (specifically the Q-function) is used to estimate the expected future cumulative reward that the agent can obtain by performing a certain action a in a given state s. Value function underestimation specifically refers to the systematic and significantly lower-than-actual true value estimation of certain actions (especially newly introduced or related to new actions) by the model after a change in the action space. This underestimation stems from the lack of sufficient interaction data for the model to accurately estimate the value of new actions or new state-action mappings resulting from action space restructuring, leading to conservative or pessimistic value estimates. The direct consequence is that the policy optimization algorithm will mistakenly avoid these underestimated but potentially optimal actions, leading to a catastrophic and sharp decline in agent performance.

[0051] Value Calibration:

[0052] The core innovative method proposed by the invention is a regularization correction technique aimed at the above-mentioned "value function underestimation" problem. The technical principle is to introduce a dynamic "lower value limit" or "value base" when the agent updates the value function. Specifically, it requires that the expected value learned under the current policy not be lower than a value determined by a more reliable reference policy (usually the policy learned by the agent in the previous stable phase). This calibration is achieved through a mathematical constraint, which acts as a safety net to prevent value estimates from falling to an unreasonable low level in the face of uncertainty, thereby ensuring the stability and correctness of policy updates, enabling the agent to smoothly adapt to environmental changes.

[0053] Online-Offline Replay Buffer:

[0054] The key data management mechanism in the present application for improving sample efficiency and learning stability. It solves the problem of data distribution drift caused by environmental changes in lifelong learning. The mechanism contains two logically separate buffers:

[0055] 1. Offline Buffer: used to store and manage all past task (i.e. under the old action space) collected historical experience data. These data contain valuable knowledge about the general dynamics and reward structure of the environment;

[0056] 2. Online Buffer: used to store the latest experience data generated by real-time interaction under the current task (i.e. under the new action space). These data best reflect the characteristics of the current environment.

[0057] During model training, the system will sample from these two buffers simultaneously according to the preset mixing ratio. This strategy enables the agent to constantly "review" old knowledge while learning to adapt to new tasks, thereby balancing adaptability (plasticity) to new environments and retention of old knowledge (stability).

[0058] As shown in Figure 1 , an embodiment of the present application is: a calibration lifelong reinforcement learning method for dynamic action space, comprising the following steps:

[0059] S1: the agent performs initial reinforcement learning training in the initial action space environment, and stores the experience data generated by interaction into the online replay buffer;

[0060] In this embodiment, the initial reinforcement learning training can be completed using mainstream deep reinforcement learning algorithms such as Proximal Policy Optimization (PPO), Deep Deterministic Policy Gradient, etc. The agent interacts with the environment under the initial action set (for example, a robot has only four basic movement actions of "forward", "backward", "left turn" and "right turn"), and the state-action-reward-next state tuple data collected is stored into the online replay buffer in real time. The goal of this stage is to let the agent learn the basic task policy under the initial action constraint, for example, to navigate from the starting point to the target point, while accumulating the initial experience data pool for subsequent lifelong learning.

[0061] S2: in response to detecting a change in the action space, transferring the historical experience data in the online replay buffer to the offline replay buffer for archival storage, and retaining the current trained policy network and value network parameters;

[0062] Further, step S2 specifically includes:

[0063] The monitoring agent monitors the expansion event of the set of executable actions of the agent, and when a new action is identified, automatically triggers the transfer and archiving operation of the historical experience data, while completely retaining the learning parameters of the policy network and the value network as the initial parameters for subsequent learning.

[0064] In this embodiment, the detection of the change in the action space can be triggered by a signal actively sent by the environment, or can be realized by self-recognition of the agent. Once the change is detected, for example, the robot obtains the new action of grasping through software update, the system will immediately start the data migration protocol. The protocol safely transfers all experience data collected under the old action space in the online buffer to the offline buffer for long-term archiving. It is crucial that the weight parameters of the policy network and the value network are completely retained instead of being randomly initialized. This approach ensures that the agent's learning in the new phase does not start from zero, but builds on existing knowledge, using the general knowledge learned about state representation, environmental dynamics, and reward signals as priori, providing a stable starting point for the rapid integration of new actions and effectively avoiding knowledge gaps.

[0065] S2a: Empty the online replay buffer to store new experience data generated under the new action space.

[0066] In this embodiment, emptying the online buffer is a key data management operation, which aims to provide a clean storage space for the new round of interaction data, prevent the mixing of experience data under the old and new action spaces, ensure the data distribution consistency of the current learning phase, and thus guarantee the timeliness and accuracy of subsequent data sampling from the online buffer.

[0067] S3: The agent interacts in the new environment containing the new action and stores the newly generated experience data in the online replay buffer;

[0068] In this embodiment, the agent begins to explore and utilize in the expanded action space (e.g., basic movement actions plus grasping action). Initially, due to the uncertainty of the value and consequences of the new action, the agent may take random attempts or use the new action based on some exploration strategy (such as entropy regularization). All interaction data generated in the new environment, including successful use of the new action to obtain high rewards and failed experiences, are recorded in real time to the online replay buffer. These data most directly reflect the characteristics of the new action space and are the core basis for the model to adapt to the new environment.

[0069] S4: In the model update phase, sample from the online replay buffer and the offline replay buffer at a pre-set ratio to form a mixed training data batch;

[0070] Further, step S4 specifically includes:

[0071] A part of data is sampled from the offline replay buffer and another part of data is sampled from the online replay buffer to form the mixed training data batch together, so as to ensure that the model can review historical knowledge while adapting to a new environment.

[0072] In this embodiment, the mixed sampling strategy is the core of balancing stability and plasticity. For example, the initial mixing ratio of offline data and online data can be set to 7:3, and the ratio of online data can be gradually increased as the training proceeds. Sampling from the offline buffer is equivalent to allowing the agent to review the successful experience in the old task regularly, which helps to consolidate existing knowledge and prevent catastrophic forgetting caused by learning new actions; sampling from the online buffer ensures that the model can quickly absorb and adapt to the latest information of the new environment. This cooperative sampling mechanism enables the update of the value function and the policy to simultaneously consider the reliability of historical experience and the real-time nature of new environment dynamics, significantly improving the sample efficiency and robustness of overall learning.

[0073] S5: updating the value network based on the mixed training data batch and using the reference value determined by the historical policy to calibrate and constrain the current value function estimate;

[0074] Further, in step S5, when updating the value network based on the mixed training data batch and using the reference value determined by the historical policy to calibrate and constrain the current value function estimate, a regularization term is introduced, and the mathematical expression of the calibration constraint is:

[0075] ;

[0076] wherein, is the learned value, Vμ is the reference value calculated from the historical policy s . μ The calibration constraint forces the current value estimate to be no lower than the reference value.

[0077] Further, the calculation process of the reference value is as follows:

[0078] Based on the historical experience data stored in the offline replay buffer, the state value is calculated using the historical policy μ as a reliable lower limit benchmark for the value function estimate.

[0079] Further, in step S5, when updating the value network, the total loss function is composed of the temporal difference loss and the calibration constraint term.

[0080] In this embodiment, the calibration operation forces the current value estimate to be no lower than the historical reference value, which is equivalent to setting a dynamic and reliable safety lower limit or value base for value learning. This scheme can actively prevent the model from systematically and pessimistically underestimating the value of all actions (including old and new actions) due to insufficient new action data or high uncertainty. Through this calibration, the update of the value function is guided towards a more optimistic and accurate direction, avoiding the strategy network from avoiding new actions that should be optimal or mastered old actions due to receiving incorrect value signals, thereby fundamentally suppressing the catastrophic decline in performance and making the learning transition process very smooth.

[0081] As shown in Figure 2 , the performance before and after the introduction of the calibration mechanism is compared. In the traditional method without calibration, the performance curve shows a cliff-like drop after the expansion point of the action space; while using the method of the present application, the performance only shows slight and short-term fluctuations, and can quickly recover and surpass the original level, verifying the significant effect of value calibration on maintaining learning stability.

[0082] S6: updating the policy network based on the calibrated value function to adapt the agent to the new action space.

[0083] Further, in step S6, the updating of the policy network based on the calibrated value function specifically includes:

[0084] Using the updated value function in step S5, the parameters of the policy network are updated by a policy gradient method or a deterministic policy optimization method to optimize the decision-making performance of the agent in the new action space.

[0085] In this embodiment, the update of the policy network depends on the calibrated value function. For example, the parameters of the policy are updated by a policy gradient method, and the direction of the gradient is guided by the calibrated value function. Since the value estimate has been calibrated, the pessimistic underestimation is avoided, and the gradient signal received by the policy network is accurate and encourages exploration. This enables the policy to confidently try and adopt new actions identified by the calibration mechanism as having potential, while robustly maintaining effective behavior in the old action space. Ultimately, the policy network can efficiently integrate new and old knowledge to output a decision-making policy with excellent performance in the expanded action space, achieving rapid and stable adaptation.

[0086] Further, after step S6, it further includes:

[0087] continuously monitoring whether the action space changes again;

[0088] If a new change is detected, steps S2 to S6 are repeated to implement lifelong learning of the agent in the environment with continuously and dynamically changing action space.

[0089] In this embodiment, the entire process constitutes a complete autonomous cycle. For example, after mastering movement and grasping, the robot can upgrade again to obtain the placing action. The system will detect this change again and automatically restart the process of data archiving, parameter reservation, hybrid sampling and value calibration. Through this cycle mechanism, the agent can cope with the expansion of the action space one after another in its entire life cycle, and seamlessly integrate new skills into its existing knowledge system, achieving true lifelong learning and continuous evolution. This has crucial value for AI systems that need to be deployed for a long time and adapt to complex changing environments, such as long-running autonomous robots and continuously evolving game AI.

[0090] In summary, the technical scheme of the present application can achieve the following remarkable technical effects:

[0091] Thanks to the value calibration mechanism, the performance of the agent only shows a small temporary decline when facing new available actions, and can quickly recover or even surpass the original level. This mechanism acts as a stabilizer, ensuring the smoothness of the learning process and avoiding the dramatic shock of the strategy and the sudden collapse of the performance.

[0092] Through the innovative online-offline hybrid replay mechanism, the present scheme can make full use of all the historical interaction data collected in the life cycle of the agent. These valuable historical data work together with newly collected data to significantly reduce the number of interaction samples required for the agent to achieve ideal performance in a new environment, greatly improving data utilization.

[0093] The present application provides a complete end-to-end solution, successfully building a lifelong reinforcement learning system that can truly face dynamic action space for reliable and continuous learning, filling a key technical gap in this field.

[0094] As shown in Figure 3 The embodiment of the present application also provides a calibration lifelong reinforcement learning system for dynamic action space, comprising:

[0095] The first environment interaction module 10 is used for initial reinforcement learning training of the agent in the initial action space environment, and stores the experience data generated by the interaction into the online replay buffer;

[0096] The action space change detection module 20 is used for transferring the historical experience data in the online replay buffer to the offline replay buffer for archiving storage in response to detecting a change in the action space, and reserving the current trained strategy network and value network parameters;

[0097] The second environment interaction module 30 is configured to make the agent interact in a new environment containing a new action, and store new generated experience data into the online replay buffer.

[0098] The mixed sampling module 40 is configured to, in the model updating stage, sample data from the online replay buffer and the offline replay buffer in a preset proportion to form a mixed training data batch.

[0099] The value calibration updating module 50 is configured to calibrate and constrain the current value function estimation based on the mixed training data batch and the reference value determined by the historical policy, and update the value network.

[0100] The policy updating module 60 is configured to update the policy network based on the calibrated value function, so that the agent adapts to the new action space.

[0101] Further, in the action space change detection module 20, the historical experience data in the online replay buffer is transferred to the offline replay buffer for archiving storage in response to detecting that the action space changes, and the current trained policy network and value network parameters are specifically retained.

[0102] The expansion event of the set of executable actions of the agent is monitored, and when a new action is identified to be added, the transfer and archiving operation of the historical experience data is automatically triggered, and the learning parameters of the policy network and the value network are completely retained as initial parameters for subsequent learning.

[0103] Further, in the mixed sampling module 40, the sampling of data from the online replay buffer and the offline replay buffer in a preset proportion to form a mixed training data batch specifically includes:

[0104] A part of data is sampled from the offline replay buffer, and another part of data is sampled from the online replay buffer, which together constitute the mixed training data batch, so as to ensure that the model can review historical knowledge while adapting to the new environment.

[0105] Further, in the value calibration updating module 50, when the value network is updated based on the mixed training data batch and the reference value determined by the historical policy is used to calibrate and constrain the current value function estimation, a regularization term is introduced, and the mathematical expression of the calibration constraint is:

[0106] ;

[0107] Wherein, is the learned value, Vμ is the reference value determined by the historical policy. s μ ​The calculated reference value, and the calibration constraint forces the current value estimate to be no less than the reference value.

[0108] Further, the reference value The calculation process is as follows:

[0109] Based on the historical experience data stored in the offline replay buffer, the historical policy μ The state value is calculated as a reliable lower limit benchmark of the value function estimate.

[0110] Further, in the value calibration update module 50, when updating the value network, the total loss function is composed of the time difference loss and the calibration constraint term.

[0111] Further, in the policy update module 60, updating the policy network based on the calibrated value function specifically includes:

[0112] Using the updated value function, the parameters of the policy network are updated by a policy gradient method or a deterministic policy optimization method to optimize the decision performance of the agent in the new action space.

[0113] Further, after the action space change detection module 20 and before the second environment interaction module 30, there is also a buffer emptying module for:

[0114] Emptying the online replay buffer to store new experience data generated in the new action space.

[0115] Further, after the policy update module 60, there is also an action space continuous monitoring module, specifically for:

[0116] Continuously monitoring whether the action space has changed again;

[0117] If a new change is detected, the operations between the action space change detection module 20 and the policy update module 60 are repeated to realize lifelong learning of the agent in the continuously and dynamically changing action space environment.

[0118] The above only describes the embodiments of the present application, and does not limit the patent scope of the present application, any equivalent structure or equivalent process transformation using the content of the present application specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A calibration lifetime reinforcement learning method for dynamic action spaces, applied to robot control systems, characterized in that, Includes the following steps: S1: Control the robot to perform initial reinforcement learning training in an initial action space environment. The initial action space includes a set of basic actions that the robot can execute. The state, actions, rewards and experience data of the next state generated during the training process are stored in the online replay buffer. S2: In response to the detection of an expansion event in the robot's action space, the historical experience data related to the basic action stored in the online replay buffer is transferred to the offline replay buffer for archiving and storage, while retaining the parameters of the currently trained policy network and value network. S3: Control the robot to interact in an extended motion space environment that includes at least one new action, wherein the new action is a robot action added through software update, and the new experience data generated during the interaction is stored in the online replay buffer; S4: During the model update phase, samples are mixed from the online replay buffer and the offline replay buffer at a preset ratio to form a batch of mixed training data. S5: Based on the batch of mixed training data, and using the reference value determined by the historical strategy to calibrate the current value function estimate, update the value network; In step S5, based on the mixed training data batch and using the reference value determined by the historical strategy to calibrate the current value function estimate, a regularization term is introduced when updating the value network. The mathematical expression of the calibration constraint is: ； in, For the value of what is learned, Vμ ( s (Based on historical strategies) μ The calculated reference value, the calibration constraint forces the current value estimate to be no less than the reference value; The reference value The calculation process is as follows: Based on the historical experience data stored in the offline playback buffer, the historical strategy is utilized. μ The state value is calculated and used as a reliable lower bound benchmark for the estimation of the value function; S6: Using the updated value function, the parameters of the policy network are updated through the policy gradient method or the deterministic policy optimization method to optimize the robot's decision control strategy in the extended action space.

2. The calibration lifetime reinforcement learning method for dynamic action space according to claim 1, characterized in that, In step S2, the step of responding to the detection of an expansion event in the robot's motion space by transferring the historical experience data related to the basic motion stored in the online replay buffer to the offline replay buffer for archiving and storage, while retaining the parameters of the currently trained policy network and value network, specifically includes: The system monitors the robot for new executable actions added through software upgrades. When a new action is detected, it automatically triggers the transfer of historical experience data related to the basic action stored in the online replay buffer to the offline replay buffer for archiving and storage. At the same time, it fully preserves the learning parameters of the policy network and value network as initial parameters for subsequent learning.

3. The calibration lifetime reinforcement learning method for dynamic action space according to claim 1, characterized in that, In step S4, the step of mixing and sampling from the online playback buffer and the offline playback buffer according to a preset ratio to form a mixed training data batch specifically includes: A portion of data is sampled from the offline replay buffer, and another portion of data is sampled from the online replay buffer to form the mixed training data batch, so as to ensure that the model can review historical knowledge while adapting to the new environment.

4. The calibration lifetime reinforcement learning method for dynamic action space according to claim 1, characterized in that, In step S5, when updating the value network, the total loss function is composed of the temporal difference loss and the calibration constraint term.

5. The calibration lifetime reinforcement learning method for dynamic action space according to claim 1, characterized in that, After step S2 and before step S3, step S2a is also included: Clear the online replay buffer to store new experience data generated in the new action space.

6. The calibration lifetime reinforcement learning method for dynamic action space according to claim 1, characterized in that, Following step S6, the following is also included: Continuously monitor whether the robot's motion space changes again; If a new change is detected, steps S2 to S6 are repeated to achieve lifelong learning and continuous optimization of the control strategy for the robot in a continuously dynamic environment of motion space.

7. A calibration lifetime reinforcement learning system for dynamic action spaces, applied to robot control, characterized in that, include: The first environment interaction module is used to control the robot to perform initial reinforcement learning training in an initial action space environment. The initial action space includes a set of basic actions that the robot can execute. The state, actions, rewards and experience data of the next state generated during the training process are stored in the online replay buffer. The motion space change detection module is used to, in response to the detection of an expansion event in the robot's motion space, transfer the historical experience data related to the basic motion stored in the online replay buffer to the offline replay buffer for archiving and storage, and retain the parameters of the currently trained policy network and value network. The second environment interaction module is used to control the robot to interact in an extended action space environment containing at least one new action. The new action is a robot action added through software update. New experience data generated during the interaction is stored in the online playback buffer. The hybrid sampling module is used to sample from the online replay buffer and the offline replay buffer at a preset ratio during the model update phase to form a hybrid training data batch. The value calibration update module is used to update the value network by calibrating the current value function estimate based on the mixed training data batch and using the reference value determined by the historical strategy. In the value calibration update module, based on the mixed training data batch and using the reference value determined by the historical strategy, a calibration constraint is applied to the current value function estimate. When updating the value network, a regularization term is introduced, and the mathematical expression of the calibration constraint is: ； in, For the value of what is learned, Vμ ( s (Based on historical strategies) μ The calculated reference value, the calibration constraint forces the current value estimate to be no less than the reference value; The reference value The calculation process is as follows: Based on the historical experience data stored in the offline playback buffer, the historical strategy is utilized. μ The state value is calculated and used as a reliable lower bound benchmark for the estimation of the value function; The strategy update module is used to update the parameters of the strategy network using the updated value function, through the strategy gradient method or the deterministic strategy optimization method, so as to optimize the decision control strategy of the robot in the extended action space.

Images