Collaborative knowledge fusion reinforcement learning method for sparse reward environment

Abstract

The invention discloses a sparse reward environment-oriented collaborative knowledge fusion reinforcement learning method, and relates to the field of collaborative knowledge fusion reinforcement learning methods. By constructing a lightweight collaborative knowledge fusion model and a dynamic reward remodeling mechanism, the problems of low intelligent agent exploration efficiency and difficulty in strategy convergence in a sparse reward environment are solved. The method comprises the following steps: constructing a reinforcement learning framework comprising a policy network and a value network; designing an action space mutation supervision mechanism and a lightweight collaborative knowledge fusion model, and generating a smooth substitution action when a strategy is detected to be unstable; and a reward function is designed in combination with the task target and the dynamic constraint, and reward remodeling is realized by activating rewards through sub-target potential energy difference and knowledge fusion. According to the method, effective intermediate feedback can be provided for the agents in the sparse reward environment, the exploration efficiency is improved, the convergence time is shortened, the stability and cross-scene migration ability of the strategy are enhanced, and an effective solution is provided for sparse reward scenes such as robot control and multi-agent game.

Description

Technical Field

[0001] The present invention proposes a collaborative knowledge fusion reinforcement learning method for a sparse reward environment, and relates to the field of collaborative knowledge fusion reinforcement learning methods. Background Art

[0002] In reinforcement learning, a sparse reward environment refers to a training scenario in which an agent receives reward feedback only when achieving a specific goal, lacking effective guidance signals during the intermediate exploration process. This problem is widely present in complex scenarios such as robotic control and multi-agent games. In robotic grasping tasks, agents receive positive rewards only when they successfully grasp an object, with no feedback for posture adjustments during movement, making it difficult for the agent to find an effective strategy through random exploration. In financial derivatives hedging scenarios, agents only receive profit and loss settlement signals at option expiration, unable to learn risk control from intermediate trading behaviors, resulting in difficulty in policy convergence. The core challenges of sparse reward environments lie in inefficient exploration, delayed reward distribution, and training instability caused by high variance. Agents may become trapped in local optima due to long periods of lack of feedback, requiring extensive behavioral interactions to converge. The competitive nature of multi-agent collaborative scenarios further exacerbates this problem. Because individual rewards are affected by the behavior of their opponents, the policy coordination mechanism under sparse rewards is even more complex. Therefore, a new algorithm is needed to coordinate knowledge and online interaction data, reduce the original sparse reward dependency through a dynamic reward reshaping mechanism, and deal with the dilemma of ineffective exploration of high-dimensional state space in reinforcement learning. Summary of the Invention

[0003] To address these issues, this paper proposes a collaborative knowledge fusion reinforcement learning method for sparse reward environments. This collaborative knowledge fusion model addresses the problems of existing reinforcement learning techniques trapped in local optimal solutions and inefficient exploration. The core of this paper is to establish a collaborative knowledge fusion model. The design goal is to establish a directed exploration mechanism guided by collaborative knowledge, improve the coverage of the effective state space, and ensure convergence to the global optimal strategy through dynamic reward compensation and maintaining strategy diversity.

[0004] According to the present invention, a collaborative knowledge fusion reinforcement learning method for a sparse reward environment includes the following specific related steps:

[0005] Step S1: Build a reinforcement learning framework including a policy network and a value network, generate action probability distribution and learn the optimal strategy during training;

[0006] Step S2: Construct a lightweight collaborative knowledge fusion model to replace the action output vector of the policy network when the model triggering conditions are met;

[0007] Step S3: Design a reward function that includes task objectives and dynamic constraints, reshape the reward after the lightweight collaborative knowledge fusion model is triggered, and feed the reshaped reward back to the parameter update process of the policy network and value network.

[0008] Step S1 specifically includes the following contents:

[0009] Step S11: Build the A2C network, introduce the advantage function to optimize the update of the policy gradient, reduce the variance and improve the stability of the training, and the value network of A2C The parameters are optimized by minimizing the temporal difference objective function, which can be expressed as:

[0010]

[0011] in, is the starting time step, is the value network parameter; is the discount factor; yes Rewards of the moment; It is the intelligent agent in State passing strategy of The state reached after the step, ; is the single-step timing difference step size, ;

[0012] Step S12: The policy network passes the generalized advantage function Calculate the policy gradient and determine each state The probability distribution of taking an action under Expressed as:

[0013]

[0014] in is the current state, is the current action, the timing difference error Expressed as:

[0015]

[0016] The total reward is , the policy gradient update formula is:

[0017]

[0018] in are policy network parameters, is the training segment length, is the bias-variance trade-off coefficient, is the policy entropy, It is the entropy regularization weight, which increases the randomness of the strategy and prevents local optimality.

[0019] Step S2 specifically includes the following contents:

[0020] Step S21: Detect the mutation value of the real-time multi-dimensional action space, quantify the mutation degree of the action in adjacent time steps through differential calculation, and define the action change rate for:

[0021]

[0022] in for The output of the moment strategy network dimensional action value, the action space interval is set at Avoid frequent shaking movements. Designing independent thresholds in multi-dimensional action space for the environment interaction time step , set the intervention flag , when the rate of change of any dimension exceeds the threshold, the intervention is activated ,otherwise :

[0023]

[0024] in is the action space dimension index, , a total of dimensional continuous action space; For the The allowed change rate threshold of the dimensional action is calibrated according to the specific action space; the historical action is cached in the buffer , in 2 Complete supervisory decision-making within the company;

[0025] Step S22: Design a collaborative knowledge strategy model to generate smooth alternative actions , the model expression is:

[0026]

[0027] in is a trainable weight matrix, The dimension of the action space is , the dimension of the state space is ; is a trainable bias vector, in the state mapping Up-calibrate the action offset so that the model can output non-zero action when zero state input; is the environmental state vector; is the inertia factor, and the parameter optimization target should be consistent with the original strategy:

[0028]

[0029] In the formula is the trade-off coefficient, , Store data for the experience pool. When using the current state and the original output of the policy network constitute the training samples; when Use exponential decay memory Keep the model adaptable.

[0030] Step S3 specifically includes the following contents:

[0031] Step S31: Constructing the total reward of reinforcement learning , including mission target rewards and dynamic constraint penalty :

[0032]

[0033] in is the number of mission objectives, ; For the The weight of the task, ; Rewards for task completion targets, is the number of constraints, is the penalty coefficient of the dynamic constraint term, ;

[0034] Step S32: The task objective reward function is designed as follows:

[0035]

[0036] in For the specific implementation of the target task reward value, For the current moment State and goal constraints The difference, is the threshold parameter;

[0037] The penalty design of dynamic constraint term is as follows:

[0038]

[0039] in For the The actual value of the constraint, , is the safety threshold;

[0040] Step S33: When the lightweight collaborative knowledge fusion model is activated, reward sparsity compensation is performed, and the knowledge fusion reward is superimposed on the basic reward. :

[0041]

[0042] in is the sub-goal potential energy difference, is a constant coefficient, :

[0043]

[0044] Is the sub-goal potential function, measuring the current state The state closest to the sub-goal Proximity:

[0045]

[0046] Activate rewards for knowledge fusion, where Status Number of times it was visited during training:

[0047]

[0048] is the state-dependent collaborative knowledge strategy alignment, is the feature difference vector between the collaborative knowledge state and the reinforcement learning state in the current state, and each component It is the collaborative knowledge and reinforcement learning in the Difference values ​​on dimension features:

[0049]

[0050] is the adaptive weight, :

[0051]

[0052] for The dynamic adjustment coefficient is in the range of ;

[0053] Step S34: Collaborative knowledge fusion has prior knowledge and reasoning ability. In the experience replay pool, it is necessary to weight the collaborative knowledge to construct the priority and define the priority function To evaluate the collaborative knowledge strategy distribution:

[0054]

[0055] in is the TD error; is a sufficiently small positive real number that makes the priority function probability greater than 0; the last term is the advantage term of the value function, To coordinate the value of knowledge action, It is the reinforcement learning action value, which is used to evaluate the degree to which the collaborative knowledge fusion in the experience pool is retrieved without considering the TD error. During the reinforcement learning training process, the priority of the collaborative knowledge will change with the performance of the agent. The probability density function is expressed as follows:

[0056]

[0057] in is the experience pool data index, is a hyperparameter, , A value close to 1 represents a high-value sample. Approaching 0 means giving up collaborative knowledge guidance;

[0058] Step S35: Since the priority mechanism changes the experience distribution in the buffer, the estimation of the expected value of the value function introduces a bias, which will affect the stability and robustness of the strategy. The bias can be reduced by introducing the importance sampling weight into the loss function of the value network. The importance sampling weight The calculation is as follows:

[0059]

[0060] in, is a dynamic coefficient, Completely non-uniform sampling occurs when Completely uniform sampling occurs when It will gradually decrease to 0 during training, adding the importance sampling weight to the value network loss function middle:

[0061]

[0062] in are network parameters, The sampling probability of the behavior strategy is updated through the loss value. The policy network updates the output strategy according to the updated value network. When the loss converges, Output the final strategy , end the training.

[0063] The beneficial effects of the present invention are:

[0064] 1. This invention addresses the difficulty of intelligent agents obtaining effective feedback in sparse reward environments. By constructing a lightweight collaborative knowledge fusion model, it dynamically replaces the output of the policy network when sudden action changes or policy instability are detected. Combined with a reward reshaping mechanism, this approach provides the intelligent agent with denser intermediate feedback signals. By designing sub-goal potential and knowledge fusion-activated rewards, the intelligent agent can quickly locate effective exploration directions in environments with minimal reward signals, avoiding being trapped in local optima or ineffective exploration due to long periods of no rewards. This method effectively reduces the time required for convergence.

[0065] 2. To address the core challenge of inefficient agent exploration in sparse reward environments, this paper constructs a multi-level reward compensation system through dynamic reward reshaping and smooth substitution of collaborative knowledge and policy alignment. The knowledge compensation reward function dynamically attenuates the collaborative knowledge weight through compliance scoring, providing additional guidance at key decision points while maintaining the original sparse reward framework. Prioritized experience replay retains samples containing key reward signals, enabling the agent to learn efficiently even with limited positive reward samples.

[0066] 3. This invention addresses the difficulty of policy transfer in sparse reward tasks by using state-dependent collaborative knowledge alignment and adaptive weights to enable the agent to automatically adjust the strength of knowledge fusion based on the current reward sparsity. During the initial training phase, when rewards are extremely sparse, the system prioritizes collaborative knowledge guidance. As valid reward samples accumulate, it gradually transitions to a policy optimization mode based primarily on environmental feedback. This adaptive mechanism not only avoids the policy degradation of traditional methods under sparse rewards but also significantly improves cross-scenario transfer capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 Flowchart of the reinforcement learning framework for collaborative knowledge models;

[0068] Figure 2 Flowchart of the training process for an agent in a sparse reward environment. DETAILED DESCRIPTION

[0069] In order to further understand the content of the present invention, the present invention is described in detail with reference to the accompanying drawings and embodiments. It should be understood that the embodiments are merely for explaining the present invention and are not intended to limit the present invention.

[0070] The specific implementation steps are as follows:

[0071] Step S1: Build a reinforcement learning framework including a policy network and a value network, generate action probability distribution and learn the optimal strategy during training;

[0072] Step S2: Construct a lightweight collaborative knowledge fusion model to replace the action output vector of the policy network when the model triggering conditions are met;

[0073] Step S3: Design a reward function that includes task objectives and dynamic constraints, reshape the reward after the lightweight collaborative knowledge fusion model is triggered, and feed the reshaped reward back to the parameter update process of the policy network and value network.

[0074] In step S1, Figure 1 As shown in step S1, in order to build a reinforcement learning framework including a policy network and a value network, the following steps need to be performed:

[0075] Step S11: Build the A2C network, introduce the advantage function to optimize the update of the policy gradient, reduce the variance and improve the stability of the training, and the value network of A2C The parameters are optimized by minimizing the temporal difference objective function, which can be expressed as:

[0076]

[0077] in, is the starting time step, is the value network parameter; is the discount factor; yes Rewards of the moment; It is the intelligent agent in State passing strategy of The state reached after the step, ; is the single-step timing difference step size, ;

[0078] Step S12: The policy network passes the generalized advantage function Calculate the policy gradient and determine each state The probability distribution of taking an action under Expressed as:

[0079]

[0080] in is the current state, is the current action, the timing difference error Expressed as:

[0081]

[0082] The total reward is , the policy gradient update formula is:

[0083]

[0084] in are policy network parameters, is the training segment length, is the bias-variance trade-off coefficient, is the policy entropy, It is the entropy regularization weight, which increases the randomness of the strategy and prevents local optimality.

[0085] In step S2, if Figure 1 As shown in step S2, a lightweight collaborative knowledge fusion model is constructed to replace the action output vector of the policy network when the model triggering condition is met. Specifically, the following steps are included:

[0086] Step S21: Detect the mutation value of the real-time multi-dimensional action space, quantify the mutation degree of the action in adjacent time steps through differential calculation, and define the action change rate for:

[0087]

[0088] in for The output of the moment strategy network dimensional action value, the action space interval is set at Avoid frequent shaking movements. Designing independent thresholds in multi-dimensional action space for the environment interaction time step , set the intervention flag , when the rate of change of any dimension exceeds the threshold, the intervention is activated ,otherwise :

[0089]

[0090] in is the action space dimension index, , a total of dimensional continuous action space; For the The allowed change rate threshold of the dimensional action is calibrated according to the specific action space; the historical action is cached in the buffer , in 2 Complete supervisory decision-making within the company;

[0091] Step S22: Figure 2 As shown, the collaborative knowledge strategy model is designed to generate smooth alternative actions , the model expression is:

[0092]

[0093] in is a trainable weight matrix, The dimension of the action space is , the dimension of the state space is ; is a trainable bias vector, in the state mapping Up-calibrate the action offset so that the model can output non-zero action when zero state input; is the environmental state vector; is the inertia factor, and the parameter optimization target should be consistent with the original strategy:

[0094]

[0095] In the formula is the trade-off coefficient, , Store data for the experience pool. When using the current state and the original output of the policy network constitute the training samples; when Use exponential decay memory Keep the model adaptable.

[0096] In step S3, Figure 1 As shown in step S3, a reward function including task objectives and dynamic constraints is designed. After the lightweight collaborative knowledge fusion model is triggered, the reward is reshaped and the reshaped reward is fed back to the parameter update process of the policy network and the value network. Specifically, the steps are as follows:

[0097] Step S31: Constructing the total reward of reinforcement learning , including mission target rewards and dynamic constraint penalty :

[0098]

[0099] in is the number of mission objectives, ; For the The weight of the task, ; Rewards for task completion targets, is the number of constraints, is the penalty coefficient of the dynamic constraint term, ;

[0100] Step S32: The task objective reward function is designed as follows:

[0101]

[0102] in For the specific implementation of the target task reward value, For the current moment State and goal constraints The difference, is the threshold parameter;

[0103] The penalty design of dynamic constraint term is as follows:

[0104]

[0105] in For the The actual value of the constraint, , is the safety threshold;

[0106] Step S33: Figure 2 As shown in the figure, when the lightweight collaborative knowledge fusion model is activated, the reward sparsity compensation is performed, and the knowledge fusion reward is superimposed on the basic reward. :

[0107]

[0108] in is the sub-goal potential energy difference, is a constant coefficient, :

[0109]

[0110] Is the sub-goal potential function, measuring the current state The state closest to the sub-goal Proximity:

[0111]

[0112] Activate rewards for knowledge fusion, where Status Number of times it was visited during training:

[0113]

[0114] is the state-dependent collaborative knowledge strategy alignment, is the feature difference vector between the collaborative knowledge state and the reinforcement learning state in the current state, and each component It is the collaborative knowledge and reinforcement learning in the Difference values ​​on dimension features:

[0115]

[0116] is the adaptive weight, :

[0117]

[0118] for The dynamic adjustment coefficient is in the range of ;

[0119] Step S34: Collaborative knowledge fusion has prior knowledge and reasoning ability. In the experience replay pool, it is necessary to weight the collaborative knowledge to construct the priority and define the priority function To evaluate the collaborative knowledge strategy distribution:

[0120]

[0121] in is the TD error; is a sufficiently small positive real number that makes the priority function probability greater than 0; the last term is the advantage term of the value function, To coordinate the value of knowledge action, It is the reinforcement learning action value, which is used to evaluate the degree to which the collaborative knowledge fusion in the experience pool is retrieved without considering the TD error. During the reinforcement learning training process, the priority of the collaborative knowledge will change with the performance of the agent. The probability density function is expressed as follows:

[0122]

[0123] in is the experience pool data index, is a hyperparameter, , A value close to 1 represents a high-value sample. Approaching 0 means giving up collaborative knowledge guidance;

[0124] Step S35: Figure 2 As shown, since the priority mechanism changes the experience distribution in the buffer, the estimation of the expected value of the value function introduces a bias, which will affect the stability and robustness of the strategy. The bias can be reduced by introducing the importance sampling weight into the loss function of the value network. The importance sampling weight The calculation is as follows:

[0125]

[0126] in, is a dynamic coefficient, Completely non-uniform sampling occurs when Completely uniform sampling occurs when It will gradually decrease to 0 during training, adding the importance sampling weight to the value network loss function middle:

[0127]

[0128] in are network parameters, The sampling probability of the behavior strategy is updated through the loss value. The policy network updates the output strategy according to the updated value network. When the loss converges, Output the final strategy , end the training.

Claims

1. A collaborative knowledge fusion reinforcement learning method for a sparse reward environment, comprising the following steps: Step S1: Build a reinforcement learning framework including a policy network and a value network, generate action probability distribution and learn the optimal strategy during training; Step S2: Construct a lightweight collaborative knowledge fusion model to replace the action output vector of the policy network when the model triggering conditions are met; Step S3: Design a reward function that includes task objectives and dynamic constraints, reshape the reward after the lightweight collaborative knowledge fusion model is triggered, and feed the reshaped reward back to the parameter update process of the policy network and value network.

2. The collaborative knowledge fusion reinforcement learning method for sparse reward environments according to claim 1, characterized in that: The step S1 includes building a reinforcement learning framework including a policy network and a value network, generating an action probability distribution and learning an optimal strategy during training, including the following steps: Step S11: Build the A2C network, introduce the advantage function to optimize the update of the policy gradient, reduce the variance and improve the stability of the training, and the value network of A2C The parameters are optimized by minimizing the temporal difference objective function, which can be expressed as: in, is the starting time step, is the value network parameter; is the discount factor; yes Rewards of the moment; It is the intelligent agent in State passing strategy of The state reached after the step, ; is the single-step timing difference step size, ; Step S12: The policy network passes the generalized advantage function Calculate the policy gradient and determine each state The probability distribution of taking an action under Expressed as: in is the current state, is the current action, the timing difference error Expressed as: The total reward is , the policy gradient update formula is: in are policy network parameters, is the training segment length, is the bias-variance trade-off coefficient, is the policy entropy, It is the entropy regularization weight, which increases the randomness of the strategy and prevents local optimality.

3. The collaborative knowledge fusion reinforcement learning method for sparse reward environments according to claim 1, characterized in that: The step S2 of constructing a lightweight collaborative knowledge fusion model to replace the action output vector of the policy network when the model triggering condition is met includes the following steps: Step S21: Detect the mutation value of the real-time multi-dimensional action space, quantify the mutation degree of the action in adjacent time steps through differential calculation, and define the action change rate for: in for The output of the moment strategy network dimensional action value, the action space interval is set at Avoid frequent shaking movements. Designing independent thresholds in multi-dimensional action space for the environment interaction time step , set the intervention flag , when the rate of change of any dimension exceeds the threshold, the intervention is activated ,otherwise : in is the action space dimension index, , a total of dimensional continuous action space; For the The allowed change rate threshold of the dimensional action is calibrated according to the specific action space; the historical action is cached in the buffer , in 2 Complete supervisory decision-making within the company; Step S22: Design a collaborative knowledge strategy model to generate smooth alternative actions , the model expression is: in is a trainable weight matrix, The dimension of the action space is , the dimension of the state space is ; is a trainable bias vector, in the state mapping Up-calibrate the action offset so that the model can output non-zero action when zero state input; is the environmental state vector; is the inertia factor, and the parameter optimization target should be consistent with the original strategy: In the formula is the trade-off coefficient, , Store data for the experience pool. When using the current state and the original output of the policy network constitute the training samples; when Use exponential decay memory Keep the model adaptable.

4. The collaborative knowledge fusion reinforcement learning method for sparse reward environments according to claim 1, characterized in that: The design in step S3 includes a reward function containing task objectives and dynamic constraints, and after the lightweight collaborative knowledge fusion model is triggered, the reward is reshaped, and the reshaped reward is fed back to the parameter update process of the policy network and the value network, including the following steps: Step S31: Constructing the total reward of reinforcement learning , including mission target rewards and dynamic constraint penalty : in is the number of mission objectives, ; For the The weight of the task, ; Rewards for task completion targets, is the number of constraints, is the penalty coefficient of the dynamic constraint term, ; Step S32: The task objective reward function is designed as follows: in For the specific implementation of the target task reward value, For the current moment State and goal constraints The difference, is the threshold parameter; The dynamic constraint penalty design is as follows: in For the The actual value of the constraint, , is the safety threshold; Step S33: When the lightweight collaborative knowledge fusion model is activated, reward sparsity compensation is performed, and the knowledge fusion reward is superimposed on the basic reward. : in is the sub-goal potential energy difference, is a constant coefficient, : Is the sub-goal potential function, measuring the current state The state closest to the sub-goal Proximity: Activate rewards for knowledge fusion, where Status Number of times it was visited during training: is the state-dependent collaborative knowledge strategy alignment, is the feature difference vector between the collaborative knowledge state and the reinforcement learning state in the current state, and each component It is the collaborative knowledge and reinforcement learning in the Difference values ​​on dimension features: is the adaptive weight, : for The dynamic adjustment coefficient is in the range of ; Step S34: Collaborative knowledge fusion has prior knowledge and reasoning ability. In the experience replay pool, it is necessary to weight the collaborative knowledge to construct the priority and define the priority function To evaluate the collaborative knowledge strategy distribution: in is the TD error; is a sufficiently small positive real number that makes the priority function probability greater than 0; the last term is the advantage term of the value function, To coordinate the value of knowledge action, It is the reinforcement learning action value, which is used to evaluate the degree to which the collaborative knowledge fusion in the experience pool is retrieved without considering the TD error. During the reinforcement learning training process, the priority of the collaborative knowledge will change with the performance of the agent. The probability density function is expressed as follows: in is the experience pool data index, is a hyperparameter, , A value close to 1 represents a high-value sample. Approaching 0 means giving up collaborative knowledge guidance; Step S35: Since the priority mechanism changes the experience distribution in the buffer, the estimation of the expected value of the value function introduces a bias, which will affect the stability and robustness of the strategy. The bias can be reduced by introducing the importance sampling weight into the loss function of the value network. The importance sampling weight The calculation is as follows: in, is a dynamic coefficient, Completely non-uniform sampling occurs when Completely uniform sampling occurs when It will gradually decrease to 0 during training, adding the importance sampling weight to the value network loss function middle: in are network parameters, The sampling probability of the behavior strategy is updated through the loss value. The policy network updates the output strategy according to the updated value network. When the loss converges, Output the final strategy , end the training.

Images