A reinforcement learning method based on expert agent guidance and mirror self-game and a computer readable storage medium

Abstract

This invention belongs to the field of artificial intelligence technology and relates to a reinforcement learning method based on expert agent guidance and mirror self-game, as well as a computer-readable storage medium. This method aims to solve the problems of low training efficiency, difficult convergence, and limited policy performance in reinforcement learning under high-dimensional action spaces. The method first constructs an expert agent in a low-dimensional action space and trains it to exceed typical metaheuristic algorithms. Then, it constructs a high-dimensional agent with a consistent state space but a complete action space. In the first stage, rapid convergence is achieved through expert guidance. In the second stage, a mirror self-game mechanism is introduced, utilizing historical version adversarial generation to drive continuous policy evolution through advantageous rewards. Entropy regularization and an experience replay buffer sliding window are combined to improve exploration capabilities and data quality. This method effectively integrates prior knowledge and autonomous exploration, significantly improving policy performance and generalization ability.

Description

Technical Field

[0001] This invention relates to the field of AI reinforcement learning technology, and in particular to a reinforcement learning method based on expert agent guidance and mirror self-game, as well as a computer-readable storage medium. Background Technology

[0002] Reinforcement learning, as a core paradigm in artificial intelligence for achieving autonomous decision-making and policy optimization, has shown significant potential in recent years for high-dimensional combinatorial optimization problems such as complex scheduling, resource allocation, and intelligent manufacturing. Its core mechanism relies on agents accumulating experience through trial and error in continuous interaction with the environment, gradually approaching the optimal policy. However, compared to paradigms like supervised learning that rely on labeled data, reinforcement learning's inherent sparse rewards, high variance in returns, and the exploration challenges posed by its large state-action space often result in inefficient and unstable training processes. Especially in discrete combinatorial optimization scenarios such as workshop scheduling and logistics route planning, the set of actionable actions grows exponentially with the problem size. Agents that directly construct the entire action space often face the curse of dimensionality, leading to slow policy convergence or even ineffective learning.

[0003] Existing technologies generally employ methods that construct low-dimensional behavior spaces based on prior human knowledge. Specifically, researchers encode heuristic rules validated in practice (such as first-in-first-out, shortest processing time priority, etc.) or efficient strategies designed by domain experts into a set of candidate behaviors for intelligent agents. Because the number of such rules is limited and they have clear engineering significance, the dimensionality of the constructed behavior space is significantly reduced, and the search complexity is drastically decreased. This allows reinforcement learning algorithms to complete training within a reasonable time and outperform traditional metaheuristic algorithms (such as genetic algorithms, particle swarm optimization, and ant colony optimization) on multiple benchmark problems. The advantage of this method lies in its effective integration of human experience and data-driven learning, avoiding blind exploration from scratch while retaining the flexibility of policy fine-tuning, thus becoming the mainstream technical approach in current industrial-grade intelligent decision-making systems.

[0004] However, as application scenarios place stringent demands on decision-making performance to approach theoretically optimal solutions, the aforementioned rule-based approach of compressing the behavior space presents a contradiction. Fundamentally, each rule can only select one or a few specific candidate options (e.g., in the workshop scheduling problem, using a first-in-first-out rule means only the earliest arriving workpiece can be selected). Using a limited set of rules / strategies as candidate behaviors results in a limited upper limit to decision-making performance, making it impossible to find the globally optimal solution. To find the globally optimal solution, all candidate options need to be selectable at each decision point (e.g., all candidate workpieces that a machine can process). However, constructing the behavior space from the set of all candidate options / actions leads to a high-dimensional behavior space, a large search space, and difficult learning. Therefore, how to solve the problem of efficient learning for intelligent agents in high-dimensional behavior spaces is a crucial issue.

[0005] Therefore, how to construct a two-stage learning paradigm that combines the rapid convergence characteristics of expert guidance with the continuous evolution capability driven by self-game theory, so as to achieve a gradual leap from an efficient rule starting point to a globally optimal strategy, has become a key technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] The purpose of this invention is to propose a reinforcement learning method and a computer-readable storage medium based on expert agent guidance and mirror self-game. By constructing a two-stage learning mechanism, in the first stage, an expert agent in a low-dimensional behavior space guides the behavior of a high-dimensional agent, achieving rapid convergence to the expert-level decision-making level. In the second stage, a mirror self-game mechanism is introduced, enabling the high-dimensional agent to continuously explore better strategies in an adversarial environment composed of its own historical versions, thereby breaking through the subspace limitations imposed by expert rules and approaching the global optimum.

[0007] To achieve the above objectives, this invention proposes a reinforcement learning method based on expert agent guidance and mirror self-game, characterized by the following steps: S1: Construct and train an expert agent, which has a low-dimensional behavior space verified by domain experts; S2: Construct a high-dimensional intelligent agent whose state space is consistent with the expert intelligent agent, and whose behavior space covers the complete set of legal actions; S3: In the first learning phase, the current environmental state is simultaneously input into the high-dimensional agent and the trained expert agent. The environmental feedback after the high-dimensional agent performs an action and the environmental feedback after the expert agent hypothetically performs an action are obtained respectively. An advantage reward is constructed based on the difference between the two to update the high-dimensional agent's strategy. S4: In the second learning phase, a historical model copy of the high-dimensional agent is periodically generated as a mirror agent and stored in the mirror agent pool. In each interaction, an opponent is selected from the mirror agent pool. An advantage reward is constructed based on the difference in environmental feedback obtained by the high-dimensional agent and the selected mirror agent performing actions in the same state to update the high-dimensional agent's strategy. S5: After training, save the high-dimensional agent model for actual decision-making.

[0008] Furthermore, in step S1, the state space of the expert agent includes state features extracted based on domain knowledge, environmental image information, and communication information in the multi-agent system; the behavior space of the expert agent consists of several heuristic rules that have been verified and are effective by domain experts; the network structure of the expert agent sequentially includes a graph neural network module, a principal component analysis module, and a multilayer perceptron module; the reward function of the expert agent is defined as the environmental feedback generated by the system between adjacent decision points.

[0009] Furthermore, the graph neural network module uses a message passing mechanism to process node and edge information in the environmental image, the principal component analysis module extracts the principal components of the state features through singular value decomposition, and the multilayer perceptron module contains at least three fully connected layers and uses a modified linear unit as the activation function.

[0010] Furthermore, the network structure of the high-dimensional agent shares the input layer and intermediate layer structure with the expert agent, with only the number of neurons in the output layer matching the size of the complete action set. An action masking mechanism is integrated in the output layer, which dynamically generates a Boolean mask vector based on the current state to force the probability of invalid actions to be zero.

[0011] Furthermore, in S3, the training termination conditions for the first learning stage include: reaching a preset maximum number of training rounds; or the average environmental feedback obtained by the high-dimensional agent within a certain number of consecutive rounds being no less than a set proportion of the feedback corresponding to the expert agent; or the performance index showing no significant improvement within a fixed-length sliding window.

[0012] Furthermore, in S4, the mirror pool maintains the most recent historical versions of several high-dimensional agents and historical snapshots with excellent performance. Its maximum capacity is a fixed value. When a new mirror agent is added and the pool is full, the worst-performing or the least selected historical version is removed first. In the second learning phase, each interaction selects the latest mirror agent with a high probability and randomly selects any historical mirror agent in the pool as the game opponent with a low probability. The high-dimensional agent and the mirror agent share the same environment simulator to ensure consistency of state transition.

[0013] Furthermore, an entropy regularization term is introduced during the policy optimization process. The policy loss function includes the weighted expected value of the original policy loss minus the policy entropy. The entropy coefficient is set to a larger value in the early stage of training and gradually reduced in the later stage of training according to a predetermined decay strategy.

[0014] Furthermore, the experience replay buffer is managed using a sliding window mechanism, retaining only the interaction data generated in the most recent N training rounds, where N is a preset positive integer.

[0015] The present invention also proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements a reinforcement learning method based on expert agent guidance and mirror self-game.

[0016] Compared with existing technologies, the advantages of this invention are as follows: By constructing a two-stage reinforcement learning framework that combines expert guidance and mirror self-game, this invention effectively reconciles the contradiction between utilizing prior knowledge and global exploration capabilities. The first stage achieves efficient learning through a low-dimensional expert agent, while the second stage relies on the mirror self-game mechanism to break rule constraints, driving continuous policy evolution and thus surpassing the expert agent to approach the global optimum. This method significantly expands the policy expression capability while maintaining training efficiency, providing a practical and advanced intelligent decision-making solution for complex combinatorial optimization problems. Attached Figure Description

[0017] Figure 1 The overall process of the two-stage learning method for enhanced learning agents; Figure 2 Diagram of expert agent network structure and learning principle; Figure 3 A diagram illustrating the principle of a two-stage learning method to enhance learning. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be further described below.

[0019] This invention proposes a reinforcement learning method based on expert agent guidance and mirror self-game, used to train a reinforcement learning agent to learn quickly and continuously, thereby improving the agent's decision-making level. The overall process is as follows: Figure 1 As shown, it includes the following steps: Step S1: Construct and train an expert agent. Construct an agent with a low-dimensional behavioral space and train it using a reinforcement learning algorithm. After training, it can surpass general metaheuristic algorithms and reach an expert level. Save the model to obtain the expert agent. Specifically, it includes the following steps: Step S1.1: Construct an expert agent, designing the agent's state space, behavior space, reward function, and network structure; The state space includes state features designed based on domain knowledge (including the state and quantity of resources / products / individuals, features related to the target value, and features representing the system's operating state), environmental image information, and communication information between agents; such as Figure 2 As shown, taking the workshop scheduling problem as an example, environmental image information can be obtained from the dissecting graph of the workshop scheduling problem; the state features designed based on domain knowledge can be the delivery date of products waiting to be processed, the processing time of each workstation, and the penalty for product delay at the current moment; The behavior adopts efficient rules designed based on domain knowledge. For example, for the workshop scheduling problem, classic scheduling rules (first-in, first-out, shortest total processing time, longest total processing time, earliest delivery date) can be used, or rules can be designed manually based on the characteristics of the optimization problem (such as maximum delay penalty, maximum equipment utilization rate). In a multi-agent system, there is communication between agents, including the state information and decision-making behavior of other agents, as well as the information after this information has been processed (e.g., through a multilayer perceptron). The reward function is the system feedback generated by the system within adjacent decision points. Taking the workshop scheduling problem as an example, if the optimization objective is the delay penalty for all products, the reward function can be defined as the newly added delay penalty value between two adjacent decision points. Network structure such as Figure 2 As shown, the system comprises a graph neural network (Graph Neural Network), a principal component analysis (PCA) model, and a multilayer perceptron. The Graph Neural Network processes image information in the environment, such as the disjunctive graph in a workshop scheduling problem. PCA extracts key information from high-dimensional state features. The multilayer perceptron further processes the input information. The Graph Neural Network module uses a message passing mechanism to iteratively aggregate node and edge information in the environmental image. The PCA module receives the high-dimensional node embeddings and original state features output by the Graph Neural Network, extracts the top k principal components through Singular Value Decomposition (SVD), and retains information with a cumulative variance contribution rate of no less than 95%. The multilayer perceptron module contains three fully connected layers, with the Rectified Linear Unit (ReLU) activation function uniformly used. The final output layer is normalized by Softmax to generate the selection probability distribution of each heuristic rule.

[0020] Step S1.2: The expert agent learning and training method is as follows: In each decision step, the agent learns based on the current state. s Generate behavior a Execute within the environment and receive rewards from environmental feedback. r and the latest status s’ , will interactive data { s , a , r , s’ Store and update agent network parameters based on interaction data; Training is performed using value-based reinforcement learning algorithms (such as the DQN algorithm) or policy-based reinforcement learning algorithms (such as the PPO algorithm); training ends when a termination condition is reached, such as training 10,000 EPOCHs. Step S1.3: After training is completed, test the performance of the agent. If it can outperform general metaheuristic algorithms, such as genetic algorithms and particle swarm algorithms, then save the agent model as the trained expert agent; if it cannot outperform them, return to step S1.2 to continue training. Step S2: Construct a high-dimensional intelligent agent, design the state space, behavior space and network structure of the high-dimensional intelligent agent, where behavior refers to all legal individuals / actions. The intelligent agent has a large behavior space and search space, making training difficult. Specifically: The state space of a high-dimensional intelligent agent is the same as that of an expert intelligent agent, so that the same input information can be used; The behavior space is the complete set of all candidate options / actions; for example, in the shop floor scheduling problem, the behavior space is all the workpieces that are currently available for the machine, not just a certain scheduling rule. The input and intermediate layers of the network structure are the same as those of the expert agent. Since the number of neurons in the output layer equals the number of actions, the high-dimensional agent has more output layers than the expert agent, and it has a masking mechanism in the output layer to mask individuals / actions that are currently unavailable. This mechanism is implemented through a Boolean mask vector m(s), where m(s) ∈ {0,1}^{|A|}, where |A| is the size of the action space, and m_i(s)=0 indicates that action i is infeasible in state s. The mask vector is dynamically generated at each decision step based on the legality of the current state, such as resource availability, task dependencies, or physical constraints. The original output logits z of the policy network becomes z' = z⊙m(s) - (1 - m(s))·M after masking, where ⊙ is the element-wise product, and M is a large positive number (e.g., 10^6) to ensure that the probability output of invalid actions approaches zero. The final action probability distribution is given by Softmax(z').

[0021] The overall architecture diagram of the two-stage reinforcement learning method is as follows: Figure 3 As shown, the first stage is expert agent-guided learning, and the second stage is mirror self-game continuous learning, including steps S3 and S4. Step S3: For the high-dimensional agent, in the first stage of learning, rapid learning is carried out based on the guidance of the expert agent. The trained expert agent and the high-dimensional agent solve the same problem. By rewarding the current agent's decision with the advantage of the expert agent, the current agent is guided to move closer to the decision of the expert agent, thereby quickly learning to the expert level. Specifically: Step S3.1: At each decision point, obtain the current state of the environment. s c , as input to the intelligent agent; Step S3.2: The current agent is based on the state input. s c Output behavior a c It is executed within the environment, and the environment provides reward feedback for the current behavior. r cFor example, after the current workpiece is completed, the system adds a delay penalty value within the decision period; Step S3.3: Apply the same environmental conditions as before the decision. s c Input is given to the expert agent, and the action is output. a e Imagine executing the action in an environment and receiving reward feedback from that environment. r e ; Step S3.4: Subtract the feedback obtained by the expert agent from the environmental feedback obtained by the current agent, and use this as the advantage reward for the current agent relative to the expert agent. r adv ,Right now r adv = r c - r e ; Step S3.5: Reward with advantages r adv Update the network parameters of the current agent to indicate how far the current agent's choice is from the expert's choice, thereby guiding the current agent to continuously learn from the expert agent and get closer to the expert agent's level; use value-based or policy-based reinforcement learning algorithms for learning optimization; Step S3.6: The termination condition for the first stage of learning can be a given number of rounds, or the feedback obtained by the current agent within a certain training interval. r c Reaching expert intelligent agents and obtaining feedback r e The performance can be improved to a certain level, such as reaching 90%; if the performance does not improve for several consecutive rounds, the first stage of training can be terminated. Determine if the first stage of training has reached the termination condition. If not, continue to step S3.1. If it has, test the performance of the current agent. If it is close to the expert level (e.g., the total round reward is greater than 90% of that of an expert agent), proceed to the second stage of learning. Otherwise, continue to the first stage of learning. Step S4: In the second stage of learning, the current high-dimensional agent is replicated in stages to obtain its mirror agent. Based on the advantage reward of the current agent to the decision-making behavior of the mirror agent, the current agent is encouraged to explore new and better strategies. Through self-game, continuous learning is achieved to break through the expert level. Specifically: Step S4.1: For the current agent, every certain number of rounds, make a copy of its mirror image with the same model structure and network parameters; Set up a mirror pool to store several recent mirror images and high-performing mirror images from historical versions. When selecting, the latest mirror image of the current agent is selected most of the time, and a historical version mirror image is randomly selected from the mirror pool with a certain probability. By introducing the mirror pool, a different opponent is randomly selected for training after a period of time to alleviate the catastrophic forgetting problem. The maximum capacity of the mirror pool P is set to a fixed value M=20. When a new mirror image is added and the pool is full, the worst-performing or the historical version that has not been selected for the longest time is removed first.

[0022] Step S4.2: The method for the current agent to interact with the environment and update the network is the same as in the first stage (steps S3.1-S3.5), except that a mirror agent replaces the expert agent in the first stage; the advantage reward becomes the difference between the current agent and the mirror agent, i.e. r adv = r c - r m ,in r m The environmental reward feedback obtained by performing the same action for the mirror agent; Reward with advantages r adv Update the current agent to indicate how much the current agent's decision has improved compared to the previous one, and guide the current agent to continuously explore new and better strategies, and carry out continuous learning through mirror self-game; To make training more effective, the following steps are taken: The current agent's policy π ( a | s Entropy regularization is applied to prevent strategies from converging prematurely to deterministic behavior, which would turn self-play into a repetitive confrontation between two fixed strategies, making it impossible to discover new strategies. Incorporating entropy reward into the policy loss encourages the policy to maintain randomness, as shown in the following formula: L policy =L original - β ⋅E[H(π(⋅∣ s ))] H(π) = -Σπ( a | s log π( a | s ) Where L policy It is the ultimate loss of the strategy, L original This is the initial loss calculated by the strategy, β is the entropy coefficient, E represents the expected value, and π(⋅| s H(π) is the policy, and H(π) is the entropy of policy π. s Indicates state, aIndicates an action; To encourage exploration, adjustments will be made dynamically. β That is, a larger β is used in the early stage of training to encourage exploration, and then gradually reduced in the later stage to achieve convergence; To improve the quality of stored data, a sliding window is set for training data storage, meaning only the most recent data is retained. N Update outdated data from each round of data. Step S4.8: The second stage of learning termination condition can be a number of rounds set manually, such as 10,000 rounds, or a number of consecutive rounds in which performance does not improve; If the second phase of learning is not completed, return to step S4.1 and repeat the above steps; Specifically, step S5, intelligent agent model storage and decision-making application, includes the following: Save the trained model, for example, as a pkl format; For a problem to be solved, the trained model is invoked, the state is input, the agent generates behavior, executes it in the environment, obtains a new state, and this process is repeated to enable the agent to solve the unseen problem.

[0023] Step S5: Save the learned high-dimensional agent model, and solve the problem by loading the trained model to realize decision-making applications; Specifically, it is applied to computer-readable storage media, on which computer programs are stored. When the computer programs are executed by a processor, a reinforcement learning method based on expert agent guidance and mirror self-play is implemented.

[0024] The above are merely preferred embodiments of the present invention and do not constitute any limitation on the present invention. Any equivalent substitutions or modifications made by those skilled in the art to the technical solutions and content disclosed in the present invention without departing from the scope of the present invention shall be deemed to have remained within the protection scope of the present invention.

Claims

1. A reinforcement learning method based on expert agent guidance and mirror self-game, characterized in that, Includes the following steps: S1: Construct and train an expert agent, which has a low-dimensional behavior space verified by domain experts; S2: Construct a high-dimensional intelligent agent whose state space is consistent with the expert intelligent agent, and whose behavior space covers the complete set of legal actions; S3: In the first learning phase, the current environmental state is simultaneously input into the high-dimensional agent and the trained expert agent. The environmental feedback after the high-dimensional agent performs an action and the environmental feedback after the expert agent hypothetically performs an action are obtained respectively. An advantage reward is constructed based on the difference between the two to update the high-dimensional agent's strategy. S4: In the second learning phase, a historical model copy of the high-dimensional agent is periodically generated as a mirror agent and stored in the mirror agent pool. In each interaction, an opponent is selected from the mirror agent pool. An advantage reward is constructed based on the difference in environmental feedback obtained by the high-dimensional agent and the selected mirror agent performing actions in the same state to update the high-dimensional agent's strategy. S5: After training, save the high-dimensional agent model for actual decision-making.

2. The reinforcement learning method based on expert agent guidance and mirror self-game as described in claim 1, characterized in that, In step S1, the state space of the expert agent includes state features extracted based on domain knowledge, environmental image information, and communication information in the multi-agent system; the behavior space of the expert agent consists of several heuristic rules that have been verified and are effective by domain experts; the network structure of the expert agent sequentially includes a graph neural network module, a principal component analysis module, and a multilayer perceptron module; the reward function of the expert agent is defined as the environmental feedback generated by the system between adjacent decision points.

3. The reinforcement learning method based on expert agent guidance and mirror self-game as described in claim 2, characterized in that, The graph neural network module uses a message passing mechanism to process node and edge information in the environmental image. The principal component analysis module extracts the principal components of the state features through singular value decomposition. The multilayer perceptron module contains at least three fully connected layers and uses a modified linear unit as the activation function.

4. The reinforcement learning method based on expert agent guidance and mirror self-game as described in claim 1, characterized in that, The network structure of the high-dimensional agent shares the input layer and intermediate layer structure with the expert agent. Only the number of neurons in the output layer matches the size of the complete action set. An action masking mechanism is integrated in the output layer. The masking mechanism dynamically generates a Boolean mask vector based on the current state to force the output of invalid actions to have a probability of zero.

5. The reinforcement learning method based on expert agent guidance and mirror self-game as described in claim 1, characterized in that, In S3, the training termination conditions for the first learning stage include: reaching the preset maximum number of training rounds; or the average environmental feedback obtained by the high-dimensional agent within a certain number of consecutive rounds being no less than the set proportion of the feedback corresponding to the expert agent; or the performance index showing no significant improvement within a fixed-length sliding window.

6. The reinforcement learning method based on expert agent guidance and mirror self-game as described in claim 1, characterized in that, In step S4, the mirror pool maintains the most recent historical versions of several high-dimensional agents and historical snapshots with excellent performance. Its maximum capacity is a fixed value. When a new mirror agent is added and the pool is full, the worst-performing or the least selected historical version is removed first. In the second learning phase, each interaction selects the latest mirror agent with a high probability and randomly selects any historical mirror agent in the pool as the game opponent with a low probability. The high-dimensional agent and the mirror agent share the same environment simulator to ensure consistency of state transition.

7. The reinforcement learning method based on expert agent guidance and mirror self-game as described in claim 1, characterized in that, An entropy regularization term is introduced during the policy optimization process. The policy loss function contains the weighted expected value of the original policy loss minus the policy entropy. The entropy coefficient is set to a larger value in the early stage of training and gradually reduced in the later stage of training according to a predetermined decay strategy.

8. The reinforcement learning method based on expert agent guidance and mirror self-game as described in claim 1, characterized in that, The experience replay buffer is managed using a sliding window mechanism, retaining only the interaction data generated in the most recent N training rounds, where N is a preset positive integer.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the reinforcement learning method based on expert agent guidance and mirror self-game as described in any one of claims 1-8.

Images