A Human-Machine Interactive Collaborative Decision-Making Method Based on Expert Knowledge Base and Reinforcement Learning in Complex Industrial Environments

Abstract

This invention provides a human-machine mutually beneficial collaborative decision-making method based on expert knowledge base and reinforcement learning for complex industrial environments. It addresses the contradiction between cognitive biases in human decision-makers and the limited dynamic decision-making capabilities of intelligent agents, overcoming the limitations of traditional intelligent agents in fuzzy and uncertain scenarios through human-machine collaboration. For human-machine collaborative decision-making, it constructs a human-machine mutually beneficial collaborative decision-making model incorporating real-time human intervention, dynamic information support from the expert knowledge base, and online optimization by the intelligent agent. It integrates a structured expert experience reasoning module from the expert knowledge base, an adaptive learning module for the reinforcement learning intelligent agent, and a two-way information interaction and control closed-loop module between humans and machines. This establishes a two-way enhancement mechanism where "expert knowledge base and intelligent agent decision-making jointly guide on-site expert decision-making, and on-site decision results inversely optimize the parameters of the expert knowledge base and the intelligent agent." Applicable to typical environments such as intelligent manufacturing and industrial production, it achieves the fusion of human decision-making and intelligent agent learning, driving the intelligent agent to evolve to a higher cognitive level.

Description

Technical Field

[0001] This invention relates to complex industrial environments, specifically to an intelligent agent-machine interactive collaborative decision-making method based on expert knowledge base and reinforcement learning in complex industrial environments. Background Technology

[0002] Against the backdrop of the rapid evolution of the Industrial Internet and the deep integration of intelligent manufacturing, modern industrial production systems are gradually developing towards higher intelligence and flexibility. The operating status of industrial equipment in complex industrial environments exhibits high-dimensionality, strong coupling, and time-varying characteristics. Decision-making needs often involve complex scenarios with multiple stages, multiple objectives, and multiple constraints, and there are strict temporal dependencies and resource competition relationships between tasks. For example, temperature regulation in chemical production must simultaneously meet safety thresholds, energy consumption limits, and product quality requirements, while equipment fault diagnosis requires combining historical cases, real-time sensor data, and the implicit experience of operators. This decision-making environment of "multiple constraints, conflicting objectives, and dynamic changes" places extremely high demands on the environmental adaptability, experience utilization efficiency, and real-time response capabilities of decision-making systems. Traditional single-entity decision-making methods are no longer sufficient to meet the actual needs of industrial scenarios.

[0003] Traditional methods relying solely on autonomous decision-making by intelligent agents exhibit significant shortcomings in complex industrial environments. Firstly, the industrial environment's state space is vast, requiring massive amounts of data to cover complex scenarios. However, the high-frequency real-time data collected in actual production is often insufficient to support rapid convergence to the optimal strategy. Secondly, tasks such as industrial fault diagnosis and process optimization heavily rely on implicit experience. Pure reinforcement learning, lacking expert guidance, is prone to getting stuck in local optima or making misjudgments. For example, an agent might fail to recognize the nonlinear relationship between vibration frequency and bearing wear, leading to incorrect parameter adjustment suggestions. Furthermore, existing agents are mostly in a "static learning" mode, making it difficult to dynamically adapt to changes in scenarios such as equipment aging and process alterations, thus hindering long-term performance optimization. On the other hand, traditional methods relying on human autonomous decision-making suffer from human cognitive biases, as humans cannot quickly understand and respond to multidimensional information and unexpected situations in complex industrial environments.

[0004] Against this backdrop, developing a "human-machine mutually beneficial collaborative decision-making method integrating expert knowledge base and reinforcement learning" has become an urgent need in complex industrial scenarios. This method leverages a two-way empowerment mechanism of "human experience guidance - agent learning - feedback optimization," fully utilizing human experts' deep understanding of complex scenarios and their grasp of implicit rules. Simultaneously, it relies on the autonomous learning capabilities of intelligent agents to accelerate experience transformation and strategy iteration: human experts compensate for the agent's experience blind spots by calibrating agent decisions in real time and supplementing new scenario experiences; the agent, in turn, transforms implicit experience into learnable policy parameters through priority experience replay and policy updates incorporating human guidance, thus feeding back into the dynamic updates of the knowledge base. This collaborative model of "experience-driven + intelligent evolution" not only solves the decision-making errors caused by the lack of experience in traditional methods but also overcomes the efficiency bottleneck of agent autonomous learning. By extracting information from an expert knowledge base and enabling intelligent agents to quickly respond to equipment status decisions, clear decision-making information is provided to assist humans in making decisions. At the same time, human decision feedback is used to optimize the knowledge base and intelligent agents, ultimately achieving a significant improvement in the environmental adaptability and long-term reliability of the decision-making system in complex industrial scenarios, providing key technical support for the intelligent development of the Industrial Internet. Summary of the Invention

[0005] Technical Challenges: Long-Term Optimization and Adaptive Evolution Mechanisms for Human-Machine Collaboration: In industrial scenarios, dynamic changes such as equipment aging, process alterations, and the application of new materials constantly introduce new tasks and constraints, requiring human-machine collaborative systems to possess long-term adaptability. Current methods suffer from "experience solidification" and "agent overfitting": human experience may become outdated due to technological iterations, while intelligent agents may fail in new scenarios due to over-reliance on historical data. How to construct a closed-loop mechanism of "human experience updating - agent learning - collaborative strategy optimization," ensuring that both evolve together to adapt to environmental changes, is a core requirement for achieving long-term system reliability.

[0006] Decision-making power allocation in dynamic scenarios: The decision-making needs of complex industrial tasks are highly dynamic. In normal, stable scenarios, humans may rely more on experience to make quick decisions; in novel, unknown scenarios, autonomous exploration by intelligent agents may be more efficient. However, current methods lack dynamic weight allocation mechanisms: fixed weights may lead to "lagging human experience" or "misjudgment by intelligent agents"; dynamic weights need to address issues such as "how to quantify scenario complexity" and "how to assess the reliability of human and intelligent agent decisions".

[0007] Real-time interaction and feedback closed-loop design for human-machine decision-making: The decision-making needs of complex industrial environments require high real-time performance in human-machine collaboration. However, current methods suffer from interaction delays and feedback gaps: after the agent generates decision suggestions, humans may be unable to quickly locate key decision points due to information overload; after human intervention, the agent may fail to understand the intervention intent, leading to execution deviations.

[0008] Technical Solution: This invention proposes a human-machine collaborative decision-making method for complex industrial environments. By constructing a closed-loop mechanism of "state perception - dual-source collaborative decision-making - expert intervention - feedback optimization," it achieves a deep integration of human expert experience and the autonomous learning capabilities of intelligent agents. This fully leverages the advantages of human experts' deep understanding of complex scenarios and their grasp of implicit rules, while simultaneously relying on the autonomous learning capabilities of intelligent agents to accelerate experience transformation and strategy optimization, significantly improving the environmental adaptability and long-term reliability of the decision-making system in complex industrial scenarios. The method is based on real-time data of industrial equipment operating status. A reinforcement learning intelligent agent generates preliminary decision suggestions, while a structured experience guidance system outputs from an expert knowledge base, forming a dual-source decision input. On-site human experts, based on multi-dimensional information such as real-time equipment status, intelligent agent decision output, and expert knowledge base experience, collaboratively calibrate the decision suggestions and expert guidance, enabling real-time intervention in the decision-making process or supplementing new scenario experiences.

[0009] First, at the decision-making entity modeling level, industrial equipment operating status data is collected in real time via sensors and synchronously transmitted to a reinforcement learning agent and an expert knowledge base system. The reinforcement learning agent generates decision recommendations based on the real-time collected equipment operating status data. It is built upon a Markov decision process framework and employs an Actor-Critic dual-network architecture: the Actor network generates decision recommendations, while the Critic network evaluates the long-term value of actions and guides action selection through policy functions. The expert knowledge base outputs expert experience guidance based on a structured experience base. Through structured feature extraction and similarity measurement, it filters highly applicable expert experience, providing accurate decision-making basis for equipment control. These two types of agents complement each other, with data-driven and experience-driven approaches respectively, constructing a collaborative decision-making foundation.

[0010] Secondly, at the level of collaborative decision-making, to ensure the safe operation of equipment and avoid the risk of misjudgment by the intelligent agent, the decision suggestions generated by the intelligent agent and the experience guidance output by the expert knowledge base are simultaneously pushed to on-site human experts. The on-site human experts then comprehensively judge, based on multi-dimensional conditions, whether to intervene in the decision-making process or supplement new experience to optimize the historical knowledge base. Specifically, the system adopts a dynamic weight allocation strategy to balance the human-machine decision-making weights: increasing the weight of human experience in routine scenarios and enhancing the weight of autonomous exploration by the intelligent agent in unknown scenarios. The weight values ​​are dynamically adjusted through a time-series decay factor and value difference to ensure rapid learning of human experience in the initial stage and a gradual transition to autonomous evolution of the intelligent agent in the later stage. When a human expert intervenes, their correction instructions are transformed into high-priority samples for reinforcement learning. The sample priority is calculated through time-series difference error, giving expert experience a higher replay weight to ensure that the intelligent agent preferentially absorbs implicit rules.

[0011] Finally, at the closed-loop optimization level, the system achieves continuous evolution of human-machine collaboration through a two-way reinforcement mechanism: if an expert chooses to intervene, the agent will combine the decision instructions after intervention with experience samples from environmental interaction feedback, and complete autonomous optimization through a priority experience replay mechanism and a policy network update method that integrates human guidance, accelerating policy convergence and improving decision robustness; if on-site experts provide new experience, the expert knowledge base will collect this experience sample and update the knowledge representation at the appropriate time according to preset rules; specifically, human intervention actions will be transformed into behavioral cloning loss terms for reinforcement learning, directly integrated into the policy network update process, forcing the agent to approximate human expertise. The system accelerates the absorption of implicit rules by incorporating the decision-making logic of experts. Simultaneously, new scenario experiences supplemented by experts are stored in the experience buffer in real time. The system triggers a full update according to preset rules, merging historical and new experiences to reconstruct the knowledge base and ensure the timeliness and completeness of knowledge representation. In addition, to avoid decision-making rigidity caused by local optima, the system forces human experts to participate in decision evaluation periodically. When new scenarios such as process changes and equipment aging are detected, online incremental training is automatically started to simultaneously optimize the policy network parameters and expert knowledge base. This enables the system to maintain stable control over known scenarios while quickly adapting to dynamic changes in unknown environments.

[0012] This technical solution, through a human-machine bidirectional empowerment mechanism, integrates the tacit experience of human experts with the learning advantages of intelligent agents, significantly improving the reliability of decision-making in complex industrial environments and providing support for the continuous evolution of intelligent manufacturing systems. Specifically, human experts, leveraging their deep understanding of industrial scenarios and long-accumulated tacit experience, play a core role in key stages; while intelligent agents, relying on their learning advantages, rapidly absorb high-value human experience and uncover data patterns through reinforcement learning, compensating for blind spots in human experience. This collaborative mechanism reduces the risk of single-agent decision-making and, through continuous knowledge iteration and strategy optimization, endows intelligent manufacturing systems with adaptive and self-evolving capabilities to cope with complex environments, providing a replicable technological paradigm for flexible production and efficient decision-making in the era of the Industrial Internet.

[0013] Beneficial effects:

[0014] (1) Enhance the reliability and security of the decision-making system. By introducing a dual-source decision-making mechanism of expert knowledge base and reinforcement learning agent and real-time intervention and feedback from human experts, the system can effectively cope with uncertainties such as misjudgment risk and strategy rigidity, and improve the credibility and security of decision-making.

[0015] (2) Enhance the dynamism of environmental adaptation and strategy convergence efficiency. By introducing priority experience playback and knowledge base incremental update mechanism, we can effectively deal with dynamic scenarios such as process changes and equipment aging, and improve the system's speed of adapting to new environments and the efficiency of strategy optimization.

[0016] (3) Achieve continuous evolution and long-term synergy of human-machine capabilities. By introducing a "decision-optimization" closed loop and an online incremental training mechanism, we can effectively address issues such as continuous environmental changes and outdated knowledge, and promote the mutual enhancement and dynamic synergy of system capabilities and expert cognition. Attached Figure Description

[0017] Figure 1 It is a flowchart of the agent feedback optimization stage of the intelligent agent human-machine interactive collaborative decision-making method based on expert knowledge base and reinforcement learning in complex industrial environments.

[0018] Figure 2 This is a schematic diagram of the main principle of the method of the present invention. Detailed Implementation

[0019] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0020] like Figure 2 As shown, this invention provides a human-machine interactive collaborative decision-making method based on expert knowledge base and reinforcement learning in complex industrial environments.

[0021] (1) In complex operation and maintenance scenarios of the Industrial Internet, the efficient and stable operation of industrial equipment faces multiple challenges, including dynamic environmental fluctuations, massive data processing, and the inheritance of implicit experience. Reinforcement learning by a single intelligent agent is susceptible to data bias and sudden environmental changes, while relying solely on expert experience is limited by the low efficiency of implicit knowledge reuse and difficulty in adapting to new scenarios. To address this, a collaborative decision-making system of "data-driven intelligent agent + structured expert knowledge" is constructed, and the model is described as follows:

[0022] The intelligent agent model: The operational status data collected by industrial equipment is processed and transformed into the state input of the reinforcement learning intelligent agent. The modeling process of the reinforcement learning intelligent agent is based on the Markov Decision Process (MDP) framework, and the core lies in defining the state-action-reward-next state tuple. ,in This indicates the current environmental state of the agent. , (a set of state spaces) This indicates the actions that the agent can perform in this state. , (for the action space set) It is to perform an action Immediate rewards from post-environmental feedback ( , (for reward space set) To perform the action The next state after transition ( ), the tuple The basic data samples constitute the interaction between the intelligent agent and the environment. Based on this, the value function is further defined. With policy function Value function Used to quantify the state of an intelligent agent. Next action The expected value of the accumulated discount reward during subsequent interactions reflects the long-term value of the state-action pair; the policy function. Indicates the agent's state Select action The probability distribution is determined. The agent employs an Actor-Critic architecture for policy optimization: the Actor network acts as the policy module, collecting interaction trajectory data through interaction with the environment and updating the policy under the guidance of the Critic network. The Critic network, as the value evaluation module, is responsible for calculating the action selected by the Actor network in the current state. long-term value This provides valuable signals for policy updates, guiding the Actor network to adjust action selection probabilities. Ultimately, this leads to continuous optimization of the strategy. Once the agent receives the corresponding environmental state input, it generates decision suggestions and pushes them to on-site human experts for their decision-making reference.

[0023] Expert Knowledge Base: After processing, the operational status data collected from industrial equipment is input into the expert knowledge base. The knowledge base then outputs corresponding expert experience guidance through structured knowledge matching analysis. The specific process is as follows: First, structured feature vectors are extracted from the input data. Then, a similarity measurement is performed based on the correlation between the input status feature vectors and candidate expert experiences, thus forming a probability distribution representing the applicability of the expert experience. Next, the top k most applicable expert experiences are selected in descending order of probability as the output results. Finally, by analyzing the key information contained in these top-k expert experiences, such as equipment operation specifications, fault handling strategies, and process parameter suggestions, direct decision-making basis is provided for tasks such as adjusting equipment operating parameters, diagnosing faults, and optimizing operating strategies in industrial scenarios, achieving accurate reuse and scenario adaptation of expert experience.

[0024] (2) Human Intervention Stage: In the intelligent decision-making scenario of the Industrial Internet, although the collaboration between intelligent agents and expert knowledge bases has greatly improved decision-making efficiency, the single data-driven or experience-driven decision-making model still has limitations—intelligent agents may fall into local optima due to data bias, and expert knowledge bases may fail due to outdated experience. In this context, the human intervention stage, as a key quality control link, ensures the accuracy of decisions and system stability through real-time calibration and periodic evaluation. The specific process is as follows: In the on-site human expert decision-making process: In the traditional human-machine collaborative decision-making process, there are two typical mechanisms: one is that humans retain the final decision-making power, that is, after the intelligent agent provides decision suggestions, the human expert directly decides the final execution plan; the other is weighted fusion decision-making, which uses preset weights to weight and combine the human decision and the intelligent agent decision results to generate the final decision. Considering the application characteristics of the industrial environment, current practice tends to adopt the model of trusting human expert decision-making: human experts directly decide the final operation mode of the equipment based on professional knowledge, on-site experience and comprehensive judgment of the overall goal. This mechanism not only meets the core requirements of industrial scenarios for decision reliability and accountability traceability, but also fully leverages the initiative and authority of human experts in complex situations. Upon receiving the agent's decision output and the expert experience output from the expert knowledge base, the on-site human expert first compares the similarity between the two. If there is a decision conflict or the agent makes a mistake, the human expert will intervene in real time. The human expert will first determine whether the agent's decision is erroneous. If so, the on-site expert will provide a real-time expert decision based on the equipment's real-time operating status to control the equipment's operation, and simultaneously push this human decision and environmental feedback to the agent for online optimization. If the experience provided by the expert knowledge base is erroneous, the on-site expert will generate targeted feedback based on the current equipment operating status characteristics and update the expert knowledge base to optimize historical experience. In addition, the system will force human experts to participate in decision evaluation at fixed intervals, which can effectively avoid decision-making deviations caused by dynamic environmental changes or the agent's local optimum tendency, significantly improving the long-term stability and strategy consistency of the agent's decisions.

[0025] (3) Agent feedback optimization stage: such as Figure 1 As shown, an agent optimization method based on a fusion mechanism of priority experience replay and human intervention is designed. In the process of reinforcement learning agent optimization, industrial scenarios place higher demands on decision-making accuracy and experience inheritance. Traditional single experience buffers (such as conventional RL experience buffers) are no longer sufficient. Often, the scarcity and high value of expert experience are overlooked, making it difficult for agents to efficiently absorb mature strategies from on-site experts. To address this, the agent feedback optimization phase employs differentiated storage and priority reinforcement mechanisms to focus on optimizing the utilization efficiency of expert experience: if the environmental feedback received by the agent originates from expert experience, it is stored in a buffer similar to the regular RL experience buffer. Expert experience buffer zone for differentiated management Meanwhile, a priority experience replay mechanism is adopted when extracting experience samples. Through dynamic weighting, expert experience is given a higher replay priority than conventional experience to ensure that the agent learns on-site expert experience first. When the agent updates the policy network, expert guidance experience is integrated into the update process. By constructing a dynamic loss function and introducing an adaptive weight mechanism, the influence of expert experience on the policy network update is adjusted.

[0026] The priority-based experience replay mechanism addresses the issue that traditional experience replay mechanisms treat all samples equally, leading to insufficient utilization of high-quality human expert experience. Therefore, this approach employs a priority sampling mechanism to dynamically increase the sampling probability of expert experience samples, thereby accelerating policy convergence and improving decision robustness. Specifically, this mechanism calculates the priority of expert experience samples using temporal difference error and the Q-dominance function, employing the formula... Determine priorities, among which It is a small positive number to ensure that the priority is always positive and to avoid the zero-error experience being completely ignored, where the timing difference term The definition is as follows:

[0027]

[0028] in: This represents the immediate reward at time t, i.e., in state t. Execute action Rewards based on real-time environmental feedback; It is a discount factor with a value range of [0,1]. It is the state of the policy network. The expression combines the value estimate of the current state-action pair with the action selection of the next state policy network to accurately quantify the contribution of samples to policy optimization. This, through a priority scheduling mechanism, significantly improves the utilization efficiency of high-quality expert experience, thereby accelerating the convergence of the reinforcement learning process and enhancing decision robustness.

[0029] The policy network update process employs a human intervention fusion mechanism, achieving efficient collaboration between expert knowledge and reinforcement learning through the construction of a dynamic loss function, thus balancing human experience and autonomous learning. The specific definition of the dynamic loss function is as follows:

[0030]

[0031] The core parameter design reflects the dynamic balance mechanism of human-machine collaboration: human-machine intervention indicators. As a binary state variable, it takes a value of 1 when human intervention occurs and 0 otherwise, thus identifying scenarios where human decision-making is involved; the device ultimately executes the action. Defined as when humans participate When no intervention is needed, the agent's policy input is used. To balance human guidance and autonomous exploration, and to prevent the policy network from converging prematurely, adaptive weights are used here. The time decay factor is dynamically adjusted, and its calculation formula is as follows:

[0032]

[0033] in: It is the time decay factor ( (where k is the number of training rounds), the degree of dependence on human experience decreases as the strategy matures; To quantify the value advantage of human decision-making compared to AI decision-making, an exponential function is used to amplify the difference in advantages and a lower limit is set; Positive weights are assigned only when human actions are superior to policy actions. Note that these adaptive weights do not participate in the backpropagation process to avoid interfering with network convergence. Through the coordinated design of the above parameters, this mechanism achieves a dynamic balance between expert knowledge injection and reinforcement learning-based autonomous optimization in policy network updates. It accurately captures key decision-making scenarios through human intervention markers and balances the differences in human-machine decision-making value using adaptive weights. Thus, while ensuring the safety of industrial control, it constructs a continuous evolutionary path for the agent based on feedback from real-world scenarios, significantly improving the decision-making efficiency and robustness of human-machine collaborative systems in complex dynamic environments.

[0034] (4) Expert knowledge base feedback optimization stage: In the actual production process, when on-site experts provide real-time expert experience, the expert knowledge base will receive and store the experience in real time. Considering that the expert knowledge base needs to support the operation of the equipment in real time and cannot perform a full update operation in the production process, the system will select a specific time window to trigger the knowledge base update according to the equipment preset rules. When updating, the newly received real-time expert experience will be merged with the existing experience in the historical expert experience base. A new expert knowledge base containing real-time experience will be constructed through the full update process, so as to ensure the real-time operation of the equipment and realize the incremental iterative optimization of expert knowledge.

Claims

1. A human-machine mutually beneficial collaborative decision-making method for intelligent agents based on expert knowledge base and reinforcement learning in complex industrial environments, characterized by the following steps: The process is as follows: First, industrial sensors are used to collect real-time data on the operating status of industrial equipment, which is then simultaneously transmitted to the reinforcement learning agent and the expert knowledge base. The reinforcement learning agent generates decision suggestions based on real-time collected equipment operation status data, and the expert knowledge base outputs expert experience guidance based on a structured experience base. To ensure the safe operation of the equipment and avoid the risk of misjudgment by the intelligent agent, the decision suggestions generated by the intelligent agent and the experience guidance output by the expert knowledge base are pushed to the on-site human experts simultaneously. The on-site human experts will then make a comprehensive judgment based on multi-dimensional conditions as to whether to intervene in the decision-making process or supplement new experience to optimize the historical knowledge base. If the experts choose to intervene, the intelligent agent will combine the decision instructions after intervention with the experience samples of environmental interaction feedback, and complete the autonomous optimization through a priority experience replay mechanism and a policy network update method that integrates human guidance, thereby accelerating policy convergence and improving decision robustness. If on-site experts provide new experiences, the expert knowledge base will collect samples of those experiences and update the knowledge representation at the appropriate time according to preset rules.

2. The method for intelligent agent-machine mutually beneficial collaborative decision-making based on expert knowledge base and reinforcement learning in complex industrial environments, as described in claim 1, is characterized by: The operational status data collected by the industrial equipment is processed and converted into the state input of the reinforcement learning agent. The modeling process of the reinforcement learning agent is based on the Markov decision process framework, and the core lies in defining the state-action-reward-next state tuple. ,in This indicates the current environmental state of the intelligent agent. , For a set of state spaces, This indicates the actions that the agent can perform in this state. , For the action space set, It is to perform an action Immediate rewards based on post-environmental feedback , For the reward space collection, To perform the action The next state after transitioning to The tuple The basic data samples constituting the interaction between the intelligent agent and the environment; based on this, the value function is further defined. With policy function Value function Used to quantify the state of an intelligent agent. Next action The expected value of the accumulated discount reward during subsequent interactions reflects the long-term value of the state-action pair; the policy function. Indicates the agent's state Select action The probability distribution; the agent adopts an Actor-Critic architecture to optimize the policy: the Actor network acts as the policy module, collecting interaction trajectory data through interaction with the environment and updating the policy under the guidance of the Critic network; the Critic network acts as the value evaluation module, responsible for calculating the action selected by the Actor network in the current state. long-term value This provides valuable signals for policy updates, guiding the Actor network to adjust action selection probabilities. Ultimately, this achieves continuous optimization of the strategy; once the agent receives the corresponding environmental state input, it will generate decision suggestions and push them to on-site human experts for their decision-making reference.

3. The method for intelligent agent-machine mutually beneficial collaborative decision-making based on expert knowledge base and reinforcement learning in complex industrial environments, as described in claim 2, is characterized by: The operational status data collected by the industrial equipment is processed and then input into an expert knowledge base. The expert knowledge base outputs corresponding expert experience guidance through structured knowledge matching analysis. The specific process is as follows: First, structured feature vectors are extracted from the input data. Then, a similarity measurement is performed based on the correlation between the input status feature vectors and candidate expert experiences, thereby forming a probability distribution representing the applicability of expert experience. Then, the top k highly applicable expert experiences are selected in descending order of probability as the output results. Finally, by analyzing the key information such as equipment operation specifications, fault handling strategies, and process parameter suggestions contained in these Top-k expert experiences, direct decision-making basis is provided for tasks such as adjusting equipment operating parameters, diagnosing faults, and optimizing operating strategies in industrial scenarios, realizing the accurate reuse and scenario adaptation of expert experience.

4. The method for intelligent agent-machine mutually beneficial collaborative decision-making based on expert knowledge base and reinforcement learning in complex industrial environments, as described in claim 3, is characterized by: In the on-site human expert decision-making process, there are two typical mechanisms in traditional human-machine collaborative decision-making: one is that humans retain the final decision-making power, that is, after the intelligent agent provides decision suggestions, the human expert directly decides the final execution plan; the other is weighted fusion decision-making, which uses preset weights to weight and combine the human decision and the intelligent agent's decision results to generate the final decision. Considering the application characteristics of the industrial environment, a model of trusting human expert decision-making is adopted: human experts, based on professional knowledge, on-site experience, and a comprehensive judgment of the overall goal, directly determine the final operation mode of the equipment; upon receiving the intelligent agent's decision output and the expert experience output from the expert knowledge base, the on-site human expert will first compare the similarity between the two: If there is a decision conflict between the two or if the agent makes a decision error, human experts will intervene in real time. The human experts will first determine whether the agent's decision is erroneous. If the agent's decision is erroneous, the on-site experts will provide real-time expert decisions based on the real-time operating status of the equipment to control the equipment's operation. The human decision and environmental feedback will be pushed to the agent simultaneously for its online optimization. If the experience provided by the expert knowledge base is erroneous, the on-site experts will generate targeted feedback based on the current characteristics of the equipment's operating status and update the expert knowledge base to optimize historical experience. In addition, the system will force human experts to participate in decision evaluation every fixed period to avoid deviations in decision direction caused by dynamic changes in the environment or the agent's tendency towards local optima.

5. The method for intelligent agent-machine mutually beneficial collaborative decision-making based on expert knowledge base and reinforcement learning in a complex industrial environment, as described in claim 4, is characterized in that: During the optimization process of a reinforcement learning agent, if the environmental feedback received by the agent comes from expert experience, it is stored in a buffer similar to the regular RL experience buffer. Expert experience buffer zone for differentiated management Meanwhile, a priority experience replay mechanism is adopted when extracting experience samples. Through dynamic weighting, expert experience is given a higher replay priority than conventional experience to ensure that the agent learns on-site expert experience first. When the agent updates the policy network, expert guidance experience is integrated into the update process. By constructing a dynamic loss function and introducing an adaptive weight mechanism, the influence of expert experience on the policy network update is adjusted.

6. The method for intelligent agent-machine mutually beneficial collaborative decision-making based on expert knowledge base and reinforcement learning in complex industrial environments, as described in claim 5, is characterized by: The priority experience replay mechanism calculates the priority of expert experience samples using time-series difference error and Q-dominance function, employing the formula... Determine priorities, among which It is a small positive number to ensure that the priority is always positive and to avoid the zero-error experience being completely ignored, where the timing difference term The definition is as follows: in: This represents the immediate reward at time t, i.e., in state t. Execute action Rewards based on real-time environmental feedback; It is a discount factor with a value range of [0,1]. It is the state of the policy network. The action to choose.

7. The method for intelligent agent-machine mutually beneficial collaborative decision-making based on expert knowledge base and reinforcement learning in complex industrial environments, as described in claim 6, is characterized by: The policy network update process employs a human intervention fusion mechanism, achieving efficient collaboration between expert knowledge and reinforcement learning through the construction of a dynamic loss function, thus balancing human experience and autonomous learning. The specific definition of the dynamic loss function is as follows: The core parameter design reflects the dynamic balance mechanism of human-machine collaboration: human-machine intervention indicators. As a binary state variable, it takes a value of 1 when human intervention occurs and 0 otherwise, thus identifying scenarios where human decision-making is involved; the device ultimately executes the action. Defined as when humans participate When no intervention is needed, the agent's policy input is used. To balance human guidance and autonomous exploration, and to prevent the policy network from converging prematurely, adaptive weights are used. The time decay factor is dynamically adjusted, and its calculation formula is as follows: in: It is the time-series decay factor. k represents the number of training rounds, and the degree of dependence on human experience decreases as the strategy matures. To quantify the value advantage of human decision-making compared to AI decision-making, an exponential function is used to amplify the difference in advantages and a lower limit is set; Ensure that positive weights are assigned only when human actions are superior to strategy actions.

8. The method for intelligent agent-machine mutually beneficial collaborative decision-making based on expert knowledge base and reinforcement learning in complex industrial environments, as described in claim 7, is characterized by: In actual production, when on-site experts provide real-time expert experience, the expert knowledge base will receive and store that experience in real time. Considering that the expert knowledge base needs to support equipment operation in real time and cannot perform a full update operation during production, the system will select a specific time window to trigger the knowledge base update according to the equipment's preset rules. During the update, the newly received real-time expert experience will be merged with the existing experience in the historical expert experience base. A brand-new expert knowledge base containing real-time experience will be built through the full update process, thereby ensuring the real-time operation of the equipment while realizing incremental iterative optimization of expert knowledge.

Images