Man-machine collaborative reinforcement and prompting learning method, model and device for complex process control

Abstract

The application discloses a man-machine collaborative reinforcement and prompt learning method, model and equipment for complex process control, and the method comprises the following steps: constructing expert review data set in the field of complex process control, designing deep neural network reinforcement learning model integrating field knowledge, constructing man-machine collaborative reinforcement model based on prompt learning, etc. Through the multi-task prompt engineering, the specific prompt is generated, the modal transformation between the prompt text and the time sequence data is carried out, and the prompt optimization of the pre-training model is completed. The generated man-machine collaborative reinforcement model integrates the field knowledge, and the pre-training model parameters and model input are updated constantly, so that the most relevant knowledge in the field can be activated and acquired better, and the model generated instruction is more in line with the industrial mechanism.

Description

Technical Field

[0001] This invention relates to the field of intelligent collaborative control of complex dynamic systems, and more specifically, to a human-machine collaborative reinforcement and prompting learning method, model, and device for complex process control. Background Technology

[0002] Complex industrial processes are characterized by inter-process coupling, variable operating conditions, and time-varying equipment characteristics. As the complexity of system control objects, control objectives, and environments becomes increasingly prominent, the high dimensionality, nonlinearity, time-varying nature, uncertainty, and incompleteness of systems make it difficult to mathematically model the control objects. The multi-level information structure makes it difficult to coordinate reliable system operation with high-performance collaborative control, posing a significant challenge to system control theories and methods based on mathematical models.

[0003] Generative artificial intelligence possesses powerful knowledge encoding and complex task reasoning capabilities. Multimodal pre-trained models integrate multi-level information and domain knowledge, enabling better representation, modeling, and prediction of complex dynamic systems. However, generative pre-trained models embed general domain knowledge, which, while broad, is not precise. By introducing relevant domain-specific knowledge through reinforcement learning from human feedback (RLHF), and establishing human-machine collaborative reinforcement and prompting learning methods that integrate domain knowledge, we can explore the model integration mechanisms and methods for domain-specific knowledge, update model parameters and inputs, and better activate and acquire the most relevant knowledge in the domain, making the model's generated instructions more consistent with industrial mechanisms. However, different downstream tasks involve different domain knowledge and specific task requirements, and currently, there is a lack of a multimodal generative pre-trained collaborative control model that can be further optimized through prompting engineering. Summary of the Invention

[0004] To address the aforementioned shortcomings of existing technologies, this invention provides a human-machine collaborative reinforcement and prompting learning method, model, and device for complex process control. Through multi-task prompting engineering, specific prompts are generated, and the prompting optimization of the pre-trained model is achieved through modal transformation between prompt text and time-series data. Simultaneously, the human-machine collaborative reinforcement model integrates domain knowledge, continuously updating the pre-trained model parameters and model inputs to better activate and acquire the most relevant domain knowledge, resulting in model-generated instructions that are more consistent with industrial mechanisms.

[0005] To achieve the above objectives, on the one hand, the present invention provides a human-machine collaborative reinforcement and prompting learning method for complex process control, characterized by comprising the following steps:

[0006] Step S1: Construct an expert review dataset in the field of complex process control: Integrate experience data and audit data from multiple fields and levels to establish an expert knowledge base covering cross-domain and cross-business areas;

[0007] Step S2: Design a deep neural network reinforcement learning model that integrates domain knowledge: realize reinforcement through human feedback, utilize domain expert knowledge to guide the model's decision-making and learning process, and adjust model parameters to obtain a multimodal generative pre-trained model.

[0008] Step S3: Construct a human-computer collaborative reinforcement model based on cue learning: Introduce cue learning to enable the human-computer collaborative reinforcement model to interact with human experts, select appropriate cue types, design suitable text cue and effective cue strategies, integrate multiple cue inputs into the model, guide the model's output, and maximize the model's performance and learning effect.

[0009] Furthermore, in step S1, the expert review dataset spans different levels and business domains. By comprehensively understanding the system's operating mechanism, an expert knowledge base is constructed, which includes domain expert knowledge, problem descriptions, example inputs and outputs, or other forms of instruction signals, to support model decision-making and reasoning.

[0010] Furthermore, in step 2, the process of designing a deep neural network reinforcement learning model that integrates domain knowledge is as follows:

[0011] Step S21: Construct a pre-trained network using a proximal optimization strategy reinforcement learning algorithm, and design the reward model structure within the reinforcement learning framework:

[0012] The reward model process is represented as follows:

[0013] R(s, a, s′)

[0014] In the formula, s represents the current state, a is the action chosen by the agent, and s′ is the next state transitioned to after performing action a; the function R represents the reward obtained by transitioning to state s′ after choosing action a in state s; the reward can be positive, negative, or zero, used to represent the goodness or badness of the behavior or the punishment for the agent; the expected cumulative reward is expressed as:

[0015]

[0016] In the formula, E represents the expected value, t represents the time step, and γ is the discount factor used to adjust the importance of future rewards;

[0017] Step S22: Using the expert review dataset as the input carrier, human expert knowledge effectively interacts with the reward model and provides real-time feedback, thereby training a reward model that integrates expert knowledge.

[0018] Step S23: Based on the evaluation and feedback of the reward model on the generative output of the pre-trained model, fine-tune its internal network parameters.

[0019] The final reward for updating the rules using its pre-trained model is represented as:

[0020] R t (θ)=r θ (y|x)-λD KL (π PPO (y|x)||π base (y|x))

[0021] In the formula, x and y are the input and output of the reinforcement model, respectively, and r θ (y|x) represents the evaluation of the reward model on the reinforcement model, D KL (π PPO (y|x)||π base (y|x)) represents the Kullback-Leibler divergence between the enhanced model output and the initial model output;

[0022] Step S24: After collecting the final reward value, the reinforcement learning update algorithm (PPO) calculates the advantage function for each time step. The objective function is obtained, and then the gradient of the objective function with respect to the policy parameters is estimated using the gradient ascent method. The policy parameters are then updated, and the above steps are repeated until the predetermined number of iterations is reached or the convergence condition is met.

[0023] The objective function maximized at each policy update is expressed as:

[0024]

[0025] In the formula, θ is the parameter of the current policy. old These are the strategy parameters from the last update. It is the advantage function, representing the state s. t Select action a t Advantage relative to expected value is used to measure the quality of an action; π θ (a t |s t ) represents the probability of the action under the current policy. It represents the action probability of the policy at the time of the last update. KL divergence is used as a loss function to measure the difference between the two probability distributions.

[0026] The advantage function is expressed as follows:

[0027]

[0028] In the formula, Q(s) t ,a t V(s) is the action-value function of the state-action pair. t ) is a state-value function;

[0029] The update rule for the reinforcement learning algorithm model is as follows:

[0030] Furthermore, in step S3, the process of constructing a human-machine collaborative reinforcement model based on cue learning is as follows:

[0031] Step S31: Collect cue data related to the target task from the multimodal generative pre-trained model, including problem description, example input and output, domain expert knowledge or other forms of guidance signals, to ensure that the cue data is closely related to the target task and has high quality; Step S32: Select appropriate cue types and design effective cue strategies to integrate into the generative model to obtain the model output, and fine-tune and adapt the model for multi-task domain.

[0032] The process of prompting learning is defined as follows:

[0033] x′=[X][Z]=f prompt (x)

[0034] In the formula, the hint function f prompt Based on the original input x, a prompt x′ is generated. [X] and [Z] are the two components of x′, which are the original input and the prompt word, and the output to be filled by the model, respectively.

[0035] Hints will help you learn the appropriate hints that will eventually be generated. for:

[0036]

[0037] In the formula, P θ (z * (x′) is the correct output z that the model gives based on the prompt x′. * The probability of.

[0038] Furthermore, in step S31, the type of the target task includes state estimation and prediction tasks and fault diagnosis / quality inspection tasks.

[0039] Furthermore, in step S4, the method for designing and generating prompts includes multi-prompt integration, imitation learning, sub-prompt combination, and sub-prompt decomposition.

[0040] Furthermore, the method for designing and generating prompts includes extracting template prompts, language prompts, attribute prompts, reliability prompts, etc. from expert experience, image semantics, and signal language, and integrating multiple prompts as the final prompt signal input into the model to manipulate the model's behavior, assist in completing various specific problems in industrial scenarios, and maximize the performance of the trained model.

[0041] On the other hand, the present invention provides a human-machine collaborative reinforcement model for complex process control, characterized in that it is a generative model constructed by the human-machine collaborative reinforcement and prompting learning method described above.

[0042] Finally, the present invention provides an electronic device characterized by having a memory storing executable program code; a processor coupled to the memory; the processor calling the executable program code stored in the memory to execute the human-machine collaborative reinforcement and prompting learning method for complex process control as described above.

[0043] Compared with the prior art, the present invention has the following technical effects:

[0044] (1) This invention utilizes the powerful knowledge encoding and complex task reasoning capabilities of generative artificial intelligence to meet the requirements of multi-task collaborative control for complex industrial processes.

[0045] (2) In industrial scenarios, this invention introduces relevant domain-specific knowledge through reinforcement learning, corrects model results, optimizes model parameters, describes system structural features, improves model interpretability, better activates and acquires the most relevant knowledge in the field, and makes the model generation instructions more in line with industrial mechanisms.

[0046] (3) This invention addresses the different domain knowledge and specific task requirements involved in different downstream tasks by further optimizing the generative pre-trained collaborative control model through prompting engineering, thereby improving the efficiency of the pre-trained model in completing downstream tasks. Attached Figure Description

[0047] Figure 1 This is a flowchart of a human-computer collaborative reinforcement and prompting learning method in one embodiment of the present invention.

[0048] Figure 2 This is a diagram illustrating the architecture of an expert review dataset in one embodiment of the present invention.

[0049] Figure 3 This is a flowchart illustrating the application of a human-machine collaborative reinforcement model based on prompting learning in a copper smelting workshop, according to one embodiment of the present invention. Detailed Implementation

[0050] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but these are not intended to limit the scope of the invention.

[0051] In the following detailed description, numerous specific details are set forth to provide a more thorough understanding of the invention. However, it will be apparent to those skilled in the art that well-known algorithms and models (such as gradient ascent, pre-trained networks) are not shown in detail to avoid obscuring the gist of the invention.

[0052] Furthermore, the execution order of actions, steps, etc. in the apparatus and methods shown in the claims, specification, and drawings can be implemented in any order, unless a specific order is explicitly specified, and as long as the output of the preceding processing is not used in the subsequent processing.

[0053] Example

[0054] See Figure 1 This embodiment provides a human-machine collaborative reinforcement and prompting learning method for complex process control, targeting complex industrial process control, such as multi-furnace collaborative control scenarios involving flash furnaces, anode furnaces, converters, blowers, and converter blowers in copper smelting, including the following steps:

[0055] Step S1: Construct an expert review dataset for complex process control: Integrate experience data and audit data from multiple fields and levels to establish an expert knowledge base covering cross-domain and cross-business operations. For a concrete, implementable example, see [link to relevant documentation]. Figure 2 The expert review dataset spans different levels and business domains. By comprehensively understanding the operating mechanism of the process system, it constructs an expert knowledge base containing domain expert knowledge, problem descriptions, example inputs and outputs, or other forms of instruction signals to support model decision-making and reasoning.

[0056] Step S2: Design a deep neural network reinforcement learning model that integrates domain knowledge: Implement reinforcement through human feedback, utilize domain expert knowledge to guide the model's decision-making and learning process, and adjust model parameters to obtain a multimodal generative pre-trained model. The expert review dataset is embedded into the human feedback reinforcement learning, and the expert knowledge base provides guidance and direction for the reinforcement learning. As a concrete example of implementation, the process of designing a deep neural network reinforcement learning model that integrates domain knowledge is as follows:

[0057] Step S21: Construct a pre-trained network using a proximal optimization strategy reinforcement learning algorithm, and design the reward model structure within the reinforcement learning framework:

[0058] The reward model process is represented as follows:

[0059] R(s,a,s ′ )

[0060] In the formula, s represents the current state, a is the action chosen by the agent, and s ′ It represents the next state after performing action 'a'; function R represents the transition to state s after selecting action 'a' in state s. ′ The reward obtained; the reward can be positive, negative, or zero, used to represent the goodness or badness of the behavior or the punishment for the agent; the agent learns a policy, that is, a mapping from state to action, to maximize the reward accumulated over time, usually expressed as the expected cumulative reward:

[0061]

[0062] In the formula, E represents the expected value, t represents the time step, and γ is the discount factor used to adjust the importance of future rewards;

[0063] Step S22: Using the expert review dataset as the input carrier, human expert knowledge effectively interacts with the reward model and provides real-time feedback, thereby training a reward model that integrates expert knowledge.

[0064] Step S23: Freeze the pre-trained network to form parameters. The frozen network is used as the initial model. The parameters of the first few layers of the pre-trained model are fixed, and the parameters of other parts of the network are fine-tuned based on the evaluation and feedback of the generative output of the pre-trained model by the reward model.

[0065] The final reward for updating the rules using its pre-trained model is represented as:

[0066] R t (θ)=r θ (y∣x)-λD KL (π PPO (y∣x)∣∣π base (y∣x))

[0067] In the formula, x and y are the input and output of the reinforcement model, respectively, and r θ (y|x) represents the reward model's evaluation of the reinforcement model, D KL (π PPO (y∣x)∣∣π base (y|x)) represents the Kullback–Leibler divergence between the enhanced model output and the initial model output;

[0068] Step S24: After collecting the final reward value, the reinforcement learning update algorithm (PPO) calculates the advantage function for each time step. The objective function is obtained, and then the gradient of the objective function with respect to the policy parameters is estimated using the gradient ascent method. The policy parameters are then updated, and the above steps are repeated until the predetermined number of iterations is reached or the convergence condition is met.

[0069] The objective function maximized at each policy update is expressed as:

[0070]

[0071] In the formula, θ is the parameter of the current policy. old These are the strategy parameters from the last update. It is the advantage function, representing the state s. t Select action a tAdvantage relative to expected value is used to measure the quality of an action; π θ (a t |s t ) represents the probability of the action under the current policy. It represents the action probability of the policy at the time of the last update. KL divergence is used as a loss function to measure the difference between the two probability distributions.

[0072] The advantage function is expressed as follows:

[0073]

[0074] In the formula, Q(s) t ,a t V(s) is the action-value function of the state-action pair. t ) is a state-value function;

[0075] The update rule for the reinforcement learning algorithm model is as follows:

[0076] Step S3: Construct a human-computer collaborative reinforcement model based on cue learning: Introduce cue learning to enable the human-computer collaborative reinforcement model to interact with human experts, select appropriate cue types, design suitable text cue and effective cue strategies, integrate multiple cue inputs into the model, guide the model output, and maximize the model's performance and learning effect.

[0077] The process of the human-computer collaborative reinforcement model based on prompting learning is as follows:

[0078] 1. Collect cue data related to the target task from multimodal generative pre-trained models. This includes problem descriptions, example inputs and outputs, domain expert knowledge, or other forms of guidance signals. Ensure that the cue data is closely related to the target task and is of high quality. Target task types include state estimation and prediction tasks, fault diagnosis / quality inspection, etc.

[0079] 2. Selecting appropriate cue types and designing effective cue strategies to integrate into the generative model yields the model output, enabling fine-tuning and adaptation for multi-task tasks within the domain. Cue design and generation methods include multi-cue ensemble, imitation learning, sub-cue combination, and sub-cue decomposition. For example, template cues, linguistic cues, attribute cues, and reliability cues are extracted from expert experience, image semantics, and signal language. These multiple cues are then integrated as the final cue signal input into the model to manipulate its behavior, assisting in solving various specific problems in industrial scenarios and maximizing the performance of the trained model. The cue learning process is defined as follows:

[0080] x′=[X][Z]=f prompt (x)

[0081] In the formula, the hint function f prompt Based on the original input x, a prompt x′ is generated. [X] and [Z] are the two components of x′, which are the original input and the prompt word, and the output to be filled by the model, respectively.

[0082] Hints will help you learn the appropriate hints that will eventually be generated. for:

[0083]

[0084] In the formula, P θ (z * (x′) is the correct output z that the model gives based on the prompt x′. * The probability of.

[0085] Figure 3 This demonstration showcases the application of a human-machine collaborative reinforcement model based on cueing learning in a copper smelting workshop. To diagnose the operational status of equipment inside a smelting furnace, the process begins with establishing a problem description, including normal operation, anomaly warnings, and fault classification. Multiple sensor data points are collected, including temperature, pressure, and flow rate. This is combined with domain expert knowledge, including the normal operating range of the equipment, typical fault models, fault causes and handling suggestions, and other guidance signals, such as operation manuals, maintenance records, images, and video data, to comprehensively gather cueing data. Then, cueing strategies are designed, including verbal cues, attribute cues, saliency cues, and reliability cues. These cues are integrated into a generative model, including text, images, and signals, to perform state estimation and fault diagnosis for the oxygen compressor module and converter blower module within the copper smelting workshop. Finally, the model is fine-tuned based on the actual conditions of the workshop to improve diagnostic accuracy, and the cueing design and model parameters are continuously optimized based on feedback.

[0086] If the aforementioned human-machine collaborative reinforcement and prompting learning method for complex process control is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Therefore, the essence of this technical solution, or the part that contributes to the prior art, or a portion of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0087] In summary, this invention provides a method, model, and device for human-machine collaborative reinforcement and prompting learning in complex process control. The method includes steps such as constructing an expert review dataset in the field of complex process control, designing a deep neural network reinforcement learning model that integrates domain knowledge, and constructing a human-machine collaborative reinforcement model based on prompting learning. This invention generates specific prompts through multi-task prompting engineering and optimizes the prompting of the pre-trained model through modal transformation between prompt text and time-series data. The generated human-machine collaborative reinforcement model integrates domain knowledge and continuously updates the parameters and inputs of the pre-trained model to better activate and acquire the most relevant domain knowledge, making the model's generated instructions more consistent with industrial mechanisms.

[0088] Those skilled in the art should understand that variations can be implemented by combining existing technology with the above embodiments, which will not be elaborated here. Such variations do not affect the essence of the present invention, and will not be elaborated here either.

[0089] The preferred embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and the devices and structures not described in detail should be understood as being implemented in a conventional manner in the art. Any person skilled in the art can make many possible variations and modifications to the technical solutions of the present invention using the methods and techniques disclosed above, or modify them into equivalent embodiments with equivalent changes, without departing from the scope of the present invention. This does not affect the essential content of the present invention. Therefore, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the present invention's technical solutions still fall within the protection scope of the present invention.

[0090] Those skilled in the art will recognize that the units, i.e., algorithm steps, of the various examples described in connection with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

Claims

1. A human-machine collaborative reinforcement and prompting learning method for complex process control, characterized by, Includes the following steps: Step S1: Construct an expert review dataset in the field of complex process control: Integrate experience data and audit data from multiple fields and levels to establish an expert knowledge base covering cross-domain and cross-business areas; Step S2: Design a deep neural network reinforcement learning model that integrates domain knowledge: realize reinforcement through human feedback, utilize domain expert knowledge to guide the model's decision-making and learning process, and adjust model parameters to obtain a multimodal generative pre-trained model. Step S3: Construct a human-computer collaborative reinforcement model based on cue learning: Introduce cue learning to enable the human-computer collaborative reinforcement model to interact with human experts, select appropriate cue types, design suitable text cue and effective cue strategies, integrate multiple cue into the human-computer collaborative reinforcement model, guide the model's output, and give full play to the model's performance and learning effect; In step 2, the process of designing a deep neural network reinforcement learning model that integrates domain knowledge is as follows: Step S21: Construct a pre-trained network using a proximal optimization strategy reinforcement learning algorithm, and design the reward model structure within the reinforcement learning framework: The reward model process is represented as follows: In the formula, Indicates the current state. It is the action chosen by the agent. It is to perform an action The function transitions to the next state. Indicates the state Select Action Then, transition to state The reward obtained; the reward can be positive, negative, or zero, used to represent the goodness or badness of the behavior or the punishment for the agent; the expected cumulative reward is expressed as: In the formula, Indicates the expected value. Indicates a time step. It is a discount factor used to adjust the importance of future rewards; Step S22: Using the expert review dataset as the input carrier, human expert knowledge effectively interacts with the reward model and provides real-time feedback, thereby training a reward model that integrates expert knowledge. Step S23: Based on the evaluation and feedback of the pre-trained model's generative output by the reward model, fine-tune its internal network parameters; The final reward for updating the rules using its pre-trained model is represented as: In the formula, y and y are the input and output of the reinforcement model, respectively. To reward the evaluation of the reinforcement model, To enhance the Kullback–Leibler divergence between the model output and the initial model output; Step S24: After collecting the final reward value, the reinforcement learning update algorithm calculates the advantage function for each time step. The objective function is obtained, and then the gradient of the objective function with respect to the policy parameters is estimated using the gradient ascent method. The policy parameters are then updated, and the above steps are repeated until the predetermined number of iterations is reached or the convergence condition is met. The objective function maximized at each policy update is expressed as: In the formula, These are the parameters of the current strategy. These are the strategy parameters from the last update. It is the dominance function, representing the state... Select Action The advantage relative to the expected value is used to measure the quality of an action. It is the probability of the action of the current strategy. It represents the action probability of the policy at the time of the last update. KL divergence is used as a loss function to measure the difference between the two probability distributions. The advantage function is expressed as follows: In the formula, It is the action-value function of the state-action pair. It is a state-value function; The update rule for the reinforcement learning algorithm model is as follows: .

2. The human-machine collaborative reinforcement and prompting learning method for complex process control according to claim 1, characterized in that, In step S1, the expert review dataset spans different levels and business domains. By comprehensively understanding the operating mechanism of the process system, an expert knowledge base is constructed, which includes domain expert knowledge, problem descriptions, example input and output instruction signals, to support model decision-making and reasoning.

3. The human-machine collaborative reinforcement and prompting learning method for complex process control according to claim 1, characterized in that, In step S3, the process of constructing a human-computer collaborative reinforcement model based on cue learning is as follows: Step S31: Collect cue data related to the target task from the multimodal generative pre-trained model, including problem description, example input and output, and guidance signals from domain expert knowledge, to ensure that the cue data is closely related to the target task and has high quality; Step S32: Select appropriate prompt types and design effective prompt strategies to integrate into the generative model to obtain model output, which is used for fine-tuning and adapting to multi-task tasks in the domain. The process of prompting learning is defined as follows: In the formula, the hint function Generate hints based on the original input. , and for The two components are the original input and prompt words, and the output to be filled by the model. Hints will help you learn the appropriate hints that will eventually be generated. for: In the formula, The model is based on the prompts Give the correct output The probability of.

4. The human-machine collaborative reinforcement and prompting learning method for complex process control according to claim 3, characterized in that, In step S31, the type of the target task includes state estimation and prediction tasks and fault diagnosis / quality inspection tasks.

5. The human-machine collaborative reinforcement and prompting learning method for complex process control according to claim 3, characterized in that, In step S4, the methods for designing and generating prompts include multi-prompt integration, imitation learning, sub-prompt combination, and sub-prompt decomposition.

6. The human-machine collaborative reinforcement and prompting learning method for complex process control according to claim 5, characterized in that, The method for designing and generating prompts includes extracting template prompts, language prompts, attribute prompts, and reliability prompts from expert experience, image semantics, and signal language, and integrating multiple prompts as the final prompt signal input into the model to manipulate the model's behavior, assist in solving various specific problems in industrial scenarios, and maximize the performance of the trained model.

7. An electronic device, characterized in that, A memory storing executable program code; a processor coupled to the memory; the processor calling the executable program code stored in the memory to execute the human-machine collaborative reinforcement and prompting learning method for complex process control as described in any one of claims 1 to 6.

Images