Multi-task multi-objective collaborative optimization and regulation robot system for pharmaceutical process

Abstract

The present application relates to the technical field of intelligent control of pharmaceutical process, and particularly relates to a multi-task and multi-target collaborative optimization robot system for a pharmaceutical process; comprising an execution agent module, an inference agent module and a self-decision module, the execution agent module comprising a data grabbing unit and a data preprocessing unit; the inference agent module comprising a model training unit and an inference unit; the self-decision module comprising a decision unit and an execution unit; the present application greatly improves the multi-task and multi-target automatic monitoring and intelligent optimization level of the traditional Chinese medicine pharmaceutical process by fusing RPA, MODPO and RL technologies, and realizes efficient data acquisition, accurate parameter optimization and intelligent feedback control.

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology for pharmaceutical processes, and in particular to a multi-task, multi-objective collaborative optimization and control robot system for pharmaceutical processes. Background Technology

[0002] In the field of traditional Chinese medicine (TCM) pharmaceutical manufacturing, the long-standing reliance on manual experience and traditional knowledge in process decision-making has significantly limited the industry's scientific and modern development. Driven by digital transformation and intelligent manufacturing, the TCM pharmaceutical industry is facing a significant opportunity for transformation and upgrading. The industry needs to accelerate its digital transformation and intelligent manufacturing to promote innovative development. TCM pharmaceutical companies need to explore new digital and intelligent manufacturing models.

[0003] During this transformation, RPA (Robotic Process Automation) technology has been widely applied in manufacturing processes due to its high efficiency, accuracy, and reliability. RPA can simulate human operations on computers, thereby automating decision-making and optimizing processes, significantly improving the efficiency and quality consistency of traditional Chinese medicine (TCM) manufacturing. Specifically, after introducing RPA technology, TCM pharmaceutical companies can automatically monitor and manage all aspects of drug production and quality control. However, despite its excellent performance in automated execution and information processing, RPA's functionality remains limited to preset rules and procedures, lacking the ability to handle complex decisions and adaptive adjustments.

[0004] To overcome these limitations and further improve automation levels, combining RPA, multi-objective direct preference optimization (MODPO), and reinforcement learning (RL) techniques has become a new direction for intelligent optimization of traditional Chinese medicine manufacturing processes. RL, through the training of a single agent, can not only effectively monitor and optimize complex production processes but also make intelligent decisions based on real-time data and environmental changes. This combination of technologies can not only rapidly process and analyze large amounts of data, providing scientific and accurate decision support, but also automate production processes, reduce human intervention and errors, thereby improving the consistency of drug quality.

[0005] Therefore, there is an urgent need to provide a multi-task, multi-objective collaborative optimization and control robot system for pharmaceutical processes. Compared with existing technologies, by integrating RPA, MODPO, and RL technologies, the system can greatly improve the level of automated monitoring and intelligent optimization of multi-task, multi-objective processes in traditional Chinese medicine manufacturing, and achieve efficient data acquisition, accurate parameter optimization, and intelligent feedback control. Summary of the Invention

[0006] This invention addresses the technical problems existing in the prior art and provides a multi-task, multi-objective collaborative optimization and control robot system for pharmaceutical processes.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] A multi-task, multi-objective collaborative optimization and control robot system for pharmaceutical processes includes an execution agent module, an inference agent module, and a self-decision-making module. The execution agent module includes a data acquisition unit and a data preprocessing unit; the inference agent module includes a model training unit and an inference unit; and the self-decision-making module includes a decision-making unit and an execution unit.

[0009] The data capture unit captures historical production data; the data preprocessing unit performs data cleaning, standardization, deduplication, and format conversion operations on the historical production data captured by the data capture unit.

[0010] The model training unit constructs a Markov decision process training model, and obtains the process model corresponding to each historical production data by training the Markov decision process training model.

[0011] The reasoning unit acquires real-time production data and inputs it into the corresponding process model to obtain optimized production data. Based on the optimized production data, multiple adjustment schemes are generated.

[0012] The decision-making unit uses the direct preference optimization algorithm to unify the multiple adjustment schemes generated by the reasoning unit to obtain a decision adjustment scheme;

[0013] The execution unit is used to control the production system to execute decision adjustment plans.

[0014] Furthermore, the Markov decision process training model includes five elements: agent, environment, policy, state, action, and reward, which are constructed through the following steps:

[0015] S101. Define a policy network to output action a given state s;

[0016] S102. Define the target policy network to calculate the target action a′;

[0017] S103. Define the Q-network to output the Q-value and optimize the policy network;

[0018] S104. Define the target Q-network, which is used to calculate the target Q-value and to optimize the Q-network.

[0019] Furthermore, the process model for each historical production data point is obtained through the following steps:

[0020] S201. Take one type of historical production data from the historical production data processed by the data preprocessing unit as the initial state s. tt represents the time for fetching historical production data, and the policy network is based on the initial state s. t Make the initial action a t ;

[0021] S202, The environment is based on the initial action a t For the initial state s t Make changes to obtain the state s at the next moment. t+1 and provide feedback rewards r to the agent. t ;

[0022] S203, Iterate through steps S201-S202 until the agent performs the target action a. ′ Stop iterating when the time comes;

[0023] S204. Set up a buffer. Store the data from each iteration in S203 as a sample in the buffer. Each sample contains the data generated in one iteration. t a t r t s t+1 ;

[0024] S205. Update the Q network set in step S103;

[0025] S206. Update the policy network set in step S101;

[0026] S207. Update the target policy network set in step S102;

[0027] S208. Perform iterative training, iterating through steps S201-S207 until the preset number of iterations is reached, to obtain a process model based on historical production data.

[0028] S209. Using the methods in steps S201-S208, obtain the process model for the next type of historical production data;

[0029] S2010. Repeat steps S201-S209 until the process model for all historical production data has been generated.

[0030] Furthermore, in step S201, the agent will, based on the state, perform an action a. t Generate multiple candidate actions Specifically:

[0031]

[0032] In the above formula, ∈ i This represents random noise drawn from a Gaussian or uniform distribution; expressed through Q... i The network calculates the Q-values ​​of the generated candidate actions separately, and then selects the candidate actions... and state s t Enter Q i Network obtained The corresponding Q value: Then calculate the probability P of each action according to the following formula:

[0033]

[0034] In the above formula, Q represents the Q-network, a represents the action, and s represents the state;

[0035] The candidate action with the highest probability is selected as the final action taken by the agent.

[0036] Furthermore, S205 specifically includes the following steps:

[0037] S2051. Extract a set number of samples from the buffer and input them into the Q network set in step S103 to calculate the Q value of each sample.

[0038] S2052. Calculate the target Q value for each sample using the target policy network set in step S102 and the target Q network set in step S104. For each sample, if state s t+1 If the state is terminated, then the target Q value of this sample is r. t Conversely, the target Q value of this sample is r. t +γ×Q_target(s t+1 ,π_target(s t+1 ), where γ is the discount factor, Q_target represents the output of the target Q network, and π_target represents the output of the target policy network;

[0039] S2053. Using the mean squared error loss function, calculate the square of the average difference between the Q values ​​of all extracted samples and the target Q value:

[0040]

[0041] In the above formula, n represents the set value. Represents the loss function. The parameters represent the loss function. Q_target represents the current output of the Q network. i (s t,j ,a t,j ) represents the target Q value;

[0042] S2054. Based on the squared value of the average difference, update the Q-network set in step S103 using the gradient descent algorithm. The specific update process is as follows:

[0043] For loss function Regarding parameters Take the derivative to obtain the gradient.

[0044]

[0045] This represents the Q-value of the j-th sample with respect to the parameter. The gradient is used to update the parameters at each step of gradient descent using the following formula:

[0046]

[0047] In the above formula, α is the learning rate.

[0048] Furthermore, S206 specifically includes the following steps:

[0049] S2061. Calculate the gradient of the policy network set in step S101 using the chain rule. The specific calculation formula is as follows:

[0050]

[0051] In the above formula, The parameters of the policy network are represented by , which are optimized using gradient ascent or descent methods. J represents the objective function, i.e., the expected cumulative reward, which is maximized by adjusting the parameters of the policy network. This represents the gradient of the objective function with respect to the policy network parameters. This represents the gradient of the Q-function with respect to action a. Indicates the policy π relative to the parameters The gradient of E is the expected function.

[0052] S2062. Based on the gradient calculated in step S2061, the policy network set in step S101 is updated using the backpropagation algorithm and gradient descent algorithm. The specific update process is as follows:

[0053] Calculate the policy gradient using the chain rule:

[0054]

[0055] Q-value pairs of actions The partial derivatives, The policy network output is related to its parameters. The partial derivatives, where D represents the state sampling space;

[0056] Using the calculated policy gradient, the parameters of the policy network are expressed by the following formula. Perform gradient ascent update:

[0057]

[0058] Furthermore, in step S207, the target policy network is updated by updating its parameters. The parameters of the target policy network are calculated and updated using the following formula:

[0059] θ_target=τθ+(1-τ)θ_target

[0060] In the above formula, θ_target represents the parameters of the target policy network, τ is a constant, and θ represents the parameters of the current target policy network.

[0061] Furthermore, the direct preference optimization algorithm in the decision-making unit is expressed by the following formula:

[0062]

[0063] In the above formula, L MODPO Let y represent the optimization function, x represent the process parameters in the production data, and the process parameters x contain multiple objectives. w The decision adjustment scheme representing relative preference, y l This indicates a relatively unfavorable decision-making adjustment plan. The probability distribution of the response y to process parameter x; r (φ,-k) ω represents the reward score for the response y with multiple parameters x, ω represents the weight between different objectives of process parameter x, and D k Let σ represent the dataset, σ represent the sigmoid function used to map the input to the (0,1) interval, β represent the temperature parameter controlling the intensity of the KL penalty, and π represent the sigmoid function used to map the input to the (0,1) interval. sft This indicates supervised fine-tuning of the language model. Let ω represent the target weight vector excluding the k-th target, T represents the matrix transpose, and k represents the current target index.

[0064] Furthermore, the data capture unit captures historical production data from the Enterprise Resource Planning System, Manufacturing Execution System, and Laboratory Information Management System. The historical production data includes, but is not limited to, product batch information, process parameter settings, and operating status of production equipment.

[0065] Furthermore, the self-decision-making module also includes an alarm unit, and the execution unit is also used to acquire real-time production data of the production site after the execution decision adjustment plan. The alarm unit has an alarm threshold for each production data. When real-time production data exceeds its corresponding alarm threshold, the alarm unit will issue an alarm.

[0066] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0067] (1) This invention integrates RPA, reinforcement learning and deep learning technologies to realize multi-task and multi-objective collaborative intelligence, automation and lean manufacturing of traditional Chinese medicine manufacturing process. It has significant effects on improving production efficiency, reducing resource consumption and production defects and has promotion and application value. It not only realizes multi-task and multi-objective collaborative self-decision-making in traditional Chinese medicine manufacturing process, but also ensures the integrity of existing production control system. It is extremely convenient to deploy and use and has broad application prospects.

[0068] (2) This invention realizes non-intrusive and automated data collection and preprocessing from multiple production systems and databases, providing comprehensive and accurate data support for process optimization; it utilizes deep reinforcement learning algorithms to perform in-depth data mining and optimization analysis, discovering key influencing factors and potential for process optimization; it uses code generation to construct a large model of the process, and combines multi-objective reinforcement learning to adaptively optimize process parameters, achieving dual optimization of process efficiency and product quality; it realizes automated execution and real-time closed-loop control of the process, reducing manual intervention. Attached Figure Description

[0069] Figure 1 This is a schematic diagram of the system of the present invention. Detailed Implementation

[0070] The technical solution of the present invention will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are not all embodiments of the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0071] like Figure 1 As shown, the present invention provides a multi-task, multi-objective collaborative optimization and control robot system for pharmaceutical processes, including an execution agent module, an inference agent module, and a self-decision-making module; the execution agent module includes a data acquisition unit and a data preprocessing unit; the inference agent module includes a model training unit and an inference unit; and the self-decision-making module includes a decision-making unit, an execution unit, and an alarm unit.

[0072] The data capture unit deploys corresponding RPA to capture historical production data from different systems such as Enterprise Resource Planning (ERP), Manufacturing Execution System (MES), and Laboratory Information Management System (LIMS) by simulating manual operation. Historical production data includes, but is not limited to, product batch information, process parameter settings, and operating status of production equipment.

[0073] The data preprocessing unit is used to perform preprocessing work such as data cleaning, standardization, deduplication, and format conversion on the historical production data captured by the data crawling unit, in preparation for subsequent data modeling.

[0074] The model training unit is used to train the process model. Utilizing preprocessed historical production data and combined with the reinforcement learning algorithm DDPG, it simulates numerous production scenarios to train a process model that accurately describes and simulates the pharmaceutical manufacturing process of traditional Chinese medicine. This process model integrates massive amounts of data from traditional Chinese medicine manufacturing and can effectively capture the mapping relationship between process parameters and product quality and production efficiency. The specific training method is as follows:

[0075] S1. Construct a Markov Decision Process (MDP) training model using formal methods. The MDP training model includes five elements: agent, environment, policy, state, action, and reward. The MDP training model is essentially an interaction model between the agent and the environment through policy, state, action, and reward. Specifically:

[0076] S101. Define a policy network, denoted as π(s,a), and initialize a deep neural network. This initialized deep neural network corresponds to the policy network structure and is used to output the action a given state s.

[0077] S102. Define the target policy network and initialize a deep neural network with the same structure as the defined target policy network to calculate the target action a. ′ .

[0078] S103. Define the Q-network, denoted as Q. π (s,a) initializes a deep neural network, which corresponds to the defined Q-network structure and is used to output the Q-value and optimize the policy network.

[0079] S104. Define the target Q-network, denoted as... Initialize a deep neural network with the same structure as the defined target Q-network to calculate the target Q-value and optimize the Q-network.

[0080] S2. Input the different historical production data processed by the data preprocessing unit into the Markov decision process training model to form a process model corresponding to each historical production data, specifically:

[0081] S201. Take one of the preprocessed historical production data as the initial state s. t t represents the time when the historical production data was captured, and the policy network is based on the initial state s. t Make the initial action a t .

[0082] Among them, the agent will determine the initial action a based on the state. t Generate multiple candidate actions Specifically:

[0083]

[0084] In the above formula, ∈ i This represents random noise drawn from a Gaussian or uniform distribution. The final action is selected by considering the exploration noise, using Q... i The network calculates the Q-values ​​of the generated candidate actions separately. i The network accepts states and actions as input and outputs a scalar value representing the expected long-term reward of performing the action in that state. Given multiple candidate actions, the network first... and state s t Enter Q i Network obtained The corresponding Q value: Then calculate the probability P of each candidate action according to the following formula:

[0085]

[0086] The candidate action with the highest probability is selected as the initial action 'a' that the agent ultimately makes. t .

[0087] S202, The environment is based on the initial action a t The initial state s t Make changes to obtain the state s at the next moment. t+1 and provide feedback rewards r to the agent. t .

[0088] S203, Iterate through steps S201-S202 until the agent performs the target action a. ′ Stop iterating when the time is right.

[0089] S204. Set up a buffer. Store the data from each iteration in S203 as a sample in the buffer. One sample includes s... t a t r t s t+1 .

[0090] S205. Update the Q network set in step S103, specifically as follows:

[0091] S2051. Extract a set number of samples from the buffer and input them into the Q network set in step S103. Calculate the Q value of each sample. The set value is less than half the number of samples in the buffer.

[0092] S2052. Calculate the target Q value for each sample using the target policy network set in step S102 and the target Q network set in step S104. For each sample (s) t a tr t s t+1 If state s t+1 If the state is terminated, then the target Q value of this sample is r. t Conversely, the target Q value of this sample is r. t +γ×Q_target(s t+1 ,π_target(s t+1 )), where γ is the discount factor, Q_target represents the output of the target Q network, and π_target represents the output of the target policy network.

[0093] S2053. Using the mean squared error loss function, calculate the squared average difference between the Q values ​​of all extracted samples and the target Q value:

[0094]

[0095] In the above formula, n represents the set value. Represents the loss function. The parameters represent the loss function. Q_target represents the current output of the Q network. i (s t,j ,a t,j ) represents the target Q value.

[0096] S2054. Based on the squared value of the average difference, update the Q-network set in step S103 using the gradient descent algorithm. The specific update process is as follows:

[0097] For loss function Regarding parameters Take the derivative to obtain the gradient.

[0098]

[0099] This represents the Q-value of the j-th sample with respect to the parameter. The gradient of the parameter represents the sensitivity of the predicted value of the corresponding state-action pair to the parameters. At each step of gradient descent, this gradient is used to update the parameters:

[0100]

[0101] Where α is the learning rate, which determines the size of the step in the gradient direction.

[0102] S206. Update the policy network set in step S101, specifically as follows:

[0103] S2061. Calculate the gradient of the policy network set in step S101 using the chain rule. The specific calculation formula is as follows:

[0104]

[0105] In the above formula, The parameters of the policy network are represented by , which are optimized using gradient ascent or descent methods. J represents the objective function, i.e., the expected cumulative reward, which is maximized by adjusting the parameters of the policy network. This represents the gradient of the objective function with respect to the policy network parameters. This represents the gradient of the Q-function with respect to action a. Indicates the policy π relative to the parameters The gradient of E is given by E, which represents the expectation function.

[0106] S2062. Based on the gradient calculated in step S2061, the policy network set in step S101 is updated using the backpropagation algorithm and gradient descent algorithm. The specific update process is as follows:

[0107] Calculate the policy gradient using the chain rule:

[0108]

[0109] Q-value pairs of actions The partial derivatives, The policy network output is related to its parameters. The partial derivative of , where D represents the state sampling space.

[0110] Use the calculated policy gradient to adjust the parameters of the policy network. Perform gradient ascent update:

[0111] S207. Update the target policy network set in step S102 by using a soft update strategy to update the parameters of the target policy network. Specifically, the parameters of the target policy network are a smoothed value of the Q network or policy network parameters, i.e., θ_target = τθ + (1-τ)θ_target, where θ_target represents the parameters of the target policy network, τ is a constant, and θ represents the parameters of the current target policy network.

[0112] S208. Perform iterative training, iterating through steps S201-S207 until the preset number of iterations is reached or the agent's performance reaches a satisfactory level, thus obtaining a process model of production data.

[0113] S209. Using the methods in steps S201-S208, obtain the process model for the next type of production data.

[0114] S2010. Repeat steps S201-S209 until the process model for all production data is generated.

[0115] The inference unit is used to acquire real-time production data from the production site, input the real-time production data into the corresponding process model, obtain optimized production data, generate multiple adjustment schemes, and thus guide the production site to adjust the corresponding parameters to achieve the purpose of optimizing product quality and improving production efficiency.

[0116] The decision-making unit utilizes a multi-objective direct preference optimization algorithm based on the reasoning unit to achieve collaborative optimization of multiple process tasks. This allows for trade-offs among different adjustment schemes, resolving decision conflicts, and summarizing them into a unified decision adjustment scheme. This decision adjustment scheme constitutes the final process sequence. The expression of the optimization function is as follows:

[0117]

[0118] In the above formula, L MODPO Let y represent the optimization function, x represent the process parameters in the production data, and the process parameters x contain multiple objectives. w The decision adjustment scheme representing relative preference, y l This indicates a decision adjustment plan that is relatively unfavorable. The probability distribution of the response y to process parameter x; r (υ,-k) ω represents the reward score for the response y of multiple parameters x. The reward score can be calculated based on multiple objectives, including evaluating the production efficiency and production cost of different combinations of process parameters x and response y; ω represents the weight between different objectives of process parameter x, and D k This represents a dataset, which includes process parameters x and corresponding decision adjustment schemes y. w and y l σ represents the sigmoid function, used to map the input to the (0,1) interval, β represents the temperature parameter controlling the intensity of the KL penalty, and π sft This indicates supervised fine-tuning of the language model. Let ω represent the target weight vector excluding the k-th target, T represents the matrix transpose, and k represents the current target index.

[0119] The execution unit is used to schedule RPA robots, implement the decision adjustment plan obtained by the decision unit into the production process, and monitor the production data of the process after the execution of the decision adjustment plan in real time, so as to grasp the real-time status of the production site.

[0120] The alarm unit sets alarm thresholds for the production data of each process. The alarm unit is also used to receive real-time production data monitored by the execution unit. When the production data exceeds the corresponding alarm threshold, the alarm unit immediately triggers a packet alarm.

[0121] This invention achieves non-intrusive, automated data acquisition and preprocessing from multiple production systems and databases, providing comprehensive and accurate data support for process optimization. It utilizes deep reinforcement learning algorithms for in-depth data mining and optimization analysis, identifying key influencing factors and potential for process optimization. A large-scale model is built using code generation, and process parameters are adaptively optimized using multi-objective reinforcement learning, achieving dual optimization of process efficiency and product quality. The system enables automated execution and real-time closed-loop control of the process, reducing manual intervention. By integrating RPA, reinforcement learning, and deep learning technologies, this invention constructs an intelligent optimization software robot system for traditional Chinese medicine (TCM) manufacturing processes. This system achieves multi-task, multi-objective collaborative intelligence, automation, and lean manufacturing of TCM manufacturing processes, significantly improving production efficiency, reducing resource consumption, and minimizing production defects. It not only realizes multi-task, multi-objective collaborative self-decision-making in TCM manufacturing processes but also ensures the integrity of existing production control systems. Its deployment and use are extremely convenient, demonstrating broad application prospects.

[0122] Finally, it should be noted that the above content is only used to illustrate the technical solution of the present invention, and is not intended to limit the scope of protection of the present invention. Simple modifications or equivalent substitutions made by those skilled in the art to the technical solution of the present invention do not depart from the essence and scope of the technical solution of the present invention.

Claims

1. A multi-task, multi-objective collaborative optimization and control robot system for pharmaceutical processes, characterized in that: It includes an execution agent module, an inference agent module, and a self-decision-making module. The execution agent module includes a data acquisition unit and a data preprocessing unit; the inference agent module includes a model training unit and an inference unit; and the self-decision-making module includes a decision-making unit and an execution unit. The data capture unit is equipped with RPA, which captures historical production data from the Enterprise Resource Planning System, Manufacturing Execution System, and Laboratory Information Management System by simulating manual operation. The historical production data includes product batch information, process parameter settings, and operating status of production equipment. The data preprocessing unit performs data cleaning, standardization, deduplication, and format conversion operations on the historical production data captured by the data capture unit. The process model for each historical production data point is obtained through the following steps: S201. Take one type of historical production data from the historical production data processed by the data preprocessing unit as the initial state. t represents the time for fetching historical production data, and the policy network is based on the initial state. Make the initial action ; S202, The environment is based on the initial action. For the initial state Make changes to obtain the state at the next moment. and provide rewards to the intelligent agent. ; S203, Iterate through steps S201-S202 until the action performed by the agent is the target action. Stop iterating when the time comes; S204. Set up a buffer. Store the data from each iteration in S203 as a sample in the buffer. Each sample contains the data generated in one iteration. , , , ; S205. Update the Q network set in step S103; S206. Update the policy network set in step S101; S207. Update the target policy network set in step S102; S208. Perform iterative training, iterating through steps S201-S207 until the preset number of iterations is reached, to obtain a process model based on historical production data. S209. Using the methods in steps S201-S208, obtain the process model for the next type of historical production data; S2010. Repeat steps S201-S209 until the process model for all historical production data is generated. S205 specifically includes the following steps: S2051. Extract a set number of samples from the buffer and input them into the Q network set in step S103 to calculate the Q value of each sample. S2052. Calculate the target Q-value for each sample using the target policy network set in step S102 and the target Q-network set in step S104. For each sample, if the state... If the state is terminated, then the target Q value of this sample is... Conversely, the target Q value of this sample is ,in As a discount factor, This represents the output of the target Q-network. This represents the output of the target policy network; S2053. Using the mean squared error loss function, calculate the squared average difference between the Q values ​​of all extracted samples and the target Q value: In the above formula, n represents the set value. Represents the loss function. The parameters represent the loss function. This represents the current output of the Q network. The target Q value; S2054. Based on the squared value of the average difference, update the Q-network set in step S103 using the gradient descent algorithm. The specific update process is as follows: For loss function Regarding parameters Take the derivative to obtain the gradient. : This represents the Q-value of the j-th sample with respect to the parameter. The gradient is used to update the parameters at each step of gradient descent using the following formula: In the above formula, It is the learning rate; S206 specifically includes the following steps: S2061. Calculate the gradient of the policy network set in step S101 using the chain rule. The specific calculation formula is as follows: In the above formula, The parameters of the policy network are represented and optimized using gradient ascent or descent methods. This represents the objective function, i.e., the expected cumulative reward, which is maximized by adjusting the parameters of the policy network. This represents the gradient of the objective function with respect to the policy network parameters. The Q function represents the action. gradient, Representation Strategy Relative to parameters gradient, Represent the expected function; S2062. Based on the gradient calculated in step S2061, the policy network set in step S101 is updated using the backpropagation algorithm and gradient descent algorithm. The specific update process is as follows: Calculate the policy gradient using the chain rule: Q-value pairs of actions The partial derivatives, The policy network output is related to its parameters. The partial derivatives, Represents the state sampling space; Using the calculated policy gradient, the parameters of the policy network are expressed by the following formula. Perform gradient ascent update: ； The model training unit constructs a Markov decision process training model, and obtains the process model corresponding to each historical production data by training the Markov decision process training model. The Markov decision process training model includes five elements: agent, environment, policy, state, action, and reward. It is constructed through the following steps: S101. Define a policy network to output action a given state s; S102. Define the target policy network to calculate the target action. ; S103. Define the Q-network to output the Q-value and optimize the policy network; S104. Define the target Q-network, which is used to calculate the target Q-value and to optimize the Q-network; The reasoning unit acquires real-time production data and inputs it into the corresponding process model to obtain optimized production data. Based on the optimized production data, multiple adjustment schemes are generated. The decision-making unit uses the direct preference optimization algorithm to unify the multiple adjustment schemes generated by the reasoning unit to obtain a decision adjustment scheme; The direct preference optimization algorithm in the decision-making unit is expressed by the following formula: In the above formula, Let x represent the process parameters in the production data, where x represents the optimization function and x contains multiple objectives. Decision adjustment schemes representing relative preferences This indicates a relatively unfavorable decision-making adjustment plan. This represents the probability distribution of the response y to process parameter x; This represents the reward score for the response y, which has multiple parameters x. This represents the weights between different objectives for the process parameter x. Represents a dataset, This refers to the sigmoid function, which maps input to the interval (0,1). The temperature parameter that controls the intensity of the KL penalty. This indicates supervised fine-tuning of the language model. Represents the target weight vector The weight vector of all targets except the k-th target, where T represents the matrix transpose and k represents the current target index; The execution unit is used to control the production system to execute decision adjustment plans.

2. The multi-task, multi-objective collaborative optimization and control robot system for pharmaceutical processes according to claim 1, characterized in that, In step S201, the agent will take an action based on its state. Generate multiple candidate actions Specifically: In the above formula, This represents random noise drawn from a Gaussian or uniform distribution; through The network calculates the Q-values ​​of the generated candidate actions separately, and then selects the candidate actions... and state ,enter Network obtained The corresponding Q value: Then calculate the probability P of each action according to the following formula: In the above formula, Q represents the Q-network, a represents the action, and s represents the state; The candidate action with the highest probability is selected as the final action taken by the agent.

3. The multi-task, multi-objective collaborative optimization and control robot system for pharmaceutical processes according to claim 1, characterized in that, In step S207, the target policy network is updated by updating its parameters. The parameters of the target policy network are calculated and updated using the following formula: In the above formula, τ represents the parameters of the target policy network, where τ is a constant. This represents the parameters of the current target policy network.

4. The multi-task, multi-objective collaborative optimization and control robot system for pharmaceutical processes according to claim 1, characterized in that, The self-decision-making module also includes an alarm unit. The execution unit is also used to acquire real-time production data of the production site after the execution decision adjustment plan. The alarm unit has an alarm threshold for each production data. When real-time production data exceeds its corresponding alarm threshold, the alarm unit will issue an alarm.

Images