Industrial blending tank control method based on deep reinforcement learning model

Abstract

The invention relates to the technical field of industrial control, and particularly discloses an industrial blending tank control method based on a deep reinforcement learning model, and the method comprises the steps: building a blending tank simulation system as an interaction environment of deep reinforcement learning; a deep reinforcement learning network of an Actor-critic structure is set; an experience replay pool is constructed based on a deep reinforcement learning network and an interaction environment, and an Actor network and a critic network are respectively trained based on the experience replay pool to obtain a strategy model and an evaluation model. According to the method, a large amount of empirical data is obtained through interaction with a simulated production device or system, learning is carried out from the large amount of empirical data, so that a control strategy is continuously optimized, the mode is totally different from PID and is similar to the thought that prediction is carried out before control of MPC, but system identification and parameter setting do not need to be carried out, and therefore the control strategy is optimized. A large amount of complex work is avoided, learning is completely carried out from data, and a certain generalization ability is achieved.

Description

Technical Field

[0001] The present invention relates to the field of industrial control technology, and in particular to a control method for an industrial blending tank based on a deep reinforcement learning model. Background Art

[0002] At present, due to its simplicity and effectiveness, more than 90% of controllers use proportional-integral-differential (PID) control, but PID is only applicable to linear steady-state systems with single input and single output. With the acceleration of industrial intelligence, there is an increasing demand for the use of artificial intelligence algorithms in the optimization and control of industrial production processes. Industrial production devices or systems generally have multiple operating variables that jointly affect multiple controlled variables, that is, multiple variables are coupled, and the operating variables and controlled variables usually have a nonlinear relationship. In addition, process industries almost all have the characteristics of large lags, which makes them unsuitable for PID algorithms. Usually, the control algorithm in industrial fields such as chemical industry is the model predictive control (MPC) algorithm. MPC is known for its good stability, but its premise is to find the steady state of the controlled variable, do system identification and parameter setting, which is very difficult and labor-intensive. The accuracy of model identification directly affects the subsequent control effect.

[0003] Deep reinforcement learning combines the advantages of sequential decision-making of reinforcement learning with the ability of deep learning to abstract states and approximate complex decisions and evaluation functions, making it very suitable for the control of complex industrial production equipment or systems. However, the common difficulty at present is that the actual production system has extremely high safety requirements and cannot allow unreliable trial and error and output actions of reinforcement learning in the early stages of training. Therefore, before applying the reinforcement learning model to the actual production system, it is necessary to use offline data or simulation data to train a model with stable and reliable performance. Summary of the invention

[0004] The purpose of the present invention is to provide a control method for an industrial blending tank based on a deep reinforcement learning model to solve the problems raised in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A control method for an industrial blending tank based on a deep reinforcement learning model, the method comprising:

[0007] Build a blending tank simulation system as an interactive environment for deep reinforcement learning;

[0008] A deep reinforcement learning network of an Actor-critic structure is set; the Actor-critic structure includes an Actor network and a Critic network, the Actor network is called a policy network, and the Critic network is called an evaluation network; wherein the number of output actions of the policy network is equal to the number of operating variables of the industrial production device;

[0009] An experience replay pool is constructed based on a deep reinforcement learning network and an interactive environment, and an Actor network and a critic network are trained based on the experience replay pool to obtain a policy model and an evaluation model. The training process is to evaluate the output of the policy model according to the evaluation model, and the evaluation process uses cumulative immediate rewards, overshoots and steady-state errors.

[0010] As a further solution of the present invention: in the stage of creating an interactive environment, the process of building a blending tank simulation system includes:

[0011] Set the state representation; set the state s t =(c1,c2,c t ), where s t represents the state at time t, c1 and c2 represent the concentrations of the two liquid raw materials respectively; c t Indicates the concentration of the blended liquid at time t;

[0012] Set action definition; Set action and They represent the flow rates u1 and u2 of the two liquid raw materials entering the blending tank respectively;

[0013] Set immediate reward; set r t+1 =α×overshoot+β×delay+γ×steady-state error; r t+1 represents the immediate reward at time t+1; the overshoot, delay and steady-state error are represented by c t It is calculated that α, β and γ are all preset correction coefficients, α=0.4, β=0.2, γ=0.4.

[0014] As a further solution of the present invention: the steps of constructing an experience replay pool based on a deep reinforcement learning network and an interactive environment, respectively training an Actor network and a critic network based on the experience replay pool, and obtaining a strategy model and an evaluation model include:

[0015] Get the state at a certain moment, input the state at that moment into the Actor network, and get the action;

[0016] Apply the action to the interactive environment to obtain the state at the next moment;

[0017] Input the next moment state into the critic network and get an immediate reward;

[0018] Collect all data and build an experience replay pool;

[0019] Based on the experience replay pool, the Actor network and the critic network are trained respectively to obtain a strategy model and an evaluation model.

[0020] As a further solution of the present invention: the step of constructing an experience replay pool based on the deep reinforcement learning network and the interactive environment, and training the Actor network and the critic network based on the experience replay pool to obtain the strategy model and the evaluation model also includes:

[0021] The step response model is used as the interactive interface model between the deep reinforcement learning network and the interactive environment. The action is input into the step response model to obtain the state and immediate reward at the next moment.

[0022] The general form of a step signal is:

[0023]

[0024] Where 1(t) represents a unit step signal, A is a preset amplitude, and a step occurs when t=t0. At this time, the step signal is:

[0025]

[0026] Among them, Δu is the amplitude of the step, that is, the change in the flow rate of the two liquid raw materials.

[0027] As a further solution of the present invention: the data structure of the samples in the experience replay pool is (s t ,a t ,r t ,s t+1 ), where s t represents the state at time t, a t Indicates t The corresponding action, r t Indicates a t The corresponding immediate reward, s t+1 Represents the state at time t+1.

[0028] As a further solution of the present invention: the number of neurons in the output layer of the Actor network is set to 2, indicating the output of two actions; the actions obey the conditional probability distribution, that is, a1, a2~π(a|s; θ); wherein a1 and a2 represent two actions, π represents the conditional probability distribution, s represents the state, and θ is a parameter of the Actor network.

[0029] As a further solution of the present invention: the Actor network is the basic MLP network; the critic network includes a first evaluation network, a second evaluation network, a first target evaluation network and a second target evaluation network, the four critic networks have the same structure, are all basic network MLP, and have the same initial parameters; the target evaluation network is used to assist the evaluation network in completing the update, and the evaluation network with a small output is selected to assist the strategy network in completing the update.

[0030] As a further solution of the present invention: the parameter updating process includes:

[0031] Read the sample in the experience replay pool and convert the state s in the sample t Input the policy network and get the predicted action and immediate reward; the sample is a four-tuple,

[0032] The sample (s t ,a t ) is input into the evaluation network to obtain Q(s t ,a t |θ q ), the s in the sample t+1 Input the policy network and get the predicted action a t+1 , will s t+1 and a t+1 Input the target evaluation network and calculate Among them, θ q represents the parameters in the evaluation network, represents the parameters in the target evaluation network;

[0033] Based on the obtained Q(s t ,a t |θ q )and Calculate the loss function of the evaluation network, Among them, r t Indicates that action a is taken at time t t The immediate reward is obtained, γ represents the preset discount factor, and the gradient descent method is used to minimize the loss function and update the parameters θ in the evaluation network. q ;

[0034] Will Input to the trained evaluation network, predict Using gradient ascent method to maximize Update the parameters θ in the policy network u .

[0035] Compared with the prior art, the beneficial effects of the present invention are: the present invention obtains a large amount of empirical data by interacting with the simulated production device or system, and learns from the large amount of empirical data, thereby continuously optimizing the control strategy. This method is completely different from PID, and has certain similarities with the idea of ​​MPC of first predicting and then controlling, but there is no need for system identification and parameter setting, thus avoiding a lot of complicated work, and learning completely from the data with a certain generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention.

[0037] Figure 1 This is a flowchart of the control method of an industrial blending tank based on a deep reinforcement learning model. DETAILED DESCRIPTION

[0038] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0039] Figure 1 The flowchart of the control method of the industrial blending tank based on the deep reinforcement learning model is as follows. In an embodiment of the present invention, a control method of the industrial blending tank based on the deep reinforcement learning model includes:

[0040] Step S100: Building a blending tank simulation system as an interactive environment for deep reinforcement learning;

[0041] Step S200: setting a deep reinforcement learning network of an Actor-critic structure; the Actor-critic structure includes an Actor network and a critic network, the Actor network is called a policy network, and the critic network is called an evaluation network; wherein the number of output actions of the policy network is equal to the number of operating variables of the industrial production device;

[0042] Step S300: construct an experience replay pool based on the deep reinforcement learning network and the interactive environment, and train the Actor network and the critic network based on the experience replay pool to obtain a strategy model and an evaluation model; wherein the training process is to evaluate the output of the strategy model according to the evaluation model, and the evaluation process uses cumulative immediate rewards, overshoots and steady-state errors.

[0043] The present invention proposes a control method for an industrial blending tank based on a deep reinforcement learning model. By using reinforcement learning as a control model, it is necessary to define the key elements of the Markov decision process, including the representation of the state, the definition of the action, and the design of the reward function, wherein the reward function is difficult to design in some scenarios. Since the ultimate goal is control, the present invention uses the commonly used indicators in the control field when designing the reward function: overshoot, delay, and steady-state error.

[0044] Moreover, deep reinforcement learning is to learn from a large amount of empirical data obtained by interacting with the environment, that is, the simulated production device or system, so as to continuously optimize the control strategy. This method is completely different from PID, and has certain similarities with the idea of ​​MPC of first predicting and then controlling. However, deep reinforcement learning does not require system identification and parameter tuning, avoiding a lot of complex work. Deep reinforcement learning is completely learned from data and has a certain generalization ability.

[0045] Specifically, the working process of the technical solution of the present invention is that at a certain moment, the state in the current blending tank (the concentration of different liquid raw materials) is obtained, and the deep reinforcement learning model applies an input action to the interactive environment constructed by the simulation system. At this time, the state in the current blending tank will change, and the changed state is obtained. At the same time, the overshoot, delay and steady-state error are obtained, and the immediate reward is calculated. The grid is continuously updated according to the immediate reward. When the conditions are met, the Actor network is output as a strategy model. When faced with control needs, only the state needs to be input to obtain the corresponding action; among them, the action mentioned in this application is the flow rate added when different liquid raw materials are added.

[0046] Furthermore, in the stage of creating an interactive environment, the process of building a blending tank simulation system includes:

[0047] Set the state representation; set the state s t =(c1,c2,c t ), where s t represents the state at time t, c1 and c2 represent the concentrations of the two liquid raw materials respectively; c t Indicates the concentration of the blended liquid at time t;

[0048] Set action definition; Set action and They represent the flow rates u1 and u2 of the two liquid raw materials entering the blending tank respectively;

[0049] Set immediate reward; set r t+1 =α×overshoot+β×delay+γ×steady-state error; r t+1 represents the immediate reward at time t+1; the overshoot, delay and steady-state error are represented by c tIt is calculated that α, β and γ are all preset correction coefficients, α=0.4, β=0.2, γ=0.4.

[0050] The above content limits the data structure of the parameters involved in this application, specifically limiting the state, action and immediate reward; wherein, the state is the input of the Actor network, and the action is the output of the Actor network. After the action is applied to the interactive environment (blending tank simulation system), it is evaluated by the critic network to obtain an immediate reward; the purpose of the Critic network is to evaluate the value of the action output by the Actor network in the current state, and its output is a scalar used to guide the optimization of the Actor network.

[0051] As a preferred embodiment of the technical solution of the present invention, the steps of building an experience replay pool based on a deep reinforcement learning network and an interactive environment, respectively training an Actor network and a critic network based on the experience replay pool, and obtaining a strategy model and an evaluation model include:

[0052] Get the state at a certain moment, input the state at that moment into the Actor network, and get the action;

[0053] Apply the action to the interactive environment to obtain the state at the next moment;

[0054] Input the next moment state into the critic network and get an immediate reward;

[0055] Collect all data and build an experience replay pool;

[0056] Based on the experience replay pool, the Actor network and the critic network are trained respectively to obtain a strategy model and an evaluation model.

[0057] The training model needs to build samples first. The above content provides the sample construction process, obtains the state at a certain moment, inputs the state at that moment into the Actor network, obtains the action, applies the action to the interactive environment, obtains the state at the next moment, inputs the state at the next moment into the critic network, and obtains an immediate reward. This is actually a sequential application process from the Actor network to the interactive environment to the critic network. The data in this process is counted to build an experience replay pool as a sample set; the data structure of the samples in the experience replay pool is (s t ,a t ,r t ,s t+1 ), where s t represents the state at time t, a t Indicates t The corresponding action, r t Indicates a t The corresponding immediate reward, st+1 Represents the state at time t+1.

[0058] After the experience replay pool is built, the Actor network and the critic network are trained based on the experience replay pool. When the preset conditions are met, the trained Actor network and the critic network are output to obtain the strategy model and the evaluation model.

[0059] Furthermore, the interaction process between the Actor network and the interactive environment needs to be explained as follows:

[0060] The step of constructing an experience replay pool based on the deep reinforcement learning network and the interactive environment, and respectively training the Actor network and the critic network based on the experience replay pool to obtain a strategy model and an evaluation model also includes:

[0061] The step response model is used as the interactive interface model between the deep reinforcement learning network and the interactive environment. The action is input into the step response model to obtain the state and immediate reward at the next moment.

[0062] The general form of a step signal is:

[0063]

[0064] Where 1(t) represents a unit step signal, A is a preset amplitude, and a step occurs when t=t0. At this time, the step signal is:

[0065]

[0066] Among them, Δu is the amplitude of the step, that is, the change in the flow rate of the two liquid raw materials.

[0067] When the input of the system (operated variables u1 and u2) changes, the output will change accordingly, that is, the system will enter a new state. Then, the overshoot, delay and steady-state error are calculated based on the data curve of the controlled variable.

[0068] Among them, the delay, also known as the lag time τ, can be directly read from the output response curve, that is, the time from the input step signal at time t = 0 to the start of the change of the response observation data at the output end, which is used to measure the response speed of the system. The steady-state error can be expressed as e ss =c(t)-r, t→∞, ss is the abbreviation of stable state in English, which means steady state, that is, when time tends to positive infinity, the difference between the system output c(t) and the reference input r (that is, the set value of the controlled variable, which is the required blended concentration in the present invention).

[0069] In actual operation, as long as the output curve is stable, it can be considered that time tends to positive infinity, and the steady-state output value c(t) can be obtained. The steady-state error is used to measure the control accuracy of the system. The overshoot σ is the instantaneous maximum max of the controlled variable, i.e. the concentration of the blended liquid, under the action of the step input. t The relative percentage error between c(t) and the steady-state value c(t), t→∞ can be expressed as The overshoot is used to measure the stability of the system in the process of responding to a step signal input. Finally, the immediate reward can be calculated.

[0070] As a preferred embodiment of the technical solution of the present invention, the number of neurons in the output layer of the Actor network is set to 2, indicating the output of two actions; the actions obey the conditional probability distribution, that is, a1, a2~π(a|s; θ); wherein a1 and a2 represent two actions, π represents the conditional probability distribution, s represents the state, and θ is a parameter of the Actor network.

[0071] In an example of the technical solution of the present invention, a specific model training solution is provided as follows:

[0072] The Actor network is the basic MLP network; the critic network includes a first evaluation network, a second evaluation network, a first target evaluation network and a second target evaluation network. The four critic networks have the same structure, which is the basic network MLP and has the same initial parameters; the target evaluation network is used to assist the evaluation network in completing the update, and the evaluation network with a small output is selected to assist the strategy network in completing the update.

[0073] The parameter updating process includes:

[0074] Read the sample in the experience replay pool and convert the state s in the sample t Input the policy network and get the predicted action and immediate reward; the sample is a four-tuple,

[0075] The sample (s t ,a t ) is input into the evaluation network to obtain Q(s t ,a t |θ q ), the s in the sample t+1 Input the policy network and get the predicted action a t+1 , will s t+1 and a t+1 Input the target evaluation network and calculate Among them, θ q represents the parameters in the evaluation network, represents the parameters in the target evaluation network;

[0076] Based on the obtained Q(s t ,a t |θ q )and Calculate the loss function of the evaluation network, Among them, r t Indicates that action a is taken at time t t The immediate reward is obtained, γ represents the preset discount factor, and the gradient descent method is used to minimize the loss function and update the parameters θ in the evaluation network. q ;

[0077] Will Input to the trained evaluation network, predict Using gradient ascent method to maximize Update the parameters θ in the policy network u .

[0078] The functions that can be achieved by the above-mentioned control method of the industrial blending tank based on the deep reinforcement learning model are all completed by a computer device, and the computer device includes one or more processors and one or more memories, and at least one program code is stored in the one or more memories. The program code is loaded and executed by the one or more processors to realize the functions of the control method of the industrial blending tank based on the deep reinforcement learning model.

[0079] The processor takes out instructions from the memory one by one, analyzes the instructions, and then completes the corresponding operations according to the instruction requirements, generating a series of control commands, so that the various parts of the computer can automatically, continuously and coordinately move to become an organic whole, realize the input of programs, the input of data, and the calculation and output of results. The arithmetic operations or logical operations generated in this process are all completed by the operator; the memory includes a read-only memory (ROM), which is used to store computer programs, and a protection device is provided outside the memory.

[0080] Exemplarily, the computer program may be divided into one or more modules, one or more modules are stored in a memory and executed by a processor to implement the present invention. One or more modules may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program in a terminal device.

[0081] Those skilled in the art will understand that the description of the above service equipment is merely an example and does not constitute a limitation on the terminal equipment. It may include more or fewer components than described above, or a combination of certain components, or different components, for example, it may include input and output devices, network access devices, buses, etc.

[0082] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal device, and uses various interfaces and lines to connect various parts of the entire user terminal.

[0083] The memory can be used to store computer programs and / or modules. The processor can realize various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as an information collection template display function, a product information release function, etc.); the data storage area can store data created according to the use of the berth status display system (such as product information collection templates corresponding to different product types, product information that different product providers need to release, etc.). In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0084] If the module / unit integrated in the terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the modules / units in the above-mentioned embodiment system, and can also be completed by instructing the relevant hardware through a computer program. The above-mentioned computer program can be stored in a computer-readable storage medium, and the computer program can realize the functions of the above-mentioned various system embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. Computer-readable media may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0085] It should be noted that, in this article, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0086] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A control method for industrial blending tank based on deep reinforcement learning model, It is characterized in that The method comprises: Build a blending tank simulation system as an interactive environment for deep reinforcement learning; A deep reinforcement learning network of an Actor-critic structure is set; the Actor-critic structure includes an Actor network and a Critic network, the Actor network is called a policy network, and the Critic network is called an evaluation network; wherein the number of output actions of the policy network is equal to the number of operating variables of the industrial production device; An experience replay pool is constructed based on a deep reinforcement learning network and an interactive environment, and an Actor network and a critic network are trained based on the experience replay pool to obtain a policy model and an evaluation model. The training process is to evaluate the output of the policy model according to the evaluation model, and the evaluation process uses cumulative immediate rewards, overshoots and steady-state errors.

2. The control method of the industrial blending tank based on the deep reinforcement learning model according to claim 1, It is characterized in that The process of building a blending tank simulation system includes: Set the state representation; set the state s t =(c 1 ,c 2 ,c t ), where s t represents the state at time t, c 1 and c 2 Respectively represent the concentration of the two liquid raw materials; c t Indicates the concentration of the blended liquid at time t; Set action definition; Set action and They represent the flow rates u of the two liquid raw materials entering the blending tank. 1 and u 2 ; Set immediate reward; set r t+1 =α×overshoot+β×delay+γ×steady-state error; r t+1 represents the immediate reward at time t+1; the overshoot, delay and steady-state error are represented by c t It is calculated that α, β and γ are all preset correction coefficients, α=0.4, β=0.2, γ=0.

4.

3. The control method of the industrial blending tank based on the deep reinforcement learning model according to claim 1, It is characterized in that The steps of constructing an experience replay pool based on a deep reinforcement learning network and an interactive environment, respectively training an Actor network and a critic network based on the experience replay pool, and obtaining a strategy model and an evaluation model include: Get the state at a certain moment, input the state at that moment into the Actor network, and get the action; Apply the action to the interactive environment to obtain the state at the next moment; Input the next moment state into the critic network and get an immediate reward; Collect all data and build an experience replay pool; Based on the experience replay pool, the Actor network and the critic network are trained respectively to obtain a strategy model and an evaluation model.

4. The control method of the industrial blending tank based on the deep reinforcement learning model according to claim 3, It is characterized in that The step of constructing an experience replay pool based on the deep reinforcement learning network and the interactive environment, and respectively training the Actor network and the critic network based on the experience replay pool to obtain a strategy model and an evaluation model also includes: The step response model is used as the interactive interface model between the deep reinforcement learning network and the interactive environment. The action is input into the step response model to obtain the state and immediate reward at the next moment. The general form of a step signal is: Where 1(t) represents a unit step signal, A is the preset amplitude, and when t=t 0 A step occurs when , and the step signal is: Among them, Δu is the amplitude of the step, that is, the change in the flow rate of the two liquid raw materials.

5. The control method of the industrial blending tank based on the deep reinforcement learning model according to claim 1, It is characterized in that The data structure of the samples in the experience replay pool is (s t ,a t ,r t ,s t+1 ), where s t represents the state at time t, a t Indicates t The corresponding action, r t Indicates a t The corresponding immediate reward, s t+1 Represents the state at time t+1.

6. The control method of the industrial blending tank based on the deep reinforcement learning model according to claim 5, It is characterized in that The number of neurons in the output layer of the Actor network is set to 2, indicating that two actions are output; the actions obey the conditional probability distribution, that is, a 1 ,a 2 ~π(a|s;θ); where a 1 and a 2 represents two actions, π represents the conditional probability distribution, s represents the state, and θ is the parameter of the Actor network.

7. The control method of the industrial blending tank based on the deep reinforcement learning model according to claim 6, It is characterized in that The Actor network is the basic MLP network; the critic network includes a first evaluation network, a second evaluation network, a first target evaluation network and a second target evaluation network. The four critic networks have the same structure, which is the basic network MLP and has the same initial parameters; the target evaluation network is used to assist the evaluation network in completing the update, and the evaluation network with a small output is selected to assist the strategy network in completing the update.

8. The control method of the industrial blending tank based on the deep reinforcement learning model according to claim 7, It is characterized in that The parameter update process includes: Read the sample in the experience replay pool and convert the state s in the sample t Input the policy network and get the predicted action and immediate reward; the sample is a four-tuple, The sample (s t ,a t ) is input into the evaluation network to obtain Q(s t ,a t |θ q ), the s in the sample t+1 Input the policy network and get the predicted action a t+1 , will s t+1 and a t+1 Input the target evaluation network and calculate Among them, θ q represents the parameters in the evaluation network, represents the parameters in the target evaluation network; Based on the obtained Q(s t ,a t |θ q )and Calculate the loss function of the evaluation network, Among them, r t Indicates that action a is taken at time t t The immediate reward is obtained, γ represents the preset discount factor, and the gradient descent method is used to minimize the loss function and update the parameters θ in the evaluation network. q ; Will Input to the trained evaluation network, predict Using gradient ascent method to maximize Update the parameters θ in the policy network u .

Images