Deep reinforcement learning optimization compensation method oriented to nonlinear intermittent process

Abstract

The invention discloses a deep reinforcement learning optimization compensation method oriented to a nonlinear intermittent process. The method comprises the following steps: expanding a three-dimensional input data matrix into a two-dimensional matrix; carrying out standardization treatment; a JY-KPLS model is constructed; a JY-KPLS model optimization problem is constructed; solving an optimization problem; calculating the similarity between the historical data and the query data; respectively screening m pieces of old data and n pieces of new data from the old process data set and the new process data set according to the similarity, and subtracting the current query data from the m pieces of old data and the n pieces of new data; a deviation sample is used as a data set, a JITL-JYKPLS local model is established, and the mismatch problem is solved; performing interaction and trial-and-error training on the compensated model and an optimization system; if the total training reward value in the current batch exceeds the total training reward value in the previous batch, the currently trained optimization system is used for inter-batch optimization; otherwise, the previous batch optimization system is used for inter-batch optimization; and outputting the final product quality. The method can significantly improve the quality of the final product.

Description

Technical Field

[0001] The present invention belongs to the technical field of operation optimization of industrial production processes, and specifically is a deep reinforcement learning optimization compensation method for nonlinear intermittent processes. Background Art

[0002] In the process industry, intermittent process, as an important branch, has the characteristics of "multiple time-varying" and is the production method of many high value-added, small-scale, and multi-operation stage products. This multi-process, small-scale production method can well adapt to the rapid changes in market demand in modern process industries. In the production fields of fine chemicals, biopharmaceuticals, food, polymer reactions, metal processing, etc., intermittent processes have a very important position and have been widely praised. With the continuous development of industry, the increasingly fierce market competition environment has put forward higher requirements for enterprises and products. Efficient and stable production processes have received more and more attention. In order to further improve product quality and production efficiency, the operation optimization of production processes has gradually become the focus of research. For intermittent processes, due to the existence of nonlinearity, time-varying and uncertainty factors, how to maximize the total production benefits while ensuring product quality is of extremely important practical significance. The research on operation optimization is usually based on quality prediction models, which can be divided into two categories: mechanism models and data-driven models. However, for more complex industrial systems, it is usually difficult to construct accurate and reliable mechanism models. Therefore, this type of mechanism model-based method is difficult to be widely used in modern industry. Data-driven modeling is favored because it does not rely on knowledge of mechanisms, is easy to build models, and has high model accuracy. However, for a new intermittent production process that has just been put into production, it is usually impossible to obtain sufficient process data due to the short operation time of the production process. In order to establish its mathematical model, it is often necessary to accumulate a large amount of production data by running the production process, which will consume a lot of manpower and financial resources, is costly, and has low modeling efficiency. Therefore, how to effectively reduce the cost of modeling while ensuring model accuracy has become an urgent problem to be solved.

[0003] Since transfer learning was introduced into the field of machine learning in the 1990s, it has received much attention and research, and has made great progress. By transferring the knowledge learned about related tasks to improve the learning process of new tasks, it is especially used to overcome the limitations of highly specialized but small data sets. In actual production processes, there are similar processes with similar process mechanisms and production processes. They may have similar mechanisms, the same or similar production equipment, or common process and quality variables. The idea of ​​establishing a process transfer model is to use the production processes that are similar to the target process and have sufficient process data in actual production, and use their sufficient data information to assist the target process that lacks production data to establish a quality prediction model.

[0004] With the increasingly fierce market competition and the increasing scale of production, the efficiency of intermittent process operation optimization has received more and more attention. The research on operation optimization is of great significance to improving the quality of the final product and the competitiveness of enterprises. Different from the research on conventional process with the goal of maximizing the quality of the final product, the optimization research of high-cost process not only needs to ensure the feasibility of the optimization strategy, but also needs to consider minimizing the optimization cost to ensure the orderly development of subsequent operations. However, although the data-driven model does not have the structural mismatch problem of the mechanism model, there is still a certain deviation between it and the actual process, that is, there is a mismatch between the process and the model. Especially for the process migration model, although similar intermittent processes have the same or similar physical and chemical principles, there are inevitably some differences. These differences will lead to more serious mismatch problems between the process migration model and the actual process, and the NCO mismatch problem of the optimization model will be more prominent, which will eventually make the optimization result obtained by the process migration model only a suboptimal solution. Therefore, in order to improve product quality, production efficiency and the comprehensive economic benefits of enterprises, it is necessary to optimize the quality of the intermittent process on the basis of solving the model mismatch problem. Summary of the invention

[0005] In response to the problems existing in the above-mentioned prior art, the present invention provides a deep reinforcement learning optimization and compensation method for nonlinear intermittent processes. The method can achieve high-efficiency and low-cost optimization of intermittent processes with little data, can solve the problem of difficulty in establishing a process model for a new intermittent process in the early stage of its production operation due to the lack of production data, can effectively reduce the production cost of the intermittent process, and can solve the problem of model mismatch, which can significantly improve the quality of the final product and the economic benefits of the enterprise.

[0006] In order to achieve the above-mentioned object, the present invention provides a deep reinforcement learning optimization compensation method for nonlinear intermittent processes, comprising an intermittent production process A and an intermittent production process B, wherein the intermittent production process A is an old production process that has been produced for a long time and has sufficient historical operation data and operation control experience, and the intermittent production process B is a new production process that is about to be put into production; at the same time, the intermittent production process A and the intermittent production process B have the same or similar production equipment, and the final products of the intermittent production process A and the intermittent production process B belong to the same product family, comprising the following steps:

[0007] Step 1: Expand the three-dimensional input data matrix of intermittent production process A and intermittent production process B into two-dimensional matrices according to the batch direction, namely X a , X b ;

[0008] Step 2: X a , Xb Each column of is processed with zero mean and unit variance, and at the same time, the output data matrix Y a , Y b Zero mean and unit variance treatment was performed;

[0009] Step 3: Construct JY-KPLS process model;

[0010] S31: Let i=1, K ai =K a , K bi =K b , Y Ji =Y J , where i represents the order of extracting latent variables;

[0011] S32: From the joint output matrix Y Ji Select any column as the initial u Ji , where u Ji =[u ai ;u bi ];

[0012] S33: Calculate φ according to formula (1) ai The score vector t ai , calculate φ according to formula (2) bi The score vector t bi :

[0013]

[0014] S34: Calculate the common load matrix q of the joint output variables according to formula (3) Ji , and normalize the load matrix q Ji =q Ji / ||q Ji ||, where q Ji T q Ji =1;

[0015]

[0016] In the formula, t Ji =[t ai ;t bi ];

[0017] S35: Calculate Y ai The score vector u ai =Y ai q Ji , calculate Y bi The score vector u bi =Y bi q Ji ;

[0018] S36: Determine u ai and u bi Whether it converges, if so, proceed to step S37, otherwise return to S33;

[0019] S37: Reduce the kernel matrix and the joint output matrix according to formulas (4), (5) and (6) respectively to obtain the corresponding residual matrix K ai+1 , K bi+1 and K Ji+1 ;

[0020]

[0021] S38: let i=i+1, and repeat S32 to S37 until n pivots are extracted;

[0022] S39: Based on the known kernel matrix K b And the obtained score matrix T b with U b , according to formula (7), the output of the JY-KPLS process model is calculated

[0023]

[0024] In the formula, is the joint matrix of the output variables of intermittent production process A and production process B, is the joint matrix of latent variables of intermittent production process A and production process B;

[0025] Step 4: Construct the optimization problem of the JY-KPLS process model according to formula (8);

[0026]

[0027] Step 5: Solve the optimization problem in formula (8) and obtain the solution x cur ;

[0028] Step 6: x cur As the query point x q , use formula (9) to calculate the similarity s(x a ,x q )、s(x b ,x q );

[0029]

[0030] Where τ∈[0,1] is the weight parameter, d(x q ,x i ) is xq and x i The Euclidean distance between i Denotes Δx q and Δx i The angle between q =x q -x q-1 , Δx i =x i -x i-1 ;

[0031] Step 7: According to the similarity, m old data are selected from the old process data set, and n new data are selected from the new process data set;

[0032] Step 8: Subtract the selected data from the current query data and use the obtained deviation samples as the data set for building the local compensation model (ΔX a ,ΔY a ) and (ΔX b ,ΔY b );

[0033] Step 9: Use the selected data to establish the JITL-JYKPLS local model, and use the trust region optimization method to solve the mismatch problem of the JITL-JYKPLS local model;

[0034] S91: According to formula (10), the optimization problem of the JITL-JYKPLS local model is constructed, that is, the optimization compensation problem based on instant learning and trust region;

[0035]

[0036] Where D is the radius of the spherical trust region, g(Δx) = ||Δx|| 2 <D is the trust region constraint;

[0037] S92: By solving formula (10), the optimized compensation value Δx is obtained, and then the manipulated variable trajectory after compensation is obtained

[0038] S93: Predicting the final manipulated variable trajectory x using the JY-KPLS process model q The corresponding comprehensive economic benefit gain If the comprehensive economic benefit gain ΔP pred If formula (11) is satisfied, then select x q As the operation variable trajectory of the batch training, the process output is predicted Get a new model training environment, otherwise, the qth training is deemed unsuccessful and continue to use the operation variable trajectory x of the previous training q-1 As the final operation variable trajectory of the current training, that is, xq =x q-1 ;

[0039]

[0040] In the formula, τ is the comprehensive economic benefit, δ is the threshold;

[0041] S94: Calculate the actual gain ΔP in economic benefits real =P(x (k-1) ,y (k-1) )-P(x (k) ,y (k) ) to update the trust region radius D; if ΔP real <αΔP pred , then let If ΔP real >αΔP pred ,and Then let D q+1 =γD q Otherwise, let D q+1 =D q ;

[0042] Step 10: Perform trial-and-error training based on the compensated model obtained in step 9 and the optimization system;

[0043] Step 11: After the batch production is completed, determine whether the total reward value of the training in the current batch exceeds the total reward value of the training in the previous batch; if it exceeds, the currently trained optimization system is used for inter-batch optimization; otherwise, the optimization system obtained by the previous batch training is used to replace the current optimization system for inter-batch optimization;

[0044] Step 12: Input the optimized parameters into the controller, use the controller to control the production system to perform production operations, and output the final product quality;

[0045] Step 13: Set k=k+1, update the modeling data, and return to step 1 for the next batch optimization.

[0046] This method adopts the Joint-Y Kernel Partial Least Squares (JY-KPLS) method of joint quality indicators and the idea of ​​Just in Time Learning (JITL), uses the production information of similar processes that are widely present in industrial processes to assist in establishing a new process model, and uses the Deep Deterministic Policy Gradient (DDPG) algorithm to optimize intermittent process operations. For inter-batch optimization, the optimization system with better training of adjacent batches is used for optimization, avoiding the situation where the optimization system with poor training is directly optimized for inter-batch optimization. At the same time, in order to ensure that the process migration model can reflect the actual working conditions of the production process in real time, new data is used for filling and updating to ensure the timeliness of the model. For the intra-batch training process, firstly, based on the idea of ​​transfer learning, the Joint-Y Kernel Partial Least Squares (JY-KPLS) method of the joint quality index is used to migrate similar old production process data to the modeling process of the new production process, so as to assist in the establishment of the new intermittent process quality prediction model, thereby solving the problem that the new production process has less data and it is difficult to accurately establish a model, and greatly reducing the modeling cost. However, if the data with large differences from the new process is blindly used for modeling, the improvement of the accuracy of the migration model may be limited. Therefore, the present invention adopts the Just in Time Learning (JITL) method to extract useful data information from the new and old process data sets in a targeted manner, and limit the data distribution range to establish the JITL-JY-KPLS local model. Due to the inherent differences between the new and old processes and the interference of unknown factors such as model uncertainty and noise, it is inevitable that there is a mismatch problem between the process migration model and the factory, which will affect the realization of subsequent optimization performance. At the same time, the Modifier Adaptation (MA) method is used to solve the gradient to compensate for the difference between the model and the actual process. It is difficult to solve the gradient information in the strongly nonlinear intermittent process, making the implementation process more difficult. In response to these problems, the present invention adopts the Trust Region (TR) method to compensate the process migration model to effectively solve the problem of model mismatch, and compensates the model on the basis of batch training, thereby correcting the reward value that guides the training of the intelligent agent, further improving the optimization performance of deep reinforcement learning, and further improving the final quality of the product and the efficiency of the optimization process, so as to improve the comprehensive economic benefits of the enterprise.

[0047] The method of the present invention can achieve high-efficiency and low-cost optimization of intermittent processes in the presence of little data. It can solve the problem of difficulty in establishing a process model for a new intermittent process in the early stage of its production operation due to lack of production data. It can effectively reduce the production cost of the intermittent process and solve the problem of model mismatch, which can significantly improve the quality of the final product and the economic benefits of the enterprise. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a schematic diagram of the production process of cobalt oxalate crystals in the present invention;

[0049] Figure 2 It is a cumulative reward curve diagram of training in different batches in the present invention;

[0050] Figure 3 It is a performance comparison curve diagram of different optimization strategies in the present invention;

[0051] Figure 4 It is a schematic diagram of the feeding rate of ammonium oxalate of different batches in the present invention;

[0052] Figure 5 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0053] The present invention will be further described below in conjunction with the accompanying drawings.

[0054] like Figures 1 to 5As shown, the present invention provides a deep reinforcement learning optimization and compensation method for nonlinear intermittent processes. The method uses a deep deterministic policy gradient algorithm (DDPG) to optimize the operation of intermittent processes. The process mainly includes two parts: intra-batch training and inter-batch optimization. Inter-batch optimization is to use the optimization system with better training of adjacent batches for optimization, so as to avoid the optimization system with poor training to directly perform inter-batch optimization. In order to ensure that the process migration model can reflect the working conditions of the actual production process in real time, new data is used for filling and updating to ensure the timeliness of the model. In the intra-batch training, the kernel partial least squares (JY-KPLS) method of the joint quality index is first used to migrate similar old production process data to the new production process, thereby solving the problem that the new production process has less data and it is difficult to accurately establish a model, and reducing the modeling cost. However, if the data with large differences from the new process is used for modeling blindly, the improvement of the accuracy of the migration model may be limited. Therefore, Just in Time Learning (JITL) is used to extract useful data information from new and old process data sets in a targeted manner, and the data distribution range is limited to establish a local model. Due to the interference of unknown factors such as model uncertainty and noise, it is difficult to avoid the mismatch between the process migration model and the factory, which in turn affects the realization of subsequent optimization performance. At the same time, the Modifier Adaptation (MA) method solves the gradient to compensate for the difference between the model and the actual process. It is difficult to solve the gradient information in the strongly nonlinear intermittent process, and this method is difficult to implement. The present invention uses the Trust Region (TR) and other gradient-free optimization methods to solve the model mismatch problem, and finally realizes the correction of the reward value of the guiding agent training, further improving the optimization performance of deep reinforcement learning.

[0055] Specifically, there are two similar batch production processes A and B. For a typical batch production process, it has J process variables, K sampling time points in a batch and I batches are collected, so that a typical three-dimensional input data matrix X∈R I×J×K and the two-dimensional output data matrix Y∈R I×K, R represents a real number set. The intermittent production process A is an old production process that has been produced for a long time and has sufficient historical operation data and operation control experience. The intermittent production process B is the current factory and is a new production process that is about to be put into production. At the same time, the production mechanisms of intermittent production processes A and B are similar, and intermittent production processes A and B have the same or similar production equipment, and the final products of intermittent production processes A and B belong to the same product family. Therefore, the operation data and control experience of intermittent production process A will help to establish the production model of intermittent production process B and optimize its operation.

[0056] The method comprises the following steps:

[0057] Step 1: Expand the three-dimensional input data matrix of intermittent production process A and intermittent production process B into two-dimensional matrices according to the batch direction, namely X a , X b ;

[0058] Step 2: X a , X b Each column of is processed with zero mean and unit variance, and at the same time, the output data matrix Y a , Y b Zero mean and unit variance treatment was performed;

[0059] Step 3: Construct JY-KPLS process model;

[0060] S31: Let i=1, K ai =K a , K bi =K b , Y Ji =Y J , where i represents the order of extracting latent variables;

[0061] S32: From the joint output matrix Y Ji Select any column as the initial u Ji , where u Ji =[u ai ;u bi ];

[0062] S33: Calculate φ according to formula (1) ai The score vector t ai , calculate φ according to formula (2) bi The score vector t bi :

[0063]

[0064] S34: Calculate the common load matrix q of the joint output variables according to formula (3)Ji , and normalize the load matrix q Ji =q Ji / ||q Ji ||, where q Ji T q Ji =1;

[0065]

[0066] In the formula, t Ji =[t ai ;t bi ];

[0067] S35: Calculate Y ai The score vector u ai =Y ai q Ji , calculate Y bi The score vector u bi =Y bi q Ji ;

[0068] S36: Determine u ai and u bi Whether it converges, if so, proceed to step S37, otherwise return to S33;

[0069] S37: Reduce the kernel matrix and the joint output matrix according to formulas (4), (5) and (6) respectively to obtain the corresponding residual matrix K ai+1 , K bi+1 and K Ji+1 ;

[0070]

[0071] S38: Let i=i+1, and repeat S32 to S37 until n principal components are extracted; the number of principal components n can be determined by cross-validation (using the prediction residual sum of the validation set to determine whether the prediction ability of the model is statistically improved after the introduction of the new component, thereby giving the optimal number of principal components n);

[0072] S39: Based on the known kernel matrix K b And the obtained score matrix T b with U b , according to formula (7), the output of the JY-KPLS process model is calculated

[0073]

[0074] In the formula, the kernel principal component matrix is the joint matrix of the output variables of intermittent production process A and production process B, is the joint matrix of latent variables of intermittent production process A and production process B. So far, the establishment of JY-KPLS process model is completed; then, the training within the batch is carried out according to steps 4 to 9;

[0075] Step 4: Construct the optimization problem of the JY-KPLS process model according to formula (8);

[0076]

[0077] Step 5: Solve the optimization problem in formula (8) and obtain the solution x cur ;

[0078] Step 6: x cur As the query point x q , use formula (9) to calculate the similarity s(x a ,x q )、s(x b ,x q );

[0079]

[0080] Where τ∈[0,1] is the weight parameter, d(x q ,x i ) is x q and x i The Euclidean distance between i Denotes Δx q and Δx i The angle between q =x q -x q-1 , Δx i =x i -x i-1 ;

[0081] Step 7: According to the similarity, m old data are selected from the old process data set, and n new data are selected from the new process data set;

[0082] Step 8: Subtract the selected data from the current query data and use the obtained deviation samples as the data set for building the local compensation model (ΔX a ,ΔY a ) and (ΔX b ,ΔY b );

[0083] Step 9: Use the selected data to establish the JITL-JYKPLS local model, and use the trust region optimization method to solve the mismatch problem of the JITL-JYKPLS local model;

[0084] S91: According to formula (10), the optimization problem of the JITL-JYKPLS local model is constructed, that is, the optimization compensation problem based on instant learning and trust region;

[0085]

[0086] Where D is the radius of the spherical trust region, g(Δx) = ||Δx|| 2 <D is the trust region constraint;

[0087] S92: By solving formula (10), the optimized compensation value Δx is obtained, and then the manipulated variable trajectory after compensation is obtained

[0088] S93: Predicting the final manipulated variable trajectory x using the JY-KPLS process model q The corresponding comprehensive economic benefit gain If the comprehensive economic benefit gain ΔP pred If formula (11) is satisfied, then select x q As the operation variable trajectory of the batch training, the process output is predicted Get a new model training environment, otherwise, the qth training is deemed unsuccessful and continue to use the operation variable trajectory x of the previous training q-1 As the final operation variable trajectory of the current training, that is, x q =x q-1 ;

[0089]

[0090] In the formula, τ is the comprehensive economic benefit, δ is the threshold;

[0091] S94: Calculate the actual gain ΔP in economic benefits real =P(x (k-1) ,y (k-1) )-P(x (k) ,y (k) ) to update the trust region radius D; if ΔP real <αΔP pred , then let If ΔP real >αΔP pred ,and Then let D q+1 =γD q Otherwise, let D q+1 =D q; Then, perform batch-to-batch deep reinforcement learning optimization according to steps 10 to 13;

[0092] Step 10: Perform trial-and-error training based on the compensated model obtained in step 9 and the optimization system;

[0093] Step 11: After the batch production is completed, determine whether the total reward value of the training in the current batch exceeds the total reward value of the training in the previous batch; if it exceeds, the currently trained optimization system is used for inter-batch optimization; otherwise, the optimization system obtained by the previous batch training is used to replace the current optimization system for inter-batch optimization;

[0094] Step 12: Input the optimized parameters into the controller, use the controller to control the production system to perform production operations, and output the final product quality;

[0095] Step 13: Set k=k+1, update the modeling data, and return to step 1 for the next batch optimization.

[0096] The present invention is described in detail below through examples to verify its feasibility:

[0097] As an important raw material, cobalt metal is widely used in the production and manufacturing of batteries, aerospace materials, and high-temperature resistant materials, and an important production indicator of cobalt metal is its average particle size. With the increasing demand for cobalt metal in the industrial field, the requirements for the quality of cobalt powder are also increasing. When the average particle size of cobalt oxalate is small, it is easy to cause many adverse effects such as filter blockage, increased drying time, and low production efficiency. Therefore, optimizing the cobalt oxalate preparation process and increasing the average particle size of its final product have a great impact on improving production efficiency and product quality. In the hydrometallurgical industry, the cobalt oxalate crystallization process is a typical intermittent production process. Figure 1 This is a schematic diagram of the production process of cobalt oxalate crystals. It mainly involves liquid phase reaction between cobalt chloride and ammonium oxalate, and a series of processes such as multiple reactions, washing, filter pressing and drying are performed to obtain the finished cobalt oxalate product. The most important step is the reaction process between ammonium oxalate and cobalt chloride, and its chemical reaction formula is shown in formula (12):

[0098] CoCl2+(NH4)2C2O4→CoC2O4↓+2NH4Cl (12);

[0099] The cobalt oxalate crystallization process mainly includes two important parts: ammonium oxalate dissolver and crystallizer. The ammonium oxalate crystallization process is carried out in the crystallizer and is constantly stirred. The specific process is as follows:

[0100] (1) Preparation of oxalic acid: a certain amount of solid oxalic acid and a certain amount of pure water are placed in an oxalic acid dissolving kettle, heated with steam until completely dissolved, and filtered to obtain relatively pure oxalic acid.

[0101] (2) Synthesis of ammonium oxalate: Add oxalic acid solution into the ammonium oxalate synthesis kettle, introduce ammonia gas or liquid ammonia, and heat to a certain temperature. There are different requirements for coarse and fine materials.

[0102] (3) Synthesis of cobalt oxalate: First, a certain amount of cobalt chloride solution is placed in a cobalt oxalate synthesis kettle, heated to a certain temperature, and then ammonium oxalate solution is introduced at a certain rate for a certain period of time. The feed is stopped, and the kettle is opened to pass the suspension into a filter press for filtration.

[0103] (4) After filtering, washing and drying once, washing, filtering and drying are performed three times, and the finished product of cobalt oxalate is obtained.

[0104] During the reaction, the reaction temperature is generally kept constant by a PI controller, and the stirring rate of the reactor is generally set to a fixed value. Therefore, the only operating variable that can affect the final cobalt oxalate particle size is the rate at which the ammonium oxalate solution is added into the reactor. Therefore, the input variable of the system is the feed rate of the ammonium oxalate solution; the output variable is the average particle size of the cobalt oxalate. The input and output variables are shown in Table 1:

[0105] Table 1: Input and output variable table

[0106]

[0107] 1) Data generation and model calibration

[0108] This embodiment uses the cobalt oxalate crystallization process as a simulation object to further verify the method of the present invention. By analyzing the cobalt oxalate mechanism process, a mechanism model is established, and the cobalt oxalate synthesis process mechanism model is used to replace the actual production process to provide reasonable modeling data for the data model. The parameter settings of the mechanism model are shown in Table 2.

[0109] Table 2: Mechanism model parameters

[0110]

[0111]

[0112] Among them, K a is the nucleation rate temperature index, K b is the growth rate temperature index, K n is the nucleation rate coefficient, K g is the growth rate coefficient, K v is the shape factor, α is the nucleation rate supersaturation index, β is the growth rate supersaturation index, and γ is the stirring rate index.

[0113] In the process of producing cobalt oxalate, the differences between the new and old batch processes mainly come from the concentration of raw materials and production process parameters, and are largely affected by the environment, geographical location and process. Table 3 shows the numerical range of the input variables of the new and old batch processes.

[0114] Table 3: Change range of input variables of new and old processes

[0115]

[0116]

[0117] As shown in Table 3, for the operation variable to be optimized, the feed rate of ammonium oxalate between batches, a segmented constantization strategy is adopted to divide the change trajectory of the ammonium oxalate feed rate within a batch into 11 constant segments according to the total batch time. In order to obtain training data, a constant value of ±0.0005 m / s is added. 3 / s pseudo-random binary signal to stimulate the dynamic characteristics of the process, to produce the variation characteristics between batches in the actual process. According to experimental measurements, the prediction error converges when the migration number is 40, and the prediction error of subsequent batches changes very little. Therefore, it was finally decided to use 40 batches of old process data and 5 batches of new process data for latent variable process migration modeling.

[0118] 2) Design of Deep Reinforcement Learning Optimization System

[0119] In order to carry out the DDPG optimization algorithm in an orderly manner and ensure that the optimization system as an intelligent agent is effectively trained, the present invention takes the cobalt oxalate synthesis process as the background and carries out the design of rewards and states. In order to ensure the training efficiency of the optimization system and reduce the computational burden of the algorithm, it is necessary to reasonably select the dimension of the state information according to the complexity of the cobalt oxalate synthesis process. Therefore, the state design of the cobalt oxalate synthesis process model is shown in formula (13):

[0120] s i =(x i1 ,x i2 ,...,x i11 ,y i ) (13);

[0121] In the formula, xi=(x i1 ,x i2 ,...,x i11 ) is the ammonium oxalate feed rate for the cobalt oxalate synthesis process, where the component x ik is the operation trajectory corresponding to different time periods in the i-th batch, k = 1, 2, ..., 11. i is the final product mass of the average particle size of cobalt oxalate.

[0122] In deep reinforcement learning, the design of the reward function is also very critical. A reasonable reward function can not only affect the convergence of the algorithm, but also affect the performance of batch optimization in the intermittent process. The design of the reward needs to consider the objective function of traditional batch optimization. In actual optimization, the dynamic optimization problem is usually converted into a static optimization problem by expanding in the batch direction. The specific optimization problem is shown in formula (14):

[0123]

[0124] In the formula, φ(x,y p ) is the cost function of the actual production process design, y p is the final product quality. g(x,y p ) is the constraint function involved in the actual production process. In order to maximize the comprehensive economic benefits of the product and ensure product quality, a reward function as shown in formula (15) is designed:

[0125]

[0126] Where, the subscript pre is the production information of the previous batch during the training process, φ exp is the expected product quality y exp The corresponding cost. The reward r consists of two parts, r1 and r2, where r1 allows the optimization system to continuously optimize the actual production process, and r2 can ensure the final optimization performance. k is the weight corresponding to each reward. In order to speed up the learning efficiency of the optimization system, the present invention adopts a mathematical driving model instead of the actual process as the training interaction object, and formula (15) is approximated to the form of formula (16):

[0127]

[0128] In the formula, x * represents the operation trajectory after compensation, It is the final product quality prediction value based on the data-driven model.

[0129] 3) Neural network parameter setting

[0130] In terms of neural network parameter setting, since the DDPG network is mainly composed of two types of networks, the executor and the evaluator, it is necessary to set the executor network and the evaluator network separately. The number of input neurons and the number of output neurons of the executor network correspond to the dimensions of the state and action respectively, while the number of input neurons of the evaluator network corresponds to the total dimension of the state and action. The number of hidden neurons in each network can be obtained through experimental testing. The specific neural network parameters are shown in Table 4, and the hyperparameters are shown in Table 5.

[0131] Table 4: Neural network parameter settings

[0132]

[0133] Table 5: Hyperparameter settings

[0134]

[0135] 4) Deep reinforcement learning optimization of intermittent processes based on process transfer model

[0136] As an important branch of machine learning, deep reinforcement learning is a product that combines a deep neural network with powerful representation capabilities with reinforcement learning with decision-making capabilities. It is usually used to deal with sequential decision-making problems, and the optimal strategy for a specified task can be finally obtained by using the experience of continuous interaction with the environment and trial and error. The present invention adopts the method of intra-batch training and inter-batch optimization. In order to solve the problem that the new intermittent process is difficult to establish a process model due to the lack of production data in the early stage of its production operation, when the deep reinforcement learning system is trained, the process migration model is used to replace the actual production process for interaction and trial and error to achieve the optimal optimization system strategy, and the process information migration of similar production processes is applied to the modeling process of the new production process by using the kernel partial least squares (Joint-Y Kernel Partial Least Squares, JY-KPLS) method of the joint quality index to assist in the establishment of a new intermittent process quality prediction model. However, due to the blind use of data with large differences from the new process for modeling, the modeling accuracy may be low. This simulation example adopts the idea of ​​real-time learning to perform local modeling of JITL-JY-KPLS. Due to the inherent differences between the old and new processes and the large number of mathematical approximations and assumptions made in the process of model building, the optimal solution obtained by optimizing and solving the model often loses its optimality or even feasibility in the actual production environment. To address this problem, trust region optimization is used in this example to solve the model mismatch problem. By using the optimization compensation of the trust region, the compensated model replaces the original data model for in-batch training, so that the correct reward guidance can be obtained, thereby improving the optimization performance of the deep reinforcement learning optimization system. In order to ensure that the self-migration model can reflect the actual working conditions of the production process in real time, new data filling is adopted to ensure the timeliness of the model. After the current batch of production is completed, the optimization system with better training for adjacent batches is used for process optimization to avoid the optimization system with poorer training from directly performing inter-batch optimization. In order to demonstrate the superiority of the optimization method proposed in the present invention, the optimization effects of the intermittent process trust region optimization method based on JY-KPLS and the intermittent process optimization method based on KPLS are compared. From Figure 2From the analysis of the changes in the cumulative rewards of training in different batches, it can be seen that the batch training method greatly improves the learning efficiency of the optimization system. With the repeated training of each batch of the optimization system and the continuous updating of the self-migration model to approach the actual process, the training effect of the optimization system is also continuously improved. Figure 3 It can be seen from the three different optimization methods that the deep reinforcement learning optimization method can achieve better optimization results while ensuring the quality of the final cobalt oxalate product. It can reduce the number of iterations required for the optimization process, give priority to finding the optimal solution, and significantly improve the optimization efficiency.

Claims

1. A deep reinforcement learning optimization compensation method for nonlinear intermittent processes, comprising an intermittent production process A and an intermittent production process B, wherein the intermittent production process A is an old production process that has been produced for a long time and has sufficient historical operation data and operation control experience, and the intermittent production process B is a new production process that is about to be put into production; at the same time, the intermittent production process A and the intermittent production process B have the same or similar production equipment, and the final products of the intermittent production process A and the intermittent production process B belong to the same product family, characterized in that: The following steps are involved: Step 1: Expand the three-dimensional input data matrix of intermittent production process A and intermittent production process B into two-dimensional matrices according to the batch direction, namely X a , X b ; Step 2: X a , X b Each column of is processed with zero mean and unit variance, and at the same time, the output data matrix Y a , Y b Zero mean and unit variance treatment was performed; Step 3: Construct JY-KPLS process model; S31: Let i=1, K ai =K a , K bi =K b , Y Ji =Y J , where i represents the order of extracting latent variables; S32: From the joint output matrix Y Ji Select any column as the initial u Ji , where u Ji =[u ai ;u bi ]; S33: Calculate φ according to formula (1) ai The score vector t ai , calculate φ according to formula (2) bi The score vector t bi : S34: Calculate the common load matrix q of the joint output variables according to formula (3) Ji , and normalize the load matrix q Ji =q Ji / ||q Ji ||, where q Ji T q Ji =1; Where, t Ji =[t ai ;t bi ]; S35: Calculate Y ai The score vector u ai =Y ai q Ji , calculate Y bi The score vector u bi =Y bi q Ji ; S36: Determine u ai and u bi Whether it converges, if so, proceed to step S37, otherwise return to S33; S37: Reduce the kernel matrix and the joint output matrix according to formulas (4), (5) and (6) respectively to obtain the corresponding residual matrix K ai+1 , K bi+1 and K Ji+1 ; S38: let i=i+1, and repeat S32 to S37 until n pivots are extracted; S39: Based on the known kernel matrix K b And the obtained score matrix T b with U b , according to formula (7), the output of the JY-KPLS process model is calculated In the formula, is the joint matrix of the output variables of intermittent production process A and production process B, is the joint matrix of latent variables of intermittent production process A and production process B; Step 4: Construct the optimization problem of the JY-KPLS process model according to formula (8); Step 5: Solve the optimization problem in formula (8) and obtain the solution x cur ; Step 6: x cur As the query point x q , use formula (9) to calculate the similarity s(x a ,x q )、s(x b ,x q ); Where τ∈[0,1] is the weight parameter, d(x q ,x i ) is x q and x i The Euclidean distance between i Denotes Δx q and Δx i The angle between q =x q -x q-1 , Δx i =x i -x i-1 ; Step 7: According to the similarity, m old data are selected from the old process data set, and n new data are selected from the new process data set; Step 8: Subtract the selected data from the current query data and use the obtained deviation samples as the data set for building the local compensation model (ΔX a ,ΔY a ) and (ΔX b ,ΔY b ); Step 9: Use the selected data to establish the JITL-JYKPLS local model, and use the trust region optimization method to solve the mismatch problem of the JITL-JYKPLS local model; S91: According to formula (10), the optimization problem of the JITL-JYKPLS local model is constructed, that is, the optimization compensation problem based on instant learning and trust region; Where D is the radius of the spherical trust region, g(Δx) = Δx 2 <D is the trust region constraint; S92: By solving formula (10), the optimized compensation value Δx is obtained, and then the manipulated variable trajectory after compensation is obtained S93: Predicting the final manipulated variable trajectory x using the JY-KPLS process model q The corresponding comprehensive economic benefit gain If the comprehensive economic benefit gain ΔP pred If formula (11) is satisfied, then select x q As the operation variable trajectory of the batch training, the process output is predicted Get a new model training environment, otherwise, the qth training is deemed unsuccessful and continue to use the operation variable trajectory x of the previous training q-1 As the final operation variable trajectory of the current training, that is, x q =x q-1 ; In the formula, τ is the comprehensive economic benefit, δ is the threshold; S94: Calculate the actual gain ΔP in economic benefits real =P(x (k-1) ,y (k-1) )-P(x (k) ,y (k) ) to update the trust region radius D; if ΔP real <αΔP pred , then let If ΔP real >αΔP pred ,and Then let D q+1 =γD q Otherwise, let D q+1 =D q ; Step 10: Perform trial-and-error training based on the compensated model obtained in step 9 and the optimization system; Step 11: After the batch production is completed, determine whether the total reward value of the training in the current batch exceeds the total reward value of the training in the previous batch; if it exceeds, the currently trained optimization system is used for inter-batch optimization; otherwise, the optimization system obtained by the previous batch training is used to replace the current optimization system for inter-batch optimization; Step 12: Input the optimized parameters into the controller, use the controller to control the production system to perform production operations, and output the final product quality; Step 13: Set k=k+1, update the modeling data, and return to step 1 for the next batch optimization.

Images