A deep reinforcement learning optimization compensation method for a nonlinear batch process

Abstract

A deep reinforcement learning optimization compensation method for nonlinear intermittent processes, which expands a three-dimensional input data matrix into a two-dimensional matrix; performs standardization processing; constructs a JY-KPLS model; constructs a JY-KPLS model optimization problem; solves the optimization problem; calculates the similarity of historical and query data; according to the similarity, m old data and n new data are selected from the old and new process data sets respectively, and the difference is calculated with the current query data; the deviation sample is used as the data set to establish a JITL-JYKPLS local model to solve the mismatch problem; the compensated model and the optimization system are interacted and trial-and-error trained; if the total reward value of the current batch training exceeds the total reward value of the previous batch training, the optimized system of the current training is used for batch-to-batch optimization; otherwise, the optimization system of the previous batch is used for batch-to-batch optimization; the final product quality is output. The method can significantly improve the quality of the final product.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of operation optimization of industrial production processes, and particularly relates to a deep reinforcement learning optimization compensation method for a nonlinear batch process. BACKGROUND

[0002] In process industry, batch process as an important branch has the characteristics of "multiple time-varying", and is the production mode of many high value-added, small-scale and multi-operation stage products. This multi-process and small-scale production mode can well adapt to the rapid changes in market demand in modern process industry. In the production fields of fine chemical industry, biopharmaceutical industry, food industry, polymer reaction and metal processing, batch process has a very important position and has been widely praised. With the continuous development of industry, the increasingly fierce market competition environment puts forward higher requirements for enterprises and products, and efficient and stable production process is paid more and more attention. In order to further improve product quality and production efficiency, the operation optimization of production process gradually becomes the focus of research. For batch process, due to the existence of nonlinearity, time-varying and uncertainty, how to maximize the total benefit while ensuring product quality has extremely important practical significance. The research on operation optimization is usually based on the quality prediction model, which can be divided into two categories: mechanism model and data-driven model. However, for a more complex industrial system, it is usually difficult to build an accurate and reliable mechanism model. Therefore, such a mechanism model-based method is difficult to be widely used in modern industry. The data-driven modeling method is favored because of its characteristics of not relying on mechanism knowledge, easy to establish model and high model accuracy. However, for a new batch production process just put into production, due to the short running time of the production process, it is usually difficult to obtain sufficient process data. In order to establish its mathematical model, it is often necessary to accumulate a large amount of production data by running the production process, which will consume a lot of manpower and financial resources, with high cost and low modeling efficiency. Therefore, how to ensure the accuracy of the model while effectively reducing the cost of modeling has become a problem to be solved.

[0003] Since the transfer learning was introduced into the field of machine learning in the 1990s, it has attracted much attention and research, and has made great progress. Through transferring the learned knowledge of related tasks to improve the learning process of new tasks, it is particularly useful for overcoming the limitations of highly specialized but small-scale data sets. In actual production process, there are similar processes with similar process mechanisms and production processes, which may be similar in mechanism, have the same or similar production equipment, or have common process and quality variables. The idea of establishing a process transfer model is to use the production processes similar to the target process and with sufficient process data in actual production to assist the target process lacking production data in establishing a quality prediction model.

[0004] With the increasingly fierce market competition and the increasing expansion of production scale, the efficiency of intermittent process operation optimization is also paid more and more attention, and the research on operation optimization is of great significance to improve the quality of final products and the competitiveness of enterprises. Unlike the research on maximizing the quality of final products as the target of the conventional process, the optimization research of high-cost process not only needs to ensure the feasibility of the optimization strategy, but also needs to consider minimizing the optimization cost to ensure the orderly development of subsequent operation. However, although the data-driven model does not have the problem of structural mismatch of the mechanism model, there is still a certain deviation between the model and the actual process, that is, there is a process and model mismatch problem. Especially for process migration models, although similar intermittent processes have the same or similar physical and chemical principles, there are inevitably some differences. These differences will lead to more serious mismatch problems between the process migration model and the actual process, and the NCO mismatch problem of the model in optimization is more prominent, and finally the optimization result obtained by the process migration model is only a suboptimal solution. Therefore, in order to improve the product quality, improve the production efficiency and the comprehensive economic benefit of enterprises, it is necessary to solve the model mismatch problem and optimize the quality of the intermittent process. SUMMARY

[0005] In view of the problems existing in the prior art, the present application provides a deep reinforcement learning optimization compensation method for a nonlinear intermittent process, which can realize efficient and low-cost optimization of the intermittent process under the condition of less data, can solve the problem that it is difficult to establish a process model for a new intermittent process in its initial production and operation period due to the lack of production data, can effectively reduce the production cost of the intermittent process, and can solve the model mismatch problem, thereby significantly improving the quality of the final product and the economic benefit of the enterprise.

[0006] In order to achieve the above purpose, the present application provides a deep reinforcement learning optimization compensation method for a nonlinear intermittent process, which comprises an intermittent production process A and an intermittent production process B, the intermittent production process A is an old production process that has been produced for a long time and has sufficient historical operation data and operation control experience, and the intermittent production process B is a new production process to be put into production; at the same time, the intermittent production process A and the intermittent production process B have the same or similar production equipment, and the final products of the intermittent production process A and the intermittent production process B belong to a product family, comprising the following steps:

[0007] Step 1: The three-dimensional input data matrix of the intermittent production process A and the intermittent production process B is unfolded into a two-dimensional matrix according to the batch direction, respectively X a 、X b ;

[0008] Step 2: X a 、Xb each column of the matrix Y a is zero-mean and unit-variance processed, and the output data matrix Y b is zero-mean and unit-variance processed;

[0009] Step three: constructing a JY-KPLS process model;

[0010] S31: Let i=1, K ai =K a , K bi =K b , Y Ji =Y J , where i represents the order of extracting latent variables;

[0011] S32: Select any column from the joint output matrix Y Ji as the initial u Ji , where u Ji =[u ai ;u bi ];

[0012] S33: Calculate the score vector t ai of φ ai according to formula (1), and calculate the score vector t bi of φ bi according to formula (2):

[0013]

[0014] S34: Calculate the common load matrix q Ji of the joint output variable according to formula (3), and normalize the load matrix q Ji =q Ji / ||q Ji ||, where q Ji T q Ji =1;

[0015]

[0016] In the formula, t Ji =[t ai ; t bi ];

[0017] S35: Calculate the score vector u ai of Y ai =Y ai q Ji , and calculate the score vector u bi of Y bi =Y bi q Ji ;

[0018] S36: judge whether u ai and u bi converge, if yes, go to step S37, otherwise, return to step S33;

[0019] S37: reduce the kernel matrix and the joint output matrix according to formula (4), (5) and (6) respectively to obtain the corresponding residual matrices K ai+1 , K bi+1 and K Ji+1 ;

[0020]

[0021] S38: let i = i + 1, repeat steps S32 to S37 until n principal components are extracted;

[0022] S39: based on the known kernel matrix K b and the obtained score matrix T b and U b , calculate the output of the JY-KPLS process model according to formula (7)

[0023]

[0024] wherein, is the joint matrix of the output variables of the batch production process A and the production process B, is the joint matrix of the latent variables of the batch production process A and the production process B;

[0025] Step four: construct the optimization problem of the JY-KPLS process model according to formula (8);

[0026]

[0027] Step five: solve the optimization problem in formula (8) and obtain the solution x cur of the optimization problem;

[0028] Step six: take x cur as the query point x q , and use formula (9) to calculate the similarity s(x a , x q ), s(x b , x q ) between the historical data and the query data;

[0029]

[0030] wherein, τ ∈ [0, 1] is a weight parameter, d(x q , x i ) is the Euclidean distance between xq and x i The Euclidean distance between them, θ i Indicate Δx q and Δx i The angle between them, Δx q =x q -x q-1 Δx i =x i -x i-1 ;

[0031] Step 7: Based on the similarity, select m old data entries from the old process dataset and n new data entries from the new process dataset;

[0032] Step 8: Subtract the filtered data from the current query data, and use the resulting deviation sample as the dataset (ΔX) for constructing the local compensation model. a ,ΔY a ) and (ΔX b ,ΔY b );

[0033] Step 9: Build a JITL-JYKPLS local model using the selected data, and use the trust region optimization method to solve the problem of mismatch in the JITL-JYKPLS local model;

[0034] S91: The optimization problem of the JITL-JYKPLS local model is constructed according to formula (10), that is, the optimization compensation problem based on real-time learning and trust region.

[0035]

[0036] In the formula, D is the radius of the spherical trust region, and g(Δx) = ||Δx|| 2 <D represents a trust region constraint;

[0037] S92: The optimized compensation value Δx is obtained by solving formula (10), and then the trajectory of the compensated operation variable is obtained.

[0038] S93: Predicting the final operational variable trajectory x using the JY-KPLS process model q Corresponding comprehensive economic benefit gain If the overall economic benefit gain ΔP pred If formula (11) is satisfied, then choose x. q The trajectory serves as the operational variable for in-batch training, thereby predicting the process output. A new model training environment is required; otherwise, the q-th training iteration is considered unsuccessful, and the trajectory of the operational variable x from the previous training iteration is continued. q-1 The trajectory, i.e., x, is the final operational variable in the current training iteration.q = x q-1 ;

[0039]

[0040] wherein τ is the comprehensive economic benefit, and δ is the threshold value;

[0041] S94: calculating the actual gain ΔP of the economic benefit real = P(x (k-1) ,y (k-1) )-P(x (k) ,y (k) ), to update the radius D of the trust region; if ΔP real < αΔP pred , then if ΔP real > αΔP pred , and then D q+1 = γD q ; otherwise, D q+1 = D q ;

[0042] Step ten: interacting with the optimization system according to the compensated model obtained in step nine, and trial-and-error training;

[0043] Step eleven: after the batch production is completed, determining whether the total reward value of the current batch training exceeds the total reward value of the previous batch training; if yes, using the current trained optimization system for batch-to-batch optimization; otherwise, using the optimization system trained in the previous batch to replace the current optimization system for batch-to-batch optimization;

[0044] Step twelve: inputting the optimized parameters into the controller, and using the controller to control the production system to perform production operations, and outputting the final product quality;

[0045] Step thirteen: setting k = k + 1, updating the modeling data, and returning to step one for the next batch optimization.

[0046] The method adopts the joint quality index kernel partial least squares method (Joint-Y Kernel Partial Least Squares, JY-KPLS) and the just in time learning (Just in Time Learning, JITL) idea, uses the production information of similar processes widely existing in the industrial process to assist in establishing a new process model, and uses the deep deterministic policy gradient algorithm (Deep Deterministic Policy Gradient, DDPG) to perform intermittent process operation optimization. For batch-to-batch optimization, the optimization system trained better is used for optimization, avoiding the case that the optimization system trained worse directly performs batch-to-batch optimization. At the same time, in order to ensure that the process migration model can reflect the working condition of the actual production process in real time, new data is used for filling and updating, ensuring the timeliness of the model. For the training process within the batch, firstly, based on the idea of transfer learning, the joint quality index kernel partial least squares (Joint-Y Kernel Partial Least Squares, JY-KPLS) method is used to migrate similar old production process data to the modeling process of the new production process, to assist in the establishment of the quality prediction model of the new intermittent process, thereby solving the problem that the new production process data is less and difficult to accurately establish the model, greatly reducing the modeling cost. However, if data with large differences from the new process is blindly used for modeling, it may limit the improvement of the precision of the migration model. Therefore, the invention uses the just in time learning (Just in Time Learning, JITL) method to extract useful data information from the new and old process data sets, and limits the data distribution range to establish a JITL-JY-KPLS local model. Due to the inherent differences between the new and old processes, as well as the interference of unknown factors such as model uncertainty and noise, it is difficult to avoid the mismatch between the process migration model and the factory, which will affect the realization of the subsequent optimization performance. At the same time, the modifier adaptation (Modifier Adaptation, MA) method is used to solve the gradient to compensate for the difference between the model and the actual process, which makes it difficult to implement the process due to the difficulty in solving the gradient information in the strong nonlinear intermittent process. In view of these problems, the invention uses the trust region (Trust Region, TR) method to compensate the process migration model, to effectively solve the model mismatch problem, by compensating the model based on the batch training, and then correcting the reward value of the intelligent agent training, further improving the optimization performance of the deep reinforcement learning, and improving the final quality of the product and the efficiency of the optimization process, so as to improve the comprehensive economic benefit of the enterprise.

[0047] The method in the application can realize high efficiency and low cost optimization of intermittent process in the case of little data, can solve the problem that it is difficult to establish a process model due to lack of production data at the initial stage of production operation of a new intermittent process, can effectively reduce the production cost of the intermittent process, can solve the problem of model mismatch, and can significantly improve the quality of the final product and the economic benefit of the enterprise. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 A schematic diagram of the production process of cobalt oxalate crystals in the application;

[0049] Figure 2 A cumulative reward curve diagram of training in different batches in the application;

[0050] Figure 3 A performance comparison curve diagram of different optimization strategies in the application;

[0051] Figure 4 A schematic diagram of the ammonium oxalate feeding rate of different batches in the application;

[0052] Figure 5 A flowchart of the application. DETAILED DESCRIPTION

[0053] The application will be further described below in combination with the drawings.

[0054] As Figures 1 to 5As shown, the present application provides a deep reinforcement learning optimization compensation method for a nonlinear batch process, which uses a deep deterministic policy gradient (DDPG) algorithm for batch process operation optimization. The process mainly includes two parts: intra-batch training and inter-batch optimization. Inter-batch optimization uses the better optimization system of adjacent batches for optimization to avoid the direct inter-batch optimization of the poor optimization system. In order to ensure that the process migration model can reflect the actual production process conditions in real time, new data is used for filling and updating to ensure the timeliness of the model. In intra-batch training, first, the joint quality index kernel partial least squares (JY-KPLS) method is used to migrate similar old production process data to the new production process, thereby solving the problem of insufficient new production process data and difficulty in accurately establishing a model, and reducing the modeling cost. However, if data with large differences from the new process is blindly used for modeling, it may limit the improvement of the migration model accuracy. Therefore, the just in time learning (JITL) method is used to extract useful data information from the new and old process data sets, and a local model is established by limiting the data distribution range. Due to the uncertainty of the model, noise and other unknown factors, the mismatch between the process migration model and the factory is difficult to avoid, which affects the realization of the subsequent optimization performance. At the same time, the modifier adaptation (MA) method is used to solve the gradient to compensate for the difference between the model and the actual process. In a strongly nonlinear batch process, it is difficult to solve the gradient information, and this method is difficult to implement. The present application uses the trust region (TR) gradient-free optimization method to solve the model mismatch problem, and finally realizes the modification of the reward value of the intelligent agent training, and further improves the optimization performance of the deep reinforcement learning.

[0055] Specifically, two similar batch production processes A and batch production processes B are included. For a typical batch production process, it has J process variables, K sampling time points in a batch, and I batch numbers are collected, so that a typical three-dimensional input data matrix X R I×J×K and a two-dimensional output data matrix Y R I×K, R represents the real number set. The batch production process A is an old production process with a long production time, sufficient historical operation data and operation control experience, the batch production process B is a new production process to be put into production, the production mechanism between the batch production process A and the batch production process B is similar, the batch production process A and the batch production process B have the same or similar production equipment, and the final products of the batch production process A and the batch production process B belong to a product family, therefore, the operation data and control experience of the batch production process A will help to establish the production model of the batch production process B and optimize the operation of the batch production process B;

[0056] The method comprises the following steps:

[0057] Step one: the three-dimensional input data matrix of the batch production process A and the batch production process B is unfolded into a two-dimensional matrix according to the batch direction, respectively X a , X b ;

[0058] Step two: the columns of X a , X b are processed with zero mean and unit variance, and the output data matrix Y a , Y b is processed with zero mean and unit variance;

[0059] Step three: constructing a JY-KPLS process model;

[0060] S31: let i=1, K ai =K a , K bi =K b , Y Ji =Y J , wherein i represents the order of extracting latent variables;

[0061] S32: selecting any column from the joint output matrix Y Ji as the initial u Ji , wherein u Ji =[u ai ;u bi ];

[0062] S33: calculating the score vector t ai of φ ai according to formula (1), and calculating the score vector t bi of φ bi according to formula (2):

[0063]

[0064] S34: calculating the common load matrix q of the joint output variable according to formula (3)Ji and normalize the loading matrix q Ji = q Ji / ||q Ji ||, where q Ji T q Ji = 1.

[0065]

[0066] where t Ji = [t ai ; t bi ];

[0067] S35: Calculate the score vector u ai of Y ai = Y ai q Ji , and calculate the score vector u bi of Y bi = Y bi q Ji ;

[0068] S36: Determine whether u ai and u bi converge, if yes, go to step S37, otherwise return to step S33;

[0069] S37: Reduce the kernel matrix and the joint output matrix according to formulas (4), (5) and (6) respectively to obtain the corresponding residual matrices K ai+1 , K bi+1 and K Ji+1 ;

[0070]

[0071] S38: Let i = i + 1, repeat steps S32 to S37 until n principal components are extracted; the number of principal components n can be determined by cross-validation (determine whether the predictive ability of the model after introducing a new component is statistically improved by the prediction residual sum of squares of the validation set, thereby giving the optimal number of principal components n);

[0072] S39: Based on the known kernel matrix K b and the obtained score matrix T b and U b , calculate the output of the JY-KPLS process model according to formula (7)

[0073]

[0074] where the kernel principal component matrix is the joint matrix of the output variables of batch process A and batch process B, is the joint matrix of the latent variables of batch process A and batch process B, thus, the JY-KPLS process model is completed; then, steps four to nine are performed for intra-batch training;

[0075] Step four: construct the optimization problem of the JY-KPLS process model according to formula (8);

[0076]

[0077] Step five: solve the optimization problem in formula (8) and obtain the solution x cur of the optimization problem;

[0078] Step six: take x cur as the query point x q , and use formula (9) to calculate the similarity s(x a , x q ), s(x b , x q ) between the historical data and the query data;

[0079]

[0080] wherein, τ ∈ [0, 1] is a weight parameter, d(x q , x i ) is the Euclidean distance between x q and x i , θ i represents the angle between Δx q and Δx i , Δx q = x q - x q-1 , Δx i = x i - x i-1 ;

[0081] Step seven: according to the similarity size, select m pieces of old data from the old process data set and select n pieces of new data from the new process data set;

[0082] Step eight: use the selected data and the current query data to perform difference, and obtain the deviation samples as the data set (ΔX a , ΔY a ) and (ΔX b , ΔY b ) for constructing the local compensation model;

[0083] Step 9: Build a JITL-JYKPLS local model using the selected data, and use the trust region optimization method to solve the problem of mismatch in the JITL-JYKPLS local model;

[0084] S91: The optimization problem of the JITL-JYKPLS local model is constructed according to formula (10), that is, the optimization compensation problem based on real-time learning and trust region.

[0085]

[0086] In the formula, D is the radius of the spherical trust region, and g(Δx) = ||Δx|| 2 <D represents a trust region constraint;

[0087] S92: The optimized compensation value Δx is obtained by solving formula (10), and then the trajectory of the compensated operation variable is obtained.

[0088] S93: Predicting the final operational variable trajectory x using the JY-KPLS process model q Corresponding comprehensive economic benefit gain If the overall economic benefit gain ΔP pred If formula (11) is satisfied, then choose x. q The trajectory serves as the operational variable for in-batch training, thereby predicting the process output. A new model training environment is required; otherwise, the q-th training iteration is considered unsuccessful, and the trajectory of the operational variable x from the previous training iteration is continued. q-1 The trajectory, i.e., x, is the final operational variable in the current training iteration. q =x q-1 ;

[0089]

[0090] In the formula, τ represents the comprehensive economic benefit, and δ represents the threshold.

[0091] S94: Calculate the actual gain ΔP for economic benefits real =P(x (k-1) ,y (k-1) )-P(x (k) ,y (k) ), to update the trust region radius D; if ΔP real <αΔP pred Then let If ΔP real >αΔP pred ,and Let D q+1 =γD q Otherwise, let D q+1 =D q; then, batch deep reinforcement learning optimization is performed according to steps ten to thirteen;

[0092] Step ten: interact with the optimization system according to the compensated model obtained in step nine, and perform trial and error training;

[0093] Step eleven: after the batch production is completed, it is determined whether the total reward value in the current batch training exceeds the total reward value in the previous batch training; if yes, the current trained optimization system is used for batch-to-batch optimization; otherwise, the optimization system trained in the previous batch is used to replace the current optimization system for batch-to-batch optimization;

[0094] Step twelve: input the optimized parameters into the controller, and use the controller to control the production system to perform production operation, and output the final product quality;

[0095] Step thirteen: let k = k + 1, update the modeling data, and return to step one for the next batch optimization.

[0096] The present application will be described in detail by examples as follows, and its feasibility will be verified:

[0097] Metal cobalt is widely used in the production of batteries, aerospace materials, and high-temperature resistant materials as an important raw material, and an important production index of metal cobalt is its average particle size. With the increasing demand for metal cobalt in the industrial field, the quality requirements for cobalt powder are also increasing. When the average particle size of cobalt oxalate is small, it is easy to cause filter screen blockage, increase drying time, and low production efficiency, and many other adverse effects. Therefore, optimizing the preparation process of cobalt oxalate and improving the average particle size of the final product have a great influence on improving the production efficiency and product quality. In the hydrometallurgical industry, the cobalt oxalate crystallization process is a typical batch production process, Figure 1 is a schematic diagram of the cobalt oxalate crystallization process. It mainly reacts liquid phase of cobalt chloride and ammonium oxalate, through a series of processes such as multiple reactions, washing, pressure filtration and drying, to obtain cobalt oxalate finished product, wherein the most important step is the reaction process of ammonium oxalate and cobalt chloride, and the chemical reaction formula is shown as formula (12):

[0098] CoCl2+ (NH4) 2C2O4→ CoC2O4↓ + 2NH4Cl (12);

[0099] The cobalt oxalate crystallization process mainly includes two important parts: ammonium oxalate dissolver and crystallizer. The ammonium oxalate crystallization process is carried out in the crystallizer and through constant stirring, the specific process is as follows:

[0100] (1) Preparation of oxalic acid: a certain amount of solid oxalic acid and a certain amount of pure water are put into an oxalic acid dissolving kettle, steam heating is performed until complete dissolution, pressure filtration is performed, and relatively pure oxalic acid is obtained.

[0101] (2) Synthesis of ammonium oxalate: oxalic acid solution is added to the ammonium oxalate synthesis kettle, ammonia gas is introduced or liquid ammonia is added, and the temperature is heated to a certain temperature, which has different requirements for coarse and fine materials.

[0102] (3) Synthesis of cobalt oxalate: a certain amount of cobalt chloride solution is first put into the cobalt oxalate synthesis kettle, heated to a certain temperature, which has different requirements for coarse and fine materials, then a certain rate of ammonium oxalate solution is introduced, and the process is continued for a certain time, the feeding is stopped, and the suspension is introduced into the filter press for pressure filtration.

[0103] (4) After one-time pressure filtration, washing and drying, the finished product cobalt oxalate is obtained after three times of washing, three times of pressure filtration and drying.

[0104] During the reaction process, the reaction temperature is generally kept constant by using a PI controller, and the stirring speed of the reaction kettle is also generally set to a fixed value, so the only operation variable that can affect the final cobalt oxalate particle size is the rate at which the ammonium oxalate solution is added to the reaction kettle. Therefore, the input variable of the system is the feeding rate of the ammonium oxalate solution; the output variable is the average particle size of the cobalt oxalate. The input and output variable conditions are shown in Table 1:

[0105] Table 1: Input and output variable table

[0106]

[0107] 1) Data generation and model correction

[0108] This embodiment uses the cobalt oxalate crystallization process as a simulation object to further verify the method of the present application. By analyzing the mechanism of cobalt oxalate, a mechanism model is established, and the mechanism model of the cobalt oxalate synthesis process is used instead of the actual production process to provide reasonable modeling data for the data model. The parameter settings of the mechanism model are shown in Table 2.

[0109] Table 2: Parameter table of mechanism model

[0110]

[0111]

[0112] wherein K a is the nucleation rate temperature index, K b is the growth rate temperature index, K n is the nucleation rate coefficient, K g is the growth rate coefficient, K v is the shape factor, α is the nucleation rate supersaturation index, β is the growth rate supersaturation index, and γ is the stirring rate index.

[0113] The differences between the old and new batch processes mainly come from the concentration of raw materials and the production process parameters, and are largely influenced by the environment and geographical location and the process. Table 3 shows the numerical range of input variables of the old and new batch processes.

[0114] Table 3: Change range table of input variables of old and new processes

[0115]

[0116]

[0117] As shown in Table 3, for the operation variable ammonium oxalate batch feed rate to be optimized, a segmented constant treatment strategy is adopted, and the ammonium oxalate feed rate change trajectory in a batch is divided into 11 constant segments according to the total length of the batch time. In order to obtain training data, the dynamic characteristics of the process are excited by adding a pseudo-random binary signal with ±0.0005 m 3 / s to generate the change characteristics between batches in the actual process. According to experimental measurement, the prediction error converges when the migration number is 40, and the prediction error of the subsequent batches changes little. Therefore, finally, 40 batches of old process data and 5 batches of new process data are used for latent variable process migration modeling.

[0118] 2) Design of deep reinforcement learning optimization system

[0119] In order to orderly carry out the DDPG optimization algorithm and ensure that the optimization system as an agent is effectively trained, the present application designs the reward and state based on the cobalt oxalate synthesis process. In order to ensure the training efficiency of the optimization system and reduce the calculation burden of the algorithm, it is necessary to reasonably select the dimension of state information according to the complexity of the cobalt oxalate synthesis process, so the state design of the cobalt oxalate synthesis process model is shown in formula (13):

[0120] s i =(x i1 ,x i2 ,...,x i11 ,y i ) (13);

[0121] In the formula, xi=(x i1 ,x i2 ,...,x i11 ) is the ammonium oxalate feed rate of the cobalt oxalate synthesis process, wherein the component x ik is the operation trajectory corresponding to different time periods in the i th batch, k = 1, 2, …, 11. y i is the final product quality of the average particle size of cobalt oxalate.

[0122] In deep reinforcement learning, the design of reward function is also very critical. A reasonable reward function can not only affect the convergence of the algorithm, but also affect the performance of intermittent process batch optimization. The design of reward needs to consider the objective function of traditional batch optimization. In actual optimization, the dynamic optimization problem is usually converted into a static optimization problem in the direction of batch expansion. The specific optimization problem is shown in formula (14):

[0123]

[0124] In the formula, φ(x,y p ) is the cost function of the actual production process design, y p is the final product quality. g(x,y p ) is the constraint function involved in the actual production process. In order to maximize the overall economic benefit of the product and ensure the quality of the product, a reward function as shown in formula (15) is designed:

[0125]

[0126] In the formula, the subscript pre is the production information of the previous batch in the training process, φ exp is the cost corresponding to the expected product quality y exp . The reward r is composed of two parts r1 and r2, wherein r1 can allow the optimization system to continuously optimize the actual production process, and r2 can ensure the final optimization performance. a k is the weight corresponding to each reward. In order to speed up the learning efficiency of the optimization system, the present application uses a mathematical driving model instead of the actual process as the training interactive object, and formula (15) is approximated to the form of formula (16):

[0127]

[0128] In the formula, x * represents the compensated operation trajectory, is the final product quality prediction value based on the data-driven model.

[0129] 3) Neural network parameter setting

[0130] In terms of neural network parameter setting, since the DDPG network is mainly composed of two types of networks, namely the executor and the evaluator, the executor network and the evaluator network need to be set respectively. The input neuron number and the output neuron number of the executor network correspond to the dimension of the state and the action respectively, and the input neuron number of the evaluator network corresponds to the total dimension of the state and the action. The number of hidden neurons of each network can be obtained through experimental test. The specific neural network parameters are shown in Table 4, and the hyperparameters are shown in Table 5.

[0131] Table 4: Neural network parameter setting

[0132]

[0133] Table 5: Hyperparameter settings

[0134]

[0135] 4) Deep reinforcement learning optimization for batch processes based on process migration model

[0136] Deep reinforcement learning, as an important branch of machine learning, is a combination of deep neural networks with powerful representation capabilities and reinforcement learning with decision-making capabilities. It is usually used to handle sequential decision-making problems, and through continuous interaction with the environment and trial-and-error experience, it can ultimately obtain the optimal strategy for a specified task. The present invention adopts the method of training within batches and optimizing between batches. To solve the problem that new batch processes lack production data at the initial stage of production, making it difficult to establish a process model, a process migration model is used to replace the actual production process for interaction and trial-and-error during the training of the deep reinforcement learning system, in order to achieve the optimal optimization system strategy. The Joint-Y Kernel Partial Least Squares (JY-KPLS) method is used to migrate process information from similar production processes to the modeling process of the new production process to assist in the establishment of the quality prediction model of the new batch process. However, due to the blind use of data that is significantly different from the new process for modeling, it may result in low modeling accuracy. The JITL-JY-KPLS local modeling is adopted in this simulation example. Due to the inherent differences between the new and old processes, as well as the large number of mathematical approximations and assumptions made during the modeling process, the optimal solution obtained by optimization based on the model often loses optimality or even feasibility in the actual production environment. To solve this problem, the trust region optimization is used in this example to solve the model mismatch problem. By using the optimization compensation of the trust region, the compensated model replaces the original data model for training within the batch, so that the correct reward guidance can be obtained, and the optimization performance of the deep reinforcement learning optimization system is improved. In order to ensure that the self-migration model can reflect the actual production process in real time, the new data filling method is used to ensure the timeliness of the model. After the current batch production is completed, the optimization system with better training performance is used for process optimization to avoid direct batch-to-batch optimization of the optimization system with poor training performance. In order to demonstrate the superiority of the optimization method proposed in this invention, the optimization effect of the JY-KPLS-based batch process trust region optimization method and the KPLS-based batch process optimization method is compared. From the comparison results, it can be seen that the optimization performance of the deep reinforcement learning optimization system based on the process migration model is better than that of the JY-KPLS-based batch process trust region optimization method and the KPLS-based batch process optimization method. Figure 2The analysis of the cumulative reward changes of the training within different batches can be obtained that the training within batches greatly improves the learning efficiency of the optimization system, and with the repeated training of each batch of the optimization system and the continuous updating of the self-migration model to approach the actual process, the training effect of the optimization system is also continuously improved. From Figure 3 As can be seen from the three different optimization methods, the deep reinforcement learning optimization method can better optimize while ensuring the quality of the final cobalt oxalate product. It can reduce the required iteration number in the optimization process, preferentially find the optimal solution, and significantly improve the optimization efficiency.

Claims

1. A deep reinforcement learning optimization compensation method for nonlinear batch processes, comprising a batch production process A and a batch production process B, wherein the batch production process A is an old production process with a long production time, sufficient historical operation data and operation control experience, and the batch production process B is a new production process to be put into production; meanwhile, the batch production process A and the batch production process B have the same or similar production equipment, and the final products of the batch production process A and the batch production process B belong to a product family, and the method is characterized in that, Comprising the following steps: Step one: the three-dimensional input data matrix of batch process A and batch process B is unfolded into a two-dimensional matrix according to the batch direction, respectively X a , X b ; Step two: for each column of X a , X b , perform zero mean and unit variance, and for each column of output data matrix Y a , Y b , perform zero mean and unit variance; Step three: build JY-KPLS process model; S31: Let , , , wherein, denotes the order of extracting the latent variables; S32: Select any column from the joint output matrix as the initial wherein ; S33: Calculate the score vector of the formula (1) S34: Calculate the score vector of the formula (2) :​​ （1）； （2）； S34: Calculate the common loading matrix of the joint output variables according to formula (3) and normalize the loading matrix where ; （3）； In the formulae, ​ S35: compute the score vector , compute the score vector ; S36: judging and whether it converges, if it converges, entering step S37, otherwise returning to S33; S37: reduce the kernel matrix and the joint output matrix according to formulas (4), (5) and (6) respectively to obtain corresponding residual matrices , and ; （4）； （5）； （6）； S38: Let S32 to S37 until the n leading elements are extracted; S39: Based on the known kernel matrix and the obtained score matrix with , the output of the JY-KPLS process model is calculated according to formula (7) ; （7）； wherein , is the joint matrix of the output variables of batch production process A and production process B, is the joint matrix of the latent variables of batch production process A and production process B; Step four: according to formula (8) to construct the optimization problem of JY-KPLS process model; （8）； Step five: Solve the optimization problem in equation (8) and obtain the solution of the optimization problem x cur ; Step six: Calculate the similarity between the query point and the historical data x cur As the query point x q , the similarity between the historical data and the query data is calculated using formula (9) ; （9）； wherein is a weight parameter, is and the Euclidean distance between denotes and the angle between , ; Step seven: according to the similarity, select m old data from the old process data set and n new data from the new process data set; Step eight: use the selected data and the current query data to make a difference, and the deviation sample obtained is used as the data set for constructing a local compensation model and ; Step nine: use the selected data to establish JITL-JYKPLS local model, and use the trust region optimization method to solve the mismatch problem of JITL-JYKPLS local model; S91: according to formula (10), construct the optimization problem of JITL-JYKPLS local model, that is, the optimization compensation problem based on instant learning and trust region; （10）； where D is the spherical trust region radius, is the trust region constraint; S92: obtain the optimized compensation value by solving formula (10) , and further obtain the compensated operation variable trajectory ; S93: predict the final manipulated variable trajectory using JY-KPLS process model corresponding comprehensive economic benefit gain , if the comprehensive economic benefit gain satisfies formula (11), select as the manipulated variable trajectory for batch training, and then predict the process output , obtain a new model training environment, otherwise, determine that the first q training is unsuccessful, and continue to use the last trained manipulated variable trajectory as the final manipulated variable trajectory for the current training, that is ; （11）； In the formula, for comprehensive economic benefits, is a threshold value; S94: Calculate the actual gain of economic benefit , update the radius of the trust domain D ; if , let ; if and , let ; otherwise, let ; Step ten: interact with the optimization system according to the compensated model obtained in step nine, and try and error training; Step eleven: after the batch production is completed, it is judged whether the total reward value in the current batch training exceeds the total reward value in the previous batch training; if it exceeds, the current trained optimization system is used for batch optimization; otherwise, the optimization system trained in the previous batch is used instead of the current optimization system for batch optimization; Step twelve: input the optimized parameters into the controller, use the controller to control the production system to carry out production operation, and output the final product quality; Step Thirteen: Order k = k +1, update the modeling data, and return to step one for the next batch of optimization.

Images