Fermentation process optimization regulation and control method based on reinforcement learning and comparative learning

Abstract

The invention relates to a fermentation process optimization regulation and control method based on reinforcement learning and comparative learning. The method comprises the following steps: acquiring historical batch data and batch data to be regulated and controlled in a penicillin fermentation process, and constructing an optimized regulation and control model by adopting a deep reinforcement learning double-actor network; optimizing an action value function by combining a loss function of comparative learning, enhancing the identification capability of similarity and difference between states, inputting historical batch data into a model for training, calculating a difference value between the penicillin yield and an expected yield, updating parameters by adopting a back propagation algorithm, and automatically adjusting hyper-parameters through Bayesian optimization; and finally, inputting data of a batch to be regulated and controlled into the trained neural network regulation and control model, and realizing penicillin yield regulation and control through iterative action output. By adopting the method, the penicillin yield can be effectively regulated and controlled, and powerful technical support and theoretical guidance are provided for optimization regulation and control of the penicillin fermentation process.

Description

Technical Field

[0001] The present invention relates to the technical field of neural networks, and more particularly to the field of optimization and regulation, specifically a method for optimizing and regulating the fermentation process based on reinforcement learning and contrastive learning. Background Art

[0002] The penicillin fermentation process plays a crucial role in modern industry. The fermentation production process of penicillin involves using Penicillium to synthesize penicillin through fermentation reactions. Optimizing this process can effectively increase the yield of penicillin, enabling sufficient antibiotic supply globally to combat various bacterial infections such as pneumonia, sepsis, tuberculosis, etc. Therefore, how to effectively regulate the yield of the penicillin fermentation process has become an important research topic in academia and industry.

[0003] Early studies adopted traditional control methods such as model predictive control (MPC), robust control, and adaptive control. Some scholars established a mechanism model and designed a non-linear model predictive control (NMPC), but this method relies on an accurate mechanism model, requires a large amount of penicillin fermentation process data, and is difficult to cope with the complexity and uncertainty of the penicillin fermentation process.

[0004] To solve the above problems, researchers proposed iterative learning control (ILC), which uses the repetitive characteristics of the penicillin fermentation process to gradually correct errors between batches, thereby improving control performance. In addition, the two-dimensional system control theory has opened up a new direction for the control of the penicillin fermentation process. This theory designs control strategies by considering both the time and batch dimensions simultaneously to solve problems such as non-linearity, model mismatch, and external disturbances in the penicillin fermentation process. However, both of the above methods have poor adaptability to the non-linear and time-varying characteristics of the penicillin fermentation process and lack self-adaptive ability.

[0005] In contrast, the penicillin fermentation process control strategy based on reinforcement learning (RL) does not rely on an accurate process model but optimizes the control strategy through interaction with the environment. To solve the dimensionality disaster problem of Q-learning in high-dimensional and continuous state spaces, deep Q-learning (DQN) combined with a deep neural network was proposed, enabling it to handle continuous state spaces and apply to the optimization of the penicillin fermentation process control. At the same time, the controller based on deep deterministic policy gradient (DDPG) combined with principal component analysis further improved the regulation performance of the penicillin fermentation process. By combining reinforcement learning with iterative learning control (ILC), it can effectively handle non-repetitive problems in the penicillin fermentation process. Although the reinforcement learning algorithm is optimized by gradually generating and adjusting action sequences, it is prone to error accumulation and has low sample usage efficiency, limiting the exploration ability of the model. In addition, manually setting hyperparameters is often susceptible to human bias and has a high risk of overfitting.

[0006] Therefore, there is an urgent need to propose a new penicillin fermentation process control method, aiming to solve the problem of overestimation of the value function in the penicillin fermentation process regulation using reinforcement learning, enhance the ability to identify the similarities and differences between states, and automatically adjust the hyperparameters through Bayesian optimization, so as to improve the control accuracy and effectively avoid overfitting. Summary of the Invention

[0007] The object of the present invention is to overcome the above-mentioned disadvantages of the prior art and provide a fermentation process optimization and regulation method based on reinforcement learning and contrast learning with strong robustness and a relatively wide application range.

[0008] To achieve the above object, the fermentation process optimization and regulation method based on reinforcement learning and contrast learning of the present invention is as follows:

[0009] The fermentation process optimization and regulation method based on reinforcement learning and contrast learning, its main feature is that the method includes the following steps:

[0010] (1) Obtain and preprocess the historical and to-be-regulated batch data of the penicillin fermentation process, and construct a deep reinforcement learning double-actor network.

[0011] (2) Optimize the action value function in combination with the contrast learning loss function, train the model using historical data through the backpropagation algorithm, and automatically adjust the model hyperparameters using Bayesian optimization.

[0012] (3) Input the to-be-regulated batch data into the trained neural network model, and realize the regulation of penicillin production through iterative action output.

[0013] Preferably, the step (1) specifically includes the following steps:

[0014] (1.1) Obtain the data of the penicillin simulation production process through the data source, including offline data and the data of the to-be-regulated batch.

[0015] (1.2) Construct a model based on the double-actor deep deterministic policy gradient algorithm to obtain the output action, that is, the adopted regulation strategy.

[0016] For each action vector a t , add the same exploration noise ε t ,

[0017]

[0018] where, represents the policy function of the i-th actor network at time t, is the corresponding network parameter. The exploration noise ε t obeys a Gaussian distribution with a mean of 0 and a standard deviation of σ, that is The noise standard deviation is set to decay dynamically with the number of iterations:

[0019] σ = λ t + σ0

[0020] where σ0 is the initial variance, λ ∈ (0, 1) is the decay coefficient, and t is the time value.

[0021] The data (s, a, r, s′) generated by each interaction between the model and the penicillin fermentation process is stored in the experience pool. When updating the parameters, n samples [(s1, a1, r1, s2),..., (s n , a n , r n , s′ n+1 )] are randomly sampled from the experience pool, and the target action of the next state s′ is calculated using the target policy network. Each sample is assigned to the independent experience pool D set for each policy network with the same probability. When updating the parameters, the target policy network only samples from the fixed experience values. For each sample (s i , a j , r j , r j , s j+1 ), the target action value generated by the target policy network is:

[0022]

[0023] where, represents the policy function of the target policy network in the state , are the parameters of the target policy network, and ε j is the noise added to the target policy network as part of the exploration.

[0024] The next state s j+1 and the target policy are used as the inputs to the target value network to calculate two target value functions The smaller value of the two is used as the estimate Q of the true value target :

[0025]

[0026] where, are the parameters of the target value network, s j is the current state, is the output action of the i-th target action network in the current state, r j+1 is the reward obtained when entering the next state s j+1 after executing the action.

[0027] Preferably, the step (2) specifically includes the following steps:

[0028] (2.1) Optimize the action value function by combining the contrastive learning loss function;

[0029] (2.2) Automatically adjust the model hyperparameters using Bayesian optimization;

[0030] (2.3) Train the model using the backpropagation algorithm with historical data.

[0031] Preferably, construct a Gaussian mapping function φ: where F and D are constants representing dimensions, M is the mapping matrix, and x is the input. Usually, F << D, and each element M in M i,j is independently sampled from a predefined Gaussian distribution For any x i and x j , φ should satisfy:

[0032] (1 - ε)||x i - x j || ≤ ||φ(x i ) - φ(x j )|| ≤ (1 + ε)||x i + x j ||

[0033] where 0 ≤ ε ≤ 1. Use the KD - tree to perform K - nearest neighbor search in the projection space and store the tuples <[φ(s), a], s, Q(s, a)>,

[0034]

[0035] where, Q(s t , a t ) represents the expected return of choosing action a t under the state s t at time t, E() represents the expectation function, γ represents the discount factor, and r τ (s τ , a τ ) represents the immediate reward generated at time step τ due to the state s τ and the action a τ . Use the cumulative reward in the historical trajectory to approximate the Q - value, and then randomly sample a batch of N samples x i (1 ≤ i ≤ N) = [φ(s i ), a i , s i , Q i . For each of the N samples of x i , query the KD - tree to obtain M nearest samples xij (1 ≤ j ≤ M), construct the following set using these samples:

[0036]

[0037] where denotes a set containing the sample x i and its neighboring samples, and x iM denotes the M-th sample similar to the sample x i . Then, use K-means clustering to obtain K categories C k partitioned based on the Q value. Construct the final set of hard negative samples:

[0038]

[0039] where denotes a set containing the sample x i and its hard negative samples, and x k denotes the sample x i the k-th hard negative sample, and k i denotes the index of the positive sample most similar to the sample x i . K represents the total number of hard negative samples. Then, the total loss function containing the contrastive learning loss function is:

[0040]

[0041] where, x i1 ,... x iK denotes the negative samples related to the sample x i , and x N1 ,..., x NK denotes the positive samples related to the sample x i .

[0042]

[0043] z q represents the latent variable obtained after passing the anchor sample through the neural network, represents the latent variable obtained after passing the positive sample through the neural network, represents the latent variable obtained after passing the hard negative sample through the neural network, and W represents the weight matrix.

[0044] Obtain n sets of random hyperparameter observation points {γ1, γ2,..., γ n}, and define the loss function as:

[0045]

[0046] where, y t (θ i , γi ) represents the true penicillin production at time t under the i-th set of parameter combinations, represents the penicillin production output by the model at time t under the i-th set of parameter combinations, T represents the total number of sampling points, and θ i represents the network parameters to be optimized by backpropagation when the hyperparameters are set to γ i :

[0047]

[0048] where α is the learning rate. Define the objective function of Bayesian optimization as:

[0049] g(γ i ) = -L(θ i , γ i )

[0050] Calculate [g(γ1), g(γ2),..., g(γ k )], and obtain the optimal solution among k sets of hyperparameter combinations The prior distribution of g(γ) can be calculated as:

[0051] g(γ 1:k ) ~ N(μ0(γ 1:k ), ∑0(γ 1:k , γ 1:k ))

[0052] where γ 1:k represents the hyperparameter sequence from 1 to k groups, μ0(·) represents the mean function, and ∑0(·) represents the kernel function. Let n = k + 1. When g(γ 1:n ) is obtained, the conditional probability distribution can be calculated as:

[0053]

[0054] where:

[0055] μ n (γ) = ∑0(γ, γ 1:n )∑0(γ 1:n , γ 1:n ) -1 (g(γ 1:n ) - μ0(γ 1:n )) + μ0(γ)

[0056]

[0057] where γ 1:n represents the hyperparameter sequence from 1 to n groups, μ n (γ) represents the posterior mean, represents the posterior variance, ∑0() represents the covariance matrix, and g(γ 1:n ) represents the value of the objective function when the hyperparameter value is γ 1:n . Then, the acquisition function EI n (γ) is obtained as follows:

[0058]

[0059]

[0060] where, represents the currently observed optimal value, represents the probability density function of the standard normal distribution, Φ(·) represents the cumulative distribution function of the standard normal distribution, and [Δ n (γ)] + represents keeping the positive part of Δ n (γ), and |Δ n (γ)| represents the absolute value of the expected improvement;. The next hyperparameter observation point is obtained as follows:

[0061] γ n+1 = argmax E I n (γ)

[0062] Repeat the above steps. When the number of optimization times reaches the preset maximum value I, the optimal hyperparameter combination γ * is obtained.

[0063] After obtaining the optimal hyperparameter combination γ * , the loss function becomes:

[0064]

[0065] where, L HM-CURL is the contrastive learning loss term, is the predicted output of the model at time t when the model parameters are set to θ n , and the hyperparameters are set to γ * , and y t (θ n ,γ * ) represents the true value. The network parameters θ under the optimal hyperparameter combination are updated by the BP backpropagation algorithm:

[0066]

[0067] where, γ * is the optimal hyperparameter combination, θ is the parameter of the network to be updated, and α is the learning rate. When the training reaches a certain number of iterations, the optimal network parameters θ * are obtained, combined with the optimal hyperparameters γ *, an optimal control model is obtained.

[0068] Preferably, the step (3) obtains the model output after training in the following manner:

[0069] After the model training is completed, at each state s t select the action that can maximize Q π (s,a) as the control signal,

[0070]

[0071] where is the value function of the i-th value network at state s t and action a t and w t are the parameters of the value network.

[0072] The optimization and regulation method for the fermentation process based on reinforcement learning and contrast learning of the present invention is adopted. The problem of overestimation of the value function is solved by parallel training of double actors, and the independence is ensured through an independent experience pool. Secondly, the action value function is optimized by combining the contrast learning loss function to enhance the identification of similarity and difference between states. Finally, Bayesian optimization is used to automatically adjust the hyperparameters to improve the control accuracy and avoid overfitting. Description of the Drawings

[0073] Figure 1 is the schematic diagram of the optimization and regulation method for the fermentation process based on reinforcement learning and contrast learning of the present invention.

[0074] Figure 2 is the comparison diagram of the regulation effects between the optimization and regulation method for the fermentation process based on reinforcement learning and contrast learning of the present invention and other methods.

[0075] Figure 3 is the comparison diagram of the change of the reward function during the training between the optimization and regulation method for the fermentation process based on reinforcement learning and contrast learning of the present invention and other methods.

[0076] Figure 4 is the comparison diagram of the penicillin concentration at the end of each batch between the optimization and regulation method for the fermentation process based on reinforcement learning and contrast learning of the present invention and other methods. Detailed Embodiments

[0077] In order to more clearly describe the technical content of the present invention, the following will be further described in combination with specific embodiments.

[0078] Before detailing the embodiments according to the present invention, it should be noted that, hereinafter, the terms "comprising", "including" or any other variants are intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or elements inherent to such process, method, article or device.

[0079] Please refer to Figure 1 As shown, the optimization and regulation method for the fermentation process based on reinforcement learning and contrast learning, wherein the method comprises the following steps:

[0080] (1) Obtain and preprocess the historical and to-be-regulated batch data of the penicillin fermentation process, and construct a deep reinforcement learning dual-actor network.

[0081] (2) Optimize the action-value function in combination with the contrast learning loss function, train the model using the historical data through the backpropagation algorithm, and automatically adjust the model hyperparameters using Bayesian optimization.

[0082] (3) Input the to-be-regulated batch data into the trained neural network model, and realize the regulation of penicillin production through iterative action output.

[0083] As a preferred embodiment of the present invention, the step (1) specifically comprises the following steps:

[0084] (1.1) Obtain the data of the penicillin simulation production process through the data source, including the offline data and the data of the to-be-regulated batch.

[0085] (1.2) Construct a model based on the dual-actor deep deterministic policy gradient algorithm to obtain the output action, that is, the adopted regulation strategy.

[0086] As a preferred embodiment of the present invention, the dual-actor network is constructed in the following manner:

[0087] For each action vector a t , add the same exploration noise ε t :

[0088]

[0089] Wherein, represents the policy function of the i-th actor network at time t, are the corresponding network parameters. The exploration noise ε t follows a Gaussian distribution with a mean of 0 and a standard deviation of σ, that is The standard deviation of the noise is set to decay dynamically with the number of iterations:

[0090] σ = λ t + σ0

[0091] Among them, σ0 is the initial variance, λ∈(0,1) is the attenuation coefficient, and t is the time value.

[0092] Store the data (s, a, r, s′) generated by each interaction between the model and the penicillin fermentation process in the experience pool. When updating the parameters, randomly select n samples [(s1, a1, r1, s2),...,(s n , a n , r n , s′ n+1 )] from the experience pool, and use the target policy network to calculate the target action of the next state s′ Each sample is assigned to the independent experience pool D set for each policy network with the same probability. When updating the parameters, the target policy network only samples from the fixed experience values. For each sample (s i , a j , r j , s j ), the target action value generated by the target policy network is: j+1 )

[0093]

[0094] Among them, represents the policy function of the target policy network in the state , are the parameters of the target policy network, and ε j is the noise added to the target policy network and is part of the exploration.

[0095] Use the next state s j+1 and the target policy as the input of the target value network to calculate two target value functions Take the smaller value of the two as the estimated Q of the true value target :

[0096]

[0097] Among them, are the parameters of the target value network, s j is the current state, is the output action of the i-th target action network in the current state, r j+1 is the reward obtained when entering the next state s j+1 after executing the action.

[0098] As a preferred embodiment of the present invention, the step (2) specifically includes the following steps:

[0099] (2.1) Optimize the action value function by combining with the contrastive learning loss function;

[0100] (2.2) Automatically adjust the model hyperparameters using Bayesian optimization;

[0101] (2.3) Train the model using the backpropagation algorithm with historical data.

[0102] As a preferred embodiment of the present invention, construct the contrastive learning loss function in the following manner:

[0103] Construct a Gaussian mapping function where F and D are constants representing dimensions, M is the mapping matrix, and x is the input. Usually F << D, and each element M in M i,j is independently sampled from a predefined Gaussian distribution For any x i and x j , φ should satisfy:

[0104] (1 - ε)||x i - x j || ≤ ||φ(x i ) - φ(x j )|| ≤ (1 + ε)||x i + x j ||

[0105] where 0 ≤ ε ≤ 1. Use the KD - tree to perform K - nearest neighbor search in the projection space, and store the tuples <[φ(s), a], s, Q(s, a)>,

[0106]

[0107] where, Q(s t , a t ) represents the expected return of selecting action a t under the state s t , E() represents the expectation function, γ represents the discount factor, r τ (s τ , a τ ) represents the immediate reward generated at time step τ due to state s τ and action a τ . Use the cumulative reward in the historical trajectory to approximate the q - value, and then randomly sample a batch of N samples x i (1 ≤ i ≤ N) = [φ(s i ), a i , s i , Q i from the constructed KD - tree. For each x iFor N samples, query the KD tree to obtain M nearest samples x ij (1 ≤ j ≤ M), use these samples to construct a set:

[0108]

[0109] where represents a set containing the sample x i and its neighboring samples, x iM represents the Mth sample similar to the sample x i . Then, use K-means clustering to obtain K categories C divided based on the Q value k Construct the final set of hard negative samples:

[0110]

[0111] where represents a set containing the sample x i and its hard negative samples, x k represents the kth hard negative sample of the sample x i , k i represents the number of the positive sample most similar to the sample x i , and K represents the total number of hard negative samples. Then, the total loss function including the contrastive learning loss function is:

[0112]

[0113] where, x i1 ,...x iK represents the negative samples related to the sample x i , x N1 ,...,x NK represents the positive samples related to the sample x i .

[0114]

[0115] z q represents the latent variable obtained by passing the anchor sample through the neural network, represents the latent variable obtained by passing the positive sample through the neural network, represents the latent variable obtained by passing the hard negative sample through the neural network, and W represents the weight matrix.

[0116] As a preferred embodiment of the present invention, perform Bayesian hyperparameter optimization in the following manner:

[0117] Obtain n sets of random hyperparameter observation points {γ1, γ2,..., γ n} and define the loss function as:

[0118]

[0119] Among them, y t (θ i ,γ i ) represents the true penicillin production at time t under the i-th set of parameter combinations, represents the penicillin production output by the model at time t under the i-th set of parameter combinations, T represents the total number of sampling points, θ i represents the network parameters to be optimized by backpropagation when the hyperparameters are set to γ i :

[0120]

[0121] Among them, α is the learning rate. Define the objective function of Bayesian optimization as:

[0122] g(γ i ) = -L(θ i ,γ i )

[0123] Calculate [g(γ1), g(γ2),..., g(γ k )], and obtain the optimal solution among the k sets of hyperparameter combinations The prior distribution of g(γ) can be calculated as:

[0124] g(γ 1:k ) ~ N(μ0(γ 1:k ), ∑0(γ 1:k ,γ 1:k ))

[0125] Among them, γ 1:k represents the hyperparameter sequence from 1 to k groups, μ0(·) represents the mean function, and ∑0(·) represents the kernel function. Let n = k + 1. After obtaining g(γ 1:n ), the conditional probability distribution can be calculated as:

[0126]

[0127] Among them:

[0128] μ n (γ) = ∑0(γ,γ 1:n )∑0(γ 1:n ,γ 1:n ) -1 (g(γ 1:n ) - μ0(γ 1:n )) + μ0(γ)

[0129]

[0130] Among them, γ1:n Denote a sequence of hyperparameters from 1 to n, μ n (γ) represents the posterior mean, represents the posterior variance, ∑0() represents the covariance matrix, g(γ 1:n ) represents the value of the objective function when the hyperparameter value is γ 1:n At this time, the acquisition function EI n (γ) is obtained in the following way:

[0131]

[0132] where, represents the currently observed optimal value, represents the probability density function of the standard normal distribution, Φ(·) represents the cumulative distribution function of the standard normal distribution, [Δ n (γ)] + represents keeping only the positive part of Δ n (γ), |Δ n (γ)| represents the absolute value of the expected improvement. The next hyperparameter observation point is obtained in the following way:

[0133] γ n+1 = argmax E I n (γ)

[0134] where, γ n+1 represents the next hyperparameter observation point, EI n (γ) represents the acquisition function. Repeat the above steps. When the number of optimization reaches the preset maximum value I, the optimal hyperparameter combination γ * is obtained.

[0135] As a preferred embodiment of the present invention, the parameters are updated by backpropagation in the following way:

[0136] After obtaining the optimal hyperparameter combination γ * , the loss function becomes:

[0137]

[0138] where, L HM-CURL is the contrastive learning loss term, is the predicted output of the model at time t when the parameters are set to θ n , and the hyperparameters are set to γ * , y t (θ n ,γ * ) represents the true value. The network parameters θ under the optimal hyperparameter combination are updated by the BP backpropagation algorithm:

[0139]

[0140] Among them, γ * is the optimal hyperparameter combination, θ is the parameter that the network needs to update, and α is the learning rate. When the training reaches a certain number of iterations, the optimal network parameter θ * is obtained, and combined with the optimal hyperparameter γ * to obtain the optimal control model.

[0141] As a preferred embodiment of the present invention, the action output by the model in the step (3) is obtained in the following manner:

[0142] In each state s t , select the action that can maximize Q π (s, a) as the control signal:

[0143]

[0144] Among them, is the value function of the i-th value network in the state s t and the action a t , and w t is the parameter of the value network.

[0145] In practical applications, during the process of regulating the penicillin fermentation process, the method based on reinforcement learning may lead to a deterioration in the regulation effect of penicillin production due to problems such as overestimation of the value function. This technical solution proposes an optimization and regulation method for the fermentation process based on reinforcement learning and contrastive learning. First, solve the problem of overestimation of the value function through dual-actor parallel training, and ensure independence through an independent experience pool. Secondly, optimize the action value function by combining the contrastive learning loss function to enhance the identification of similarity and difference between states. Finally, use Bayesian optimization to automatically adjust the hyperparameters to improve the control accuracy and avoid overfitting.

[0146] The optimization and regulation method for the fermentation process based on reinforcement learning and contrastive learning mainly includes the following three main stages:

[0147] 1. Data preprocessing and model construction;

[0148] 2. Model hyperparameter optimization and parameter update;

[0149] 3. Regulate the penicillin fermentation process according to the trained model.

[0150] In a specific embodiment of the present invention, the above three main stages will be further described in detail below:

[0151] 1. Data preprocessing and model construction:

[0152] The present invention adopts the penicillin simulation platform designed by Birol et al. in 2002. The main variable descriptions are shown in Table 1, including 15 characteristic variables and the quality variable penicillin titer. 80 batches of data are generated using the simulation platform, with the generation time for each batch of data being 400 hours and the data sampling interval being 1 hour. The first 60 batches of data are used as the training set, and the last 20 batches of data are used as the test set. The training set is used to train the neural network model and adjust the model parameters, and the test set is used to independently evaluate the generalization performance of the model.

[0153] Table 1 Description of Penicillin Simulation Variables

[0154]

[0155] Since the traditional regulation method based on reinforcement learning will affect the performance of the penicillin fermentation process model regulation due to the problem of overestimating the value function, the present invention reduces the problem of overestimating the value function in the form of a dual actor to improve the regulation effect.

[0156] 2. Model Hyperparameter Optimization and Parameter Update:

[0157] Obtain n groups of random hyperparameter observation points {γ1, γ2,..., γ n} and define the loss function as:

[0158]

[0159] where y t (θ i , γ i ) represents the true penicillin production at time t under the i-th group of parameter combinations, represents the penicillin production output by the model at time t under the i-th group of parameter combinations, T represents the total number of sampling points, and θ i represents the network parameters to be optimized by backpropagation when the hyperparameters are set to γ i :

[0160]

[0161] where α is the learning rate. Define the objective function of Bayesian optimization as:

[0162] g(γ i ) = -L(θ i , γ i )

[0163] Calculate [g(γ1), g(γ2),..., g(γ k )] to obtain the optimal solution among the k groups of hyperparameter combinations The prior distribution of g(γ) can be calculated as:

[0164] g(γ 1:k )~N(μ0(γ 1:k ),∑0(γ 1:k ,γ 1:k ))

[0165] where γ 1:k represents a sequence of hyperparameters from 1 to k groups, μ0(·) represents the mean function, and ∑0(·) represents the kernel function. Let n = k + 1. After obtaining g(γ 1:n ), the conditional probability distribution can be calculated as:

[0166]

[0167] where:

[0168] μ n (γ) = ∑0(γ,γ 1:n )∑0(γ 1:n ,γ 1:n ) -1 (g(γ 1:n ) - μ0(γ 1:n )) + μ0(γ)

[0169]

[0170] where γ 1:n represents a sequence of hyperparameters from 1 to n groups, μ n (γ) represents the posterior mean, represents the posterior variance, ∑0() represents the covariance matrix, g(γ 1:n ) represents the value of the objective function when the hyperparameter value is γ 1:n . Then, the acquisition function EI n (γ) is obtained in the following way:

[0171]

[0172] where, represents the currently observed optimal value, represents the probability density function of the standard normal distribution, Φ(·) represents the cumulative distribution function of the standard normal distribution, [Δ n (γ)] + represents keeping the positive part of Δ n (γ), and |Δ n (γ)| represents the absolute value of the expected improvement. The next hyperparameter observation point is obtained in the following way:

[0173] γ n+1 = argmax E I n (γ)

[0174] where γn+1 Denotes the next hyperparameter observation point, EI n (γ) represents the acquisition function. Repeat the above steps. When the number of optimization reaches the preset maximum value I, the optimal hyperparameter combination γ is obtained * . Obtain the optimal hyperparameter combination γ * After that, the loss function becomes:

[0175]

[0176] where L HM-CURL is the contrastive learning loss term, is the prediction output of the model at time t when the parameters are set to θ n , and the hyperparameters are set to γ * , y t (θ n ,γ * ) represents the true value. The network parameters θ under the optimal hyperparameter combination are updated by the BP backpropagation algorithm:

[0177]

[0178] where γ * is the optimal hyperparameter combination, θ is the parameter that the network needs to update, and α is the learning rate. When the training reaches a certain number of iterations, the optimal network parameters θ are obtained * , combined with the optimal hyperparameters γ * , the optimal control model is obtained.

[0179] 3. Regulate the penicillin fermentation process according to the trained model:

[0180] According to the obtained optimal model, at each state s t , select the action that can maximize Q π (s,a) as the control signal,

[0181]

[0182] where is the value function of the i-th value network at state s t and action a t , and w t are the parameters of the value network.

[0183] The experimental model effect of this technical solution is as follows:

[0184] Table 2 shows the performance comparison of three models in different batches, with penicillin concentration and the obtained reward value as evaluation indicators. The results show that the proposed DAHmBoNet exhibits the best performance in all batches, significantly outperforming the other two models. In terms of penicillin concentration regulation, the regulation method based on DAHmBoNet continuously improves from 0.7108 in batch 2 to 1.3826 in batch 12, and maintains a stable level between 1.3855 and 1.3834 between batch 15 and batch 20. In contrast, the penicillin production concentration based on IL-RLC and DDPG increases at a slower rate and shows certain fluctuations in the later stage. In addition, at batch 12, the penicillin concentration regulated by DAHmBoNet is 0.1449 and 0.3302 higher than that of IL-RLC and DDPG respectively, indicating its significant advantage in optimizing penicillin concentration.

[0185] Penicillin concentration synthesized in each batch of Table 2 and rewards obtained by the model

[0186]

[0187]

[0188] In terms of the performance of the reward value, DAHmBoNet is also superior to the DDPG and IL-RLC methods. At batches 6 and 8, the reward values of DAHmBoNet reach 666.45 and 734.82 respectively, significantly higher than 439.45 and 492.60 of IL-RLC, and 378.59 and 424.35 of DDPG. This advantage is particularly obvious in the later stage. The reward values of DAHmBoNet remain stable (from 734.12 to 735.34) from batch 12 to batch 20, while the reward values of IL-RLC and DDPG fluctuate greatly and are always lower than DAHmBoNet. Generally speaking, DAHmBoNet not only shows a faster response speed and higher learning ability, but also demonstrates extremely strong robustness and generalization ability. These results indicate that DAHmBoNet has significant performance advantages in the task of optimizing the synthesized penicillin concentration and is a better choice for solving the optimization and regulation problem of the penicillin fermentation process.

[0189] Figure 2 Shows the penicillin concentration control effects of the three methods after training for a certain number of batches. As can be seen from the figure, the concentration change of DAHmBoNet almost completely remains within the allowable error range during the entire sampling time period and always closely adheres to the target concentration, demonstrating extremely high control accuracy. In contrast, IL-RLC approaches the target value in some time periods, but deviates from the target concentration in many places, with a large error range. The performance of DDPG is the worst, and its concentration value almost always deviates from the target and has a large fluctuation range, making it difficult to meet the target concentration requirements.

[0190] Figure 3 The reward changes of three methods in the 10th batch are shown. The reward value of DAHmBoNet always remains at the highest level throughout the control process and shows small fluctuations, indicating the strongest persistence and robustness of its strategy optimization ability. Although the reward value of IL-RLC has increased, the gap with DAHmBoNet is still large, and the fluctuation range is significantly larger than that of DAHmBoNet, reflecting its lack of stability. The growth of the reward value of DDPG tends to flatten out, and the final convergence value is still the lowest, indicating that its strategy has not been fully optimized during long-term training.

[0191] Figure 4 The change trends of the terminal penicillin concentration of three models in different batches are shown. The experimental results show that DAHmBoNet shows significant advantages in all batches. Specifically, the concentration of DAHmBoNet increases rapidly in the early batches, from 0.71 g / L in batch 2 to 1.38 g / L in batch 12, and then gradually converges. (In contrast, the concentrations of IL-RLC and DDPG increase more slowly in the early stage and show large fluctuations in the later stage. In addition, the standard deviation of the concentration of DAHmBoNet is small, indicating that its model has higher stability and robustness in the concentration optimization task. From the overall trend, DAHmBoNet performs better than other models in all batches. Its rapid increase and stable optimization ability indicate that the model can effectively regulate the penicillin fermentation process.

[0192] Compared with other methods, this method constructs an optimization and regulation framework based on the data of the penicillin fermentation process. This framework solves the problem of overestimation of the value function through parallel training of double actors and ensures the independence between the double actors through an independent experience pool. Secondly, the loss function of contrastive learning is introduced into the training process of the action value function, and by maximizing the feature differences between positive and negative samples, the policy network's accurate capture of the similarities and differences of different states is enhanced. Finally, the automatic adjustment of hyperparameters is realized through Bayesian optimization, further improving the control accuracy and avoiding overfitting. In summary, the proposed fermentation process optimization and regulation method based on reinforcement learning and contrastive learning is effective and practical.

[0193] In the description of this specification, the descriptions referring to terms such as "one embodiment", "some embodiments", "example", "specific example", or "embodiment" etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0194] Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

[0195] By adopting the fermentation process optimization and control method based on reinforcement learning and contrast learning of the present invention, the problem of overestimation of the value function is solved through parallel training of double actors, and the independence between the double actors is ensured through an independent experience pool. Secondly, the loss function of contrast learning is introduced into the training process of the action value function, and by maximizing the feature differences between positive and negative samples, the policy network's accurate capture of the similarities and differences of different states is enhanced. Finally, the automatic adjustment of hyperparameters is realized through Bayesian optimization, further improving the control accuracy and avoiding overfitting, providing strong guidance for the regulation of the penicillin fermentation production process.

[0196] In this specification, the present invention has been described with reference to its specific embodiments. However, it is obvious that various modifications and variations can still be made without departing from the spirit and scope of the present invention. Therefore, the specification and drawings should be regarded as illustrative rather than restrictive.

Claims

1. A fermentation process optimization and control method based on reinforcement learning and contrastive learning, characterized in that: The method comprises the following steps: (1) Obtain and preprocess the historical and batch data of penicillin fermentation process and construct a deep reinforcement learning dual-actor network; (2) The action-value function is optimized by combining the contrastive learning loss function, the model is trained using the back-propagation algorithm using historical data, and the model hyperparameters are automatically adjusted using Bayesian optimization; (3) The batch data to be regulated is input into the trained neural network model, and penicillin production regulation is achieved through iterative action output.

2. The fermentation process optimization and control method based on reinforcement learning and contrast learning according to claim 1, characterized in that: The step (1) specifically comprises the following steps: (1.1) Obtain penicillin simulation production process data through data sources, including offline data and batch data to be regulated; (1.2) Adopt regulatory strategies for the acquired data and construct a dual-actor network structure.

3. The fermentation process optimization and control method based on reinforcement learning and contrast learning according to claim 2, characterized in that: The step (1.2) described above constructs a dual-actor network structure in the following manner: For each action vector a t , adding the same exploration noise ε t : in, is the i-th action vector, represents the policy function of the ith actor network at time t, is the corresponding network parameter, exploring the noise ε t It obeys a Gaussian distribution with a mean of 0 and a standard deviation of σ, that is The noise standard deviation is set to decay dynamically with the number of iterations: s = l t +σ0 Among them, σ0 is the initial variance, λ∈(0,1) is the attenuation coefficient, and t is the time value; The data (s, a, r, s′) generated by each interaction between the model and the penicillin fermentation process is stored in the experience pool. When the parameters are updated, n samples [(s1, a1, r1, s2), ..., (s n ,a n ,r n ,s′ n+1 )], and use the target policy network to calculate the target action for the next state s′ Each sample is assigned with the same probability to an independent experience pool D set for each policy network. i In the parameter update, the target policy network only samples at fixed experience values. For each sample (s j ,a j ,r j ,s j+1 ), the target action value generated by the target policy network is: in, Indicates that the target policy network is in state The policy function is is the parameter of the target policy network, ε j Noise added to the target policy network as part of exploration; The next state s j+1 Target strategy As input to the target value network, two target value functions are calculated The smaller of the two is taken as the estimate of the true value Q target : in, is the parameter of the target value network, s j is the current state, is the output action of the i-th target action network in the current state, r j+1 Enter the next state s after executing the action j+1 The rewards obtained when.

4. The fermentation process optimization and control method based on reinforcement learning and contrast learning according to claim 3, characterized in that: The step (2) specifically comprises the following steps: (2.1) Optimize the action value function by combining contrastive learning loss function; (2.2) Automatically adjust model hyperparameters using Bayesian optimization; (2.3) The model is trained by updating parameters using the back-propagation algorithm based on historical data.

5. The fermentation process optimization and control method based on reinforcement learning and contrast learning according to claim 4 is characterized in that: The step (2.1) constructs the contrastive learning loss function in the following way: Construct a Gaussian mapping function where F and D are constants representing dimensions, M is a mapping matrix, x is the input, typically F << D, and each element M in M i,j is independently sampled from a predefined Gaussian distribution For any x i and x j , φ should satisfy: (1-e)‖x i -x j ‖≤‖φ(x i )-φ(x j )‖≤(1+ε)‖x i +x j ‖ Where 0≤ε≤1, use KD tree to perform K nearest neighbor search in the projection space, and store tuples <[φ(s),a],s,Q(s,a)>: Among them, Q(s t ,a t ) represents the state s at time t t Next select action a t The expected return, E() represents the expected function, γ represents the discount factor, r τ (s τ ,a τ ) means that at time step τ, due to the state s τ and action a τ The generated immediate reward uses the cumulative reward in the historical trajectory to approximate the Q value, and randomly samples a batch of N samples x from the constructed KD tree i (1≤i≤N)=[φ(s i ),a i ,s i ,Q i ], for each x i N samples, query the KD tree to get the M nearest samples x ij (1≤j≤M), and construct the set in this way: in, Represents a sample x i and its collection of neighboring samples, x iM Represents the sample x i For similar M samples, K-means clustering is used to obtain K categories C based on Q values. k , construct the final set of difficult negative samples: in, Represents a sample x i and its set of hard negative samples, x ik Represents sample x i The kth hard-to-classify negative sample, k i Represents the sample x i The number of the most similar positive sample, K represents the total number of difficult negative samples, and the total loss function including the contrastive learning loss function is: Among them, x i1 ,...x iK Represents the sample x i The relevant negative samples, x N1 ,...,x NK Represents the sample x i The relevant positive samples, Among them, z q Represents the hidden variable obtained after the anchor sample passes through the neural network, Represents the hidden variable obtained after the positive sample passes through the neural network, Represents the hidden variables obtained after the difficult negative samples pass through the neural network, and W represents the weight matrix.

6. The method for optimizing and controlling the penicillin fermentation process based on the reinforcement learning dual-actor strategy and contrastive learning hard-to-distinguish negative sample mining according to claim 5, characterized in that: The step (2.2) described above performs Bayesian hyperparameter optimization in the following manner: Get n sets of random hyperparameter observation points and define the loss function L(θ i ,γ i )for: Among them, y t (θ i ,γ i ) represents the actual penicillin production at time t under the i-th group of parameter combinations, represents the penicillin production output by the model at time t under the i-th group of parameter combinations, T represents the total number of sampling points, θ i Indicates that when the hyperparameter is set to γ i The following network parameters need to be optimized by back-propagation: Among them, α is the learning rate, and the objective function of Bayesian optimization is defined as: g(γ i )=-L(θ i ,c i ) Calculate [g(γ1),g(γ2),...,g(γ k )], and obtain the optimal solution among k sets of hyperparameter combinations The prior distribution of g(γ) is: g(γ 1:k )~N(μ0(γ 1:k ),∑0(γ 1:k ,c 1:k )) Among them, γ 1:k represents 1 to k sets of hyperparameter sequences, μ0(·) represents the mean function, and ∑0(·) represents the kernel function; Let n = k + 1, when we get g(γ 1:n ), the conditional probability distribution is: m n (γ)=∑0(γ,γ 1:n )∑0(γ 1:n ,c 1:n ) -1 (g(γ 1:n )-μ0(γ 1:n ))+μ0(c) Among them, γ 1:n Represents 1 to n sets of hyperparameter sequences, μ n (γ) represents the posterior mean, represents the posterior variance, ∑0() represents the covariance matrix, g(γ 1:n ) indicates that the hyperparameter value is γ 1:n When the value of the objective function is , the acquisition function EI is obtained as follows n (γ): in, represents the currently observed optimal value, represents the probability density function of the standard normal distribution, Φ(·) represents the cumulative distribution function of the standard normal distribution, [Δ n (γ)] + Indicates retention Δ n The positive part of (γ), |Δ n (γ)| represents the absolute value of the expected improvement; Get the next hyperparameter observation point as follows: c n+1 =argmaxEI n (c) Among them, γ n+1 represents the next hyperparameter observation point, EI n (γ) represents the acquisition function. Repeat the above steps. When the number of optimizations reaches the preset maximum value I, the optimal hyperparameter combination γ is obtained. * .

7. The fermentation process optimization and control method based on reinforcement learning and contrast learning according to claim 6, characterized in that: The step (2.3) described above performs back propagation to update parameters in the following manner: Get the optimal hyperparameter combination γ * After that, the loss function becomes: Among them, L HM-CURL is the contrastive learning loss term, For the model, the parameters are set to θ n , the hyperparameter is set to γ * The predicted output of the model at time t, y t (θ n ,γ * ) represents the true value, and the network parameters θ under the optimal hyperparameter combination are updated through the BP back propagation algorithm: Among them, γ * is the optimal hyperparameter combination, θ is the parameter that the network needs to update, α is the learning rate, and when the training reaches a certain number of iterations, the optimal network parameter θ is obtained * , combined with the optimal hyperparameter γ * , and obtain the optimal control model.

8. The fermentation process optimization and control method based on reinforcement learning and contrast learning according to claim 7, characterized in that: The step (3) obtains the model output in the following manner: In each state s t Next, choose the one that maximizes Q π The action of (s,a) is used as a control signal: in, is the i-th value network in state s t and action a t The value function under t are the parameters of the value network.

Images