Q-learning-based dye vat process temperature adaptive PID control method and related device

Abstract

The invention discloses a dye vat process temperature self-adaptive PID control method based on Q-learning and a related device, and relates to the technical field of intelligent control, a pre-established intelligent agent environment learning model based on a Q-learning algorithm is adopted, PID control parameters at the current moment corresponding to current dye vat data input into the model are iteratively updated, and the dye vat process temperature self-adaptive PID control method based on the Q-learning algorithm is obtained. The method comprises the following steps: when the iteration stop condition is met, obtaining a PID control signal at the current moment, then generating an opening signal of a dye vat burner according to the control signal, and controlling the dye vat burner to heat the dye liquor to the target process temperature at the current moment through the opening signal. According to the method, the Q-learning algorithm is adopted to intelligently control the opening degree of the burner, so that the temperature in the dye vat can be quickly responded and accurately adjusted, and it is ensured that the temperature of dye liquor in the dye vat can be attached to a target process temperature curve.

Description

Technical Field

[0001] The present application relates to the field of intelligent control technology, and in particular to a Q-learning-based dye vat process temperature adaptive PID control method and related devices. Background Art

[0002] Dyeing control of dyeing machines is a complex and delicate process. Dyeing control mainly controls the temperature of dye liquid in the dye vat to execute a temperature process curve that changes according to time. Specifically, the temperature process curve includes multiple sections of heating at different rates, multiple sections of cooling at different rates, and multiple sections of heat preservation for different time periods.

[0003] The temperature of dye liquor is an important process parameter in the dyeing process. However, the control accuracy of the dye liquor temperature in the prior art is not high. Specifically, the dye liquor temperature controlled by the dyeing machine during the actual heating or cooling process is not in line with the temperature process curve, which will produce fabrics with color differences and color spots of low quality, reducing the success rate of dyeing in one time; during the dyeing process, the heat transfer system composed of dye liquor, fabric, equipment, etc. needs to reach a thermal equilibrium state. If the dye liquor temperature is not in line with the temperature process curve, the thermal equilibrium time will be extended, and the entire dyeing cycle will be extended accordingly. Summary of the invention

[0004] In view of the above problems, this application provides a Q-learning-based dye vat process temperature adaptive PID control method and related devices to improve the control accuracy of the dye solution temperature in the dyeing machine. The specific scheme is as follows:

[0005] The first aspect of the present application provides a Q-learning-based dye vat process temperature adaptive PID control method, comprising:

[0006] Get the current dye vat data; the current dye vat data includes the current heating rate, the current temperature deviation and the current dye solution temperature. The current temperature deviation is the deviation between the current target process temperature and the current dye solution temperature.

[0007] The current dye vat data is input into a pre-established intelligent agent environment learning model based on the Q-learning algorithm to obtain the PID control signal at the current moment; the intelligent agent environment learning model based on the Q-learning algorithm is configured to iteratively update the PID control parameters at the current moment corresponding to the current heating rate, the current temperature deviation and the current dye solution temperature using an ε-greedy strategy until the iteration stop condition is met;

[0008] The opening signal of the dye vat burner is generated according to the PID control signal at the current moment, so as to control the dye vat burner to heat the dye solution temperature to the target process temperature at the current moment in an opening state corresponding to the opening signal.

[0009] In a possible implementation, the ε-greedy strategy includes an exploration strategy and an utilization strategy. The iteration stop condition is that the number of iterations reaches a preset number of iterations. The ε-greedy strategy is used to iteratively update the current PID control parameters corresponding to the current heating rate, the current temperature deviation, and the current dye solution temperature until the iteration stop condition is met, including:

[0010] determining at least one set of PID control parameters to be utilized through a discovery strategy;

[0011] Selecting a set of PID control parameters from at least one set of PID control parameters to be used according to the utilization strategy to obtain the updated PID control parameters, determining the updated PID control parameters as the PID control parameters at the current moment, and calculating the value function value of the PID control parameters at the current moment;

[0012] Based on the value function value, the iterative execution determines at least one set of PID control parameters to be used through the exploration strategy; according to the utilization strategy, a set of PID control parameters is selected from the at least one set of PID control parameters to be used, and the PID control parameters updated this time are obtained. The PID control parameters updated this time are determined as the PID control parameters at the current moment, and the value function value of the PID control parameters at the current moment is calculated until the number of iterations reaches the preset number of iterations, and the PID control parameters at the current moment are output.

[0013] In one possible implementation, at least one set of PID control parameters to be used is determined by an exploration strategy, including:

[0014] At least one set of PID control parameters is determined within a preset adjustment range according to a preset exploration probability; the preset adjustment range is determined according to a preset adjustment coefficient and the current number of iterations; the preset adjustment range is negatively correlated with the number of iterations; the preset exploration probability is determined according to the attenuation rate of this iteration and the current number of iterations; the attenuation rate is negatively correlated with the number of iterations.

[0015] In a possible implementation, selecting a set of PID control parameters from at least one set of PID control parameters to be utilized according to the utilization strategy to obtain the updated PID control parameters includes:

[0016] A set of PID control parameters is selected from at least one set of PID control parameters to be used by adopting a gradient updating method.

[0017] In one possible implementation, a set of PID control parameters is selected from at least one set of PID control parameters to be used by adopting a gradient update method, including:

[0018] Calculate the gradient value of the current PID control parameter;

[0019] The PID control parameters for this update are determined from at least one set of PID control parameters to be used according to a preset learning rate and a gradient value.

[0020] In a possible implementation, calculating the value function value of the PID control parameter at the current moment includes:

[0021] The value function value of the PID control parameters at the current moment is comprehensively calculated according to the preset temperature reward rule and the preset error reward rule.

[0022] The second aspect of the present application provides a Q-learning-based dyeing vat process temperature adaptive PID control device, comprising:

[0023] An acquisition unit, an input unit and a control unit; wherein:

[0024] An acquisition unit is used to acquire current dye vat data; the current dye vat data includes a current heating rate, a current temperature deviation and a current dye solution temperature, and the current temperature deviation is: a deviation between a current target process temperature and a current dye solution temperature;

[0025] An input unit is used to input the current dye vat data into a pre-established intelligent agent environment learning model based on the Q-learning algorithm to obtain a PID control signal at the current moment; the intelligent agent environment learning model based on the Q-learning algorithm is configured to: adopt an ε-greedy strategy to iteratively update the PID control parameters at the current moment corresponding to the current heating rate, the current temperature deviation and the current dye solution temperature until an iteration stop condition is met;

[0026] The control unit is used to generate an opening signal of the dye vat burner according to the PID control signal at the current moment, so as to control the dye vat burner to heat the dye solution to the target process temperature at the current moment in an opening state corresponding to the opening signal.

[0027] The third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the Q-learning-based adaptive PID control method for dyeing vat process temperature as described in any one of the above.

[0028] In a fourth aspect, the present application provides a dye vat process temperature adaptive PID control device based on Q-learning, comprising at least one processor and a memory connected to the processor, wherein:

[0029] The memory is used to store computer programs;

[0030] The processor is used to execute the computer program so that the dye vat process temperature adaptive PID control device based on Q-learning can implement the dye vat process temperature adaptive PID control method based on Q-learning as described in any one of the above.

[0031] In a fifth aspect, the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the Q-learning-based dyeing vat process temperature adaptive PID control method as described in any one of the above.

[0032] By means of the above technical scheme, the Q-learning-based dye vat process temperature adaptive PID control method and related device provided by the present application adopts a pre-established intelligent agent environment learning model based on the Q-learning algorithm, iteratively updates the PID control parameters at the current moment corresponding to the current dye vat data input into the model, until the iteration stop condition is met, and obtains the PID control signal at the current moment, and then generates the dye vat burner opening signal according to the control signal, and controls the dye vat burner through the opening signal to heat the combustion temperature to the target process temperature at the current moment. The present application adopts the Q-learning algorithm to realize the intelligent control of the burner opening, thereby realizing the rapid response and precise adjustment of the temperature in the dye vat, and ensuring that the dye liquid temperature of the dye vat can fit the target process temperature curve. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0034] Figure 1 A schematic diagram of the control principle of the PID control algorithm provided in this application;

[0035] Figure 2 A schematic diagram of a flow chart of a dye vat process temperature adaptive PID control method based on Q-learning provided in this application;

[0036] Figure 3 A flow chart showing an example of a Q-learning-based adaptive PID control method for dye vat process temperature provided in this application;

[0037] Figure 4 An example diagram of the training process of the Q-learning algorithm provided in this application;

[0038] Figure 5This is a flow chart of an example of a Q-learning-based adaptive PID control method for dye vat process temperature provided in this application;

[0039] Figure 6 A schematic diagram of the structure of the Q-learning-based dye vat process temperature adaptive PID control device provided in this application;

[0040] Figure 7 This is a schematic diagram of the structure of the Q-learning-based dye vat process temperature adaptive PID control device provided in this application. DETAILED DESCRIPTION

[0041] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0042] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0043] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0044] In the field of textile dyeing, accurate control of the temperature in the dye vat is the key to ensuring dyeing quality. Specifically, it is necessary to accurately control the burner opening according to the preset temperature process curve and the specific execution time of the temperature process curve to control the temperature of the dye liquid in the dye vat, reduce the deviation between the temperature of the dye liquid in the dye vat and the process curve, so as to ensure the dyeing time of the fabric and improve the success rate of dyeing in one time.

[0045] The existing dyeing vat process temperature control generally adopts the traditional PID control algorithm. The parameter adjustment of this algorithm in the temperature control process relies on manual experience and is difficult to adapt to the complex and changeable process environment and dynamic temperature changes.

[0046] For details, see Figure 1 , a schematic diagram of the control principle of the PID control algorithm provided in this application.

[0047] The traditional PID control algorithm usually executes the control process by the PID controller. The PID controller is the core of the entire temperature control. It mainly calculates the deviation between the temperature process curve and the temperature of the dye liquid in the dye vat, and generates a control signal based on the proportional part, integral part and differential part of the deviation, and adjusts the valve opening of the gas burner through the control signal.

[0048] like Figure 1 As shown, r(t) is the given value in the temperature process curve, c(t) is the actual output value, that is, the temperature of the dye liquid in the dye vat, e(t) is the deviation between the temperature process curve and the temperature of the dye liquid in the dye vat, and u(t) is the control signal generated based on the proportional part, integral part and differential part of the above deviation.

[0049] It should be noted that the PID controller is very sensitive to parameter adjustment. Small parameter changes will lead to large performance differences. Therefore, the PID control algorithm requires experienced and skilled staff in the temperature control process. Moreover, in actual applications, the PID controller cannot adapt to the need for real-time adjustment of control parameters due to changes in operating conditions (changes in raw material composition, aging of equipment, fluctuations in ambient temperature, etc.). The PID controller cannot adapt to such dynamic changes, and the control effect will deteriorate accordingly.

[0050] It can be seen that the effect of temperature control based on the traditional PID control algorithm is not good, that is, the accuracy of temperature control is low.

[0051] In order to solve the above problems, a high-precision temperature control technology is urgently needed. Therefore, this application provides a dye vat process temperature adaptive PID control method and related devices based on Q-learning. This temperature control method can quickly respond to temperature deviations and adjust the opening of the gas burner through an intelligent algorithm, thereby achieving accurate prediction and control of the dye liquid temperature in the dye vat.

[0052] Optional, see Figure 2 , a flow chart of a Q-learning-based adaptive PID control method for dyeing vat process temperature provided in the present application.

[0053] It should be noted that this application is based on a pre-trained intelligent agent environment learning model based on the Q-learning algorithm to achieve rapid response and precise adjustment to the temperature change of the dye liquid in the dye vat. The model is trained based on the Q-learning algorithm, which is a classic reinforcement learning algorithm. The algorithm is based on the idea of ​​value iteration and estimates the value function (also called Q value) corresponding to each state action to guide the intelligent agent to choose the best action in each state.

[0054] Specifically, the Q-learning algorithm simulates the learning and decision-making process of the intelligent agent in the environment and adaptively adjusts the control parameters of the PID controller to achieve precise control of the dye temperature in the dye vat. This adaptive control strategy can effectively adapt to dynamically changing process conditions and environmental interference, and improve the accuracy and stability of dye vat process temperature control.

[0055] like Figure 2 As shown in FIG. 1 , the Q-learning-based dyeing vat process temperature adaptive PID control method includes the following steps:

[0056] Step 101, obtaining current dye vat data; the current dye vat data includes the current heating rate, the current temperature deviation and the current dye solution temperature, and the current temperature deviation is: the deviation between the current target process temperature and the current dye solution temperature.

[0057] For the intelligent agent environment learning model based on the Q-learning algorithm, the premise for controlling the temperature of the dye liquid in the dye vat is to determine the current state data of the intelligent agent.

[0058] The current state data of the agent is the current dyeing tank data, which specifically includes the current heating rate, the current temperature deviation and the current dyeing liquid temperature. The current temperature deviation is the deviation between the current target process temperature and the current dyeing liquid temperature.

[0059] Optionally, obtain the current heating rate, current temperature deviation and current dye solution temperature.

[0060] Step 102: input the current dye vat data into a pre-established intelligent agent environment learning model based on the Q-learning algorithm to obtain a PID control signal at the current moment; the intelligent agent environment learning model based on the Q-learning algorithm is configured to iteratively update the PID control parameters at the current moment corresponding to the current heating rate, the current temperature deviation and the current dye solution temperature using an ε-greedy strategy until the iteration stop condition is met.

[0061] The operating logic of the intelligent agent environment learning model based on the Q-learning algorithm is: use the ε-greedy strategy to iteratively update the PID control parameters at the current moment corresponding to the current heating rate, the current temperature deviation and the current dye solution temperature until the iteration stop condition is met.

[0062] The intelligent agent environment learning model based on the Q-learning algorithm mainly relies on the execution of the Q-learning algorithm to obtain the optimal PID control parameters corresponding to the current dye vat data. Next, the Q-learning algorithm is briefly introduced:

[0063] In the Q-learning algorithm, the agent will be in different states (state, referred to as s) in the environment, and each state will correspond to an action (action, referred to as a), that is, each state corresponds to an action. After executing the corresponding action in the current state, the agent will receive a reward (reward, referred to as r) and transfer to a new state. It should be noted that in this application, the state data is the heating rate, temperature deviation and dye solution temperature, and the action data is the PID control parameter.

[0064] In addition, the agent has a Q-table in different environments, which stores all the information of the state space (state_space) and action space (action_space) in the current environment, as well as the Q value obtained after executing the corresponding action (action) in each state (state). For example, Q(s, a) represents the expected cumulative reward (also called the value function value) of executing action a in state s, and the Q-learning algorithm approximates the optimal strategy by iteratively updating the new Q value.

[0065] Each time the Q value is updated iteratively, it is calculated using the update formula, which can be as follows:

[0066]

[0067] Among them, Q(s, a) represents the Q value of executing action a in state s; α is the learning rate, which is mainly used for the step size of each Q value update; r is the immediate reward obtained by the agent from the environment after executing action a; is a discount factor, which is used to represent the discounted value of future rewards; s′ is the new state transferred to after executing action a. is the maximum Q-value of all possible actions in the new state s′.

[0068] It should be noted that the above update formula is based on the Bellman optimal equation. In reinforcement learning algorithms, the Bellman optimal equation is often used to solve the optimal strategy, that is, to find a set of state-to-action mappings so that the cumulative reward obtained under this strategy is maximized.

[0069] Simply put, the execution process of the Q-learning algorithm mainly includes initialization, action selection, action execution, and updating the Q table.

[0070] For initialization, the main process is to create a Q table. As mentioned above, the Q table stores all the information in the state space and action space, as well as the Q value of the corresponding action executed in each state.

[0071] Specifically, the state space includes but is not limited to the heating rate rate, the deviation error and the current temperature current_temp of the dye vat. It can be understood that when the temperature process curve is in the insulation section, the heating rate rate is 0, and the deviation is the difference between the target process temperature and the current dye solution temperature; the action space includes the parameters of the PID control parameters, namely the proportion Kp, the integral Ki and the differential Kd.

[0072] First, each state in the state space and each variable in the action space are divided. For example, the heating rate rate in the state space is divided from 0 to 1.2 at an interval of 0.1, the deviation error is divided from -3 to 3 at an interval of 0.1, and the current temperature current_temp of the dye vat is divided from 10 to 150 at an interval of 1; the proportion Kp is divided from 0.5 to 10 at an interval of 0.5, the integral Ki is divided from 0.1 to 3 at an interval of 0.2, and the differential Kd is divided from 0.1 to 1 at an interval of 0.1.

[0073] It should be noted that the purpose of division during the initialization process is to discretize the state space data and action space data. If not discretized, the continuous state space data and action space data will occupy a very large system memory.

[0074] Afterwards, all values ​​in the Q table are initialized, that is, initial values ​​are assigned to the state space data and action space data in the Q table, and the initial values ​​are generally 0 or a small random number.

[0075] Then, set the key parameters, including the learning rate α, the discount factor , exploration rate ε and decay rate λ.

[0076] Learning rate α and discount factor The role of has been explained above. Next, we will introduce the decay rate λ and the exploration rate ε.

[0077] The decay rate λ is mainly used to control the speed at which the exploration rate ε decreases over time or the number of iterations. When the decay rate λ is greater than 1 and close to 1, the exploration rate ε decreases very slowly, indicating that the agent can maintain a high exploration rate ε for a long time, and has more time and opportunities to explore new actions; when the decay rate λ is less than 1 but close to 1, the exploration rate ε decreases faster, and the agent will transition from the exploration phase to the utilization phase faster, that is, the agent will turn to utilizing the known best action faster; when the decay rate λ is small, the exploration rate ε will decrease rapidly, and the agent will soon turn to mainly utilizing the known best action instead of exploring new actions.

[0078] The exploration rate ε is closely related to the execution process of the Q-learning algorithm. Next, the execution process of the Q-learning algorithm in this application is introduced.

[0079] The execution process of the Q-learning algorithm adopts the ε-greedy strategy, which includes the exploration strategy and the utilization strategy. The exploration strategy is mainly implemented based on the exploration rate ε, while the utilization strategy is implemented based on the utilization probability 1-ε. The exploration rate ε is used to balance the relationship between the exploration phase and the utilization phase of the Q-learning algorithm during execution. If the exploration rate ε is set too high, the agent may spend too much time on exploration actions and ignore the use of known actions; if ε is set too low, the agent may stop exploring too early, thereby missing the best action that may have a higher reward. It can be seen that a reasonable ε can improve the performance of the Q-learning algorithm.

[0080] First, get the current state of the agent from the environment.

[0081] In the exploration phase, at least one set of PID control parameters to be used is determined through an exploration strategy, and at least one set of PID control parameters is determined within a preset adjustment range according to a preset exploration probability.

[0082] The preset exploration probability is the exploration rate ε. For example, the initial value of the exploration rate ε can be the default value 0.1, or it can be set according to the specific environment. The exploration rate ε is calculated by the decay rate of this iteration and the current number of iterations, that is, the exploration rate ε used in each iterative update process is different, and the calculation formula can be: . It can be seen that the exploration rate ε will gradually decrease with the increase of the number of iterations, by reducing the proportion of exploration and increasing the proportion of utilization, that is, as the agent learns more about the environment, it will focus more on choosing known good actions. It should be noted that the decay rate is negatively correlated with the number of iterations, that is, as the number of iterations increases, the decay rate gradually decreases, which means that as the agent's understanding of the environment deepens, it will tend to use existing actions more and more, choose actions with higher Q values, and reduce the frequency of random exploration.

[0083] The preset adjustment range is the preset parameter adjustable range of the PID control parameter. The preset adjustment range is determined according to the preset adjustment system and the current number of iterations, and the preset adjustment range is negatively correlated with the number of iterations, that is, as the number of iterations increases, the preset adjustment range of the PID control parameter becomes smaller.

[0084] Optionally, an action is randomly selected near the current state for exploration based on the ε-exploration strategy. Randomly selecting an action near the current state for exploration can avoid random action selection from a large range, thereby reducing fluctuations in the entire temperature control system caused by excessive changes in action.

[0085] For example, the action PID control parameters include the parameter proportion Kp, the parameter integral Ki and the parameter differential Kd, and a range parameter is preset for each parameter. Assuming that the current action is ( , , ), the range parameters corresponding to the above parameters are , and Then, for the parameter ratio Kp, the preset adjustment range is [ , ], and similarly, the preset adjustment range of parameter integral Ki and parameter differential Kd can be obtained.

[0086] For the above , and The value of can be dynamically adjusted using a function related to the number of iterations. For example, the adjustment formula can be: .

[0087] in, is the initial Kp variation range, β is the adjustment coefficient, and t is the current number of iterations. For β, it is less than 1 and can be set based on experience.

[0088] It should be noted that as the number of iterations increases, the preset adjustment range for each PID control parameter will gradually narrow. That is, a larger action range can be used for exploration initially. As the number of iterations increases, the optional action range will be narrowed. This allows for extensive exploration of different action combinations in the early stages, and more fine-tuning of parameters in the later stages.

[0089] After determining at least one set of PID control parameters to be used through the exploration phase, it is time to execute the Q-learning algorithm.

[0090] Optionally, a set of PID control parameters is selected from at least one set of PID control parameters to be utilized according to the utilization strategy to obtain the PID control parameters updated this time, the PID control parameters updated this time are determined as the PID control parameters at the current moment, and the value function value of the PID control parameters at the current moment is calculated.

[0091] It should be noted that in this application, a set of PID control parameters is selected from at least one set of PID control parameters to be used by a gradient update method. That is, the gradient value of the current PID control parameter is calculated, and the PID control parameter for this update is determined from at least one set of PID control parameters to be used according to a preset learning rate and a gradient value.

[0092] Specifically, in the probability 1-ε utilization stage, the agent uses gradient updates to select tasks instead of directly selecting the action with the maximum Q value.

[0093] First, the gradient value of the current PID control parameter is calculated.

[0094] Taking the parameter ratio Kp as an example, assume that the current parameter value of this parameter is , on this basis, add a small increment , get the new parameter value , keep other parameters unchanged, use the current state and new parameter values ​​to run for a short period of time, and calculate the value of the objective function under these two parameters and .

[0095] It should be noted that the objective function is mainly used to calculate the reward value. Assuming that action a is taken in state s, the reward obtained after a series of state transitions is R (s, a). The definition formula of the objective function is as follows:

[0096]

[0097] Among them, J() represents the objective function and E represents the calculation of the expected cumulative reward.

[0098] The gradient calculation formula of the parameter ratio Kp can be:

[0099]

[0100] Similarly, the gradient values ​​of parameter integral Ki and parameter differential Kd can be calculated.

[0101] The calculated gradient value is used to determine the direction of parameter update. Specifically, if the gradient of the parameter ratio Kp , indicating that increasing Kp will increase the objective function. In this case, Kp should be reduced and the update direction is negative. On the contrary, if the gradient value of Kp is less than 0, Kp should be increased and the update direction is positive.

[0102] The determination of the updating direction of the parameter integral Ki and the parameter differential Kd is similar to that of the parameter proportion Kp.

[0103] After that, the new parameter values ​​are calculated. The new parameter values ​​are mainly calculated by the preset learning rate and gradient value.

[0104] For the parameter ratio Kp, the update formula can be:

[0105]

[0106] in, is the current parameter value, is the updated parameter value.

[0107] The update method of parameter integral Ki and parameter differential Kd is the same as that of parameter proportion Kp

[0108] Finally, new PID control parameters consisting of an updated parameter proportion Kp, an updated parameter integral Ki and an updated parameter differential Kd are obtained. The new PID control parameters are the PID control parameters updated this time.

[0109] It can be seen that in the utilization stage of this application, the intelligent agent adopts the gradient update method to select the PID control parameters for this update from the actions explored in the exploration stage, rather than directly selecting the action with the maximum Q value, so as to better balance the relationship between exploration and utilization and improve the performance and adaptability of the entire temperature control system.

[0110] Then, the updated PID control parameter is determined as the PID control parameter at the current moment, and the value function value of the PID control parameter at the current moment, that is, the Q value, is calculated.

[0111] Finally, based on the value function value, the iteration execution determines at least one set of PID control parameters to be used through the exploration strategy; according to the utilization strategy, a set of PID control parameters is selected from at least one set of PID control parameters to be used, and the updated PID control parameters are obtained, and the updated PID control parameters are determined as the PID control parameters at the current moment, and the value function value of the PID control parameters at the current moment is calculated until the number of iterations reaches the preset number of iterations, and the PID control parameters at the current moment are output. It can be understood that the number of iterations reaching the preset number of iterations is the iteration stop condition.

[0112] It should be noted that in the Q-learning algorithm, the design of the value function, also known as the reward function, is crucial. The value function can directly affect the learning effect of the entire Q-learning algorithm and the quality of the final strategy.

[0113] The setting of the reward function is mainly designed from three aspects: temperature, error and comprehensive factors.

[0114] For the temperature level, there are two situations: one is to reward when the temperature is close to the target, and the other is to reward when the target temperature is reached quickly.

[0115] If the current temperature is close to the target process temperature, a positive reward can be given. For example, the absolute value of the difference between the current temperature and the target process temperature is calculated. The smaller the difference, the higher the reward. The reward formula can be:

[0116]

[0117] Among them, A is an adjustment coefficient, and its size is determined according to the actual situation. A is set to ensure that the reward is within an appropriate range. A is mainly set based on experience, and A in different scenarios is generally different.

[0118] If the target process temperature is reached in a shorter time, a larger positive reward can be given. Specifically, this can be done by recording the time step required to reach the target process temperature. The shorter the time step, the higher the reward. Assuming that it takes n time steps to reach the target temperature, the reward formula can be expressed as:

[0119]

[0120] Among them, B is an adjustment coefficient, which is set based on experience, and B in different scenarios is generally different.

[0121] For the error level, if the error decreases, a positive reward is given. The size of the reward can be proportional to the magnitude of the error reduction.

[0122] For example, let the error of the previous time step be , the error at the current time step is , the reward formula can be:

[0123]

[0124] Among them, C is an adjustment coefficient, which is set based on experience, and B is generally different in different scenarios.

[0125] For the comprehensive factor level, the above rewards based on temperature and error are combined.

[0126] Exemplarily, the total reward formula can be expressed as:

[0127]

[0128] Moreover, different weights can be assigned to different parts of the reward according to the actual situation, so that the reward can be more in line with the actual situation.

[0129] It should also be noted that after each iteration calculates the value function value of the PID control parameter at the current moment, the Q table needs to be updated. The update formula of the Q table is:

[0130]

[0131] Step 103 : generating an opening signal of the dye vat burner according to the PID control signal at the current moment, so as to control the dye vat burner to heat the dye solution to the target process temperature at the current moment at an opening state corresponding to the opening signal.

[0132] The PID control signal at the current moment is used to generate the burner opening signal to control the dye vat burner to heat the dye solution temperature to an opening state closest to the target process temperature.

[0133] It should also be noted that the training of the intelligent agent environment learning model based on the Q-learning algorithm is based on historical dye vat data.

[0134] Specifically, historical dye vat data within a preset period is obtained, and the historical dye vat data includes the target process temperature, dye vat dye liquid temperature, and heating rate. After obtaining the historical dye vat data within the preset period, these data need to be preprocessed. Taking the dye vat dye liquid temperature as an example, if the temperature change rate exceeds twice the heating rate, it is considered an abnormal value, and the temperature at the previous moment plus the heating rate can be used as the dye vat dye liquid temperature at the current moment, thereby reducing the impact of abnormal values. Assuming that the heating rate of the dye vat dye liquid is 1.2℃ / min, and the data collection time interval is 1min, then the twice heating rate is (1.2 / 1)×2=2.4℃. If the data collection time interval is 20s (1 / 3 of 1min), then the twice heating rate is (1.2 / 3)×2=0.8℃.

[0135] Optional, see Figure 3 , a flow chart of an example of a Q-learning-based adaptive PID control method for dyeing vat process temperature provided in the present application.

[0136] like Figure 3 As shown, first, the process parameters are obtained, that is, the current dye vat data, and the process parameters are input into the intelligent agent environment learning model based on the Q-learning algorithm. The model executes the Q-learning algorithm, outputs the burner opening signal, and then controls the burner opening through the PLC, so as to achieve the purpose of heating the dye liquid in the dye vat.

[0137] In summary, the Q-learning-based adaptive PID control method for dye vat process temperature provided in the present application adopts a pre-established intelligent agent environment learning model based on the Q-learning algorithm, iteratively updates the PID control parameters at the current moment corresponding to the current dye vat data input into the model, until the iteration stop condition is met, and obtains the PID control signal at the current moment, and then generates an opening signal of the dye vat burner according to the control signal, and controls the dye vat burner through the opening signal to heat the combustion temperature to the target process temperature at the current moment. The present application adopts the Q-learning algorithm to realize the intelligent control of the opening of the burner, thereby achieving rapid response and precise adjustment of the temperature in the dye vat, ensuring that the temperature of the dye liquid in the dye vat can fit the target process temperature curve.

[0138] Optional, see Figure 4 , an example diagram of the training process of the Q-learning algorithm provided in this application.

[0139] First, obtain the historical data of the dye vat within the preset time, and then process the abnormal values ​​of the dye temperature in the dye vat historical data. After that, create a Q table and set the learning rate α and discount factor , exploration rate ε and decay rate λ and other key parameters.

[0140] Then, the current state of the agent is obtained from the current environment, an action is selected through the ε-greedy exploration strategy, the selected action is executed, the reward and new state given by the environment are observed, the Q value of the corresponding state work in the Q table is updated, and the new state s′ is used as the current state until the training ends when the set maximum number of iterations is reached.

[0141] Optional, see Figure 5 , a flow chart showing an example of a Q-learning-based adaptive PID control method for dye vat process temperature provided in this application.

[0142] First, the process temperature curve is obtained, which includes the heating rate, the target temperature of each stage, and the insulation time of the insulation stage. Then, according to the information of the process curve and the process target temperature of the algorithm at each moment, the temperature of the dye vat dye liquid is obtained regularly through the timed task, and the heating rate, process target temperature, and dye vat dye liquid temperature are used as state inputs to the Q-learning reinforcement learning adaptive PID algorithm. After the algorithm is inferred, the offset of the burner valve opening is output, and the offset of the opening is added to the current valve opening as the final burner valve opening. The burner valve opening instruction is sent to the PLC for control. The burner increases or decreases the amount of gas entering according to the size of the valve opening to heat the dye vat dye liquid, thereby controlling the change of the dye vat dye liquid temperature and ensuring that the dye vat dye liquid temperature can closely follow the changes in the process curve.

[0143] In summary, this application is based on the provided Q-learning reinforcement learning adaptive PID algorithm to intelligently control the opening of the burner, ensure that the combustion temperature in the dye vat can closely follow the target temperature process curve, realize accurate prediction and control of the dye liquid temperature in the dye vat, reduce temperature deviation, and thus shorten the dyeing cycle. It can not only reduce temperature fluctuations and improve dyeing efficiency and quality, but also reduce operational difficulty through intelligent management and improve overall production efficiency.

[0144] The above introduces the Q-learning-based dyeing vat process temperature adaptive PID control method provided in the embodiment of the present application. The following will introduce a device for executing the above-mentioned Q-learning-based dyeing vat process temperature adaptive PID control method.

[0145] See also Figure 6 , Figure 6 This is a schematic diagram of the structure of the Q-learning-based dyeing vat process temperature adaptive PID control device provided in this application. Figure 6 As shown, the device includes: an acquisition unit 10, an input unit 20 and a control unit 30; wherein:

[0146] The acquisition unit 10 is used to acquire current dyeing vat data; the current dyeing vat data includes the current heating rate, the current temperature deviation and the current dyeing liquid temperature, and the current temperature deviation is: the deviation between the current target process temperature and the current dyeing liquid temperature;

[0147] The input unit 20 is used to input the current dye vat data into a pre-established intelligent agent environment learning model based on the Q-learning algorithm to obtain a PID control signal at the current moment; the intelligent agent environment learning model based on the Q-learning algorithm is configured to: adopt an ε-greedy strategy to iteratively update the PID control parameters at the current moment corresponding to the current heating rate, the current temperature deviation and the current dye solution temperature until an iteration stop condition is met;

[0148] The control unit 30 is used to generate an opening signal of the dye vat burner according to the PID control signal at the current moment, so as to control the dye vat burner to heat the dye solution to the target process temperature at the current moment in an opening state corresponding to the opening signal.

[0149] In one embodiment, the ε-greedy strategy includes an exploration strategy and an exploitation strategy, and the iteration stop condition is that the number of iterations reaches a preset number of iterations. The input unit 20 is specifically used to:

[0150] determining at least one set of PID control parameters to be utilized through a discovery strategy;

[0151] Selecting a set of PID control parameters from at least one set of PID control parameters to be used according to the utilization strategy to obtain the updated PID control parameters, determining the updated PID control parameters as the PID control parameters at the current moment, and calculating the value function value of the PID control parameters at the current moment;

[0152] Based on the value function value, the iterative execution determines at least one set of PID control parameters to be used through the exploration strategy; according to the utilization strategy, a set of PID control parameters is selected from the at least one set of PID control parameters to be used, and the PID control parameters updated this time are obtained. The PID control parameters updated this time are determined as the PID control parameters at the current moment, and the value function value of the PID control parameters at the current moment is calculated until the number of iterations reaches the preset number of iterations, and the PID control parameters at the current moment are output.

[0153] In one embodiment, the input unit 20 is specifically configured to:

[0154] At least one set of PID control parameters is determined within a preset adjustment range according to a preset exploration probability; the preset adjustment range is determined according to a preset adjustment coefficient and the current number of iterations; the preset adjustment range is negatively correlated with the number of iterations; the preset exploration probability is determined according to the attenuation rate of this iteration and the current number of iterations; the attenuation rate is negatively correlated with the number of iterations.

[0155] In one embodiment, the input unit 20 is specifically configured to:

[0156] A set of PID control parameters is selected from at least one set of PID control parameters to be used by adopting a gradient updating method.

[0157] In one embodiment, the input unit 20 is specifically configured to:

[0158] Calculate the gradient value of the current PID control parameter;

[0159] The PID control parameters for this update are determined from at least one set of PID control parameters to be used according to a preset learning rate and a gradient value.

[0160] In one embodiment, the input unit 20 is specifically configured to:

[0161] The value function value of the PID control parameters at the current moment is comprehensively calculated according to the preset temperature reward rule and the preset error reward rule.

[0162] The present application also provides a Q-learning-based dye vat process temperature adaptive PID control device. Figure 7As shown, it shows a schematic diagram of the structure of a dye vat process temperature adaptive PID control device suitable for implementing the Q-learning-based dye vat process temperature adaptive PID control device in the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 7 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0163] like Figure 7 As shown, the Q-learning-based dyeing vat process temperature adaptive PID control device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0164] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0165] Also provided in an embodiment of the present application is a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the Q-learning-based dyeing vat process temperature adaptive PID control methods provided in the embodiments of the present application.

[0166] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any one of the Q-learning-based dyeing vat process temperature adaptive PID control methods provided in the embodiments of the present application.

[0167] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0168] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the method of each embodiment of the present application.

[0169] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0170] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, a computer, a training device or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device or data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integration. Available media can be magnetic media, (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or semiconductor media (e.g., solid-state hard disk (SSD)), etc.

Claims

1. The Q-learning-based adaptive PID control method for dyeing vat process temperature is characterized by: include: Get the current dye vat data; The current dye vat data includes the current heating rate, the current temperature deviation and the current dye solution temperature, wherein the current temperature deviation is the deviation between the current target process temperature and the current dye solution temperature; The current dye vat data is input into a pre-established intelligent agent environment learning model based on a Q-learning algorithm to obtain a PID control signal at the current moment; the intelligent agent environment learning model based on the Q-learning algorithm is configured to: adopt an ε-greedy strategy to iteratively update the PID control parameters at the current moment corresponding to the current heating rate, the current temperature deviation and the current dye solution temperature until an iteration stop condition is met; An opening signal of the dye vat burner is generated according to the PID control signal at the current moment, so as to control the dye vat burner to heat the dye solution temperature to the target process temperature at the current moment in an opening state corresponding to the opening signal.

2. The Q-learning-based adaptive PID control method for dyeing vat process temperature according to claim 1, characterized in that: The ε-greedy strategy includes an exploration strategy and an utilization strategy, the iteration stop condition is that the number of iterations reaches a preset number of iterations, and the ε-greedy strategy is used to iteratively update the current PID control parameters corresponding to the current heating rate, the current temperature deviation and the current dye solution temperature until the iteration stop condition is met, including: Determining at least one set of PID control parameters to be utilized by the exploration strategy; Selecting a set of PID control parameters from the at least one set of PID control parameters to be utilized according to the utilization strategy to obtain the PID control parameters updated this time, determining the PID control parameters updated this time as the PID control parameters at the current moment, and calculating the value function value of the PID control parameters at the current moment; Based on the value function value, iteratively execute the at least one group of PID control parameters to be used determined by the exploration strategy; select a group of PID control parameters from the at least one group of PID control parameters to be used according to the utilization strategy to obtain the PID control parameters updated this time, determine the PID control parameters updated this time as the PID control parameters at the current moment, and calculate the value function value of the PID control parameters at the current moment, until the number of iterations reaches the preset number of iterations, and output the PID control parameters at the current moment.

3. The Q-learning-based adaptive PID control method for dyeing vat process temperature according to claim 2, characterized in that: Determining at least one set of PID control parameters to be used by the exploration strategy includes: The at least one set of PID control parameters is determined within a preset adjustment range according to a preset exploration probability; the preset adjustment range is determined according to a preset adjustment coefficient and a current number of iterations; the preset adjustment range is negatively correlated with the number of iterations; the preset exploration probability is determined according to the attenuation rate of this iteration and the current number of iterations; the attenuation rate is negatively correlated with the number of iterations.

4. The Q-learning-based adaptive PID control method for dye vat process temperature according to claim 2, characterized in that: Selecting a set of PID control parameters from the at least one set of PID control parameters to be utilized according to the utilization strategy to obtain the updated PID control parameters includes: A set of PID control parameters is selected from the at least one set of PID control parameters to be used by adopting a gradient updating method.

5. The Q-learning-based adaptive PID control method for dyeing vat process temperature according to claim 4, characterized in that: The step of selecting a set of PID control parameters from the at least one set of PID control parameters to be used by adopting a gradient updating method comprises: Calculating the gradient value of the current PID control parameter; The PID control parameters for this update are determined from the at least one set of PID control parameters to be used according to a preset learning rate and the gradient value.

6. The Q-learning-based adaptive PID control method for dyeing vat process temperature according to claim 2, characterized in that: The calculating the value function value of the PID control parameter at the current moment includes: The value function value of the PID control parameter at the current moment is comprehensively calculated according to the preset temperature reward rule and the preset error reward rule.

7. The Q-learning-based dye vat process temperature adaptive PID control device is characterized by: include: An acquisition unit, an input unit and a control unit; wherein: The acquisition unit is used to acquire current dye vat data; The current dye vat data includes the current heating rate, the current temperature deviation and the current dye liquor temperature, wherein the current temperature deviation is the deviation between the current target process temperature and the current dye liquor temperature; The input unit is used to input the current dye vat data into a pre-established intelligent agent environment learning model based on the Q-learning algorithm to obtain a PID control signal at the current moment; the intelligent agent environment learning model based on the Q-learning algorithm is configured to: adopt an ε-greedy strategy to iteratively update the PID control parameters at the current moment corresponding to the current heating rate, the current temperature deviation and the current dye solution temperature until an iteration stop condition is met; The control unit is used to generate an opening signal of the dye vat burner according to the PID control signal at the current moment, so as to control the dye vat burner to heat the dye solution to the target process temperature at the current moment in an opening state corresponding to the opening signal.

8. A computer program product, characterized in that It includes computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the Q-learning-based adaptive PID control method for dyeing vat process temperature as described in any one of claims 1 to 6.

9. A Q-learning-based dye vat process temperature adaptive PID control device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the Q-learning-based dyeing vat process temperature adaptive PID control device can implement the Q-learning-based dyeing vat process temperature adaptive PID control method as described in any one of claims 1 to 6.

10. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the Q-learning-based adaptive PID control method for dyeing vat process temperature as described in any one of claims 1 to 6.

Images