Control system, method and equipment for preparing dust reducing agent based on reinforcement learning

Abstract

The invention discloses a control system, method and equipment for preparing a dust-reducing agent based on reinforcement learning, and the system comprises a sensing unit which is used for collecting the technological parameters in the process of synthesizing and preparing the dust-reducing agent in real time, and comprises the steps: respectively measuring the mass of each raw material, calculating the proportion, and transmitting the proportion to a control unit; measuring the temperature, the air humidity, the pH value and the liquid level height in the reaction kettle; the decision-making unit is used for interacting with a preparation environment based on a target reinforcement learning algorithm and the data collected by the sensing unit so as to learn a target control strategy of the process parameters; and the execution unit is used for adjusting one or more technological parameters for synthesizing and preparing the dust-reducing agent according to the target control strategy output by the decision-making unit. Through the scheme of the invention, the optimization can be continuously learned, and the generalization ability is improved so as to adapt to complex and changeable working conditions.

Description

Technical Field

[0001] The present application relates to the field of control, and in particular to a control system, method and equipment for preparing dust suppressant based on reinforcement learning. Background Art

[0002] Dust suppressant is a chemical agent used to suppress the flying of dust. It is widely used in coal, electricity, metallurgy and other industries. It is of great significance to improve the working environment, protect the health of employees, and increase the service life of equipment. However, the preparation process of dust suppressant involves the ratio of multiple raw materials and the control of process parameters. If it cannot be accurately controlled, the performance of the dust suppressant will be unstable, affecting its use effect. Therefore, it is very important to develop an efficient and intelligent dust suppressant preparation control system to ensure the quality of dust suppressant and improve production efficiency. At present, the dust suppressant preparation control mainly adopts the traditional PID control method, which sets the target value, measures the actual value, calculates the deviation value, and adjusts the control amount according to the deviation size to make the actual value as close to the target value as possible. Although this control method realizes the automatic adjustment of process parameters to a certain extent, there are still some problems: first, PID control relies on an accurate mathematical model, and the dust suppressant preparation process is complex, involving multiple physical and chemical reactions, and it is difficult to establish an accurate mathematical model; second, PID control belongs to feedback control, there is a certain hysteresis, and it cannot respond to the fluctuation of process parameters in time; third, PID control lacks self-learning and self-adaptation capabilities, and it is difficult to cope with changes in factors such as raw material quality and environmental conditions. These problems lead to low control accuracy in the dust suppressant preparation process, unstable product quality, and high energy and material consumption.

[0003] In order to solve the above problems, researchers began to explore the application of intelligent control technology in the preparation of dust suppressants. Among them, fuzzy control based on expert experience and data-driven intelligent optimization algorithms (such as neural networks, genetic algorithms, etc.) have attracted attention. These methods have overcome the shortcomings of PID control to a certain extent and improved the accuracy and adaptability of control. However, the establishment of fuzzy control rules depends on expert experience and is greatly affected by human factors; intelligent optimization algorithms require a large amount of historical data for training, while the working conditions of dust suppressant preparation are changeable, and sample data is difficult to fully cover, resulting in insufficient algorithm generalization ability.

[0004] Therefore, there is an urgent need for a technical solution that can continuously learn and optimize, improve generalization capabilities, and adapt to complex and changing working conditions. Summary of the invention

[0005] In order to solve the deficiencies of the prior art, the embodiments of the present application provide a control system, method and device for preparing dust suppression agent based on reinforcement learning. The present application solves the technical problems such as insufficient generalization ability of the prior art algorithm.

[0006] An embodiment of the present application provides a control system for preparing dust suppressants based on reinforcement learning, comprising: a perception unit, a decision unit and an execution unit; wherein the perception unit is used to collect process parameters in the process of synthesizing and preparing dust suppressants in real time, including: measuring the masses of various raw materials respectively to calculate the ratio, and measuring the temperature, air humidity, pH and liquid level in the reactor; the decision unit is used to interact with the preparation environment based on the target reinforcement learning algorithm and the data collected by the perception unit to learn the target control strategy of the process parameters; the execution unit is used to adjust one or more process parameters of the synthesis of the dust suppressant according to the target control strategy output by the decision unit.

[0007] In a possible implementation, the target reinforcement learning algorithm and the data collected by the perception unit interact with the preparation environment to learn the target control strategy of the process parameters, including: using the data collected by the perception unit as the input of the target reinforcement learning algorithm, and constructing the current state to obtain the target adjustment value of the preparation process parameter; transmitting the target adjustment value to the execution unit to generate control instructions for the target process equipment; calling the perception unit to collect the adjusted process parameter data, and calculating the instant reward value; updating the intelligent agent table according to the current state and instant reward value constructed by the process parameter data to obtain a new control strategy; performing iterative calculations and interactive adjustments to obtain the target control strategy when the intelligent agent table converges.

[0008] In a possible implementation, the data collected by the perception unit is used as the input of the target reinforcement learning algorithm, and the current state is constructed to obtain the target adjustment value of the preparation process parameter, including: preprocessing the data collected by the perception unit, and extracting the target key features from the preprocessed data to construct a feature vector describing the current process parameter state; inputting the feature vector into the target reinforcement learning algorithm, and the target reinforcement learning algorithm is used to select random actions multiple times based on a first probability based on a greedy strategy to construct a random action library; select the current optimal action in the random action library according to a second probability; and map the current optimal action back to the mathematical space of the process parameters to obtain the target adjustment value of the preparation process parameters.

[0009] In a possible implementation, the calling perception unit collects adjusted process parameter data and calculates an immediate reward value, including: obtaining a sample of the current dust reduction agent being prepared, and obtaining the dust reduction efficiency of the sample through a dust concentration sensor; based on the collected adjusted process parameter data, calculating the sum of the material cost and energy cost of the current raw material ratio to obtain the preparation cost; based on the dust reduction efficiency and preparation cost, constructing a target reward function to calculate the immediate reward value.

[0010] In a possible implementation, the current state and instant reward value constructed according to the process parameter data are used to update the intelligent agent table to obtain a new control strategy, including: preprocessing the adjusted process parameter data to obtain a multi-dimensional state vector; using the state vector and the instant reward value as a new set of samples, and performing a discount calculation on the intelligent agent table to obtain a table update value; and accumulating the table update value on the intelligent agent table to update the intelligent agent table.

[0011] In a possible implementation, the target reward function is constructed based on the dust reduction efficiency and the preparation cost to calculate the immediate reward value, including: ,in, Indicates in status Execute control instructions The immediate return value, represents the first weight, Indicates the dust concentration when no dust suppressant is used. Indicates the dust concentration after using the dust suppressant sample, represents the second weight, Indicates The amount of raw materials used, Indicates The unit price of raw materials, Indicates the total number of raw material types, represents the energy consumption during the preparation process, Represents the price per unit of energy.

[0012] In a possible implementation, the state vector and the instant reward value are used as a set of new samples, and the agent table is discounted to obtain the table update value, including: ,in, Indicates the updated value of the table. represents the learning rate, represents the immediate return value, represents the discount factor, Indicates the next state All possible actions The maximum value among the corresponding Q values, It is the state-action value function, which represents the expected value of the long-term cumulative reward that the agent can obtain by interacting with the environment after taking action a in state s.

[0013] An embodiment of the present application also provides a control method for preparing dust suppressants based on reinforcement learning, including: real-time collection of process parameters in the process of synthesizing and preparing dust suppressants; based on the target reinforcement learning algorithm and the data collected by the perception unit, interacting with the preparation environment to learn the target control strategy of the process parameters; and adjusting one or more process parameters of the synthesized preparation of dust suppressants according to the target control strategy output by the decision unit.

[0014] An embodiment of the present application also provides a control device for preparing dust reduction agents based on reinforcement learning, comprising: a processor, a memory, and a system bus; wherein the processor and the memory are connected via the system bus; the memory is used to store one or more programs, and the one or more programs include instructions, which, when executed by the processor, enable the processor to execute the method described in the above embodiment.

[0015] In a control system, method and device for preparing dust suppressants based on reinforcement learning as provided above, the embodiments of the present application learn the optimal control strategy through the interaction between the reinforcement learning agent and the environment, and can learn from the operating experience of experts to obtain the initial control strategy, avoid exploration from scratch, and improve learning efficiency. Further, in some embodiments, reinforcement learning can optimize the control strategy through continuous trial and error, adapt to changes in working conditions, overcome the lag of PID control and the empirical dependence of fuzzy control. Further, in some embodiments, reinforcement learning does not require a large amount of historical data, but gradually improves the control strategy through autonomous learning of the agent, reduces the requirements for data volume, and improves the generalization ability of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 A schematic block diagram of a control system for preparing dust suppression agents based on reinforcement learning provided in an embodiment of the present application; Figure 2 A flow chart of a target control strategy for learning process parameters provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present application.

[0019] Those skilled in the art will appreciate that the terms "first", "second" and the like in the embodiments of the present application are only used to distinguish different steps, devices or modules, etc., and do not represent any specific technical meaning, nor do they represent the necessary logical order between them. It should also be understood that in the embodiments of the present application, "multiple" may refer to two or more, and "at least one" may refer to one, two or more. It should also be understood that for any component, data or structure mentioned in the embodiments of the present application, in the absence of explicit limitation or contrary revelation given in the context, it can generally be understood as one or more. In addition, the term "and / or" in the present application is only a kind of association relationship describing the associated objects, indicating that there can be three kinds of relationships, for example, A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present application generally indicates that the associated objects before and after are an "or" relationship. It should also be understood that the description of each embodiment in the present application emphasizes the differences between the embodiments, and the same or similar parts can refer to each other. For the sake of brevity, they will not be repeated one by one.

[0020] At the same time, it should be understood that, for ease of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present application and its application or use. The techniques, methods and devices known to ordinary technicians in the relevant fields may not be discussed in detail, but where appropriate, the techniques, methods and devices should be considered as part of the specification. It should be noted that similar numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0021] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0022] Figure 1A schematic block diagram of a control system for preparing dust suppressants based on reinforcement learning provided in an embodiment of the present application. It should be understood that the system shown in the figure is exemplary and not restrictive. This means that the system architecture involved is not limited to a specific form or design, but is presented as an example. In other words, the architecture shown in the figure can be regarded as a way of expression to clearly describe related concepts and relationships, and does not exclude other forms of architecture. Therefore, when interpreting the architecture in the picture, it should be understood that the model is flexible and diverse, and its purpose is to provide an exemplary description rather than a restrictive provision on a specific form.

[0023] like Figure 1 As shown, the system includes a sensing unit 101, a decision unit 102 and an execution unit 103. Specifically, the sensing unit 101 is used to collect process parameters in the process of synthesizing and preparing dust suppressants in real time. Specifically, the masses of various raw materials are measured respectively to calculate the proportion, and the temperature, air humidity, pH and liquid level in the reactor are measured.

[0024] In one embodiment, the sensor of the sensing unit in the present application can be set as: a raw material ratio sensor, which uses a weighing sensor to measure the mass of various raw materials respectively and calculate the ratio. A temperature sensor, which uses a thermocouple or a thermal resistor to measure the temperature in the reactor. A humidity sensor, which uses a capacitive humidity sensor to measure the air humidity in the reactor. A pH sensor, which uses a glass electrode pH meter to measure the acidity and alkalinity of the reaction solution. A liquid level sensor, which uses an ultrasonic liquid level meter to measure the liquid level in the reactor. The data of each sensor is transmitted to the decision unit through a bus or a wireless network for processing.

[0025] The decision unit 102 is used to interact with the preparation environment based on the target reinforcement learning algorithm and the data collected by the perception unit to learn the target control strategy of the process parameters. It should be understood that the decision unit is the core of this system, and the reinforcement learning algorithm is used to optimize the dust suppression agent preparation process. Reinforcement learning is a machine learning method that learns the optimal strategy through trial and error interaction with the environment. Its basic principle is as follows: the agent (Agent) at each moment Observe the state of the environment , and according to the strategy Take Action ; Environment according to action Entering a new state , and give the agent a reward ; The agent is rewarded and the new state Adjustment strategy , so that the long-term cumulative reward is maximized. Strategy π(a|s) is the probability distribution of the agent choosing action a in state s. In the dust suppression agent preparation system, the strategy determines how to adjust the control instructions (actions) according to the current process parameters (state).

[0026] The embodiment of the present application adopts a value-based reinforcement learning algorithm, namely, the Q-learning algorithm. The core of the Q-learning algorithm is to learn a Q function, which represents the state Next Action The long-term cumulative reward expectation of: ,in, is a discount factor used to balance the importance of current rewards and future rewards. represents the expected value; k is the index of the time step, indicating the number of decay steps of future rewards, represents the discount factor after the kth step. The Q function can be iteratively updated by the Bellman equation: ,in, is the learning rate, which controls the magnitude of each update.

[0027] In the embodiment of the present application, the state Various parameters in the preparation process of dust suppressants, including raw material ratio, temperature, humidity, pH value, etc.; Action To adjust the working parameters of each actuator, the raw material feed amount, heating power, stirring speed, etc. can be adjusted; the reward r is designed according to the performance indicators of the prepared dust suppressant (such as dust suppression efficiency, cost, etc.). The better the performance, the higher the reward.

[0028] In order to improve the convergence speed and stability of the Q-learning algorithm, this application adopts the following improvement measures: experience replay, transfer samples obtained by the interaction between the agent and the environment The training data is stored in the experience pool, and a batch of samples are randomly selected from the experience pool for training each time, which breaks the correlation between samples and improves the training efficiency and stability. Double Q-Network uses two Q networks with the same structure but different parameters. and , each time an update is made, one network is responsible for selecting the optimal action, and the other network is responsible for evaluating the action The target network uses two Q networks with the same structure but different parameters. and ,in For the current network, For the target network, the parameters of the current network are copied to the target network every certain number of steps, which improves the stability of training.

[0029] Strategy is a classic method for balancing exploration and exploitation in reinforcement learning, using probability Randomly select actions (explore the effects of unknown actions) with probability Select the action with the largest current Q value (using the known optimal action). In the preparation of dust suppressant, Allows the system to try new combinations of process parameters (exploration) while leveraging existing experience to maintain efficiency (exploitation).

[0030] Based on the above improvement measures, the Q-learning algorithm process of this system is as follows: Initialize the current Q network and the target Q network , experience pool , exploration rate Repeat the following steps until convergence: Strategy selection action , that is, The probability of randomly selecting an action is The probability of choosing The action with the largest value. Execute the action , observe the reward and the new state , the sample will be transferred Store in experience pool From the experience pool Randomly select a batch of transfer samples from For each sample, calculate the target value: . Update the current Q network Parameters, minimize the loss function: ,in, is the parameter of the current Q network. At certain intervals, the parameters of the current Q network are copied to the target Q network: , and reduce the exploration rate .

[0031] Furthermore, in one embodiment, the data collected by the perception unit is used as the input of the target reinforcement learning algorithm, and the current state is constructed to obtain the target adjustment value for the raw material ratio; the target adjustment value is transmitted to the execution unit to generate the control instruction of the target process equipment; the perception unit is called to collect the adjusted process parameter data, and the instant reward value is calculated; the intelligent agent table is updated according to the current state and instant reward value constructed by the process parameter data to obtain a new control strategy; iterative calculation and interactive adjustment are performed to obtain the target control strategy when the intelligent agent table converges.

[0032] The execution unit 103 is used to adjust one or more process parameters of the synthesis and preparation of the dust suppression agent according to the target control strategy output by the decision unit. It should be noted that the execution unit is responsible for adjusting the working parameters of each actuator according to the control strategy output by the decision unit to realize automatic control of the dust suppression agent preparation process.

[0033] Specifically, the raw material feeding system controls the feeding amount and feeding speed of each raw material according to the feedback of the raw material ratio sensor and the output of the decision unit to ensure the accuracy of the raw material ratio. Temperature control system: According to the feedback of the temperature sensor and the output of the decision unit, the power of the heater is controlled to maintain the temperature in the reactor within the optimal range. Humidity control system controls the working state of the humidifier or dehumidifier according to the feedback of the humidity sensor and the output of the decision unit to maintain the humidity in the reactor within the optimal range. pH control system controls the dosage of pH regulator according to the feedback of the pH sensor and the output of the decision unit to maintain the pH value of the reaction liquid within the optimal range. Stirring control system controls the speed and stirring time of the stirring motor according to the output of the decision unit to ensure that the reactants are fully mixed.

[0034] The working parameters of each actuator are adjusted through the PID control algorithm, that is, according to the deviation between the current state and the target state, the change value of the adjustment amount is calculated and iterated continuously until the target state is reached. The mathematical expression of the PID control algorithm is: ,in, To control the amount, is the deviation, They are proportional, integral and differential coefficients respectively, and can be debugged according to the specific control object and performance requirements.

[0035] Figure 2 A flow chart of a target control strategy for learning process parameters provided in an embodiment of the present application. It should be noted that reinforcement learning is a machine learning method that learns the optimal strategy through the interaction between an agent and an environment. , the agent observes the state of the environment , according to the current strategy Select an action , after execution, the environment state is transferred to , and the agent receives a reward This process can be done with a five-tuple To describe, among them, Represents the state space, which contains all possible environmental states; represents the action space, which contains all the actions that the agent can perform; represents the state transition probability, Indicates in status Next action After transfer to state probability; represents the reward function, Indicates in status Next action Instant rewards received; Represents a discount factor that balances immediate rewards and long-term rewards.

[0036] The goal of reinforcement learning is to learn an optimal policy , so that the cumulative reward obtained by the agent in the process of interacting with the environment is maximized: ,in, represents the expected value, represents the cumulative reward. The optimal strategy The goal is to maximize the cumulative reward (i.e., balance dust removal efficiency and cost). In Q-learning, the strategy is usually implemented through a greedy strategy: select the action with the largest current Q value (exploitation), or randomly explore new actions (exploration).

[0037] Specifically, at step S201, the data collected by the perception unit is used as the input of the target reinforcement learning algorithm, and the current state is constructed to obtain the target adjustment value of the preparation process parameter. The embodiment of the present application adopts a reinforcement learning algorithm based on Q-learning to learn the optimal control strategy by continuously interacting with the environment. The core of the Q-learning algorithm is the agent table, which stores the expected reward of each action taken in each state. The agent table can be represented by a two-dimensional array To indicate that Indicates the status, Indicates action.

[0038] In the process of preparing dust suppressant, the state We can use a five-dimensional vector To represent, they correspond to the proportion, temperature, humidity, pH and liquid level. Action a can be a four-dimensional vector , indicating the adjustment amount for each raw material ratio.

[0039] In one embodiment, the perception unit collects data such as the ratio, temperature, humidity, pH, and liquid level in real time and transmits them to the decision unit. The decision unit uses these data as the input of the reinforcement learning algorithm and takes an action a according to the current state s, that is, adjusts the ratio of each raw material. For example, the perception unit collects the following data in real time through various sensors. Ratio: A: B: C: D = 2: 3: 4: 1, temperature in the reactor: T = 80 ° C, air humidity: H = 60%, pH: pH = 7.5, liquid level: L = 2.5m.

[0040] These data will be used as input to the reinforcement learning algorithm to optimize the control strategy of the decision-making unit. The current status is , that is, the ratio is , the temperature is 80℃, the humidity is 60%, the pH is 7.5, and the liquid level is 2.5m. The decision-making unit selects an action based on the agent table. , that is, increase the ratio of A by 0.1, decrease the ratio of B by 0.2, increase the ratio of C by 0.1, and keep the ratio of D unchanged.

[0041] Furthermore, in another embodiment, the data collected by the sensing unit is preprocessed, and the target key features are extracted from the preprocessed data to construct a feature vector describing the current state of the process parameters; the feature vector is input into a target reinforcement learning algorithm, and the target reinforcement learning algorithm is used to, based on a greedy strategy, select the target key features according to the first probability Select random actions multiple times to build a random action library; according to the second probability 1- The current optimal action is selected from the random action library; and the current optimal action is mapped back to the mathematical space of the process parameters to obtain the target adjustment value of the preparation process parameters.

[0042] At step S202, the target adjustment value is transmitted to the execution unit to generate a control instruction for the target process equipment. Specifically, the execution unit adjusts the operating state of each process equipment according to the control instruction of the decision unit, such as adjusting the feed valve, heater power, etc. In the embodiment described above, the execution unit performs the operation according to this action instruction. , adjust the feed valve and adjust the actual ratio of A, B, C, and D to 2.1:2.8:4.1:1.

[0043] In step S203, the perception unit is called to collect the adjusted process parameter data and calculate the instant reward value. In this application, the dust suppression agent preparation process is modeled as a Markov decision process (MDP), and its state space Including process parameters such as raw material ratio, temperature, pH value, stirring speed, etc., the action space A includes the operations of adjusting various process parameters. The reward function R is designed according to product quality indicators (such as dust suppression efficiency, cost, etc.), for example: ,in, and is the weight coefficient, which is used to balance the importance of different quality indicators.

[0044] Specifically, a sample of the dust suppression agent currently being prepared is obtained, and the dust suppression efficiency of the sample is obtained through a dust concentration sensor. The dust suppression effect of dust suppressant on dust can be calculated using the following formula: ,in, Indicates the dust concentration when no dust suppressant is used. Indicates the dust concentration after using dust suppressant.

[0045] In order to get and , dust concentration sensors can be installed at the production site to monitor the changes in dust concentration in real time. For example, a laser dust sensor can be used, which calculates the dust concentration by measuring the scattering intensity of the laser beam by dust particles. For example, at a certain moment, the dust concentration without using dust suppressant is , the dust concentration after using dust suppressant is reduced to , then the dust suppression efficiency at this time is: .

[0046] Based on the collected and adjusted process parameter data, the sum of the material cost and energy cost of the current raw material ratio is calculated to obtain the preparation cost. The cost of dust suppressant preparation mainly includes raw material cost and energy cost. The following formula can be used to calculate the preparation cost of dust suppressant per unit mass: ,in, Indicates The amount of raw materials used, Indicates The unit price of raw materials, represents the energy consumption during the preparation process, Represents the price per unit of energy.

[0047] In order to obtain these parameters, various sensors and instruments, such as flow meters and electric meters, can be installed at the production site to monitor the raw material usage and energy consumption in real time. At the same time, it is also necessary to establish a price database for raw materials and energy and update it regularly according to market conditions.

[0048] For example, to prepare 1 ton of dust suppressant, Ton of raw material A (unit price Yuan / ton), Tons of raw material B (unit price Yuan / ton), power consumption E=500 kWh (unit price Yuan / degree), then the preparation cost of dust suppressant per unit mass is: .

[0049] Get dust suppression efficiency and cost After that, they can be used as important inputs of the reinforcement learning algorithm to construct the reward function r. Based on the dust reduction efficiency and preparation cost, a target reward function is constructed to calculate the immediate return value. Specifically, it can be expressed as: ,in, Indicates in status Execute control instructions The immediate return value, represents the first weight, which can be 0.73. Indicates the dust concentration when no dust suppressant is used. Indicates the dust concentration after using the dust suppressant sample, represents the second weight, which can be 0.27. Indicates The amount of raw materials used, Indicates The unit price of raw materials, Indicates the total number of raw material types, represents the energy consumption during the preparation process, Represents the price per unit of energy.

[0050] At each time step, the reinforcement learning algorithm selects an action (such as adjusting the raw material ratio, increasing the temperature, etc.) according to the current state (such as raw material ratio, temperature, pH value, etc.), and obtains a new state and reward after execution (calculated based on the new dust suppression efficiency and cost). The parameters of the strategy network are updated accordingly, and continuous learning and optimization are carried out to finally obtain an optimal control strategy that can automatically adjust process parameters, maximize dust suppression efficiency, and minimize costs.

[0051] At step S204, the agent table is updated according to the current state and instant reward value constructed by the process parameter data to obtain a new control strategy. The adjusted process parameter data is preprocessed to obtain a multi-dimensional state vector; the state vector and the instant reward value are used as a set of new samples, and the agent table is discounted to obtain a table update value; the table update value is accumulated on the agent table to update the agent table. The table update value can be expressed as: ,in, Indicates the updated value of the table. represents the learning rate, represents the immediate return value, represents the discount factor, Indicates the next state All possible actions The maximum value among the corresponding Q values, It is the state-action value function, which represents the expected value of the long-term cumulative reward that the agent can obtain by interacting with the environment after taking action a in state s.

[0052] For example, the current state is s=(2, 3, 4, 1, 80, 60, 7.5, 2.5), that is, the ratio is A: B: C: D = 2: 3: 4: 1, the temperature is 80℃, the humidity is 60%, the pH is 7.5, and the liquid level is 2.5m. The decision-making unit selects an action a=(0.1, -0.2, 0.1, 0) based on the agent table, that is, to increase the ratio of A by 0.1, reduce the ratio of B by 0.2, increase the ratio of C by 0.1, and keep the ratio of D unchanged. According to this action instruction, the execution unit adjusts the feed valve and adjusts the actual ratio of A, B, C, and D to 2.1: 2.8: 4.1: 1. The perception unit collects the adjusted data and finds that the energy consumption has been reduced by 1%, and calculates the immediate reward r=0.5.

[0053] The decision-making unit updates the corresponding values ​​in the agent table according to the new state s'=(2.1, 2.8, 4.1, 1,) and the immediate reward R=0.5: , such as the learning rate , discount factor , then the value of action a taken in state s in the agent table will be updated to: By repeating this process continuously, the agent table will converge to the optimal value, and the decision unit will also learn the optimal control strategy, that is, at step S205, iterative calculation and interactive adjustment are performed to obtain the target control strategy when the agent table converges.

[0054] Furthermore, an embodiment of the present application also provides a control device for preparing dust reduction agents based on reinforcement learning, comprising: a processor, a memory, and a system bus; the processor and the memory are connected via the system bus; the memory is used to store one or more programs, and the one or more programs include instructions, which, when executed by the processor, enable the processor to execute any of the above methods.

[0055] Furthermore, an embodiment of the present application also provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute any of the above methods.

[0056] It can be known from the description of the above implementation mode that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0057] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0058] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0059] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A control system for preparing dust suppression agent based on reinforcement learning, characterized in that: include: Perception unit, decision unit and execution unit; among them, The sensing unit is used to collect the process parameters in the process of synthesizing and preparing the dust suppressant in real time, including: measuring the mass of various raw materials respectively to calculate the proportion, and measuring the temperature, air humidity, pH and liquid level in the reactor; The decision unit is used to interact with the preparation environment based on the target reinforcement learning algorithm and the data collected by the perception unit to learn the target control strategy of the process parameters; The execution unit is used to adjust one or more process parameters for synthesizing and preparing the dust suppression agent according to the target control strategy output by the decision unit.

2. The control system according to claim 1, characterized in that: in, The target-based reinforcement learning algorithm and the data collected by the perception unit interact with the preparation environment to learn the target control strategy of the process parameters, including: The data collected by the sensing unit is used as the input of the target reinforcement learning algorithm, and the current state is constructed to obtain the target adjustment value of the preparation process parameter; transmitting the target adjustment value to the execution unit to generate a control instruction for the target process equipment; Call the perception unit to collect the adjusted process parameter data and calculate the instant feedback value; Update the agent table based on the current state and immediate reward value constructed by process parameter data to obtain a new control strategy; Iterative calculations and interactive adjustments are performed to obtain the target control strategy when the agent table converges.

3. The control system according to claim 2, characterized in that: in, The data collected by the sensing unit is used as the input of the target reinforcement learning algorithm, and the current state is constructed to obtain the target adjustment value of the preparation process parameter, including: Preprocess the data collected by the sensing unit and extract the target key features from the preprocessed data to construct a feature vector describing the current state of the process parameters; Inputting the feature vector into a target reinforcement learning algorithm, wherein the target reinforcement learning algorithm is used to select random actions multiple times according to a first probability based on a greedy strategy to build a random action library; Selecting a current optimal action from the random action library according to a second probability; The current optimal action is mapped back to the mathematical space of the process parameters to obtain the target adjustment value of the preparation process parameters.

4. The control system according to claim 2, characterized in that: in, The calling sensing unit collects the adjusted process parameter data and calculates the instant feedback value, including: Obtain a sample of the currently prepared dust reduction agent, and obtain the dust reduction efficiency of the sample through a dust concentration sensor; Based on the collected and adjusted process parameter data, the sum of the material cost and the energy cost of the current raw material ratio is calculated to obtain the preparation cost; Based on the dust reduction efficiency and preparation cost, a target reward function is constructed to calculate an immediate reward value.

5. The control system according to claim 2, characterized in that: in, The current state and instant reward value constructed according to the process parameter data are used to update the agent table to obtain a new control strategy, including: Processing the adjusted process parameter data to obtain a multi-dimensional state vector; Take the state vector and the instant reward value as a new set of samples and discount the agent table to get the table update value; Accumulate the table update value on the agent table to update the agent table.

6. The control system according to claim 4, characterized in that: in, Based on the dust reduction efficiency and preparation cost, a target reward function is constructed to calculate the immediate return value, including: ， in, Indicates in status Execute control instructions The immediate return value, represents the first weight, Indicates the dust concentration when no dust suppressant is used. Indicates the dust concentration after using the dust suppressant sample, represents the second weight, Indicates The amount of raw materials used, Indicates The unit price of raw materials, Indicates the total number of raw material types, represents the energy consumption during the preparation process, Represents the price per unit of energy.

7. The control system according to claim 5, characterized in that: in, The state vector and the instant reward value are used as a new set of samples, and the agent table is discounted to obtain the table update value, including: ， in, Indicates the updated value of the table. represents the learning rate, represents the immediate return value, represents the discount factor, Indicates the next state All possible actions The maximum value among the corresponding Q values, It is the state-action value function, which represents the expected value of the long-term cumulative reward that the agent can obtain by interacting with the environment after taking action a in state s.

8. A control method for preparing dust suppression agent based on reinforcement learning, characterized in that: include: Real-time collection of process parameters during the synthesis and preparation of dust suppression agents; Based on the target reinforcement learning algorithm and the data collected by the perception unit, it interacts with the preparation environment to learn the target control strategy of the process parameters; According to the target control strategy output by the decision-making unit, one or more process parameters for synthesizing and preparing the dust suppressant are adjusted.

9. A control device for preparing dust suppression agent based on reinforcement learning, characterized in that: include: A processor, a memory, and a system bus; wherein the processor and the memory are connected via the system bus; The memory is used to store one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by the processor, the processor is caused to perform the method of claim 8.

Images