Raw material mixing control method and control system based on multi-agent reinforcement learning

Abstract

The application provides a raw material mixing control method and system based on multi-agent reinforcement learning and a computer readable medium. The raw material mixing control method comprises: regarding each tank as an independent agent, constructing a multi-agent reinforcement learning model, the model parameters of the multi-agent reinforcement learning model comprising: state, action and reward, the state comprising the actual weight of each tank, the action comprising adjusting the feeding speed of a single tank, and the reward comprising: giving a positive reward if the mixing ratio reaches the target ratio; training the multi-agent reinforcement learning model to obtain a trained model; obtaining the current state of n tanks in the raw material mixing device, inputting the current state into the trained model, and the trained model outputting a target action corresponding to the current state of each tank; and controlling the action of each tank as the target action.

Description

TECHNICAL FIELD

[0001] The present application mainly relates to the field of intelligent control, and in particular to a raw material mixing control method based on multi-agent reinforcement learning, a control system and a computer readable medium storing computer program code. BACKGROUND

[0002] Large industrial equipment often needs complex and precise control. For example, in raw material mixing control, a weighing sensor is usually installed under each tank to obtain the weight information of each tank, and the discharge speed of each tank is controlled by weight change and material level information. Meanwhile, other characteristic sensor data of each weighing sensor, such as voltage, current, temperature, load, etc., are also transmitted to the control terminal, and the control terminal displays these data to the user for manual intervention when problems occur. However, the current control method has the following problems:

[0003] (1) The discharge speed of each tank cannot be accurately controlled by weight and material level information alone;

[0004] (2) The stirring rate of the raw material mixing tank is usually constant, and there is currently a lack of intelligent means to automatically control the stirring rate;

[0005] (3) Once an abnormality occurs in a tank, the entire system will be in a downtime state, and manual troubleshooting is required, which affects work efficiency;

[0006] (4) Control parameters need to be set manually, which is easily affected by noise, interference and other factors, and the control accuracy is not high.

[0007] Therefore, it is desirable to use a more intelligent control method to overcome the above problems. SUMMARY

[0008] The technical problem to be solved by the present application is to provide a raw material mixing control method based on multi-agent reinforcement learning, a control system and a computer readable medium storing computer program code, which can realize intelligent control of raw material mixing.

[0009] To address the aforementioned technical problems, this application provides a raw material mixing control method based on multi-agent reinforcement learning, used to control a raw material mixing device. The raw material mixing device includes n material tanks, where n is a positive integer greater than or equal to 2. The method includes: treating each material tank as an independent agent and constructing a multi-agent reinforcement learning model. The model parameters of the multi-agent reinforcement learning model include: state, action, and reward. The state includes the actual weight of each material tank. The action includes adjusting the feeding speed of a single material tank. The reward includes: giving a positive reward if the mixing ratio reaches a target ratio; training the multi-agent reinforcement learning model to obtain a trained model; obtaining the current state of the n material tanks in the raw material mixing device, inputting the current state into the trained model, and the trained model outputting the target action corresponding to the current state of each material tank; and controlling the action of each material tank to be the target action.

[0010] In one embodiment of this application, the steps of constructing a multi-agent reinforcement learning model include: obtaining n policies corresponding to n agents; calculating the cumulative expected reward for each agent based on the n policies; calculating the policy gradient of the agent based on the cumulative expected reward, wherein the policy gradient is related to the action value function of the agent; predicting the action value function using a Transformer network to obtain a prediction function; and obtaining a deterministic policy gradient under a deterministic policy using the prediction function.

[0011] In one embodiment of this application, the step of calculating the cumulative expected reward for each agent based on the n policies includes:

[0012] The cumulative expected reward J(θ) of the i-th agent is calculated using the following formula. i ):

[0013]

[0014] Where θ=[θ1,θ2,…,θ n ] represents the policy parameters of n agents, θ i Let π represent the policy parameters of the i-th agent, where π = [π1, π2, ..., π]. n ] represents the policy of n agents. Indicates a policy parameter θ i The policy executed by the i-th agent. This indicates that the i-th agent is in the policy The following action a i p π Let γ represent the probability that the agent is in state s under policy π, γ represent the discount factor, t represent the number of time steps the agent takes to execute an action, and a represent the probability that the agent is in state s. irepresents an action taken by the ith agent, r i,t represents an action taken by the ith agent at the tth time step, i represents an immediate reward obtained after.

[0015] In an embodiment of the present application, the step of calculating the policy gradient of the agent according to the cumulative expected reward comprises: calculating the policy gradient of the ith agent J(θ i ) according to the following formula:

[0016]

[0017] wherein o i represents the observation of the ith agent, x = [o1, o2, …, o n ] represents an observation vector, i.e., a state of the multi-agent reinforcement learning model, represents the action value function of the ith agent, represents an optimization function of the policy gradient.

[0018] In an embodiment of the present application, the prediction function is obtained according to the following formula:

[0019]

[0020] wherein Attention represents a Transformer attention model, and MLP represents a fully connected layer, represents the prediction function, wherein the Transformer attention model is used to process the state x into a fixed-length vector.

[0021] In an embodiment of the present application, the step of obtaining the deterministic policy gradient under the deterministic policy using the prediction function comprises:

[0022] calculating the deterministic policy gradient of the ith agent according to the following formula:

[0023]

[0024] wherein D represents an experience storage containing tuples (x, x', a1, …, a n , r1, …, r n ), the tuples representing rewards r1, …, r n obtained when the agent performs actions a1, …, a n in a state x and actions taken in the next state x', represents an optimization function of the deterministic policy gradient.

[0025] In an embodiment of the present application, the loss function corresponding to the optimization function of the determined strategy gradient of the ith agent is

[0026]

[0027]

[0028] wherein, is a target strategy set with a delay parameter θ' i r i represents an immediate reward obtained by the ith agent after taking action a i , and j represents the jth agent, j = i + 1.

[0029] In an embodiment of the present application, the step of predicting the action value function by using the Transformer network comprises: inputting the state x into the Transformer network, the Transformer network outputting a state encoding vector h x = Encoder(x), wherein Encoder represents an encoder in the Transformer structure; representing the action a i of each agent as a one-hot encoding vector b i , and connecting the one-hot encoding vector b i and the state encoding vector h x to form an input vector v i = [h x , b i |, wherein b i = β1a1+…+β n a n , and β1+…+β n = 1; inputting the input vector v i into the Transformer network to obtain the prediction function (x, a1,..., a n ), wherein the Transformer network comprises the Transformer attention model and the fully connected layer.

[0030] In an embodiment of the present application, after the state x is input into the Transformer network, the following formula is used for preprocessing:

[0031] Input new = TEM(Input origin ) + (PE x Output EPL )

[0032] wherein, Input origin represents the state x, TEM represents a word embedding module, PE represents a position encoding, Output EPL represents the output of an external parameter layer EPL, which is updated in real time during model training, Input new represents a new state after pre-processing and entering the Transformer network.

[0033] In an embodiment of the present application, the Transformer network comprises at least two multi-layer perceptrons and at least two multi-head attention modules.

[0034] In an embodiment of the present application, the multi-layer perceptrons and the multi-head attention modules process the new state Input new according to the following formula:

[0035] Inout MHAM = MLP1 (Input new ) + MLP2 (Input new )

[0036]

[0037] wherein, MLP is a multi-layer perceptron comprising a plurality of fully connected layers, Inout MHAM represents the input of the multi-head attention module, represents the output of the multi-head attention module, MHAM represents a multi-head attention module, and RC represents a residual connection.

[0038] In an embodiment of the present application, the Transformer network further comprises a normalization layer NL and a feed-forward network layer FFN, and the output of the multi-head attention module is further processed according to the following formula:

[0039]

[0040]

[0041] wherein, represents the output of the nth multi-head attention module, and Output represents the output of the Transformer network.

[0042] In an embodiment of the present application, the Transformer network outputs the state encoding vector h x = Encoder (x) after being executed multiple times.

[0043] In an embodiment of the present application, the state further includes any of the following: the feeding speed of each tank, the tilting state of each tank, the temperature and humidity of each tank, the temperature of the mixing pool, the mixture ratio in the mixing pool, the distance between the tank and the mixing pool.

[0044] In an embodiment of the present application, the action further includes any of the following: adjusting the stirring speed of the mixing pool, adjusting the tank posture, adjusting the position of the tank discharge port, adjusting the distance between the tank and the mixing pool.

[0045] In an embodiment of the present application, the reward further includes any of the following: giving a negative reward if the mixture ratio deviates from the target ratio; giving a negative reward if the tank has problems such as tilting or weight abnormalities; giving a positive reward if the task is completed within the specified time; giving a positive reward if the damage or downtime of the equipment is short after completing the task; giving a positive reward if the equipment has no failure after completing the task.

[0046] In an embodiment of the present application, further comprising: establishing a time-sensitive network, taking each of the tanks as a slave node in the time-sensitive network, setting a master node, and synchronizing the clock of each of the slave nodes in real time by the master node when the raw material mixing device is running.

[0047] The present application also proposes a raw material mixing control system based on multi-agent reinforcement learning for controlling a raw material mixing device, the raw material mixing device including n tanks, n being a positive integer greater than or equal to 2, characterized in that it comprises: a memory for storing instructions executable by a controller; and the controller for executing the instructions to implement the method as described above.

[0048] The present application also proposes a computer readable medium storing computer program code, which, when executed by a processor, implements the method as described above.

[0049] The raw material mixing control method of the present application highly intelligently controls the action of each tank through a multi-agent reinforcement learning model, is high in precision, easy to operate, does not require human intervention, and can realize intelligent control of the raw material mixing device. By using a Transformer structure instead of a traditional neural network structure, the model convergence can be accelerated and the prediction accuracy can be improved. In addition, the real-time performance of the system is further improved by combining a time-sensitive network, and the reliable transmission of data can be ensured to improve the stability of the system. BRIEF DESCRIPTION OF DRAWINGS

[0050] The accompanying drawings are included to provide a further understanding of the present application, and are incorporated in and constitute apart of this application, illustrate embodiments of the present application, and together with the description serve to explain the principles of the present application. In the drawings:

[0051] Figure 1 is an exemplary flow chart of a raw material mixing control method according to an embodiment of the present application;

[0052] Figure 2 is an exemplary structural schematic diagram of a multi-agent reinforcement learning model;

[0053] Figure 3 is a Transformer network structure according to an embodiment of the present application;

[0054] Figure 4 is a structural schematic diagram of a raw material mixing control system according to an embodiment of the present application;

[0055] Figure 5 is a system block diagram of a raw material mixing control system according to an embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some examples or embodiments of the present application, and for those skilled in the art, the present application can also be applied to other similar scenarios without creative labor. Unless the context clearly indicates otherwise or otherwise stated, the same reference numbers in the drawings represent the same structure or operation.

[0057] As shown in the present application and claims, unless the context clearly indicates otherwise or otherwise stated, the words "one", "a", "an", and / or "the" do not specifically refer to the singular, but can also include the plural. Generally speaking, the terms "comprise" and "include" only indicate the inclusion of the steps and elements explicitly identified, and these steps and elements do not constitute an exclusive list, and the method or device can also include other steps or elements.

[0058] The foregoing is considered as illustrative only of the principles of the application. Further, since numerous modifications and changes will readily occur to those skilled in the art, it is not desired to limit the application to the exact construction and practice described. Accordingly, all such variations are intended to be included within the scope of the present application as defined in the following claims, along with full equivalents thereof.

[0059] It is also to be understood that the terminology used herein is for the purpose of describing the particular embodiments only and is not intended to be limiting. It is further noted that, as used in this specification, the singular form "a", "an" and "the" include plural references unless the context clearly dictates otherwise. It is to be understood that the terms "comprising", "including", "containing", and "having" are to be construed open- ended, i.e., to mean including but not limited to. It is also to be understood that the terms "coupled" and "coupling" refer to direct or indirect coupling or linking, and are not necessarily intended to denote a direct connection between two elements. It is further noted that the terms "coupled" and "coupling" are used synonymously with the terms "connected" and "connecting", respectively. It is to be understood that the terms "connected" and "connecting" are not to be construed as necessarily meaning directly connected or directly connecting.

[0060] Flow diagrams as can be used in accordance with embodiments of the present application to illustrate the operation of a system, according to embodiments of the present application. It will be understood that the acts shown in the figures are not necessarily to be performed in the order shown. Rather, various acts can be performed in different orders or concurrently. Further, other acts can be added or removed from the processes shown.

[0061] The raw material mixing control method of the present application is used to control a raw material mixing device, which includes n tanks, where n is a positive integer greater than or equal to 2. The function of the raw material mixing device is to input materials from the n tanks into a raw material mixing pool, respectively, to form a desired mixed raw material. Each tank has a corresponding sensor group, which can include a weighing sensor, a flow meter, a temperature and humidity sensor, a distance sensor, etc., to obtain the weight, feeding speed, inclination state, temperature and humidity, mixture ratio, distance between the tank and the mixing pool, etc. of the tank in real time or periodically. Therefore, it can be considered that all the parameter information required in the raw material control method of the present application is available. The present application does not limit the specific raw material.

[0062] Figure 1is an exemplary flowchart of a raw material mixing control method according to an embodiment of the present application. Referring to Figure 1 The raw material mixing control method according to the embodiment includes the following steps, as shown in

[0063] Step S110: Construct a multi-agent reinforcement learning model by taking each tank as an independent agent, and the model parameters of the multi-agent reinforcement learning model include state, action and reward, the state includes the actual weight of each tank, the action includes adjusting the feeding speed of a single tank, and the reward includes giving a positive reward if the mixing ratio reaches the target ratio;

[0064] Step S120: Train the multi-agent reinforcement learning model to obtain a trained model;

[0065] Step S130: Obtain the current state of the n tanks in the raw material mixing device, input the current state into the trained model, and the trained model outputs the target action corresponding to the current state of each tank; and

[0066] Step S140: Control the action of each tank as the target action.

[0067] According to the raw material mixing control method of the present application, the action of each tank can be highly intelligently controlled by the multi-agent reinforcement learning model, which is high in precision, easy to operate, does not require human intervention, and can realize intelligent control of the raw material mixing device. The steps S110-S140 described above are specifically explained below.

[0068] Figure 2 is an exemplary structural schematic diagram of a multi-agent reinforcement learning model. Referring to Figure 2 As shown, a multi-agent reinforcement learning model includes multiple agents, such as agents 1-n, corresponding to n tanks, and each tank is taken as an independent agent.

[0069] In step S110, the structure of the constructed multi-agent reinforcement learning model is as shown in Figure 2 Each agent has corresponding observations, actions, policies and rewards. In a Markov process, the learner and decision maker is called an agent, and the part interacting with the agent is called an environment. The agent and the environment constantly interact, and this interaction process can be regarded as multiple time points. At each time point, the agent selects an action according to a certain policy based on the state of the environment, and the environment is transferred to the next state according to a certain state transition probability, and at the same time, the agent is fed back a reward according to the good or bad state. The agent can adjust its policy according to the feedback of the environment, continue to learn in the environment, and finally learn an optimal policy that can obtain the most or optimal reward. A complete Markov decision process can be represented by a sequence composed of state, action and reward at each time point. Figure 2The observation shown represents the observed state of the environment. In this application, the state includes at least the actual weight of each tank, the action includes at least adjusting the feeding speed of a single tank, and the reward includes at least a positive reward if the mixing ratio reaches the target ratio.

[0070] In some embodiments, the states also include any of the following: the feeding rate of each tank, the tilt state of each tank, the temperature and humidity of each tank, the temperature of the mixing tank, the proportion of the mixture in the mixing tank, and the distance between the tanks and the mixing tank. It should be noted that, to obtain these states, the raw material mixing device should also be equipped with corresponding signal sensing devices, such as weighing sensors, flow meters, accelerometers, temperature and humidity meters, distance sensors, etc.

[0071] In some embodiments, the actions further include any of the following: adjusting the stirring speed of the mixing tank, adjusting the orientation of the material tank, adjusting the position of the material tank outlet, and adjusting the distance between the material tank and the mixing tank. Similarly, in order to perform these actions, the raw material mixing device should have corresponding adjustment devices.

[0072] In some embodiments, the reward also includes any of the following:

[0073] If the mixing ratio deviates from the target ratio, a negative reward will be given;

[0074] If the material tank is tilted or has abnormal weight, a negative reward will be given;

[0075] If the task is completed within the specified time, a positive reward will be given;

[0076] If the equipment is damaged or the downtime is short after the task is completed, a positive reward will be given;

[0077] If the equipment does not malfunction after the task is completed, a positive reward will be given.

[0078] The above-mentioned positive and negative rewards both belong to Figure 2 The rewards shown are illustrated. It can be understood that positive and negative rewards are corresponding, and the corresponding adjustments can also be in opposite directions.

[0079] In step S120, relevant datasets are collected to train the constructed multi-agent reinforcement learning model, thereby obtaining the trained model.

[0080] In some embodiments, the steps of constructing a multi-agent reinforcement learning model include:

[0081] Step S210: Obtain n policies corresponding to n agents;

[0082] Step S220: Calculate the cumulative expected reward for each agent based on the n policies;

[0083] Step S230: Calculate the agent's policy gradient based on the accumulated expected reward. The policy gradient is related to the agent's action value function.

[0084] Step S240: Use a Transformer network to predict the action value function to obtain the prediction function; and

[0085] Step S250: Use the prediction function to obtain the gradient of the deterministic policy under the deterministic policy.

[0086] In step S210, the strategy is a function approximator with adjustable parameters, such as a deep neural network. Using θ = [θ1, θ2, ..., θ...] n ] represents the policy parameters of n agents, π = [π1, π2, ..., π] n Let represent the policy of n agents. During model learning, the policy parameters are continuously updated based on actions, states, and rewards to find the optimal policy that maximizes the expected cumulative long-term reward received during the task. Each agent's policy is a probability distribution over its action space.

[0087] In step S220, the cumulative expected reward J(θ) of the i-th agent can be calculated using the following formula. i ):

[0088]

[0089] Among them, θ = [θ1, θ2,..., θ n ] represents the policy parameters of n agents, θ i Let π represent the policy parameters of the i-th agent, where π = [π1, π2, ..., π]. n ] represents the policy of n agents. Indicates a policy parameter θ i The policy executed by the i-th agent. This indicates that the i-th agent is in the policy The following action a i p π Let γ represent the probability that the agent is in state s under policy π, γ represent the discount factor, t represent the number of time steps the agent takes to execute an action, and a represent the probability that the agent is in state s. i r represents the action taken by the i-th agent. i，t This indicates that the i-th agent takes action a at time step t. i The instant reward received afterward.

[0090] In step S230, the inferred policy gradient J(θ) of the i-th agent can be calculated using the following formula. i ):

[0091]

[0092] wherein o i represents the observation of the i-th agent, x = [o1, o2,..., o n represents the observation vector, i.e., the state of the multi-agent reinforcement learning model, represents the action value function of the i-th agent, represents the optimization function of the policy gradient.

[0093] In step S240, the prediction function can be obtained by using the following formula:

[0094]

[0095] wherein Attention represents the Transformer attention model, and MLP represents the fully connected layer, represents the prediction function, wherein the Transformer attention model is used to process the state x into a fixed-length vector.

[0096] It should be noted that the present application uses the Transformer network structure instead of the traditional neural network structure, which can accelerate the model convergence and improve the prediction accuracy. The Transformer network usually includes an encoder and a decoder. Figure 3 is the Transformer network of an embodiment of the present application, wherein the two-way multi-layer perceptron and the multi-head attention module are respectively part of the structure of the encoder and the decoder. In the Transformer structure, the encoder and the decoder usually have the same structure, and the data processing process of encoding and decoding is realized by the multi-layer perceptron 310, 311 and the multi-head attention module 320, 321.

[0097] In some embodiments, the step of predicting the action value function by using the Transformer network comprises:

[0098] Step S241: inputting the state x into the Transformer network, and outputting a state encoding vector h x = Encoder(x) by the Transformer network, wherein Encoder represents the encoder in the Transformer structure; d can be a preset fixed value. According to this step, the length of the state encoding vector is d regardless of the length of the state x.

[0099] Step S242: representing the action a i of each agent as a one-hot vector b iand the one-hot encoded vector b i With the state encoding vector h x Connect them to form an input vector v i =[h x b i ], where b i =β1a1+…+β n a n , where β1+…+β n =1;

[0100] Step S243: Input vector v i The input is fed into the Transformer network to obtain the prediction function. (x, a1, ..., a) n The Transformer network includes a Transformer attention model and fully connected layers. (See reference...) Figure 3 The Transformer attention model includes two multi-head attention modules 320 and 321, and each multilayer perceptron 310 and 311 contains multiple fully connected layers.

[0101] exist Figure 3 In the illustrated embodiment, the Transformer network includes two multilayer perceptrons 310 and 311 and two multi-head attention modules 320 and 321. In other embodiments, it may include other numbers of multilayer perceptrons and multi-head attention modules.

[0102] refer to Figure 3 After state x is input into the Transformer network, it is preprocessed using the following formula:

[0103] Input new =TEM(Input) origin )+(PE×Output EPL (4)

[0104] Among them, Input origin Indicates state x, TEM represents word embedding module 331, PE represents position encoding 332, and Output... EPL This represents the output of the extrinsic parameter layer 333 (EPL), which is updated in real time during model training. Input new This represents the new state that enters the Transformer network after preprocessing. The positional encoding (332) encodes the position using the true values ​​of the action, reward, and state; it can be a token encoding from a language model. The extrinsic parameter layer is a self-trained parameter module used to correct the positional encoding.

[0105] In some embodiments, the multilayer perceptron and multi-head attention module process the new state Input using the following formula. new :

[0106] Inout MHAM =MLP1(Input) new )+MLP2(Input new (5)

[0107]

[0108] MLP is a multilayer perceptron containing multiple fully connected layers, where Inout MHAM This represents the input to the multi-head attention module. This represents the output of one of the multi-head attention modules. MHAM stands for Multi-head Attention Module, and RC stands for Residual Connection.

[0109] refer to Figure 3 As shown, the Transformer network also includes a regularization layer 340 (NL) and a feedforward network layer 350 (FFN). The output of the multi-head attention module further undergoes the processing steps represented by the following formula:

[0110]

[0111] Output = NL(RC(NL) output )+FFN(NL output (8)

[0112] in, represents the output of the nth multi-head attention module, and Output represents the output of the Transformer network.

[0113] refer to Figure 3 The Transformer network outputs a state encoding vector h after being executed N times. x =Encoder(x). In some embodiments, N = 6. The number of times N can be determined based on the combined evaluation results of rewards and actions.

[0114] In step S250, the gradient of the policy determination for the i-th agent is calculated using the following formula.

[0115]

[0116] Where D represents an experience store containing tuples (x, x′, a1, ..., a1). n r1, ..., r n), the tuple represents that the agent obtains rewards r1,..., r n when performing actions a1,..., a n in state x, and takes actions a1,..., a , which represents an optimization function for determining the policy gradient.

[0117] To obtain the optimal solution of the optimization function, the following loss function can be used as

[0118]

[0119]

[0120] wherein, is a target policy set with a delay parameter θ' i , r i represents the immediate reward obtained by the i-th agent after taking action a i , j represents the j-th agent, and j = i + 1.

[0121] According to the above-described steps, the policy gradient is determined, the policy of each agent is adjusted, and after training, the model parameters of the optimal multi-agent reinforcement learning model can be obtained. By constructing the above multi-agent reinforcement learning model, the raw material mixing control system can learn the appropriate working state through training, ensuring the normal operation of the device; at the same time, it also learns how to set appropriate discharge speeds for each tank, stirring rates of the mixing pool, and the like according to the current state under abnormal conditions, so as to ensure that the system can quickly recover to the normal operating state. After the raw material mixing control method starts to run, the control process can be completed autonomously, which can reduce or even eliminate the need for human intervention, greatly improving the system operation efficiency, as well as the stability and robustness of the mixing control system.

[0122] Figure 4 is a structural schematic diagram of a raw material mixing control system according to an embodiment of the present application. Referring to Figure 4 , the raw material mixing control system 400 includes n tanks and a system control end 410. The system control end 410 is in communication connection with each tank, and each tank is also connected to a raw material mixing pool 420. The raw material mixing control system shown in Figure 4 can be used to execute the raw material mixing control method described above. The system control end 410 can be specifically implemented as a computer program carrier, a server, a computer, or the like.

[0123] Specifically, in some embodiments, the raw material mixing control method further comprises: establishing a Time-Sensitive Network (TSN), taking each tank as a slave node in the TSN, setting a master node, and synchronizing the clock of each slave node in real time by the master node when the raw material mixing device is running. Reference Figure 4 After the TSN is established, the system control end 410 is taken as the TSN master node, and each tank is taken as a TSN slave node, as shown in Figure 4 The data flow between the TSN master node and the TSN slave node is bidirectional, ensuring the information interaction between the tanks and the system control end 410, and also allowing each tank to share its own information with other tanks.

[0124] The TSN network can provide a high-precision clock synchronization mechanism, so that the time of different nodes is synchronized to the nanosecond level, thereby enabling high-precision data synchronization between different nodes. When the entire system starts running, the TSN master node synchronizes the clock of each TSN slave node, realizing the real-time data transmission. At the same time, relying on the TSN, multiple data streams can be transmitted simultaneously, and by controlling the flow, the real-time data can be transmitted preferentially, thereby ensuring the real-time performance. The TSN also supports priority setting for data streams, which can prioritize different data streams and ensure timely transmission of high-priority data streams, further improving the real-time performance of the system. Moreover, the TSN guarantees the network delay, ensuring the timeliness of real-time data, thereby making the system more reliable; through data redundancy and retransmission mechanism, it can ensure the reliable transmission of data and improve the stability of the system. The TSN ensures the real-time performance and reliability of data interaction in the raw material mixing control system 400 of the present application, and the raw material mixing control method of the present application combines the TSN communication network architecture to ensure the rate synchronization and control synchronization of each tank, mixing pool, and system control end, realizes distributed system control, collaborative decision-making, and resource sharing, improves the adaptability of raw material mixing control, and does not require human intervention.

[0125] The present application also includes a raw material mixing control system comprising a memory and a processor. The memory is used to store instructions executable by the processor; the processor is used to execute the instructions to realize the raw material mixing control method described above.

[0126] Figure 5 is a system block diagram of the raw material mixing control system of an embodiment of the present application. Reference Figure 5As shown, the raw material mixing control system 500 can include an internal communication bus 501, a processor 502, a read only memory (ROM) 503, a random access memory (RAM) 504, and a communication port 505. When applied on a personal computer, the raw material mixing control system 500 can also include a hard disk 506. The internal communication bus 501 can enable data communication among the components of the raw material mixing control system 500. The processor 502 can make decisions and issue prompts. In some embodiments, the processor 502 can be composed of one or more processors. The communication port 505 can enable data communication between the raw material mixing control system 500 and external devices. In some embodiments, the raw material mixing control system 500 can send and receive information and data from a network through the communication port 505. The raw material mixing control system 500 can also include different forms of program storage units and data storage units, such as the hard disk 506, the read only memory (ROM) 503, and the random access memory (RAM) 504, which can store various data files used by the computer processing and / or communication, and possible program instructions executed by the processor 502. The processor executes these instructions to implement the main part of the method. The results of the processor processing are transmitted to the user equipment through the communication port, and displayed on the user interface.

[0127] The raw material mixing control method described above can be implemented as a computer program, stored in the hard disk 506, and loaded into the processor 502 for execution to implement the raw material mixing control method of the present application.

[0128] The present application also includes a computer readable medium storing computer program codes, which, when executed by a processor, implement the raw material mixing control method described above.

[0129] When the raw material mixing control method is implemented as a computer program, it can also be stored in a computer readable storage medium as an article of manufacture. For example, the computer readable storage medium can include, but is not limited to, magnetic storage devices (e.g., hard disk, floppy disk, magnetic strips), optical disks (e.g., compact disk (CD), digital versatile disk (DVD)), smart cards, and flash memory devices (e.g., electrically erasable programmable read only memory (EPROM), card, stick, key drive). In addition, the various storage media described herein can represent one or more devices and / or other machine-readable media for storing information. The term "machine-readable medium" can include, but is not limited to, wireless channels and various other media (and / or storage media) that are capable of storing, containing, and / or carrying code and / or instructions and / or data.

[0130] It should be understood that the above-described embodiments are only illustrative. The embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or any combination thereof. For a hardware implementation, the processing units can be implemented within one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, and / or other electronic units designed to perform the functions described herein and / or combinations thereof.

[0131] Some aspects of the application can be performed entirely in hardware, entirely in software (including firmware, resident software, micro-code, etc.), or in a combination of hardware and software. The above hardware or software can be referred to as a "block," "module," "engine," "unit," "component," or "system." The processor can be one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DAPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, or combinations thereof. Furthermore, aspects of the application can be manifested as a computer product in a computer readable medium including computer program code. For example, the computer readable medium can include, but is not limited to, magnetic storage devices (e.g., hard disk; floppy disk; magnetic strips), optical disks (e.g., compact disk (CD); digital versatile disk (DVD)), smart cards, and flash storage devices (e.g., card; stick; key drive).

[0132] The computer readable medium can include a propagated data signal with computer program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal can take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any combination thereof. Computer readable medium can be any medium that can be read by a computer, including, but not limited to, storage devices, memory devices, and communication devices. The computer program code can be transmitted as a carrier wave, for example, on a wireless signal, over an electrical cable, over the Internet, or over a combination of these and / or other mediums.

[0133] Having described above several aspects of the basic concept, it is clear that the above-described application is merely exemplary, and that many modifications, improvements and / or perturbations can be incorporated into the present application by one of ordinary skill in the art. Such modifications, improvements and perturbations are thus suggested by this application, and remain within the spirit and scope of the exemplary embodiments of the present application.

[0134] Also, certain terminology has been used in the description for the sake of clarity. For example, the terms "one embodiment," "an embodiment," and "some embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Consequently, the terms "an embodiment," "one embodiment," and "some embodiments" appear multiple times in the specification do not necessarily refer to the same embodiment, but can refer to different embodiments. Furthermore, the terms "a" or "an," as used herein, mean "one or more" when applied to any feature in the specification.

[0135] Some embodiments use numerical designations to describe components, quantities of attributes. It is to be understood that such numerical designations used in the description of embodiments can, in some examples, be modified by the adjectives "about," "approximately," or "substantially." Unless otherwise stated, "about," "approximately," or "substantially" indicate that the stated numerical value allows for a variation of ±20%. Accordingly, numerical values used in the specification and claims of some embodiments are approximations. The approximation is provided considering reasonable variation in individual measurements, as desired in the particular embodiment. In some embodiments, numerical values should be considered to be open ended in the absence of expressly stated constraints. Although the numerical ranges and parameters setting forth the broad scope of some embodiments are approximations, the numerical values set forth in the specific examples are reported as precisely as practicable.

Claims

1. A raw material mixing control method based on multi-agent reinforcement learning for controlling a raw material mixing device, the raw material mixing device comprising n tanks, n being a positive integer greater than or equal to 2, characterized in that, Comprise: each of the tank as an independent agent, the construction of multi-agent reinforcement learning model, the model parameters of the multi-agent reinforcement learning model include: state, action and reward, the state includes the actual weight of each tank, the action includes adjusting the feeding speed of a single tank, the reward includes: if the mixing ratio reaches the target ratio, give positive reward; training the multi-agent reinforcement learning model to obtain a trained model; obtain the current state of n tanks in the raw material mixing device, input the current state into the trained model, and the trained model outputs the target action corresponding to the current state of each tank; and control the action of each tank as the target action.

2. The raw material mixing control method according to claim 1, characterized by, The step of constructing a multi-agent reinforcement learning model comprises: obtaining n strategies corresponding to n agents; calculate the cumulative expected reward of each agent according to the n strategies; calculate the strategy gradient of the agent according to the cumulative expected reward, which is related to the action value function of the agent; use the Transformer network to predict the action value function to obtain a prediction function; and use the prediction function to obtain the deterministic strategy gradient under the deterministic strategy.

3. The raw material mixing control method according to claim 2, characterized by, The step of calculating the cumulative expected reward of each agent according to the n strategies comprises: The cumulative expected reward J(0i) of the ith agent is calculated using the following equation: i ): wherein θ = [θ1, θ2, …, θn] represents the strategy parameters of n agents, θi represents the strategy parameters of the i-th agent, π = [π1, π2, …, πn] represents the strategies of n agents, and πi represents the strategy of the i-th agent. n ] represents the strategy parameters of n agents, θ i represents the strategy parameters of the i-th agent, π = [π1, π2, …, π n represents the strategies of n agents, and πi represents the strategy of the i-th agent. represents the strategy executed by the i-th agent with strategy parameters θ i , represents the action a of the i-th agent under strategy πi, represents the action a i of the i-th agent under strategy πi, π represents the probability of the agent being in state s under strategy π, γ represents the discount factor, t represents the time step number of the agent executing the action, a i represents the action taken by the i-th agent, and r i,t represents the immediate reward obtained by the i-th agent after taking action a i at the t-th time step.

4. The raw material mixing control method according to Claim 3, characterized by, The step of calculating the policy gradient of the agent according to the cumulative expected reward comprises: calculating the policy gradient J(θ i ) of the i-th agent according to the following formula: wherein o i represents the observation of the i-th agent, x = [o1, o2, …, oi, …, on] represents an observation vector, i.e. a state of the multi-agent reinforcement learning model, n represents the observation of the i-th agent, x = [o1, o2, …, oi, …, on] represents an observation vector, i.e. a state of the multi-agent reinforcement learning model, represents the action value function of the i-th agent, represents the optimization function of the policy gradient.

5. The raw material mixing control method according to Claim 4, characterized by, use the following formula to obtain the prediction function: where Attention denotes a Transformer attention model and MLP denotes a fully connected layer, denotes the prediction function, where the Transformer attention model is used to process the state x into a fixed-length vector.

6. The raw material mixing control method according to Claim 5, characterized by, The step of using the prediction function to obtain the deterministic strategy gradient under the deterministic strategy comprises: The deterministic policy gradient for the i-th agent is calculated using the following equation where D denotes an empirical store containing tuples (x, x', a1,..., an, r1,..., rn) representing the rewards rn,..., r1obtained by the agent when performing actions a1,..., anin state xand the actions taken in the next state x', and n r1,..., r n n, respectively, and n n denotes an optimization function for determining the policy gradient.​​ 7. The raw material mixing control method according to Claim 6, characterized by, The loss function corresponding to the optimization function of the determined strategy gradient of the ith agent is wherein, is a target policy set with delay parameter θ' i i represents the immediate reward obtained by the i-th agent after taking action a i , j represents the j-th agent, j = i + 1.​ 8. The raw material mixing control method according to claim 5, characterized by, The step of using the Transformer network to predict the action value function comprises: inputting the state x into the Transformer network, the Transformer network outputting a state encoding vector h of length d x = Encoder(x), where Encoder represents an encoder in the Transformer structure; The actions of each agent a i Represented as a one-hot encoded vector b i and the one-hot encoded vector b i With the state encoding vector h x Connect them to form an input vector v i =[h x ,b i ], where b i =β1a1+…+β n a n , where β1+…+β n =1; inputting the input vector v i into the Transformer network for obtaining the prediction function wherein the Transformer network comprises the Transformer attention model and the fully connected layer.

9. The raw material mixing control method according to Claim 8, characterized by, After the state x is input into the Transformer network, it is preprocessed by the following formula: Input new = TEM(Input origin )+(PE×Output EPL ) wherein, Input origin denotes the state x, TEM denotes a word embedding module, PE denotes a position encoding, Output EPL denotes the output of an external parameter layer EPL, which is updated in real time during model training, Input new denotes the new state after pre-processing and entering the Transformer network.

10. The raw material mixing control method according to Claim 9, characterized by, The Transformer network comprises at least two multi-layer perceptrons and at least two multi-head attention modules.

11. The raw material mixing control method according to Claim 10, characterized by, The multi-layer perceptron and the multi-head attention module process the new state Input using the following formula new : Inout MHAM = MLP1 (Input new ) + MLP2 (Input new ) where MLP is a multi-layer perceptron containing multiple fully connected layers, Inout MHAM denotes an input of the multi-head attention module, denotes an output of the multi-head attention module, MHAM denotes a multi-head attention module, RC denotes a residual connection.

12. The raw material mixing control method according to Claim 11, characterized by, The Transformer network further comprises a regularization layer NL and a feedforward network layer FFN, and the output of the multi-head attention module is further subjected to the processing steps represented by the following formula: Output = NL(RC(NL output )+ FFN(NL output )) wherein, denotes the output of the nth multi-head attention module, and Output denotes the output of the Transformer network.

13. The raw material mixing control method according to claim 12, wherein The Transformer network outputs the state encoding vector h after performing multiple times x = Encoder(x).

14. The raw material mixing control method according to Claim 1, wherein The state further comprises any of the following: the feeding speed of each tank, the inclination state of each tank, the temperature and humidity of each tank, the temperature of the mixing pool, the mixture ratio in the mixing pool, the distance between the tank and the mixing pool.

15. The raw material mixing control method according to Claim 1, wherein The action further comprises any of the following: adjusting the stirring speed of the mixing pool, adjusting the posture of the tank, adjusting the position of the tank discharge port, adjusting the distance between the tank and the mixing pool.

16. The raw material mixing control method according to Claim 1, wherein The reward further comprises any of the following: if the mixing ratio deviates from the target ratio, give negative reward; if the tank has problems such as inclination or abnormal weight, give negative reward; if the task is completed within a specified time, give positive reward; if the damage or downtime of the equipment is short after completing the task, give positive reward; if the equipment does not malfunction after completing the task, give positive reward.

17. The raw material mixing control method according to Claim 1, wherein Further comprising: establishing a time-sensitive network, taking each of the tanks as a slave node in the time-sensitive network, setting a master node, and synchronizing the clock of each of the slave nodes in real time when the raw material mixing device is running.

18. A raw material mixing control system based on multi-agent reinforcement learning for controlling a raw material mixing device, the raw material mixing device comprising n tanks, n being a positive integer greater than or equal to 2, characterized in that, Comprise: a memory for storing instructions executable by a controller; a controller for executing the instructions to implement the method of any of claims 1-17.

19. A computer readable medium having stored computer program code which, when executed by a processor, implements the method of any of claims 1-17.

Images