Raw material mixing control method and control system based on multi-agent reinforcement learning

Abstract

The invention provides a raw material mixing control method and system based on multi-agent reinforcement learning and a computer readable medium. The raw material mixing control method comprises the steps that each material tank serves as an independent agent, a multi-agent reinforcement learning model is constructed, model parameters of the multi-agent reinforcement learning model comprise states, actions and rewards, the states comprise the actual weight of each material tank, the actions comprise the adjustment of the feeding speed of the single material tank, and the rewards comprise the actual weight of each material tank. The reward comprises: if the mixing proportion reaches a target proportion, giving a positive reward; training the multi-agent reinforcement learning model to obtain a trained model; the current states of n charging buckets in the raw material mixing device are obtained, the current states are input into the trained model, and the trained model outputs target actions corresponding to the current states of the charging buckets; and controlling the action of each material tank to be the target action.

Description

Technical Field

[0001] The present application mainly relates to the field of intelligent control, and in particular to a raw material mixing control method, a control system and a computer-readable medium storing computer program code based on multi-agent reinforcement learning. Background Art

[0002] Large-scale industrial equipment often requires complex and precise control. For example, raw material mixing control typically involves installing load cells under multiple tanks to obtain weight information for each tank. Weight changes and material level information are then used to control the discharge rate of each tank. Each load cell also transmits other characteristic sensor data, such as voltage, current, temperature, and load, to the control terminal, which displays this data to the user, facilitating manual intervention if problems arise. However, current control methods present the following issues:

[0003] (1) It is impossible to accurately control the discharge speed of each tank only by using information such as weight and material level;

[0004] (2) The stirring rate of the raw material mixing tank is usually constant, and there is currently a lack of intelligent means to automatically control the stirring rate;

[0005] (3) Once a tank fails, the entire system will be shut down, requiring manual troubleshooting and resolution, impacting work efficiency.

[0006] (4) It is necessary to manually set the control parameters, which is easily affected by factors such as noise and interference, and the control accuracy is not high.

[0007] Therefore, it is expected to adopt a more intelligent control method to overcome the above problems. Summary of the Invention

[0008] The technical problem to be solved by the present application is to provide a raw material mixing control method, a control system and a computer-readable medium storing computer program code based on multi-agent reinforcement learning, which can realize intelligent control of raw material mixing.

[0009] To solve the above technical problems, the present application provides a raw material mixing control method based on multi-agent reinforcement learning, which is used to control a raw material mixing device, wherein the raw material mixing device includes n material tanks, where n is a positive integer greater than or equal to 2, and includes: treating each of the material tanks as an independent agent, constructing a multi-agent reinforcement learning model, and the model parameters of the multi-agent reinforcement learning model include: state, action and reward, the state includes the actual weight of each material tank, the action includes adjusting the feeding speed of a single material tank, and the reward includes: giving a positive reward if the mixing ratio reaches the target ratio; training the multi-agent reinforcement learning model to obtain a trained model; obtaining the current state of the n material tanks in the raw material mixing device, inputting the current state into the trained model, and the trained model outputting the target action corresponding to the current state of each material tank; and controlling the action of each material tank to be the target action.

[0010] In one embodiment of the present application, the steps of constructing a multi-agent reinforcement learning model include: obtaining n strategies corresponding to n said agents; calculating the cumulative expected reward of each said agent based on the n strategies; inferring the policy gradient of the agent based on the cumulative expected reward, wherein the policy gradient is related to the action-value function of the agent; using a Transformer network to predict the action-value function to obtain a prediction function; and using the prediction function to obtain a deterministic policy gradient under a deterministic strategy.

[0011] In one embodiment of the present application, the step of calculating the cumulative expected reward of each agent according to the n strategies includes:

[0012] The following formula is used to calculate the cumulative expected reward J(θ i ):

[0013]

[0014] Where θ=[θ1,θ2,…,θ n ] represents the strategy parameters of n agents, θ i represents the strategy parameters of the i-th agent, π=[π1,π2,…,π n ] represents the strategy of n agents, Represents a policy parameter θ i The strategy executed by the i-th agent, Indicates the strategy of the i-th agent The next action a i , p π represents the probability that the agent is in state s under the strategy π, γ represents the reduction factor, t represents the number of time steps for the agent to perform the action, and a irepresents the action taken by the i-th agent, r i,t Indicates that the i-th agent takes action a at the t-th time step i Instant rewards after.

[0015] In one embodiment of the present application, the step of inferring the policy gradient of the agent according to the cumulative expected reward includes: using the following formula to calculate the inferred policy gradient J(θ i ):

[0016]

[0017] Among them, i represents the observation of the ith agent, x=[o1,o2,…,o n ] represents the observation vector, i.e., the state of the multi-agent reinforcement learning model, represents the action-value function of the ith agent, represents the optimization function of the policy gradient.

[0018] In one embodiment of the present application, the following formula is used to obtain the prediction function:

[0019]

[0020] Among them, Attention represents the Transformer attention model, MLP represents the fully connected layer, Represents the prediction function, wherein the Transformer attention model is used to process the state x into a fixed-length vector.

[0021] In one embodiment of the present application, the step of using the prediction function to obtain a deterministic policy gradient under a deterministic policy includes:

[0022] The following formula is used to calculate the deterministic policy gradient of the i-th agent:

[0023]

[0024] Where D represents an experience storage containing tuples (x, x′, a1,…, a n ,r1,…,r n ), the tuple represents the agent performing actions a1,…,a in state x n The rewards r1,…,r n and the action to be taken at the next state x′, Represents the optimization function for determining the policy gradient.

[0025] In one embodiment of the present application, the loss function corresponding to the optimization function for determining the policy gradient of the i-th agent is

[0026]

[0027]

[0028] in, is a function with a delay parameter θ′ i The target strategy set, r i Indicates that the i-th agent is taking action a i The instant reward obtained after , j represents the jth agent, j = i + 1.

[0029] In one embodiment of the present application, the step of using a Transformer network to predict the action value function includes: inputting the state x into the Transformer network, and the Transformer network outputting a state encoding vector h of length d x =Encoder(x), where Encoder represents the encoder in the Transformer structure; the action a of each agent is converted into i Represented as a one-hot encoded vector b i , and the one-hot encoding vector b i With the state encoding vector h x Connect them to form an input vector v i =[h x , b i |, where b i =β1a1+…+β n a n , where β1+…+β n =1; the input vector v i Input into the Transformer network to obtain the prediction function (x,a1,...,a n ), wherein the Transformer network includes the Transformer attention model and the fully connected layer.

[0030] In one embodiment of the present application, after the state x is input into the Transformer network, it is preprocessed using the following formula:

[0031] Input new =TEM(Input origin )+(PE×Output EPL )

[0032] Among them, Input origin Represents the state x, TEM represents the word embedding module, PE represents the position encoding, Output EPL Represents the output of the external parameter layer EPL, which is updated in real time during model training. new Indicates the new state of the Transformer network after preprocessing.

[0033] In one embodiment of the present application, the Transformer network includes at least 2 multi-layer perceptrons and at least 2 multi-head attention modules.

[0034] In one embodiment of the present application, the multi-layer perceptron and the multi-head attention module use the following formula to process the new state Input new :

[0035] Inout MHAM =MLP1(Input new )+MLP2(Input new )

[0036]

[0037] Among them, MLP is a multi-layer perceptron containing multiple fully connected layers. MHAM represents the input of the multi-head attention module, Represents the output of the multi-head attention module, MHAM represents the multi-head attention module, and RC represents the residual connection.

[0038] In one embodiment of the present application, the Transformer network further includes a regularization layer NL and a feed-forward network layer FFN, and the output of the multi-head attention module further undergoes the processing steps represented by the following formula:

[0039]

[0040]

[0041] in, Represents the output of the nth multi-head attention module, and Output represents the output of the Transformer network.

[0042] In one embodiment of the present application, the Transformer network outputs the state encoding vector h after executing multiple times. x =Encoder(x).

[0043] In one embodiment of the present application, the state also includes any of the following items: the feeding speed of each material tank, the tilt state of each material tank, the temperature and humidity of each material tank, the temperature of the mixing tank, the mixture ratio in the mixing tank, and the distance between the material tank and the mixing tank.

[0044] In one embodiment of the present application, the action further includes any of the following: adjusting the stirring speed of the mixing tank, adjusting the posture of the material tank, adjusting the position of the material tank outlet, and adjusting the distance between the material tank and the mixing tank.

[0045] In one embodiment of the present application, the reward also includes any of the following items: a negative reward is given if the mixing ratio deviates from the target ratio; a negative reward is given if the material tank has problems such as tilt or abnormal weight; a positive reward is given if the task is completed within the specified time; a positive reward is given if the damage or downtime of the equipment is short after the task is completed; a positive reward is given if the equipment does not malfunction after the task is completed.

[0046] In one embodiment of the present application, it also includes: establishing a time-sensitive network, using each of the material tanks as a slave node in the time-sensitive network, setting a master node, and when the raw material mixing device is running, the master node synchronizes the clock of each of the slave nodes in real time.

[0047] In order to solve the above technical problems, the present application also proposes a raw material mixing control system based on multi-agent reinforcement learning, which is used to control a raw material mixing device, wherein the raw material mixing device includes n material tanks, where n is a positive integer greater than or equal to 2, and is characterized in that it includes: a memory for storing instructions executable by a controller; and a controller for executing the instructions to implement the method described above.

[0048] In order to solve the above technical problems, the present application further proposes a computer-readable medium storing computer program codes, which implement the above method when executed by a processor.

[0049] The raw material mixing control method of the present application uses a multi-agent reinforcement learning model to highly intelligently control the action of each material tank. It has high precision, easy operation, and no human intervention is required, and can achieve intelligent control of the raw material mixing device. By adopting the Transformer structure instead of the traditional neural network structure, the model convergence can be accelerated and the prediction accuracy can be improved. In addition, the combination of time-sensitive networks further improves the real-time performance of the system, and can ensure the reliable transmission of data and improve the stability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings are included to provide a further understanding of the present application. They are incorporated into and constitute a part of this application. The accompanying drawings illustrate embodiments of the present application and, together with this specification, serve to explain the principles of the present application. In the accompanying drawings:

[0051] Figure 1 is an exemplary flow chart of a raw material mixing control method according to an embodiment of the present application;

[0052] Figure 2 This is an exemplary structural diagram of a multi-agent reinforcement learning model;

[0053] Figure 3 This is the Transformer network structure of an embodiment of the present application;

[0054] Figure 4 This is a structural diagram of a raw material mixing control system according to an embodiment of the present application;

[0055] Figure 5 This is a system block diagram of a raw material mixing control system according to an embodiment of the present application. DETAILED DESCRIPTION

[0056] To more clearly illustrate the technical solutions of the embodiments of this application, the following is a brief introduction to the drawings required for describing the embodiments. Obviously, the drawings described below are merely examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without inventive effort. Unless otherwise apparent from the context or otherwise noted, the same reference numerals in the figures represent the same structure or operation.

[0057] As used in this application and the claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not intended to refer to the singular but may include the plural. Generally speaking, the terms "comprises" and "include" only indicate the inclusion of the steps and elements specifically identified, and these steps and elements do not constitute an exclusive list. A method or apparatus may also include other steps or elements.

[0058] Unless otherwise specifically stated, the relative arrangement of the parts and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present application. At the same time, it should be understood that, for ease of description, the sizes of the various parts shown in the drawings are not drawn according to actual proportional relationships. The techniques, methods and equipment known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, the techniques, methods and equipment should be considered as part of the authorization specification. In all examples shown and discussed here, any specific values ​​should be interpreted as being merely exemplary and not as limitations. Therefore, other examples of the exemplary embodiments may have different values. It should be noted that similar numbers and letters represent similar items in the following figures, and therefore, once an item is defined in one figure, it does not need to be further discussed in subsequent figures.

[0059] Furthermore, it should be noted that the use of terms such as "first" and "second" to define components is solely for the purpose of distinguishing the corresponding components. Unless otherwise stated, these terms have no special meaning and therefore should not be construed as limiting the scope of protection of this application. Furthermore, while the terms used in this application are selected from commonly known and commonly used terms, some terms mentioned in this specification may have been selected by the applicant at his or her discretion, and their detailed meanings are explained in the relevant sections of this description. Furthermore, this application should be understood not only by the actual terms used, but also by the meaning implied by each term.

[0060] Flowcharts are used in this application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the various steps may be processed in reverse order or simultaneously. Furthermore, other operations may be added to these processes, or one or more operations may be removed from these processes.

[0061] The raw material mixing control method of the present application is used to control a raw material mixing device, which includes n material tanks, where n is a positive integer greater than or equal to 2. The function of the raw material mixing device is to input the materials from the n material tanks into a raw material mixing pool respectively to form the required mixed raw materials. Each material tank has a corresponding sensor group, which may include a weighing sensor, a flow meter, a temperature and humidity sensor, a distance sensor, etc., for obtaining the weight, feeding speed, tilt state, temperature and humidity, mixture ratio, distance between the material tank and the mixing pool, etc. of the material tank in real time or regularly. Therefore, it can be considered that the parameter information needed in the raw material control method of the present application is all available. The present application does not limit the specific raw materials.

[0062] Figure 1This is an exemplary flow chart of a raw material mixing control method according to an embodiment of the present application. Figure 1 As shown, the raw material mixing control method of this embodiment includes the following steps:

[0063] Step S110: Treat each tank as an independent agent and construct a multi-agent reinforcement learning model. The model parameters of the multi-agent reinforcement learning model include: state, action, and reward. The state includes the actual weight of each tank, the action includes adjusting the feeding speed of a single tank, and the reward includes a positive reward if the mixing ratio reaches the target ratio.

[0064] Step S120: training a multi-agent reinforcement learning model to obtain a trained model;

[0065] Step S130: obtaining the current states of n material tanks in the raw material mixing device, inputting the current states into the trained model, and the trained model outputting the target action corresponding to the current state of each material tank; and

[0066] Step S140: Control the action of each material tank to be the target action.

[0067] According to the raw material mixing control method of the present application, the multi-agent reinforcement learning model can be used to highly intelligently control the movement of each material tank, with high precision and convenient operation, without the need for human intervention, and can achieve intelligent control of the raw material mixing device. The following is a detailed description of the above steps S110 to S140.

[0068] Figure 2 This is an exemplary structural diagram of a multi-agent reinforcement learning model. Figure 2 As shown, a multi-agent reinforcement learning model includes multiple agents, for example, agents 1 to n, corresponding to n tanks, and each tank is regarded as an independent agent.

[0069] In step S110, the structure of the constructed multi-agent reinforcement learning model is as follows: Figure 2 As shown in Figure 1. Each agent has corresponding observations, actions, strategies, and rewards. In a Markov process, the learning and decision-making entity is called the agent, and the part that interacts with the agent is called the environment. The agent and the environment continuously interact, and this interaction process can be viewed as multiple moments. At each moment, the agent selects an action based on the state of the environment and a certain strategy. The environment transitions to the next state based on a certain state transition probability, and at the same time, a reward is given to the agent based on the quality of the current state. The agent can adjust its strategy based on the feedback from the environment and continue learning in the environment, ultimately learning an optimal strategy that maximizes or optimizes the reward. The complete Markov decision process can be represented by a sequence consisting of the state, action, and reward at each moment. Figure 2The observation shown in is the observed state of the environment. In this application, the state at least includes the actual weight of each tank, the action at least includes adjusting the feeding rate of a single tank, and the reward at least includes: if the mixing ratio reaches the target ratio, then a positive reward is given.

[0070] In some embodiments, the status further includes any of the following: the feed rate of each material tank, the tilt state of each material tank, the temperature and humidity of each material tank, the temperature of the mixing tank, the ratio of the mixture in the mixing tank, and the distance between the material tanks and the mixing tank. It should be noted that to obtain these statuses, the raw material mixing device should also be equipped with corresponding signal sensing devices, such as weighing sensors, flow meters, accelerometers, thermometers and hygrometers, and distance sensors.

[0071] In some embodiments, the actions further include any of the following: adjusting the stirring speed of the mixing tank, adjusting the posture of the material tank, adjusting the position of the material tank outlet, and adjusting the distance between the material tank and the mixing tank. Similarly, in order to perform these actions, the raw material mixing device should be equipped with corresponding adjustment devices.

[0072] In some embodiments, the reward further includes any of the following:

[0073] If the mixing ratio deviates from the target ratio, a negative reward is given;

[0074] If the tank has problems such as tilt or abnormal weight, a negative reward will be given;

[0075] If the task is completed within the specified time, a positive reward will be given;

[0076] If the equipment damage or downtime is short after completing the task, a positive reward will be given;

[0077] If the device does not malfunction after the task is completed, a positive reward is given.

[0078] The above positive rewards and negative rewards are all Figure 2 As shown in , it can be understood that positive rewards and negative rewards are corresponding, and the corresponding adjustment directions can also be opposite.

[0079] In step S120, relevant data sets are collected to train the constructed multi-agent reinforcement learning model to obtain a trained model.

[0080] In some embodiments, the steps of constructing a multi-agent reinforcement learning model include:

[0081] Step S210: Obtain n strategies corresponding to n agents;

[0082] Step S220: Calculate the cumulative expected reward of each agent based on n strategies;

[0083] Step S230: Calculate the agent's policy gradient based on the accumulated expected reward. The policy gradient is related to the agent's action-value function.

[0084] Step S240: using the Transformer network to predict the action-value function to obtain a prediction function; and

[0085] Step S250: using the prediction function to obtain the deterministic policy gradient under the deterministic policy.

[0086] In step S210, the strategy is a function approximator with adjustable parameters, such as a deep neural network. n ] represents the strategy parameters of n agents, π=[π1,π2,…,π n ] represents the policies of n agents. During model learning, the policy parameters are continuously updated based on the actions, states, and rewards to find the optimal policy that maximizes the expected cumulative long-term reward received during the task. Each agent's policy is a probability distribution over its action space.

[0087] In step S220, the following formula can be used to calculate the cumulative expected reward J(θ i ):

[0088]

[0089] Among them, θ = [θ1, θ2,..., θ n ] represents the strategy parameters of n agents, θ i represents the strategy parameters of the ith agent, π=[π1,π2,…,π n ] represents the strategy of n agents, Represents a policy parameter θ i The strategy executed by the i-th agent, Indicates the strategy of the i-th agent The next action a i , p π represents the probability that the agent is in state s under the strategy π, γ represents the reduction factor, t represents the number of time steps for the agent to perform the action, and a i represents the action taken by the i-th agent, r i，t Indicates that the i-th agent takes action a at the t-th time step i Instant rewards after.

[0090] In step S230, the following formula can be used to calculate the inferred policy gradient J(θ i ):

[0091]

[0092] Among them, i represents the observation of the ith agent, x = [o1, o2, ..., o n ] represents the observation vector, i.e., the state of the multi-agent reinforcement learning model, represents the action-value function of the ith agent, represents the optimization function of the policy gradient.

[0093] In step S240, the prediction function may be obtained using the following formula:

[0094]

[0095] Among them, Attention represents the Transformer attention model, MLP represents the fully connected layer, Represents the prediction function, wherein the Transformer attention model is used to process the state x into a fixed-length vector.

[0096] It should be noted that this application uses the Transformer network structure instead of the traditional neural network structure, which can accelerate model convergence and improve prediction accuracy. The Transformer network usually includes an encoder and a decoder. Figure 3 This is a Transformer network according to an embodiment of the present application, in which two multi-layer perceptrons and a multi-head attention module are respectively used as part of the encoder and decoder structures. In the Transformer structure, the encoder and decoder generally have the same structure, and the encoding and decoding data processing processes are jointly implemented by the multi-layer perceptrons 310 and 311 and the multi-head attention modules 320 and 321.

[0097] In some embodiments, the step of using a Transformer network to predict an action-value function includes:

[0098] Step S241: Input the state x into the Transformer network, and the Transformer network outputs a state encoding vector h of length d x =Encoder(x), where Encoder represents the encoder in the Transformer structure; d can be a preset fixed value. According to this step, no matter what the length of the state x is, the length of the state encoding vector is d.

[0099] Step S242: The action a of each agent i Represented as a one-hot encoding vector b i, and the one-hot encoding vector b i With the state encoding vector h x Connect them to form an input vector v i =[h x , b i ], where b i =β1a1+…+β n a n , where β1+…+β n =1;

[0100] Step S243: Input vector v i Input into the Transformer network to obtain the prediction function (x,a1,...,a n ), where the Transformer network includes the Transformer attention model and the fully connected layer. Figure 3 , the Transformer attention model includes two multi-head attention modules 320 and 321, and each multi-layer perceptron 310 and 311 contains multiple fully connected layers.

[0101] exist Figure 3 In the illustrated embodiment, the Transformer network includes two multi-layer perceptrons 310 and 311 and two multi-head attention modules 320 and 321. In other embodiments, other numbers of multi-layer perceptrons and multi-head attention modules may also be included.

[0102] refer to Figure 3 , after the state x is input into the Transformer network, it is preprocessed using the following formula:

[0103] Input new =TEM(Input origin )+(PE×Output EPL ) (4)

[0104] Among them, Input origin represents state x, TEM represents word embedding module 331, PE represents position encoding 332, Output EPL Represents the output of the external parameter layer 333 (EPL). The external parameter layer 333 is updated in real time during model training. new Represents the new state of the Transformer network after preprocessing. Positional encoding 332 is performed using the actual values ​​of the action, reward, and state, and can be a token encoding in a language model. The external parameter layer is a self-trained parameter module used to calibrate the positional encoding.

[0105] In some embodiments, the multi-layer perceptron and multi-head attention module use the following formula to process the new state Input new :

[0106] Inout MHAM =MLP1(Input new )+MLP2(Input new ) (5)

[0107]

[0108] Among them, MLP is a multi-layer perceptron containing multiple fully connected layers. MHAM represents the input of the multi-head attention module, represents the output of one of the multi-head attention modules, MHAM represents the multi-head attention module, and RC represents the residual connection.

[0109] refer to Figure 3 As shown, the Transformer network also includes a regularization layer 340 (NL) and a feed-forward network layer 350 (FFN). The output of the multi-head attention module also undergoes the processing steps represented by the following formula:

[0110]

[0111] Output=NL(RC(NL output )+FFN(NL output )) (8)

[0112] in, Represents the output of the n-th multi-head attention module, and Output represents the output of the Transformer network.

[0113] refer to Figure 3 , the Transformer network outputs the state encoding vector h after executing N times x = Encoder(x). In some embodiments, N = 6. The number N can be determined based on the comprehensive evaluation results of the reward and the action.

[0114] In step S250, the following formula is used to calculate the determination policy gradient of the i-th agent:

[0115]

[0116] Where D represents an experience storage containing tuples (x, x′, a1, ..., a n , r1,...,r n), the tuple indicates that the agent performs actions a1,...,a in state x n The rewards r1,...,r n and the action to be taken at the next state x′, represents the optimization function that determines the policy gradient.

[0117] In order to obtain the optimal solution of the optimization function, the following loss function can be used:

[0118]

[0119]

[0120] in, is a function with a delay parameter θ′ i The target strategy set, r i Indicates that the i-th agent is taking action a i The instant reward obtained after , j represents the jth agent, j = i + 1.

[0121] According to the steps described above, the strategy gradient can be determined, and the strategy of each agent can be adjusted. After training, the model parameters of the optimal multi-agent reinforcement learning model can be obtained. This application constructs the above-mentioned multi-agent reinforcement learning model, and through training, the raw material mixing control system can learn the appropriate working state to ensure the normal operation of the device; at the same time, it also learns how to set the appropriate discharge speed for each material tank, the stirring rate of the mixing tank, etc. according to the current state in an abnormal state, so as to ensure that the system can quickly return to normal operating state. After the raw material mixing control method starts running, the control process can be completed autonomously, which can reduce or even eliminate the need for human intervention, greatly improving the system operation efficiency, as well as the stability and robustness of the mixed control system.

[0122] Figure 4 This is a schematic diagram of the structure of a raw material mixing control system according to an embodiment of the present application. Figure 4 The raw material mixing control system 400 includes n material tanks and a system control terminal 410. The system control terminal 410 establishes a communication connection with each material tank, and each material tank is also connected to a raw material mixing pool 420. Figure 4 The raw material mixing control system shown in the figure is used to execute the raw material mixing control method described above. The system control terminal 410 can be specifically implemented as a computer program carrier, a server, a computer, etc.

[0123] Specifically, in some embodiments, the raw material mixing control method further includes: establishing a time-sensitive network (TSN), treating each material tank as a slave node in the TSN, setting a master node, and when the raw material mixing device is running, the master node synchronizes the clock of each slave node in real time. Figure 4 After TSN is established, the system control terminal 410 is used as the TSN master node, and each tank is used as a TSN slave node. Figure 4 As shown, the data flow between the TSN master node and the TSN slave node is bidirectional, ensuring information exchange between the material tank and the system control terminal 410, and also allowing each material tank to share its own information with other material tanks.

[0124] The TSN network provides a high-precision clock synchronization mechanism, synchronizing the time between different nodes to the nanosecond level. This enables high-precision data synchronization between nodes. When the entire system begins operation, the TSN master node synchronizes the clocks of each TSN slave node, ensuring real-time data transmission. Furthermore, TSN supports the simultaneous transmission of multiple data streams. By controlling the flow rate, real-time data can be prioritized, ensuring real-time performance. TSN also supports data stream prioritization, enabling the timely transmission of high-priority data streams, further improving the system's real-time performance. Furthermore, by ensuring network latency, TSN ensures the timeliness of real-time data, making the system more reliable. Data redundancy and retransmission mechanisms ensure reliable data transmission and improve system stability. TSN ensures the real-time and reliability of data interaction in the raw material mixing control system 400 of the present application. The raw material mixing control method of the present application is combined with the TSN communication network architecture to ensure the rate synchronization and control synchronization of each material tank, mixing pool, and system control end, realizes distributed system control, collaborative decision-making and resource sharing, and improves the adaptability of raw material mixing control without human interference.

[0125] The present application also includes a raw material mixing control system, comprising a memory and a processor, wherein the memory is used to store instructions executable by the processor; and the processor is used to execute the instructions to implement the raw material mixing control method described above.

[0126] Figure 5 This is a system block diagram of a raw material mixing control system according to an embodiment of the present application. Figure 5As shown, the raw material mixing control system 500 may include an internal communication bus 501, a processor 502, a read-only memory (ROM) 503, a random access memory (RAM) 504, and a communication port 505. When used on a personal computer, the raw material mixing control system 500 may also include a hard disk 506. The internal communication bus 501 enables data communication between the components of the raw material mixing control system 500. The processor 502 can make decisions and issue prompts. In some embodiments, the processor 502 may be composed of one or more processors. The communication port 505 enables data communication between the raw material mixing control system 500 and the outside world. In some embodiments, the raw material mixing control system 500 can send and receive information and data from a network via the communication port 505. The raw material mixing control system 500 may also include various forms of program storage units and data storage units, such as a hard disk 506, a read-only memory (ROM) 503, and a random access memory (RAM) 504, capable of storing various data files used for computer processing and / or communication, as well as possible program instructions executed by the processor 502. The processor executes these instructions to implement the main components of the method. The processing result of the processor is transmitted to the user device through the communication port and displayed on the user interface.

[0127] The above-mentioned raw material mixing control method can be implemented as a computer program, stored in the hard disk 506, and loaded into the processor 502 for execution to implement the raw material mixing control method of the present application.

[0128] The present application also includes a computer-readable medium storing computer program code, which, when executed by a processor, implements the raw material mixing control method described above.

[0129] When the raw material mixing control method is implemented as a computer program, it can also be stored in a computer-readable storage medium as an article of manufacture. For example, computer-readable storage media can include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips), optical disks (e.g., compact disks (CDs), digital versatile disks (DVDs)), smart cards, and flash memory devices (e.g., electrically erasable programmable read-only memories (EPROMs), cards, sticks, key drives). In addition, the various storage media described herein can represent one or more devices and / or other machine-readable media for storing information. The term "machine-readable medium" can include, but is not limited to, wireless channels and various other media (and / or storage media) that can store, contain, and / or carry code and / or instructions and / or data.

[0130] It should be understood that the embodiments described above are merely illustrative. The embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or any combination thereof. For hardware implementation, the processor may be implemented within one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, and / or other electronic units designed to perform the functions described herein, or a combination thereof.

[0131] Some aspects of the present application can be performed entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The above hardware or software can be referred to as "data blocks", "modules", "engines", "units", "components" or "systems". The processor can be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DAPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors or combinations thereof. In addition, various aspects of the present application may be expressed as computer products located in one or more computer-readable media, which include computer-readable program code. For example, computer-readable media may include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, tapes...), optical disks (e.g., compact disks CDs, digital versatile disks DVDs...), smart cards, and flash memory devices (e.g., cards, sticks, key drives...).

[0132] A computer-readable medium may include a propagated data signal embodying computer program code, for example, in baseband or as part of a carrier wave. The propagated signal may be in a variety of forms, including electromagnetic, optical, etc., or a suitable combination thereof. A computer-readable medium may be any computer-readable medium other than a computer-readable storage medium that can be connected to an instruction execution system, apparatus, or device to communicate, propagate, or transmit the program for use. The program code on the computer-readable medium may be transmitted via any suitable medium, including radio, cable, fiber optic cable, radio frequency signal, or similar medium, or any combination of the above.

[0133] The basic concepts have been described above. It will be apparent to those skilled in the art that the above disclosures are merely illustrative and do not constitute limitations on this application. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and amendments to this application. Such modifications, improvements, and amendments are suggested in this application and remain within the spirit and scope of the exemplary embodiments of this application.

[0134] At the same time, this application uses specific terms to describe the embodiments of this application. For example, "one embodiment," "an embodiment," and / or "some embodiments" refer to a certain feature, structure, or characteristic related to at least one embodiment of this application. Therefore, it should be emphasized and noted that "one embodiment," "an embodiment," or "an alternative embodiment" mentioned twice or multiple times in different locations in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures, or characteristics in one or more embodiments of this application may be appropriately combined.

[0135] In some embodiments, numbers are used to describe the quantity of components and attributes. It should be understood that such numbers used in the description of the embodiments are modified by the modifiers "about", "approximately" or "substantially" in some examples. Unless otherwise stated, "about", "approximately" or "substantially" indicate that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the description and claims are approximate values, which may change according to the required features of individual embodiments. In some embodiments, the numerical parameters should take into account the specified significant digits and adopt the general method of retaining digits. Although the numerical domains and parameters used to confirm the breadth of their range in some embodiments of the present application are approximate values, in specific embodiments, the settings of such numerical values ​​are as accurate as possible within the feasible range.

Claims

1. A raw material mixing control method based on multi-agent reinforcement learning, used to control a raw material mixing device, wherein the raw material mixing device comprises n material tanks, where n is a positive integer greater than or equal to 2, and is characterized in that: include: Each of the material tanks is taken as an independent intelligent agent to construct a multi-agent reinforcement learning model, wherein the model parameters of the multi-agent reinforcement learning model include: state, action and reward, wherein the state includes the actual weight of each material tank, the action includes adjusting the feeding speed of a single material tank, and the reward includes: if the mixing ratio reaches the target ratio, a positive reward is given; Training the multi-agent reinforcement learning model to obtain a trained model; Obtaining current states of n material tanks in the raw material mixing device, inputting the current states into the trained model, and the trained model outputting a target action corresponding to the current state of each material tank; and The action of controlling each of the material tanks is the target action.

2. The raw material mixing control method according to claim 1, characterized in that: The steps to build a multi-agent reinforcement learning model include: Obtaining n strategies corresponding to the n agents; Calculate the cumulative expected reward of each of the agents according to the n strategies; Inferring a policy gradient of the agent based on the accumulated expected reward, wherein the policy gradient is related to an action value function of the agent; Predicting the action-value function using a Transformer network to obtain a prediction function; and The prediction function is used to obtain a deterministic policy gradient under a deterministic policy.

3. The raw material mixing control method according to claim 2, characterized in that: The step of calculating the cumulative expected reward of each of the agents according to the n strategies comprises: The following formula is used to calculate the cumulative expected reward J(θ i ): Where θ=[θ1,θ2,…,θ n ] represents the policy parameters of n agents, θ i represents the policy parameters of the ith agent, π=[π1,π2,…,π n ] represents the strategy of n agents, Represents a policy parameter θ i The strategy executed by the ith agent, Indicates the strategy of the i-th agent The next action a i , p π represents the probability that the agent is in state s under strategy π, γ represents the reduction factor, t represents the number of time steps that the agent performs the action, and a i represents the action taken by the ith agent, r i,t Indicates that the i-th agent takes action a at the t-th time step i Instant rewards after.

4. The raw material mixing control method according to claim 3, characterized in that: The step of inferring the agent's policy gradient according to the accumulated expected reward includes: using the following formula to calculate the inferred agent's policy gradient J(θ i ): Among them, i represents the observation of the ith agent, x = [o1, o2, …, o n ] represents the observation vector, i.e., the state of the multi-agent reinforcement learning model, represents the action value function of the ith agent, Represents the optimization function of the policy gradient.

5. The raw material mixing control method according to claim 4, characterized in that: The prediction function is obtained using the following formula: Among them, Attention represents the Transformer attention model, MLP represents the fully connected layer, Represents the prediction function, wherein the Transformer attention model is used to process the state x into a vector of fixed length.

6. The raw material mixing control method according to claim 5, characterized in that: The steps of using the prediction function to obtain a deterministic policy gradient under a deterministic policy include: The following formula is used to calculate the deterministic policy gradient of the i-th agent: Where D represents an experience storage containing tuples (x, x′, a1,…, a n ,r1,…,r n ), the tuple represents the agent performing actions a1,…,a in state x n The rewards r1,…,r n and the action to take at the next state x′, Represents the optimization function for determining the policy gradient.

7. The raw material mixing control method according to claim 6, characterized in that: The loss function corresponding to the optimization function of the determined policy gradient of the i-th agent is in, is a function with a delay parameter θ′ i The target strategy set, r i Indicates that the i-th agent takes action a i The instant reward obtained after , j represents the jth agent, j = i + 1.

8. The raw material mixing control method according to claim 5, characterized in that: The steps of using the Transformer network to predict the action value function include: The state x is input into the Transformer network, and the Transformer network outputs a state encoding vector h of length d. x =Encoder(x), where Encoder represents the encoder in the Transformer structure; Each agent's action a i Represented as a one-hot encoded vector b i , and the one-hot encoding vector b i With the state encoding vector h x Connect them to form an input vector v i =[h x ,b i ], where b i =β1a1+…+β n a n , where β1+…+β n =1; The input vector v i Input into the Transformer network to obtain the prediction function The Transformer network includes the Transformer attention model and the fully connected layer.

9. The raw material mixing control method according to claim 8, characterized in that: After the state x is input into the Transformer network, it is preprocessed using the following formula: Input new =TEM(Input origin )+(PE×Output EPL ) Among them, Input origin represents the state x, TEM represents the word embedding module, PE represents the position encoding, Output EPL Represents the output of the external parameter layer EPL, which is updated in real time during model training. new Represents the new state of the Transformer network after preprocessing.

10. The raw material mixing control method according to claim 9, characterized in that: The Transformer network includes at least 2 multi-layer perceptrons and at least 2 multi-head attention modules.

11. The raw material mixing control method according to claim 10, characterized in that: The multi-layer perceptron and the multi-head attention module use the following formula to process the new state Input new : Inout MHAM =MLP1(Input new )+MLP2(Input new ) Among them, MLP is a multi-layer perceptron with multiple fully connected layers. MHAM represents the input of the multi-head attention module, Represents the output of the multi-head attention module, MHAM represents the multi-head attention module, and RC represents the residual connection.

12. The raw material mixing control method according to claim 11, characterized in that: The Transformer network also includes a regularization layer NL and a feed-forward network layer FFN. The output of the multi-head attention module also undergoes the processing steps represented by the following formula: Output=NL(RC(NL output )+FFN(NL output )) in, Represents the output of the nth multi-head attention module, and Output represents the output of the Transformer network.

13. The raw material mixing control method according to claim 12, characterized in that: The Transformer network is executed multiple times and then outputs the state encoding vector h x =Encoder(x).

14. The raw material mixing control method according to claim 1, characterized in that: The state also includes any of the following items: the feeding speed of each material tank, the tilting state of each material tank, the temperature and humidity of each material tank, the temperature of the mixing tank, the mixture ratio in the mixing tank, and the distance between the material tank and the mixing tank.

15. The raw material mixing control method according to claim 1, characterized in that: The actions also include any of the following: adjusting the stirring speed of the mixing tank, adjusting the posture of the material tank, adjusting the position of the material tank outlet, and adjusting the distance between the material tank and the mixing tank.

16. The raw material mixing control method according to claim 1, characterized in that: The rewards also include any of the following: If the mixture ratio deviates from the target ratio, a negative reward is given; If the tank has problems such as tilt or abnormal weight, a negative reward will be given; If the task is completed within the specified time, a positive reward is given; If the damage or downtime of the equipment is short after completing the task, a positive reward will be given; If the device does not malfunction after the task is completed, a positive reward is given.

17. The raw material mixing control method according to claim 1, characterized in that: It also includes: establishing a time-sensitive network, using each of the material tanks as a slave node in the time-sensitive network, setting a master node, and when the raw material mixing device is running, the master node synchronizes the clock of each of the slave nodes in real time.

18. A raw material mixing control system based on multi-agent reinforcement learning, used to control a raw material mixing device, wherein the raw material mixing device comprises n material tanks, where n is a positive integer greater than or equal to 2, characterized in that: include: a memory for storing instructions executable by the controller; A controller, configured to execute the instructions to implement the method according to any one of claims 1 to 17.

19. A computer readable medium storing computer program code, wherein the computer program code, when executed by a processor, implements the method according to any one of claims 1 to 17.

Images