Multi-agent synergistic compatibility method based on Copula thought

Abstract

The invention discloses a multi-agent collaborative compatibility method based on a Copula thought, which is a multi-agent collaborative method based on full-link data driving, integrating priori knowledge, expert knowledge and mechanism knowledge, integrating a Copula function and tracking a target value. The method comprises the following steps: S1, establishing a multi-agent collaborative system; s2, the pre-combustion mixing agent, the combustion agent and the component detection agent respectively reinforce learning action decisions based on the Markov process; s3, putting a plurality of intelligent agents together to learn a copula-based strategy; and S4, through information interaction between the pre-combustion mixing intelligent agent and the combustion intelligent agent and between the combustion intelligent agent and the component detection intelligent agent, enabling the pre-combustion mixing intelligent agent to understand the hysteresis quality, the combustion condition and the detection condition, and making a compatibility decision. According to the invention, compatibility decision is carried out on the basis of the relationship between the agents of Copula, the traditional manual compatibility debugging method based on experience is broken through, the compatibility is controlled by adopting full-link data driving, and the compatibility effect is finer and more obvious.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-agent collaborative matching, and in particular to a multi-agent collaborative matching method based on the Copula concept. Background Art

[0002] In the face of a long lag and highly complex process, raw material matching is generally based on experience. The batching method of the traditional sintering test (gasifier raw material matching) can directly reflect the quality of the batching, but this method is time-consuming and labor-intensive. Or simple theoretical methods, experimental methods, etc., due to the current domestic sintering plant (hazardous waste) raw materials are of many types and poor stability, making the traditional batching method based on experience extremely computational and labor-intensive, and has obvious uncertainty.

[0003] The accuracy of the raw material and fuel ratio determines the chemical composition and stability of the sintered ore (hazardous waste), which is a key factor influencing the economy and technical performance of the blast furnace.

[0004] In the actual production process, sometimes two groups of batching schemes with similar particle size, composition, process parameters, etc. will show huge differences in sintered ore quality indicators. This is because the impact of the high temperature and complex system of the sintering process on iron ore is ignored, and there is a lack of understanding of the characteristics of iron ore in high temperature systems.

[0005] The traditional batching method has the following disadvantages: 1) The traditional model method is established with the lowest batching cost and alkalinity within the appropriate range as the objective function, and multi-objective constraint optimization problem; generally linear programming is used to solve the optimization or heuristic algorithm (such as genetic algorithm, etc.), which has low computational efficiency. 2) The traditional data source may only come from the final detection part, or the upstream compatibility data; the data source is relatively single and cannot well describe the real process phenomenon. 3) Most of the information interaction between multiple agents does not consider the distribution of actions and states too much, and does not fully consider the dependencies between agents.

[0006] In order to solve the above technical problems, the present invention provides a multi-agent collaborative formulation method based on the Copula concept, which is more sophisticated and more effective than the traditional formulation method controlled by experts. Summary of the invention

[0007] The purpose of the invention is to provide a multi-agent collaborative matching method based on the Copula concept, breaking the traditional experience-based manual matching debugging method, using data-driven to control the matching, and the matching effect is more refined and obvious.

[0008] To solve the above technical problems, an embodiment of the present invention provides a multi-agent collaborative matching method based on the Copula idea, which is a multi-agent collaborative method based on full-link data drive, integrating prior knowledge, expert knowledge, mechanism knowledge, and Copula functions to track target values.

[0009] Among them, the multi-agent collaborative matching method based on Copula idea includes the following steps: S1. Establish a multi-agent collaborative system, including pre-combustion mixing agent, combustion agent and component detection agent. Multiple agents collaborate to use copula to model the nodes representing variables and the dependency structure between nodes. S2, pre-combustion mixing agent, combustion agent and component detection agent are based on Markov process reinforcement learning action decision-making; S3, put the pre-combustion mixing agent, combustion agent and component detection agent together to learn the copula-based strategy; S4. Through information interaction between the pre-combustion mixing agent and the combustion agent, and between the combustion agent and the component detection agent, the pre-combustion mixing agent is made to understand the hysteresis, combustion conditions, and detection conditions, and make a compatibility decision; In steps S2-S4, each agent incorporates expert knowledge, prior knowledge, and mechanism knowledge when making action decisions and rewards.

[0010] The specific steps of step S2 are: No. Agent actions ( ): , ,......, , ; N1 represents the total number of actions of the first agent, N2 represents the total number of actions of the second agent, ..., N1 and N2 are both subscripts of the total number; No. Agent state ( ): , ,......, , ; M1 represents the total number of states of the first agent, M2 represents the total number of states of the second agent, ..., M1, M2, and M3 are all subscripts of the total number; Each agent receives a reward from the environment as a function of its state and action : , and individual observations associated with the state (associated with the state) ,award ,in is the discount factor, is the total duration; , the distribution of state transitions among the three agents, where For the three agent states The probability distribution under is the state space; For time, ,action ,Agent Observation , the probability at this time is , Joint Strategy .

[0011] Among them, step S3 puts the pre-combustion mixing agent, the combustion agent and the component detection agent together to learn the copula-based strategy, and decomposes the parameter joint strategy into: , For the Agents in parameters The marginal strategy under is the corresponding cumulative distribution function, For the three Action transformation The copula density function is, It is obtained by transforming the probability of marginal actions.

[0012] Among them, step S4 includes four processes: S4.1. Learn the marginal action distribution of each agent’s current state, taking a state s as input and outputting the marginal action distribution parameters of the three agents given the input state: S4.1.1. Recording state-action pairs ; S4.1.2. Calculate each agent in this state Conditional marginal action distribution under ; S4.1.3. Calculate the maximum likelihood value for each action through SGD , obtain the marginal action distribution of each agent based on the existing data fitting calculation; S4.2. Learning copula functions S4.2.1, The cumulative probability distribution of ; S4.2.2、Move Normalized distribution value ;in, , ; S4.2.3. Calculation of Copula Density , calculate the maximum likelihood estimate through SGD ; S4.3. Determine the joint action of all agents based on the current state S4.3.1 Copula value ; S4.3.2. Use inverse probability transformation to the original action space ; S4.3.3. Obtaining the joint actions of all agents ;Here, expert knowledge is also combined with prior knowledge and mechanism knowledge to prevent actions from violating common sense; S4.4. Obtain the trajectory, i.e. the action-based Markov diagram of all states in the global (before combustion, combustion, detection) S4.4.1. Given the current state, repeatedly predict the agent's actions and generate new trajectories. The states are nodes and the actions are weights on the edges of the graph, which are equivalent to the direction of the action in the next state. The details are as follows: Enter the current state , through step S4.3, the joint action under the current state is obtained , combined with the current environment , and each agent The reward function is combined with the expert experience to set the weight wj, R = w_1×R1+w_2×R2+w_3×R3, and the state of each agent is obtained. ; S4.4.2. Execute the action generated in step S4.4.1 and obtain the updated state from the environment: , is a cycle length. As time goes by, each time node will have a state action, thus forming a large Markov graph based on state and action movement; S4.5. Repeat steps S4.3 and S4.4 until a qualified compatibility decision is obtained.

[0013] The present invention also provides a multi-agent collaborative matching system based on the Copula concept, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the computer executes the computer program instructions, the above-mentioned matching method can be implemented.

[0014] The present invention also provides a computer-readable storage medium, on which are stored computer program instructions that can be executed by a processor. When the computer executes the program instructions, the above-mentioned matching method can be implemented.

[0015] The beneficial effects of the above technical solution of the present invention are as follows: 1. The present invention proposes to make compatibility decisions based on the relationship between Copula agents, breaking the traditional experience-based manual compatibility debugging method and using full-link data drive to control the compatibility, making the compatibility effect more refined and obvious.

[0016] 2. The present invention is based on the integration of expert knowledge based on the Copula function, etc., and can capture the dependencies between random variables. The learned connection density can be easily visualized, and the correlation between the actions of the agents can be intuitively analyzed. When the marginal strategy of the agent changes, but the dependency between different agents remains unchanged, a new joint strategy can be effectively obtained by exchanging the margins of the new agent while keeping the copula and other margins unchanged. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a principle block diagram of the present invention; Figure 2 This is a flow chart of step S4 in the present invention. DETAILED DESCRIPTION

[0018] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0019] The embodiment of the present invention provides a multi-agent collaborative matching method based on the Copula concept, which is a multi-agent collaborative method based on full-link data drive, integrating prior knowledge, expert knowledge, mechanism knowledge, integrating Copula functions, and tracking target values. Figure 1 shown.

[0020] A multi-agent collaborative matching method based on Copula idea includes the following steps: S1. Establish a multi-agent collaborative system, including pre-combustion mixing agent, combustion agent and component detection agent. Multiple agents collaborate to use copula to model the nodes representing variables and the dependency structure between nodes. The joint probability and dependency structure of multiple agents can be effectively represented by copula. Copula is a convenient and effective tool for exploring the correlation between random variables, and can also be used to obtain the probability of encounter between two variables. Agents collaborate to use copula to model the nodes representing the variables and the dependency structure between nodes. When using multivariate methods, the copula function provides an advantage that it can select the one-dimensional marginal distribution of the variables and provides a high degree of flexibility in representing multivariate dependency structures. For a two-dimensional link function, according to Sklar's theorem, the unique link function Can give: ,in and is a continuous random variable. Copula is a powerful statistical tool to capture dependencies between random variables to explicitly model correlation and coordination in multi-agent systems. The proposed model is able to learn the margins that capture the local action pattern of each individual agent separately, as well as the copula function that captures the dependency structure between agents individually and completely.

[0021] S2, pre-combustion mixing agent, combustion agent and component detection agent are based on Markov process reinforcement learning action decision-making; The specific steps of step S2 are: No. Agent actions ( ): , ,......, , ; N1 represents the total number of actions of the first agent, N2 represents the total number of actions of the second agent, ..., N1 and N2 are both subscripts of the total number; No. Agent state ( ): , ,......, , ; M1 represents the total number of states of the first agent, M2 represents the total number of states of the second agent, ..., M1, M2, and M3 are all subscripts of the total number; Each agent receives a reward from the environment as a function of its state and action : , and individual observations associated with the state (associated with the state) ,award ,in is the discount factor, is the total duration; , the distribution of state transitions among the three agents, where For the three agent states The probability distribution under is the state space; For time, ,action ,Agent Observation , the probability at this time is , Joint Strategy .

[0022] S3, put the pre-combustion mixing agent, combustion agent and component detection agent together to learn the copula-based strategy; Step S3 combines the pre-combustion mixing agent, the combustion agent, and the component detection agent to learn the copula-based strategy, and decomposes the parameter joint strategy into: , For the Agents in parameters The marginal strategy under is the corresponding cumulative distribution function, For the three Action transformation The copula density function is, It can be obtained by transforming the probability of marginal actions.

[0023] S4. Through the information exchange between the pre-combustion mixing agent and the combustion agent, and between the combustion agent and the component detection agent, the pre-combustion mixing agent can understand the hysteresis, combustion status, and detection status, and make a matching decision. The flow chart is as follows Figure 2 shown.

[0024] Step S4 includes four processes: S4.1. Learn the marginal action distribution of each agent’s current state, taking a state s as input and outputting the marginal action distribution parameters of the three agents given the input state: S4.1.1. Recording state-action pairs ; S4.1.2. Calculate each agent in this state Conditional marginal action distribution under ; S4.1.3. Calculate the maximum likelihood value for each action through SGD , the marginal action distribution of each agent needs to be calculated based on the existing data; S4.2. Learning copula functions S4.2.1, The cumulative probability distribution of ; S4.2.2. Action Normalized distribution value ;in, , ; S4.2.3. Calculation of Copula Density , calculate the maximum likelihood estimate through SGD ; S4.3. Determine the joint action of all agents based on the current state S4.3.1 Copula value ; S4.3.2. Use inverse probability transformation to the original action space ; S4.3.3. Obtaining the joint actions of all agents ;This step also combines expert knowledge with prior knowledge and mechanism knowledge to prevent actions from violating common sense; S4.4. Obtaining trajectories can be understood as the Markov diagram based on actions for all states of the global (before combustion, combustion, detection) S4.4.1. Given the current state, repeatedly predict the agent's actions and generate new trajectories. The states are nodes and the actions are weights on the edges of the graph, which are equivalent to the direction of the action in the next state. The details are as follows: Enter the current state , through step S4.3, the joint action under the current state is obtained , combined with the current environment , and each agent The reward function is combined with the expert experience to set the weight wj, R = w_1×R1+w_2×R2+w_3×R3, and the state of each agent is obtained. ; S4.4.2. Execute the action generated in step S4.4.1 and obtain the updated state from the environment: , is a cycle length. As time goes by, each time node will have a state action, and a Markov diagram based on state and action movement is obtained; S4.5. Repeat steps S4.3 and S4.4 until a qualified compatibility decision is obtained. The conditions for determining a qualified compatibility decision include but are not limited to: a. Good combustion, generally referring to the temperature reaching a certain requirement; b. Good compatibility, from combustion to the final test, meeting the requirements, and stable indicators (such as indicators such as alkalinity in sintering).

[0025] In steps S2-S4, each agent incorporates expert knowledge, prior knowledge, and mechanism knowledge when making action decisions and rewards.

[0026] The present invention also provides a multi-agent collaborative matching system based on the Copula concept, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the computer executes the computer program instructions, the above-mentioned matching method can be implemented.

[0027] Furthermore, the present invention also provides a computer-readable storage medium, on which are stored computer program instructions that can be executed by a processor. When the computer executes the program instructions, the above-mentioned matching method can be implemented.

[0028] Compared with the traditional compatibility method, the present invention has the following advantages: 1. Break away from the traditional manual compounding and debugging method based on experience, and adopt data-driven control of compounding.

[0029] 2. Most existing multi-agent interaction modeling works usually assume that agents make independent decisions based on their observations, ignoring the complex dependencies between agents. Therefore, the present invention proposes to make matching decisions based on the relationship between agents based on Copula.

[0030] 3. In the present invention, multiple intelligent agents make action decisions based on Markov process reinforcement learning, and fully understand the optimal state of each intelligent agent through data and mechanisms. Different intelligent agents may have completely different goals and state-action representations.

[0031] 4. In the present invention, the Copula function incorporates expert knowledge, etc., and can capture the dependencies between random variables. The difficulty lies in designing a suitable reward function for complex tasks with common implicit goals for multiple agents. The learned connection density can be easily visualized, and the correlation between the actions of the agents can be intuitively analyzed. When the marginal strategy of the agent changes, but the dependency between different agents remains unchanged, a new joint strategy can be effectively obtained by exchanging the margins of the new agent while keeping the copula and other margins unchanged.

[0032] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A multi-agent collaborative matching method based on Copula concept, characterized in that: It is a multi-agent collaborative method based on full-link data drive, integrating prior knowledge, expert knowledge, mechanism knowledge, and Copula functions to track target values.

2. The multi-agent collaborative matching method based on Copula concept according to claim 1 is characterized in that: The steps include: S1. Establish a multi-agent collaborative system, including pre-combustion mixing agent, combustion agent and component detection agent. Multiple agents collaborate to use copula to model the nodes representing variables and the dependency structure between nodes. S2, pre-combustion mixing agent, combustion agent and component detection agent are based on Markov process reinforcement learning action decision-making; S3, put the pre-combustion mixing agent, combustion agent and component detection agent together to learn the copula-based strategy; S4. Through information interaction between the pre-combustion mixing agent and the combustion agent, and between the combustion agent and the component detection agent, the pre-combustion mixing agent is made to understand the hysteresis, combustion conditions, and detection conditions, and make a compatibility decision; In steps S2-S4, each agent incorporates expert knowledge, prior knowledge, and mechanism knowledge when making action decisions and rewards.

3. The multi-agent collaborative matching method based on Copula concept according to claim 2 is characterized in that: The specific steps of step S2 are: No. Agent actions ( ): , ,......, , ; N1 represents the total number of actions of the first agent, N2 represents the total number of actions of the second agent, ..., N1 and N2 are both subscripts of the total number; No. Agent state ( ): , ,......, , ; M1 represents the total number of states of the first agent, M2 represents the total number of states of the second agent, ..., M1, M2, and M3 are all subscripts of the total number; Each agent receives a reward from the environment as a function of its state and action : , and individual observations associated with the state ,award ,in is the discount factor, is the total duration; , the distribution of state transitions among the three agents, where For the three agent states The probability distribution under is the state space; For time, ,action , Agent Observation , the probability at this time is , Joint Strategy .

4. The multi-agent collaborative matching method based on Copula concept according to claim 2 is characterized in that: Step S3 combines the pre-combustion mixing agent, the combustion agent, and the component detection agent to learn the copula-based strategy, and decomposes the parameter joint strategy into: ， For the Agents in parameters The marginal strategy under is the corresponding cumulative distribution function, For the three Action transformation The copula density function is, where It is obtained by transforming the probability of marginal actions.

5. The multi-agent collaborative matching method based on Copula concept according to claim 2 is characterized in that: Step S4 includes four processes: S4.

1. Learn the marginal action distribution of each agent’s current state, taking a state s as input and outputting the marginal action distribution parameters of the three agents given the input state: S4.1.

1. Recording state-action pairs ; S4.1.

2. Calculate each agent in this state Conditional marginal action distribution under ; S4.1.

3. Calculate the maximum likelihood value for each action through SGD , obtain the marginal action distribution of each agent based on the existing data fitting calculation; S4.

2. Learning copula functions S4.2.1, The cumulative probability distribution of ; S4.2.2、Move Normalized distribution value ;in, , ; S4.2.

3. Calculation of Copula Density , calculate the maximum likelihood estimate through SGD ; S4.

3. Determine the joint action of all agents based on the current state S4.3.1 Copula value ; S4.3.

2. Use inverse probability transformation to the original action space ; S4.3.

3. Obtaining the joint actions of all agents ;Here, expert knowledge is also combined with prior knowledge and mechanism knowledge to prevent actions from violating common sense; S4.

4. Obtaining Markov diagram based on state and action movement S4.4.

1. Repeatedly predict the agent's actions given the current state and generate new trajectories as follows: Enter the current state , through step S4.3, the joint action under the current state is obtained , combined with the current environment , and each agent The reward function is combined with the expert experience to set the weight wj, R = w_1×R1+w_2×R2+w_3×R3, and the state of each agent is obtained. ; 4.4.

2. Execute the action generated in step S4.4.1 and obtain the updated state from the environment: , is a cycle length. As time goes by, each time node will have a state action, and a Markov diagram based on state and action movement is obtained; S4.

5. Repeat steps S4.3 and S4.4 until a qualified compatibility decision is obtained.

6. A multi-agent collaborative matching system based on Copula concept, characterized in that: The invention comprises a memory, a processor and computer program instructions stored in the memory and executable by the processor. When the computer executes the computer program instructions, the compatibility method as claimed in any one of claims 1 to 5 can be implemented.

7. A computer-readable storage medium, characterized in that: Computer program instructions that can be executed by a processor are stored thereon, and when the computer executes the program instructions, the compatibility method as described in any one of claims 1 to 5 can be implemented.

Images