Multi-agent control method and device

Abstract

The present invention provides a multi-agent control method and device, which relates to the field of artificial intelligence technology. The method includes: inputting the state information of the current moment into the multi-agent control model, obtaining the target action of each agent at the next moment output by the multi-agent control model; based on the target action of each agent at the next moment, generating the control instructions corresponding to each agent at the next moment, so as to control each multi-agent based on the control instructions corresponding to each agent at the next moment; wherein each agent is used to patrol the target area; the multi-agent control model is constructed based on the Transformer network model and the reinforcement learning network model. The multi-agent control method and device provided by the present invention can combine the Transformer network model and the reinforcement learning network model with simpler calculation process and higher calculation efficiency to realize the control of multi-agents, and can improve the efficiency of controlling multi-agents.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multi-agent control method and device. Background Art

[0002] Reinforcement Learning (RL), also known as reinforcement learning, evaluation learning or enhanced learning, is one of the paradigms and methodologies of machine learning. It can be used to describe and solve the problem of how an intelligent agent can maximize its rewards or achieve specific goals through learning strategies during its interaction with the environment. Traditional reinforcement learning network models may include but are not limited to DQN network model (Deep Q-learning Network), PG network model (Policy Gradient) and AC network model (Actor-Critic). Based on the traditional reinforcement learning network model, control of a single intelligent agent can be achieved.

[0003] In certain business scenarios, such as business scenarios where multiple patrol robots patrol a target area, it is necessary to control multiple agents at the same time. In the prior art, multi-agent control algorithms can be used to control multiple agents. However, the existing multi-agent control algorithms have complex calculation processes and low calculation efficiency, resulting in low efficiency in controlling multiple agents based on the existing multi-agent control algorithms. How to achieve more efficient control of multiple agents based on a reinforcement learning network model with a simpler calculation process and higher calculation efficiency is a technical problem that needs to be solved urgently in this field. Summary of the invention

[0004] The present invention provides a multi-agent control method and device, which are used to solve the defect of low efficiency in controlling multi-agents in the prior art and achieve more efficient control of multi-agents.

[0005] The present invention provides a multi-agent control method, comprising:

[0006] Obtain the attribute information of each agent and the area information of the target area as the current state information;

[0007] Inputting the state information at the current moment into the multi-agent control model, and obtaining the target action of each agent at the next moment output by the multi-agent control model;

[0008] Based on the target action of each agent at the next moment, a control instruction corresponding to each agent at the next moment is generated, so as to control each of the multiple agents based on the control instruction corresponding to each agent at the next moment;

[0009] Among them, each of the intelligent agents is used to patrol the target area; the multi-agent control model is constructed based on the Transformer network model and the reinforcement learning network model.

[0010] According to a multi-agent control method provided by the present invention, the multi-agent control model includes: a Transformer layer and a reinforcement learning layer;

[0011] Accordingly, the step of inputting the state information at the current moment into the multi-agent control model and obtaining the target action of each agent at the next moment output by the multi-agent control model specifically includes:

[0012] Input the state information of the current moment into the Transformer layer, and obtain the executable actions of each agent at the next moment output by the Transformer layer;

[0013] The state information at the current moment and the executable actions of each intelligent agent at the next moment are input into the reinforcement learning layer, and the reinforcement learning layer determines one of the executable actions of each intelligent agent at the next moment as the target action of each intelligent agent at the next moment, and then obtains the target action of each intelligent agent at the next moment output by the reinforcement learning layer.

[0014] According to a multi-agent control method provided by the present invention, the loss function of the multi-agent control model includes: a reward loss function;

[0015] The reward loss function is used to describe minimizing the intersection area between the patrolled areas of any two intelligent agents within a preset time period since each intelligent agent starts patrolling the target area and the area of ​​the unpatrolled area in the target area.

[0016] According to a multi-agent control method provided by the present invention, the Transformer layer includes: a format conversion unit and an action determination unit;

[0017] Accordingly, the state information of the current moment is input into the Transformer layer, and the executable action of each agent at the next moment output by the Transformer layer is obtained, which specifically includes:

[0018] The state information at the current moment is input into the format conversion unit layer, and the format conversion unit sorts the agents based on the position information of each agent at the current moment, thereby obtaining the agent attribute information sequence at the current moment output by the format conversion unit; wherein the agent attribute information includes the agent position information;

[0019] The agent attribute information sequence at the current moment and the area information of the target area are input into the action determination unit, and the executable actions of each agent at the next moment output by the action determination unit are obtained.

[0020] According to a multi-agent control method provided by the present invention, the reinforcement learning layer includes: a probability distribution unit and a result output unit;

[0021] Accordingly, the step of inputting the state information at the current moment and the executable action of each agent at the next moment into the reinforcement learning layer, and obtaining the target action of each agent at the next moment output by the reinforcement learning layer, specifically includes:

[0022] Inputting the state information at the current moment and the executable actions of each agent at the next moment into the probability distribution unit, and obtaining the probability distribution of the executable actions of each agent at the next moment output by the probability distribution unit;

[0023] The probability distribution of the executable actions of each agent at the next moment is input into the result output unit, and the target action of each agent at the next moment output by the result output unit is obtained.

[0024] According to a multi-agent control method provided by the present invention, the attribute information includes: the remaining power value, the visible range radius and the location information of the agent.

[0025] The present invention also provides a multi-agent control device, comprising:

[0026] An information acquisition module is used to obtain the attribute information of each agent and the area information of the target area as the current state information;

[0027] A model calculation module, used to input the state information at the current moment into the multi-agent control model, and obtain the target action of each agent at the next moment output by the multi-agent control model;

[0028] An agent control module, for generating a control instruction corresponding to each agent at the next moment based on the target action of each agent at the next moment, so as to control each of the multiple agents based on the control instruction corresponding to each agent at the next moment;

[0029] Among them, each of the intelligent agents is used to patrol the target area; the multi-agent control model is constructed based on the Transformer network model and the reinforcement learning network model.

[0030] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements any of the multi-agent control methods described above.

[0031] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the multi-agent control methods described above.

[0032] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the multi-agent control methods described above.

[0033] The multi-agent control method and device provided by the present invention inputs state information including attribute information of each agent at the current moment and area information of the target area into the multi-agent control model, obtains the target action of each agent at the next moment output by the above-mentioned multi-agent control model, and generates control instructions corresponding to the agent at the next moment based on the target action of each agent at the next moment, so as to control each agent based on the control instructions corresponding to each agent at the next moment. It can combine the Transformer network model and the reinforcement learning network model with simpler calculation process and higher calculation efficiency to realize the control of multi-agents, and can improve the efficiency of controlling multi-agents. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 It is a flow chart of the multi-agent control method provided by the present invention;

[0036] Figure 2 It is a structural schematic diagram of a traditional DQN network model in the multi-agent control method provided by the present invention;

[0037] Figure 3 It is a structural schematic diagram of a multi-agent control model in the multi-agent control method provided by the present invention;

[0038] Figure 4 It is a structural schematic diagram of an action determination unit in the multi-agent control method provided by the present invention;

[0039] Figure 5It is a structural schematic diagram of the multi-agent control device provided by the present invention;

[0040] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] In the description of the invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0043] Figure 1 Schematic diagram of the multi-agent control method provided by the present invention. Figure 1 The multi-agent control method of the present invention is described. Figure 1 As shown, the method includes: step 101, obtaining attribute information of each intelligent agent and area information of the target area as the current state information; wherein each intelligent agent is used to patrol the target area.

[0044] It should be noted that the intelligent agents in the embodiments of the present invention may include but are not limited to intelligent entities such as drones, vehicles, and patrol robots. A plurality of the above intelligent agents may patrol a specified limited area or a specified limited space.

[0045] The target area is a pre-determined area that needs to be patrolled. Multiple agents can patrol the target area.

[0046] When any intelligent agent patrols the target area, it can respond to the received control instructions and complete the corresponding actions.

[0047] The action of the agent may include a moving direction. In the embodiment of the present invention, 1° may be used as a step length, and the moving direction may be defined as a direction of rotating 1° to 360° in a clockwise direction. For example, the action of the agent may include moving forward at a certain moving speed in a direction after rotating 30° in a clockwise direction.

[0048] It should be noted that the moving speeds of the agents may be the same or different. When the moving speeds of the agents are the same, the moving distances of each agent in the same time period are equal.

[0049] It should be noted that the agents in the embodiment of the present invention are the same agents, that is, the moving speeds of the agents are the same. In addition, the agents keep moving forward at all times. Therefore, for any agent, the action of the agent may only include the moving direction, that is, the action of the agent may only include the angle of clockwise rotation.

[0050] The intelligent agent can observe an area within a certain range based on the visual sensors on the intelligent agent. The visual range of the intelligent agent is related to the performance parameters of the above-mentioned visual sensors and is an inherent attribute of the intelligent agent.

[0051] The attribute information of the agent at the current moment can be used to describe the state of the agent at the current moment. The attribute information of the agent may include at least one of the remaining power of the agent, the visual range, the location information, the moving speed, and the distance between the agent and other agents. The attribute information of the agent is not specifically limited in the embodiments of the present invention.

[0052] Optionally, the attribute information of the agent can be vectorized and represented by an attribute vector. Each dimension of the attribute vector can represent a type of attribute information of the agent.

[0053] For each intelligent agent, the attribute information of the intelligent agent at the current moment can be obtained in a variety of ways. For example, the attribute information of the intelligent agent at the current moment can be obtained based on the controller of the intelligent agent and various sensors.

[0054] The area information of the target area at the current moment can be used to describe the patrol status of the target area at the current moment. The area information of the target area can include but is not limited to the areas patrolled by each agent within a preset time since each agent started patrolling the target area and the areas not yet patrolled.

[0055] The area information of the target area at the current moment may be obtained in a variety of ways, for example, the area information of the target area at the current moment may be obtained through various sensors.

[0056] Step 102: Input the current state information into the multi-agent control model to obtain the target action of each agent at the next moment output by the multi-agent control model; wherein the multi-agent control model is constructed based on the Transformer network model and the reinforcement learning network model.

[0057] It should be noted that, in general, the reinforcement learning network model can better solve problems with finite Markov properties. For the automatic resource scheduling strategy, the reinforcement learning network model itself is a sequential decision problem with finite Markov properties.

[0058] Figure 2 Schematic diagram of the structure of the traditional DQN network model in the multi-agent control method provided by the present invention. Figure 2 As shown in the figure, the traditional DQN network model includes Q-Network unit, Target-Network unit and ExperienceReplay unit. The Q-Network unit and the Target-Network unit have the same network structure, and the weight of the Q-Network unit can be updated to the Target-Network unit after multiple trainings.

[0059] For any intelligent agent used to patrol the target area, it is difficult to make the intelligent agent pay attention to other intelligent agents in the target area based on the traditional DQN network model, that is, it is difficult to obtain the relationship between any two intelligent agents in the target area. The above relationship may include the distance between the above two intelligent agents at the current moment, etc.

[0060] Therefore, in the embodiment of the present invention, a multi-agent control model is constructed based on the Transformer network model and the reinforcement learning network model, and the multi-agent control model is trained to obtain a trained multi-agent control model. The trained multi-agent control model can be used to obtain and output the target action of each agent at the next moment based on the state information at the current moment. The target action is the action that the agent needs to perform at the next moment.

[0061] Optionally, the reinforcement learning network model in the embodiment of the present invention may be a DQN network model, a PG network model (Policy Gradient) or an AC network model (Actor-Critic), etc. In the embodiment of the present invention, the specific reinforcement learning network model is not limited.

[0062] Preferably, since the DQN network model has the advantages of strong versatility, simple calculation process and high calculation efficiency, a multi-agent control model can be constructed based on the Transformer network model and the DQN network model in an embodiment of the present invention.

[0063] Based on the Transformer network model and the reinforcement learning network model, a multi-agent control model is constructed, and the multi-agent control model is trained. After the trained multi-agent control model is obtained, the current state information can be input into the multi-agent control model.

[0064] The above multi-agent control model can obtain and output the target action of each agent at the next moment based on the state information at the current moment.

[0065] Step 103: Based on the target action of each agent at the next moment, a control instruction corresponding to each agent at the next moment is generated, so as to control each agent based on the control instruction corresponding to each agent at the next moment.

[0066] Specifically, after obtaining the target action of each agent at the next moment output by the multi-agent control model, the control instructions corresponding to each agent at the next moment can be generated based on the target action of each agent at the next moment.

[0067] For each agent, after generating the control instruction corresponding to the agent at the next moment, the control instruction corresponding to the agent at the next moment can be sent to the controller of the agent. The controller of the agent can respond to the control instruction corresponding to the agent at the next moment and control the action of the agent at the next moment.

[0068] The embodiment of the present invention inputs state information including attribute information of each agent at the current moment and area information of the target area into a multi-agent control model, obtains the target action of each agent at the next moment output by the multi-agent control model, and generates control instructions corresponding to the agent at the next moment based on the target action of each agent at the next moment. Based on the control instructions corresponding to each agent at the next moment, each agent is controlled. The Transformer network model and a reinforcement learning network model with a simpler calculation process and higher calculation efficiency can be combined to realize the control of multiple agents, which can improve the efficiency of controlling multiple agents.

[0069] Based on the contents of the above embodiments, the attribute information includes: remaining power, visible range and location information.

[0070] Specifically, for any intelligent agent used to patrol the target area, the attribute information of the intelligent agent may include the remaining power, visual range and location information of the intelligent agent.

[0071] Optionally, a plane rectangular coordinate system may be established in the target area, and the position information of any intelligent agent may be represented by the coordinates in the plane rectangular coordinate system.

[0072] Optionally, e can be used to represent the remaining power of the intelligent body; r represents the visible radius of the intelligent body; x and y represent the horizontal and vertical coordinates of the intelligent body in the above coordinate system.

[0073] The attribute information of the agent in the embodiment of the present invention includes the remaining power, visual range and location information of the above-mentioned agent, which can provide a data basis for the multi-agent control model to more efficiently and accurately obtain the target action of each agent at the next moment.

[0074] Figure 3 Schematic diagram of the structure of the multi-agent control model in the multi-agent control method provided by the present invention. Figure 3 As shown in the figure, the multi-agent control model includes: Transformer layer and reinforcement learning layer.

[0075] Accordingly, the state information at the current moment is input into the multi-agent control model to obtain the target action of each agent at the next moment output by the multi-agent control model, specifically including: inputting the state information at the current moment into the Transformer layer to obtain the executable action of each agent at the next moment output by the Transformer layer.

[0076] It should be noted that the Transformer layer is built based on the Transformer network model, and the reinforcement learning layer is built based on the DQN reinforcement learning network model.

[0077] like Figure 3 As shown, Figure 3 The Status in the data may indicate the current status information. After obtaining the current status information, the current status information may be stored in the Center Storage unit.

[0078] It should be noted that the Center Storage unit may be a cache unit.

[0079] The current state information can be obtained from the Center Storage unit, and the current state information can be input into the Transformer layer.

[0080] The Transformer layer can obtain and output the executable actions of each agent at the next moment based on the state information at the current moment.

[0081] It should be noted that, for any agent, the number of executable actions of the agent at the next moment may be one or more.

[0082] Optionally, the executable actions of each agent at the next moment can be vectorized to obtain the action vector corresponding to each agent at the next moment. Each dimension in the action vector corresponding to any agent can represent an executable action of the agent at the next moment.

[0083] The state information at the current moment and the executable actions of each agent at the next moment are input into the reinforcement learning layer, and the reinforcement learning layer determines one of the executable actions of each agent at the next moment as the target action of each agent at the next moment, and then obtains the target action of each agent at the next moment output by the reinforcement learning layer.

[0084] Specifically, the current state information can be obtained from the Center Storage unit, and the current state information and the executable actions of each agent at the next moment can be input into the reinforcement learning layer.

[0085] like Figure 3 As shown in Figure 1, the reinforcement learning layer includes Q-Network unit, Target-Network unit and ExperienceReplay unit. The state vector in the Q-Network unit represents the state information at the current moment.

[0086] The mechanism of the reinforcement learning layer is as follows:

[0087] S0→A0→R1→S1→…→S t-1 →A t-1 →R t →S t →…

[0088] Among them, S0 represents the state information at the initial moment, which is the moment when each agent starts patrolling the target area; A0 represents the target action of each agent at the initial moment; R1 represents the reward value of each agent at the first moment after each agent performs the target action; S t-1 Represents the state information at time t-1; A t-1 represents the target action of each agent at time t-1; R t It represents the reward value of each agent at the tth moment after each agent performs the target action.

[0089] like Figure 3 As shown, Figure 3 The Reward in represents the reward value of each agent at the next moment after each agent performs the target action. Figure 3 Status- in it indicates the status information at the next moment.

[0090] The Transformer layer combined with the reinforcement learning layer can create a mapping like the following:

[0091]

[0092] Among them, a i represents the executable action of agent i at the next moment, a i ∈{V1,V2,…,V k-1 ,V k}, i∈{1,2,…,I}, I represents the number of agents, V k Indicates a clockwise rotation of k degrees; represents the target action of agent i at the next moment; χ represents the state information at the current moment; θ represents the model parameters of the deep learning layer.

[0093] For each agent, the reinforcement learning layer can determine one of the executable actions of the agent at the next moment as the target action of the agent at the next moment based on the state information at the current moment and the executable actions of the agent at the next moment, and then output the target action of each agent at the next moment.

[0094] The embodiment of the present invention inputs the current state information into the Transformer layer in the multi-agent control model, obtains the executable action of each agent at the next moment output by the above Transformer layer, inputs the current state information and the executable action of each agent at the next moment into the reinforcement learning layer in the multi-agent control model, and the reinforcement learning layer determines one of the executable actions of each agent at the next moment as the target action of each agent at the next moment, and then obtains the target action of each agent at the next moment output by the above reinforcement learning layer. The relationship between the agents can be determined based on the Transformer layer, and the executable action of each agent at the next moment can be determined based on the above relationship, thereby realizing the control of multi-agents based on the traditional reinforcement learning network model.

[0095] Based on the contents of the above embodiments, the loss function of the multi-agent control model includes: a reward loss function.

[0096] The reward loss function is used to describe the minimization of the intersection area between the patrolled areas of any two agents within a preset time since each agent starts patrolling the target area and the area of ​​the unpatrolled area in the target area.

[0097] Specifically, when multiple agents patrol a target area, the patrol objectives may include more efficient patrols of the target area and fewer unpatrolled areas.

[0098] More efficient patrols can be represented by the intersection area between any two patrol areas of the target area within a preset time period since each agent started patrolling the target area. The smaller the intersection area, the higher the patrol efficiency.

[0099] Fewer unpatrolled areas can be represented by the area of ​​the unpatrolled area in the target area within a preset time period since each agent starts patrolling the target area. The smaller the area of ​​the unpatrolled area, the fewer unpatrolled areas there are.

[0100] The reward loss function can be expressed by the following formula:

[0101] Υ(Γ,S,A)=αφ(Γ,S,A)+βψ(Γ,S,A),0<α,β≤1

[0102] Among them, φ(Γ,S,A) represents the intersection area loss function; ψ(Γ,S,A) represents the unpatrolled area loss function; α and β represent weights.

[0103] The intersection area loss function φ(Γ,S,A) can be expressed by the following formula:

[0104]

[0105] Wherein, avg means finding the average value; Circle(i,j) means the intersection area between the patrol areas of agent i and agent j within the preset time since each agent started patrolling the target area; r means the visible radius of the agent; ξ = 0.1.

[0106] The loss function of the unpatrolled area ψ(Γ,S,A) can be expressed by the following formula:

[0107]

[0108] Among them, ΔH t It represents the change in the area of ​​the unpatrolled area in the target area at time t within the preset time since each agent started patrolling the target area.

[0109] Based on the reward loss function, the multi-agent control model can be trained. The training goal is to minimize the reward loss function, that is, minimize the intersection area loss function and minimize the unpatrolled area loss function. The specific process of training the multi-agent control model is as follows: Step 1, set the maximum number of iterations, batch size, the number of encoders and decoders in the Transformer layer, and the number of heads, initialize each sample agent, and initialize the Center Storage unit of the reinforcement learning layer.

[0110] Step 2: Start this iteration, input the sample state information at the current moment into the Transformer layer in training, and obtain each predicted executable action of each sample agent at the next moment output by the Transformer layer; wherein, the sample state information at the current moment includes the attribute information of each sample agent and the regional information of the sample area; the attribute information of the sample agent includes the remaining power, visible range and location information of the sample agent; the regional information of the sample area includes but is not limited to the areas that each sample agent has patrolled and the areas that have not been patrolled within the preset time since each sample agent started patrolling the sample area; t-1 represents the current moment.

[0111] Step 3: After inputting each predicted executable action of each sample agent at the next moment into the deep learning layer in training, the deep learning layer in training can traverse each sample agent (or implement it in parallel), determine the predicted target action of each sample agent at the next moment with the delta greedy strategy, and obtain the sample state information at the next moment, as well as the reward value of each sample agent at the next moment after each sample agent executes the predicted target action, that is, obtain S′ t-1 , A′ t-1 , R′ t and S′ t , then S′ t-1 , A′ t-1 , R′ t and S′ t Stored in the Experience Replay unit in the deep learning layer. Where t-1 represents the current moment; S′ t-1 Represents the sample status information at the current moment; A′ t-1 represents the target action of each sample agent at the current moment; R′ t It represents the reward value of each sample agent at the next moment after each sample agent performs the predicted target action; S′ t Indicates the status information at the next moment.

[0112] Step 4: Select a batch of data in the Experience Replay unit, use the output of the Target-Network unit in the reinforcement learning layer as the Groud-truth (real data), and train the reinforcement learning layer and the Transformer layer based on the above Groud-truth and reward loss function.

[0113] Step 5: Based on the results of this iteration, update the above hyperparameters, and repeat steps 2 to 5, provided that the number of iterations does not exceed the maximum number of iterations, until the number of iterations is greater than the maximum number of iterations, to obtain a trained multi-agent control model.

[0114] In the embodiment of the present invention, the multi-agent control model is trained based on the reward-loss function. The reward-loss function is determined based on the intersection area between the patrolled areas of any two agents within a preset time since each agent starts patrolling the target area and the area of ​​the unpatrolled area in the target area. Based on the trained multi-agent control model, more efficient patrols of the target area and fewer unobserved areas can be achieved.

[0115] Based on the contents of the above embodiments, the Transformer layer includes: a format conversion unit and an action determination unit.

[0116] Accordingly, the state information at the current moment is input into the Transformer layer, and the executable action of each agent at the next moment output by the Transformer layer is obtained, which specifically includes: the state information at the current moment is input into the format conversion unit layer, and the format conversion unit sorts each agent based on the position information of each agent at the current moment, and then obtains the agent attribute information sequence at the current moment output by the format conversion unit; wherein the attribute information of the agent includes the position information of the agent.

[0117] Specifically, for agent i used to patrol the target area, the location information index(i) of agent i at the current moment is as follows:

[0118] index(i)=i x +i y

[0119] Among them, i x and i y Represents the coordinate value of agent i in the above plane rectangular coordinate system at the current moment.

[0120] Based on the position information index(i) of each agent at the current moment, each agent is sorted in ascending order, and the agent attribute information sequence obtained at the current moment is as follows:

[0121] G=[g1,g2,…,g i ,…,g I-1 ,g I ] T

[0122] Among them, G represents the agent sequence at the current moment; g i Represents the attribute information (attribute vector) of agent i at the current moment.

[0123] The agent attribute information sequence at the current moment and the area information of the target area are input into the action determination unit, and the executable action of each agent at the next moment output by the action determination unit is obtained.

[0124] Figure 4 is a schematic diagram of the structure of the action determination unit in the multi-agent control method provided by the present invention. Since the input of the action determination unit is the agent attribute information sequence at the current moment, the agent attribute information sequence at the current moment has been vectorized and contains the position information of each agent, therefore, compared with the traditional Transformer network model, such as Figure 4 As shown, the action determination unit in the embodiment of the present invention removes the original Embedding unit and Position-Embedding unit in the traditional Transformer network model.

[0125] Figure 4 The Inputs of the action determination unit are the agent attribute information sequence at the current moment, and the Outputs are the agent attribute information sequence at the current moment after the right shift operation.

[0126] After the agent attribute information sequence at the current moment and the area information of the target area are input into the action determination unit, the action determination unit can obtain and output the executable action of each agent at the next moment based on the agent attribute information sequence at the current moment and the area information of the target area.

[0127] The embodiment of the present invention inputs the state information of the current moment into the format conversion unit in the Transformer layer, obtains the agent attribute information sequence of the current moment output by the above-mentioned format conversion unit, inputs the agent attribute information sequence of the current moment and the area information of the target area into the action determination unit in the Transformer layer, obtains the executable action of each agent at the next moment output by the action determination unit, and can convert the state information of the current moment into a vectorized information sequence that can be recognized by the action determination unit through the format conversion unit, can determine the relationship between the agents based on the action determination unit, and determine the executable action of each agent at the next moment based on the above-mentioned relationship, thereby realizing the control of multiple agents based on the traditional reinforcement learning network model.

[0128] Based on the contents of the above embodiments, the reinforcement learning layer includes: a probability distribution unit and a result output unit.

[0129] Correspondingly, the state information at the current moment and the executable actions of each agent at the next moment are input into the reinforcement learning layer, and the target action of each agent at the next moment output by the reinforcement learning layer is obtained, which specifically includes: inputting the state information at the current moment and the executable actions of each agent at the next moment into the probability distribution unit, and obtaining the probability distribution of the executable actions of each agent at the next moment output by the probability distribution unit.

[0130] Specifically, after the state information at the current moment and the executable actions of each agent at the next moment are input into the probability distribution unit, the probability distribution unit can obtain and output the probability distribution of the executable actions of each agent at the next moment. Among them, corresponding to any agent, the probability distribution of the executable actions of the agent at the next moment can be used to describe the possibility of the agent executing each executable action at the next moment. The probability distribution of the executable actions of the agent at the next moment may include the probability value of the agent executing each executable action at the next moment.

[0131] like Figure 3 As shown, the action vector in the Q-Network unit represents the probability distribution of the executable actions of each agent at the next moment. The probability distribution of the executable actions of each agent at the next moment can be represented by the probability vector corresponding to each agent at the next moment. The probability vector corresponding to agent i at the next moment is as follows:

[0132]

[0133] Among them, p t represents the probability that agent i rotates t° clockwise at the next moment, t∈{1,2,…,k}.

[0134] The probability distribution of the executable actions of each agent at the next moment is input into the result output unit, and the target action of each agent at the next moment output by the result output unit is obtained.

[0135] Specifically, after the probability distribution of the executable actions of each intelligent agent at the next moment is input into the result output unit, the result output unit can use the executable action with the highest probability in the probability distribution of the executable actions of each intelligent agent at the next moment as the target action of each intelligent agent at the next moment, and then obtain the target action of each intelligent agent at the next moment output by the result input unit.

[0136] The embodiment of the present invention inputs the state information of the current moment and the executable actions of each intelligent agent at the next moment into the probability distribution unit in the reinforcement learning layer, obtains the probability distribution of the executable actions of each intelligent agent at the next moment output by the above probability distribution unit, inputs the probability distribution of the executable actions of each intelligent agent at the next moment into the result output unit in the reinforcement learning layer, obtains the target action of each intelligent agent at the next moment output by the above result output unit, and can more accurately and efficiently obtain the target action of each intelligent agent at the next moment based on the reinforcement learning layer, and can more efficiently control multiple intelligent agents.

[0137] Figure 5 Schematic diagram of the structure of the multi-agent control device provided by the present invention. Figure 5The multi-agent control device provided by the present invention is described. The multi-agent control device described below and the multi-agent control method provided by the present invention described above can be referred to each other. Figure 5 As shown, the device includes: an information acquisition module 501, a model calculation module 502 and an intelligent agent control module 503.

[0138] The information acquisition module 501 is used to acquire the attribute information of each agent and the area information of the target area as the current state information.

[0139] The model calculation module 502 is used to input the current state information into the multi-agent control model and obtain the target action of each agent at the next moment output by the multi-agent control model.

[0140] The agent control module 503 is used to generate a control instruction corresponding to each agent at the next moment based on the target action of each agent at the next moment, so as to control each multi-agent based on the control instruction corresponding to each agent at the next moment.

[0141] Among them, each intelligent agent is used to patrol the target area; the multi-agent control model is built based on the Transformer network model and the reinforcement learning network model.

[0142] Specifically, the information acquisition module 501, the model calculation module 502 and the agent control module 503 are electrically connected.

[0143] For each agent, the information acquisition module 501 can obtain the attribute information of the agent at the current moment in a variety of ways, for example, based on the controller of the agent and various sensors, the attribute information of the agent at the current moment can be obtained. The information acquisition module 501 can obtain the area information of the target area at the current moment in a variety of ways, for example, through various sensors, the area information of the target area at the current moment can be obtained.

[0144] The model calculation module 502 inputs the current state information into the multi-agent control model. The multi-agent control model can obtain and output the target action of each agent at the next moment based on the current state information.

[0145] The agent control module 503 can generate a control instruction corresponding to each agent at the next moment based on the target action of each agent at the next moment. For each agent, after generating the control instruction corresponding to the agent at the next moment, the control instruction corresponding to the agent at the next moment can be sent to the controller of the agent. The controller of the agent can respond to the control instruction corresponding to the agent at the next moment and control the action of the agent at the next moment.

[0146] The embodiment of the present invention inputs state information including attribute information of each agent at the current moment and area information of the target area into a multi-agent control model, obtains the target action of each agent at the next moment output by the multi-agent control model, and generates control instructions corresponding to the agent at the next moment based on the target action of each agent at the next moment. Based on the control instructions corresponding to each agent at the next moment, each agent is controlled. The Transformer network model and a reinforcement learning network model with a simpler calculation process and higher calculation efficiency can be combined to realize the control of multiple agents, which can improve the efficiency of controlling multiple agents.

[0147] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the multi-agent control method, which includes: obtaining the attribute information of each agent and the area information of the target area as the state information at the current moment; inputting the state information at the current moment into the multi-agent control model, obtaining the target action of each agent at the next moment output by the multi-agent control model; based on the target action of each agent at the next moment, generating the control instruction corresponding to each agent at the next moment, so as to control each multi-agent based on the control instruction corresponding to each agent at the next moment; wherein each agent is used to patrol the target area; the multi-agent control model is constructed based on the Transformer network model and the reinforcement learning network model.

[0148] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0149] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multi-agent control method provided by the above methods, which includes: obtaining attribute information of each agent and area information of the target area as state information at the current moment; inputting the state information at the current moment into the multi-agent control model, and obtaining the target action of each agent at the next moment output by the multi-agent control model; based on the target action of each agent at the next moment, generating control instructions corresponding to each agent at the next moment, so as to control each multi-agent based on the control instructions corresponding to each agent at the next moment; wherein each agent is used to patrol the target area; the multi-agent control model is constructed based on the Transformer network model and the reinforcement learning network model.

[0150] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the multi-agent control method provided by the above-mentioned methods, the method comprising: obtaining attribute information of each agent and area information of the target area as state information at the current moment; inputting the state information at the current moment into the multi-agent control model, and obtaining the target action of each agent at the next moment output by the multi-agent control model; based on the target action of each agent at the next moment, generating control instructions corresponding to each agent at the next moment, so as to control each multi-agent based on the control instructions corresponding to each agent at the next moment; wherein each agent is used to patrol the target area; the multi-agent control model is constructed based on the Transformer network model and the reinforcement learning network model.

[0151] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0152] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-agent control method, characterized in that: include: Acquire attribute information of each agent and area information of the target area as the current state information, wherein the attribute information of the agent includes at least one of the remaining power of the agent, the visual range, the position information, the moving speed, and the distance between the agent and other agents, and the area information of the target area includes the areas that have been patrolled by each agent and the areas that have not been patrolled within a preset time period since each agent starts patrolling the target area; Inputting the state information at the current moment into the multi-agent control model, and obtaining the target action of each agent at the next moment output by the multi-agent control model; Based on the target action of each agent at the next moment, a control instruction corresponding to each agent at the next moment is generated, so as to control each of the multiple agents based on the control instruction corresponding to each agent at the next moment; Wherein, each of the intelligent agents is used to patrol the target area; the multi-agent control model is constructed based on the Transformer network model and the reinforcement learning network model; Multi-agent control model, including: Transformer layer and reinforcement learning layer; Accordingly, the step of inputting the state information at the current moment into the multi-agent control model and obtaining the target action of each agent at the next moment output by the multi-agent control model specifically includes: Input the state information of the current moment into the Transformer layer, and obtain the executable actions of each agent at the next moment output by the Transformer layer; The state information at the current moment and the executable actions of each agent at the next moment are input into the reinforcement learning layer, and the reinforcement learning layer determines one of the executable actions of each agent at the next moment as the target action of each agent at the next moment, thereby obtaining the target action of each agent at the next moment output by the reinforcement learning layer; The reinforcement learning layer includes: a probability distribution unit and a result output unit; Accordingly, the step of inputting the state information at the current moment and the executable action of each agent at the next moment into the reinforcement learning layer, and obtaining the target action of each agent at the next moment output by the reinforcement learning layer, specifically includes: Inputting the state information at the current moment and the executable actions of each agent at the next moment into the probability distribution unit, and obtaining the probability distribution of the executable actions of each agent at the next moment output by the probability distribution unit; The probability distribution of the executable actions of each agent at the next moment is input into the result output unit, and the target action of each agent at the next moment output by the result output unit is obtained.

2. The multi-agent control method according to claim 1, characterized in that: The loss function of the multi-agent control model includes: a reward loss function; The reward loss function is used to describe minimizing the intersection area between the patrolled areas of any two intelligent agents within a preset time period since each intelligent agent starts patrolling the target area and the area of ​​the unpatrolled area in the target area.

3. The multi-agent control method according to claim 1, characterized in that: The Transformer layer includes: a format conversion unit and an action determination unit; Accordingly, the state information of the current moment is input into the Transformer layer, and the executable action of each agent at the next moment output by the Transformer layer is obtained, which specifically includes: The state information at the current moment is input into the format conversion unit layer, and the format conversion unit sorts the agents based on the position information of each agent at the current moment, thereby obtaining the agent attribute information sequence at the current moment output by the format conversion unit; wherein the agent attribute information includes the agent position information; The agent attribute information sequence at the current moment and the area information of the target area are input into the action determination unit, and the executable actions of each agent at the next moment output by the action determination unit are obtained.

4. The multi-agent control method according to claim 1 or 2, characterized in that: The attribute information includes: the remaining power value, the visible range radius and the location information of the intelligent body.

5. A multi-agent control device, characterized in that: include: An information acquisition module is used to acquire the attribute information of each agent and the area information of the target area, as the current state information, the attribute information of the agent includes at least one of the remaining power of the agent, the visual range, the position information, the moving speed and the distance between the agent and other agents, and the area information of the target area includes the areas that have been patrolled by each agent and the areas that have not been patrolled within a preset time since each agent started patrolling the target area; A model calculation module, used to input the state information at the current moment into the multi-agent control model, and obtain the target action of each agent at the next moment output by the multi-agent control model; An agent control module, for generating a control instruction corresponding to each agent at the next moment based on the target action of each agent at the next moment, so as to control each of the multiple agents based on the control instruction corresponding to each agent at the next moment; Wherein, each of the intelligent agents is used to patrol the target area; the multi-agent control model is constructed based on the Transformer network model and the reinforcement learning network model; Multi-agent control model, including: Transformer layer and reinforcement learning layer; Accordingly, the step of inputting the state information at the current moment into the multi-agent control model and obtaining the target action of each agent at the next moment output by the multi-agent control model specifically includes: Input the state information of the current moment into the Transformer layer, and obtain the executable actions of each agent at the next moment output by the Transformer layer; The state information at the current moment and the executable actions of each agent at the next moment are input into the reinforcement learning layer, and the reinforcement learning layer determines one of the executable actions of each agent at the next moment as the target action of each agent at the next moment, thereby obtaining the target action of each agent at the next moment output by the reinforcement learning layer; The reinforcement learning layer includes: a probability distribution unit and a result output unit; Accordingly, the step of inputting the state information at the current moment and the executable action of each agent at the next moment into the reinforcement learning layer, and obtaining the target action of each agent at the next moment output by the reinforcement learning layer, specifically includes: Inputting the state information at the current moment and the executable actions of each agent at the next moment into the probability distribution unit, and obtaining the probability distribution of the executable actions of each agent at the next moment output by the probability distribution unit; The probability distribution of the executable actions of each agent at the next moment is input into the result output unit, and the target action of each agent at the next moment output by the result output unit is obtained.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the multi-agent control method as described in any one of claims 1 to 4 is implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multi-agent control method as described in any one of claims 1 to 4 is implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the multi-agent control method as described in any one of claims 1 to 4 is implemented.

Images