Multi-agent surrounding method and system based on dynamic neighbor agent potential field

Abstract

The invention belongs to the technical field of multi-agents, and particularly relates to a multi-agent surrounding method and system based on a dynamic neighbor agent potential field. The multi-agent surrounding method based on the dynamic neighbor agent potential field comprises the following steps: considering the dynamic neighbor agent potential field and a set multi-agent surrounding success condition, and determining a surrounding agent formation; the surrounding agent carries out surrounding of the target agent, if the target agent is successfully captured, surrounding of a neighbor surrounding agent is carried out based on dynamic neighbor agent potential energy, otherwise, the target agent continues to be captured until the target agent is successfully captured; and when all the neighbor surrounding agents successfully surround the target agent, completing multi-agent surrounding. According to the method, the problem of group collaboration of the surrounding agents in a multi-agent surrounding task is solved, the effect of each surrounding agent is exerted, the multi-agent surrounding efficiency is improved, and efficient surrounding of the multi-agents in a complex environment is achieved.

Description

Technical Field

[0001] The present invention belongs to the field of multi-agent technology, and in particular relates to a multi-agent roundup method and system based on a dynamic neighboring agent potential field. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] The most important thing in the multi-agent roundup task is to consider its specific definition, that is, how the multi-agent can reflect the advantages of the group and the cooperation ability between the roundup agents. In the process of the multi-agent roundup task, the roundup agent should maintain a certain roundup formation with other neighboring agents as it approaches the target agent. The roundup formation should not be too loose due to large intervals, nor should it be too tight due to small intervals. When the roundup agents move in coordination, the roundup formation with moderate distance and stability within a certain range is the best.

[0004] The complexity of the environment is mainly determined by the number and distribution of static obstacles in the environment and the conditions for successful capture. There are many environmental factors involved in multi-agent capture tasks, including the target agent, static obstacles in the environment, other capture agents, etc. This requires the capture agent to have the ability to adapt to complex environments and have strong planning and control capabilities.

[0005] The actual scenes of multi-agent roundups are generally complex and full of obstacles. Therefore, how to achieve the rationality and stability of multi-agent roundups in complex environments is a problem that needs to be solved urgently. At the same time, how to solve the problem of group coordination of roundup agents in multi-agent roundups tasks, so as to give full play to the role of each roundup agent and improve the efficiency of multi-agent roundups, is also a difficult problem of multi-agent roundups. Summary of the invention

[0006] In order to solve the above problems, the present invention proposes a multi-agent capture method and system based on dynamic neighbor agent potential field, which considers the dynamic neighbor agent potential field force and the multi-agent capture success conditions to capture the target agent, thereby realizing efficient capture of multi-agents in complex environments.

[0007] According to some embodiments, a first solution of the present invention provides a multi-agent roundup method based on a dynamic neighbor agent potential field, using the following technical solutions:

[0008] A multi-agent hunting method based on dynamic neighbor agent potential field, comprising:

[0009] Obtain multi-agent capture environment information;

[0010] Based on the acquired capture environment information, set the multi-agent capture success conditions;

[0011] Considering the potential field of dynamic neighboring agents and the set multi-agent capture success conditions, determine the capture agent formation;

[0012] The capturing agent captures the target agent in the determined capturing agent formation. If the current capturing agent successfully captures the target agent, the neighbor capturing agent is captured based on the dynamic neighbor agent potential energy. Otherwise, the capturing of the target agent continues until the target agent is successfully captured.

[0013] The neighbor roundup agent rounds up the target agent according to the set multi-agent roundup success conditions;

[0014] When all neighboring hunting agents successfully hunt the target agent, the hunting of the agents ends and the multi-agent hunting is completed.

[0015] As a further technical limitation, during the process of the capturing intelligent agent capturing the target intelligent agent in the determined capturing intelligent agent formation, if the current capturing intelligent agent successfully captures the target intelligent agent, the capturing intelligent agent will be treated as an obstacle and move with the target intelligent agent while maintaining a constant relative position. At this time, the linear velocity of the target intelligent agent is lost by half; the neighboring capturing intelligent agents continue to capture the target intelligent agent, expand the successful capture range of the remaining capturing intelligent agents, and complete the capture of the target intelligent agent.

[0016] As a further technical limitation, the dynamic neighbor intelligent agent potential field force generated based on the dynamic neighbor intelligent agent potential field guides the movement of the encircling intelligent agent to form a encircling formation with a preset distance between the encircling intelligent agents, and when the distance between the encircling intelligent agents is greater than the preset distance, attraction is generated, and when the distance between the encircling intelligent agents is less than the preset distance, repulsion is generated.

[0017] As a further technical limitation, the conditions for successful multi-agent roundup are: when the target agent enters the successful roundup range of the first roundup agent, the target agent's linear velocity is lost, the current roundup agent follows the target agent and moves synchronously at the same linear velocity, and the current target agent is regarded as an obstacle that follows the target agent; when the target agent enters the successful roundup range of other neighboring agents, the roundup mission is successful.

[0018] As a further technical limitation, in the process of rounding up neighbor roundup agents based on the dynamic neighbor agent potential, the state space, action space and reward function of multi-agent roundup are designed in turn.

[0019] Furthermore, the capturing agent avoids obstacles and neighboring agents starting from the initial position. In the process of capturing the target agent, under different states, based on the distance information and angle information in the acquired multi-agent capturing environment information, the relative position of the current capturing agent and the target agent is determined to complete the design of the state space; the action space is the scale factor of the static obstacle repulsion field and the dynamic neighboring agent potential field force distance parameter; the reward function at least includes a capture success reward, an obstacle collision penalty, a neighboring agent collision penalty, a distance guidance reward and a step redundancy penalty.

[0020] According to some embodiments, the second solution of the present invention provides a multi-agent capture system based on a dynamic neighbor agent potential field, which adopts the following technical solutions:

[0021] A multi-agent roundup system based on dynamic neighbor agent potential field, comprising:

[0022] An acquisition module, which is configured to acquire the capture environment information of the multi-agent;

[0023] A setting module, which is configured to set a multi-agent roundup success condition based on the acquired roundup environment information;

[0024] A determination module, which is configured to determine the formation of the encircling agents by considering the dynamic neighboring agent potential field and the set multi-agent encirclement success conditions;

[0025] The round-up module is configured to have the round-up agents round up the target agent in the determined round-up agent formation. If the current round-up agent successfully captures the target agent, the neighbor round-up agents round up the target agent based on the dynamic neighbor agent potential energy. Otherwise, the round-up of the target agent continues until the target agent is successfully captured. The neighbor round-up agents round up the target agent according to the set multi-agent round-up success conditions. When all the neighbor round-up agents successfully round up the target agent, the round-up of the agents ends and the multi-agent round-up is completed.

[0026] According to some embodiments, a third solution of the present invention provides a computer-readable storage medium, which adopts the following technical solution:

[0027] A computer-readable storage medium stores a program, which, when executed by a processor, implements the steps of a multi-agent roundup method based on a dynamic neighbor agent potential field as described in the first solution of the present invention.

[0028] According to some embodiments, a fourth solution of the present invention provides an electronic device, which adopts the following technical solution:

[0029] An electronic device comprises a memory, a processor and a program stored in the memory and running on the processor, wherein when the processor executes the program, the steps of the multi-agent encirclement method based on the dynamic neighbor agent potential field as described in the first scheme of the present invention are implemented.

[0030] According to some embodiments, a fifth solution of the present invention provides a computer program product, which adopts the following technical solution:

[0031] A computer program product includes software code, wherein the program in the software code executes the steps in the multi-agent hunting method based on dynamic neighbor agent potential field as described in the first embodiment of the present invention.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] The present invention considers the potential field force of dynamic neighboring agents and the successful conditions for multi-agent capture to capture the target agent, optimizes the successful conditions for capture and the movement of the target agent in the multi-agent capture task, and realizes efficient capture of multiple agents in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings in the specification that constitute a part of this embodiment are used to provide a further understanding of this embodiment. The schematic embodiments of this embodiment and their descriptions are used to explain this embodiment and do not constitute improper limitations on this embodiment.

[0035] Figure 1 Schematic diagram of the successful conditions of the roundup mission in the first embodiment of the present invention; wherein, Figure 1 (a) is a schematic diagram of the first stage of the successful capture mission. Figure 1 (b) is a schematic diagram of the second stage of the successful roundup mission;

[0036] Figure 2 Schematic diagram of the unreasonable formation of the encircling intelligent agent in the first embodiment of the present invention; wherein, Figure 2 (a) is a schematic diagram of a tight encirclement formation. Figure 2 (b) is a loose and tight encirclement formation;

[0037] Figure 3 It is a flowchart of a multi-agent roundup algorithm based on a dynamic neighbor agent potential field in Embodiment 1 of the present invention;

[0038] Figure 4 Schematic diagram of a training environment for a multi-agent roundup algorithm based on a dynamic neighbor agent potential field in Embodiment 1 of the present invention;

[0039] Figure 5 Schematic diagram of a multi-agent capture process based on a dynamic neighbor agent potential field in Embodiment 1 of the present invention; wherein: Figure 5(a) is a schematic diagram of the multi-agent hunting process at the 100th time step. Figure 5 (b) is a schematic diagram of the multi-agent hunting process at the 400th time step. Figure 5 (c) is a schematic diagram of the multi-agent hunting process at the 700th time step. Figure 5 (d) is a schematic diagram of the multi-agent hunting process at the 1000th time step. Figure 5 (e) is a schematic diagram of the multi-agent hunting process at the 1300th time step. Figure 5 (f) is a schematic diagram of the multi-agent hunting process at the 1600th time step. Figure 5 (g) in the figure is a schematic diagram of the multi-agent hunting process at the 1900th time step. Figure 5 (h) is a schematic diagram of the multi-agent roundup process at the 2200th time step. Figure 5 (i) is a schematic diagram of the multi-agent hunting process at the 2500th time step;

[0040] Figure 6 A schematic diagram of rewards for a multi-agent roundup algorithm based on a dynamic neighbor agent potential field in Embodiment 1 of the present invention;

[0041] Figure 7 Schematic diagram of a complex test environment for a multi-agent roundup algorithm based on a dynamic neighbor agent potential field in Embodiment 1 of the present invention;

[0042] Figure 8 A schematic diagram of a test environment for four encircling intelligent agents in Embodiment 1 of the present invention;

[0043] Fig. 9 Schematic diagram of the test environment for five encircling intelligent agents in Embodiment 1 of the present invention. DETAILED DESCRIPTION

[0044] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0045] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0046] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0047] In the present invention, terms such as "upper", "lower", "left", "right", "front", "back", "vertical", "horizontal", "side", "bottom", etc. indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings. They are relational words determined only for the convenience of describing the structural relationships of the various parts or elements of the present invention, and do not specifically refer to any part or element in the present invention and should not be understood as limitations on the present invention.

[0048] In the present invention, terms such as "fixed connection", "connected", "connection", etc. should be understood in a broad sense, indicating that it can be fixedly connected, integrally connected or detachably connected; it can be directly connected or indirectly connected through an intermediate medium. For relevant scientific research or technical personnel in this field, the specific meanings of the above terms in the present invention can be determined according to specific circumstances, and they cannot be understood as limitations on the present invention.

[0049] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0050] Embodiment 1

[0051] Embodiment 1 of the present invention introduces a multi-agent encirclement method based on a dynamic neighbor agent potential field.

[0052] When executing a multi-agent roundup task, as the number of roundup agents gradually increases, the deep reinforcement learning method is used for training. The generated experience sample data will show a substantial increase as the number of roundup agents increases, which will cause the training time to be too long or even unable to converge. When rounding up isomorphic agents, the present invention can adopt a centralized training and decentralized execution training strategy, and set the state space and action space of each roundup agent to be consistent. Therefore, when the same state is observed in the environment, the actions taken by the roundup agent are also likely to be consistent. During the training process, the current roundup agent can use the experience sample data of other neighboring agents. This strategy can significantly improve the utilization efficiency of the experience sample data.

[0053] In order to solve the problem of group coordination of the capturing agents in the multi-agent capturing task and give full play to the role of each capturing agent to improve the efficiency of the multi-agent capturing, this embodiment introduces the dynamic neighbor agent potential field (DNPF) to improve the capturing effect; by studying the related issues of multi-agent capturing under the conditions of different numbers of capturing agents, the influence of the dynamic neighbor agent potential field on the success rate of the capturing is analyzed.

[0054] This embodiment analyzes the problems related to the multi-agent roundup task and proposes corresponding solutions, defines the conditions for successful multi-agent roundup in a complex environment, and designs the state space, action space and reward function of the multi-agent roundup algorithm based on the dynamic neighbor agent potential field; considers the dynamic neighbor agent potential field, and the multi-agent roundup algorithm process based on the dynamic neighbor agent potential field, and verifies the effectiveness of the improved method by designing simulation experiments.

[0055] The successful conditions of the roundup task are considered more realistically. When the roundup task is actually carried out, the target agent's movement speed will not disappear immediately when it enters the successful roundup range of the roundup agent, that is, the target agent will not stop instantly. The multi-agent roundup task is carried out in a two-dimensional continuous environment, such as Figure 1 The success conditions of the roundup mission shown are set in two stages: Figure 1 As shown in (a) in the first stage, after the target agent enters the successful capture range of the first capture agent, the target agent loses half of its linear velocity, and the current capture agent moves synchronously with the target agent at the same linear velocity, and the current target is regarded as an obstacle that follows the target agent. Figure 1 As shown in (b) in the figure, the neighboring agent is subject to the repulsive force F of the dynamic obstacle. repulse_obstacle ,The second stage is when the target agent enters the successful capture range of other neighbor agents, which is regarded as a success of this capture mission.

[0056] This embodiment adopts the dynamic neighbor agent potential field multi-agent roundup algorithm as the guiding strategy for the roundup agent, where the artificial potential field is an algorithm to guide the roundup agent to move in the environment, and the dynamic neighbor agent potential field is an algorithm to guide the team collaboration of the roundup agent.

[0057] In a multi-agent capture task, ideally the capture agents need to form a suitable formation to capture the target agent, such as Figure 2 As shown in Figure 2, a formation that is too loose or too tight is not conducive to capturing the target agent: Figure 2 As shown in (a) in the figure, if the formation of the encircling agents is too tight, it is easy to cause collisions between the encircling agents, and it is difficult to form an encirclement of the target agent. The target agent is easy to escape from the edge of the encirclement formation, resulting in the failure of the encirclement mission. Figure 2 As shown in (b) in the figure, if the formation of the encircling agents is too loose, there will be a large gap in the middle of the encirclement formed by the encircling agents, and the target agent can easily escape from the gap in the encirclement formation, resulting in the failure of the encirclement mission.

[0058] Depend on Figure 2It can be seen that it is very important to reasonably design the formation of the encircling intelligent agent. Since the environment in which the encircling intelligent agent is located uses an artificial potential field algorithm to guide the movement of the encircling intelligent agent, it is considered to construct a dynamic neighbor intelligent agent potential field based on the original gravitational field from the target intelligent agent and the repulsive field from the static obstacles to guide the encircling intelligent agent to form an encirclement formation that maintains a certain distance in the process of encircling the target intelligent agent. Ideally, it can guide the encircling intelligent agent to form a trend of encircling the target intelligent agent and to avoid collisions with neighboring intelligent agents.

[0059] The dynamic neighbor agent potential field force generated by the dynamic neighbor agent potential field should guide the movement of the encircling agent to form an encircling formation at a certain distance, generate attraction when the encircling agent moves away from a certain distance, and generate repulsion when the encircling agent approaches a certain distance. When designing the dynamic neighbor agent potential field force, the calculation formula of gravity of the artificial potential field method is shown in formula (1):

[0060]

[0061] Among them, P goal is the position of the target agent at the current time step, P agent The position of the rounded-up agent for the current time step.

[0062] The calculation formula of the potential field gravity of the dynamic neighbor agent is shown in formula (2):

[0063]

[0064] Among them, F att_fri,i represents the gravitational force on the capture agent i from the neighboring agents, j∈O(i) represents that the capture agent j is a neighboring agent within the observation range of the capture agent i. j and P i Represent the positions of the current time step of the round-up agent j and the round-up agent i respectively.

[0065] The repulsive force calculation formula of the dynamic neighbor agent potential field is shown in formula (3):

[0066]

[0067] Among them, F rep_fri,i represents the calculation of the gravitational force on the capture agent i from the neighboring agents, r obs is the observation radius of the encircling agent. In this chapter, in order to reflect the cooperative encirclement and capture ability of the encircling agent, the observation radius is set to 1m, and distance is the potential field force distance parameter of the dynamic neighbor agent.

[0068] In summary, the calculation formula of the potential field force of the dynamic neighbor agent is shown in formula (4):

[0069]

[0070] Among them, the dynamic neighbor agent potential field force F fri,i is positive or negative determined by the distance ||P j - P i || between the pursuit agent j and i at the current time step and the distance parameter distance. When the distance between the pursuit agents j and i is too close and less than the distance distance, that is, ||P j - P i || < distance, then F fri,i is calculated as a negative value, and the neighbor agent potential field force acts as a repulsive force; when the distance between the pursuit agents j and i is too close and greater than the distance distance, that is, ||P j - P i || > distance, then F fri,i is calculated as a positive value, and the neighbor agent potential field force acts as an attractive force; theoretically, it meets the expectation of a reasonable pursuit formation design.

[0071] In the multi-agent pursuit task, based on the dynamic neighbor agent potential field algorithm to guide the movement of the pursuit agents, to improve the pursuit efficiency, enhance the cooperation ability among the pursuit agents, and guide the formation of a pursuit formation among the pursuit agents.

[0072] In the multi-agent pursuit task, the pursuit agents start from the initial position to avoid obstacles and neighbor agents. During the process of pursuing the target agent, in different states, the information obtained from the environment can be divided into distance information and angle information. If the distance information and angle information of the current pursuit agent with respect to the target agent, the nearest obstacle, and neighbor agents are known, its relative position can be obtained.

[0073] The distance information includes the distance s dis_cur_goal between the pursuit agent and the target agent at the current time step, the distance s dis_cur_obs between the pursuit agent and the nearest obstacle at the current time step, and the distance s dis_cur_fri between the current pursuit agent and other neighbor agents at the current time step. The distance information s dis set is:

[0074] s dis = (s dis_cur_goal , s dis_cur_obs , s dis_cur_fri ) (5)

[0075] The angle information includes the included angle s angle_cur_goal between the pursuit agent and the target agent at the current time step, the included angle s angle_cur_obs, the angle s between the current time step round-up agent and other neighboring agents angle_cur_fri . Angle information angle The collection is:

[0076] s angle =(s angle_cur_goal ,s angle_cur_obs ,s angle_cur_fri ) (6)

[0077] The current time step termination flag is done and is set to 0. When this round ends, done is set to 1.

[0078] In summary, the state space is designed as

[0079] s=(s dis ,s angle ,done) (7)

[0080] The action space is designed as the static obstacle repulsion field scale factor and the dynamic neighbor agent potential field force distance parameter. The action space is set to A = [a1, a2, a3, a4, a5, a6, a7, a8, a9], where η represents the set value of the static obstacle repulsion field scale factor, and the specific value is obtained from the experiment in the case analysis; distance represents the set value of the dynamic neighbor agent potential field force distance parameter, and the specific value is obtained from multiple tests. The action space design of the multi-agent roundup algorithm based on the dynamic neighbor agent potential field is shown in Table 1.

[0081] Table 1 Action space design of multi-agent hunting algorithm based on dynamic neighbor agent potential field

[0082]

[0083] During the training process of deep reinforcement learning, the agent needs to observe the state of the environment in which it is located and input these states into the network. According to the current strategy, the agent will take corresponding actions. At the same time, the environment will give the agent corresponding rewards according to the actions of the agent at different time steps, and then adjust the weight parameters of the training network and continue to iterate. The agent receives rewards from the beginning of training and continues to accumulate. The goal is to maximize the total reward. Therefore, in the training of the policy network, the design of the reward function is particularly critical.

[0084] In the design of the reward function, the relevant influence of the neighboring agents participating in the roundup task in the multi-agent is added. The specific reward function design is:

[0085] (1) Successful capture reward R successThe reward that the capture agent gets for successfully capturing the target agent within the maximum number of iterations in this round. This is a common reward in the process of reinforcement learning when the agent completes a given task. Set R success =50.

[0086] (2) Obstacle collision penalty R col_obs When the hunting agent collides with an obstacle in the environment, it will be punished accordingly. This is a form of punishment in the multi-agent hunting task. col_obs =-20.

[0087] (3) Neighbor Agent Collision Penalty R col_fri When the current hunting agent collides with a neighboring agent in the environment, it will be punished accordingly. This is a form of punishment that takes into account the coordination of multiple agents in multi-agent tasks. col_fri =-10.

[0088] (4) Distance guidance reward R guide The reward feedback of the roundup agent at different time steps is set to solve the problem of sparse rewards during the learning process. guide =d cur_goal_last -d cur_goal *1.01.

[0089] (5) Step redundancy penalty R step The step penalty that the agent receives at each time step can avoid the problem of redundant steps in the training process to the greatest extent. step =-1.

[0090] In summary, at the end of this round, the cumulative reward value obtained during the training of the round-up agent is:

[0091] R=R success +R col_obs +R col_fri +∑(R guide +R step ) (8)

[0092] like Figure 3The overall process of the multi-agent roundup algorithm based on the dynamic neighbor agent potential field is as follows: in the multi-agent roundup task in a complex environment, first, the roundup agent rounds up the target agent according to the training strategy. If the current roundup agent successfully captures the target agent, the roundup agent will be treated as an obstacle and move with the target agent while maintaining a constant relative position; at the same time, the target agent loses half of its linear velocity; the successful roundup range of the remaining roundup agents is expanded; finally, the remaining roundup agents round up the target agent according to the training strategy within the maximum number of iterations of this round until all the roundup agents are successfully captured, and the roundup task is completed.

[0093] Case Analysis

[0094] In order to solve the problem of multi-agent roundup, this embodiment designs a multi-agent roundup simulation environment based on the dynamic neighbor agent potential field; builds a multi-agent roundup training environment based on the dynamic neighbor agent potential field, including 3 roundup agents, 1 target agent and static obstacles. In order to verify the effectiveness and rationality of the multi-agent roundup algorithm based on the dynamic neighbor agent potential field, simple test environments and complex test environments are set up for experimental verification; in order to verify the scalability and robustness of the multi-agent roundup algorithm based on the dynamic neighbor agent potential field, the number of roundup agents is increased in the test environment for scalability experiments, and finally the simulation experiment results are analyzed.

[0095] The training environment of multi-agent hunting algorithm based on dynamic neighbor agent potential field is as follows: Figure 4 As shown in the figure, in order to be closer to the realistic multi-agent roundup scenario, a more complex multi-agent roundup simulation experiment is designed. Seven rectangular static obstacles are set in the training environment. The center point coordinates of the four lateral obstacles are (1.5, 1), (3.7, 1), (3.7, 2.8), and (1.5, 2.8), and the center point coordinates of the three longitudinal obstacles are (0.75, 2), (3, 2), and (5.25, 2). The length and width of the rectangular static obstacle are 0.8m×0.2m.

[0096] The multi-agent roundup based on soft actor-critic (SAC) and dynamic neighbor agent potential field is shown in Table 2:

[0097] Table 2 Pseudo code of multi-agent hunting algorithm based on SAC and dynamic neighbor agent potential field

[0098]

[0099] The multi-agent hunting process based on the dynamic neighbor agent potential field is as follows: Figure 5 As shown, specifically:

[0100] like Figure 5 As shown in (a), first, three hunting agents are initialized at equal intervals in the upper area of ​​the training environment, and one target agent is randomly initialized in the lower area of ​​the training environment.

[0101] like Figure 5 As shown in (b), the target agent moves to the right, the two encircling agents on the left pass through the upper middle aisle, and the encircling agent on the right is preparing to pass through the right aisle, forming a encirclement trend.

[0102] like Figure 5 As shown in (c) in the figure, since the target agent moves much faster than the capturing agent, when the target agent suddenly moves to the left, the three capturing agents gradually move downward and pass through the middle aisle one by one.

[0103] like Figure 5 As shown in (d) in the figure, the target intelligent agent is subject to the repulsive force of the static obstacle and the surrounding intelligent agent on the left, which restricts the target intelligent agent from passing through the leftmost aisle. At this time, the surrounding intelligent agent below is ready to pass through the middle aisle below to block the target intelligent agent.

[0104] like Figure 5 As shown in (e) in the figure, the three capturing agents maintain a reasonable capturing formation and move further towards the target agent. At this time, the target agent is trapped in the corner and is ready to move to the right.

[0105] like Figure 5 As shown in (f) in the figure, the lower capturing agent is closest to the target agent and is ready to capture the target agent.

[0106] like Figure 5 As shown in (g), the distance between the lower capturing agent and the target agent is close to the successful capture range. The left capturing agent is ready to intercept the target agent from the left side of the obstacle to prevent the target agent from escaping in the opposite direction. The right capturing agent is ready to pass through the middle aisle below.

[0107] like Figure 5 As shown in (h), the lower encircling agent successfully captures the target agent, and the encircling agents on the left and right sides expand the successful encirclement range and move towards the target agent.

[0108] like Figure 5 As shown in (i), the target agent is captured by the remaining capturing agents, and the multi-agent capturing task is successful.

[0109] In order to verify the rationality of the designed dynamic neighbor agent potential field force and the effectiveness of the multi-agent roundup algorithm based on the dynamic neighbor agent potential field, a multi-agent roundup task simulation experiment was set up in a two-dimensional continuous space environment. The training environment includes 3 roundup agents, 1 target agent, and static obstacles. The multi-agent roundup algorithm based on the dynamic neighbor agent potential field was trained for 6000 rounds. The original reward graph and the reward graph after the sliding average operation are shown in Figure 2. Figure 6 shown.

[0110] like Figure 6 As shown in the figure, the horizontal axis is used to represent the number of training rounds, and the vertical axis is used to represent the total cumulative reward value obtained by the three hunting agents in one round. Among them, the blue curve represents the total reward value of each round of the multi-agent hunting algorithm based on the dynamic neighbor agent potential field in the simulation experimental environment, that is, the original reward; the red curve represents the reward value after sliding average processing, that is, the sliding average reward, and the sliding window value is 50. It can be seen from the reward graph that the multi-agent hunting algorithm based on the dynamic neighbor agent potential field tends to converge around 4000 rounds and remains stable in subsequent rounds.

[0111] Successfully saved the training strategy, such as Figure 7 As shown in the figure, a complex test environment with seven static obstacles is set up, and the coordinates of their center points are (1.5, 1), (3.7, 1), (3.7, 2.8), (1.5, 2.8), (2.6, 1.9), (0.6, 1.9), and (4.8, 1.9), respectively. The length and width of the first five static obstacles are 0.8m×0.2m, and the length and width of the last two static obstacles are 0.6m×0.2m.

[0112] In the test environment, the test success rate obtained by using the dynamic neighbor agent potential field algorithm as the hunting strategy for the hunting agent and the test success rate of the multi-agent hunting algorithm based on the dynamic neighbor agent potential field are shown in Table 3.

[0113] Table 3 Test success rates of different algorithms and different test environments

[0114]

[0115]

[0116] It can be seen from Table 3 that the test success rate of the multi-agent hunting algorithm based on APF-DDQN in a simple test environment is 73.3%, and the test success rate in a complex test environment is 67.7%; the test success rate of the multi-agent hunting algorithm based on the dynamic neighbor agent potential field is 79.5% in a simple environment, and the test success rate in a complex test environment is 76.9%; the success rate in a simple test environment is increased by 8.46%, and the success rate in a complex test environment is increased by 13.59%, which proves that the multi-agent hunting algorithm based on the dynamic neighbor agent potential field has strong generalization to different test environments, and also verifies the rationality of the design of the dynamic neighbor agent potential field force in this chapter.

[0117] The scalability of a multi-agent algorithm refers to the ability of the system to effectively handle larger amounts of data and more agents, and to expand and adjust as needed when faced with problems of different scales and complexities. The scalability of a multi-agent algorithm is crucial for practical applications. Only when the system has good scalability can it complete tasks in real time or near real time, thereby improving the stability and flexibility of the system.

[0118] Scalability is an important indicator for evaluating multi-agent reinforcement learning algorithms. In order to verify the scalability of the multi-agent roundup algorithm based on the dynamic neighbor agent potential field, the training strategy was deployed to the test environment of 4 roundup agents and 5 roundup agents respectively to conduct scalability experiments while keeping the training strategy unchanged. The four roundup agent test environments are as follows: Figure 8 As shown, the five round-up agent test environments are as follows Fig. 9 As shown in Table 4, the scalability test data of the multi-agent hunting algorithm based on the dynamic neighbor agent potential field is shown.

[0119] Table 4 Scalability test data of multi-agent hunting algorithm based on dynamic neighbor agent potential field

[0120]

[0121] As can be seen from Table 4, when the multi-agent round-up algorithm based on the dynamic neighbor agent potential field is deployed to 4 round-up agents, the test success rate is 83.9% in a simple test environment and 83.6% in a complex test environment; when the training strategy is deployed to 5 round-up agents, the test success rate is 87.5% in a simple environment and 89.8% in a complex test environment. After analysis, it can be seen that this is due to the increase in the number of round-up agents and static obstacles, which restricts the freedom of movement of the target agent under certain circumstances, so that the target agent is relatively easy to capture in a complex test environment. The scalability of the multi-agent round-up algorithm based on the dynamic neighbor agent potential field is verified.

[0122] Based on the multi-agent round-up task, this embodiment takes into account the difficulties brought by the actual round-up situation, increases the complexity of the experimental environment, optimizes the round-up success conditions and the target agent movement process of the multi-agent round-up task, proposes dynamic neighbor agent potential field force, and trains the round-up agent to obtain the optimal strategy by combining it with the artificial potential field and the flexible actor-critic algorithm; the results show that the reward curve convergence effect of the multi-agent round-up algorithm based on the dynamic neighbor agent potential field is obvious, and the dynamic neighbor agent potential field force can effectively guide the round-up agent to form a reasonable round-up formation to round up the target agent, thereby improving the round-up efficiency. Adding more round-up agents to conduct experiments successfully verified the scalability of the multi-agent round-up algorithm based on the dynamic neighbor agent potential field.

[0123] Embodiment 2

[0124] Embodiment 2 of the present invention introduces a multi-agent capture system based on a dynamic neighbor agent potential field.

[0125] A multi-agent roundup system based on dynamic neighbor agent potential field, comprising:

[0126] An acquisition module, which is configured to acquire the capture environment information of the multi-agent;

[0127] A setting module, which is configured to set a multi-agent roundup success condition based on the acquired roundup environment information;

[0128] A determination module, which is configured to determine the formation of the encircling agents by considering the dynamic neighboring agent potential field and the set multi-agent encirclement success conditions;

[0129] The round-up module is configured to have the round-up agents round up the target agent in the determined round-up agent formation. If the current round-up agent successfully captures the target agent, the neighbor round-up agents round up the target agent based on the dynamic neighbor agent potential energy. Otherwise, the round-up of the target agent continues until the target agent is successfully captured. The neighbor round-up agents round up the target agent according to the set multi-agent round-up success conditions. When all the neighbor round-up agents successfully round up the target agent, the round-up of the agents ends and the multi-agent round-up is completed.

[0130] The detailed steps are the same as the multi-agent encirclement method based on the dynamic neighbor agent potential field provided in Example 1, and will not be repeated here.

[0131] Embodiment 3

[0132] Embodiment 3 of the present invention provides a computer-readable storage medium.

[0133] A computer-readable storage medium stores a program thereon, which, when executed by a processor, implements the steps of a multi-agent encirclement method based on a dynamic neighbor agent potential field as described in Embodiment 1 of the present invention.

[0134] The detailed steps are the same as the multi-agent encirclement method based on the dynamic neighbor agent potential field provided in Example 1, and will not be repeated here.

[0135] Embodiment 4

[0136] A fourth embodiment of the present invention provides an electronic device.

[0137] An electronic device comprises a memory, a processor and a program stored in the memory and running on the processor, wherein when the processor executes the program, the steps of the multi-agent encirclement method based on the dynamic neighbor agent potential field as described in the first embodiment of the present invention are implemented.

[0138] The detailed steps are the same as the multi-agent encirclement method based on the dynamic neighbor agent potential field provided in Example 1, and will not be repeated here.

[0139] Embodiment 5

[0140] Embodiment 5 of the present invention provides a computer program product.

[0141] A computer program product includes software code, wherein the program in the software code executes the steps in the multi-agent encirclement method based on the dynamic neighbor agent potential field as described in the first embodiment of the present invention.

[0142] The detailed steps are the same as the multi-agent encirclement method based on the dynamic neighbor agent potential field provided in Example 1, and will not be repeated here.

[0143] The above description is only a preferred embodiment of the present embodiment and is not intended to limit the present embodiment. For those skilled in the art, the present embodiment may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present embodiment shall be included in the protection scope of the present embodiment.

Claims

1. A multi-agent hunting method based on dynamic neighbor agent potential field, characterized in that: include: Obtain multi-agent capture environment information; Based on the acquired capture environment information, set the multi-agent capture success conditions; Considering the potential field of dynamic neighboring agents and the set multi-agent capture success conditions, determine the capture agent formation; The capturing agent captures the target agent in the determined capturing agent formation. If the current capturing agent successfully captures the target agent, the neighbor capturing agent is captured based on the dynamic neighbor agent potential energy. Otherwise, the capturing of the target agent continues until the target agent is successfully captured. The neighbor roundup agent rounds up the target agent according to the set multi-agent roundup success conditions; When all neighboring hunting agents successfully hunt the target agent, the hunting of the agents ends and the multi-agent hunting is completed.

2. A multi-agent hunting method based on a dynamic neighbor agent potential field as claimed in claim 1, characterized in that: During the process of the capturing intelligent agent capturing the target intelligent agent in the determined capturing intelligent agent formation, if the current capturing intelligent agent successfully captures the target intelligent agent, the capturing intelligent agent will be treated as an obstacle and move with the target intelligent agent while maintaining a constant relative position. At this time, the linear velocity of the target intelligent agent is lost by half; the neighboring capturing intelligent agents continue to capture the target intelligent agent, expand the successful capture range of the remaining capturing intelligent agents, and complete the capture of the target intelligent agent.

3. A multi-agent hunting method based on dynamic neighbor agent potential field as claimed in claim 1, characterized in that: The dynamic neighbor intelligent body potential field force generated based on the dynamic neighbor intelligent body potential field guides the movement of the encircling intelligent body to form an encircling formation with a preset distance between the encircling intelligent bodies. When the distance between the encircling intelligent bodies is greater than the preset distance, attraction is generated, and when the distance between the encircling intelligent bodies is less than the preset distance, repulsion is generated.

4. A multi-agent hunting method based on a dynamic neighbor agent potential field as claimed in claim 1, characterized in that: The multi-agent capture success condition is: when the target agent enters the capture success range of the first capture agent, the target agent loses its linear velocity, the current capture agent moves synchronously with the target agent at the same linear velocity, and the current target agent is regarded as an obstacle that moves with the target agent; When the target agent enters the successful capture range of other neighboring agents, the capture mission is successful.

5. A multi-agent hunting method based on dynamic neighbor agent potential field as claimed in claim 1, characterized in that: In the process of rounding up neighbor roundup agents based on the dynamic neighbor agent potential, the state space, action space and reward function of multi-agent roundup are designed in turn.

6. A multi-agent hunting method based on dynamic neighbor agent potential field as claimed in claim 5, characterized in that: The capturing agent starts from the initial position and avoids obstacles and neighboring agents. In the process of capturing the target agent, under different states, based on the distance information and angle information in the multi-agent capturing environment information obtained, the relative position of the current capturing agent and the target agent is determined to complete the design of the state space; the action space is the scale factor of the static obstacle repulsion field and the dynamic neighboring agent potential field force distance parameter; the reward function at least includes a capture success reward, an obstacle collision penalty, a neighboring agent collision penalty, a distance guidance reward and a step redundancy penalty.

7. A multi-agent capture system based on dynamic neighbor agent potential field, characterized in that: include: An acquisition module, which is configured to acquire the capture environment information of the multi-agent; A setting module, which is configured to set a multi-agent roundup success condition based on the acquired roundup environment information; A determination module, which is configured to determine the formation of the encircling agents by considering the dynamic neighboring agent potential field and the set multi-agent encirclement success conditions; A capture module is configured to capture a target intelligent body in the determined capture intelligent body formation by the capture intelligent body, and if the current capture intelligent body successfully captures the target intelligent body, the neighbor capture intelligent body is captured based on the dynamic neighbor intelligent body potential, otherwise the capture of the target intelligent body continues until the target intelligent body is successfully captured; The neighbor rounding-up agent rounds up the target agent according to the set multi-agent rounding-up success conditions; when all the neighbor rounding-up agents have successfully rounded up the target agent, the rounding-up of the agent ends and the multi-agent rounding-up is completed.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the multi-agent hunting method based on a dynamic neighbor agent potential field as described in any one of claims 1 to 6 are implemented.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the steps of the multi-agent hunting method based on dynamic neighbor agent potential field as described in any one of claims 1-6 are implemented.

10. A computer program product comprising software code, characterized in that The program in the software code executes the steps of the multi-agent hunting method based on dynamic neighbor agent potential field as described in any one of claims 1-6.

Images