Multi-agent interaction method and agents

Abstract

The invention discloses a multi-agent interaction method and agents, and belongs to the technical field of agents. The multi-agent interaction method comprises the steps that all agents collect corresponding image information, and the image information comprises relative distance information and mutual view angle information between the agents and other types of agents; the intelligent agent generates a target control instruction based on the image information; the intelligent body moves based on the target control instruction; wherein the intelligent agent is obtained based on reward function training with the purpose that the distance between the intelligent agent and other types of intelligent agents reaches the target condition, and under the condition that the intelligent agent is the first type of intelligent agent, the target condition is distance minimization; and under the condition that the intelligent agent is the second type of intelligent agent, the target condition is distance maximization. According to the multi-agent interaction method provided by the invention, chasing of multiple agents in a complex scene can be realized, so that the interaction effect among the multiple agents is improved.

Description

Technical Field

[0001] This application belongs to the technical field of agents, and particularly relates to an interaction method for multiple agents and an agent. Background Art

[0002] Artificial intelligence technology is being increasingly widely applied to people's daily work, study, and entertainment. In related technologies, the focus is mainly on the interaction between an agent and a static environment. However, in a multi-agent environment, both the agents and the environment are changing, and each agent is "breaking" the strategies of the others, making the existing training methods inapplicable to the multi-agent environment and affecting the interaction effect between multiple agents. Summary of the Invention

[0003] This application aims to solve at least one of the technical problems existing in the prior art. For this purpose, this application proposes an interaction method for multiple agents and an agent, which can realize the pursuit of multiple agents in a complex scenario, thereby improving the interaction effect between multiple agents.

[0004] In a first aspect, this application provides an interaction method for multiple agents, and the method includes: Each agent collects corresponding image information, and the image information includes the relative distance information and mutual perspective information between the agent and other types of agents; each of the agents is in the same scenario; The agent generates a target control instruction based on the image information; The agent moves based on the target control instruction; where The agent is trained based on a reward function with the goal of the distance between the agent and other types of agents reaching a target condition. When the agent is a first type of agent, the target condition is distance minimization; when the agent is a second type of agent, the target condition is distance maximization.

[0005] According to the interaction method for multiple agents of this application, a reward function is designed based on the relative distance information and mutual perspective information between agents, and the agents are trained based on the reward function with the goal of the distance between the agent and other types of agents reaching a target condition. Thus, in the case where there are multiple agents in the environment, each agent can generate a corresponding target control instruction for movement based on the collected image information, so as to realize the pursuit of multiple agents in a complex scenario, thereby improving the interaction effect between multiple agents.

[0006] According to an embodiment of this application, the reward function is trained through the following steps: Obtain the sample image information collected by the agent at the current acquisition moment; Input the sample image information, the expected distance, and the expected relative viewing angle into the agent, and obtain the actual distance and the actual relative viewing angle of the agent from other types of agents when the agent moves according to the predicted control instruction. Train the reward function based on the difference between the actual distance and the expected distance, and the difference between the actual relative viewing angle and the expected relative viewing angle.

[0007] According to an embodiment of the present application, the training of the reward function based on the difference between the actual distance and the expected distance, and the difference between the actual relative viewing angle and the expected relative viewing angle includes: Construct a collision penalty function based on the collision situation between the agent and the obstacles in the scene when the agent moves according to the predicted control instruction. Update the reward function based on the collision penalty function.

[0008] According to an embodiment of the present application, the updating of the reward function based on the collision penalty function includes: Obtain the environmental perception information recognized by the agent based on the sample image information. Construct an exploration reward function based on the environmental perception information. Update the reward function based on the exploration reward function and the collision penalty function.

[0009] According to an embodiment of the present application, the agent is trained based on a reward function with the goal of the distance between the agent and other types of agents reaching a target condition, including: In the first alternating training stage, when the other types of agents move based on a preset control instruction, train the reward function with the goal of the distance between the agent and the other types of agents reaching the target condition, and obtain the predicted control instruction corresponding to the current training stage. In the non-first alternating training stage, when the other types of agents move based on the predicted control instruction obtained in the previous training stage, train the reward function with the goal of the distance between the agent and the other types of agents reaching the target condition.

[0010] According to an embodiment of the present application, the agent is trained based on a reward function with the goal of the distance between the agent and other types of agents reaching a target condition, including: Correspondingly set the training goal and the scene corresponding to the training stage according to the training stage. In the scenario corresponding to the current training stage, train the reward function with the training objective corresponding to the current training stage, and use the trained reward function obtained in the current training stage as the initial reward function for the next training stage.

[0011] According to an embodiment of the present application, the agent generates a target control instruction based on the image information, including: The agent generates the target control instruction based on the image information, which is used to indicate the moving direction and / or motion posture of the agent.

[0012] In a second aspect, the present application provides an interaction device for multiple agents, and the device includes: A first processing module, configured to enable each agent to collect corresponding image information, where the image information includes relative distance information and mutual viewing angle information between the agent and other types of agents; each of the agents is in the same scenario; A second processing module, configured to enable the agent to generate a target control instruction based on the image information; A third processing module, configured to enable the agent to move based on the target control instruction; where The agent is trained based on a reward function with the goal of the distance between the agent and other types of agents reaching a target condition. When the agent is a first type of agent, the target condition is distance minimization; when the agent is a second type of agent, the target condition is distance maximization.

[0013] According to the interaction device for multiple agents of the present application, a reward function is designed based on the relative distance information and mutual viewing angle information between agents, and the agents are trained based on the reward function with the goal of the distance between the agent and other types of agents reaching a target condition. Therefore, in the case where there are multiple agents in the environment, each agent can generate corresponding target control instructions based on the collected image information to move, so as to achieve the pursuit of multiple agents in a complex scenario, thereby improving the interaction effect between multiple agents.

[0014] In a third aspect, the present application provides an agent, and the agent moves based on the multi-agent interaction method as described in the first aspect.

[0015] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the multi-agent interaction method as described in the first aspect above.

[0016] Fifth aspect, the present application provides a computer program product, including a computer program which, when executed by a processor, implements the interaction method of multi-agent as described in the first aspect above.

[0017] One or more of the above technical solutions in the embodiments of the present application have at least one of the following technical effects: Design a reward function based on the relative distance information and mutual perspective information between agents, aiming at the distance between an agent and other types of agents reaching the target condition, and train the agent based on the reward function, so that in the case of multiple agents in the environment, each agent can generate corresponding target control instructions for movement according to the collected image information respectively, so as to achieve the pursuit of multiple agents in a complex scene, thereby improving the interaction effect between multiple agents.

[0018] Furthermore, design a collision penalty function based on the collision situation between the agent and the obstacles in the scene, and optimize the reward function based on the collision penalty function, so that on the basis of the distance between the agent and other types of agents reaching the target condition, the agent can also learn to cope with actions such as jumping, squatting, and running of the obstacles in the complex scene, bypass the obstacles in the scene, and achieve the pursuit of multiple agents in the complex scene, further improving the interaction effect.

[0019] Even further, through the design of a multi-stage distributed RL training process, gradually increase the training difficulty, and dynamically adjust the reward function during the training process, which can accelerate the convergence speed of agent training, thereby improving the training speed and generalization ability of the agent, and making it more suitable for the pursuit interaction of multiple agents in a complex environment.

[0020] Still further, by setting an alternating training method for two types of agents, the agents can continuously adapt to and optimize their respective behaviors when the strategies of other types of agents change, increasing the generality of the algorithm, further improving the interaction effect, and having universality.

[0021] The additional aspects and advantages of the present application will be partially given in the following description, partially become obvious from the following description, or be understood through the practice of the present application. Description of the Drawings

[0022] The above and / or additional aspects and advantages of the present application will become obvious and easy to understand from the description of the embodiments in conjunction with the following drawings, where: Figure 1 is one of the flow diagrams of the interaction method of multi-agent provided by the embodiment of the present application; Figure 2 is the second flow diagram of the interaction method of multi-agent provided by the embodiment of the present application; Figure 3It is a schematic structural diagram of an interaction device for multi - agents provided by an embodiment of the present application; Figure 4 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0023] Next, the technical solutions in the embodiments of the present application will be clearly described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art belong to the scope of protection of the present application.

[0024] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are usually of the same category, and do not limit the number of objects. For example, the first object can be one or multiple. In addition, "and / or" in the specification and claims means at least one of the connected objects, and the character " / " generally indicates an "or" relationship between the associated objects before and after.

[0025] Next, in conjunction with the accompanying drawings, the multi - agent interaction method, multi - agent interaction device, electronic device, and readable storage medium provided by the embodiments of the present application will be described in detail through specific embodiments and their application scenarios.

[0026] Among them, the multi - agent interaction method can be applied to a terminal, and specifically can be executed by hardware or software in the terminal.

[0027] The multi - agent interaction method provided by the embodiments of the present application, the execution subject of this multi - agent interaction method can be an electronic device or a functional module or functional entity in the electronic device that can implement this multi - agent interaction method. The electronic devices mentioned in the embodiments of the present application include but are not limited to mobile phones, tablet computers, computers, cameras, and wearable devices, etc. Here, the electronic device is used as an example of the execution subject to illustrate the multi - agent interaction method provided by the embodiments of the present application.

[0028] As Figure 1 shown, this multi - agent interaction method includes: Step 110, Step 120, and Step 130.

[0029] Step 110: Each agent collects corresponding image information, and each agent is in the same scene; In this step, the agent can be an entity object, such as an embodied agent or a robot, etc.; or the agent can also be a virtual object.

[0030] Among them, the virtual objects may include, but are not limited to: virtual characters, virtual animals, and other virtual objects with motion characteristics.

[0031] The scene can be a physical scene or a virtual scene. The virtual scene can include virtual spaces, such as one or more virtual rooms, etc.

[0032] It can be understood that in the virtual scene, the virtual scene can include multiple agents. While each agent interacts with other static environmental features in the virtual scene, the agents can also interact with each other.

[0033] It should be noted that in this application, multiple agents in the same scene can be divided into at least two categories, namely, the first type of agent and the second type of agent. Among them, the first type of agent is used to chase the second type of agent, and the second type of agent is used to escape from the chase of the first agent.

[0034] Each agent can collect image information for characterizing the scene environment situation. The image information includes the relative distance information and the mutual perspective information between the agent and other types of agents.

[0035] Among them, the relative distance information includes the distance between the agents, which is used to characterize the distance between the agents; the mutual perspective information includes the relative perspective, which is used to characterize the postures such as the orientations of the agents. By introducing the mutual perspective information, it can ensure that each agent has the correct posture on the basis of meeting the distance requirement, making its behavior more natural, realistic, and efficient.

[0036] In the actual execution process, the relative distance information and the mutual perspective information can be obtained from the image information through algorithms such as image recognition.

[0037] Step 120: The agent generates a target control instruction based on the image information; In this step, the agent can make a prediction according to the image information to obtain a target control instruction that makes the distance between it and other types of agents approach the target condition.

[0038] Among them, the target condition is determined based on the type of the agent. When the agent is the first type of agent, the target condition is distance minimization; when the agent is the second type of agent, the target condition is distance maximization.

[0039] For the first type of agent, the target control instruction can be used as a tracking strategy to indicate that it should chase the second type of agent as soon as possible; for the second type of agent, the target control instruction can be used as an escape strategy to indicate that it should not be chased by the first type of agent for as long as possible.

[0040] In some embodiments, step 120 may include: The agent generates a target control instruction based on the image information, which is used to indicate the moving direction and / or motion posture of the agent.

[0041] In this embodiment, the moving direction may include moving forward, backward, left or right, etc.

[0042] The motion posture may include: squatting down, standing up, jumping, walking or running, etc.

[0043] During the actual execution process, the agent collects image information and automatically predicts, based on the image information, a target control instruction that can make the distance between it and other types of agents approach the target condition, which is used to indicate its moving direction and motion posture.

[0044] Step 130: The agent moves based on the target control instruction.

[0045] In this step, the agent is trained based on a reward function with the goal of the distance between the agent and other types of agents reaching the target condition.

[0046] The reward function can be designed and constructed based on the A3C (Asynchronous Advantage Actor-Critic) algorithm. The reward function can be correspondingly designed based on the type of the agent. For the first type of agent, the reward function should drive them to approach and capture the target faster and more efficiently. For the second type of agent, the reward function should encourage them to escape better or maintain a distance.

[0047] In some embodiments, bilateral confrontation can also be introduced during the training process, and the reward function should reflect a zero-sum game (such as one type of agent scoring and the other type of agent losing points).

[0048] In some embodiments, the training can also be balanced by dynamically adjusting the reward function or introducing an adaptive confrontation mechanism, so as to reduce situations where if the first type of agent is too strong, the second type of agent has no hope; if the second type of agent is too fast, the first type of agent cannot learn, etc.

[0049] In some embodiments, the reward for the agent to pass through obstacles can also be appropriately increased to encourage the agent to effectively pass through the obstacles.

[0050] The design of the reward function should consider both the distance between the first type of agent and the second type of agent and their mutual perspectives / orientations, so that the agent not only has to approach, but also should adopt the "correct posture" to pursue or escape, making the behavior more natural and efficient.

[0051] During the training process, the training samples corresponding to different types of agents can be the same, but their training objectives are opposite. When the agent is a first-type agent, its training objective is to minimize the distance; when the agent is a second-type agent, its training objective condition is to maximize the distance. Multi-stage distributed RL training process design and multi-stage alternating training processes can be adopted to accelerate the training of each agent.

[0052] The specific training process will be described in the following embodiments and will not be elaborated here for the time being.

[0053] Taking a virtual scenario that includes two agents, with the task instruction: pick up the toy car in the room and snatch the toy car as an example for illustration.

[0054] In the actual application process, each agent should first identify the environmental scenario, find the location of the toy car and pick up the toy car. After picking up the toy car, the agent that picks up the toy car is updated to an escaping agent (i.e., the second-type agent), and the agent that does not pick up the toy car is updated to a chasing agent (i.e., the first-type agent). The chasing agent needs to chase the escaping agent to snatch the toy car; while the escaping agent needs to avoid the chase of the chasing agent and protect the toy car from being snatched.

[0055] Each agent respectively collects image information to identify the relative distance information and mutual perspective information between itself and the other agent based on the image information. The chasing agent generates a target control instruction for approaching the escaping agent based on the relative distance information and mutual perspective information, and moves based on this target control instruction; at the same time, the escaping agent generates a target control instruction for moving away from the chasing agent based on the relative distance information and mutual perspective information, and moves based on this target control instruction; during the movement process, each agent collects new image information again and generates a new target control instruction based on the new image information to enter the next round of control, and so on in a loop, so as to achieve the chase between multiple agents.

[0056] If the chasing agent chases the escaping agent and snatches the toy car, the task can be ended. In some embodiments, the chasing agent and the escaping agent can also be swapped in identity, and the chasing strategy from step 110 to step 130 can be executed again.

[0057] It should be noted that in the present application, multiple agents within a scenario can learn independently. That is, while the first type of agent moves based on the manner from step 110 to step 130, the second type of agent also moves based on the manner from step 110 to step 130. Each agent "breaks" the strategies of each other. For example, based on the current scenario and the chasing strategy of the first type of agent, the second type of agent can generate a new escape strategy; the first type of agent will further learn a new tracking strategy based on the new escape strategy learned by the second type of agent, so as to achieve intelligent chasing among multiple agents.

[0058] In some embodiments, the image information may further include obstacle information within the scenario, and the target control instruction may further include an instruction for instructing the agent to avoid obstacles.

[0059] According to the multi-agent interaction method provided by the embodiments of the present application, a reward function is designed based on the relative distance information and mutual perspective information between agents. With the goal that the distance between an agent and other types of agents reaches the target condition, the agent is trained based on the reward function. Thus, in the case where there are multiple agents in the environment, each agent can respectively generate corresponding target control instructions according to the collected image information to move, so as to achieve the chasing of multiple agents in a complex and dynamically changing scenario, thereby improving the interaction effect among multiple agents.

[0060] The following specifically describes the training method of the agent.

[0061] In some embodiments, the reward function can be trained through the following steps: Obtain the sample image information collected by the agent at the current acquisition moment; Input the sample image information, the expected distance, and the expected relative perspective into the agent, and obtain the actual distance and actual relative perspective of the agent from other types of agents when moving according to the predicted predicted control instruction. Train the reward function based on the difference degree between the actual distance and the expected distance, and the difference degree between the actual relative perspective and the expected relative perspective.

[0062] In this embodiment, the sample image information may include the actual relative distance information and actual mutual perspective information between the agent and other types of agents in the training scenario within the current video frame.

[0063] The expected distance and the expected relative perspective are the distance and relative perspective that are expected for this agent to achieve with other types of agents in the next video frame. The expected relative perspective can be user-defined, such as set within a target angle, such as within 60° or 55°.

[0064] For different types of agents, the set values of the expected distances are different. Taking the agent as the first type of agent as an example, the expected distance corresponding to the first type of agent should be a relatively small distance; taking the agent as the second type of agent as an example, the expected distance corresponding to the second type of agent should be a relatively large distance.

[0065] For the first type of agent, input the sample image information, the expected distance, and the expected relative viewing angle into the agent. Train the agent with the goal of minimizing the distance between the agent and other types of agents. Output a predictive control instruction for enabling the agent to minimize the distance from other types of agents as much as possible, and control the movement of the agent according to the movement direction and / or motion posture indicated by the predictive control instruction, and obtain the new actual distance and actual relative viewing angle between the agent and other agents in the next video frame.

[0066] Then, based on the degree of difference between the actual distance and the expected distance, such as the distance magnitude; and the degree of difference between the actual relative viewing angle and the expected relative viewing angle, train the reward function. It can be understood that the smaller the degree of difference, the higher the corresponding reward score; conversely, the lower the reward score.

[0067] Input the current new actual distance and actual relative viewing angle, the expected distance, and the expected relative viewing angle into the agent. Train the agent with the goal of minimizing the distance between the agent and other types of agents. Output a new predictive control instruction, and control the movement of the agent according to the movement direction and / or motion posture indicated by the new predictive control instruction, and repeat this process until convergence, and end the training of the agent.

[0068] For the second type of agent, input the sample image information, the expected distance, and the expected relative viewing angle into the agent. Train the agent with the goal of maximizing the distance between the agent and other types of agents. Output a predictive control instruction for enabling the agent to maximize the distance from other types of agents as much as possible, and control the movement of the agent according to the movement direction and / or motion posture indicated by the predictive control instruction, and obtain the new actual distance and actual relative viewing angle between the agent and other agents in the next video frame.

[0069] Then, based on the degree of difference between the actual distance and the expected distance, such as the distance magnitude; and the degree of difference between the actual relative viewing angle and the expected relative viewing angle, train the reward function. It can be understood that the smaller the degree of difference, the higher the corresponding reward score; conversely, the lower the reward score. The training steps are similar to those of the first type of agent and will not be elaborated here.

[0070] It should be noted that during the training process, other types of agents may be in a stationary state or a moving state. Other types of agents may move according to a fixed motion strategy or according to the same training method.

[0071] According to the multi-agent interaction method provided by the embodiments of the present application, a reward function is designed based on the relative distance information and mutual perspective information between agents, aiming to make the distance between an agent and other types of agents reach the target condition. Based on the reward function, the agent is trained, so that in the case of multiple agents in the environment, each agent can generate corresponding target control instructions for movement according to the collected image information, so as to realize the pursuit of multiple agents in a complex scene, thereby improving the interaction effect between multiple agents.

[0072] In some embodiments, training the reward function based on the difference degree between the actual distance and the expected distance, and the difference degree between the actual relative perspective and the expected relative perspective may include: Constructing a collision penalty function based on the collision situation between the agent and the obstacles in the scene when the agent moves according to the predicted control instruction; Updating the reward function based on the collision penalty function.

[0073] In this embodiment, the collision situation is used to characterize whether the agent collides with the obstacle and the number of collisions, etc. Taking the scene as a virtual room as an example, the obstacles include but are not limited to: sofas, tables, chairs, cabinets, and doors in the room. It can be understood that the more times of collision occur, the higher the corresponding collision penalty score is.

[0074] In the actual execution process, the agent can also be trained to recognize the obstacles in the environment based on the sample image information and the labels of each obstacle in the sample image information, and the agent is trained with the goal of generating a predicted control instruction for the agent to perform obstacle avoidance operations. For example, in the case of encountering a step, the agent performs the action of "jumping"; obtaining the collision situation between the agent and the obstacle during the execution of the target control instruction, obtaining the collision penalty score, subtracting the collision penalty score from the reward score, and using the new score as the reward score to train the agent, so that the generated predicted control intelligence can make the distance between the agent and other types of agents reach the target condition, and can further avoid the obstacles in the scene.

[0075] According to the multi-agent interaction method provided by the embodiments of the present application, a collision penalty function is designed based on the collision situation between the agent and the obstacles in the scene, and the reward function is optimized based on the collision penalty function, so that on the basis that the distance between the agent and other types of agents reaches the target condition, the agent can also learn to perform actions such as jumping, squatting, and running to cope with the obstacles in the complex scene, avoid the obstacles in the scene, realize the pursuit of multiple agents in the complex scene, and further improve the interaction effect.

[0076] In some embodiments, when the agent is a second - type agent, updating the reward function based on the collision penalty function may include: Obtain the environmental perception information recognized by the agent based on the sample image information; Construct an exploration reward function based on the environmental perception information; Update the reward function based on the exploration reward function and the collision penalty function.

[0077] In this embodiment, the environmental perception information is the perception or exploration situation of the agent about other objects in the scene where it is located.

[0078] In some embodiments, an exploration reward function can be further constructed. The exploration reward score is used to evaluate the agent's exploration of the environment in the scene. For example, when a new object or a new other agent in the scene is recognized, the higher the exploration degree, the higher the corresponding exploration reward score.

[0079] During the actual execution process, the reward function can be designed based on the following steps: First, the reward function corresponding to the first - type agent Parameters: - dis2target_now: The current distance to the target - direction_error: The angular error towards the target - dis_exp: The expected distance to the target - is_collision: Whether a collision occurs Return: - reward: The comprehensive reward value """ # If dis_exp is not passed, use the default self.dis_exp if dis_exp is None: dis_exp = self.dis_exp # Update the object attributes and record the current distance and angle self.dis2target = dis2target_now self.angle2target = direction_error # If the angular error is less than 60 degrees, consider the error negligible and set it to 0 if abs(direction_error)<60: direction_error = 0 else: # The part exceeding 60 degrees is normalized to the interval [0, 1] proportionally direction_error = (abs(direction_error) - 60) / 60 # Calculate the error between the expected distance and the current distance e_dis = abs(dis_exp - dis2target_now) # Normalize the distance error (with 300 as the benchmark) e_dis_relative = e_dis / 300 # Relational reward, the closer to the correct direction and the smaller the distance error, the higher the reward self.tracker_relation_reward = - direction_error - e_dis_relative # Calculate the final reward, deduct the collision penalty reward = self.tracker_relation_reward - (self.collision_punishment if is_collision else 0) # If very close to the target and the angle error is small, add extra reward if dis2target_now <= 60 and abs(self.angle2target) < 50: reward += 10 # Update the object attribute to record the reward self.r_tracker = reward return reward def reward_target(self, dis2target_now, is_collision=False, explor_reward=0): """ Second, the reward function corresponding to the second type of agent: Parameters: - dis2target_now: The current distance to the target - is_collision: Whether a collision occurs - explor_reward: Exploration reward Returns: - reward: Comprehensive reward value """ # Basic reward formula: tracker relationship reward + collision penalty + exploration reward reward = - self.tracker_relation_reward \ - (self.collision_punishment if is_collision else 0) \ + explor_reward # (Optional) Limit the lower bound of the reward # reward = max(reward, -1) # Record the reward self.r_target = reward return reward According to the multi-agent interaction method provided by the embodiments of the present application, a reward function is designed based on the relative distance information and mutual perspective information between agents, the collision situation between the agent and obstacles in the scene, and the exploration situation of the agent in the scene, so that on the basis that the distance between the agent and other types of agents reaches the target condition, the agent can further avoid obstacles in the scene and enhance the perception ability of the environment, realizing multi-agent pursuit in a complex scene and further improving the interaction effect.

[0080] In some embodiments, the agent is trained based on a reward function with the goal of the distance between the agent and other types of agents reaching the target condition, and may include: Set the training target and the corresponding scene for the training stage accordingly; In the scene corresponding to the current training stage, train the reward function with the training target corresponding to the current training stage, and use the trained reward function obtained in the current training stage as the initial reward function for the next training stage.

[0081] In this embodiment, multiple training stages can also be set to train the agent in stages. For example, multiple training stages are set from simple to complex according to the complexity of the scene or function. Set the training target and scene in the corresponding stage, and conduct staged training on the agent. By means of step by step, it helps the agent to master the task skills from simple to complex.

[0082] Taking the training of the first type of agent as an example, in the first training stage, the scenario is set as a relatively simple environment scenario, such as a single first type of agent facing a stationary or randomly moving escape object. Taking the first type of agent tracking the escape object and moving in the direction of the escape object as the training goal, focusing on learning basic movements and tracking, such as learning actions like running and jumping, to train the reward function.

[0083] In the second training stage, the number of agents is increased in the scenario, and chasing is introduced. The training goal is set to minimize the distance between the first type of agent and the escape object, focusing on learning chasing to adjust the reward function trained in the first stage.

[0084] In the third training stage, obstacles and complex maps can be further added to the scenario, such as randomly adding different obstacles. Taking obstacle avoidance and optimal path learning as the training goal, training the path planning strategy to adjust the reward function obtained in the second stage, so that the agent learns actions such as jumping, squatting, and running to cope with obstacles in complex scenarios.

[0085] In the fourth training stage, the second type of agent can be selected as the escape object. For example, the second type of agent is set as an intelligent escape object that can generate target control instructions based on the collected image information for movement to increase the distance from the first type of agent. The training goal is set to minimize the distance between the first type of agent and the second type of agent, strengthening the game awareness and adaptive ability, and further optimizing the reward function obtained in the previous stage of training.

[0086] The training process for the second type of agent is similar to that of the first agent. Only the training goal needs to be set to maximize the distance between the first type of agent and the second type of agent, which is not elaborated in this application.

[0087] In this application, by setting reasonable promotion criteria at each stage, the agent can grow continuously in an environment with gradually increasing challenges, and finally master complex multi-agent chasing strategies. Dynamically changing the ratio of distance and angle in the reward function during the training process can increase the model training speed and generalization ability.

[0088] According to the multi-agent interaction method provided by the embodiments of this application, through the design of a multi-stage distributed RL training process, the training difficulty is gradually increased, and the reward function is dynamically adjusted during the training process, which can accelerate the convergence speed of agent training, thereby improving the training speed and generalization ability of the agent, making it more suitable for multi-agent chasing interactions in complex environments.

[0089] In some embodiments, the agent is trained based on a reward function with the goal of the distance between the agent and other types of agents reaching the target condition, and may include: In the first alternating training phase, with other types of agents moving based on preset control instructions, the objective is to make the distance between the agent and other types of agents reach the target condition, train the reward function, and obtain the predictive control instruction corresponding to the current training phase. In the non-first alternating training phase, with other types of agents moving based on the predictive control instructions obtained from the previous phase of training, the objective is to make the distance between the agent and other types of agents reach the target condition, and train the reward function.

[0090] In this embodiment, a multi-stage alternating training process can also be set up, such as performing alternating training on the first type of agent and the second type of agent to dynamically adjust the reward function.

[0091] The preset control instruction can be an initial action execution sequence or a fixed policy defined by the user. In some embodiments, the preset control instruction can also be the control instruction obtained after independently training the agent, such as the control instruction generated for the agent after the completion of the first training phase or the third training phase. The present application does not limit this here.

[0092] It should be noted that in the non-first alternating training phase, the predictive control instructions obtained by other types of agents from the previous phase of training may be updated correspondingly as the training phase is updated. By making the distance between the agent and other types of agents reach the target condition with other types of agents moving based on the predictive control instructions obtained from the previous phase of training, and training the reward function, the control instruction of the current agent can be adjusted correspondingly according to the change of the control instruction of other types of agents, so as to gradually approach the optimal strategy.

[0093] In the first alternating training phase, since there has been no alternating training between different types of agents before, the movement of the second type of agent can be controlled according to the control instruction obtained after independently training the second type of agent. In this scenario, the first type of agent is trained with the objective of minimizing the distance between the first type of agent and the second type of agent.

[0094] Then enter the second alternating training phase. The movement of the first type of agent is controlled by the control instruction generated by the trained first type of agent. In this scenario, the second type of agent is trained with the objective of maximizing the distance between the second type of agent and the first type of agent.

[0095] Then enter the third alternating training phase. The movement of the second type of agent is controlled by the control instruction generated by the second type of agent trained in the previous training phase. In this scenario, the first type of agent is trained with the objective of minimizing the distance between the first type of agent and the second type of agent, and so on in a cycle until convergence.

[0096] In other embodiments, it is also possible to first control the movement of the first type of agent according to the control instructions obtained after independently training the first type of agent, and in this scenario, train the second type of agent with the goal of maximizing the distance between the second type of agent and the first type of agent.

[0097] In other embodiments, it is also possible to first control the movement of other types of agents according to the control instructions obtained after independently training the other types of agents, and in this scenario, train the current type of agent with the goal that the distance between the current type of agent and the other types of agents meets the target conditions, such as: Training Phase 1 (Chaser Training), the chaser explores the environment and chases the escapee. The escapee acts according to a fixed strategy (such as moving randomly or moving according to a certain rule), calculates the reward score of the chaser, and factors such as distance, perspective, and whether it is captured can be considered.

[0098] Training Phase 2 (Escapee Training), the escapee explores the environment and tries to avoid being chased. The chaser acts according to a fixed strategy (still chasing). Calculate the reward score of the escapee, and factors such as distance, angle, and survival time can be considered.

[0099] Then, enter the alternating training phase, alternately train in multiple training cycles, and alternately update the strategies of the chaser and the escapee to adapt to the strategy changes of the other party; the strategy of each agent will be adjusted according to the behavior of the other party, so as to gradually approach the optimal strategy.

[0100] According to the multi-agent interaction method provided by the embodiments of the present application, by setting the alternating training method for two types of agents, the agents can continuously adapt and optimize their respective behaviors when the strategies of other types of agents change, increasing the generality of the algorithm and further improving the interaction effect, and having universality.

[0101] Such as Figure 2 As shown, in some embodiments, the following basic system architecture can be adopted to implement the above method.

[0102] Environment layer (Environment), used to define the simulation space, map boundaries, obstacles, physical rules (such as speed limits, collision detection), and provide state observation, action execution, and reward feedback interfaces.

[0103] Agent layer (Agents), including chasers (Chasers) and runners (Runners). Among them, the goal of the chaser is to capture the runner. The goal of the runner is to avoid the chaser. Each agent has independent perception, decision-making, and action modules.

[0104] The training control layer conducts multi-stage alternating training to train the escape and pursuit strategies of the two types of agents, accelerating the model convergence speed and increasing the model generalization.

[0105] In the multi-agent interaction method provided by the embodiments of the present application, the execution subject may be a multi-agent interaction device. In the embodiments of the present application, taking the multi-agent interaction device executing the multi-agent interaction method as an example, the multi-agent interaction device provided by the embodiments of the present application is described.

[0106] The embodiments of the present application also provide a multi-agent interaction device.

[0107] As Figure 3 shown, the multi-agent interaction device includes: a first processing module 310, a second processing module 320, and a third processing module 330.

[0108] The first processing module 310 is configured to enable each agent to collect corresponding image information, where the image information includes relative distance information and mutual viewing angle information between the agent and other types of agents; each agent is in the same scene; The second processing module 320 is configured to enable the agent to generate a target control instruction based on the image information; The third processing module 330 is configured to enable the agent to move based on the target control instruction; where the agent is trained based on a reward function with the goal of the distance between the agent and other types of agents reaching a target condition. In the case where the agent is a first type of agent, the target condition is distance minimization; in the case where the agent is a second type of agent, the target condition is distance maximization.

[0109] According to the multi-agent interaction device provided by the embodiments of the present application, a reward function is designed based on the relative distance information and mutual viewing angle information between agents, and the agent is trained based on the reward function with the goal of the distance between the agent and other types of agents reaching a target condition. Thus, in the case where there are multiple agents in the environment, each agent can respectively generate corresponding target control instructions for movement based on the collected image information, so as to achieve the pursuit of multiple agents in a complex scene, thereby improving the interaction effect between multiple agents.

[0110] In some embodiments, the device may further include a fourth processing module, configured to: Obtain the sample image information collected by the agent at the current acquisition moment; Input the sample image information, the expected distance, and the expected relative viewing angle into the agent, and obtain the actual distance and actual relative viewing angle of the agent from other types of agents when moving according to the predicted predicted control instruction; Train the reward function based on the difference degree between the actual distance and the expected distance, and the difference degree between the actual relative viewing angle and the expected relative viewing angle.

[0111] In some embodiments, the fourth processing module may further be configured to: Construct a collision penalty function based on the collision situation between the agent and the obstacles in the scene when the agent moves according to the predicted predictive control instruction; Update the reward function based on the collision penalty function.

[0112] In some embodiments, the fourth processing module may further be configured to: Obtain the environmental perception information recognized by the agent based on the sample image information; Construct an exploration reward function based on the environmental perception information; Update the reward function based on the exploration reward function and the collision penalty function.

[0113] In some embodiments, the fourth processing module may further be configured to: In the first alternating training stage, when other types of agents move based on the preset control instruction, train the reward function with the goal that the distance between the agent and other types of agents reaches the target condition, and obtain the predictive control instruction corresponding to the current training stage; In the non-first alternating training stage, when other types of agents move based on the predictive control instruction obtained from the previous stage training, train the reward function with the goal that the distance between the agent and other types of agents reaches the target condition.

[0114] In some embodiments, the fourth processing module may further be configured to: Correspondingly set the training objective and the scene corresponding to the training stage according to the training stage; Within the scene corresponding to the current training stage, train the reward function with the training objective corresponding to the current training stage, and use the trained reward function obtained in the current training stage as the initial reward function for the next training stage.

[0115] In some embodiments, the second processing module 320 may further be configured to: The agent generates a target control instruction for indicating the moving direction and / or motion posture of the agent based on the image information.

[0116] The multi-agent interaction device in the embodiments of the present application can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices other than terminals. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a handheld computer, a vehicle-mounted electronic device, a Mobile Internet Device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc. It can also be a server, a Network Attached Storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, etc. The embodiments of the present application do not make specific limitations.

[0117] The multi-agent interaction device in the embodiments of the present application can be a device with an operating system. The operating system can be the Android operating system, the IOS operating system, or other possible operating systems. The embodiments of the present application do not make specific limitations.

[0118] The multi-agent interaction device provided by the embodiments of the present application can implement Figure 1 and Figure 2 each process implemented by the method embodiments. To avoid repetition, it will not be elaborated here.

[0119] The embodiments of the present application also provide an agent.

[0120] The agent can be an entity object, such as an embodied agent or a robot, etc.; or, the agent can also be a virtual object, such as a virtual character or a virtual animal in a virtual scene.

[0121] The agent can be a first-type agent or a second-type agent. The agent can move based on the multi-agent interaction method described in any of the above embodiments. This application will not elaborate here.

[0122] According to the agent provided by the embodiments of the present application, a reward function is designed based on the relative distance information and mutual perspective information between agents. Aiming at the distance between the agent and other types of agents reaching the target condition, the agent is trained based on the reward function. Thus, in the case where there are multiple agents in the environment, each agent can respectively generate corresponding target control instructions for movement according to the collected image information to achieve the pursuit of multiple agents in a complex scenario, thereby improving the interaction effect between multiple agents.

[0123] In some embodiments, as Figure 4 shown, the embodiments of the present application also provide an electronic device 400, including a processor 401, a memory 402, and a computer program stored on the memory 402 and executable on the processor 401. When the program is executed by the processor 401, it implements each process of the above-mentioned embodiment of the multi-agent interaction method and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0124] It should be noted that the electronic device in the embodiments of the present application includes the above-mentioned mobile electronic device and non-mobile electronic device.

[0125] The embodiments of the present application also provide a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements each process of the above-mentioned embodiment of the multi-agent interaction method and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0126] Wherein, the processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory ROM, random access memory RAM, magnetic disk or optical disc, etc.

[0127] The embodiments of the present application also provide a computer program product, including a computer program, which implements the above-mentioned multi-agent interaction method when executed by a processor.

[0128] Wherein, the processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory ROM, random access memory RAM, magnetic disk or optical disc, etc.

[0129] The embodiments of the present application further provide a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement each process of the above-mentioned embodiment of the multi-agent interaction method and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0130] It should be understood that the chip mentioned in the embodiments of the present application may also be referred to as a system-on-chip, system chip, chip system, or system-on-chip, etc.

[0131] It should be noted that in this document, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including that element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the reverse order according to the functions involved. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, the features described with reference to certain examples may be combined in other examples.

[0132] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-described embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions for causing a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present application.

[0133] The embodiments of the present application have been described above in conjunction with the accompanying drawings. However, the present application is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present application, those of ordinary skill in the art can also make many forms without departing from the purpose of the present application and the scope protected by the claims, and all of them belong to the protection scope of the present application.

[0134] In the description of this specification, the description referring to terms such as "one embodiment", "some embodiments", "schematic embodiments", "examples", "specific examples", or "some examples", etc. means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0135] Although the embodiments of the present application have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and purposes of the present application, and the scope of the present application is defined by the claims and their equivalents.

Claims

1. A multi-agent interaction method, characterized in that: include: Each agent collects corresponding image information, wherein the image information includes relative distance information and mutual viewing angle information between the agent and other agents of the same type; Each of the intelligent agents is in the same scene; The intelligent agent generates a target control instruction based on the image information; The intelligent agent moves based on the target control instruction; wherein, The intelligent agent is trained based on a reward function with the goal of achieving a target condition for the distance between the intelligent agent and other types of intelligent agents. When the intelligent agent is a first type of intelligent agent, the target condition is minimization of the distance; When the agent is a second type of agent, the target condition is distance maximization.

2. The multi-agent interaction method according to claim 1, characterized in that: The reward function is trained by the following steps: Obtaining sample image information collected by the agent at the current collection moment; Inputting the sample image information, expected distance and expected relative viewing angle into the agent, and obtaining the actual distance and actual relative viewing angle of the agent from other agents when the agent moves according to the predicted control instruction; The reward function is trained based on the difference between the actual distance and the expected distance, and the difference between the actual relative viewing angle and the expected relative viewing angle.

3. The multi-agent interaction method according to claim 2, characterized in that: The training of the reward function based on the difference between the actual distance and the expected distance, and the difference between the actual relative viewing angle and the expected relative viewing angle, comprises: Constructing a collision penalty function based on the collision between the agent and obstacles in the scene when the agent moves according to the predicted control command obtained by prediction; Based on the collision penalty function, the reward function is updated.

4. The multi-agent interaction method according to claim 3, characterized in that: The updating of the reward function based on the collision penalty function comprises: Acquire environmental perception information recognized by the agent based on the sample image information; Based on the environmental perception information, construct an exploration reward function; The reward function is updated based on the exploration reward function and the collision penalty function.

5. The multi-agent interaction method according to any one of claims 1 to 4, characterized in that: The intelligent agent is trained based on a reward function with the goal of achieving a target condition for the distance between the intelligent agent and other intelligent agents, including: In the first alternating training phase, when the other type of intelligent agent moves based on the preset control instruction, the reward function is trained with the goal of the distance between the intelligent agent and the other type of intelligent agent reaching the target condition, and the prediction control instruction corresponding to the current training phase is obtained; In the non-first alternating training stage, when the other type of intelligent agent moves based on the predicted control instruction obtained in the previous stage of training, the reward function is trained with the goal of achieving the target condition for the distance between the intelligent agent and the other type of intelligent agent.

6. The multi-agent interaction method according to any one of claims 1 to 4, characterized in that: The intelligent agent is trained based on a reward function with the goal of achieving a target condition for the distance between the intelligent agent and other intelligent agents, including: According to the training stage, the training target and the scene corresponding to the training stage are set accordingly; In the scene corresponding to the current training stage, the reward function is trained with the training target corresponding to the current training stage, and the trained reward function obtained in the current training stage is used as the initial reward function of the next training stage.

7. The multi-agent interaction method according to any one of claims 1 to 4, characterized in that: The intelligent agent generates a target control instruction based on the image information, including: The intelligent agent generates the target control instruction for indicating the moving direction and / or motion posture of the intelligent agent based on the image information.

8. A multi-agent interactive device, characterized in that: include: A first processing module is used to enable each agent to collect corresponding image information, wherein the image information includes relative distance information and mutual viewing angle information between the agent and other agents of the same type; Each of the intelligent agents is in the same scene; A second processing module, configured to enable the agent to generate a target control instruction based on the image information; The third processing module is used to make the intelligent agent move based on the target control instruction; wherein, The intelligent agent is trained based on a reward function with the goal of achieving a target condition for the distance between the intelligent agent and other types of intelligent agents. When the intelligent agent is a first type of intelligent agent, the target condition is minimization of the distance; When the agent is a second type of agent, the target condition is distance maximization.

9. An intelligent agent, characterized in that: The agent moves based on the multi-agent interaction method as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multi-agent interaction method as described in any one of claims 1 to 7 is implemented.

Images