Multi-robot game simulation method, simulation platform, electronic device and storage medium

Abstract

The present invention provides a multi-robot game simulation method, a simulation platform, an electronic device and a storage medium, belonging to the field of artificial intelligence technology. The method includes running at least one environment instance, each environment instance is provided with at least one own robot and at least one opponent robot; in the current training round, the training strategy shared by each environment instance and the opponent strategy corresponding to each opponent robot are determined, the first control instruction corresponding to the training strategy is sent to the own robot and the second control instruction corresponding to the opponent strategy is sent to the opponent robot, so that the own robot and the opponent robot perform the game task in the environment instance. The present invention configures an opponent robot for the own robot in each environment instance, improves the diversity of confrontation training experience, realizes full exploration of the environment instance, and the opponent robot in each environment instance plays a confrontation game with the own robot sharing the strategy according to the opponent strategy, which is helpful to formulate a more robust and universal strategy.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multi-robot game simulation method, a simulation platform, an electronic device and a storage medium. Background Art

[0002] In recent years, multi-agent games have become an important research direction in the field of artificial intelligence, providing a controllable and reproducible environment for conducting various experiments. By simulating the confrontation and collaboration of multiple agents in a dynamic environment, researchers can better understand the behavior of complex systems and promote the development of intelligent decision-making systems.

[0003] Existing multi-agent game simulation platforms usually generate game strategies through reinforcement learning. Reinforcement learning usually only considers the interaction between the agent and the environment, and the generated strategies have certain limitations. Summary of the invention

[0004] The present invention provides a multi-robot game simulation method, a simulation platform, an electronic device and a storage medium, which solve the problem that a multi-agent game simulation platform generates game strategies through reinforcement learning. Reinforcement learning usually only considers the interaction between the agent and the environment, and the generated strategy has certain limitations.

[0005] The present invention provides a multi-robot game simulation method, comprising:

[0006] Running at least one environment instance, each of which is provided with at least one own robot and at least one opponent robot;

[0007] In the current training round, the training strategy shared by each of the environment instances and the opponent strategy corresponding to each of the opponent robots are determined, and the first control instruction corresponding to the training strategy is sent to the own robot and the second control instruction corresponding to the opponent strategy is sent to the opponent robot, so that the own robot and the opponent robot perform the game task in the environment instance.

[0008] As an embodiment, it also includes: obtaining the state space, action space and game winning rate of the robot in each of the environment instances after executing the game task process, updating the training strategy according to the state space and the action space, judging whether to update the opponent strategy according to the game winning rate, and entering the next training round based on the updated training strategy and the opponent strategy / updated opponent strategy until the preset training stop condition is reached.

[0009] As an embodiment, the determining of the training strategy shared by each of the environment instances and the opponent strategy corresponding to each of the opponent robots includes:

[0010] Acquire the current confrontation situation of the own robot in each of the environment instances, determine the current action space of the own robot in each of the environment instances according to the current confrontation situation, and use the current action space as the target point of the own robot;

[0011] Inputting the target point into a preset path planner to obtain a planned path for the robot to move to the target point, and obtaining the training strategy according to the target point and the planned path;

[0012] A strategy is extracted from a preset strategy pool based on a self-game algorithm, and the extracted strategy is used as the opponent strategy.

[0013] As an embodiment, the current confrontation situation includes a current state space. Correspondingly, determining the current action space of the own robot in each of the environment instances according to the current confrontation situation includes:

[0014] The current state space corresponding to each of the environment instances is input into a preset reinforcement learning network to obtain the target point of the own robot in each of the environment instances.

[0015] As an embodiment, judging whether to update the opponent strategy according to the game winning rate, and entering the next training round based on the updated training strategy and the opponent strategy / updated opponent strategy until a preset training stop condition is reached, includes:

[0016] If the game winning rate is not greater than a preset threshold, enter the next training round based on the updated training strategy and the opponent's strategy until a preset training stop condition is reached;

[0017] If the game winning rate is greater than a preset threshold, the training strategy is added to the strategy pool; based on the self-game algorithm, strategies are extracted from the strategy pool for each opponent robot, the extracted strategies are used as updated opponent strategies, and the next training round is entered based on the updated training strategy and the updated opponent strategy until the preset training stop condition is reached.

[0018] As an embodiment, the opponent strategy in each of the environment instances is different.

[0019] As an embodiment, the own robot and the opponent robot both include a chassis, a gimbal, wheels arranged on the chassis, and a laser radar, a first joint for driving the wheels to rotate is arranged between the chassis and the wheels, and a second joint for driving the gimbal to rotate is arranged between the chassis and the gimbal. Correspondingly, before running at least one environment instance, each of which is provided with at least one own robot and at least one opponent robot, the following is further included:

[0020] Each wheel of the own robot or the opponent robot is modeled separately to obtain a refined model of the own robot or the opponent robot.

[0021] The present invention also provides a multi-robot game simulation platform, comprising:

[0022] An operation module, used for operating at least one environment instance, each of which is provided with at least one own robot and at least one opponent robot;

[0023] A training module is used to determine, in a current training round, a training strategy shared by each of the environment instances and an opponent strategy corresponding to each of the opponent robots, send a first control instruction corresponding to the training strategy to the own robot and send a second control instruction corresponding to the opponent strategy to the opponent robot, so that the own robot and the opponent robot perform a game task in the environment instance.

[0024] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the multi-robot game simulation method described above is implemented.

[0025] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the multi-robot game simulation method as described in any one of the above is implemented.

[0026] The multi-robot game simulation method, simulation platform, electronic device and storage medium provided by the present invention configure opponent robots for own robots in each environment instance by running environment instances in parallel, thereby improving the diversity of adversarial training experience and achieving full exploration of environment instances. The opponent robots in each environment instance all play adversarial games with their own robots that share strategies according to the opponent's strategies, which helps to formulate more robust and universal strategies. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0028] Figure 1 It is a flow chart of the multi-robot game simulation method provided by the present invention.

[0029] Figure 2 It is a schematic diagram of the parallel operation of multiple environment instances provided by the present invention.

[0030] Figure 3 is a schematic diagram of any environment example provided by the present invention.

[0031] Figure 4 It is a grid map schematic diagram of any environment example provided by the present invention.

[0032] Figure 5 It is a training flow diagram of the training strategy provided by the present invention.

[0033] Figure 6 It is a structural schematic diagram of the multi-robot game simulation platform provided by the present invention.

[0034] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention.

[0035] Figure 8 It is a schematic diagram of the algorithm code of the parallel self-game reinforcement learning algorithm provided by the present invention. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] It should be noted that all actions of acquiring signals, information or data in the present invention are performed in compliance with the corresponding data protection laws and policies of the location and with the authorization given by the owner of the corresponding device.

[0038] The simulation platform provides a controllable and reproducible environment for conducting various experiments, which helps researchers to develop and evaluate various methods effectively, efficiently and safely. In recent years, the development of agent confrontation game simulation platforms has mainly focused on some simple game tasks, such as Go, poker, and some arcade games. Multi-agent games usually involve not only a huge search space, but also require complex decision-making capabilities such as cooperation and confrontation between multiple agents, incomplete information reasoning, and long-term planning. These characteristics make this type of environment an ideal platform for studying deep reinforcement learning and multi-agent cooperation and confrontation. In order to further study the self-robot game problem in practical applications, related research has also begun to adopt With the development of cooperative-competitive game platforms, some simulators that support physics engines (such as Gazebo, Mujoco, Unity, etc.) are used to enhance realism and narrow the reality gap, and on this basis, some simple self-robot game confrontation tasks are completed.

[0039] The application of reinforcement learning in the field of self-robotics has achieved many successes, such as legged self-robotics movement, robotic arm grasping, self-robotics navigation, etc. In reinforcement learning, the agent observes the environment and performs actions, and the environment gives rewards based on the agent's behavior. Reinforcement learning updates the behavior strategy from the feedback of the environment by controlling the interaction between the agent and the environment to maximize the long-term reward. Reinforcement learning does not rely on pre-labeled training data, but the amount of data required to train the strategy using reinforcement learning will also increase with the increase of task complexity. In recent years, multi-agent games have become an important research direction in the field of artificial intelligence, and its typical applications include intelligent transportation, military simulation, financial market simulation, etc. By simulating the confrontation and cooperation of multiple agents in a dynamic environment, researchers can better understand the behavior of complex systems and promote the development of intelligent decision-making systems.

[0040] Most existing simulation platforms do not support multi-environment parallelism and only perform training in a single environment, resulting in insufficient exploration.

[0041] As the number of environments and the number of robots on one's side increases, existing simulation platforms are often unable to maintain real-time simulation, or their performance drops sharply in large-scale scenarios. For example, when real-time confrontations involve dozens or even hundreds of intelligent agents, the load on the physics engine and decision-making algorithm is too high, resulting in a significant decrease in simulation speed, which affects the efficiency of strategy learning.

[0042] In addition, most current multi-agent game platforms do not introduce physical characteristics to the environment and agents, and cannot accurately simulate complex physical phenomena in the real world. Friction, collision, sensor feedback, etc. in real environments are often simplified in existing platforms. Although some environments use physical engines to enhance their authenticity, the kinematic models of the robots in these environments are generally simplified to a certain extent, resulting in the inability of the agents' behaviors to fully correspond to the actual scenes, thus affecting the simulation effect and the practical applicability of the research results.

[0043] To this end, the present invention provides a multi-robot game simulation method, simulation platform, electronic device and storage medium, which at least solve the defect of insufficient exploration degree of the existing multi-agent game platform, and on this basis, further make up for the deficiencies in physical modeling accuracy, strategy learning efficiency and other aspects. The present invention is described in detail below with reference to the accompanying drawings.

[0044] Figure 1 : is a flow chart of the multi-robot game simulation method provided by the present invention, such as Figure 1 As shown, the present invention provides a multi-robot game simulation method, including steps S100 to S200.

[0045] Step S100, running at least one environment instance, each of which is provided with at least one own robot and at least one opponent robot. All environment instances are run in parallel, and the number of own robots and the number of opponent robots may correspond one to one. Specifically, the agents in each environment instance may be evenly divided into two groups, one group being the own robots and the other group being the opponent robots. The environment instances may be the same or different, and the present invention is described by taking different environment instances as examples.

[0046] Step S200, in the current training round, determine the training strategy shared by each of the environment instances and the opponent strategy corresponding to each of the opponent robots, send the first control instruction corresponding to the training strategy to the own robot and send the second control instruction corresponding to the opponent strategy to the opponent robot, so that the own robot and the opponent robot perform the game task in the environment instance. The training strategy is used to characterize the strategy of controlling the own robot to move in the environment instance and attack the opponent robot, and the opponent strategy is used to characterize the strategy of controlling the opponent robot to move in the environment instance and attack the own robot. The first control instruction and the second control instruction are speed vector instructions for driving the own robot and the opponent robot to move and attack, respectively.

[0047] Figure 2 is a schematic diagram of the parallel operation of multiple environment instances provided by the present invention, Figure 3 is a schematic diagram of any environment example provided by the present invention, such as Figure 2 and Figure 3 As shown, the present invention supports large-scale parallel reinforcement learning training, and multiple environment instances are run in parallel during training. Figure 2 and Figure 3 Each box in is an environment instance, and the friendly robot and the opponent robot play games to accumulate experience. The training strategy is shared by all environment instances, and the opponent strategy of the opponent robot in each environment instance can be different. The game task in the present invention is a confrontational game task. The friendly robot and the opponent robot fire projectiles at each other. The intelligent agent that is hit a certain number of times is disabled and loses the ability to move and shoot. The goal of both the friendly robot and the opponent robot is to disable the opponent's two friendly robots to win.

[0048] Figure 4 is a grid map schematic diagram of any environment example provided by the present invention, such as Figure 4 As shown, it is assumed that each environment instance contains four agents, which are divided into two groups for confrontation game. The environment instance can be a rectangular area, 8 meters long and 5 meters wide. Several obstacles are symmetrically set in the venue of the environment instance to block the movement and collision of the own robot. The obstacles make the entire confrontation task more challenging, and the own robot needs to plan a strategy to gain an advantage in the confrontation.

[0049] It can be understood that the present invention runs environment instances in parallel, configures opponent robots for its own robots in each environment instance, improves the diversity of adversarial training experience, and achieves full exploration of environment instances. The opponent robots in each environment instance play adversarial games with its own robots that share strategies based on the opponent's strategies, which helps to formulate more robust and universal strategies.

[0050] On the basis of the above embodiment, as an optional embodiment, the self-robot and the opponent robot both include a chassis, a gimbal, wheels and a laser radar arranged on the chassis, a first joint for driving the wheel to rotate is arranged between the chassis and the wheel, and a second joint for driving the gimbal to rotate is arranged between the chassis and the gimbal. Correspondingly, before running at least one environment instance, each of which is provided with at least one self-robot and at least one opponent robot, the following steps are also included.

[0051] Each wheel of the own robot or the opponent robot is modeled separately to obtain a refined model of the own robot or the opponent robot.

[0052] The present invention focuses on the physical realism of the robot, and performs accurate physical modeling on the robot. The individual modeling of each wheel of the own robot or the opponent robot specifically refers to the individual modeling of the roller of each wheel of the own robot or the opponent robot, and the omnidirectional movement is achieved through the friction between the roller and the ground. The wheel can be a Mecanum wheel. The rotation of the gimbal is subject to angular velocity restrictions, and the own robot or the opponent robot can only fire projectiles in the direction pointed by the gimbal.

[0053] Both the robot of the own side and the robot of the opponent have a shooting function, that is, they can fire projectiles at the enemy robot, and this function is realized by ray simulation. When firing a projectile, a ray is fired from the projectile firing port of the robot along the direction pointed by the pan-tilt, and the information of the first object contacted by the ray and the distance the ray has traveled are obtained. By judging the information of the hit object, it can be judged whether the firing hits the enemy robot. In the process of robot confrontation game, the robots of both sides are constantly moving, and the projectile firing direction is the same as the current direction of the pan-tilt, so it is necessary to control the pan-tilt to point to the enemy robot. For this reason, the present invention sets an automatic aiming system for the robot to effectively attack the enemy robot. The automatic aiming system takes into account the distance to the target and the obstacles in the field at the same time, and manipulates the robot to aim at the enemy robot that is not blocked by obstacles and is closest, and then rotates its pan-tilt according to the position of the locked target enemy robot for aiming, and the rotation of the pan-tilt is controlled by a PD controller (integral-differential controller). When the pan-tilt is consistent with the target direction, the projectile is fired.

[0054] It can be understood that the present invention performs accurate physical modeling of the robot's movement device, shooting device and perception device, so that the simulation environment is as consistent as possible with the robot in the actual system in terms of movement and perception, so as to improve the simulation authenticity.

[0055] Based on the above embodiment, as an optional embodiment, the determining of the training strategy shared by each of the environment instances and the opponent strategy corresponding to each of the opponent robots includes the following steps.

[0056] Step S210, obtaining the current confrontation situation of the own robot in each of the environment instances, determining the current action space of the own robot in each of the environment instances according to the current confrontation situation, and using the current action space as the target point of the own robot. The current confrontation situation is used to characterize the respective positions, health comparison, projectile data comparison and potential conflict environment information of the own robot and the opponent robot.

[0057] Optionally, as an embodiment, the current confrontation situation includes a current state space, which includes the robot's current position, current health and current number of projectiles. Correspondingly, determining the current action space of the own robot in each of the environmental instances based on the current confrontation situation includes: inputting the current state space corresponding to each of the environmental instances into a preset reinforcement learning network to obtain the target point of the own robot in each of the environmental instances.

[0058] Among them, the target point refers to the position point that can quickly defeat the opponent robot. For example, when the distance between the own robot and the opponent robot is far, the target point of the own robot is set around the same opponent robot according to the current confrontation situation, so as to achieve the goal of flanking the opponent robot and quickly kill it, thereby gaining an advantage.

[0059] It should be noted that the friendly robot and the opponent robot share the same state space and action space, and the state space / current state space may also include the execution time of the game task.

[0060] Step S220, input the target point into a preset path planner to obtain a planned path for the robot to move to the target point, and obtain the training strategy according to the target point and the planned path.

[0061] Step S230, extracting a strategy from a preset strategy pool based on a self-game algorithm, and using the extracted strategy as the opponent strategy. The order of step S210 and step S230 can be adjusted, and the present invention does not limit this.

[0062] Figure 5It is a training flow diagram of the training strategy provided by the present invention, such as Figure 5 As shown, the present invention provides a hierarchical strategy structure for robot confrontation game tasks, and the training strategy includes a high-level target point release strategy for controlling the own robot team and a low-level path planning strategy for controlling the own robot body. Step S210 is used to determine the high-level strategy, and step S220 is used to determine the low-level strategy. Among them, the game task can be regarded as a Markov process, and the high-level strategy is generated through reinforcement learning. The rewards used for reinforcement learning include three items, namely, the hit reward , Disability Reward and winning rewards In addition, the rewards of the opponent robot team are used as an additional penalty term for training the own robot team.

[0063] The training strategy and the opponent strategy are determined through steps S210-S230, and speed vector instructions are sent to the own robot and the opponent robot in the environment instance respectively, and then the environmental perception feedback fed back by the own robot and the opponent robot is obtained. After the own robot and the opponent robot move and attack according to the strategy respectively, a new confrontation situation is formed. The present invention calculates rewards according to the new confrontation situation, and inputs the calculated rewards into the reinforcement learning network to achieve strategy update. It should be noted that the frequency of sending speed vector instructions and obtaining environmental perception feedback is higher than the frequency of reward calculation for the new confrontation situation.

[0064] The self-playing algorithm is an algorithm that trains an intelligent agent through self-playing. It is often used in the fields of reinforcement learning and deep learning, especially in game problems. The basic idea of ​​self-playing is to let the intelligent agent play against itself without an external opponent, and continuously optimize its strategy and decision-making ability through this game process, and use the data generated by self-playing to update its strategy. After each game, the strategy will be adjusted according to the result of winning or losing to improve the performance in the future game.

[0065] It can be understood that the present invention simplifies the hierarchical strategy structure of robot adversarial game tasks, promotes continuous adaptation and improvement of training strategies through parallel self-games, and diversifies the opponent strategies of different opponent robots, which helps to formulate more robust and universal strategies.

[0066] On the basis of the above embodiment, as an optional embodiment, the multi-robot game simulation method provided by the present invention also includes the following steps.

[0067] Step S300, obtaining the state space, action space and game winning rate of the own robot in each of the environment instances after executing the game task process, updating the training strategy according to the state space and the action space, judging whether to update the opponent strategy according to the game winning rate, and entering the next training round based on the updated training strategy and the opponent strategy / updated opponent strategy until the preset training stop condition is reached.

[0068] As an embodiment, judging whether to update the opponent strategy according to the game winning rate, entering the next training round based on the updated training strategy and the opponent strategy / updated opponent strategy until a preset training stop condition is reached includes the following steps.

[0069] Step S310, if the game winning rate is not greater than a preset threshold, enter the next training round based on the updated training strategy and the opponent's strategy until a preset training stop condition is reached.

[0070] Step S320, if the game winning rate is greater than a preset threshold, the training strategy is added to the strategy pool; based on the self-game algorithm, strategies are extracted from the strategy pool for each opponent robot, the extracted strategies are used as updated opponent strategies, and the next training round is entered based on the updated training strategy and the updated opponent strategy until the preset training stop condition is reached.

[0071] During the training process, the present invention maintains a strategy pool, which stores historical versions of the training strategy, and the opponent strategies used in different environments are extracted from this strategy pool. The sampling method is determined by a specific self-game algorithm. At the end of a training round, the training strategy is updated using the experience collected from all environments, and then the average winning rate of the training strategy with opponents in all environments in the previous training round is calculated. If the winning rate exceeds a preset threshold, the current training strategy is added to the strategy pool and the opponent strategies in all environment instances are resampled.

[0072] The present invention records the algorithm of the training strategy of the robot as a parallel self-game reinforcement learning algorithm. The algorithm code is as follows: Figure 8 shown.

[0073] It can be understood that the present invention provides a group of dynamic opponent robots for the own robot, promoting the continuous adaptation and improvement of the training strategy through parallel self-game.

[0074] On the basis of the above embodiment, as an optional embodiment, in the existing technology, it is usually necessary to divide the simulation calculation into two parts: the GPU is responsible for simulation and graphics rendering, and the CPU is responsible for processing the physics engine, reward calculation and observation calculation. After that, these data will be transmitted back to the GPU for training or further action generation. The frequent data exchange between the GPU and the CPU is the main bottleneck. Unlike the prior art, the present invention can directly perform physical simulation on the GPU by using a GPU-accelerated simulation backend, and keep all training data such as observations, rewards and actions in the GPU, avoiding frequent data transmission, and directly obtaining simulation results on the GPU through the API supported by PyTorch tensors, realizing an end-to-end GPU accelerated training pipeline, effectively avoiding excessive load and improving training efficiency.

[0075] The multi-robot game simulation platform provided by the present invention is described below. The multi-robot game simulation platform described below and the multi-robot game simulation method described above can be referenced to each other.

[0076] Figure 6 Schematic diagram of the structure of the multi-robot game simulation platform provided by the present invention. Figure 6 As shown, the present invention also provides a multi-robot game simulation platform, including the following modules.

[0077] An operation module 610 is used to operate at least one environment instance, each of which is provided with at least one own robot and at least one opponent robot;

[0078] The training module 620 is used to determine the training strategy shared by each of the environment instances and the opponent strategy corresponding to each of the opponent robots in the current training round, send the first control instruction corresponding to the training strategy to the own robot and send the second control instruction corresponding to the opponent strategy to the opponent robot, so that the own robot and the opponent robot perform the game task in the environment instance.

[0079] As an embodiment, it also includes an update module, which is used to obtain the state space, action space and game winning rate of the robot in each of the environment instances after executing the game task process, update the training strategy according to the state space and the action space, determine whether to update the opponent strategy according to the game winning rate, and enter the next training round based on the updated training strategy and the opponent strategy / updated opponent strategy until the preset training stop condition is reached.

[0080] As an embodiment, the training module 620 is further used for:

[0081] Acquire the current confrontation situation of the own robot in each of the environment instances, determine the current action space of the own robot in each of the environment instances according to the current confrontation situation, and use the current action space as the target point of the own robot;

[0082] Inputting the target point into a preset path planner to obtain a planned path for the robot to move to the target point, and obtaining the training strategy according to the target point and the planned path;

[0083] A strategy is extracted from a preset strategy pool based on a self-game algorithm, and the extracted strategy is used as the opponent strategy.

[0084] As an embodiment, the current confrontation situation includes a current state space, and correspondingly, the training module 620 is further used for:

[0085] The current state space corresponding to each of the environment instances is input into a preset reinforcement learning network to obtain the target point of the own robot in each of the environment instances.

[0086] As an embodiment, the update module is further used for:

[0087] If the game winning rate is not greater than a preset threshold, enter the next training round based on the updated training strategy and the opponent's strategy until a preset training stop condition is reached;

[0088] If the game winning rate is greater than a preset threshold, the training strategy is added to the strategy pool; based on the self-game algorithm, strategies are extracted from the strategy pool for each opponent robot, the extracted strategies are used as updated opponent strategies, and the next training round is entered based on the updated training strategy and the updated opponent strategy until the preset training stop condition is reached.

[0089] As an embodiment, the opponent strategy in each of the environment instances is different.

[0090] As an embodiment, the own robot and the opponent robot both include a chassis, a gimbal, wheels arranged on the chassis and a laser radar, a first joint for driving the wheels to rotate is provided between the chassis and the wheels, and a second joint for driving the gimbal to rotate is provided between the chassis and the gimbal. Correspondingly, before running at least one environment instance, each of which is provided with at least one own robot and at least one opponent robot, a modeling module is also included, and the modeling module is used to individually model each of the wheels of the own robot or the opponent robot to obtain a refined model of the own robot or the opponent robot.

[0091] It should be noted that the multi-robot game simulation platform provided by the present invention can execute the multi-robot game simulation method described in any of the above embodiments during specific operation, and has the technical effects corresponding to the method, which will not be elaborated in this embodiment.

[0092] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the multi-robot game simulation method, the method comprising: running at least one environment instance, each of which is provided with at least one own robot and at least one opponent robot; in the current training round, determining the training strategy shared by each of the environment instances and the opponent strategy corresponding to each of the opponent robots, sending the first control instruction corresponding to the training strategy to the own robot and sending the second control instruction corresponding to the opponent strategy to the opponent robot, so that the own robot and the opponent robot perform the game task in the environment instance.

[0093] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0094] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multi-robot game simulation method provided by the above-mentioned methods, which includes: running at least one environment instance, each of which is provided with at least one own robot and at least one opponent robot; in the current training round, determining the training strategy shared by each of the environment instances and the opponent strategy corresponding to each of the opponent robots, sending the first control instruction corresponding to the training strategy to the own robot and sending the second control instruction corresponding to the opponent strategy to the opponent robot, so that the own robot and the opponent robot perform game tasks in the environment instance.

[0095] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the multi-robot game simulation method provided by the above-mentioned methods, the method comprising: running at least one environment instance, each of the environment instances being provided with at least one own robot and at least one opponent robot; in a current training round, determining a training strategy shared by each of the environment instances and an opponent strategy corresponding to each of the opponent robots, sending a first control instruction corresponding to the training strategy to the own robot and sending a second control instruction corresponding to the opponent strategy to the opponent robot, so that the own robot and the opponent robot perform game tasks in the environment instance.

[0096] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0097] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-robot game simulation method, characterized in that: Use the GPU-accelerated simulation backend to perform physical simulation directly on the GPU, leaving all the training data of observations, rewards, and actions on the GPU. Through the API supported by PyTorch tensors, the simulation results can be obtained directly on the GPU to achieve an end-to-end GPU-accelerated training pipeline, including: Running at least one environment instance, each of which is provided with at least one own robot and at least one opponent robot; In the current training round, the training strategy shared by each of the environment instances and the opponent strategy corresponding to each of the opponent robots are determined, and a first control instruction corresponding to the training strategy is sent to the own robot and a second control instruction corresponding to the opponent strategy is sent to the opponent robot, so that the own robot and the opponent robot perform the game task in the environment instance; the training strategy is used to characterize the strategy of controlling the own robot to move in the environment instance and attack the opponent robot, and the opponent strategy is used to characterize the strategy of controlling the opponent robot to move in the environment instance and attack the own robot; multiple environment instances are run in parallel during training, and the training strategy is shared by all environment instances, and the opponent strategy of the opponent robot in each environment instance is different.

2. The multi-robot game simulation method according to claim 1, characterized in that: Also includes: The state space, action space and game winning rate of the own robot in each of the environment instances after executing the game task process are obtained, the training strategy is updated according to the state space and the action space, and whether to update the opponent strategy is determined according to the game winning rate. The next training round is entered based on the updated training strategy and the opponent strategy / updated opponent strategy until a preset training stop condition is reached.

3. The multi-robot game simulation method according to claim 1, characterized in that: The determining of the training strategy shared by each of the environment instances and the opponent strategy corresponding to each of the opponent robots includes: Acquire the current confrontation situation of the own robot in each of the environment instances, determine the current action space of the own robot in each of the environment instances according to the current confrontation situation, and use the current action space as the target point of the own robot; Inputting the target point into a preset path planner to obtain a planned path for the robot to move to the target point, and obtaining the training strategy according to the target point and the planned path; A strategy is extracted from a preset strategy pool based on a self-game algorithm, and the extracted strategy is used as the opponent strategy.

4. The multi-robot game simulation method according to claim 3, characterized in that: The current confrontation situation includes a current state space. Correspondingly, determining the current action space of the own robot in each of the environment instances according to the current confrontation situation includes: The current state space corresponding to each of the environment instances is input into a preset reinforcement learning network to obtain the target point of the own robot in each of the environment instances.

5. The multi-robot game simulation method according to claim 2, characterized in that: The determining whether to update the opponent strategy according to the game winning rate, and entering the next training round based on the updated training strategy and the opponent strategy / updated opponent strategy until a preset training stop condition is reached, includes: If the game winning rate is not greater than a preset threshold, enter the next training round based on the updated training strategy and the opponent's strategy until a preset training stop condition is reached; If the game winning rate is greater than a preset threshold, the training strategy is added to the strategy pool; based on the self-game algorithm, strategies are extracted from the strategy pool for each opponent robot, the extracted strategies are used as updated opponent strategies, and the next training round is entered based on the updated training strategy and the updated opponent strategy until the preset training stop condition is reached.

6. The multi-robot game simulation method according to any one of claims 1 to 5, characterized in that: The opponent strategy in each of the environment instances is different.

7. The multi-robot game simulation method according to any one of claims 1 to 5, characterized in that: The own robot and the opponent robot each include a chassis, a pan-tilt platform, wheels arranged on the chassis, and a laser radar, a first joint for driving the wheels to rotate is arranged between the chassis and the wheels, and a second joint for driving the pan-tilt platform to rotate is arranged between the chassis and the pan-tilt platform. Correspondingly, before running at least one environment instance, each of which is provided with at least one own robot and at least one opponent robot, the following further includes: Each wheel of the own robot or the opponent robot is modeled separately to obtain a refined model of the own robot or the opponent robot.

8. A multi-robot game simulation platform, characterized in that: Use the GPU-accelerated simulation backend to perform physical simulation directly on the GPU, leaving all the training data of observations, rewards, and actions on the GPU. Through the API supported by PyTorch tensors, the simulation results can be obtained directly on the GPU to achieve an end-to-end GPU-accelerated training pipeline, including: An operation module, used for operating at least one environment instance, each of which is provided with at least one own robot and at least one opponent robot; A training module is used to determine, in the current training round, the training strategy shared by each of the environment instances and the opponent strategy corresponding to each of the opponent robots, send the first control instruction corresponding to the training strategy to the own robot and send the second control instruction corresponding to the opponent strategy to the opponent robot, so that the own robot and the opponent robot perform the game task in the environment instance; the training strategy is used to characterize the strategy of controlling the own robot to move in the environment instance and attack the opponent robot, and the opponent strategy is used to characterize the strategy of controlling the opponent robot to move in the environment instance and attack the own robot; multiple environment instances are run in parallel during training, the training strategy is shared by all environment instances, and the opponent strategy of the opponent robot in each environment instance is different.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the multi-robot game simulation method as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multi-robot game simulation method as claimed in any one of claims 1 to 7 is implemented.

Images