A multi-robot navigation reinforcement learning method based on buffer voronoi cell

Abstract

The application discloses a kind of multi-robot navigation reinforcement learning methods based on buffer Voronoi unit, it is related to multi-agent navigation technical field.Based on the current position information of robot, the corresponding Voronoi unit is constructed, and each boundary of Voronoi unit is translated inwardly by the distance of the physical radius of robot, to generate buffer Voronoi unit;The distance between the current navigation point and each adjacent robot navigation point is calculated;If there is distance less than the physical radius of robot, then the current navigation point is offset angle in counterclockwise direction along the boundary of Voronoi unit, to generate new navigation point;New navigation point is projected on the boundary of buffer Voronoi unit, to obtain the current target navigation point;According to the self-observation information and external observation information of robot, determine the observation space, generate the action of robot based on the observation space and the trained strategy network, and determine the linear velocity and angular velocity of robot in the next iteration round based on action.The method improves the success rate of robot obstacle avoidance.

Description

Technical Field

[0001] This application relates to the field of multi-agent navigation technology, and in particular to a multi-robot navigation reinforcement learning method based on buffered Voronoi units. Background Technology

[0002] In obstacle avoidance and path planning of multi-robot systems, the Buffered Voronoi Cell (BVC) is an important method for safe area partitioning. Traditional Voronoi diagrams assign non-overlapping safe areas to each robot. To address sensor noise and localization uncertainty in multi-robot systems, existing technologies have proposed methods such as Probabilistic-BVC and B-UAVC, and have also explored heuristic right-hand rules or novel deadlock recovery mechanisms to handle passage congestion problems in obstacle-dense environments. Robot obstacle avoidance is a core issue in multi-robot systems.

[0003] Deep Reinforcement Learning (DRL) is widely used in the design of obstacle avoidance strategies for multi-robots. Some methods directly map raw sensor data into control commands in an end-to-end manner, while many other DRL strategies rely on environmental models to improve generalization and computational efficiency. However, since neural networks require a fixed input dimension, many methods assume a constant number of obstacles in the environment, limiting their applicability in real-world dynamic scenarios. Existing techniques utilize Long Short-Term Memory (LSTM) networks to handle a variable number of neighboring robots, or introduce attention mechanisms through Synthetic Aperture Radar Learning (SARL) to assess the importance of surrounding agents. However, existing methods either assign excessive weight to the tail of the input sequence, lack sufficient modeling for bidirectional collision avoidance, and focus less on the autonomous decision-making optimization of individual robots, resulting in low obstacle avoidance success rates. Summary of the Invention

[0004] Therefore, it is necessary to provide a multi-robot navigation reinforcement learning method based on buffered Voronoi units to address the aforementioned technical problems. This method improves the obstacle avoidance success rate of robots.

[0005] The following technical solution is adopted in this specification: This specification provides a multi-robot navigation reinforcement learning method based on buffered Voronoi units, including: For each iteration of the robot navigation process, for each robot, the current position information of the robot is obtained, the corresponding Voronoi unit is constructed based on the current position information of the robot, and each boundary of the Voronoi unit is translated inward by the distance of the robot's physical radius to generate a buffered Voronoi unit. Select the current navigation point within the robot's Voronoi unit, obtain multiple neighboring robot navigation points within the perception range, and calculate the distance between the current navigation point and each neighboring robot navigation point; If a distance is less than the robot's physical radius, a potential deadlock risk is identified, and the current navigation point is shifted counterclockwise along the Voronoi cell boundary by an angle. To generate new navigation points; The new navigation point is projected onto the boundary of the buffer Voronoi cell to obtain the current target navigation point; the projection operation is used to ensure that the robot is within the buffer Voronoi cell as it moves toward the current target navigation point. Based on the coordinates of the current target navigation point, external observation information is determined. The robot's self-observation information and external observation information are combined to form the observation space. This observation space is then input into a trained policy network to generate the robot's actions. Based on these actions, the robot's linear velocity and angular velocity at the next moment are determined. Self-observation information includes the robot's current velocity, orientation, desired velocity, and physical radius. External observation information includes the first... The buffer Voronoi unit region generated by the neighboring robot, the first The navigation points of the neighboring robots and the first The distance between adjacent robots or obstacles and the relative positional relationship between the robot's current position and the target position.

[0006] Optionally, the weight coefficients in the policy network loss function are dynamically adjusted according to the environment density and task requirements; the loss function of the policy network is: ; in, The loss value of the policy network. This represents the speed difference between the robot's current speed and its desired speed. For target navigation point With the Nearby robot navigation points The Euclidean distance between them For the preset safety radius, The robot's current speed and direction relative to its target navigation point. g∗ The angle between the directions, This is the absolute difference between the robot's current speed amplitude and its desired speed amplitude. These are all weighting coefficients, used to adjust the contribution ratio of each item in the loss function. This indicates that among all neighboring robots, the distance to the navigation point is greater than the safe radius and there is at least one neighboring robot. This indicates that the distance to the navigation point is less than the safe radius. dn Target navigation points generated for the robot With the Nearby robot navigation points The Euclidean distance between them.

[0007] Optionally, the observation space is determined based on the robot's self-observation information and external observation information, specifically including: The first GRU subnetwork, employing bidirectional gated recurrent units, processes external observation information along the positive direction to generate the first hidden state. ; The second GRU subnetwork, employing bidirectional gated recurrent units, processes external observation information in the reverse direction to generate the second hidden state. To capture the temporal dependencies and spatial contextual relationships between neighboring robots; First hidden state Second hidden state Concatenate the vectors to generate a fixed-dimensional vector. ; Fixed-dimensional vector With self-observation information Combined, a fixed-length integrated observation vector is generated; The integrated observation vector is normalized to generate a BVC spatial state representation; The BVC spatial state representation is defined as the observation space.

[0008] Optionally, the self-observation information is of fixed length; the external observation information is of variable length.

[0009] Optionally, angle The calculation formula is: ; in, For offset angle, , The distance between the first neighboring robot navigation point and the current navigation point. For the first m The distance between the current navigation point and the nearest robot navigation point For the robot's physical radius, max To obtain the maximum value, min To obtain the minimum value.

[0010] Optionally, the Voronoi element is the set of all points whose Euclidean distance to the robot's current position is less than their Euclidean distance to any other robot; the formula for the Voronoi element is: ; in, It is a Voronoi unit. Let be a geometric point in space, used to determine whether the geometric point belongs to a robot's Voronoi region. To represent two-dimensional Euclidean space, For the first i A robot, For the first j A robot, where || represents the Euclidean norm.

[0011] Optionally, the linear velocity and angular velocity of the robot in the next iteration are determined based on the action, specifically including: Obtain the robot's actions; the actions include two-dimensional velocity increments; the two-dimensional velocity increments are: ; in, For two-dimensional velocity increments, for x velocity increment in direction, for y Increment of velocity in the direction; Calculate the robot's velocity at the next moment based on the two-dimensional velocity increment; the robot's velocity at the next moment: ; in, For robots t The velocity vector at time +1 For robots t The velocity vector at time t, For two-dimensional velocity increments; Based on the velocity amplitude constraint, the robot's velocity vector at the next moment is decomposed into linear velocity and angular velocity.

[0012] Optionally, the linear velocity is: ; in, for t+ Linear velocity at time 1 for t+ The velocity vector at time 1, where || represents the Euclidean norm. The robot's current orientation and velocity vector The angle between them; Angular velocity is: ; in, for t+1 angular velocity at time t, This is the preset rotation adjustment time constant.

[0013] Optionally, the policy network adopts a two-layer fully connected network structure.

[0014] This specification provides a multi-robot navigation reinforcement learning device based on buffered Voronoi units, including: The generation module is used to obtain the current position information of each robot in each iteration of the robot navigation process, construct the corresponding Voronoi unit based on the current position information of the robot, and translate each boundary of the Voronoi unit inward by the distance of the robot's physical radius to generate a buffered Voronoi unit. The calculation module is used to select the current navigation point within the robot's Voronoi unit, obtain multiple neighboring robot navigation points within the perception range, and calculate the distance between the current navigation point and each neighboring robot navigation point. The judgment module is used to determine the potential deadlock risk if there is a distance smaller than the robot's physical radius, and then offset the current navigation point counterclockwise along the Voronoi cell boundary by an angle. To generate new navigation points; The projection module is used to project the new navigation point onto the boundary of the buffer Voronoi cell to obtain the current target navigation point; the projection operation is used to ensure that the robot is within the buffer Voronoi cell during its movement toward the current target navigation point. The determination module is used to determine external observation information based on the coordinates of the current target navigation point. It defines the robot's self-observation information and external observation information as the observation space, inputs this space into the trained policy network to generate the robot's actions, and determines the robot's linear velocity and angular velocity at the next moment based on the actions. The self-observation information includes the robot's current velocity, orientation, desired velocity, and physical radius; the external observation information includes the first... The buffer Voronoi unit region generated by the neighboring robot, the first The navigation points of the neighboring robots and the first The distance between adjacent robots or obstacles and the relative positional relationship between the robot's current position and the target position.

[0015] This specification provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described multi-robot navigation reinforcement learning method based on buffered Voronoi units.

[0016] This specification provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described multi-robot navigation reinforcement learning method based on buffered Voronoi units.

[0017] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects: For each iteration of the robot navigation process, for each robot, the current position information of the robot is obtained, and a corresponding Voronoi cell is constructed based on the current position information. Each boundary of the Voronoi cell is then translated inward by the robot's physical radius to generate a buffer Voronoi cell. Within the robot's Voronoi cell, the current navigation point is selected, and multiple neighboring robot navigation points within the perception range are obtained. The distance between the current navigation point and each neighboring robot navigation point is calculated. If any distance is less than the robot's physical radius, a potential deadlock risk is identified, and the current navigation point is shifted counterclockwise along the Voronoi cell boundary by an angle. The method generates a new navigation point; projects this new navigation point onto the boundary of a buffer Voronoi unit to obtain the current target navigation point; the projection operation ensures that the robot remains within the buffer Voronoi unit during its movement towards the current target navigation point; the observation space is determined based on the robot's self-observation information and external observation information; the robot's actions are generated based on the observation space and the trained policy network; and the linear and angular velocities of the robot in the next iteration are determined based on the actions. External observation information enhances the robot's ability to perceive changes in the external environment. The dual-layer observation input method not only enhances feature representation capabilities but also flexibly handles different numbers of neighboring robots, improving environmental adaptability. This method utilizes buffer Voronoi units as safety boundaries and combines them with a policy network to optimize control commands, significantly improving obstacle avoidance success rates in complex dynamic scenarios. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 This document provides a flowchart illustrating a multi-robot navigation reinforcement learning method based on buffered Voronoi units. Figure 2The following is a schematic diagram of Voronoi cells and buffer Voronoi cells provided by the present invention, taking five robots as an example. (a) is a schematic diagram of Voronoi cells, and (b) is a schematic diagram of buffer Voronoi cells. Figure 3 The present invention provides a schematic diagram of generating navigation points for four ring-shaped robots A, B, C, and D, wherein (a) is a schematic diagram of the robot's current position and global target point, and (b) is a schematic diagram of the intersection of the robot's current navigation points and the desired direction of movement. Figure 4 This is an example diagram illustrating the correction of navigation points in an embodiment of the present invention; Figure 5 A schematic diagram of the navigation strategy and evaluation neural network architecture based on BiGRUs provided by this invention; Figure 6 A flowchart of a multi-robot distributed navigation method based on buffered Voronoi units provided by the present invention; Figure 7 This specification provides a schematic diagram of a multi-robot navigation reinforcement learning device based on buffered Voronoi units; Figure 8 This specification provides a schematic diagram of a computer device for implementing a multi-robot navigation reinforcement learning method based on buffered Voronoi units. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in this specification without creative effort are within the scope of protection of this application.

[0021] Devices such as desktop computers, servers, and laptops are capable of executing the solutions described in this manual. For ease of explanation, the following description will focus on servers as the primary execution method.

[0022] Existing reward functions are mostly designed based on target distance or collision time, which can easily lead to a trade-off between traffic efficiency and safety. For example, while RL-RVO improves obstacle avoidance success rate, it significantly reduces traffic efficiency due to excessive risk avoidance.

[0023] This invention proposes a multi-robot navigation reinforcement learning method based on buffered Voronoi units. Addressing the issues of deadlock in dense environments or sacrificing efficiency for safety in existing distributed multi-robot obstacle avoidance methods, this method proposes a distributed navigation strategy integrating buffered Voronoi units and reinforcement learning. This method constructs a BVC region using only robot position information and uses it as the observation input for reinforcement learning. A policy network based on bidirectional recurrent modules maps the current state to navigation actions, dynamically determining the robot's running speed. The reward function of the policy network is designed based on the distance between the BVC region and the target navigation point to predict and reduce collision risk in advance. Experiments show that this invention has significant advantages in success rate, average speed, and travel time, and is applicable to various complex dynamic scenarios.

[0024] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0025] Figure 1 This is a flowchart illustrating a multi-robot navigation reinforcement learning method based on buffered Voronoi units, as described in this specification. The method includes the following steps: S101: For each iteration of the robot navigation process, for each robot, obtain the robot's current position information, construct the corresponding Voronoi unit based on the robot's current position information, and translate each boundary of the Voronoi unit inward by the distance of the robot's physical radius to generate a buffered Voronoi unit.

[0026] In one exemplary embodiment, the Voronoi cell is the set of all points whose Euclidean distance to the robot's current position is less than the Euclidean distance to any other robot.

[0027] Specifically, the current position information of each robot in the environment is obtained, and Voronoi units and buffered Voronoi units are constructed based on the position information. Assume that the environment contains... n A robot Current location as a set of seed points .

[0028] For each robot Building robots Voronoi unit Defined as all to The Euclidean distance is less than that to any other arbitrary The set of points at a distance, the robot Voronoi unit For formula (1): (1); in, It is a Voronoi unit. These are geometric points in space, used to determine whether a geometric point belongs to a robot's Voronoi region. For two-dimensional Euclidean space, For the first i A robot, For the first j A robot, where || represents the Euclidean norm.

[0029] Set the safety radius based on the robot's physical radius. Voronoi unit Each boundary is shifted inward by a distance Buffered Voronoi Cells (BVCs) are generated to serve as robots. The safe operating area.

[0030] Figure 2 This is a schematic diagram of Voronoi cells and buffer Voronoi cells, using five robots as an example, provided by the present invention. A, B, C, D, and E represent the five robots. Figure 2 Figure (a) in the figure is a schematic diagram of Voronoi cells. Figure 2 Figure (b) in the diagram is a schematic diagram of the buffer Voronoi cells.

[0031] S102: Select the current navigation point within the robot's Voronoi unit, obtain multiple neighboring robot navigation points within the perception range, and calculate the distance between the current navigation point and each neighboring robot navigation point.

[0032] The current navigation point is the intersection of the target direction and the BVC boundary.

[0033] For the current robot, in its Voronoi unit Select the current navigation point Acquire within the sensing range Navigation points of neighboring robots Calculate the current navigation point Distance between each neighboring navigation point .

[0034] S103: If there is a distance smaller than the robot's physical radius, a potential deadlock risk is identified, and the current navigation point is offset counterclockwise along the Voronoi cell boundary by an angle. To generate new navigation points.

[0035] If there exists any If this is not the case, a potential deadlock risk is identified, and the navigation point needs to be offset. The offset angle is calculated counterclockwise along the Voronoi cell boundary. Generate new navigation points The offset is determined by the minimum distance. Decide.

[0036] In one exemplary embodiment, the offset angle The calculation formula is formula (2): (2); in, For offset angle, , The distance between the first neighboring robot navigation point and the current navigation point. For the first m The distance between the current navigation point and the nearest robot navigation point For the robot's physical radius, max To obtain the maximum value, min To obtain the minimum value.

[0037] Figure 3 This invention provides a schematic diagram of four ring-shaped robots A, B, C, and D generating navigation points. (a) shows the robot's current position and global target point; (b) shows the intersection of the robot's current navigation points and the desired direction of movement. Figure 3 As shown in Figure (a), A, B, C, and D represent four mobile robots (Agent A, B, C, and D), respectively, with the black solid dots indicating the current position of the mobile robots; g_a, g_b, g_c, and g_d are the global target points of robots A, B, C, and D, respectively, represented by small orange dots; different colored areas represent the Voronoi units corresponding to each robot, that is, the distance from any point in this area to the corresponding robot is less than or equal to the distance to other robots; Figure 3 In Figure (b), the red asterisk indicates the intersection of all the robot's current navigation points, and the black arrow indicates the robot's current velocity direction or desired motion direction.

[0038] Figure 4 This is an example diagram of correcting navigation points in an embodiment of the present invention, such as... Figure 4As shown, A, B, C, and D represent four mobile robots, with their current positions indicated by solid gray dots; g_a, g_b, g_c, and g_d are the global goal points of robots A, B, C, and D, respectively, represented by small orange dots; g'_a, g'_b, g'_c, and g'_d represent the target navigation points reselected by each robot within the buffer Voronoi unit according to the collision avoidance strategy, indicated by red asterisks (*); different colored areas represent the Voronoi units corresponding to each robot, i.e., the distance from any point in this area to the corresponding robot is less than or equal to the distance to other robots; black arrows indicate the robot's current velocity direction or desired motion direction; dashed arrows point from the original goal point to the adjusted navigation point, indicating the path deviation caused by the robot's collision avoidance requirements; red asterisks indicate navigation points optimized by the BVC mechanism, located within their respective Voronoi units and far from the conflict area; the central intersection point (intersection of intersecting lines) represents the boundary intersection point of multiple robot Voronoi units, which is a potential collision risk area.

[0039] The sensing range is set according to specific engineering practices. For example, in one embodiment of the present invention, the sensing range is 5 meters.

[0040] S104: Project the new navigation point onto the boundary of the buffer Voronoi cell to obtain the current target navigation point; the projection operation is used to ensure that the robot is within the buffer Voronoi cell during its movement toward the current target navigation point.

[0041] New navigation point after offset correction Projecting onto the boundary of the buffered Voronoi cell yields the current target navigation point. This projection operation ensures that the robot remains inside the BVC while moving towards the target point, thus avoiding the risk of collisions caused by approaching the Voronoi boundary.

[0042] Projection operation with new navigation point Using the buffer Voronoi cell as the projection point, a vertical projection is made onto each boundary segment, resulting in multiple candidate projection points; from these candidate projection points, valid projection points located on the boundary segments of the buffer Voronoi cell are selected; the distance from the new navigation point is then calculated. The nearest valid projection point is determined as the current target navigation point.

[0043] S105: Determine external observation information based on the coordinates of the current target navigation point, define the robot's self-observation information and external observation information as the observation space, input the observation space into the trained policy network to generate the robot's actions, and determine the robot's linear velocity and angular velocity at the next moment based on the actions; self-observation information includes the robot's current velocity, orientation, desired velocity, and physical radius; external observation information includes the first... The buffer Voronoi unit region generated by the neighboring robot, the first The navigation points of the neighboring robots and the first The distance between adjacent robots or obstacles and the relative positional relationship between the robot's current position and the target position.

[0044] In an exemplary embodiment, the observation space is determined based on the robot's self-observation information and external observation information, specifically including: using a first GRU subnetwork with bidirectional gated recurrent units to process the external observation information in the positive direction to generate a first hidden state. The second GRU subnetwork, employing bidirectional gated cyclic units, processes external observation information in the reverse direction to generate the second hidden state. To capture the temporal dependencies and spatial contextual relationships between neighboring robots; the first hidden state Second hidden state Concatenate the vectors to generate a fixed-dimensional vector. ; fixed-dimensional vector With self-observation information By combining these methods, a fixed-length integrated observation vector is generated; the integrated observation vector is then normalized to generate a BVC spatial state representation; and this BVC spatial state representation is then used as the observation space.

[0045] Specifically, the geometric information of the buffer Voronoi unit (including boundary point coordinates, area, and distance to the target point) is encoded as the observation state of the reinforcement learning agent. The policy network adopts a bidirectional recurrent neural network structure based on a gated recurrent unit (GRU) to map the observation state into continuous control actions (such as linear velocity and angular velocity). The reward function is designed to be positively correlated with the distance from the buffer Voronoi unit to the target navigation point, so as to simultaneously optimize passage efficiency and obstacle avoidance safety.

[0046] The synthesized observation vectors, after normalization, are fed into two neural networks for further analysis. One network is the policy execution network (Actor), responsible for mapping spatial states to actions. The other is the policy evaluation network (Critic), used to evaluate the value of each state-action pair.

[0047] The policy execution network employs a two-layer fully connected network structure, which can effectively convert the spatial state of the input into a velocity mean. The actual actions taken by the robot are derived from... The velocity increments are obtained by random sampling from a Gaussian distribution centered on the network. This design helps to find a good balance between exploration and exploitation. Meanwhile, the policy evaluation network... Also composed of two fully connected layers, it aims to encode the spatial state and output the corresponding state-action value function to guide the parameter optimization process of the policy execution network.

[0048] In order to extract spatial features more accurately from buffered Voronoi cell (BVC) sequence data and guide the robot's motion decisions, this invention divides the observation information into a fixed-length self-observation part and a variable-length external observation part.

[0049] It is worth noting that this invention employs a bidirectional gated recurrent unit (Bi-GRU) module when processing external observation information. This module contains two GRU sub-networks that process the input sequence in the forward and reverse directions, respectively, thereby effectively capturing the temporal dependencies and spatial context relationships between neighboring robots.

[0050] After processing by the Bi-GRU module, two final hidden states are obtained. and The two are concatenated to form a vector of fixed dimensions. This achieves effective compression and unified representation of external observation information. Next, the above... With self-observation information Combined, a fixed-length integrated observation vector is formed. The BVC space state representation is then processed through a normalization layer and used as the input to the subsequent neural network.

[0051] Furthermore, the neural network architecture proposed in this invention has good scalability and can adapt to different numbers of neighboring robots without pre-setting the environmental topology, thus making it suitable for distributed multi-robot navigation tasks in complex dynamic scenarios.

[0052] In one exemplary embodiment, the self-observation information is of fixed length; the external observation information is of variable length.

[0053] Specifically, the reinforcement learning framework is used to construct policy neural networks to establish a mapping relationship between robot observation data and its control actions; the framework consists of five core components: reward function, policy execution network (Actor), policy evaluation network (Critic), robot state and action space.

[0054] The observation space is divided into two parts: self-observation information and external observation information; self-observation information ,ori Including the robot's current speed Orientation towards ori, desired speed and physical radius External observation information , including the first within the perception range The buffer Voronoi unit region generated by the neighboring robots Its navigation points and the distance to the neighbor or obstacle. ).

[0055] Action space is defined as two-dimensional velocity increments The robot's speed will be updated in the next moment. Furthermore, the velocity amplitude is limited to a certain range. To facilitate motion control, the velocity at the next moment is decomposed into linear velocity. With angular velocity .

[0056] In an exemplary embodiment, the weight coefficients in the policy network loss function are dynamically adjusted according to the environmental density and task requirements; the loss function of the policy network is formula (3): (3); in, The loss value of the policy network. This represents the speed difference between the robot's current speed and its desired speed. For target navigation point With the Nearby robot navigation points The Euclidean distance between them For the preset safety radius, Let g be the angle between the robot's current velocity direction and the direction pointing to its target navigation point g*. This is the absolute difference between the robot's current speed amplitude and its desired speed amplitude. These are all weighting coefficients, which are dynamically adjusted based on environmental density and task requirements to regulate the contribution ratio of each element in the loss function. This indicates that among all neighboring robots, the distance to the navigation point is greater than the safe radius and there is at least one neighboring robot. This indicates that the distance to the navigation point is less than the safe radius. dn Target navigation points generated for the robot With the Nearby robot navigation points The Euclidean distance between them.

[0057] Specifically, the reward function is designed to guide the robot to gradually learn the sub-tasks required to achieve the final navigation goal, and is particularly suitable for path planning and obstacle avoidance problems in long-term or complex environments. Traditional distance-based reward designs are difficult to match with the characteristics of the Buffered Voronoi Cell (BVC) method. Therefore, this invention proposes a novel multi-condition reward function to enhance the robot's obstacle avoidance and navigation capabilities within the BVC region.

[0058] If the distance of all neighboring robots ,but If it exists And angle difference ,but If it exists And angle difference ,but .in, Indicates the current speed With expected speed The differences between them; Navigation points generated for the current robot With the Nearby robot navigation points The distance between them; For the safety radius; when all When this occurs, encourage the robot to move in the target direction; if there is... Then the navigation point needs to be adjusted according to the angle difference.

[0059] Angle difference in the reward function Defined as current speed With expected speed The cosine similarity between them, i.e. cosine The closer its value is to 1, the more consistent the directions of the two directions are; when When the system prioritizes increasing the speed to improve traffic flow efficiency; when When the direction deviation is large, it indicates that the orientation needs to be corrected first.

[0060] velocity amplitude difference Defined as the difference between the magnitude of the current velocity and the desired velocity, i.e. This is used to measure the degree of matching in terms of speed magnitude; when A large value indicates that the robot's speed is too fast or too slow, and adjustments need to be made to maintain stable movement.

[0061] Through the above multi-condition reward mechanism, the system can automatically configure behavioral strategies in different scenarios: when there is no risk of collision, the robot is encouraged to accelerate forward; when there is a potential collision and the directions are inconsistent, a penalty is imposed to prompt the robot to slow down and avoid the collision; when there is a risk of collision but the directions are consistent, the robot is encouraged to fine-tune its speed while maintaining its direction to pass safely.

[0062] Weighting coefficients in the reward function The parameters can be dynamically adjusted according to environmental density and task requirements; in the experimental setup of this invention, the following parameter values ​​are used: For example, in high-density environments, this can be achieved by increasing... The weights of relevant terms are used to strengthen directional consistency constraints, thereby improving collision avoidance capabilities and ensuring the safety and stability of the system in complex scenarios.

[0063] In an exemplary embodiment, determining the robot's linear velocity and angular velocity for the next iteration based on the action specifically includes: acquiring the robot's action; the action being a two-dimensional velocity increment as shown in formula (4): (4); in, For two-dimensional velocity increments, for x velocity increment in direction, for y The velocity increment in the direction.

[0064] Based on the two-dimensional velocity increment, the robot's velocity at the next moment is calculated; the robot's velocity at the next moment is given by formula (5): (5); in, For robots t The velocity vector at time +1 For robots t The velocity vector at time t, For two-dimensional velocity increments; Based on the velocity amplitude constraint, the robot's velocity vector at the next moment is decomposed into linear velocity and angular velocity.

[0065] In an exemplary embodiment, the linear velocity is given by formula (6): (6); in, for t linear velocity at time t, for t The velocity vector at time t, || represents the Euclidean norm. The robot's current orientation and velocity vector The angle between them.

[0066] Angular velocity is given by formula (7): (7); in, for t angular velocity at time t, This is the preset rotation adjustment time constant.

[0067] The policy execution network (Actor) generates actions based on the comprehensive observation vectors, while the policy evaluation network (Critic) evaluates the value of the action according to the reward function. Together, they form the Actor-Critic architecture, achieving policy iteration through collaborative optimization. This enables the robot to autonomously learn safe and efficient navigation behaviors in dynamic multi-robot environments. The core objective of the reinforcement learning framework is to enable the robot to efficiently reach the target location while avoiding collisions with neighboring robots or obstacles, all while satisfying the safety constraints of the buffered Voronoi unit and optimizing navigation point selection and velocity control.

[0068] In one exemplary embodiment, the policy network employs a two-layer fully connected network structure.

[0069] Figure 5 This is a schematic diagram of the navigation strategy and evaluation neural network architecture based on BiGRUs provided by the present invention, as shown below. Figure 5 As shown, O_sur represents the environmental perception observation data acquired at different time steps, and ⊕ represents the feature fusion operation, which is the observation at the current time ( ) and historical context information ( ) to be spliced ​​or weighted and merged. Represents the splicing result; BiGRU module: a bidirectional gated recurrent unit, used to process time-series observation sequences and capture long-term dependencies. The two parallel branches on the right: the upper branch: the policy network, outputting linear velocity instructions in the action space (…). This determines the robot's next movement behavior; the lower branch: evaluates the network and outputs the state-action value function. Q ( s,a or state value function V ( s The neural network architecture is used to evaluate the quality of the current policy and guide policy updates. It includes: an observation input layer (LayerNormalization, LN) that receives the temporal observation sequence generated by buffered Voronoi units; an O_sur bidirectional gated recurrent unit (BiGRU) module that encodes the historical and future contextual information of the observation sequence; a policy network branch that outputs the robot's linear velocity action based on the encoded hidden states; and an evaluation network branch that outputs the corresponding state-action value for policy optimization. This represents the fused observation sequence before it is input into the BiGRU module. This represents the value estimate at the initial time step. This represents the average value of all states within a single episode. Action represents an action.

[0070] Figure 6 The flowchart of the multi-robot distributed navigation method based on buffered Voronoi units provided by the present invention is as follows: Figure 6 As shown, the current position information of each robot in the environment is obtained; Voronoi units and buffered Voronoi units are constructed; an initial navigation point is selected within the Voronoi unit; the navigation point is pre-offset corrected based on the spatial positional relationship of the navigation points of neighboring robots; the corrected navigation point is projected onto the boundary of the buffered Voronoi unit to generate the final safe navigation point; the BVC geometric features are used as input to the reinforcement learning policy network; and distributed safe navigation is achieved by outputting control commands through the Actor-Critic framework.

[0071] The beneficial effects of this invention are as follows: (1) A multi-robot distributed navigation method based on buffered Voronoi units (BVC) and reinforcement learning framework is proposed. By constructing a safety constraint region and a navigation point prediction mechanism, the method effectively solves the problems of deadlock, low passage efficiency and insufficient obstacle avoidance safety in dense environments. This method uses BVC as a safety boundary and combines it with a policy network to optimize control instructions, which significantly improves the obstacle avoidance capability and passage efficiency of the system in complex dynamic scenarios.

[0072] (2) A comprehensive observation space combining self-observation information and external observation information was designed. The self-observation part includes key parameters such as current speed, orientation, expected speed, and robot radius, while the external observation includes information such as the navigation point positions and distances of neighboring robots. This dual-layer observation input method not only enhances the feature representation capability but also flexibly responds to different numbers of neighboring robots, improving the system's adaptability and robustness.

[0073] (3) The Bi-Gated Recurrent Unit (Bi-GRU) module is used to process external observation information, which realizes the effective encoding of variable-length neighbor information. The fixed-dimensional vector output by the Bi-GRU module is concatenated with the self-observation information to form a standardized spatial state representation, which further improves the model's ability to perceive changes in the external environment and the accuracy of decision-making;

[0074] (4) A novel multi-condition reward function is introduced, which dynamically adjusts the reward value according to the current environmental state, guiding the robot to gradually learn the sub-tasks required to achieve the final navigation goal. This reward function finely controls the robot's behavior through parameters such as angle difference and velocity amplitude difference, ensuring that it can efficiently complete the path planning task while avoiding collisions. In addition, the weight coefficients can be flexibly adjusted according to specific application scenarios, further enhancing the system's versatility and practicality;

[0075] (5) Compared with the prior art, the multi-robot distributed navigation method of the present invention achieves efficient adaptation and precise control to complex dynamic environments by integrating BVC constraints and reinforcement learning framework. This method not only reduces the dependence on large-scale labeled data, but also significantly improves cross-scene performance with a reduced number of parameters, providing a reliable solution for edge real-time analysis of multi-robot systems in practical applications.

[0076] The above are one or more embodiments of the multi-robot navigation reinforcement learning method based on buffered Voronoi units provided in this specification. Based on the same idea, this specification also provides a corresponding multi-robot navigation reinforcement learning device based on buffered Voronoi units, such as... Figure 7 As shown.

[0077] Figure 7 A schematic diagram of a multi-robot navigation reinforcement learning device based on buffered Voronoi units provided in this specification includes: The generation module 701 is used to obtain the current position information of the robot for each iteration in the robot navigation process, construct the corresponding Voronoi unit based on the current position information of the robot, and translate each boundary of the Voronoi unit inward by the distance of the robot's physical radius to generate a buffered Voronoi unit.

[0078] The calculation module 702 is used to select the current navigation point within the robot's Voronoi unit, obtain multiple neighboring robot navigation points within the perception range, and calculate the distance between the current navigation point and each neighboring robot navigation point.

[0079] The judgment module 703 is used to determine the potential deadlock risk if there is a distance smaller than the robot's physical radius, and to offset the current navigation point counterclockwise along the Voronoi cell boundary by an angle. To generate new navigation points.

[0080] Projection module 704 is used to project the new navigation point onto the boundary of the buffer Voronoi cell to obtain the current target navigation point; the projection operation is used to ensure that the robot is within the buffer Voronoi cell during its movement toward the current target navigation point.

[0081] The determination module 705 is used to determine external observation information based on the coordinates of the current target navigation point, define the robot's self-observation information and external observation information as the observation space, input the observation space into the trained policy network to generate the robot's actions, and determine the robot's linear velocity and angular velocity at the next moment based on the actions; the self-observation information includes the robot's current velocity, orientation, desired velocity, and physical radius; the external observation information includes the first... The buffer Voronoi unit region generated by the neighboring robot, the first The navigation points of the neighboring robots and the first The distance between adjacent robots or obstacles and the relative positional relationship between the robot's current position and the target position.

[0082] Specific limitations regarding the buffered Voronoi unit-based multi-robot navigation reinforcement learning device can be found in the above description of the limitations of the buffered Voronoi unit-based multi-robot navigation reinforcement learning method, and will not be repeated here. Each module in the aforementioned buffered Voronoi unit-based multi-robot navigation reinforcement learning device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0083] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 A reinforcement learning method for multi-robot navigation based on buffered Voronoi units is provided.

[0084] This instruction manual also provides Figure 8 The schematic diagram of the computer device shown is as follows: Figure 8 At the hardware level, the computer device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to achieve the above-mentioned functions. Figure 1 A reinforcement learning method for multi-robot navigation based on buffered Voronoi units is provided.

[0085] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0086] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A multi-robot navigation reinforcement learning method based on buffered Voronoi units, characterized in that, The method includes the following: (The environment contains multiple robots.) For any given moment during the robot navigation process, for each robot, the current position information of the robot is obtained, the corresponding Voronoi unit is constructed based on the current position information of the robot, and each boundary of the Voronoi unit is translated inward by the distance of the robot's physical radius to generate a buffered Voronoi unit; Select the current navigation point within the robot's Voronoi unit, obtain multiple neighboring robot navigation points within the perception range, and calculate the distance between the current navigation point and each neighboring robot navigation point; If there is a distance smaller than the robot's physical radius, then the current navigation point is offset counterclockwise along the Voronoi cell boundary by an angle. To generate new navigation points; The new navigation point is projected onto the boundary of the buffer Voronoi cell to obtain the current target navigation point; the projection operation is used to ensure that the robot is within the buffer Voronoi cell during its movement toward the current target navigation point. Based on the coordinates of the current target navigation point, external observation information is determined. The robot's self-observation information and external observation information are combined to form the observation space. This observation space is then input into a trained policy network to generate the robot's actions. Based on these actions, the robot's linear velocity and angular velocity at the next moment are determined. The self-observation information includes the robot's current velocity, orientation, desired velocity, and physical radius. The external observation information includes the first... The buffer Voronoi unit region generated by the neighboring robot, the first The navigation points of the neighboring robots and the first The distance between adjacent robots or obstacles and the relative positional relationship between the robot's current position and the target position.

2. The multi-robot navigation reinforcement learning method based on buffered Voronoi units as described in claim 1, characterized in that, The loss function of the policy network is: ； in, The loss value of the policy network. This represents the speed difference between the robot's current speed and its desired speed. For target navigation point With the Nearby robot navigation points The Euclidean distance between them For the preset safety radius, The robot's current speed and direction relative to its target navigation point. g∗ The angle between the directions, This is the absolute difference between the robot's current speed amplitude and its desired speed amplitude. These are all weighting coefficients, which are dynamically adjusted based on environmental density and task requirements. This indicates that among all neighboring robots, the distance to the navigation point is greater than the safe radius and there is at least one neighboring robot. This indicates that the distance to the navigation point is less than the safe radius. dn Target navigation points generated for the robot With the Nearby robot navigation points The Euclidean distance between them.

3. The multi-robot navigation reinforcement learning method based on buffered Voronoi units as described in claim 1, characterized in that, The process of defining the robot's self-observation information and external observation information as the observation space specifically includes: The first GRU subnetwork is used to process external observation information in the forward direction to generate the first hidden state. ; A second GRU subnetwork is used to process external observation information in the reverse direction to generate a second hidden state. To capture the temporal dependencies and spatial contextual relationships between neighboring robots; the first GRU subnetwork and the second GRU subnetwork constitute a bidirectional gated loop unit; The first hidden state and the second hidden state Concatenate the vectors to generate a fixed-dimensional vector. ; Fixed-dimensional vector With self-observation information Combined, a fixed-length integrated observation vector is generated; The comprehensive observation vector is normalized to generate a BVC spatial state representation; The BVC spatial state representation is defined as the observation space.

4. The multi-robot navigation reinforcement learning method based on buffered Voronoi units as described in claim 1, characterized in that, The angle The calculation formula is: ； in, For angle, , The distance between the first neighboring robot navigation point and the current navigation point. For the first m The distance between the current navigation point and the nearest robot navigation point For the robot's physical radius, max To obtain the maximum value, min To obtain the minimum value.

5. The multi-robot navigation reinforcement learning method based on buffered Voronoi units as described in claim 1, characterized in that, The Voronoi unit is the set of all points whose Euclidean distance to the robot's current position is less than their Euclidean distance to any other robot; the formula corresponding to the Voronoi unit is: ； in, It is a Voronoi unit. Let be a geometric point in space, used to determine whether the geometric point belongs to a robot's Voronoi region. To represent two-dimensional Euclidean space, For the first i A robot, For the first j A robot, where |||| represents the Euclidean norm.

6. The multi-robot navigation reinforcement learning method based on buffered Voronoi units as described in claim 1, characterized in that, The action includes a two-dimensional velocity increment; the two-dimensional velocity increment includes x velocity increment in direction and y Increment of velocity in the direction; The determination of the robot's linear velocity and angular velocity at the next moment based on the action specifically includes: Based on the aforementioned two-dimensional velocity increment, the robot's velocity vector at the next moment is calculated; the robot's velocity vector at the next moment is: ； in, For robots t The velocity vector at time +1 For robots t The velocity vector at time t, For two-dimensional velocity increments; Based on the velocity amplitude constraint, the robot's velocity vector at the next moment is decomposed into linear velocity and angular velocity.

7. The multi-robot navigation reinforcement learning method based on buffered Voronoi units as described in claim 6, characterized in that, The linear velocity is: ； in, for t+ Linear velocity at time 1 for t+ The velocity vector at time 1, where || represents the Euclidean norm. The robot's current orientation and velocity vector The angle between them; The angular velocity is: ； in, for t+ Angular velocity at time 1 This is the preset rotation adjustment time constant.

8. The multi-robot navigation reinforcement learning method based on buffered Voronoi units as described in claim 1, characterized in that, The policy network adopts a two-layer fully connected network structure.

Images