A method, apparatus and equipment for multi-robot patrolling

Abstract

This application discloses a multi-robot patrolling method, apparatus, and device, relating to the field of robot control technology. It determines target patrol actions by analyzing local observation data through a strategy network onboard the robot, achieving autonomous patrolling based on local observations without the need for pre-planning, making it suitable for long-term tasks. After executing the target patrol action, it constructs historical experience using single-step marginal entropy gain, and updates the strategy network and value network onboard the robot based on this historical experience and the global joint value obtained from a centralized hybrid network. Mathematically, this ensures that the greedy strategy can approach the global optimum, allowing it to adapt to environmental changes without pre-planning. It drives the robot to actively explore areas where entropy has increased due to long-term neglect, completely eliminating local oscillations and improving training stability.

Description

Technical Field

[0001] This application relates to the field of robot control technology, specifically to a method, apparatus, and equipment for multi-robot patrolling. Background Technology

[0002] The core task of multi-robot patrol is to enable multiple robots to perform long-term, continuous patrol tasks within a graph or mesh environment, thereby achieving area security monitoring. Long-term goals typically include node coverage, uniformity of access frequency, and unpredictability. Currently, the mainstream technical solutions in this field can be divided into three main categories:

[0003] Planning-based methods rely on classic coverage control, path planning, and scheduling algorithms. They typically take a global perspective, planning theoretically optimal or near-optimal patrol paths for the robot team. Planning methods are global and provable, with theoretically guaranteed performance once planning is complete. However, their drawbacks include high computational complexity and poor scalability. As the patrol environment topology (graph structure) grows larger or the patrol time span increases, the space and time computation required by planning algorithms increase exponentially. This makes them difficult to apply to large patrol environments and long-term tasks, lacking practical feasibility.

[0004] Heuristic-based methods guide each robot's behavior by designing simple local rules. Common heuristics include those based on neighborhood repulsion, information potential fields, and partitioned recycling. The robot makes real-time decisions based on its local perception information (such as the positions of neighboring robots). Heuristic methods are typically distributed, relatively simple to implement, and offer good real-time performance. However, their drawback is the lack of global guarantees, making them prone to getting trapped in local optima. Because decisions are based solely on local information, heuristic methods lack a global perspective, leading to inefficient behaviors such as chasing hotspots and oscillations within the robot team. In other words, robots may cluster towards recently visited areas while ignoring long-unvisited areas, failing to achieve truly uniform coverage.

[0005] Learning-based methods utilize machine learning techniques such as multi-agent reinforcement learning (MARL) and imitation learning to allow robots to autonomously learn patrol strategies through interaction with their environment. They can handle constraints related to partial observability and weak communication. The advantage of learning-based methods is their adaptability to complex and dynamic environments, independent of precise environmental models, and the ability to learn effective behaviors through data-driven approaches. However, their disadvantages include high sensitivity to reward design and unstable training. The performance of learning methods heavily depends on the design of the reward function, but in real-world patrol tasks, rewards are often sparse (only obtained upon visiting nodes), and it is difficult to distinguish the contribution of each robot, leading to highly unstable training and difficulty in converging to the ideal policy. Summary of the Invention

[0006] The purpose of this application is to provide a multi-robot patrol method, apparatus, and equipment, which solves the problems of existing technologies being unable to cope with long-term tasks, having local oscillations, ignoring long-term unvisited areas, and having difficulty distinguishing the contributions of each robot, resulting in a very unstable training process.

[0007] This application is achieved through the following technical solution:

[0008] The first aspect of this application provides a multi-robot patrol method, including:

[0009] For each patrolling robot, local observation data is acquired through the robot, and the local observation data is analyzed through the policy network on the robot itself to determine the target patrol action;

[0010] The robot is controlled to perform its corresponding target patrol action, and the single-step marginal entropy gain and local observation data at the next moment are obtained after the target patrol action is performed.

[0011] Based on the local observation data, target patrol actions, single-step marginal entropy gain, and the local observation data at the next moment, historical experience is constructed and placed into the historical experience pool.

[0012] The historical experience in the historical experience pool is analyzed by the local value network carried by each robot to obtain the local value corresponding to each robot. Based on the local value corresponding to each robot and the real-time global state corresponding to all robots, the global joint value is obtained through a centralized hybrid network.

[0013] Based on the global joint value, the local value network and centralized hybrid network carried by the robot are updated, and the updated local value network is used to obtain local value and the updated centralized hybrid network is used to obtain global joint value.

[0014] Based on the historical experience in the historical experience pool and the updated local value network, obtain the objective function value for limiting the update magnitude of the policy.

[0015] Based on the objective function value, the policy network onboard the robot is updated, and the updated network is used for the next robot patrol.

[0016] In one possible implementation, for each patrolling robot, local observation data is acquired by the robot, and the local observation data is analyzed through the policy network onboard the robot to determine the target patrol action, including:

[0017] For each patrolling robot, its own state information, topological neighborhood information, and communication domain teammate information are obtained to obtain the corresponding local observation data.

[0018] The robot analyzes the local observation data using its own onboard strategy network to determine the target patrol action; the target patrol action refers to the position node of the robot in the patrol environment topology map corresponding to the area to be patrolled at the next moment.

[0019] In one possible implementation, obtaining the single-step marginal entropy gain after performing the target patrol action includes:

[0020] For any location node in the patrol environment topology map corresponding to the area to be patrolled, obtain the first access probability corresponding to the location node, and obtain the first Shannon entropy corresponding to the first access probability.

[0021] Obtain the second access probability corresponding to the location node selected for the target patrol action, and obtain the second Shannon entropy corresponding to the second access probability;

[0022] Based on the first Shannon entropy and the second Shannon entropy, determine the single-step marginal entropy gain after performing the target patrol action.

[0023] In one possible implementation, the historical experience in the historical experience pool is analyzed through a local value network carried by each robot to obtain the local value corresponding to each robot, including:

[0024] For any robot, a joint trajectory of historical local observations and actions is generated using historical experience from the robot's corresponding historical experience pool.

[0025] The robot's local value is determined by analyzing the combined trajectory of historical local observations and actions using a local value network.

[0026] In one possible implementation, a global joint value is obtained through a centralized hybrid network based on the local value corresponding to each robot and the real-time global state corresponding to all robots, including:

[0027] Obtain the real-time location node set and global environment heat map of all robots to obtain the real-time global state of all robots;

[0028] The centralized hybrid network is used to analyze the local value of each robot and its real-time global state to determine the global joint value.

[0029] In one possible implementation, the local value network and centralized hybrid network carried by the robot are updated based on the global joint value, including:

[0030] For any robot, the timing difference error is obtained based on the historical experience in the robot's historical experience pool, and the dominance function value is obtained based on the timing difference error.

[0031] Based on the advantage function value and the global joint value in the previous update process, the estimated value of the cumulative marginal return of the discount is determined; wherein, in the first update process, the global joint value in the previous update process is set to zero;

[0032] The error value is obtained based on the estimated cumulative marginal return of the discount and the global joint value in the current update process;

[0033] Based on the error value, the network parameters of the local value network and the centralized hybrid network are updated through backpropagation.

[0034] In one possible implementation, for any given robot, a temporal difference error is obtained based on the historical experience in the robot's corresponding historical experience pool, and a dominance function value is obtained based on the temporal difference error, including:

[0035] For any given robot, based on the historical experience in the robot's corresponding historical experience pool, the temporal difference error is obtained as follows:

[0036] ;

[0037] In the formula, The timing difference error at time step t; The single-step marginal entropy gain is obtained based on the historical experience in the robot's corresponding historical experience pool, representing the value gained on the historical trajectory. Above, the robot performs target patrol actions and transfers location nodes. The entropy change that occurs afterward; As a discount factor, For the real-time global state at time step t Based on this, the global joint value output by the centralized hybrid network; For the real-time global state at the next time step Based on this, the global joint value output by the centralized hybrid network;

[0038] The dominance function value is obtained based on the time-series difference error:

[0039] ;

[0040] In the formula, The dominant function value, For smoothing parameters, The time difference error at time step t+k is the time index, and T is the total number of time steps.

[0041] Based on the advantage function value and the global joint value from the previous update, the estimated cumulative marginal return after discount is determined as follows:

[0042] ;

[0043] In the formula, This is the estimated cumulative marginal return after discounting. This refers to the global joint value from the previous update process.

[0044] In one possible implementation, based on the historical experience in the historical experience pool and the updated local value network, the objective function value for determining the magnitude of the policy update is:

[0045] ;

[0046] ;

[0047] In the formula, To limit the objective function value of the policy update magnitude, For the network parameters of the policy network, Where i is the batch size, i is the sample index, and t is the time step. The advantage function value is obtained based on historical experience in the historical experience pool and the updated local value network. This represents the probability ratio between the old and new strategies. This is a truncation function used to restrict... exist Within the range; To truncate parameters, Here is the entropy regularization coefficient. For policy entropy regularization, For the current policy network in local observation data Select action The probability, For the policy network in the last update process, based on local observation data Select action The probability of.

[0048] A second aspect of this application provides a multi-robot patrol device, comprising:

[0049] The action determination module is used to acquire local observation data of each patrolling robot and analyze the local observation data through the policy network on the robot itself to determine the target patrol action.

[0050] The data acquisition module is used to control the robot to perform its corresponding target patrol action, and to acquire the single-step marginal entropy gain after the target patrol action is performed and the local observation data at the next moment.

[0051] An experience storage module is used to construct historical experience based on the local observation data, target patrol actions, single-step marginal entropy gain, and local observation data at the next moment, and to put the historical experience into a historical experience pool.

[0052] The joint analysis module is used to analyze the historical experience in the historical experience pool through the local value network carried by each robot, to obtain the local value corresponding to each robot, and to obtain the global joint value through a centralized hybrid network based on the local value corresponding to each robot and the real-time global state corresponding to all robots.

[0053] The first update module is used to update the local value network and centralized hybrid network carried by the robot based on the global joint value, and subsequently use the updated local value network to obtain local value and the updated centralized hybrid network to obtain global joint value.

[0054] The target determination module is used to obtain the objective function value of the limit policy update range based on the historical experience in the historical experience pool and the updated local value network.

[0055] The second update module is used to update the policy network carried by the robot itself according to the objective function value, and to use the updated policy network for the next robot patrol.

[0056] The first aspect of this application provides an electronic device, including a processor and a memory;

[0057] The memory stores computer-executed instructions;

[0058] The processor executes computer execution instructions stored in the memory, causing the processor to perform the multi-robot patrol method as described in any of the first aspects.

[0059] Compared with the prior art, this application has the following advantages and beneficial effects:

[0060] This application provides a multi-robot patrol method, apparatus, and device. It determines target patrol actions by analyzing local observation data through a strategy network onboard the robot, achieving autonomous patrol based on local observation. This eliminates the need for pre-planning paths and is suitable for long-term tasks. After executing the target patrol action, it constructs historical experience using single-step marginal entropy gain. Based on this historical experience and the global joint value obtained from a centralized hybrid network, it updates the strategy network and value network onboard the robot. Mathematically, this ensures that the greedy strategy can approach the global optimum, allowing it to adapt to environmental changes without pre-planning. It drives the robot to actively explore areas where entropy has increased due to long-term neglect, completely eliminating local oscillations and improving training stability. Attached Figure Description

[0061] To more clearly illustrate the technical solutions of the exemplary embodiments of this application, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the drawings:

[0062] Figure 1 A flowchart illustrating a multi-robot patrol method provided in this application embodiment;

[0063] Figure 2 This is a schematic diagram of the structure of a multi-robot patrol device provided in an embodiment of this application;

[0064] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0065] The attached diagram shows the markings and corresponding component names:

[0066] 201-Action determination module, 202-Data acquisition module, 203-Experience storage module, 204-Joint analysis module, 205-First update module, 206-Target determination module, 207-Second update module, 301-Memory, 302-Processor, 303-Bus. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the embodiments and accompanying drawings. The illustrative embodiments and descriptions of this application are only for explaining this application and are not intended to limit this application.

[0068] To facilitate understanding of the technical solutions described in the embodiments of this application by those skilled in the art, a multi-robot patrol scenario is first introduced. Assume the patrol environment topology corresponding to the area to be patrolled is G(V,E), and N robots patrol this topology simultaneously. This requires not only planning the patrol routes for individual robots but also coordinating patrols among them. Here, V is the set of location nodes in the patrol environment topology, and E is the set of edges between location nodes. In this scenario, the embodiments of this application propose a multi-robot patrol method. This method determines the target patrol action by analyzing local observation data through the policy network onboard the robot, achieving autonomous patrol based on local observation. It eliminates the need for pre-planning paths and is applicable to long-term tasks. After executing the target patrol action, a single-step marginal entropy gain is used to construct historical experience. Based on this historical experience and the global joint value obtained from the centralized hybrid network, the policy network and value network onboard the robot are updated. Mathematically, this ensures that the greedy policy can approximate the global optimum, allowing for adaptive environmental changes without pre-planning. This drives the robot to actively explore areas where entropy has increased due to long-term neglect, completely eliminating local oscillations and improving training stability.

[0069] like Figure 1 As shown in the figure, this application provides a multi-robot patrol method, including:

[0070] S101. For each patrolling robot, local observation data is acquired through the robot, and the local observation data is analyzed through the strategy network on the robot itself to determine the target patrol action.

[0071] S102. Control the robot to perform its corresponding target patrol action, and obtain the single-step marginal entropy gain and the local observation data at the next moment after performing the target patrol action;

[0072] S103. Based on the local observation data, target patrol actions, single-step marginal entropy gain, and the local observation data at the next moment, construct historical experience and put the historical experience into the historical experience pool.

[0073] S104. Analyze the historical experience in the historical experience pool through the local value network carried by each robot to obtain the local value corresponding to each robot, and obtain the global joint value through a centralized hybrid network based on the local value corresponding to each robot and the real-time global state corresponding to all robots.

[0074] S105. Based on the global joint value, update the local value network and centralized hybrid network carried by the robot, and subsequently use the updated local value network to obtain local value and the updated centralized hybrid network to obtain global joint value.

[0075] S106. Based on the historical experience in the historical experience pool and the updated local value network, obtain the objective function value for limiting the update magnitude of the policy.

[0076] S107. Update the policy network on the robot itself according to the objective function value, and use the updated policy network for the next robot patrol.

[0077] In one possible implementation, for each patrolling robot, local observation data is acquired by the robot, and the local observation data is analyzed through the policy network onboard the robot to determine the target patrol action, including:

[0078] S101.1 For each patrolling robot, obtain its own state information, topological neighborhood information, and communication domain teammate information to obtain the corresponding local observation data of the robot.

[0079] Local observation data consists of its own state information, topological neighborhood information, and communication domain teammate information, aiming to comprehensively perceive the environmental topology and cooperative status.

[0080] For example, the robot's own state information may include: the topology ID (IdentityDocument) of the node where the robot is currently located, used for positioning in the patrol environment topology map; and the normalized global coordinates of the node where the robot is currently located, used to assist in spatial perception.

[0081] Topological neighborhood information is defined as the set of neighboring nodes that are directly connected to the current node in the patrol environment topology graph. It includes the IDs of all neighboring nodes and their corresponding edge weights, which directly determine the robot's current traversable action space.

[0082] A fixed communication radius is set, assuming that all other robots within this radius can establish ideal communication. Ideal communication means that communication indicators (such as communication delay) meet preset communication conditions. Then, the teammate information in the communication domain includes the IDs of all other robots within the communication radius that can establish ideal communication, as well as their relative coordinates to this robot. If there are no teammates within the communication range, or the number of teammates is less than the maximum preset value, the corresponding feature vector is padded with zeros to maintain the consistency of the input dimension.

[0083] S101.2 Analyze the local observation data through the strategy network on the robot itself to determine the target patrol action; the target patrol action refers to the position node of the robot in the patrol environment topology map corresponding to the area to be patrolled at the next moment.

[0084] To capture long-term dependencies and topological relationships during the patrol process, a Transformer is used as the backbone of the policy network, thus forming a T-Actor (T-policy network). For the aforementioned heterogeneous inputs, the T-Actor adopts a "hierarchical encoding-fusion-temporal modeling" structure, as follows:

[0085] Heterogeneous feature encoding layer: For discrete features (such as various IDs), learnable embedding layers with shared weights are used to map integer indices to dense vectors. ;in, For ID embedding vectors, Indicates length is A real vector; The robot's identity ID is represented as a string of length 1. A real-valued vector. For continuous features (such as coordinates and weights), a non-linear projection is performed using an independent multilayer perceptron, mapping the feature vector to a feature vector. ;in, The feature vectors corresponding to continuous features. The continuous feature is represented as a length of A real vector.

[0086] Feature Fusion Layer: Since the number of topological neighbors is variable, the first step is to perform mean pooling on the feature vectors of all neighbor nodes to generate a fixed-dimensional local topological feature vector. The encoded self-feature, aggregated topological neighbor features, and processed communication partner features are then concatenated. Finally, a fully connected fusion layer maps the concatenated vector to a comprehensive state vector at time step t. This serves as a single token for the Transformer (a deep learning model architecture based on self-attention). This indicates that the integrated state vector satisfies the model dimension. That is to say, each integrated state vector consists of 128 real numbers.

[0087] Temporal Feature Extraction: Input Sequence: The sequence of the robot's comprehensive state vectors over T time steps. As input, sinusoidal positional codes are superimposed to preserve the temporal order information of the trajectory without recursion. An L=4-layer Transformer Encoder Block (encoding layer with a self-attention mechanism architecture) is used; a multi-head self-attention mechanism with 8 heads and a hidden layer dimension of 256 is employed.

[0088] Output layer: The features encoded by the Transformer are passed through a fully connected layer (MLP) and a Softmax function to output the probability distribution of actions for the current topological neighbor nodes. The target patrol actions were obtained. This indicates that the policy network on the i-th robot identifies all historical observation information from the initial time to time step t. It outputs the action the robot will take at the current moment. The probability of.

[0089] In one possible implementation, obtaining the single-step marginal entropy gain after performing the target patrol action includes:

[0090] S102.1 For any location node in the patrol environment topology map corresponding to the area to be patrolled, obtain the first access probability corresponding to the location node, and obtain the first Shannon entropy corresponding to the first access probability.

[0091] For example, the first Shannon entropy can be obtained through the following technical solutions, which may include:

[0092] set up Given the joint patrol trajectory of the robots, node v is on the trajectory. The cumulative weighted access count is determined as follows:

[0093] ;

[0094] In the formula, For position node v in the trajectory The cumulative weighted access count in the data. For indicator functions, If true, the function value is set to 1; otherwise, the function value is set to 0. Given the real-time global state, if the real-time global state includes the location node v, then... Established; The importance weight of node v (default is 1, but greater than 1 for specific areas that require special patrols).

[0095] Based on the cumulative weighted access count, the probability of the first access corresponding to the location node can be determined as follows:

[0096] ;

[0097] In the formula, Let V be the first access probability corresponding to the location node, and let V be the set of all location nodes in the patrol environment topology graph.

[0098] The formula for obtaining Shannon entropy is: In the formula, For Shannon entropy.

[0099] Based on the formula for obtaining Shannon entropy, assuming the historical trajectory from time 0 to time t is... , , Let m be the set of all robot position nodes at time step m. Let m be the set of all robot actions at time step m, where m = 1, 2, ..., t, and t is the current time step. Then, the first Shannon entropy corresponding to the first access probability can be obtained as: ;

[0100] S102.2, Obtain the second access probability corresponding to the location node selected for the target patrol action, and obtain the second Shannon entropy corresponding to the second access probability: In the formula, The location node selected for carrying out target patrol actions.

[0101] S102.3. Based on the first Shannon entropy and the second Shannon entropy, determine the single-step marginal entropy gain after performing the target patrol action.

[0102] Based on the first and second Shannon entropies mentioned above, the single-step marginal entropy gain can be determined as follows: In the formula, This represents the single-step marginal entropy gain.

[0103] If a location node has never been visited, it will significantly increase the support set of the distribution, usually resulting in a large positive gain. If a location node has already been visited, its gain depends on the current distribution state. Due to the concavity and submodularity of the entropy function, if the previous visit frequency of a location node is much lower than the average level, repeated visits will make the distribution more uniform, resulting in a positive gain. If a location node has been over-visited, revisiting it will make the distribution more skewed, and the gain will decrease sharply or even approach zero.

[0104] By combining sub-modular strategy optimization theory, the action-induced single-step marginal entropy gain is directly used as the reward signal. This mathematically ensures that the greedy strategy can approach the global optimal solution, driving the robot to actively explore regions where the entropy (uncertainty) has increased due to long-term neglect, thus completely eliminating local oscillations.

[0105] In one possible implementation, the historical experience in the historical experience pool is analyzed through a local value network carried by each robot to obtain the local value corresponding to each robot, including:

[0106] S104.1 For any robot, generate a joint trajectory of historical local observation and action using the historical experience in the historical experience pool corresponding to the robot;

[0107] S104.2 Analyze the historical local observation and action joint trajectory through the local value network carried by the robot to determine the local value corresponding to the robot.

[0108] Each robot maintains an independent local value network to estimate its local utility on the current historical trajectory. To address the long temporal dependencies in patrol tasks and maintain consistency with T-Actor feature extraction capabilities, the local value network employs a Transformer with a self-attention mechanism. The input to the local value network is not a single-time observation, but rather the robot's historical observation-action joint trajectory. This trajectory includes all historical position nodes of the robot up to time step t-1 and the historical target patrol actions, and can be represented as... In the formula, For the i-th robot, the historical local observation data at the initial time step, For the i-th robot, the historical target patrol action at the initial time step is... For the historical local observation data of the i-th robot at time step t, The historical target patrol action of the i-th robot at time step t-1.

[0109] By aggregating historical information through a self-attention mechanism, the problem of state inference in partially observable environments is solved. Finally, a scalar value is output, representing the cumulative value estimate of the robot's current policy based on the complete sequence of states from history to the present, rather than just the evaluation based on the current instantaneous state, thus obtaining the robot's corresponding local value.

[0110] In one possible implementation, a global joint value is obtained through a centralized hybrid network based on the local value corresponding to each robot and the real-time global state corresponding to all robots, including:

[0111] S104.3 Obtain the real-time location node set and global environment heat map of all robots to obtain the real-time global state of all robots;

[0112] For example, a global environment heatmap can be denoted as... The cumulative number of visits to all nodes in the graph is the key basis for calculating the single-step marginal entropy gain. This global state is only obtained and used on the centralized training server. In actual physical deployment (execution phase), the centralized hybrid network is removed, and the robot does not need to obtain the global heatmap. It only needs to run the local value network or policy network.

[0113] Optionally, the global state acquisition mechanism can be as follows: the real-time location node set is directly extracted from memory by the simulation engine and input to the Q-Critic supernetwork without delay. Q-Critic refers to a Q-value network composed of local value networks and a centralized hybrid network, used to form a reinforcement learning architecture with T-Actor. The integration of Transformer into T-Actor and Q-Critic endows the robot with full-lifecycle memory capabilities. In large-scale patrol environment topology mapping, it can effectively reduce revisit intervals, and performance does not drastically degrade as the size of the patrol environment topology increases.

[0114] S104.4 Analyze the local value of each robot and the real-time global state through the centralized hybrid network to determine the global joint value.

[0115] Centralized hybrid networks can employ a feedforward neural network structure, with weights generated by a supernetwork. The supernetwork takes the global state as input and outputs the inter-layer weights and biases of the hybrid network. This ensures consistency between local and global optima (i.e.,...). This represents the partial derivative of the global joint value with respect to the individual local value, showing how much the global value changes as the individual value increases (the value being greater than or equal to zero is to maintain monotonicity, i.e., consistency between the individual and the whole). The weight matrix generated by the hypernetwork is processed by an absolute value function, forcing all mixed weights to be non-negative. This means that if any robot increases its local value, the global joint value will not decrease, thus guiding individual behavior to converge towards the team goal.

[0116] In one possible implementation, the global joint value can be expressed as:

[0117] ;

[0118] In the formula, For the sake of overall joint value, For centralized hybrid networks, The local value corresponding to the first robot. The local value corresponding to the Nth robot. For the overall joint value.

[0119] In one possible implementation, the local value network and centralized hybrid network carried by the robot are updated based on the global joint value, including:

[0120] S105.1 For any robot, obtain the time-series difference error based on the historical experience in the historical experience pool corresponding to the robot, and obtain the dominance function value based on the time-series difference error;

[0121] In one possible implementation, for any given robot, a temporal difference error is obtained based on the historical experience in the robot's corresponding historical experience pool, and a dominance function value is obtained based on the temporal difference error, including:

[0122] For any given robot, based on the historical experience in the robot's corresponding historical experience pool, the temporal difference error is obtained as follows:

[0123] ;

[0124] In the formula, The timing difference error at time step t; The single-step marginal entropy gain is obtained based on the historical experience in the robot's corresponding historical experience pool, representing the value gained on the historical trajectory. Above, the robot performs target patrol actions and transfers location nodes. The entropy change that occurs afterward; This is the discount factor, with a value ranging from 0 to 1; For the real-time global state at time step t Based on this, the global joint value output by the centralized hybrid network; For the real-time global state at the next time step Based on this, the global joint value output by the centralized hybrid network;

[0125] The dominance function value is obtained based on the time-series difference error:

[0126] ;

[0127] In the formula, The dominant function value, This is a smoothing parameter, located between [0,1]. In applications, it can be set to 0.975 to balance the bias and variance. The time difference error at time step t+k is the time index, and T is the total number of time steps.

[0128] S105.2. Determine the estimated value of the cumulative marginal return of the discount based on the advantage function value and the global joint value in the previous update process; wherein, in the first update process, the global joint value in the previous update process is set to zero;

[0129] Based on the advantage function value and the global joint value from the previous update, the estimated cumulative marginal return after discount is determined as follows:

[0130] ;

[0131] In the formula, This is the estimated cumulative marginal return after discounting. This refers to the global joint value from the previous update process.

[0132] S105.3. Obtain the error value based on the estimated cumulative marginal return of the discount and the global joint value in the current update process;

[0133] S105.4. Based on the error value, update the network parameters of the local value network and the network parameters of the centralized hybrid network through backpropagation.

[0134] In one possible implementation, based on the historical experience in the historical experience pool and the updated local value network, the objective function value for determining the magnitude of the policy update is:

[0135] ;

[0136] ;

[0137] In the formula, To limit the objective function value of the policy update magnitude, For the network parameters of the policy network, Where i is the batch size, i is the sample index, and t is the time step. The advantage function value is obtained based on historical experience in the historical experience pool and the updated local value network. This represents the probability ratio between the old and new strategies. This is a truncation function used to restrict... exist Within the range; The truncation parameter is used to limit the ratio of the old and new strategies, preventing instability caused by excessively large strategy updates. This is the entropy regularization coefficient; This is the policy entropy regularization term, which can be calculated based on the probability distribution output by the current policy network. For the current policy network in local observation data Select action The probability, For the policy network in the last update process, based on local observation data Select action The probability of the objective function is given. To facilitate understanding of the objective function value by those skilled in the art, its underlying principle is explained.

[0138] The cumulative marginal return of the discount at time step t is defined as:

[0139]

[0140] In the formula, To accumulate marginal returns for discounts, To take an action at time step t+k and transition to the node The resulting single-step marginal entropy gain.

[0141] Based on the cumulative marginal return of the above discounts, the strategy gradient can be determined as follows:

[0142] ;

[0143] In the formula, For policy gradient, To calculate the average value of multiple sampled trajectory data, To calculate the gradient of the parameters, indicating the direction of parameter update; For network parameters are At that time, given the current local observation Under these conditions, the policy network takes action. The probability of action This is the index of the next target node that the robot decides to go to at time step t. The value range is the set of all neighboring nodes connected to the current node.

[0144] Embedding the aforementioned policy gradient into the PPO-Clip (Proximal Policy Optimization with Clipped Objective) objective of MAPPO (Multi-Agent Proximal Policy Optimization), we can obtain an objective function that limits the policy update magnitude. By maximizing the objective function to update the policy network parameters, we can make it approach the global optimum uniform coverage, thus forming a training algorithm suitable for sub-module reward functions and realizing the training of the policy network.

[0145] Based on the objective function value that limits the policy update magnitude, an end-to-end learning framework based on MAPPO can be implemented. This framework models patrol decisions as a Markov Decision Process (MDP), allowing the policy network to directly output actions based on current observations without pre-planning paths. When the environmental topology changes or node weights dynamically change, the patrol center of gravity can be adaptively adjusted without retraining or replanning. This also reduces system complexity, and the PPO-Clip mechanism ensures a stable convergence process.

[0146] like Figure 2 As shown in the figure, this application provides a multi-robot patrol device, including:

[0147] The action determination module 201 is used to acquire local observation data of each patrolling robot and analyze the local observation data through the policy network on the robot itself to determine the target patrol action.

[0148] The data acquisition module 202 is used to control the robot to perform its corresponding target patrol action, and to acquire the single-step marginal entropy gain after the target patrol action is performed and the local observation data at the next moment.

[0149] The experience storage module 203 is used to construct historical experience based on the local observation data, target patrol actions, single-step marginal entropy gain and the local observation data at the next moment, and put the historical experience into the historical experience pool.

[0150] The joint analysis module 204 is used to analyze the historical experience in the historical experience pool through the local value network carried by each robot, to obtain the local value corresponding to each robot, and to obtain the global joint value through a centralized hybrid network based on the local value corresponding to each robot and the real-time global state corresponding to all robots.

[0151] The first update module 205 is used to update the local value network and the centralized hybrid network carried by the robot based on the global joint value, and subsequently use the updated local value network to obtain local value and the updated centralized hybrid network to obtain global joint value.

[0152] The target determination module 206 is used to obtain the objective function value of the limit policy update range based on the historical experience in the historical experience pool and the updated local value network.

[0153] The second update module 207 is used to update the policy network carried by the robot itself according to the objective function value, and to use the updated policy network for the next robot patrol.

[0154] The multi-robot patrol device provided in this application embodiment can execute the above-described method and technical solution. Its principle and beneficial effects are similar, and will not be repeated here.

[0155] like Figure 3 As shown, based on the same inventive concept, this application also provides an electronic device, including a memory 301 and a processor 302; the memory 301 and the processor 302 are interconnected via a bus 303.

[0156] The memory 301 stores computer-executed instructions;

[0157] The processor 302 executes computer execution instructions stored in the memory 301, causing the processor 302 to perform a multi-robot patrol method as described in any embodiment of this application.

[0158] For specific examples, memory may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, first-in-first-out (FIFO) memory, and / or first-in-last-out (FILO) memory, etc.; specifically, processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor may be implemented using at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), PLA (Programmable Logic Array). Furthermore, the processor may include a main processor and coprocessors. The main processor, also known as the CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state.

[0159] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the multi-robot patrol method described in any of the above embodiments.

[0160] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the multi-robot patrol method described in any of the above embodiments.

[0161] All or part of the steps in the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable memory. When the program is executed, it performs the steps of the above method embodiments; and the aforementioned memory (storage medium) includes: read-only memory (ROM), RAM, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof.

[0162] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0163] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0164] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0165] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0166] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A multi-robot patrol method, characterized by, include: For each patrolling robot, local observation data is acquired through the robot, and the local observation data is analyzed through the policy network on the robot itself to determine the target patrol action; The robot is controlled to perform its corresponding target patrol action, and the single-step marginal entropy gain and local observation data at the next moment are obtained after the target patrol action is performed. Based on the local observation data, target patrol actions, single-step marginal entropy gain, and the local observation data at the next moment, historical experience is constructed and placed into the historical experience pool. The historical experience in the historical experience pool is analyzed by the local value network carried by each robot to obtain the local value corresponding to each robot. Based on the local value corresponding to each robot and the real-time global state corresponding to all robots, the global joint value is obtained through a centralized hybrid network. Based on the global joint value, the local value network and centralized hybrid network carried by the robot are updated, and the updated local value network is used to obtain local value and the updated centralized hybrid network is used to obtain global joint value. Based on the historical experience in the historical experience pool and the updated local value network, obtain the objective function value for limiting the update magnitude of the policy. Based on the objective function value, the policy network on the robot itself is updated, and the updated policy network is used for the next robot patrol. Obtain the single-step marginal entropy gain after executing the target patrol action, including: For any location node in the patrol environment topology map corresponding to the area to be patrolled, obtain the first access probability corresponding to the location node, and obtain the first Shannon entropy corresponding to the first access probability. Obtain the second access probability corresponding to the location node selected for the target patrol action, and obtain the second Shannon entropy corresponding to the second access probability; Based on the first Shannon entropy and the second Shannon entropy, determine the single-step marginal entropy gain after performing the target patrol action.

2. The multi-robot patrol method of claim 1, wherein, For each patrolling robot, local observation data is acquired through the robot, and this local observation data is analyzed using the robot's onboard policy network to determine the target patrol actions, including: For each patrolling robot, its own state information, topological neighborhood information, and communication domain teammate information are obtained to obtain the corresponding local observation data. The robot analyzes the local observation data using its own onboard strategy network to determine the target patrol action; the target patrol action refers to the position node of the robot in the patrol environment topology map corresponding to the area to be patrolled at the next moment.

3. The multi-robot patrol method according to claim 1, characterized in that, By analyzing the historical experience in the historical experience pool through the local value network carried by each robot, the local value corresponding to each robot is obtained, including: For any robot, a joint trajectory of historical local observations and actions is generated using historical experience from the robot's corresponding historical experience pool. The robot's local value is determined by analyzing the combined trajectory of historical local observations and actions using a local value network.

4. The multi-robot patrol method according to claim 1, characterized in that, Based on the local value of each robot and the real-time global state of all robots, a centralized hybrid network is used to obtain the global joint value, including: Obtain the real-time location node set and global environment heat map of all robots to obtain the real-time global state of all robots; The centralized hybrid network is used to analyze the local value of each robot and its real-time global state to determine the global joint value.

5. The multi-robot patrol method according to claim 1, characterized in that, Based on the aforementioned global joint value, the local value network and centralized hybrid network mounted on the robot are updated, including: For any robot, the timing difference error is obtained based on the historical experience in the robot's corresponding historical experience pool, and the dominance function value is obtained based on the timing difference error. Based on the advantage function value and the global joint value in the previous update process, the estimated value of the cumulative marginal return of the discount is determined; wherein, in the first update process, the global joint value in the previous update process is set to zero; The error value is obtained based on the estimated cumulative marginal return of the discount and the global joint value in the current update process; Based on the error value, the network parameters of the local value network and the centralized hybrid network are updated through backpropagation.

6. The multi-robot patrol method according to claim 5, characterized in that, For any given robot, based on the historical experience in the robot's corresponding historical experience pool, a temporal difference error is obtained, and the dominance function value is obtained based on the temporal difference error, including: For any given robot, based on the historical experience in the robot's corresponding historical experience pool, the temporal difference error is obtained as follows: ； In the formula, The timing difference error at time step t; The single-step marginal entropy gain is obtained based on the historical experience in the robot's corresponding historical experience pool, representing the value gained on the historical trajectory. Above, the robot performs target patrol actions and transfers location nodes. The entropy change that occurs afterward; As a discount factor, For the real-time global state at time step t Based on this, the global joint value output by the centralized hybrid network; For the real-time global state at the next time step Based on this, the global joint value output by the centralized hybrid network; The dominance function value is obtained based on the time-series difference error: ； In the formula, The dominant function value, For smoothing parameters, The time difference error at time step t+k is the time index, and T is the total number of time steps. Based on the advantage function value and the global joint value from the previous update, the estimated cumulative marginal return after discount is determined as follows: ； In the formula, This is the estimated cumulative marginal return after discounting. This refers to the global joint value from the previous update process.

7. The multi-robot patrol method according to claim 1, characterized in that, Based on the historical experience in the historical experience pool and the updated local value network, the objective function value for limiting the policy update magnitude is obtained as follows: ； ； In the formula, To limit the objective function value of the policy update magnitude, For the network parameters of the policy network, Where i is the batch size, i is the sample index, and t is the time step. The advantage function value is obtained based on historical experience in the historical experience pool and the updated local value network. This represents the probability ratio between the old and new strategies. This is a truncation function used to restrict... exist Within the range; To truncate parameters, Here is the entropy regularization coefficient. For policy entropy regularization, For the current policy network in local observation data Select action The probability, For the policy network in the last update process, based on local observation data Select action The probability is given by T, where T represents the total number of time steps taken.

8. A multi-robot patrol device, wherein the multi-robot patrol device is used to perform the multi-robot patrol method according to any one of claims 1 to 7, characterized in that, include: The action determination module is used to acquire local observation data of each patrolling robot and analyze the local observation data through the policy network on the robot itself to determine the target patrol action. The data acquisition module is used to control the robot to perform its corresponding target patrol action, and to acquire the single-step marginal entropy gain after the target patrol action is performed and the local observation data at the next moment. An experience storage module is used to construct historical experience based on the local observation data, target patrol actions, single-step marginal entropy gain, and local observation data at the next moment, and to put the historical experience into a historical experience pool. The joint analysis module is used to analyze the historical experience in the historical experience pool through the local value network carried by each robot, to obtain the local value corresponding to each robot, and to obtain the global joint value through a centralized hybrid network based on the local value corresponding to each robot and the real-time global state corresponding to all robots. The first update module is used to update the local value network and centralized hybrid network carried by the robot based on the global joint value, and subsequently use the updated local value network to obtain local value and the updated centralized hybrid network to obtain global joint value. The target determination module is used to obtain the objective function value of limiting the policy update magnitude based on the historical experience in the historical experience pool and the updated local value network. The second update module is used to update the policy network carried by the robot itself according to the objective function value, and to use the updated policy network for the next robot patrol.

9. An electronic device, characterized in that, Including processor and memory; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the multi-robot patrol method as described in any one of claims 1 to 7.

Images