A method for enhancing patrol path diversity of multiple robots based on reinforcement learning

Abstract

The present application relates to a kind of based on reinforcement learning's multi-robot patrol path diversity enhancement method, belong to multi-robot cooperative control technical field, solve the existing technology path homogenization, training instability, poor scalability, path predictability high problem.It includes the model of constructing sub-strategy cluster with multiple different initial parameter sub-strategy and intelligent decision fusion layer, sub-strategy parallel output action probability sequence, fusion layer combines system state and random disturbance and outputs action instruction;Multiple scale multi-topology path point set is generated, and the model is trained to convergence by alternately freezing parameters;The path planning is carried out to the path point set to be planned using the trained model, and the complete patrol path is obtained.The path diversity and patrol efficiency are unified, and the system stability, scalability and security active defense capability are improved.

Description

Technical Field

[0001] This invention relates to the field of multi-robot cooperative control technology, and in particular to a method for enhancing the diversity of multi-robot patrol paths based on reinforcement learning. Background Technology

[0002] In modern security systems, multi-robot collaborative patrols have become a key means to improve regional monitoring efficiency and enhance security capabilities. Such systems not only require robots to fully cover the monitored area and eliminate blind spots, but also to proactively deter potential intrusions through patrol activities. However, the fixed or highly predictable nature of patrol paths can severely weaken security effectiveness—intruders can evade robot detection by analyzing historical patrol data, causing the security system to lose its practical protective value. Therefore, generating highly diverse and unpredictable patrol paths while ensuring complete area coverage and patrol response efficiency has become a core technical challenge that multi-robot security patrol systems urgently need to address.

[0003] Existing multi-robot patrol path allocation technologies are mainly divided into two categories:

[0004] One type is the traditional static partitioning path allocation method. The core of this method is to pre-divide the entire monitoring area into several fixed sub-regions and assign a dedicated patrol area to each robot. For example, after dividing the area using a grid map, a sequential traversal closed-loop path including key nodes is planned for each robot. Its advantages lie in its simple implementation logic and low development cost, but it has significant limitations: the partitioning scheme remains fixed after initialization, making it unable to perceive the global dynamic state of the system. Performance degrades significantly in large-scale or complex structural scenarios, specifically manifested in the task allocation calculation time increasing superlinearly with the expansion of the area size. Simultaneously, the system architecture is highly rigid; when the number of robots needs to be dynamically increased or decreased, the entire partition topology must be reconstructed and all paths replanned, resulting in extremely poor flexibility and scalability.

[0005] Another type is the end-to-end path planning method based on reinforcement learning, which has emerged in recent years. This type of method models the global multi-robot path allocation problem as a sequence generation task. It directly outputs the time-ordered path point sequence of all robots through a single policy network and introduces special separators to identify the task boundaries of different robots. It achieves perception of the global environment state through a unified decision-making mechanism, which to some extent avoids the task fragmentation problem caused by traditional static partitioning. Under the assumption of robot homogeneity, it can get rid of strong partitioning constraints through a single model and achieve dynamic connection of cross-regional patrol paths. However, these methods still have key drawbacks: centralized decision-making architecture and single-objective optimization mechanisms can easily lead to irreversible convergence of strategies to the same static optimal solution, resulting in highly homogeneous patrol paths; even if traditional multi-objective optimization algorithms such as NSGA-II are used to initially generate Pareto solution sets, the solution sets will continuously collapse during iteration due to the lack of a continuous and effective dynamic diversity maintenance mechanism, making it difficult to maintain path heterogeneity in the long term; and random perturbation improvement methods such as ε-greedy, due to the lack of structural guidance and target adaptability of perturbations, cannot construct statistically significant non-steady-state path distributions in long-term tasks, making it difficult to fundamentally break through the bottleneck of path pattern solidification.

[0006] In summary, existing technologies cannot effectively solve problems such as path homogenization, unstable system training, and poor scalability while ensuring patrol coverage and response efficiency. Summary of the Invention

[0007] Based on the above analysis, the embodiments of the present invention aim to provide a method for enhancing the diversity of multi-robot patrol paths based on reinforcement learning, in order to solve existing problems.

[0008] On one hand, embodiments of the present invention provide a method for enhancing the diversity of multi-robot patrol paths based on reinforcement learning, comprising the following steps: A robot patrol model is constructed, which includes a sub-policy cluster with multiple sub-policies with differentiated initial parameters and an intelligent decision fusion layer. Each sub-policy outputs an action probability sequence to the intelligent decision fusion layer in parallel according to the system state. The intelligent decision fusion layer outputs action instructions based on each action probability sequence, the system state, and added random perturbations. Randomly generate path point sets of different sizes and topologies. For each path point set, use the robot patrol model to perform path planning to generate a decision sequence. Use the decision sequence to construct a loss function. Then, by alternately freezing the parameters of the sub-policy cluster and the intelligent decision fusion layer, the robot patrol model is collaboratively trained until convergence. The trained robot patrol model is used to plan a complete patrol path from the set of points to be planned.

[0009] Based on the further improvement of the above method, the collaborative training is achieved through a two-stage alternating iteration. In the first stage, the parameter gradient of the intelligent decision fusion layer is frozen, the sub-policy clusters are optimized, and a perturbation mechanism is introduced. In the second stage, the parameter gradient of all sub-policy clusters is frozen, the intelligent decision fusion layer is optimized, and a random perturbation term is superimposed. The process is iterated until the robot patrol model converges.

[0010] Based on the further improvement of the above method, each iteration is for a different set of path points, and z independent path plannings are performed through the robot patrol model to generate z distinct decision sequences. Based on the z decision sequences, a sub-policy cluster loss function and an intelligent decision fusion layer loss function are constructed respectively, and the parameters of the sub-policy cluster and the intelligent decision fusion layer are optimized based on the sub-policy cluster loss function and the intelligent decision fusion layer loss function respectively.

[0011] Based on the further improvement of the above method, in the stage of optimizing the sub-policy cluster, the binary mask obtained by sampling from the Bernoulli distribution is used. As an adaptive perturbation factor, some paths generated by sub-policies are randomly disabled. At the same time, policy entropy loss and the decision sequence are introduced to jointly construct a sub-policy cluster loss function to train the sub-policy cluster.

[0012] Based on a further improvement of the above method, the sub-policy cluster loss function is: , in, ( ) represents the j-th sub-strategy, θ j Let be the parameter of the j-th sub-strategy, and b be the balance factor. z is the number of planning iterations, k is the number of sub-policies, s is the given instance, and A i For the decision sequence, L(A) i ) is the decision sequence A i The cost, i∈{1,2,…,z}, j∈{1,2,…,k}, The entropy loss is the policy entropy loss.

[0013] Where β is the weighting coefficient, H(π( |S t )) is the sub-policy in system state S t The policy entropy, S, is the probability of choosing a certain action. t Let t be the system state at time t.

[0014] Based on the further improvement of the above method, in the stage of optimizing the intelligent decision fusion layer, the parameter gradients of all sub-policies are frozen, a decision fusion loss function is constructed based on z decision sequences, and the intelligent decision fusion layer is optimized using the decision fusion loss function.

[0015] Based on a further improvement of the above method, the formula for the decision fusion loss function is as follows: Where z is the number of planning iterations, and A i For the decision sequence, L(A) i ) is the decision sequence A i The cost, b is the balance factor; The cost formula for the decision sequence is: , Among them, A i,c For decision sequence A i The path sequence of the c-th robot, a t For the action assignment at time t, || ||2 represents the Euclidean distance.

[0016] Based on further improvements to the above method, the differentiated initial parameters include at least one of the following: the neural network architecture of the sub-strategy, the initial weights, the network depth, and the connection method.

[0017] Based on a further improvement of the above method, the system state includes a partial decision sequence s consisting of the path points assigned at time t. t And the set x of all unassigned waypoints and robots at time t. t The decision sequence is based on the s when completing path planning. t Obtained by splitting according to the robot.

[0018] Based on the further improvement of the above method, the action instructions include switching robots and selecting path points. Switching robots means ending the path planning of the previous robot and starting the path planning of the new robot. Selecting path points means adding a new path point after the current robot's path.

[0019] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects: 1. By constructing a sub-policy cluster with differentiated initial parameters, and combining Bernoulli mask adaptive perturbation and policy entropy loss mechanism, each sub-policy is forced to evolve towards a complementary solution space, avoiding excessive policy convergence. This achieves the essential diversity and unpredictability of multi-robot patrol paths, significantly reduces the probability of intruders evading detection, and enhances the proactive defense capability of the security system. 2. By using a two-stage alternating training and parameter gradient freezing mechanism, training interference between the sub-policy cluster and the intelligent decision fusion layer is isolated. At the same time, standard normal distribution random perturbation is embedded in the intelligent decision fusion layer, which realizes the unity of model training stability and dynamic adaptability of decision output. This ensures both the coverage integrity and response efficiency of patrol paths and enhances the system's robustness to environmental changes. 3. Through the modular architecture of the decoupled sub-policy cluster and intelligent decision fusion layer, the system supports the dynamic addition and deletion of policy units and the flexible adjustment of task scale. Combined with the parallel update design of sub-policy in the training phase, the system can quickly adapt to scenarios such as the increase or decrease of the number of robots and the dynamic addition of path points without global model reconstruction, which significantly improves the scalability of the system and the computational efficiency in large-scale scenarios.

[0020] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0021] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0022] Figure 1 This is a flowchart of the multi-robot patrol path diversity enhancement method based on reinforcement learning in an embodiment of the present invention. Detailed Implementation

[0023] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0024] A specific embodiment of the present invention discloses a method for enhancing the diversity of multi-robot patrol paths based on reinforcement learning, such as... Figure 1 As shown, it includes the following steps: S1: Construct a robot patrol model that includes a cluster of sub-policies with multiple sub-policies having differentiated initial parameters and an intelligent decision fusion layer; each sub-policy outputs an action probability sequence in parallel to the intelligent decision fusion layer according to the system state, and the intelligent decision fusion layer outputs action instructions based on each action probability sequence, the system state, and added random perturbations.

[0025] Specifically, the differentiated initial parameters include at least one of the following: the neural network architecture of the sub-strategy, the initial weights, the network depth, and the connection method.

[0026] It should be noted that the sub-policies adopt a Transformer architecture, with each sub-policy's Transformer containing a self-attention layer and a feedforward network. The number of attention heads in the self-attention layer is set to a value between 8 and 16, while the hidden layer dimension of the feedforward network is uniformly 512-dimensional, and the activation function is ReLU. The number of network layers for each sub-policy is set to a different configuration of 3 to 5 layers, and the initial weights are randomly generated using a Xavier normal distribution to ensure the heterogeneity of the initial parameters. The intelligent decision fusion layer is a multilayer perceptron (MLP), employing a fully connected structure consisting of an input layer, a hidden layer, and an output layer. The input layer dimension is determined by the number of sub-policies and the action space dimension, specifically the product of the number of sub-policies, the number of robots, and the number of path points, superimposed with a 10-dimensional global feature vector of the system. The hidden layer consists of two layers: the first hidden layer has a dimension of 64 dimensions, and the second has a dimension of 32 dimensions, with the activation function being ReLU. The output layer dimension is consistent with the action space dimension, and the activation function is Softmax. The differentiated configurations of attention heads, network depth, and initial weights for each sub-policy match the design requirements of differentiated initial parameters. This ensures, from the architectural structure and parameter source, that each sub-policy develops different feature extraction preferences and function approximation capabilities, producing different latent representations for the same state input. The lightweight, fully connected structure of the MLP synergizes with the heterogeneous architecture of the sub-policy cluster, adapting to the differentiated outputs of multiple sub-policies and providing a structural foundation for subsequent dynamic weight calculation and decision fusion.

[0027] S2: Randomly generate path point sets of different sizes and topologies. For each path point set, use the robot patrol model to perform path planning to generate a decision sequence. Use the decision sequence to construct a loss function. Then, use the parameters of the sub-policy cluster and the intelligent decision fusion layer to perform collaborative training on the robot patrol model until convergence.

[0028] It should be noted that a set of path points P = {p1, p2, ..., pn} containing n path points is randomly generated. n}, each path point Represented as ,in , Representing path points , coordinate A set of m robots initial position Represented as ,in , These represent the robot's initial state. , coordinate, .

[0029] Specifically, the collaborative training is achieved through a two-stage alternating iteration. In the first stage, the parameter gradients of the intelligent decision fusion layer are frozen, the sub-policy cluster is optimized, and a perturbation mechanism is introduced. In the second stage, the parameter gradients of all sub-policies are frozen, the intelligent decision fusion layer is optimized, and a random perturbation term is superimposed. The process is iterated until the robot patrol model converges.

[0030] It should be noted that the parameter gradient freezing operation in the two-stage training is achieved by explicitly masking the backpropagation gradient of the corresponding network. Freezing the gradient of the fusion layer during the sub-policy update stage is to prevent changes in the parameters of the intelligent decision fusion layer from interfering with the independent evolution of the sub-policies. Freezing the parameter gradients of all sub-policies during the intelligent decision fusion layer update stage is to allow the intelligent decision fusion layer to focus on learning the integration rules of the multi-source policy outputs, ensuring that each module achieves collaborative optimization under stable architectural constraints.

[0031] Specifically, for each iteration, for a different set of path points, z independent path plannings are performed using the robot patrol model to generate z distinct decision sequences. Based on the z decision sequences, a sub-policy cluster loss function and an intelligent decision fusion layer loss function are constructed respectively. Based on the sub-policy cluster loss function and the intelligent decision fusion layer loss function, the parameters of the sub-policy cluster and the intelligent decision fusion layer are optimized respectively.

[0032] It should be noted that each iteration targets a different set of path points, performing z independent path planning operations using the robot patrol model to generate z distinct decision sequences A. i (i=1,2,…,z) The scope of strategy exploration is expanded through multiple path planning iterations, improving the model's adaptability to different path allocation scenarios. A sub-policy cluster loss function and an intelligent decision fusion layer loss function are constructed based on the z decision sequences. The loss value is the average of the z training results, which reduces the random error of a single path planning iteration, making the loss more closely reflect the model's true optimization state and ensuring the stability and convergence of the training process.

[0033] It should be noted that the decision sequence A in a single path planning iteration... i = (A i1 A i2 ,…,A im ) contains the paths of all robots, where the path A of any robot is... ij =(r ij ,p k ,p k+1 ,…,p k+x ), And satisfy The first to arrive mAfter merging the sets of path points traversed by each robot and removing duplicate path points, the resulting set is exactly equal to the set of path points. P .

[0034] Specifically, in the stage of optimizing the sub-policy cluster, the binary mask obtained by sampling from the Bernoulli distribution is used. As an adaptive perturbation factor, some paths generated by sub-policies are randomly disabled. At the same time, policy entropy loss and the decision sequence are introduced to jointly construct a sub-policy cluster loss function to train the sub-policy cluster.

[0035] It should be noted that during the parameter gradient freeze of the intelligent decision fusion layer, each sub-policy is trained independently and in parallel. The parameter update process is independent of each other and unaffected by the parameter state of the intelligent decision fusion layer. During training, each sub-policy interacts with the environment, which consists of the path point set and the robot. Each sub-policy receives the system state at the current time step as input, performs feature extraction and action probability distribution prediction based on its own heterogeneous initial parameters, and outputs an action probability sequence. The robot executes the action with the highest probability. After execution, the updated system state is obtained, and the updated state is input into the sub-policy to obtain the action for the next time step. This process continues until all path points in the path point set have been traversed. The actions at each time step are sorted by time to obtain the decision sequence of that sub-policy. Each sub-policy repeats the above process z times to obtain z different decision sequences.

[0036] The intelligent decision fusion layer does not participate in the generation of decision sequences at this stage, but only provides a stable environment for the independent evolution of sub-policies through parameter freezing. For the decision sequences generated by each sub-policy, random disabling decisions are made using binary masks obtained by sampling from a Bernoulli distribution. Each decision sequence corresponds to an independent mask element decision result, thereby disrupting the consistency of short-term reward signals among different sub-policies. Each sub-policy constructs a loss function by combining its own decision sequence and policy entropy loss, and updates only its own network parameters through backpropagation. The policy entropy loss optimizes path allocation efficiency while maintaining the randomness and diversity of each sub-policy's decisions, preventing them from converging to a fixed decision pattern.

[0037] Specifically, the sub-policy cluster loss function is: , in, ( ) represents the j-th sub-strategy, θ j Let be the parameter of the j-th sub-strategy, and b be the balance factor. z is the number of planning iterations, k is the number of sub-policies, s is the given instance, and A i For the decision sequence, L(A) i ) is the decision sequence A iThe cost, i∈{1,2,…,z}, j∈{1,2,…,k}, The entropy loss is the policy entropy loss.

[0038] Where β is the weighting coefficient, H(π( |S t )) is the sub-policy in system state S t The policy entropy, S, is the probability of choosing a certain action. t Let t be the system state at time t.

[0039] It should be noted that the sampling probability of the Bernoulli distribution can be dynamically configured according to the training scenario, and the sampling results... The matrix is ​​a z-row, k-column binary numerical matrix containing only 0s and 1s, where i∈{1,2,…,z} corresponds to z path planning iterations, and j∈{1,2,…,k} corresponds to k sub-policies. Each element in the matrix uniquely corresponds to the decision sequence generated by the j-th sub-policy in the i-th path planning iteration. A value of 0 indicates that the decision sequence of the corresponding sub-policy is disabled, and a value of 1 indicates that the decision sequence is retained. This random disabling method disrupts the consistency of short-term reward signals among different sub-policies, forcing each sub-policy to evolve towards a complementary solution space. This provides a differentiated training basis for the sub-policy cluster loss function, ensuring the accuracy of the correlation between loss calculation and decision sequences.

[0040] It's important to note that policy entropy is an indicator of the randomness and diversity of a policy's decisions in the current state. Higher entropy indicates a more even probability distribution of different actions chosen by the policy, resulting in greater decision diversity and preventing sub-policies from converging to identical decision logic. Lower entropy indicates a greater tendency for the policy to choose specific actions, leading to stronger decision determinism and potentially causing path pattern solidification. By incorporating policy entropy loss into the sub-policy cluster loss function, we can optimize path allocation efficiency while forcibly maintaining the decision differences between sub-policies. The formula for policy entropy is... , Where, p i Let be the probability that the sub-policy selects the i-th action under the system state. f=[f1,f2,…,f m+n ] represents the original score sequence for each action output by the sub-policy, where m is the number of robots and n is the number of path points.

[0041] It should be noted that each element in f corresponds to the original score of a candidate action. Candidate actions include all robots and waypoints; therefore, the length of f is the same as the total number of elements in the robot set and the waypoint set. Each sub-policy will convert its own output f into the corresponding p. i The sequence is the sequence of action probabilities for the sub-policy, p = [p1, p2, ... p].m+n This fully describes the sub-policy's preference for selecting all candidate actions. Subsequently, the action probability sequences of each sub-policy are input into the intelligent decision fusion layer. The intelligent decision fusion layer assigns adaptive weights to these sequences based on the system state and integrates them, then adds random perturbations to generate the final action command, realizing the coordinated implementation of multi-sub-policy decisions.

[0042] Specifically, in the stage of optimizing the intelligent decision fusion layer, the parameter gradients of all sub-policies are frozen, a decision fusion loss function is constructed based on the z decision sequences, and the intelligent decision fusion layer is optimized using the decision fusion loss function.

[0043] It should be noted that the neural network used in the intelligent decision fusion layer is a multilayer perceptron. This perceptron dynamically weights the action probability distributions output by each sub-strategy through a fully connected structure. The weight allocation is adaptively adjusted based on the current system state, ensuring that different sub-strategy proposals receive appropriate attention according to scenario requirements. The added random perturbation is sampled from a standard normal distribution. Its purpose is to introduce appropriate randomness into the fusion decision, preventing the final output action commands from forming a fixed pattern and further enhancing the diversity of path allocation. The decision fusion loss function is used to quantify the deviation between the path allocation cost corresponding to the fused action command and the average cost. Backpropagation is used to adjust the network parameters of the multilayer perceptron and optimize the weight allocation rules, ensuring that the fused action command can both take into account the decision advantages of each sub-strategy and meet the efficiency requirements of path allocation. During this optimization stage, since the parameter gradients of all sub-strategies are frozen, their output action probability distributions remain stable, providing a reliable training basis for the weight learning of the intelligent decision fusion layer, ensuring the learning effect of the fusion strategy, and achieving collaborative adaptation between the sub-strategy cluster and the intelligent decision fusion layer. In each round of training, the same set of path points is used to perform z independent path plannings. The decision fusion loss value is the average of the z training results. This reduces the random error of a single path planning, ensures the training stability and convergence effect of the intelligent decision fusion layer, and enables the optimized intelligent decision fusion layer to accurately integrate the decision advantages of each sub-strategy and output action commands that take into account both efficiency and diversity.

[0044] Specifically, the formula for the decision fusion loss function is as follows: Where z is the number of planning iterations, and A i For the decision sequence, L(A) i ) is the decision sequence A i The cost, b is the balance factor.

[0045] Specifically, the cost formula for the decision sequence is as follows: , Among them, A i,c For decision sequence Ai The path sequence of the c-th robot, a t For the action command at time t, || ||2 represents the Euclidean distance.

[0046] It should be noted that decision sequence A i The cost is the maximum path length of all robots in the sequence, meaning the overall patrol time is determined by the robot with the longest path, ensuring a balanced path allocation.

[0047] It should be noted that the action instructions ,in, For the output of each sub-policy, Ψ For intelligent decision fusion layer, Let ε be the parameters of the intelligent decision fusion layer, ε~N(0,I) be the random perturbation, and I be the identity matrix.

[0048] It should be noted that the model convergence condition is that the changes in the sub-policy cluster loss function and the intelligent decision fusion layer loss function are both less than the preset value in n consecutive iterations.

[0049] S3: Use the trained robot patrol model to plan the path of the set of path points to obtain the complete patrol path.

[0050] It should be noted that, for the set of path points to be planned, the trained robot patrol model will gradually generate a complete decision sequence through dynamic updates of system status and selection of action commands, and finally decompose it into the exclusive patrol path for each robot.

[0051] Specifically, the system state includes a partial decision sequence s consisting of the path points assigned at time t. t And the set x of all unassigned waypoints and robots at time t. t The decision sequence is based on the s when completing path planning. t Obtained by splitting according to the robot.

[0052] It should be noted that the system state at time t , where s t Let x be a partial decision sequence consisting of the path points assigned at time t. t Let t be the set of all unassigned path points and robots at time t. The system state is continuously updated with each execution of an action command. For example, suppose there are 2 robots and 3 path points, and the initial system state is... ,in , The first allocation of robots Then the system status is updated to ,in The second allocation was chosen as the robot. Adding path point p1 updates the system status to... ,in .

[0053] Specifically, the action instructions include switching robots and selecting waypoints. Switching robots means ending the path planning of the previous robot and starting the path planning of the new robot. Selecting waypoints means adding a new waypoint to the path of the current robot.

[0054] It should be noted that the action instruction that can be executed at time t is a. t , must meet Where "\" represents the set difference operation, and P is the set of path points. Let a be a set of robots. t The constraint, which is the difference between the union of the pathpoint set and the robot set and the set of assigned action instructions, corresponds to the system state update logic, ensuring that a t From x only t Select from the unassigned elements in the action space to avoid repetitive actions. All action instructions that can be executed at any time constitute the action space B, B=P∪R.

[0055] It should be noted that the robot switching action is triggered when the current robot's path length reaches d% of the total distance of all unassigned path points, and the path planning of the next robot is started. The percentage d can be dynamically configured according to the actual scenario, with an example value of 30%~50% (e.g., d=40%). That is, when the total length of the current robot's assigned path reaches 40% of the total distance of the remaining unassigned path points, the robot switching is triggered. If the total distance of the remaining unassigned path points is 0, the current robot path planning is directly terminated to ensure that the path lengths of each robot are balanced and to avoid overloading a single robot.

[0056] It should be noted that after all action instructions are completed, the state sequence s t This will be integrated into a complete decision sequence, namely the patrol paths of each robot. The integration rules are based on the state sequence s. t The split implementation: s t Path points between two adjacent robot identifiers (e.g., r1 and r2) belong to the preceding robot (r1), and all path points after the last robot identifier belong to that robot (r2). n ); for example, s t =(r1,p1,p2,r2,p3,p4,r3,p5), then the integrated path of robot 1 is (r1,p1,p2), the path of robot 2 is (r2,p3,p4), and the path of robot 3 is (r3,p5), ensuring that the correspondence between the path sequence and the robots is unambiguous.

[0057] It should be noted that the practical application can be carried out offline. After obtaining the complete patrol path, the path for each robot is sent to the corresponding robot, and the robot patrols according to the preset path. The practical application process of this invention also supports online dynamic adjustment, which is achieved through a modular architecture of decoupled sub-policy clusters and intelligent decision fusion layer. It can adapt to changes in environment and task without global reconstruction of the model. When a new path point is added during the process, the system can directly add the new path point to the unassigned path point and robot set x. t When the robot malfunctions, it only needs to be switched from x. t The system removes faulty robot identifiers. When the number of robots needs to be dynamically increased or decreased, the sub-policy cluster can flexibly add or remove policy units. The intelligent decision fusion layer does not need to be retrained; it only performs dynamic weight integration based on the abstract decision output of the sub-policies, avoiding direct processing of the underlying environmental state. Simultaneously, the parallel update mechanism of the sub-policies used in the training phase makes the path planning time of a large-scale cluster approach that of a traditional single-model solution. This significantly improves the system's adaptability and computational efficiency when dynamically adding or removing robots, changing task scale, or experiencing sudden environmental changes, ensuring that the continuity and diversity of patrol paths remain unaffected.

[0058] Those skilled in the art will understand that all or part of the processes implementing the methods of the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0059] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for enhancing the diversity of multi-robot patrol paths based on reinforcement learning, characterized in that, Includes the following steps: A robot patrol model is constructed, which includes a sub-policy cluster with multiple sub-policies with differentiated initial parameters and an intelligent decision fusion layer. Each sub-policy outputs an action probability sequence to the intelligent decision fusion layer in parallel according to the system state. The intelligent decision fusion layer outputs action instructions based on each action probability sequence, the system state, and added random perturbations. Randomly generate path point sets of different sizes and topologies. For each path point set, use the robot patrol model to perform path planning to generate a decision sequence. Use the decision sequence to construct a loss function. Then, by alternately freezing the parameters of the sub-policy cluster and the intelligent decision fusion layer, the robot patrol model is collaboratively trained until convergence. The trained robot patrol model is used to plan a complete patrol path from the set of points to be planned.

2. The method according to claim 1, characterized in that, The collaborative training is achieved through a two-stage alternating iteration. In the first stage, the parameter gradients of the intelligent decision fusion layer are frozen, the sub-policy clusters are optimized, and a perturbation mechanism is introduced. In the second stage, the parameter gradients of all sub-policy clusters are frozen, the intelligent decision fusion layer is optimized, and random perturbation terms are superimposed. The process is repeated until the robot patrol model converges.

3. The method according to claim 1, characterized in that, Each iteration targets a different set of path points, performing z independent path planning operations using a robot patrol model to generate z distinct decision sequences. Based on these z decision sequences, a sub-policy cluster loss function and an intelligent decision fusion layer loss function are constructed, and the parameters of the sub-policy cluster and intelligent decision fusion layer are optimized based on these loss functions.

4. The method according to claim 3, characterized in that, In the phase of optimizing the sub-policy cluster, a binary mask obtained by sampling from the Bernoulli distribution is used. As an adaptive perturbation factor, some paths generated by sub-policies are randomly disabled. At the same time, policy entropy loss and the decision sequence are introduced to jointly construct a sub-policy cluster loss function to train the sub-policy cluster.

5. The method according to claim 4, characterized in that, The sub-policy cluster loss function is: ， in, ( ) represents the j-th sub-strategy, θ j Let be the parameter of the j-th sub-strategy, and b be the balance factor. z represents the number of planning iterations, k represents the number of sub-policies, s represents the given instance, and A i For the decision sequence, L(A) i ) is the decision sequence A i The cost, i∈{1,2,…,z}, j∈{1,2,…,k}, The entropy loss is the policy entropy loss. Where β is the weighting coefficient, H(π( |S t )) is the sub-policy in system state S t The policy entropy, S, is the probability of choosing a certain action. t Let t be the system state at time t.

6. The method according to claim 3, characterized in that, In the stage of optimizing the intelligent decision fusion layer, the parameter gradients of all sub-policy clusters are frozen, a decision fusion loss function is constructed based on z decision sequences, and the intelligent decision fusion layer is optimized using the decision fusion loss function.

7. The method according to claim 6, characterized in that, The formula for the decision fusion loss function is as follows: Where z is the number of planning iterations, and A i For the decision sequence, L(A) i ) is the decision sequence A i The cost, b is the balance factor; The cost formula for the decision sequence is: ， Among them, A i,c For decision sequence A i The path sequence of the c-th robot, a t For the action assignment at time t, || ||2 represents the Euclidean distance.

8. The method according to claim 1, characterized in that, The differentiated initial parameters include at least one of the following: neural network architecture of the sub-strategy, initial weights, network depth, and connection method.

9. The method according to claim 1, characterized in that, The system state includes a partial decision sequence s consisting of the path points assigned at time t. t And the set x of all unassigned waypoints and robots at time t. t The decision sequence is based on the s when completing path planning. t Obtained by splitting according to the robot.

10. The method according to claim 1, characterized in that, Action commands include switching robots and selecting waypoints. Switching robots ends the path planning of the previous robot and starts the path planning of the new robot. Selecting waypoints adds a new waypoint to the current robot's path.

Images