Optimization sub-target generation method, coverage path planning method, device and medium

Abstract

The application discloses an optimization sub-target generation method, a coverage path planning method, equipment and a medium, and the method comprises the steps of: providing a multi-scale observation space for a high-level strategy and a bottom-level strategy of hierarchical reinforcement learning; defining a set of hierarchical reinforcement learning frameworks with the high-level strategy and the bottom-level strategy; training and using a set of adjacency networks on a set of multi-scale maps, which are used to map specified environment state coordinates to a set of embedding spaces, so that the Euclidean distance in the embedding space can be used to effectively represent the accessibility or transition cost between two points in the real environment; and using the adjacency network in the hierarchical reinforcement learning framework to constrain the sub-target generated by the high-level strategy, and preferentially selecting a locally accessible sub-target. The application can effectively reduce the sub-target search space of the high-level strategy, so that the sub-target generated by the high-level strategy has both global strategic and local feasibility.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned aerial vehicle path planning, and particularly relates to an optimized sub-target generation method, a coverage path planning method, equipment and a storage medium. BACKGROUND

[0002] Coverage path planning (CPP) is a core technology of intelligent agents such as unmanned aerial vehicles and autonomous robots in many applications, and its goal is to plan a path to maximize the coverage range of the target area under the constraints of obstacle avoidance, endurance and the like. Among them, the method based on reinforcement learning, especially hierarchical reinforcement learning (HRL), has attracted widespread attention due to its potential in dealing with complex decision-making problems.

[0003] In a typical HRL framework, the high-level strategy is responsible for formulating long-term strategic goals (i.e., "sub-targets"), while the low-level strategy is responsible for executing these sub-targets. However, when applied to complex CPP tasks, existing HRL methods generally face a serious challenge, namely the "sub-target space explosion" problem: the action space of the high-level strategy (i.e., the space formed by all possible sub-targets) is exceptionally large, making it difficult to effectively explore. More specifically, the high-level strategy usually makes decisions based on a low-resolution global map to grasp the overall situation, but it lacks awareness of the complexity of the local environment. Therefore, the sub-targets generated by the high-level strategy may seem reasonable from a global perspective, but when executed by the low-level strategy, they may be completely inaccessible due to local obstacles. This information gap between the high-level strategy and the low-level execution capability leads to a large number of ineffective exploration attempts by the agent, severely reducing learning efficiency and convergence speed, and making it difficult to obtain a high-quality coverage path.

[0004] Therefore, how to effectively reduce the sub-target search space of the high-level strategy so that the sub-targets generated by it are both globally strategic and locally feasible is a technical problem that needs to be solved in the current HRL in the CPP field.

[0005] To this end, the present application proposes an optimized sub-target generation method and a coverage path planning method to solve the above technical problems. SUMMARY

[0006] The main purpose of the present application is to provide an optimized sub-target generation method and a coverage path planning method to solve the technical problems of low exploration efficiency, slow convergence speed and low final coverage rate caused by "sub-target space explosion" and information discontinuity between high and low level strategies when applying hierarchical reinforcement learning to coverage path planning.

[0007] The present application solves the above technical problems by adopting the following technical solutions:

[0008] An optimization sub-goal generation method for optimizing the path sub-goal of an agent's moving trajectory, comprising:

[0009] A multi-scale observation space is provided for the high-level strategy and low-level strategy of hierarchical reinforcement learning, including a low-resolution global map for global strategic planning and a high-resolution local map for local precise execution;

[0010] A set of hierarchical reinforcement learning frameworks with high-level strategies and low-level strategies are defined;

[0011] A set of parameter sets are trained and used on a set of multi-scale maps for the adjacency network to map the specified environment state coordinates to a set of embedding spaces, so that the Euclidean distance in the embedding space can be used to effectively represent the accessibility or transition cost between two points in the real environment;

[0012] The adjacency network is used in the hierarchical reinforcement learning framework to constrain the sub-goals generated by the high-level strategy , and to preferentially select locally accessible sub-goals.

[0013] Preferably, the high-level strategy generates a sub-goal at a time step frequency , and the low-level strategy is responsible for executing the sub-goal within steps.

[0014] Preferably, the multi-scale map is defined with a local cropping function and a global downsampling function, wherein:

[0015] The local cropping function is used to crop a local view of size from the centered preset map ;

[0016] The global downsampling function is used to average pool the preset map with a window size of to generate a low-resolution global view;

[0017] For any set of agents, at each time step , the high-level observation and the low-level observation are generated according to the current position in the preset map .

[0018] Preferably, the training process of the adjacency network includes:

[0019] collecting state transition trajectories of any set of agents in a specified environment using either a random policy or an expert policy;

[0020] for any state pair in the state transition trajectories , if the time step difference between the state pair is less than or equal to a preset threshold , the state pair is set as adjacent, otherwise, it is set as non-adjacent;

[0021] using the labeled data, training an adjacency network by a contrastive loss function to reduce the distance between adjacent state pairs in the embedding space and increase the distance between non-adjacent state pairs in the embedding space.

[0022] Preferably, in the training process of the high-level policy, an adjacency constraint loss term is introduced, which uses the adjacency constraint network to evaluate the reachability between the candidate subgoal and the current state of the agent, and punishes the selection of subgoals with poor reachability (i.e., far distance in the embedding space), thereby limiting the generation of subgoals of the high-level policy within a locally feasible range.

[0023] Preferably, for the trained adjacency network , a set of total loss functions are used to constrain the subgoals generated by the high-level policy, specifically including:

[0024] L1. Using the trained adjacency network , the behavior of the high-level policy to select subgoals is constrained to its approximate shortest transition distance, where there is an adjacency loss, the form of which is a hinge loss function, defined as:

[0025]

[0026] wherein, is the approximate shortest transition distance between states calculated by the embedding vector distance output by the adjacency network, and k is the adjacency step threshold;

[0027] L2. Defining a pseudo landmark as a temporary guide direction, where there is a landmark loss, defined as:

[0028]

[0029] wherein, is the L2 norm, and the landmark loss is used to measure the proximity of the subgoal to the pseudo landmark in the embedding space;

[0030] L3. Combining the above loss terms with the standard reinforcement learning loss to form the final total loss function of the high-level policy, defined as:

[0031]

[0032] where, L3 is the standard reinforcement learning loss, the reward loss term for high-level policy learning, is the weight hyperparameter of the adjacency loss for balancing the strength of the local feasibility constraint, is the weight hyperparameter of the landmark loss for balancing the strength of the global guidance.

[0033] A coverage path planning method, to overcome the "local shortsightedness" problem caused by relying solely on local feasibility constraints, a landmark-guided global exploration mechanism is introduced, specifically including:

[0034] S1. In the area of the preset map , a double strategy is used to sample landmark points;

[0035] S2. According to the unexplored area on the current global map, one or more centroids of the current most needed coverage area are dynamically determined as temporary final targets through connected component analysis and other methods;

[0036] S3. Based on any of the above-mentioned optimization sub-goal generation methods, in the high-level decision-making, a temporary graph containing the current state, all sampled landmarks and dynamic task targets is constructed, and the edge weight is calculated using the adjacency constraint network to plan an optimal high-level path, wherein the goal of the high-level policy is set to follow the direction of the path, and a "pseudo landmark" sub-goal is generated towards the first landmark on the path as the generated guide sub-goal;

[0037] S4. Looping the above S1-S3 steps, determine whether the task is terminated by specifying the monitoring task indicators, and after the task is terminated, a complete coverage path planning is completed.

[0038] Preferably, the double strategy of sampling landmark points in the S1 step specifically includes:

[0039] Coverage-based landmark sampling: In the state space explored by the agent, the Farthest Point Sampling (FPS) algorithm is used to sample a series of landmarks on the boundary of the explored area to encourage the agent to expand towards the unexplored area;

[0040] Based on the target-based landmark sampling, a novel novelty measurement method using random network distillation (RND) is used to maintain a priority queue of high novelty states in the uncovered target area, and to sample landmarks with specified exploration value from it to guide the agent to efficiently "fill in" the blank areas of the coverage task.

[0041] Preferably, the S3 step specifically includes the following operation process:

[0042] Based on any of the above-mentioned optimization sub-target generation methods, the high-level strategy executes the process to generate the centroid of the current uncovered area identified by the guiding sub-target as a temporary final target ;

[0043] A graph containing the current state, all sampled landmarks and the temporary final target is constructed, and the adjacency network is calculated to calculate the edge weight;

[0044] The shortest path from the current state to the temporary final target is solved on the graph as the optimal high-level path, and the first landmark of the path is selected as the high-level strategy direction;

[0045] Based on a set of pseudo-landmarks towards the high-level strategy direction and the landmark loss term , the high-level strategy is guided to generate sub-targets close to the pseudo-landmarks.

[0046] In another aspect, the application also discloses a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to make the processor execute the steps of the above method.

[0047] In another aspect, the application also discloses a computer device, which comprises a memory and a processor, and the memory stores a computer program, and the computer program is executed by the processor to make the processor execute the steps of the above method.

[0048] From the above technical solutions, the application provides an optimization sub-target generation method and a coverage path planning method. Compared with the prior art, the application has the following advantages:

[0049] 1. The adjacency constraint network provided by the application can effectively filter a large number of locally unreachable sub-targets, thereby avoiding invalid attempts of the bottom-level strategy, making each decision of the high-level strategy more feasible, and greatly improving the exploration efficiency.

[0050] 2.The landmark-guided global exploration mechanism constructed by the application can provide clear and long-term strategic direction for high-level strategy, effectively overcome local optimal trap, enable the agent to learn the strategy covering the entire complex environment more quickly, significantly accelerate the convergence speed of the model, and obtain higher final coverage.

[0051] 3.The application decouples local feasibility constraints and global strategy guidance, can form a clear two-stage optimization framework, and each module (such as the adjacent network and the landmark sampling strategy) can be independently optimized and replaced, has good flexibility and scalability, and can adapt to different complexity of coverage planning tasks.

[0052] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the application, nor to limit the scope of the application. Other features of the application will become apparent from the following description. Of course, any product implementing the application does not necessarily need to achieve all the advantages mentioned above. BRIEF DESCRIPTION OF DRAWINGS

[0053] The accompanying drawings, which form a part of this specification, are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification. The illustrations are shown to explain the application and are not intended to limit the application in any way. In the drawings:

[0054] Figure 1 It is a schematic diagram of the overall flow of the embodiments of the application;

[0055] Figure 2 It is a schematic diagram of the hierarchical reinforcement learning algorithm framework of the application;

[0056] Figure 3 It is a schematic diagram of the functional structure of the adjacent network of the application;

[0057] Figure 4 It is a schematic diagram of the landmark-based global exploration path selection process of the application;

[0058] Figure 5 It is a schematic diagram of the exploration trajectory of the baseline HRL method of the application;

[0059] Figure 6 It is a schematic diagram of the exploration trajectory of the method of the application;

[0060] Figure 7 It is a schematic diagram of the final coverage performance comparison of the HGRL method and the baseline HRL method of the application. DETAILED DESCRIPTION

[0061] The technical solutions of the present application will be described clearly and completely below in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. The embodiments in the present application and the features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0062] In the embodiments, refer to the drawings in detail Figure 1 to the drawings in detail Figure 7 .

[0063] As shown in the drawings Figure 1 , the layered reinforcement learning coverage path planning method based on sub-target generation optimization proposed in the embodiments of the present application is implemented in a system containing high-level strategy and low-level strategy, and sequentially includes:

[0064] (1) The optimization sub-target generation method adopts a local feasibility constraint step: through training and using an adjacency network (RNN), ), the environmental state coordinates are mapped to an embedding space, in which the Euclidean distance represents the reachability between states; and the sub-target generated by the high-level strategy is constrained by using the network, and the locally reachable sub-target is preferentially selected;

[0065] In the local feasibility constraint step, the adjacency network is trained by contrastive learning, and the constraint on the high-level strategy is realized by adding an adjacency loss term (Ladj) that selects a distant sub-target to the loss function of the high-level strategy, The form of the adjacency loss term (Ladj) is a hinge loss function, wherein is the approximate shortest transition distance between states calculated by the embedding vector distance output by the adjacency network, and k is the adjacency step threshold;

[0066] (2) The coverage path planning method adopts a global exploration guide step: through a landmark guide mechanism, a group of landmark points are strategically sampled, and the global exploration direction is planned for the high-level strategy based on the landmark points, so as to guide it to generate sub-targets with global strategic vision;

[0067] The landmark guiding mechanism in the global exploration guiding step employs two sampling strategies in parallel: one is landmark sampling for coverage, which uses the Farthest Point Sampling (FPS) algorithm to sample landmarks on the boundary of the explored area to promote exploration of unknown space; the other is landmark sampling based on targets, which uses a novelty measurement method such as Random Network Distillation (RND) to sample landmarks with high exploration value in the uncovered target area.

[0068] In a specific implementation, the specific operation process of the method includes:

[0069] STEP1. Obtain the system model, task definition, and environment information of the hierarchical reinforcement learning framework.

[0070] The specific implementation is:

[0071] S101. System dynamics model: determine the kinematics or dynamics model of the agent (such as a UAV). For this embodiment, the agent moves in a discrete grid environment, and the update of its position follows , where is an action selected from the action space , and is a motion mapping function.

[0072] S102. Task definition: define the Coverage Path Planning (CPP) task. This includes defining the environment map (containing obstacles, no-fly zones, etc.), the target area map to be covered , and the initial state of the agent (such as the initial position and the initial battery level ).

[0073] S103. Hierarchical reinforcement learning framework: set the basic structure of the hierarchical reinforcement learning (HRL), refer to Appendix Figure 2 , including the high-level policy and the low-level policy . The high-level policy generates a sub-goal with a frequency (e.g., every 10 low-level time steps), and the low-level policy is responsible for executing the sub-goal within steps.

[0074] In addition, regarding parameter selection and model expansion, it is further noted that the selection of in the adjacency network parameters balances the range of local constraints and computational complexity, and can generally be set as the execution period of the low-level policy.

[0075] STEP2. Constructing multi-scale observation space and adjacency network.

[0076] The specific implementation is:

[0077] S201. Multi-scale map processing for providing multi-scale observation space for high-level strategy and low-level strategy: define two map processing functions: one is a local cropping function , which is used to crop a local view of size from the centered map . The second is a global down-sampling function , which is used to average pool the map with a window of to generate a low-resolution global view. Through these two functions, at each time step , a high-level observation and a low-level observation are generated according to the current position , where the high-level strategy receives a low-resolution global environment map , and the low-level strategy receives a high-resolution local environment map centered on the agent ;

[0078] S202. Adjacency network construction: design a neural network with a parameter , whose input is coordinates and output is a low-dimensional embedding vector. This network needs to be pre-trained in the initial stage.

[0079] Further, the actual use function of the adjacency network is referred to in the attached Figure 3 , which shows how it represents the reachability of states in the physical space through embedding in the adjacency space.

[0080] In a specific embodiment, the adjacency constraint network set in this part is used to filter a large number of locally unreachable sub-goals to avoid invalid attempts by the low-level strategy, so that each decision of the high-level strategy is more feasible, which greatly improves the exploration efficiency.

[0081] STEP3. Collect trajectories and train adjacency network.

[0082] Specifically, before formal policy training, a pre-training phase is completed, including the following steps:

[0083] S301. Trajectory data collection: let the agent explore the environment using a random policy or an expert policy, and collect a large number of state transition trajectories;

[0084] S302. For any pair of states in the trajectory, if their time step difference is less than or equal to a preset threshold are considered contiguous; otherwise, non-contiguous

[0085] S303 uses the labeled data to train the contiguity network contiguity network The goal is to make the distance between contiguous state pairs small, and non-contiguous state pairs large.

[0086] Finally, the contiguity network is able to map physical reachability to embedding space distance, and becomes an efficient reachability evaluation agent.

[0087] STEP4. Calculate the contiguity loss using the contiguity network.

[0088] The specific implementation is as follows:

[0089] S401. Define the contiguity loss : Use the trained contiguity network to constrain the behavior of the high-level policy selection subgoal to its approximate shortest transition distance. Define the contiguity loss as , where is the current position coordinate, and here is the distance threshold in the embedding space, which can be determined by the distance distribution of the contiguity / non-contiguity pairs in the pre-training phase; STEP5. Collect landmarks and calculate landmark loss.

[0090] The specific implementation is as follows:

[0091] S501. Sample landmarks: Run the farthest point sampling (FPS) algorithm to select a set of coverage landmarks

[0092] from the experience replay pool. Run the random network distillation (RND) algorithm to select a set of target landmarks from the un-covered area;

[0093] S502. Plan the global path: Refer to the attached Figure 4 , identify the centroid of the un-covered area as the temporary final goal , construct a temporary graph with the current position as the starting point, the landmarks as intermediate nodes, and the temporary final goal as the end point. The temporary graph contains the current state, all sampled landmarks, and the temporary final goal. Use the contiguity network to estimate the edge weights, and run the value iteration algorithm to solve the shortest path from the current state to the temporary final goal on the graph, as the optimal path from , and determine the first landmark on the path​ As strategic direction;

[0094] S503. Generate sub-goals: According to Calculate pseudo-landmarks towards strategic direction ,

[0095] S503. Define landmark loss : Use pseudo-landmarks , Landmark loss is defined as ;

[0096] STEP6. Integrate total loss, guide high-level policy to generate sub-goals.

[0097] Integrate total loss: Combine the above loss terms with standard reinforcement learning loss (such as TD error) to form the final total loss function of high-level policy ;

[0098] Where the loss weight , The two sets of hyperparameters are used to control the strength of local feasibility constraints and global guidance, and their values can be balanced through experimental tuning to balance exploration and utilization.

[0099] Finally, the complete optimization direction of the high-level policy can be determined, which integrates task rewards, local feasibility constraints, and global strategic guidance.

[0100] The specific implementation is:

[0101] S601. Low-level execution: Low-level policy Receive sub-goals given by high-level And local observations , Generate specific actions And interact with the environment;

[0102] S602. Data storage: Store the generated experience tuples Into the experience replay buffer;

[0103] S603. Policy update: Sample data from the buffer, update the parameters of the high-level policy network And low-level policy network According to the loss function defined in step S104 (for high-level) and standard TD error (for low-level).

[0104] STEP7. The decision-making-execution-learning loop composed of steps S105 and S106 is executed continuously until the task is completed, where the task is terminated by monitoring the task indicators (such as coverage rate or remaining power). After the task is terminated, the system completes a complete coverage path planning.

[0105] In summary, the landmark-guided global exploration mechanism constructed by the method can provide clear and long-term strategic direction for high-level strategy, effectively overcome the local optimal trap, enable the agent to learn the strategy covering the entire complex environment faster, significantly speed up the convergence of the model, and obtain higher final coverage. Meanwhile, the method decouples the local feasibility constraint and the global strategic guidance, can form a clear two-stage optimization framework, and each module (such as the adjacency network and the landmark sampling strategy) can be optimized and replaced independently, has good flexibility and scalability, and can adapt to different complexity of coverage planning tasks.

[0106] In addition, in order to facilitate the understanding of the present application, in the further interpretation of the present embodiment, the main symbols and terms used in the present specification and their meanings are listed as follows:

[0107] is denoted as an adjacency network, which is used as a neural network model, and the parameter set thereof is is used to map the environment state coordinates to the embedding space, and the distance in the space represents the reachability between states;

[0108] is denoted as a high-level strategy, which is used as a high-level decision unit in hierarchical reinforcement learning (HRL), and is responsible for generating a subgoal;

[0109] is denoted as a low-level strategy, which is used as a low-level execution unit in hierarchical reinforcement learning (HRL), and is responsible for executing a subgoal within a given time;

[0110] is denoted as a time step frequency, which is used as the frequency of generating a subgoal by the high-level strategy (i.e. one per low-level time step), and is also the upper limit of the time step difference defining the adjacency state pair;

[0111] is denoted as a subgoal, which is the goal state generated by the high-level strategy at time step ;

[0112] is denoted as a local cropping function, which is used to crop a local view of size from the original map ;

[0113] is denoted as a global down-sampling function, which is used to average pool the original map with a window to generate a low-resolution global view;

[0114] denotes the current position of the agent, i.e. the coordinate of the agent at time step ;

[0115] denotes the high-level observation, i.e. the low-resolution global environment map received by the agent at time step ;

[0116] denotes the low-level observation, i.e. the high-resolution local environment map received by the agent at time step ;

[0117] denotes a pair of states, i.e. two points in the trajectory of the agent;

[0118] denotes the contrastive loss function, used to train the adjacency network , so that the adjacency states are close in the embedding space and the non-adjacency states are far apart;

[0119] denotes the adjacency loss, which is generally a hinge loss function, used to measure the reachability of the subgoal selected by the high-level policy and the current state in the embedding space;

[0120] denotes the approximate shortest transition distance between states calculated by the distance of the embedding vectors output by the adjacency network;

[0121] denotes the embedding space distance threshold, used to define the threshold of the adjacency loss, determined based on the distance distribution in the pre-training phase;

[0122] denotes the pseudo landmark, used as a landmark point to temporarily guide the direction of the high-level policy;

[0123] denotes the landmark loss, used to measure the closeness of the subgoal to the pseudo landmark in the embedding space;

[0124] is the L2 norm, generally representing the Euclidean distance between vectors;

[0125] denotes the final total loss function of the high-level policy, which has ;

[0126] denotes the standard reinforcement learning loss, used as the reward loss term (such as TD error) for learning the high-level policy;

[0127] weight hyper-parameter denoted as adjacency loss for balancing the strength of local feasibility constraints;

[0128] weight hyper-parameter denoted as landmark loss for balancing the strength of global guidance;

[0129] denoted as motion mapping function, for converting low-level policy action to the actual motion of the agent, updating its position;

[0130] denoted as action, for representing the specific operation performed by the agent at time step ;

[0131] denoted as environment map, containing environmental information such as obstacles, no-fly zones, etc;

[0132] denoted as target area map to be covered, for representing the area that needs to be explored and covered by the agent.

[0133] denoted as initial position of the agent;

[0134] denoted as initial battery level of the agent;

[0135] denoted as coverage landmarks, a set of landmark points selected from the boundary of the explored area through farthest point sampling (FPS), for promoting exploration;

[0136] denoted as target landmarks, a set of landmark points with high exploration value selected from the unexplored area through novelty measurement method;

[0137] denoted as temporary final target, i.e. the dynamically determined centroid of the current most needed coverage area;

[0138] denoted as selected landmark, i.e. the first landmark on the shortest path from the current state to the temporary final target as the strategic direction of the high-level policy;

[0139] denoted as experience tuple, for representing the record of the agent's interaction with the environment, including the current state, executed action, immediate reward, next state, and current sub-goal.

[0140] In a set of specific embodiment applications, there are the following application designs:

[0141] In this embodiment, the agent is set as a drone in a preset discrete grid map environment. The map performs area coverage tasks. It may include complex environmental features such as obstacles and no-fly zones (as shown in the attached document). Figure 5 (Example environment shown).

[0142] The state of an agent is determined by its current position. Altitude status and battery power Common description. The agent's positional changes are set to follow: .

[0143] in, It is a motion mapping function that maps the agent in the action space. Selected action (Including directional movements such as "East", "North", "West", "South", "Takeoff" and "Landing") are mapped to actual coordinate changes.

[0144] Landing status of the agent It is set to: if the "landing" action is performed, It becomes true (indicating a zero-altitude landing state); if the "takeoff" maneuver is executed, It becomes false (indicating that it is in cruise altitude flight mode); other actions remain unchanged. .

[0145] The agent's battery power It is set so that the battery level decreases by one unit for each action performed.

[0146] Task completion status is displayed on the map. Measured by changes in coverage, i.e., the agent's field of vision. (Set as) Add a new overlay cell within a square area.

[0147] The specific operation flow of the optimized sub-target generation method and the coverage path planning method implemented based on the method disclosed in the corresponding implementation embodiment of the present invention is as follows:

[0148] STEP 1. Obtain the system model, task definition, and environmental information of the hierarchical reinforcement learning framework.

[0149] System dynamics model and task definition: The agent's motion follows the above settings. The task objective is to maximize the target area. The coverage area is considered, while also taking into account constraints such as obstacle avoidance and power consumption.

[0150] Hierarchical Reinforcement Learning Framework: This embodiment adopts a hierarchical reinforcement learning (HRL) framework, the overall functional structure of which is shown in FIG. 1. In this embodiment, both layers use the DDQN algorithm. Figure 2

[0151] High-level policy is responsible for formulating long-term, strategic goals at a time step frequency to generate a subgoal . In this embodiment, is set to 10.

[0152] Low-level policy is responsible for executing the subgoal within a step.

[0153] STEP 2. Constructing a multi-scale observation space and adjacency network

[0154] Multi-scale map processing: provides a multi-scale observation space for the high-level policy and the low-level policy . This includes: a local cropping function : used to crop a local view of size from a pre-set map centered at the current position of the agent. In this embodiment, is set to 10; a global down-sampling function is used to average pool the pre-set map with a window size of to generate a low-resolution global view. In this embodiment, is set to 4. At each time step , the agent generates a high-level observation and a low-level observation according to the current position .

[0155] Adjacency network construction: an adjacency network is constructed (the parameter set is ), which is configured as a multi-layer perceptron (MLP) containing two hidden layers (128 neurons per layer) and outputs a 32-dimensional embedding vector. FIG. 2 illustrates the function of the adjacency network, i.e., how to represent the reachability of states in the physical space through embedding in the adjacency space. Figure 3

[0156] STEP 3. Collect trajectories and train the adjacency network

[0157] Trajectory data collection: in the environment, let the agent explore using a random policy or an expert policy to collect a large number of state transition trajectories.​​

[0158] Adjacent state definition: for any state pair in the trajectory , if the time step difference between them is less than or equal to a preset threshold , then the state pair is set as adjacent; otherwise, it is set as non-adjacent.

[0159] Adjacent network training: using a contrastive loss function is trained . The loss aims to reduce the distance between adjacent state pairs in the embedding space, while increasing the distance between non-adjacent state pairs.

[0160] Pre-training result: the trained is able to map physical reachability to Euclidean distance in the embedding space , and determine the corresponding embedding space distance threshold .

[0161] STEP4. Calculate the adjacent loss using the adjacent network

[0162] Using the trained adjacent network , the adjacent loss is calculated . Wherein, is the approximate shortest transfer distance calculated by the embedding vector distance between the current position output by and the sub-target .

[0163] This loss constrains the behavior of the high-level policy to select sub-targets to its approximate shortest transfer distance, forcing it to prefer locally reachable sub-targets.

[0164] STEP5. Collect landmarks and calculate landmark loss

[0165] First, landmark sampling is performed, including coverage-based landmark sampling : using the farthest point sampling algorithm to sample landmarks on the boundary of the explored area. In this embodiment, the number of samples is set to 20; and target-based landmark sampling : using novel metric methods such as random network distillation to sample landmarks with high exploration value from unexplored target areas. In this embodiment, the number of samples is set to 20. Identify the centroid of the unexplored area on the current global map as the temporary final target .

[0166] Construct a graph containing the union of the current state , all sampled landmarks , and . Through the trained adjacent network Calculate the weights (i.e., embedding distances) of each edge in the graph.

[0167] Solve on this graph from the current state arrive The shortest path is selected as the optimal high-level path. The first landmark on this shortest path is chosen. As a strategic direction for high-level strategy. (The following is an appendix.) Figure 4 It demonstrates the global exploration path selection process guided by landmarks.

[0168] according to The coordinate information is used to define a pseudo-landmark oriented towards the higher-level strategy direction. High-level strategy exist (especially including) Guided by ), generate a location close to the pseudo landmark. sub-targets According to pseudo-landmarks Calculate landmark loss .

[0169] STEP 6. Integrate the total loss to guide high-level strategy in generating sub-objectives.

[0170] Compare the above loss terms with the standard reinforcement learning loss. Combined, forming a high-level strategy The final total loss function: . The (adjacency loss weight) is set to 10 to strongly emphasize local feasibility constraints, ensuring that the goals generated by high-level strategies are achievable within a finite number of steps and avoiding ineffective exploration. The (landmark loss weight) is set to 1.0, which aims to provide effective global strategic guidance for high-level strategies, but without excessively restricting the flexibility of local exploration, so as to balance exploration and utilization.

[0171] Based on the loss function, sub-objective and policy update operations are performed in real time at each low-level time step. Low-level execution: low-level policy. Receive sub-objectives given by the higher-level policy and local observation Generate specific actions And interact with the environment, where: (1) Data storage: the experience tuples generated by each interaction. (2) Policy update: Sample data from the experience replay buffer and update the policy accordingly. (right ) and standard TD error (for ), Update the high-level policy network and low-level policy networks parameters. In this embodiment, a deep Q network (DDQN) is used as the base algorithm.

[0172] STEP7. The decision-execute-learn loop constituted by STEP1 to STEP6 is continuously executed until the task is completed

[0173] The task is terminated by monitoring the task indicators (such as coverage or remaining power). After the task is terminated, the system completes a complete coverage path planning.

[0174] It should be further pointed out that, in the accompanying Figure 5 The exploration trajectory of the baseline HRL method (not including the core technology of the present application) is shown in FIG. 2. As can be seen from the figure, the baseline method may have the problems of local wandering or low exploration efficiency, and the trajectory in some areas appears redundant; in the accompanying Figure 6 The exploration trajectory of the method of the present application (integrating the adjacency constraint and landmark guiding mechanism) is shown in FIG. 3. Compared with FIG. 2, the trajectory of the method of the present application shows stronger global guidance, can more efficiently traverse the complex environment, can avoid unnecessary repeated exploration, and directly moves to the key area that has not been covered. Figure 5

[0175] In addition, in the specific embodiment explanation combined with FIG. 4, FIG. 5 shows the comparison curve diagram of the final coverage performance of the method of the present application (HGRL) and the baseline HRL method. The blue curve in the figure represents the method of the present application (HGRL), and the red curve represents the baseline HRL method. From the curve trend, it can be seen that the HGRL method has a faster coverage rate in the early stage of learning, and the convergence speed is significantly better than the baseline HRL. When finally converging, the coverage rate reached by the HGRL method is also significantly higher than that of the baseline HRL, indicating that the method of the present application can more thoroughly complete the coverage task. Figure 7 Figure 7 Based on the experimental results shown in the above figures, the method of the present application is significantly better than the traditional baseline HRL method in terms of coverage rate and convergence speed, which effectively verifies the effectiveness and necessity of each step of the present application (including the multi-scale observation space, the adjacency constraint, the landmark guiding, etc.).

[0176] Based on the above-mentioned method disclosed in the present embodiment, compared with the prior art, the following significant advantages are possessed:

[0177] (1) Improve exploration efficiency: the adjacency constraint network

[0178] (1) Improve exploration efficiency: the adjacency constraint network ​​(1) Efficient filtering of a large number of locally unreachable sub-goals, avoiding invalid attempts of the underlying strategy, making each decision of the high-level strategy more feasible, thereby greatly improving the exploration efficiency. (2) Accelerate convergence and improve coverage rate: The landmark-guided global exploration mechanism constructed can provide a clear long-term strategic direction for the high-level strategy, effectively overcome local optimal traps, so that the agent can learn the strategy to cover the entire complex environment faster, significantly speed up the convergence of the model, and obtain higher final coverage rate. (3) High flexibility and scalability: The invention decouples local feasibility constraints and global strategic guidance to form a clear two-stage optimization framework, in which each module (such as the adjacency network and landmark sampling strategy) can be optimized and replaced independently, with good flexibility and scalability, and can adapt to different complexity of coverage planning tasks.

[0179] In summary, it needs to be further pointed out that although the invention is mainly described and verified for discrete grid environment, the core idea (using learnable distance metric for local constraint, using graph planning and landmark for global guidance) can be applied to a wider range of scenarios, for example: Continuous space: change the coordinate input to continuous value, the adjacency network structure remains unchanged. Nonlinear system: the bottom strategy of HRL can be replaced by a controller that can handle nonlinear dynamics. Multi-agent system: landmark information can be shared among multiple agents, or the positions of other agents can be regarded as dynamic obstacles to achieve collaborative coverage.

[0180] In another aspect, the application also discloses a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to make the processor execute the steps of the above method.

[0181] In another aspect, the application also discloses a computer device, which comprises a memory and a processor, and the memory stores a computer program, and the computer program is executed by the processor to make the processor execute the steps of the above method.

[0182] In another embodiment provided by the application, a computer program product containing instructions is also provided, which makes a computer execute the optimization sub-goal generation method and the coverage path planning method in any of the above embodiments when the computer program product is run on the computer.

[0183] It can be understood that the system provided by the embodiments of the application corresponds to the method provided by the embodiments of the application, and the explanation, examples and beneficial effects of the related content can refer to the corresponding part in the above method.

[0184] The application also provides an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus,

[0185] a memory for storing a computer program;

[0186] a processor for implementing the optimization sub-goal generation method and the coverage path planning method when executing the program stored in the memory.

[0187] The communication bus mentioned in the electronic device can be a peripheral component interconnect bus or an extended industry standard architecture bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0188] The communication interface is used for communication between the electronic device and other devices.

[0189] The memory can include a random access memory and can also include a non-volatile memory, such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0190] The processor mentioned above can be a general-purpose processor, including a central processing unit, a network processing unit, etc.; can also be a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0191] It should be further noted that the electronic device also includes a terminal device, which can also be referred to as a terminal, a user equipment, a mobile station, a mobile terminal, etc. The terminal device can be a mobile phone, a smart television, a wearable device, a tablet computer, a computer with wireless transceiver function, a virtual reality terminal device, an augmented reality terminal device, a wireless terminal in industrial control, a wireless terminal in unmanned driving, a wireless terminal in remote surgery, a wireless terminal in smart power grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, etc. The embodiments of the present application do not limit the specific technology and specific device form of the terminal device.

[0192] In the above embodiments, all or part can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired or wireless manner. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. containing one or more available media sets. The available media can be magnetic media, optical media, or semiconductor media, etc.

[0193] The above merely provides the preferred embodiments of the present application, but should not be used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

[0194] In addition, it should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, motion condition, etc. between components in a certain posture, and if the certain posture changes, the directional indications also change accordingly.

[0195] In addition, if the embodiments of the present application involve descriptions of "first", "second", etc., the descriptions of "first", "second", etc. are only for description purposes, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the features limited by "first", "second" can explicitly or implicitly include at least one of the features. In addition, the meaning of "and / or" appearing throughout the text includes three parallel solutions. Taking "A and / or B" as an example, it includes A solution, or B solution, or A and B solutions. In addition, in the embodiments of the present application, "multiple" means more than two. In addition, the technical solutions of each embodiment can be combined with each other, but it must be based on the fact that a person of ordinary skill in the art can realize it, and when the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist, nor is it within the protection scope required by the present application.

Claims

1. An optimization sub-goal generation method for optimizing path sub-goals of an agent's movement trajectory, characterized in that, Comprising: Defining a set of hierarchical reinforcement learning framework with high-level strategy and low-level strategy; training and using a set of parameters on a set of multi-scale maps for adjacency networks for mapping a specified environment state coordinate to a set of embedding spaces, where the Euclidean distance in the embedding spaces is used to characterize reachability between states; Utilizing the adjacency network in a hierarchical reinforcement learning framework To the high-level policy The generated sub-goals are constrained, and the locally reachable sub-goals are preferentially selected; The abutment network The training procedure of the abutment network comprises: Collecting state transition trajectories of any set of agents in a specified environment using random policy or expert policy; For any state pair in the state transition trajectory if the time step difference of the state pair is less than or equal to a preset threshold , the state pair is set as adjacent, otherwise, set as non-adjacent; Using the annotated data, the adjacency network is trained by a contrastive loss function training the adjacency network reducing distances between pairs of states that are adjacent in the embedding space and increasing distances between pairs of states that are not adjacent in the embedding space; For the trained adjacency network , a set of total loss functions is used to constrain the sub-goals generated by the high-level policy, which specifically includes: L1. Utilizing a trained adjacency network the behavior constraints of the high-level policy selection subgoal the approximate shortest transition distance, there is an adjacency loss in the form of a hinge loss function, defined as: wherein, is the inter-state approximate minimum transition distance calculated from the embedding vector distance output by the adjacency network, k is the adjacency step threshold value; L2. Define a pseudo landmark as a temporary guiding direction, where there is a landmark loss defined as: wherein, is the L2 norm, landmark loss for measuring proximity of a sub-target to pseudo-landmarks in embedding space; proximity; L3. The above loss terms are combined with a standard reinforcement learning loss In combination, forming the final total loss function for the high-level policy, defined as: where, is denoted as the standard reinforcement learning loss, the reward loss term for high-level policy learning, is denoted as the adjacency loss is denoted as the weight hyperparameter for balancing the strength of the local feasibility constraint, is denoted as the landmark loss is denoted as the weight hyperparameter for balancing the strength of the global guidance.

2. The method of claim 1, wherein, In the hierarchical reinforcement learning framework: the high-level policy at a time step generates a sub-goal , and the low-level policy is responsible for executing the sub-goal within steps.

3. The method of claim 2, wherein the optimization sub-object generation method is characterized by, The multi-scale map is defined with a local cropping function and a global down-sampling function, wherein: Local cropping function , for cropping a local view of a size from the centered pre-set map ​​ Global down-sampling function , for generating a low-resolution global view by averaging-pooling the preset map with a window of . Each time step for any group of agents According to the current position in the preset map Generate high-level observations and low-level observations .

4. A coverage path planning method characterized by, Comprising: S1. Adopting double strategy to sample landmark points in the preset map region; S2. Dynamically determining one or more current most-needed coverage area centroids as temporary final targets according to the current global map uncovered areas; S3. Generating a guide sub-target based on the optimization sub-target generation method of any one of claims 1-3; S4. Looping the above S1-S3 steps, judging whether the task is terminated by specifying the monitoring task indicators, and the task termination is a complete coverage path planning; The double strategy of sampling landmarks in the S1 step specifically includes: Based on coverage landmark sampling, using the farthest point sampling algorithm to sample landmarks on the boundary of the explored area; Based on target landmark sampling, using a metric method such as random network distillation to sample landmarks with specified exploration value in the uncovered target area.

5. The coverage path planning method of claim 4, wherein, The S3 step specifically includes the following operation process: Based on the optimized sub-target generation method according to any one of claims 1-3, during the execution of the high-level strategy, a guide sub-target is generated to dynamically identify the centroid of currently uncovered areas as a temporary final target. ; constructing a graph comprising the current state, all sampled landmarks and the temporary final goal, and traversing the graph by the adjacency network computing edge weights; solving on the map the shortest path from the current state to the temporary final goal as the optimal high-level path and selecting the first landmark of this path as a high-level strategy direction; based on a set of pseudo landmarks oriented towards the high-level policy direction and a landmark loss term that guides the high-level policy to generate sub-goals that are close to the pseudo landmarks.

6. A computer-readable storage medium, characterized in that, A computer program is stored, and the computer program is executed by a processor to make the processor execute the steps of the optimization sub-target generation method of any one of claims 1 to 3 or the steps of the coverage path planning method of any one of claims 4 to 5.

7. A computer device, characterized by Comprising a memory and a processor, the memory stores a computer program, and the computer program is executed by the processor to make the processor execute the steps of the optimization sub-target generation method of any one of claims 1 to 3 or the steps of the coverage path planning method of any one of claims 4 to 5.

Images