Unmanned aerial vehicle-unmanned vehicle cooperative path planning method based on deep reinforcement learning

Abstract

The application discloses a UAV-UAV cooperative path planning method based on deep reinforcement learning, and relates to the technical field of path planning. The application realizes the cooperative decision and path optimization of cross-platform multi-agent in a complex task scene by modeling the state and cooperative path planning of the UAV and the UAV task in a unified space grid, and combining the state value evaluation mechanism in deep reinforcement learning. Through task segment splitting and the construction of the coupling relationship between state nodes, the application effectively extracts the key frame state and inter-frame cooperative information in the task execution process, embeds a reinforcement learning framework on this basis, and significantly enhances the perception ability of path decision to task logic and spatial dynamics. By calculating the frame-pair structure path matrix as a guide factor for policy learning and value updating, the application can realize efficient path integration and resource scheduling of the UAV-UAV in multiple tasks and multiple time periods.

Description

Technical Field

[0001] This invention relates to the field of path planning technology, and in particular to a UAV-vehicle cooperative path planning method based on deep reinforcement learning. Background Technology

[0002] With the continuous development of unmanned systems technology and breakthroughs in intelligent algorithms, collaborative operations between drones and unmanned vehicles (UAVs) are becoming an important means to improve execution efficiency and operational quality in complex task scenarios. Traditional path planning methods, often centered on a single vehicle, struggle to address the needs of resource coordination, task allocation, and path optimization among multiple agents. In recent years, artificial intelligence, particularly deep reinforcement learning, has demonstrated superior performance in policy search and optimal path decision-making in high-dimensional state spaces and has been gradually applied to the collaborative scheduling and path optimization of heterogeneous multi-agent systems. However, in UAV-UAV collaborative operations, due to their different motion characteristics, operational areas, and energy and task constraints, existing methods still face significant challenges in handling complex spatiotemporal constraints and improving collaborative path planning efficiency. In particular, there is a lack of systematic modeling and efficient training strategies for task coupling, temporal consistency, and state decision guidance.

[0003] For example, CN114779758A discloses a path planning method for a heterogeneous collaborative system of UAVs and unmanned vehicles. This method mainly uses a dynamic UAV assembly strategy to achieve the synchronous completion of transfer tasks and operation tasks, thereby compressing the overall operation time and improving system efficiency. During path planning, it emphasizes parallel transfers and visits between task points, but it does not uniformly model the collaborative path structure of UAVs and unmanned vehicles, nor does it introduce deep reinforcement learning for strategy optimization of key state nodes. Furthermore, this scheme does not consider the transfer density characteristics of keyframe state sequences in the spatial grid, making it difficult to capture the spatial and temporal coupling patterns of operation tasks. It also lacks guiding evaluation indicators for the collaborative relationships between states, leading to path redundancy and scheduling conflicts in complex task segments.

[0004] Secondly, CN115079701A proposes a path planning method for collaborative operation between unmanned vehicles (UAVs) and drones. By establishing a set of target points and stopping points, a path planning model is constructed and an optimization objective function is solved to achieve efficient execution of collaborative inspection tasks. While this method improves the solution efficiency and comprehensively considers charging and waiting constraints, it focuses on cost optimization at the path level, neglecting the dynamic changes of key nodes in the execution of the task segment by both the UAV and the UAV, as well as the spatial interaction characteristics between collaborative paths. Furthermore, this method does not introduce a reinforcement learning policy network for end-to-end policy optimization, failing to fully utilize the state evolution patterns and task label information during task transfer, making it difficult to achieve robust and highly collaborative path output results in large-scale complex operational environments. Summary of the Invention

[0005] In view of the problems existing in the collaborative path planning technology of drones and unmanned vehicles, this invention is proposed.

[0006] Therefore, the problem to be solved by the present invention is how to guide the sampling priority of state pairs by constructing key frame state sequences and frame-pair structure path matrices, and optimize and update the policy network by combining the spatial continuity and task consistency of frame-pair cooperative nodes. This solves the problems of insufficient utilization of path structure information and lack of guidance in state decision-making in existing cooperative path optimization methods.

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0008] In a first aspect, the present invention provides a method for UAV-Vehicle cooperative path planning based on deep reinforcement learning, comprising: constructing a bidirectional state graph of UAV and Vehicle according to the spatial grid of the operation area and the task requirements; splitting multiple task segments according to the task requirements and extracting the state sequence with the highest transition probability in each task segment as the task key frame state sequence; mapping the key frame state sequence in the spatial grid, extracting state intersection nodes with temporal overlap and task coupling characteristics, and encoding them to form a frame-pair structure path matrix; using the frame-pair structure path matrix as a state transition guiding factor and embedding it into the state-action value function Q(S,A) in reinforcement learning, and updating the value network weights in combination with the path transition flow density between key frames; backtracking and optimizing the task key frame state sequence and frame-pair structure path matrix according to the updated value network weights, and outputting a set of cooperative paths for UAV and Vehicle.

[0009] As a preferred embodiment of the UAV-Vehicle cooperative path planning method based on deep reinforcement learning described in this invention, the bidirectional state graph includes UAV state nodes and Vehicle state nodes identified by the grid center point; the construction of the bidirectional state graph includes: selecting the center point coordinates of each grid cell and mapping them to a set N of UAV state nodes. u and the set of autonomous vehicle state nodes N g Based on the grid topology, set the adjacent connection edge set between state nodes, constrain the direction to be passable, and assign direction weight matrices to each; for N u With N g If there are node pairs in the node set whose spatial overlap and task coupling are both greater than a set threshold, a cross-connection edge set is constructed to form a cross-domain inter-frame connection. The adjacent connection edge set and the cross-connection edge set are combined to form a bidirectional state graph, in which all edges are enabled bidirectionally by default, and the direction weight determines the priority migration direction.

[0010] As a preferred embodiment of the UAV-Vehicle cooperative path planning method based on deep reinforcement learning described in this invention, the extraction of state cross nodes includes: under the time window sliding mechanism, scanning all state nodes in adjacent time steps in the key frame state sequence, judging the task coupling based on the task segment label and work area number to which the state node belongs, and selecting state node pairs with a spatial distance lower than a set distance threshold as state cross node pairs.

[0011] As a preferred embodiment of the UAV-Vehicle cooperative path planning method based on deep reinforcement learning described in this invention, the formation of the frame-pair structured path matrix includes: evaluating the temporal overlap of the state crossover node pairs, determining whether the task persistence is in the same scheduling cycle between adjacent key frames, and constructing a set of frame-pair cooperative node pairs; uniformly nesting and encoding the spatial coordinates, task segment labels, and time indices of each state node in the frame-pair cooperative node pairs to construct the frame-pair structured path matrix.

[0012] As a preferred embodiment of the UAV-Vehicle cooperative path planning method based on deep reinforcement learning described in this invention, the state-action value function Q(S,A) embedded in reinforcement learning includes: sequentially encoding the starting state node and the next frame state node in each pair of frame-pair cooperative node pairs in the frame-pair structure path matrix into state pairs, constructing a set of state pairs, and uniformly mapping them into state embedding units in the state-action value function Q(S,A).

[0013] As a preferred embodiment of the UAV-Autonomous Vehicle Cooperative Path Planning Method based on Deep Reinforcement Learning described in this invention, the following steps are taken: updating the value network weights includes: traversing all frame-pair cooperative node pairs in the frame-pair structure path matrix, extracting the spatial coordinate differences and task segment labels corresponding to adjacent time index frames; combining the spatial coordinate differences and task segment labels to form a state transition factor vector, which is then input into the state embedding layer of the value network to capture the spatial dynamic characteristics of state transitions under different task segments; and calculating the path number density per unit space under different task segments based on the frame-pair structure path matrix between adjacent key frames in the key frame state sequence, using this as a path transition flow density index, and adjusting the sampling weights according to the path number density during the value network training sample sampling.

[0014] As a preferred embodiment of the UAV-Vehicle cooperative path planning method based on deep reinforcement learning described in this invention, the backtracking optimization of the task keyframe state sequence and frame-pair structure path matrix includes: segmenting and sorting each state node in the keyframe state sequence with the updated value network weights; re-labeling the priority of state nodes within each task segment according to the sorting results to form a keyframe state sequence with segment priority labels; under the segment priority labels, backtracking the frame-pair structure path matrix in each task segment, extracting the minimum value path between state cooperative node pairs, and labeling the state intersection nodes within the path segment to generate preliminary cooperative path fragments within the task segment; concatenating the preliminary cooperative path fragments within each task segment according to the time index sequence; and filtering out path fragments with inconsistent tasks or time breaks according to the task label continuity and time span tolerance rules between nodes in the frame-pair structure path matrix to obtain a continuous and effective frame-pair cooperative path sequence.

[0015] As a preferred embodiment of the UAV-Vehicle cooperative path planning method based on deep reinforcement learning described in this invention, the output UAV-Vehicle cooperative path set includes: grouping the UAV state nodes and Vehicle state nodes involved in the cooperative path sequence within a task segment for consecutive valid frames, and combining the task segment label and time index to output the UAV path set and Vehicle path set as the final cooperative path set.

[0016] In a second aspect, the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, wherein the computer program instructions, when executed by the processor, implement the steps of the UAV-unmanned vehicle cooperative path planning method based on deep reinforcement learning as described in the first aspect of the present invention.

[0017] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program instructions are executed by a processor, they implement the steps of the UAV-unmanned vehicle cooperative path planning method based on deep reinforcement learning as described in the first aspect of the present invention.

[0018] The beneficial effects of this invention are as follows: By modeling the state of UAV and unmanned vehicle (UAV) tasks and coordinating path planning under a unified spatial grid, and combining the state value evaluation mechanism in deep reinforcement learning, this invention achieves cross-platform multi-agent collaborative decision-making and path optimization in complex task scenarios. Through task segmentation and the construction of coupling relationships between state nodes, this invention effectively extracts key frame states and inter-frame collaborative information during task execution. Based on this, a reinforcement learning framework is embedded, significantly enhancing the path decision-making's ability to perceive task logic and spatial dynamics. By calculating the frame-pair structure path matrix as a guiding factor for policy learning and value updating, this invention can achieve efficient path integration and resource scheduling of UAVs and UAVs under multiple tasks and time periods. In summary, this invention significantly improves the rationality, continuity, and scheduling efficiency of multi-agent collaborative path planning, and is suitable for collaborative operation tasks of unmanned systems in large-area, complex terrain, or high-frequency scheduling scenarios. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of a drone-vehicle cooperative path planning method based on deep reinforcement learning. Detailed Implementation

[0021] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0022] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0023] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0024] As mentioned in the technical background above, traditional path planning methods often focus on a single agent, making it difficult to address the needs of resource coordination, task allocation, and path optimization among multiple agents. In recent years, artificial intelligence, particularly deep reinforcement learning, has demonstrated superior performance in policy search and optimal path decision-making in high-dimensional state spaces, and has been gradually applied to the collaborative scheduling and path optimization of heterogeneous multi-agent systems. However, in UAV-vehicle collaborative operations, due to the different motion characteristics, operating areas, and energy and task constraints of the two, existing methods still face significant challenges in handling complex spatiotemporal constraints and improving the efficiency of collaborative path planning. In particular, there is a lack of systematic modeling and efficient training strategies for task coupling, temporal consistency, and state decision guidance.

[0025] Figure 1 This is a flowchart of a drone-vehicle cooperative path planning method based on deep reinforcement learning according to an embodiment of the present invention. Figure 1 As shown, the UAV-Vehicle cooperative path planning method based on deep reinforcement learning includes:

[0026] S1: Based on the spatial grid of the work area and the task requirements, construct a two-way state diagram for UAVs and unmanned vehicles. Based on the task requirements, split the task into multiple task segments and extract the state sequence with the highest transition probability in each task segment as the task key frame state sequence.

[0027] S1.1: Rasterize the work area to generate a spatial grid with equal spacing and spatial resolution, and construct a basic set of state nodes containing UAV state nodes and unmanned vehicle state nodes at the grid center point.

[0028] It should be noted that conventional methods often use equilateral quadrilateral or hexagonal cells in spatial grids to cover the target area and simplify path calculation. This invention uses an equilateral rectangular grid structure to meet the platform navigation system's compatibility requirements for ground georeferenced references, and sets a uniform spatial resolution parameter to control the actual spatial size covered by each grid cell.

[0029] Based on this grid, sets of UAV status nodes and sets of unmanned vehicle status nodes are constructed respectively. In specific operations, the center point of each grid is projected onto the spatial operation level of the UAV and the unmanned vehicle respectively, and attributes such as node type and task distribution identifier are marked.

[0030] To support subsequent task segmentation and path optimization operations, the entire work area is further divided into several spatially continuous and logically independent sub-regions based on the gridding, which are defined as work areas and numbered.

[0031] S1.2: Combining task requirement indicators, the work area is semantically divided according to task attributes, the distribution range of each type of task in the grid is defined, and each task is logically segmented in grid unit to generate a task segment sequence.

[0032] This invention supports the parallel existence of multiple types of tasks, such as inspection tasks, sampling tasks, and path coverage tasks. First, the task requirement document is parsed to extract key indicators, such as required operation frequency, time window, and spatial distribution parameters, forming a task semantic vector. Then, based on this task semantic vector and raster information, matching calculations are performed to map each type of task to a corresponding spatial region, and each region is marked with a grid cell as the smallest unit.

[0033] Furthermore, the principles for constructing task segments mainly include: first, spatial connectivity, meaning that adjacent grids should have continuous passage capability in physical space; second, task consistency, meaning that nodes within the same task segment should have consistent task semantic labels; and third, platform compatibility, meaning that task segments should be efficiently executed by the specified platform. Based on these rules, several coherent sub-regions are divided within the spatial distribution area of ​​each type of task, and each sub-region is defined as a task segment, ultimately generating a sequence of task segments.

[0034] S1.3: Based on the UAV state nodes and unmanned vehicle state nodes, construct a bidirectional state graph with directional constraints and cross-connection relationships, and embed time synchronization tags in the graph for subsequent frame-to-pair structure matching. The specific operation steps include the following:

[0035] S1.3.1: In the spatial grid set, select the center point coordinates of each grid cell and map them to the UAV state node set N. u and the set of autonomous vehicle state nodes N g It also marks the job area number and task segment label to which the status node belongs.

[0036] S1.3.2: In N u With N g Within the node set, adjacent connection edges are set between nodes according to the grid topology, with the constraint direction being the traversable direction (such as flight path wind direction or terrain accessibility), and each direction is assigned a weight matrix. For example, first, it is determined whether each node and its four neighboring nodes (up, down, left, and right) have spatial reachability, such as whether a drone is restricted to a no-fly zone or an unmanned vehicle is restricted by road breaks, etc. Under the premise of satisfying reachability, connection edges are constructed, and the direction weight matrix is ​​set according to parameters such as heading offset and terrain slope.

[0037] S1.3.3: For N u With N gIf there are node pairs in the node set whose spatial overlap and task coupling are both greater than a set threshold, a cross-connection edge set is constructed to form a cross-domain inter-frame connection.

[0038] S1.3.4: A bidirectional state graph is formed by combining the adjacent connection edge set and the cross connection edge set. All edges are enabled bidirectionally by default, but the direction weight determines the priority migration direction, forming an adjustable state transition graph.

[0039] S1.4: Traverse each task segment sequence, extract the state sequence from the bidirectional state graph based on the path with the maximum transition probability, and set it as the task key frame state sequence.

[0040] Specifically, each task segment is traversed to identify the subset of state nodes covered by the task segment and construct a subgraph in the state graph. In the subgraph, starting from the starting node of the task segment, the optimal path algorithm (such as Dijkstra's algorithm or a variant of dynamic programming, etc., is run using the maximum transition probability criterion) to search for the path with the highest probability from the starting point to the end point, which is the most likely task progression trajectory. All state nodes contained therein constitute the keyframe set.

[0041] S2: Based on the keyframe state sequence, it is mapped in the spatial grid, extracts the state intersection nodes with time overlap and task coupling characteristics, and encodes them to form a frame-pair structure path matrix.

[0042] S2.1: Map the center point coordinates of each frame in the task keyframe state sequence to the corresponding spatial grid cell number, and mark the keyframe index and task segment label in the spatial grid node set in chronological order.

[0043] S2.2: Under the time window sliding mechanism, scan all state nodes in adjacent time steps in the key frame state sequence, determine the task coupling based on the task segment label and work area number to which the state node belongs, and select node pairs with a spatial distance lower than the set distance threshold as candidate node pairs for state crossover.

[0044] Specifically, in the sliding window, pairing is based on temporal proximity; then, two criteria are used to filter cross-candidate node pairs: (1) whether the task segment labels are different but belong to the same work area number; and (2) whether the spatial distance is lower than a set distance threshold. This multi-condition joint judgment can avoid erroneous matching caused by fuzzy task similarity, and ensure the physical accessibility of spatial collaboration.

[0045] In terms of operational details, spatial distance can be determined using Euclidean distance or Mahalanobis distance, without being limited to a single criterion. Furthermore, weighting factors based on terrain information, such as obstacle factors or elevation factors, are allowed. Simultaneously, to improve the efficiency of candidate node pair filtering, an inverted index table based on region ID and task label is constructed to enable fast proximity queries within a local area.

[0046] S2.3: Perform time-series overlap assessment on cross-candidate node pairs, determine whether the task persistence is in the same scheduling cycle between adjacent key frames, and construct a set of frame-to-pair cooperative node pairs as state cooperative binding units.

[0047] S2.4: The spatial coordinates, task segment labels and time indices of each node in the frame-to-pair cooperative node pair are uniformly nested and encoded to construct a frame-to-pair structure path matrix that includes task category, spatial location and time synchronization relationship.

[0048] It should be noted that the frame-pair structure path matrix is ​​defined as a three-dimensional nested tensor structure. During the encoding process, each node is converted into a vector of a fixed format and combined to form a frame-pair vector group. This nested structure preserves all the cross-semantic information between node pairs, especially the time synchronization relationship and the degree of coupling between task categories.

[0049] Compared to conventional two-dimensional state transition matrices, the frame-pair structure matrix of this invention supports dual state modeling for heterogeneous tasks, solving the problem that traditional methods cannot express the collaborative relationship between two subjects; time synchronization information is embedded in the encoding, which facilitates the rapid retrieval of collaborative paths in the scheduling graph.

[0050] S3: The frame-pair structure path matrix is ​​used as a state transition guiding factor and embedded into the state-action value function Q(S,A) in reinforcement learning. The value network weights are updated in combination with the key inter-frame path transition flow density.

[0051] S3.1: Encode the starting state node and the next frame state node in each frame-pair cooperative node pair in the frame-pair structure path matrix into a state pair in sequence, construct a set of state pairs, and uniformly map them into the state embedding unit in the state-action value function Q(S,A).

[0052] In this way, the evolution path of each pair of state nodes can be encoded into a learnable dynamic state structure, effectively overcoming the sparse update problem caused by state discretization.

[0053] S3.2: For each pair of states, extract the change values ​​of the task segment label and spatial distance in the path connection relationship under the adjacent time index, and inject them into the value network as state transition factors to fit the state transition trend within the task segment.

[0054] S3.2.1: Traverse frames - for all frames in the structure path matrix - for each pair of cooperative nodes, extract the spatial coordinate differences and task segment labels between adjacent time index frames.

[0055] S3.2.2: Combine spatial coordinate differences and task segment labels to form a state transition factor vector, and input it into the state embedding layer of the value network to capture the spatial dynamic characteristics of state transitions under different task segments.

[0056] In the value network, the state embedding representation is adjusted by using the state transition factor vector, and the differentiated modeling of the migration pattern is realized by using task segment labels, thereby improving the network's fitting accuracy of the state migration trend within the task segment.

[0057] It should be noted that, to further improve the fitting ability of the state-action value function Q(S,A) to the state transition trend of keyframes, for each state pair, the Euclidean spatial distance between the two states is extracted from the frame-pair structure path matrix, and the corresponding task segment is labeled. Simultaneously, based on the temporal index difference and the task segment identifier, a state transition factor vector is encoded and injected as additional input into the state embedding layer of the Q network. This guides the network to focus on the spatial continuity and rhythmic features of state evolution within the task segment.

[0058] It should be noted that the state transition factor vector not only serves as an input feature for modeling state transition trends, but also participates in the gradient backpropagation of the loss function during training, adjusting the network's ability to dynamically perceive transitions in each task segment. Compared to conventional state encoding, this mechanism incorporates a coupled modeling strategy of spatial change rate and task segment semantic information, significantly improving the network's generalization ability and transition fitting accuracy in multi-task scenarios.

[0059] In this way, the method essentially introduces structural awareness into the state transition process, enabling the value network to not only understand the transition probabilities between states, but also to discern the evolutionary patterns of transition paths under task segments.

[0060] S3.3: Based on the frame-pair structure path matrix between adjacent frames in the keyframe state sequence, calculate the path number density per unit space under different task segments, which serves as the path transfer flow density index. Adjust the sampling weights according to the path number density during the value network training sample sampling, including the following steps:

[0061] S3.3.1: Based on the keyframe state sequence, count the number of spatial paths within different task segments in the frame-to-frame structure path matrix of each pair of adjacent frames, and calculate the corresponding spatial grid volume. Divide the number of spatial paths by the corresponding spatial grid volume to obtain the path number density, which serves as the path flow intensity index for that task segment. This index can dynamically reflect the concentration of state transitions in different regions of the task segment.

[0062] S3.3.2: Using path number density as an indicator of path transfer flow density, a task segment weighted sampling strategy is constructed. The sampling weights are adjusted according to path number density during the sampling of training samples in the value network to enhance the learning attention of high flow density task segments.

[0063] Specifically, let the basic sampling probability of the training samples be P0, then for each sample belonging to the task segment T... k The actual sampling weights Here, f(·) is the normalization function used to ensure the stability of the overall sampling distribution. This strategy can effectively increase the probability of samples in high flux density areas being selected during training, guiding the network to concentrate learning resources on regions with higher transfer complexity.

[0064] S3.3.3: Dynamically update the distribution of training samples for the value network based on the sampling weight adjustment strategy to ensure that the network achieves better value function fitting performance in active areas of the spatial path and key task segments.

[0065] As training iterations continue, the path density changes with the introduction of new samples, thus requiring a dynamic adjustment mechanism for the sampling weight strategy. This invention, based on a sliding window mechanism, recalculates and adjusts the actual sampling weights after each training round, ensuring the value network maintains sufficient focus on key regions throughout the training cycle, thereby achieving a dynamic balance between global generalization ability and local accuracy.

[0066] It should be noted that the state transition factor vector constructed in S3.2 does not have active sampling capability. Instead, it needs to be combined with the path flow density index in S3.3 to dynamically adjust which states are given priority for training. The unit path density within the task segment in S3.3 essentially reflects the information-rich areas within the task segment. These areas are more likely to carry high-complexity state transition features and are the key areas for value function learning.

[0067] S3.4: During the value network update process, the spatial continuity and task label consistency between frame-to-pair cooperative nodes are used as value feedback indicators. The weights are adjusted by combining the path transfer flow density to optimize the action value estimation of state embedding in each task segment.

[0068] Specifically, the spatial displacement trend of frame-to-pair collaborative nodes is calculated, and state connections without continuous physical trajectories are filtered out. Simultaneously, the task segment labels of state nodes are compared, and only state pairs with consistent labels are retained for action value estimation. Based on this, dynamic weighting is applied according to the path transition flow density index, with high-density state pairs receiving higher feedback weights and low-density state pairs receiving relatively lower feedback values. These three factors form a jointly regulated value feedback system, providing a directional, density-aware optimization gradient to the action value estimation network within the task segment at the state embedding layer, significantly improving the accuracy of critical path learning and the action discrimination capability within the task segment.

[0069] S4: Based on the updated value network weights, backtrack and optimize the task key frame state sequence and frame-pair structure path matrix, and output the collaborative path set of UAV and unmanned vehicle.

[0070] S4.1: The state nodes in the keyframe state sequence are segmented and sorted using the updated value network weights. The priority of the state nodes in each task segment is re-labeled based on the sorting results, forming a keyframe state sequence with segment priority labels.

[0071] First, for the keyframe state nodes in each task segment, the maximum action value under different action contexts is obtained using the trained state-action value function Q(S,A). All state nodes within the task segment are set as follows: The values ​​are sorted in descending order to obtain a priority sorting sequence; then, each state node is assigned a segment priority label to identify the execution priority weight within the task segment, forming a labeled keyframe state sequence.

[0072] This priority sorting not only provides a reference for path backtracking, but also ensures the orderly scheduling of subsequent path splicing tasks.

[0073] S4.2: Under the segmentation priority label, backtrack the frame-pair structure path matrix in each task segment, extract the minimum value path between state cooperative node pairs, and mark the state intersection nodes within the path segment to generate preliminary cooperative path fragments within the task segment.

[0074] Without using priority labels in S4.1, select several frame-pair collaboration nodes with the highest priority from the key frame state set of the task segment, and retrieve all feasible path sets between them in the frame-pair structure path matrix; for each path, select the path with the minimum path value as the collaboration optimal path based on the cumulative value of each state node in the path and the overall value of the defined path; and mark the multi-role state intersection nodes (i.e., nodes that exist in both the UAV and the unmanned vehicle state space) that appear in the path as interaction anchor points within the task segment, which are used for splicing subsequent path segments and task linkage.

[0075] S4.3: Concatenate the preliminary collaborative path segments within each task segment according to the time index sequence. Based on the task label continuity and time span tolerance rules between nodes in the frame-pair structure path matrix, filter out path segments with inconsistent tasks or time breaks to obtain a continuous and valid frame-pair collaborative path sequence.

[0076] Ideally, the best path segments within all task segments are concatenated according to their time indices to form a candidate path sequence. To ensure the effectiveness of the concatenated paths and task consistency, two path concatenation rules are constructed:

[0077] Only adjacent path segments p are allowed k ,p k+1 Satisfy τ(p) k )=τ(p k+1 ), where τ(·) represents the task segment label where the path segment is located;

[0078] Limit the time index difference between the starting nodes of adjacent path segments to no more than a set time threshold.

[0079] The path segments that satisfy the above rules are concatenated into the final frame-to-pair cooperative path sequence to ensure spatial continuity and consistency of task execution logic.

[0080] S4.4: For consecutive valid frames, the UAV status nodes and unmanned vehicle status nodes involved in the cooperative path sequence are grouped within the task segment. Combining the task segment label and time index, the UAV path set and unmanned vehicle path set are output as the final cooperative path set.

[0081] The algorithm iterates through consecutive valid frame-pair cooperative path sequences, dividing them into UAV path node sets and unmanned vehicle path node sets based on the platform type of the nodes. Within each task segment, the node sets are grouped according to time indices to construct a task segment path set. Finally, the path sets within all task segments are aggregated to output a cooperative path set. This cooperative path set will serve as the scheduling input for UAVs and unmanned vehicles to perform cooperative tasks.

[0082] This embodiment also provides a computer device suitable for the UAV-unmanned vehicle cooperative path planning method based on deep reinforcement learning, including a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the UAV-unmanned vehicle cooperative path planning method based on deep reinforcement learning as proposed in the above embodiment.

[0083] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0084] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, it implements the UAV-autonomous vehicle cooperative path planning method based on deep reinforcement learning as proposed in the above embodiment.

[0085] In summary, this invention achieves collaborative decision-making and path optimization for cross-platform multi-agent systems in complex task scenarios by performing state modeling and collaborative path planning for UAV and unmanned vehicle (UAV) tasks within a unified spatial grid, combined with the state value evaluation mechanism in deep reinforcement learning. Through task segmentation and the construction of coupling relationships between state nodes, this invention effectively extracts key frame states and inter-frame collaborative information during task execution. Embedding a reinforcement learning framework on this basis significantly enhances the path decision-making's ability to perceive task logic and spatial dynamics. By calculating the frame-pair structure path matrix as a guiding factor for policy learning and value updating, this invention enables efficient path integration and resource scheduling for UAVs and UAVs across multiple tasks and time periods. In conclusion, this invention significantly improves the rationality, continuity, and scheduling efficiency of multi-agent collaborative path planning, and is suitable for collaborative operation tasks of unmanned systems in large-area, complex terrain, or high-frequency scheduling scenarios.

[0086] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for UAV-UAV cooperative path planning based on deep reinforcement learning, characterized in that: include: Based on the spatial grid of the work area and the task requirements, a two-way state diagram of UAV and unmanned vehicle is constructed. Multiple task segments are divided based on the task requirements, and the state sequence with the highest transition probability in each task segment is extracted as the task key frame state sequence. Based on the key frame state sequence, mapping is performed in the spatial grid, and state intersection nodes with time overlap and task coupling characteristics are extracted and encoded to form a frame-pair structure path matrix. The frame-to-pair structure path matrix is ​​used as a state transition guiding factor and embedded into the state-action value function Q(S, A) in reinforcement learning. The value network weights are updated by combining the key inter-frame path transition flow density. Based on the updated value network weights, backtrack and optimize the key frame state sequence and frame-pair structure path matrix of the task, and output the collaborative path set of UAV and unmanned vehicle; The bidirectional state diagram includes UAV state nodes and unmanned vehicle state nodes with the grid center point as the location identifier. The construction of the bidirectional state diagram includes: selecting the center point coordinates of each grid cell and mapping them to a set of UAV state nodes. and autonomous vehicle state node set Based on the grid topology, set the adjacent connection edge set between state nodes, constrain the direction to the passable direction, and assign direction weight matrices to each; and If there are node pairs in the node set whose spatial overlap and task coupling are both greater than a set threshold, a cross-connection edge set is constructed to form a cross-domain inter-frame connection. The adjacent connection edge set and the cross-connection edge set are combined to form a bidirectional state graph, in which all edges are enabled bidirectionally by default, and the direction weight determines the priority migration direction.

2. The deep reinforcement learning based UAV-UAVC cooperative path planning method of claim 1, wherein: The extraction of the state crossover node includes: Under the time window sliding mechanism, all state nodes in adjacent time steps in the key frame state sequence are scanned. The task coupling is judged based on the task segment label and work area number to which the state node belongs. State node pairs with a spatial distance lower than a set distance threshold are selected as state crossover node pairs. 3.The method of claim 2, wherein: The formation of the frame-to-pair structure path matrix includes: The temporal overlap of the state cross-node pairs is evaluated, and it is determined whether the task persistence is in the same scheduling cycle between adjacent key frames, and a set of frame-to-pair cooperative node pairs is constructed. The spatial coordinates, task segment labels, and time indices of each state node in the frame-to-pair collaborative node pair are uniformly nested and encoded to construct a frame-to-pair structure path matrix. 4.The method of claim 1, wherein: The state-action value function Q(S, A) embedded in reinforcement learning includes: The starting state node and the next frame state node in each frame-pair cooperative node pair in the frame-pair structure path matrix are sequentially encoded into state pairs to construct a set of state pairs, and uniformly mapped into state embedding units in the state-action value function Q(S, A).

5. The deep reinforcement learning based UAV-UAVC cooperative path planning method of claim 4, wherein: Updating the value network weights includes: Traverse frames - for all frames in the structure path matrix - for each pair of cooperative nodes, extract the spatial coordinate differences and task segment labels between adjacent time index frames; The spatial coordinate differences and task segment labels are combined to form a state transition factor vector, which is then input into the state embedding layer of the value network to capture the spatial dynamic characteristics of state transitions under different task segments. Based on the frame-pair structure path matrix between adjacent keyframes in the keyframe state sequence, the path number density per unit space under different task segments is calculated as a path transfer flow density index, and the sampling weight is adjusted according to the path number density in the value network training sample sampling. 6.The method of claim 1, wherein: The key frame state sequence and frame-pair structure path matrix of the backtracking optimization task include: The state nodes in the keyframe state sequence are segmented and sorted using the updated value network weights. The priority of the state nodes in each task segment is re-labeled according to the sorting results, forming a keyframe state sequence with segment priority labels. Under the segmentation priority label, backtrack the frame-pair structure path matrix in each task segment, extract the minimum value path between state cooperative node pairs, and mark the state intersection nodes within the path segment to generate preliminary cooperative path fragments within the task segment. The initial collaborative path segments within each task segment are spliced ​​together according to the time index sequence. Based on the continuity of task labels between nodes and the time span tolerance rules in the frame-pair structure path matrix, path segments with inconsistent tasks or time breaks are filtered out to obtain a continuous and valid frame-pair collaborative path sequence.

7. The deep reinforcement learning based UAV-UAVC cooperative path planning method of claim 6, wherein: The output set of collaborative paths for drones and unmanned vehicles includes: For consecutive valid frames, the UAV status nodes and unmanned vehicle status nodes involved in the cooperative path sequence are grouped within the task segment. Combining the task segment label and time index, the UAV path set and unmanned vehicle path set are output as the final cooperative path set.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the UAV-Autonomous Vehicle Cooperative Path Planning Method based on Deep Reinforcement Learning as described in any one of claims 1 to 7.

9. A computer readable storage medium having stored thereon a computer program, characterized in that: When the computer program is executed by the processor, it implements the steps of the UAV-unmanned vehicle cooperative path planning method based on deep reinforcement learning as described in any one of claims 1 to 7.

Images