Unmanned aerial vehicle cluster task planning system of task space unified view

Abstract

The invention relates to an unmanned aerial vehicle cluster task planning system of a task space unified view. The unmanned aerial vehicle cluster task planning system comprises an LLM-based high-level task allocation decision module, an SAC-based low-level path planning control module, an RAG knowledge integration-based multi-modal representation learning module and a closed-loop self-adaption and continuous improvement module. An advanced task allocation decision module performs advanced reasoning and thinking chain task decomposition by using LLM, and dynamically optimizes an agent reward function; the low-level path planning control module optimizes continuous path planning of the unmanned aerial vehicle based on SAC reinforcement learning; the multi-modal representation learning module fuses the heterogeneous perception data and the historical experience of RAG retrieval, and constructs a unified potential state representation aided decision; and each module is tightly coupled with a closed-loop self-adaptive mechanism, and self-adaptive adjustment reward signals are fed back in real time. The system effectively solves the problems of path crossing, non-uniform resource allocation and the like, improves generalization ability and decision-making precision, and realizes faster convergence and better performance in a complex scene.

Description

TECHNICAL FIELD

[0001] The present application relates to a UAV task planning technology, in particular to a UAV cluster task planning system with a unified view of task space. BACKGROUND

[0002] With the progress of science and technology and the advent of artificial intelligence, traditional equipment is rapidly developing towards unmanned. Unmanned aerial vehicles are widely used in detection and reconnaissance, search and rescue, logistics distribution, power inspection, geographic mapping and other fields because of their flexibility and high environmental adaptability. However, due to the limited individual capacity of a single unmanned node or device, it is difficult to complete the task independently in many task scenarios, so unmanned cluster cooperation gradually shows its advantages. Unmanned clusters can demonstrate more significant coordination and intelligence. The cluster can take advantage of flexible networking communication and information sharing within the cluster to expand the cluster's situation awareness and task execution capability. Compared with humans, they have higher survivability, mobility and environmental adaptability.

[0003] During the execution of the task by the unmanned cluster, task planning is a key step. Task planning mainly includes task allocation and path planning. Task allocation is a process of reasonably allocating task targets, resources and execution order in the unmanned aerial vehicle cluster according to task requirements, resource capabilities and environmental constraints. Task allocation algorithms can be divided into two categories: centralized algorithms and distributed algorithms. As a traditional typical algorithm category, the core architecture of the centralized algorithm includes a central coordinator as the system decision center. The coordinator maintains real-time data interaction with all affiliated agents through a dedicated communication link, relies on its global information processing capability to guide the negotiation process among multiple agents, and finally determines the task allocation scheme based on the overall system performance optimization principle. Centralized algorithms mainly include Hungarian algorithm, integer linear programming, genetic algorithm, particle swarm optimization, dynamic programming and simulated annealing. These algorithms rely on global calculation of the central controller, are suitable for static or small-scale scenarios, can guarantee theoretical optimality, but have limitations such as high computational complexity, strong communication dependence and insufficient dynamic adaptability. Unlike centralized algorithms, in distributed algorithms, each agent dynamically negotiates the task allocation scheme based on local information or limited global information through local communication, autonomous decision-making and coordination mechanisms, and finally realizes global target optimization. Distributed algorithms mainly include market auction mechanism algorithm, distributed Markov decision method, swarm intelligence optimization algorithm, game theory negotiation and distributed constraint optimization, which complement centralized algorithms.

[0004] Path planning refers to the process of finding an optimal or feasible motion path from the starting point to the target point in a specific environment for the UAV. The current path planning methods for UAVs can be divided into search-based path planning algorithms, sampling-based path planning algorithms, evolutionary algorithm-based path planning algorithms, and reinforcement learning-based path planning algorithms. The search-based path planning algorithms include A* algorithm, Dijkstra algorithm, D* Lite algorithm, and LPA* algorithm, which are suitable for path generation in global static environment; the sampling-based path planning algorithms include Rapidly-exploring Random Tree (RRT) and its optimized version RRT*, Probabilistic Roadmap (PRM), and Informed RRT* with adaptive sampling optimization, which deal with high-dimensional space and dynamic obstacles through random sampling; the evolutionary algorithm-based path planning methods include genetic algorithm, particle swarm optimization algorithm, and ant colony algorithm, which utilize swarm intelligence to solve multi-objective optimization problems; the reinforcement learning-based path planning algorithms include Q-learning, Deep Q Network (DQN), Actor-Critic framework, and Proximal Policy Optimization (PPO), which realize dynamic decision optimization through autonomous environmental interaction.

[0005] In the existing task planning method for unmanned cluster, first, the task is assigned to the unmanned cluster, and then the result of the task assignment is input into the path planning module to generate the required path. This processing method only solves one problem or solves two problems step by step, and does not consider the kinematic constraints of the UAV itself when assigning tasks, only considers the relationship between tasks, and the input of path planning is the result of task assignment. Although this reduces the complexity of problem solving, it does not consider the coupling relationship between task assignment and path optimization in actual tasks, resulting in problems such as path intersection and uneven resource allocation in the unmanned cluster. SUMMARY

[0006] In view of the problems of path intersection and uneven resource allocation in the existing unmanned cluster task planning system, a task space unified view unmanned cluster task planning system is proposed.

[0007] The technical solution of the present application is as follows:

[0008] A task space unified view unmanned cluster task planning system, comprising a high-level task assignment decision module based on LLM, a low-level path planning control module based on SAC, a multi-modal representation learning module based on RAG knowledge integration, and a closed-loop adaptive and continuous improvement module.

[0009] The system uses a large language model LLM in the advanced task allocation decision module for advanced reasoning and CoT task decomposition; the LLM iteratively decomposes the task into subtasks and dynamically optimizes the reward function of the agent as the conditions change; the low-level path planning control module optimizes the continuous motion path planning instructions for each UAV based on SAC reinforcement learning; at the same time, the multi-modal representation learning module fuses heterogeneous inputs into a unified latent state representation, while the RAG retrieves relevant past experiences from the UAV knowledge base, and the retrieved context information is combined with real-time sensor data to provide information for current UAV decision-making; these components are tightly coupled with the closed-loop adaptive and continuous improvement module, in which real-time feedback from the advanced task allocation decision module and the multi-modal representation learning module adaptively adjusts the reward signal;

[0010] The system contains two layers of strategies: an advanced decision layer using the PPO algorithm and CoT reasoning, and a low-level control layer based on flexible action-evaluation SAC reinforcement learning; this separation ensures that the high-level strategy allocates and coordinates task targets for the UAV swarm, while the low-level strategy optimizes the continuous motion path planning instructions for each UAV.

[0011] Further, the LLM-based advanced task allocation decision module is as follows:

[0012] In the LLM-based advanced task allocation decision module, let s t be the global state of the UAV at time t, summarize the current target, environmental factors and group semantic graph where represents the data obtained by the UAV i sensor at time t, the internal state and part of the observation value, and n is the number of UAVs; by designing a large language model LLM and training it through the PPO algorithm; let represent the high-level action at time t, and optimize the following objective:

[0013]

[0014] where L PPO represents the objective function of PPO, θ represents the parameters of the current policy network, r t (θ) represents the probability ratio, represents the advantage function, and clip(·) represents limiting r t (θ) to the range [1-ε, 1+ε]; represents the probability of selecting the high-level action t (θ) under the state s represents the probability of selecting the action under the same state by the old policy network.​

[0015] In each decision cycle, the global state s t is input to the large language model LLM, which then internally performs the chain-of-thought CoT on the semantic graph, as follows:

[0016]

[0017] where f CoT (·) denotes the hidden reasoning process of the LLM; the final text output of the large language model LLM is parsed into high-level actions

[0018] Further, the low-level path planning control module based on SAC is as follows:

[0019] In the low-level path planning control module based on SAC, after each drone i is assigned a sub-task or partial goal, the drone i generates its individual SAC policy on a finer time scale Let x i,t denote the local state from which the drone i originates and the sub-task is assigned, and u i,t denote the continuous control action of the drone i; the objective function of maximizing entropy can be represented using the SAC algorithm as follows:

[0020] J actor (φ) = E[αlogπ — (u i,t | x i,t )- Q θ (x i,t , u i,t )]

[0021] J critic (θ) = E[(Q θ (x i,t , u i,t )- y i,t ) 2 ]

[0022]

[0023] where J actor (φ) denotes the loss function of the Actor policy network, φ denotes the parameters of the Actor policy network, α is the entropy coefficient, and π φ denotes the action distribution generated by the Actor policy network; J critic (θ) denotes the loss function of the Critic value function network, θ denotes the parameters of the Critic value function network, Q θ denotes the Q value estimate of the current state-action pair, and y i,tdenotes the target Q-value; r i,t denotes the immediate reward, γ denotes the discount factor, x i,t+1 denotes the state at the next time step, u i,t+1 denotes the action at the next time step.

[0024] Further, the multi-modal representation learning module based on RAG knowledge integration is specifically as follows:

[0025] In the multi-modal representation learning module based on RAG knowledge integration, the inputs of heterogeneous sensors are fused into a unified latent state representation; on this basis, the retrieval-enhanced generation RAG module continuously retrieves relevant historical knowledge from the knowledge base; the retrieved context information is combined with real-time sensor data to provide information for the decision-making of the unmanned aerial vehicle;

[0026] Define a representation learning module R ζ which integrates external knowledge into the state representation of the agent; let h t denote the latent state of the agent at time t, which summarizes historical observations and knowledge; the input of the learning module R ζ is the previous latent state h t-1 , new observation o t and previous knowledge base and outputs the updated latent state representation h t ; the latent state update can be expressed as:

[0027]

[0028] The above formula defines the knowledge-based state update: the latent state h t of the agent is a function of the previous latent state h t-1 , the latest observation o t and the relevant knowledge retrieved from ; the discrete time step t = 1, 2, … corresponds to the sequential decision-making process in the unmanned aerial vehicle task, and the knowledge base at time t is denoted as which is used as a memory to store semantic information accumulated over time; let be represented as a set of embedding items:

[0029]

[0030] where each is a d-dimensional vector that encodes a piece of knowledge at time t; this repository contains unmanned aerial vehicle state data, environmental semantics or task context facts, all of which are embedded in a common vector space; the size N can grow or adjust as the task progresses; in order to ensure rigor, the state evolution is described in probabilistic terms; in each step, to represent the augmented state of the agent; given a new observation o t , the update from to can be represented as:

[0031]

[0032] where h t is the current latent state, h t-1 is the latent state at the previous time step, is the current knowledge base, is the knowledge base at the previous time step, o t is the current observation, and P(·) denotes a probability distribution; the above equation can be decomposed as: updating h t according to the old state, knowledge, and new observation; updating by incorporating the newly inferred knowledge; This knowledge-based representation learning is crucial for partially observable UAV environments, where enhances o t to produce a robust latent state h t ;

[0033] Knowledge summarization based on large language models:

[0034] In the retrieval-augmented generation RAG module, the agent retrieves relevant information from and provides it as context to the large language model LLM; define a query vector q t ∈R d , which comes from the agent's state or task requirements at time t; the similarity function S(q t ,x i ) measures the matching degree of each knowledge item with the query vector; let be sorted in descending order of S(q t ,x i ); the top k knowledge items retrieved are as shown in the knowledge retrieval formula:

[0035]

[0036] Then, the summarization function g(·) converts into C t , i.e., a concise knowledge summary for the large language model LLM:

[0037]

[0038] The large language model LLM will then receive additional context C t ;

[0039] Knowledge-based large language model query and update:

[0040] By introducing a query generation function f query (·), which is used to simulate the large language model generating queries automatically according to its internal reasoning state, to retrieve and dynamically update the knowledge base:

[0041]

[0042] Where is the latent context or text-based context of the large language model LLM at time t; after calculating q t , the first k knowledge items will be retrieved according to the knowledge retrieval formula, thereby providing the knowledge set Then the large language model LLM generates an output, and the action a t is generated through the strategy π:

[0043] a t =π(h t ,C t )

[0044] After obtaining the observation at time step t+1, the system infers the new knowledge increment at the current time The update function f update (·) fuses this new knowledge increment into the existing knowledge base to generate an updated knowledge state

[0045]

[0046] The knowledge base update adopts a simple set merging method and a more detailed structure alignment and fusion strategy to ensure the consistency and structure optimization of the knowledge.

[0047] Further, the closed-loop adaptive and continuous improvement module is as follows:

[0048] The closed-loop adaptive and continuous improvement module adopts a closed-loop adaptive mechanism, in which the large language model LLM continuously modifies the reward function during the entire reinforcement learning process; the large language model LLM formulates the reward structure according to the task target, and as the agent interacts with the environment and receives feedback, the large language model LLM dynamically updates the reward to ensure consistency with the changing target and conditions;

[0049] Closed-loop adaptive process:

[0050] Consider a reinforcement learning environment with state space S, action space A, and transition model p(s t+1 |s t ,a​t Let R t Let represent the reward function at time t; the closed-loop adaptive iterative process is as follows:

[0051] Step 1) Initialization: At t=0, the LLM sets the initial reward function R0 according to the task objective;

[0052] Step 2) Agent Execution and Feedback: Under policy π, the agent selects an action a. t Receive reward r t =R t (s t ,a t ), and transition to the new state s t+1 The environment will provide additional performance feedback.

[0053] Step 3) LLM Reward Update: LLM uses environment feedback to update the reward function:

[0054]

[0055] in Extract key performance indicators (KPIs) from state transitions and summarize them, while the function... Then, based on this feedback, the previous reward function R... t Adjustments will be made; in response, R will be updated by increasing the penalty for collision events and increasing the reward for achieving sub-objectives. t This increases the success rate; this will generate a new reward function R. t+1 This makes its observation performance more consistent with the mission objectives and the agent's objectives;

[0056] Step 4) Policy Improvement: The agent uses the updated reward function R t+1 To improve its strategy;

[0057] The above closed-loop cycle repeats continuously, and the reward function and strategy are continuously improved to guide the agent to achieve the best behavior under the current task objective;

[0058] Convergence analysis and non-stationary reward stability:

[0059] A step size parameter is introduced into the reward adjustment to limit the magnitude of each modification; when the Large Language Model (LLM) proposes a change Δ to the reward function, the reward function R is conservatively adjusted. t+1 Updated to:

[0060] R t+1 =R t +βΔ

[0061] Where R tThe reward function at time t, beta represents the step size, used to reduce the influence of delta, to avoid too large changes in rewards; delta represents the adjustment proposed by the large language model LLM according to the current task performance.

[0062] The beneficial effects of the present application are:

[0063] (1) Large language model driven advanced task reasoning: The present application designs a planning module driven by a large language model LLM, which uses chain-of-thought reasoning to perform task planning and decomposition. This LLM-based controller can interpret complex tasks and decompose them into subtasks for the drone swarm, while dynamically optimizing the goal and reward structure as the task progresses. This method ultimately enables the system to have an interpretable high-level decision-making process and adapt to changing goals.

[0064] (2) Multi-modal representation learning based on RAG knowledge integration: The present application develops a deep representation learning pipeline that fuses heterogeneous sensor inputs (vision, radar, depth) into a unified latent state representation. On this basis, the retrieval and augmentation generation module continuously retrieves relevant historical knowledge from the task memory database. The retrieved context information is combined with real-time sensor data to provide information for the drone's decision-making, effectively providing the agent with a form of memory and situation awareness, thereby improving generalization ability and decision-making accuracy.

[0065] (3) Closed-loop reinforcement learning with adaptive reward shaping: The present application integrates the above modules with a low-level multi-agent flexible action-evaluation SAC reinforcement learning algorithm in a closed-loop manner. The LLM and RAG modules work together to adaptively shape the reward function and subtask goals of the SAC agent during training and execution. This closed-loop design enables the drone swarm to quickly adjust its behavior in response to environmental changes or unexpected events, resulting in faster convergence speed and superior performance in complex scenarios compared to static reward reinforcement learning methods. BRIEF DESCRIPTION OF DRAWINGS

[0066] Figure 1 Figure 1 is a diagram of the present application's task space unified view of the drone swarm task planning system. DETAILED DESCRIPTION

[0067] The present application will be described in detail below in conjunction with the accompanying drawings and specific embodiments. The present embodiment is implemented on the basis of the technical solution of the present application, and gives a detailed implementation method and specific operation process, but the protection scope of the present application is not limited to the following examples.

[0068] The application provides a UAV cluster task planning system with a unified view of task space, and a specific scheme is as follows: the system comprises an LLM-based high-level task allocation decision module, an SAC-based low-level path planning control module, a multi-modal representation learning module based on RAG knowledge integration, and a closed-loop adaptive and continuous improvement module.

[0069] Traditional multi-agent UAV methods usually rely on standard reinforcement learning (RL) or hand-tuned rules, and are difficult to cope with high-dimensional perception inputs and changing task objectives. Unlike traditional methods, the system uses a large language model (LLM) for high-level reasoning and chain of thought (CoT) task decomposition in the high-level task allocation decision module. The LLM iteratively decomposes tasks into subtasks and dynamically optimizes the reward function of the agent as conditions change. The low-level path planning control module optimizes the continuous motion path planning instructions for each UAV based on SAC reinforcement learning. At the same time, the multi-modal representation learning module based on RAG knowledge integration fuses heterogeneous inputs (such as vision, radar, and depth map) into a unified latent state representation, while the retrieval-enhanced generation RAG continuously retrieves relevant past experiences from the UAV knowledge base, and the retrieved context information is combined with real-time sensor data to provide information for current UAV decision-making. These components are tightly coupled with the closed-loop adaptive and continuous improvement module, in which real-time feedback from the LLM-based high-level task allocation decision module and the multi-modal representation learning module based on RAG knowledge integration adaptively adjusts the reward signal.

[0070] The UAV cluster task planning system with a unified view of task space proposed by the application has the following structure Figure 1 As shown in the figure, the system is a hierarchical solution, comprising two layers of strategies: a high-level decision layer using proximal policy optimization (PPO) algorithm and chain of thought (CoT) reasoning, and a low-level control layer based on soft actor-critic (SAC) reinforcement learning. This separation ensures that the high-level strategy allocates and coordinates the task objectives of the UAV swarm, while the low-level strategy optimizes the continuous motion path planning instructions for each UAV.

[0071] 1. LLM-based high-level task allocation decision module

[0072] In the LLM-based high-level task allocation decision module, let s t be the global state of the UAV at time t, summarize the current target, environmental factors and group semantic graph where represents the visual, radar, depth, etc. data obtained by the sensors of UAV i at time t, the internal state such as battery, speed, etc. and part of the observation values, and n is the number of UAVs. By designing a large language model LLM and training it through the proximal policy optimization PPO algorithm. Assuming represents the high-level action (e.g. sub-task allocation) at time t, the following objective is optimized:

[0073]

[0074] where l PPO represents the objective function of PPO, θ represents the parameters of the current policy network, r t (θ) represents the probability ratio, represents the advantage function, and clip(·) represents limiting r t (θ) to the range of [1-ε, 1+ε]. represents the probability of the current policy network selecting the high-level action t under the state s . represents the probability of the old policy network selecting the action under the same state.

[0075] In each decision cycle, the global state s t is input into the large language model LLM, and then the chain of thought (CoT) is internally executed on the semantic graph, as follows:

[0076]

[0077] where f CoT (·) represents the hidden reasoning process of LLM. This chain of thought (CoT) gives an interpretable decomposition on how to adjust the sub-task priorities or target weights. The final text output of the large language model LLM is parsed into high-level actions to ensure that the entire system can dynamically adjust the high-level task allocation strategy according to the real-time group state.

[0078] 2. Low-level path planning control module based on SAC

[0079] In the low-level path planning control module based on SAC, after each UAV i is allocated a sub-task or a partial target, the UAV i generates its individual SAC policy Let x i,t represent the local state of the UAV i derived from and the allocated sub-task, and u i,t represent the continuous control action (e.g. thrust, yaw) of the UAV i. The objective function of maximizing entropy can be represented by using the SAC algorithm:

[0080] J actor (φ) = E[αlog π φ (u i,t ∣x i,t )-Q θ (x i,t ,u i,t )]

[0081] J critic (θ) = E[(q θ (x i,t ,u i,t )-y i,t ) 2 ]

[0082]

[0083] where J actor (φ) denotes the loss function of the Actor policy network, φ denotes the parameters of the Actor policy network, α is the entropy coefficient, and π φ denotes the action distribution generated by the Actor policy network. J critic (θ) denotes the loss function of the Critic value function network, θ denotes the parameters of the Critic value function network, Q θ denotes the Q value estimation of the current state-action pair, y i,t denotes the target Q value. r i,t denotes the immediate reward, γ denotes the discount factor, x i,t+1 denotes the state at the next moment, and u i,t+1 denotes the action at the next moment. Therefore, each UAV will use the immediate sensor data from to improve its control action for path planning to handle subtle changes such as obstacle avoidance or energy saving.

[0084] 3. Multi-modal representation learning module based on RAG knowledge integration

[0085] In the multi-modal representation learning module based on RAG knowledge integration, the inputs of heterogeneous sensors (such as vision, radar, and depth map) are fused into a unified latent state representation. On this basis, the retrieval-augmented generation (RAG) module continuously retrieves relevant historical knowledge from the knowledge base. The retrieved context information is combined with real-time sensor data to provide information for the decision-making of the UAV, thereby effectively providing a form of memory and situation awareness for the agent, thereby improving the generalization ability and decision-making accuracy.

[0086] Define a representation learning module R ζ, which integrates external knowledge into the agent’s state representation. Let h t denote the latent state of the agent at time t, which summarizes the history of observations and knowledge. The learning module R ζ takes as input the previous latent state h t-1 , a new observation o t , and the previous knowledge base and outputs an updated latent state representation h t . The latent state update can be expressed as:

[0087]

[0088] The above equation defines the knowledge-based state update: the latent state h t of the agent at time t is a function of the previous latent state h t-1 , the latest observation o t , and the relevant knowledge retrieved from . Discrete time steps t = 1, 2,... correspond to sequential decision-making processes in the UAV mission, and the knowledge base at time t (denoted as ) serves as a memory that stores accumulated semantic information over time. The knowledge base is represented as a set of embedding items:

[0089]

[0090] where each is a d-dimensional vector that encodes a piece of knowledge at time t. This repository contains UAV state data, environmental semantics, or mission context facts, all embedded into a common vector space. The size N can grow or adjust as the mission progresses. To ensure rigor, the state evolution is described in probabilistic terms. At each step, represents the augmented state of the agent. Given a new observation o t , the update from to can be expressed as:

[0091]

[0092] where h t denotes the current latent state, h t-1 denotes the latent state at the previous time step, denotes the current knowledge base, denotes the knowledge base at the previous time step, o t denotes the current observation, and P(·) denotes a probability distribution. The above equation can be decomposed as: (1) updating h t according to the old state, knowledge, and new observation (via R ζ); (2) updating by incorporating newly inferred knowledge This knowledge-based representation learning is crucial for partially observable UAV environments, where will enhance o t to produce robust latent states h t .

[0093] (1) Knowledge summarization based on large language models

[0094] In the retrieval-augmented generation RAG module, the agent retrieves relevant information from and provides it as context to the large language model LLM. Define a query vector q t ∈R d , which comes from the agent's state at time t or task requirements. The similarity function S(q t ,x i ) measures the degree of matching between each knowledge item and the query vector. Let be sorted in descending order of S(q t ,x i ). The top k knowledge items retrieved are as follows:

[0095]

[0096] Then, the summarization function g(·) converts into C t , i.e., a concise knowledge summary for the large language model LLM:

[0097]

[0098] The large language model LLM will then receive additional context C t , ensuring that its output is based on relevant and up-to-date task knowledge. This approach not only reduces uncertainty but also improves the factual correctness of the large language model LLM.

[0099] (2) Knowledge-based large language model query and update

[0100] By introducing a query generation function f query (·), which simulates the large language model's ability to generate queries based on its internal reasoning state, to retrieve and dynamically update the knowledge base :

[0101]

[0102] where is the latent context or text-based context of the large language model LLM at time t. Calculate q tAfterwards, the top-k knowledge items are retrieved according to the knowledge retrieval formula, thus providing a knowledge set Then the large language model LLM generates an output, producing an action a through the policy π t :

[0103] a t = π(h t , C t )

[0104] This indicates that the actions of the UAV depend on both the latent state h t and the retrieved contextual summary C t .

[0105] After obtaining the observation at time step t+1, the system infers a new knowledge increment at the current time The update function f update (·) fuses this new knowledge increment into the existing knowledge base Generating an updated knowledge state

[0106]

[0107] The knowledge base update adopts a simple set-merging approach and a more detailed structure alignment and fusion strategy to ensure the consistency and structure optimization of knowledge. By learning when and how to query or update, the agent can best manage its knowledge, thus ensuring the success of the task. This closed-loop design of timely retrieval and continuous learning of new data is very important in complex UAV tasks.

[0108] 4. Closed-loop adaptive and continuous improvement module

[0109] The closed-loop adaptive and continuous improvement module adopts a closed-loop adaptive mechanism, in which the large language model (LLM) continuously modifies the reward function throughout the reinforcement learning process. The large language model (LLM) formulates the reward structure according to the task goals, and as the agent interacts with the environment and receives feedback, the large language model (LLM) dynamically updates the reward to ensure consistency with the changing goals and conditions.

[0110] (1) Closed-loop adaptive process

[0111] Consider a reinforcement learning environment with state space S, action space A, and transition model p(s t+1 |s t ,a t ). Let R t denote the reward function at time t. The closed-loop adaptive iterative process is as follows:

[0112] 1) Initialization: At t=0, the LLM sets the initial reward function R0 according to the task objective.

[0113] 2) Agent execution and feedback: Under policy π, the agent selects an action a. t Receive reward r t =R t (s t ,a t ), and transition to the new state s t+1 The environment (or human supervisor) will provide additional performance feedback (e.g., recent success rate, number of collisions).

[0114] 3) LLM Reward Update: LLM uses environment feedback to update the reward function:

[0115]

[0116] in Extract key performance indicators (KPIs) from state transitions and summarize them (e.g., agent success rate or number of collisions), while functions... Then, based on this feedback, the previous reward function R... t Make adjustments. For example, It may return feedback indicating a low recent success rate and a high collision frequency. In response, R will be updated by increasing the penalty for collision events and increasing the reward for achieving sub-objectives. t This will increase the success rate. This will generate a new reward function R. t+1 This makes its observation performance more consistent with the mission objectives and the agent's capabilities.

[0117] 4) Policy Improvement: The agent uses the updated reward function R. t+1 To improve its strategy.

[0118] The aforementioned closed-loop cycle repeats continuously, and the reward function and strategy are constantly improved to guide the agent to achieve optimal behavior under the current task objective.

[0119] (2) Convergence analysis and non-stationary reward stability

[0120] To ensure stability during actual training, a simple control mechanism for reward updates driven by LLM in large language models is designed. A step size parameter is introduced into the reward adjustment to limit the magnitude of each modification.

[0121] Specifically, when a large language model (LLM) proposes a change Δ to the reward function (e.g., the amount of change or the direction of adjustment), the reward function R is conservatively adjusted. t+1 Updated to:

[0122] Rt+1 = R t + βΔ

[0123] wherein R t represents the reward function at time t, β represents the step size, used to reduce the influence of Δ, to avoid too large changes in rewards. Δ represents the adjustment proposed by the large language model LLM according to the current task performance.

[0124] As the number of training increases, a decay strategy is designed for β and the frequency of reward update is reduced, so that the reward adjustment amplitude gradually decreases, or even stops eventually, when the agent approaches the optimal performance. This reward update strategy using small steps and gradually decaying ensures the smoothness of the reward changes, and allows the agent policy to have enough time to adapt to each adjustment.

[0125] The above-described embodiments only express one implementation of the present application, which is described in more detail and in more detail, but it cannot be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the scope of protection of the present application patent should be subject to the appended claims.

Claims

1. A UAV swarm mission planning system for a unified view of the task space, characterized by, The system includes a high-level task allocation decision module based on LLM, a low-level path planning control module based on SAC, a multi-modal representation learning module based on RAG knowledge integration, and a closed-loop adaptive and continuous improvement module. In the high-level task allocation decision module, the large language model LLM is used for advanced reasoning and CoT task decomposition. LLM iteratively decomposes tasks into subtasks and dynamically optimizes the reward function of the agent as conditions change. The low-level path planning control module optimizes the continuous motion path planning instructions of each UAV based on SAC reinforcement learning. Meanwhile, the multi-modal representation learning module integrates heterogeneous inputs into a unified latent state representation, and the retrieval-enhanced generation RAG continuously retrieves relevant past experiences from the UAV knowledge base. The retrieved context information combined with real-time sensor data provides information for the current UAV decision-making. These components are tightly coupled with the closed-loop adaptive and continuous improvement module, in which real-time feedback from the high-level task allocation decision module and the multi-modal representation learning module adaptively adjusts the reward signal. The system contains two layers of strategies: a high-level decision-making layer using the proximal policy optimization (PPO) algorithm and CoT reasoning, and a low-level control layer based on flexible action-evaluation (SAC) reinforcement learning. This separation ensures that the high-level strategy allocates and coordinates task objectives for the UAV swarm, while the low-level strategy optimizes the continuous motion path planning instructions for each UAV.

2. The UAV swarm mission planning system of unified view of task space of claim 1, wherein, The high-level task allocation decision module based on LLM is as follows: In the advanced task allocation decision module based on LLM, let s t be the global state of the UAV at time t, summarize the current target, environmental factors and group semantic graph Where represents the data obtained by the sensor of the UAV i at time t, the internal state and part of the observation value, and n is the number of UAVs; by designing a large language model LLM and training through the proximal policy optimization PPO algorithm; suppose represents the high-level action at time t, and the following objective is optimized: where L PPO represents the objective function of PPO, θ represents the parameters of the current policy network, r t (θ) represents the probability ratio, represents the advantage function, and clip(·) represents limiting r t (θ) to the range of [1-ε, 1+ε]; represents the probability of the current policy network selecting a high-level action t under the state s ; represents the probability of the old policy network selecting the action under the same state; In each decision cycle, the global state s t As the input of the large language model LLM, then internally perform the chain of thought CoT on the semantic graph, the formula is as follows: where f CoT (·) denotes the hidden inference process of the LLM; Final text output of large language model LLM is parsed into high-level actions 3. The UAV swarm mission planning system of unified view of task space of claim 1, wherein, The low-level path planning control module based on SAC is as follows: When each drone i is assigned a sub-task or a partial goal in the low-level path planning control module based on SAC , the drone i generates its individual SAC policy on a finer time scale Let x i,t denote the local state of the drone i resulting from and the assigned sub-task, u i,t denote the continuous control action of the drone i; the objective function to optimize using the SAC algorithm can be expressed as: J actor (φ) = E[αlogπ φ (u i,t ∣x i,t )-Q θ (x i,t ,u i,t )] J critic (θ) = E[(Q θ (x i,t ,u i,t )-y i,t ) 2 ] where J actor (φ) denotes the loss function of the Actor policy network, φ denotes the parameters of the Actor policy network, and α is the entropy coefficient. φ denotes the action distribution generated by the Actor policy network; J critic (θ) denotes the loss function of the Critic value function network, θ denotes the parameters of the Critic value function network, and Q θ denotes the Q value estimate of the current state-action pair, y i,t denotes the target Q value; r i,t denotes the immediate reward, γ denotes the discount factor, and x i,t+1 denotes the state at the next time, and u i,t+1 denotes the action at the next time.

4. The UAV swarm mission planning system of unified view of task space of claim 1, wherein, The multi-modal representation learning module based on RAG knowledge integration is as follows: In the multi-modal representation learning module based on RAG knowledge integration, the inputs from heterogeneous sensors are integrated into a unified latent state representation. Based on this, the retrieval-enhanced generation RAG module continuously retrieves relevant historical knowledge from the knowledge base. The retrieved context information combined with real-time sensor data provides information for the UAV's decision-making. Define a representation learning module R ζ that integrates external knowledge into the agent's state representation; let h t denote the agent's latent state at time t, which summarizes historical observations and knowledge; the input of the learning module R ζ is the previous latent state h t-1 , the new observation o t , and the previous knowledge base and outputs the updated latent state representation h t ; the latent state update can be represented as: The above equation defines the knowledge-based state update: the latent state h t of the agent as a function of the previous latent state h t-1 , the most recent observation o t , and the relevant knowledge retrieved from The discrete time steps t = 1, 2,... correspond to a sequential decision-making process in the drone mission, and the knowledge base at time t is denoted as which serves as a memory to store the accumulated semantic information over time; and is represented as a set of embedding items: where each is a d-dimensional vector encoding a piece of knowledge for time t; The repository contains drone state data, environmental semantics or task context facts, all embedded into a common vector space; size N can grow or adjust as the task progresses; to ensure rigor, state evolution is described in probabilistic terms; at each step, to represent the augmented state of the agent; given a new observation o t , the update can be represented as: to ​ Where h t h represents the current potential state. t-1 This represents the potential state at the previous time step. This represents the current knowledge base. This indicates the knowledge base of the previous time step, o t Let P(·) represent the current observation, and let P(·) represent the probability distribution. The above formula can be decomposed into: updating h based on the old state, knowledge, and new observations. t Update by incorporating newly inferred knowledge This knowledge-based representation learning is crucial for some observable drone environments, in which It will enhance o t To generate a robust potential state h t ; Knowledge summarization based on large language models: In the retrieval augmentation generation (RAG) module, the agent retrieves relevant information from and provides it as context to the large language model (LLM); define a query vector q t ∈R d , which comes from the state of the agent at time t or the task requirements; the similarity function S(q t ,x i ) measures the matching degree of each knowledge item with the query vector; let be arranged in descending order of S(q t ,x i ); the top k knowledge items retrieved are shown in the following knowledge retrieval formula: Then, the summarization function g(·) will convert to C t i.e., a concise knowledge summary of the large language model LLM: A large language model LLM would then receive the additional context C t ; Large language model query and update based on knowledge: By introducing a query generation function f query (·), for simulating large language models to automatically generate queries based on their internal reasoning states to retrieve and dynamically update knowledge bases ​ wherein is a latent context or text-based context of a large language model LLM at time t; q is calculated t After that, the first k knowledge items are retrieved according to the knowledge retrieval formula, thereby providing a knowledge set Then the large language model LLM generates an output, which produces an action a through the strategy π t : a t = π(h t, C t ) Upon obtaining the observation at time step t+1, the system infers a new knowledge increment at the current time update function f update (·) fuses the new knowledge increment into the existing knowledge base generates an updated knowledge state The knowledge base updating adopts a simple set merging manner and adopts a more detailed structure alignment and fusion strategy to ensure the consistency of knowledge and structure optimization.

5. The UAV swarm mission planning system of unified view of task space of claim 1, wherein, The closed-loop adaptive and continuous improvement module is as follows: The closed-loop adaptive and continuous improvement module adopts a closed-loop adaptive mechanism, in which the large language model LLM continuously modifies the reward function during the entire reinforcement learning process. The large language model LLM formulates the reward structure based on the task objective, and dynamically updates the reward as the agent interacts with the environment and receives feedback, ensuring consistency with the changing objectives and conditions. Closed-loop adaptive process: Consider a reinforcement learning environment with state space S, action space A, transition model p(s t+1 |s t ,a t ) and reward function R t at time t. The closed-loop adaptive iterative process is as follows: Step 1) Initialization: At t = 0, the LLM sets the initial reward function R0 according to the task objective; Step 2) Agent execution and feedback: Under policy π, the agent selects an action a t , receives a reward r t = R t (s t , a t ), and transitions to a new state s t+1 ; the environment provides additional performance feedback; Step 3) LLM reward update: LLM updates the reward function using environmental feedback: in Extract key performance indicators (KPIs) from state transitions and summarize them, while the function... Then, based on this feedback, the previous reward function R... t Adjustments will be made; in this regard, R will be updated by increasing the penalty for collision events and increasing the reward for achieving sub-objectives. t This increases the success rate; this will generate a new reward function R. t+1 This makes its observation performance more consistent with the mission objectives and the agent's objectives; Step 4) Policy improvement: the agent improves its policy using the updated reward function R t+1 ; The above closed-loop cycle repeats continuously, and the reward function and strategy are continuously improved to guide the agent to achieve the best behavior under the current task objective. Convergence analysis and non-stationary reward stability: A step parameter is introduced in the reward adjustment to limit the magnitude of each modification; when the large language model LLM proposes a change suggestion Δ to the reward function, the reward function R t+1 is updated to: R t+1 = R t + βΔ where R t represents the reward function at time t, β represents the step size, used to reduce the impact of Δ, to avoid too large changes in rewards; Δ represents the adjustment proposed by the large language model LLM according to the current task performance.

Images