An unmanned cluster task planning method based on an AI agent

Abstract

This invention relates to an unmanned swarm task planning method based on AI agents, belonging to the technical field of artificial intelligence. It solves the problems of insufficient inference ability, sparse and delayed feedback, and static environmental perception in existing AI agents for unmanned swarm task planning. The method includes: acquiring task requirement text and initial task environment information as the root node; performing multiple rounds of improved Monte Carlo tree search to construct a search tree until a preset first termination condition is met; in the improved Monte Carlo tree search: selecting a path from the root node to a leaf node based on the action value estimated by the current Q-value model; generating multiple candidate actions at the leaf node and performing real-time evaluation; adding candidate actions to the search tree based on the evaluation results; when a preset second termination condition is met, scoring the path, generating training samples, and optimizing the Q-value model; and selecting the path with the highest accumulated action value in the search tree as the task planning scheme.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an unmanned swarm task planning method based on AI intelligent agents. Background Technology

[0002] An unmanned swarm system refers to a collaborative system composed of a large number of autonomous or semi-autonomous unmanned units (such as unmanned vehicles and drones). Its core advantages include strong robustness, high task execution efficiency, and flexible configuration. When planning tasks for an unmanned swarm, it is necessary to effectively decompose complex tasks into sub-tasks and rationally allocate them to individuals within the swarm.

[0003] In recent years, Large Language Models (LLMs) have been evolving from passive knowledge retrieval devices to proactive decision-making agents. By enhancing the domain knowledge cognition level of large models through supervised fine-tuning (SFT) and alignment with human preferences, they are then constructed into intelligent agents with perception, planning, action, and reflection capabilities. This represents a typical application paradigm in the current "large model + robotics" field. When handling unmanned swarm task planning, these AI agents can leverage the semantic understanding, logical reasoning, and zero-shot generalization capabilities of LLMs to decompose abstract high-level instructions into concrete sequences of meta-actions, ultimately driving unmanned units to complete the instructions.

[0004] However, existing task planning methods usually generate the final solution directly based on probability, lacking the ability to explore multiple paths and think ahead during reasoning. This can easily lead to logical loopholes or decision illusions, making it difficult to guarantee robustness and optimality under complex tasks. Furthermore, the feedback mechanism of existing methods usually relies on the overall score after the task is completed, ignoring the evaluation of intermediate states during the execution process. This results in sparse and delayed feedback signals, making it difficult for the agent to locate specific decision-making errors and limiting its self-reflection and learning efficiency. Summary of the Invention

[0005] Based on the above analysis, the embodiments of the present invention aim to provide an unmanned swarm task planning method based on AI agents, in order to solve the problems of insufficient inference ability, sparse and delayed feedback, and static environmental perception of existing AI agents in unmanned swarm task planning.

[0006] The objective of this invention is mainly achieved through the following technical solutions: On the one hand, this invention provides a method for unmanned swarm task planning based on AI intelligent agents, including the following steps: Obtain the task requirement text and initial task environment information; Using the task requirement text and the initial task environment information as the root node, a search tree is constructed by performing multiple rounds of improved Monte Carlo tree search until a preset first termination condition is met. In each round of improved Monte Carlo tree search: based on the action value estimated by the current Q-value model, a path leading to a leaf node is selected from the root node; multiple candidate actions are generated at the leaf node, and the candidate actions are evaluated in real-time using a pre-trained process reward model; a new branch is created for the candidate action based on the evaluation result and added to the search tree; when a preset second termination condition is met, the selected path is scored, and backpropagation is performed along the selected path based on the scoring result to generate training samples for optimizing the Q-value model and to optimize and train the Q-value model. Starting from the root node, select the path with the highest cumulative action value in the search tree as the task planning scheme.

[0007] Furthermore, generating multiple candidate actions at the leaf node includes: generating prompt word templates based on the state of the leaf node and preset actions, and generating multiple candidate actions and corresponding selection probability values ​​using the task planning big model; The step of creating a new branch for the candidate action based on the evaluation result and adding it to the search tree includes: sorting the evaluation scores of each candidate action from high to low, and retaining the K candidate actions with the highest scores as retained actions; Create a child node for each of the retained actions, with the leaf node as the parent node and the retained action as the connecting edge, and add each of the child nodes to the search tree.

[0008] Furthermore, the step of selecting a path from the root node to a leaf node based on the action value estimated by the current Q-value model includes: iteratively executing the following steps from the root node until a leaf node is reached: Based on the current node state, the action value of each child node of the current node is obtained using the current Q-value model; wherein, the node state of the root node includes the task requirement text and the task initial environment information; the node state of nodes other than the root node includes the task requirement text, the historical step sequence from the root node to the current node, and the environment information of the current node. Based on the action value of each child node, the selection probability value of the action corresponding to each child node, and the cumulative number of visits to the current node and each child node, the child node is selected using the following formula: ; in, Represents a node Selected child node actions; Indicates at node The value of the action obtained by performing action a; Indicates at node The selection probability value of the next selection action a; Represents a node The cumulative number of visits; Represents a node The cumulative number of visits to the child nodes corresponding to the action 'a'; This indicates the exploration of hyperparameters.

[0009] Furthermore, the large-scale task planning model is fine-tuned through the following steps: Obtain a fine-tuning dataset that includes task planning instructions and the corresponding expected output actions of the model; Based on the fine-tuned dataset, the general large language model is fine-tuned using the LoRA method to obtain the task planning large model.

[0010] Furthermore, the Q-value model is constructed based on the task planning big model; wherein, a multilayer perceptron is used to replace the language modeling head in the task planning big model; the multilayer perceptron takes the hidden state of the last layer of the task planning big model as input and outputs the action value of the state-action pair.

[0011] Furthermore, the Q-value model is initially trained using the following steps: Obtain multiple complete state-action trajectories from the initial state to the end of the task and discretize them to obtain the state-action sequence corresponding to each trajectory; Each trajectory is scored and normalized to obtain the trajectory score for each trajectory; For each state-action pair in the state-action sequence of each trajectory, the average score of the trajectory including the state-action pair is taken as the action value of the state-action pair. The training dataset for the Q-value model is constructed based on each state-action pair and its corresponding action value as labels. Load the training dataset of the Q-value model, train the Q-value model using a weighted loss function including a mean squared error loss function and a conservative Q-learning function, and obtain the initial Q-value model when the training converges; wherein, the formula of the conservative Q-learning function is: ; in, The training dataset represents the Q-value model; This represents the state of the state-action pair q in the training dataset; This represents the action of the state-action pair q in the training dataset; This represents the action value label for state-action relationship q; π represents the action value of the current Q-value model for the state-action pair q; π represents the learning policy. This represents the action under the learning strategy π. The expected value of the action; This represents the trainable parameters of the Q-value model.

[0012] Furthermore, the step of backpropagating along the selected path based on the scoring results to generate training samples for optimizing the Q-value model and optimizing and training the Q-value model includes: Based on the scoring results, the action value of each node on the path is updated using the following formula: ; in, Indicates at node Next action The value of the updated action; Indicates at node Next action The value of the action obtained; Represents a node Next action The cumulative number of visits to the corresponding child nodes; R represents the scoring result; Based on each state-action pair on the selected path and the corresponding updated action value, an optimized dataset is obtained to optimize the Q-value model. Load the optimized dataset of the Q-value model, train the Q-value model using a weighted loss function including the mean squared error loss function and the conservative Q-learning function, and obtain the optimized Q-value model when the training converges.

[0013] Furthermore, the process reward model is a lightweight Transformer encoder that takes the task requirement text, the historical step sequence from the root node to the leaf node, and the candidate actions as input, and outputs an evaluation score for the candidate actions; wherein, the evaluation score includes a task execution time efficiency score, a task interruption count score, a resource consumption score, a task completion coverage score, and a total score.

[0014] Furthermore, the process reward model is trained using the following steps: Obtain multiple complete state-action trajectories from the initial state to the end of the task and discretize them to obtain the state-action sequence corresponding to each trajectory; Each trajectory is scored and normalized to obtain the trajectory score for each trajectory; Based on the trajectory scores of each trajectory, the score for each action in each trajectory is obtained as the process reward value using the following formula: ; in, Indicates action Process reward value; Indicates including actions The set of trajectories; Indicates including actions trajectory Trajectory score; Indicates including actions The number of trajectories; Construct a training dataset for the process reward model; wherein each training sample in the training dataset is a positive and negative sample pair consisting of the task requirement text, a concatenated sequence of historical steps from the root node to each node, and the action with the highest process reward value and the action with the lowest process reward value among the actions starting from each node. Load the training dataset of the process reward model, train the process reward model using the pairwise ranking loss function, and obtain the trained process reward model when the training converges; wherein, the formula of the pairwise ranking loss function is: ; in, The training dataset represents the process reward model; Represents a node The corresponding positive sample action; Represents a node The corresponding negative sample action; Indicates the dataset Calculate the expectation of all positive and negative sample pairs in the given condition. Represents a node The corresponding positive sample action's process reward value; Represents a node The corresponding process reward value for the negative sample action; Represents the Sigmoid function; This represents the trainable parameters of the process reward model.

[0015] Furthermore, the first termination condition is reaching the preset maximum number of search rounds or the optimal path remaining unchanged for a preset number of consecutive rounds; The second termination condition is that the depth of the current search path has reached the preset maximum depth or the current leaf node has reached the task objective specified in the task requirement text.

[0016] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects: 1. By using Monte Carlo tree search, the present invention enables the AI ​​agent to perform a mental rehearsal based on environmental information before outputting each action. Compared with one-step solution generation, the present invention enables the agent to explore multiple planning paths and select the optimal solution through the process reward model (PRM) and Q-value model, which significantly improves the logical rigor of the planning scheme when the agent is dealing with complex and long-cycle tasks.

[0017] 2. This invention introduces a process reward model (PRM) to evaluate and prune each intermediate action in the task planning process in real time. This solves the problem of sparse and delayed feedback signals caused by existing methods that rely solely on the final task score. This enables the agent to accurately locate decision-making errors, improves self-reflection ability and learning efficiency, and effectively controls the size of the search tree through the pruning mechanism, thereby reducing computational overhead.

[0018] 3. In the selection phase of tree search, this invention uses the long-term value estimate of each candidate action output by the Q-value model as the utilization term and exploration term of the predictive upper confidence bound tree search (PUCT) algorithm to jointly determine the path selection. This enables the agent to achieve a balance between exploring unknown paths and utilizing known high-value paths. At the same time, the Q-value model continuously iterates and optimizes by collecting real reward data through backpropagation. As the number of search rounds increases, its estimation accuracy continuously improves, achieving continuous optimization of task planning.

[0019] 4. This invention scores the real paths generated in each round of Monte Carlo tree search, constructs optimized training samples based on the scoring results and state-action pairs, and optimizes the Q-value model. As the number of search rounds increases, the estimation accuracy of the Q-value model continuously improves, guiding subsequent searches to focus more on high-value areas.

[0020] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0021] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0022] Figure 1 This is a flowchart illustrating an unmanned swarm task planning method based on an AI agent, as described in an embodiment of the present invention. Figure 2 This is a schematic diagram of the improved Monte Carlo tree search process in an embodiment of the present invention. Detailed Implementation

[0023] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0024] A specific embodiment of the present invention discloses an unmanned swarm task planning method based on AI intelligent agents, such as... Figure 1 As shown, it includes the following steps S1-S3: Step S1: Obtain the task requirement text and the initial environment information of the task.

[0025] Specifically, the task requirement text is input through a human-computer interaction interface or obtained by converting voice commands into text through a speech recognition system. The task requirement text includes at least information such as task objectives, task area, task constraints, and task priority, providing a clear direction for subsequent task planning.

[0026] The initial environmental information for the task includes environmental information collected in real time by multimodal sensors; wherein, the sensors include, but are not limited to, optical cameras, radar, GPS, meteorological sensors, etc.; the collected unstructured multimodal data is converted into structured text descriptions through a general multimodal large model as the initial environmental information for the task.

[0027] Step S2: Using the task requirement text and the initial task environment information as the root nodes, perform multiple rounds of improved Monte Carlo tree search to construct a search tree until the preset first termination condition is met.

[0028] Specifically, current methods of using intelligent agents for task planning typically output multiple instructions at once. Although there is a reflection mechanism, it is only corrected after all tasks have been completed, rather than searching in advance. Once a deviation occurs in the early steps, all subsequent planning is based on the error.

[0029] Therefore, this embodiment constructs a search tree for unmanned cluster task planning. The root node of the search tree represents the task requirement text input by the user and the initial task environment information. Each node other than the root node represents the environment state when the current action instruction is executed, including the task requirement text, the historical step sequence from the root node to the current node, and the environment information of the current node. Each edge represents a step action instruction executed from the state of the parent node to the state of the child node. The maximum depth of the search tree can be preset according to computing resources and task complexity.

[0030] The first termination condition is reaching a preset maximum number of search rounds or maintaining the optimal path for a preset number of consecutive rounds. For example, the preset number of rounds can be set to 8.

[0031] Furthermore, such as Figure 2 As shown, each round of the improved Monte Carlo tree search includes the following steps S21-S25: Step S21: Based on the action value estimated by the current Q-value model, select a path from the root node to the leaf node.

[0032] Specifically, Monte Carlo Tree Search (MCTS) is a heuristic search algorithm for decision optimization. Its core idea is to sample the decision space through random simulation and iteratively optimize the search direction based on the sampling results to approximate the optimal decision under limited computing resources.

[0033] The Q-value model is used to estimate the expected long-term reward of performing each action in each node state. The Q-value model is pre-trained using historical trajectory data in the initial stage. During the search process, it is updated online using real reward samples generated by backpropagation after each search round, so that its estimation accuracy continuously improves with the increase of search rounds, thereby more accurately guiding subsequent path selection. In step S21, based on the action value output by the current Q-value model, starting from the root node, the child node with the highest value is selected layer by layer until the leaf node is reached, completing the path selection for the current search round.

[0034] It should be noted that when performing a Monte Carlo tree search for the first time, the search tree only includes the root node, which is also the leaf node. In this case, there is no need to generate a path from the root node to the leaf node, and step S22 can be executed directly.

[0035] Step S22: Generate multiple candidate actions at the leaf node, including: generating prompt word templates based on the state of the leaf node and the preset action, and generating multiple candidate actions and corresponding selection probability values ​​using the task planning big model.

[0036] Specifically, the task planning model is a large language model based on an open-source foundation model, efficiently fine-tuned using the LoRA method. The open-source foundation model can be arbitrarily selected according to different tasks, such as Qwen and Llama. These models all adopt the standard Transformer decoder architecture and have powerful language understanding and generation capabilities. The LoRA fine-tuning process does not directly adjust the massive parameters of the original model, but indirectly affects the model's behavior by injecting a small number of trainable parameters, thereby achieving model customization at extremely low cost.

[0037] More specifically, when fine-tuning the task planning model, a fine-tuning dataset is obtained, including task planning instructions and corresponding expected output actions and selection probability values, for supervised training of the task planning model; the expected output actions can be written by experts. In this embodiment, the instruction dataset is stored in JSON format, with each line representing a question-answer pair object containing two key-value pairs, where the "input" key corresponds to the task planning instruction, and the "output" key corresponds to the expected output action and selection probability value.

[0038] Configure a trainable low-rank adapter in the linear projection layer of the Transformer layer of the task planning big model, and freeze the original parameters of the task planning big model.

[0039] Specifically, in each layer of the task planning model, the key linear projection layer is found. This layer is usually a layer with a large number of parameters and carries the core computing functions. In this embodiment, the key linear projection layers include: the query (Q) projection layer, key (K) projection layer, and value (V) projection layer in the self-attention module, the first linear projection layer and the second linear projection layer in the feedforward network (FFN), and the output projection layer.

[0040] For each linear projection layer, LoRA achieves efficient parameter fine-tuning by adding a low-rank decomposition next to the original weight matrix. This is achieved by multiplying two matrices A and B, where matrix A reduces dimensionality and matrix B increases it. The intermediate layer dimension (LoRA's rank) is lora_r, thus simulating the eigenrank. The formula is as follows: ; in, Represents the original weights of the linear projection layer h; This represents the updated weights of the linear projection layer h; This represents the scaling factor of LoRA, used to adjust the degree of influence of LoRA updates. In this embodiment, it can be set to 64. The rank of LoRA determines the width of the low-dimensional space. A larger rank provides stronger model expressive power, but also increases the number of parameters and computational overhead. In this embodiment, it can be set to 64. The dimension reduction matrix represents the linear projection layer i, which is used to project input information from a high-dimensional space to a low-dimensional space to capture key change patterns in the input. The matrix representing the dimension increase of the linear projection layer h is used to project the representation in the low-dimensional space back to the original output dimension, generating the final weight update based on the pattern interpretation.

[0041] More specifically, the LoRA Dropout rate (lora_dropout) is set to 0.05 and applied to the LoRA layer during training. This randomly sets the output activation values ​​of matrix A to zero with a 5% probability, preventing the adapter from over-relying on any single activation path and forcing it to learn more robust features. This improves generalization ability and prevents overfitting. Dropout is turned off during inference.

[0042] Load the fine-tuning dataset and train the configured low-rank adapter parameters with the goal of minimizing the cross-entropy loss between the output of the task planning model and the desired task planning scheme and selection probability value. The training ends when the loss function value converges, and the trained low-rank adapter is obtained.

[0043] The parameters of the trained low-rank adapter are combined with the original parameters of the task planning model to obtain the fine-tuned task planning model.

[0044] Furthermore, the action generation prompt word template is a structured instruction used to guide the task planning large model to generate candidate actions and their selection probabilities, including role declaration, task information, completed steps, current state, and output format constraints.

[0045] For example, the preset action generation prompt template can be: You are an unmanned swarm task planning intelligent agent; [Overall Task] {Task Requirements Text} [Steps completed] {Historical sequence of steps, listing the actions executed in chronological order} [Current Status] {Current Environment Information} Based on the information above, generate {n} possible next actions and provide the probability of choosing each action (the sum of all probabilities is 1). Different action plans should reflect different tactical characteristics (such as detection priority, mission priority, obstacle avoidance priority, energy saving priority, etc.). Output format:

Action 1

[0046] Replace “{task requirement text}” in the prompt word template with the task requirement text in the node status; replace “{historical step sequence, listing the actions executed in order}” in the prompt word template with the historical step sequence from the root node to the current node in the node status; replace “{current environment information}” in the prompt word template with the environmental information of the current node in the node status; replace “{n}” in the prompt word template with the number of candidate actions to be generated according to the task timeliness requirements, for example, it can be set to 6, to obtain the action generation prompt word. Input the action generation prompt word into the task planning model to obtain the candidate actions planned for the current node and the corresponding selection probability value.

[0047] More specifically, the candidate actions generated by the task planning model are natural language text. After the candidate actions in text form are converted into machine instructions by the existing scheme conversion model, they are verified using the rule-based JSON Schema format. By predefining the legal format of unmanned unit instructions, only edges that conform to the schema specification are retained, and invalid branches with incorrect format or that cannot be parsed are removed from the candidate actions, ensuring that only legal instructions that can actually be executed are retained in the search tree.

[0048] Furthermore, step S21, based on the action value estimated by the current Q-value model, selects a path from the root node to a leaf node, including: starting from the root node, iteratively executing the following steps S211-S212 until a leaf node is reached: Step S211: Based on the current node state, use the current Q-value model to obtain the action value of each child node of the current node.

[0049] Specifically, traditional Monte Carlo tree search typically employs a rollout strategy in the value estimation stage. This involves randomly selecting actions from the current node until a terminal state is reached, and then obtaining a reward value based on the final result. Each simulation requires executing the complete state-action sequence until termination, leading to excessively high computational costs and low search efficiency. Furthermore, the variance of rollout simulations is extremely high; the same initial state can yield drastically different reward values ​​depending on different random paths, requiring numerous simulations to converge to a stable estimate, further exacerbating the computational burden. This embodiment, in the path selection stage, uses a constructed Q-value model to directly estimate the expected cumulative reward of state-action pairs using a neural network. This replaces the value estimation required by traditional Monte Carlo tree search through multiple rollout simulations, allowing value assessment to be completed in a single forward inference, significantly reducing computational overhead.

[0050] Furthermore, the Q-value model is constructed based on the task planning big model; wherein, a multilayer perceptron is used to replace the language modeling head in the task planning big model; the multilayer perceptron takes the hidden state of the last layer of the task planning big model as input, and after passing through two layers of fully connected networks and activation function mapping, outputs the action value of the state-action pair, that is, the expected cumulative reward that can be obtained by performing the action in the current state.

[0051] Specifically, the Q-value model, based on the decoder part of the task planning big model, shares the same parameter structure as the fine-tuned task planning big model, ensuring that the Q-value model can deeply understand the semantic representation of the task planning model.

[0052] Furthermore, the Q-value model is initially trained using the following steps S2111-S2115: Step S2111: Obtain multiple complete state-action trajectories from the initial state to the end of the task and discretize them to obtain the state-action sequence corresponding to each trajectory.

[0053] Specifically, by performing Monte Carlo sampling from historical task execution records or simulation environments, multiple environmental state-action sequences from the initial task requirements to task completion are obtained. Since some action parameters in unmanned swarm control are continuous (such as travel coordinates and speed), this embodiment employs an Agent Q strategy to divide the continuous action range into several discrete intervals, obtaining the discretized state-action sequences corresponding to each trajectory, represented as follows: ;in, This represents the m-th complete state-action trajectory; Represents the state-action pair t in the trajectory; This represents the length of the trajectory m.

[0054] Step S2112: Score each trajectory and normalize it to obtain the trajectory score for each trajectory.

[0055] Specifically, an evaluation algorithm R is used to score each complete trajectory. In this embodiment, the evaluation algorithm uses four unified dimensions for scoring, with a total score of 100 points, including: task execution time efficiency (30 points), number of task interruptions (20 points), resource consumption (10 points), and task completion coverage (40 points). The scores of these four dimensions are summed to obtain the raw score of each trajectory. To facilitate model training, the raw score is normalized to the [0,1] interval as the final score for each trajectory.

[0056] Step S2113: For each state-action pair in the state-action sequence of each trajectory, the average score of the trajectory including the state-action pair is taken as the action value of the state-action pair.

[0057] Specifically, the action value of each state-action pair is obtained using the following formula: ; in, Representing state-action pairs The value of the action; Represents state-action pairs in all trajectories Total number of occurrences; This includes state-action pairs. The trajectory score of trajectory k.

[0058] Understandably, if a state-action pair frequently appears in high-scoring trajectories, its action value label is higher; conversely, if it mostly appears in low-scoring trajectories, its label is correspondingly lower. By taking the average, noise from individual trajectories can be eliminated, more stably reflecting the true quality of the action.

[0059] Step S2114: Construct the training dataset of the Q-value model based on each state-action pair and its corresponding action value as a label.

[0060] Specifically, the training dataset for the Q-value model is a set of triples, including the states in each state-action pair. ,action and action value tags .

[0061] Step S2115: Load the training dataset of the Q-value model, train the Q-value model using a weighted loss function including the mean squared error loss function and the conservative Q-learning function, and obtain the initial Q-value model when the training converges.

[0062] Specifically, when training the Q-value model, the loss function... for: ; ; ; in, This represents the mean squared error loss function; Represents the conservative Q-learning function; The weights represent the conservative Q-learning function; The training dataset represents the Q-value model; This represents the state of the state-action pair q in the training dataset; This represents the action of the state-action pair q in the training dataset; This represents the action value label for state-action relationship q; π represents the action value of the current Q-value model for the state-action pair q; π represents the learning policy. This represents the action under the learning strategy π. The expected value of the action; This represents the trainable parameters of the Q-value model.

[0063] Understandably, given the vast action space of unmanned swarms, Q-value models are prone to overestimation, i.e., giving artificially high Q-values ​​for unseen actions. By adding a Conservative Q-Learning (CQL) function to the loss function, the Q-value exponent of state-action pairs in the experience pool is maximized while the expected Q-value of the action under the current policy is minimized, which effectively reduces the Q-value estimation of unexplored actions and improves the model's generalization ability and training stability.

[0064] After each training round, the loss value on the validation set is calculated. When the validation loss does not decrease for several consecutive rounds or reaches the preset number of training rounds, training is stopped, the current model parameters are saved, and the initial Q-value model is obtained. After training is completed, the initial Q-value model is used in the path selection stage of Monte Carlo tree search. It should be noted that in the subsequent search process, the Q-value model will also be continuously optimized through the training samples generated by backpropagation to obtain the current Q-value model used in each round, thereby continuously improving the decision-making ability.

[0065] Step S212: Based on the action value of each child node, the selection probability value of the action corresponding to each child node, and the cumulative number of visits to the current node and each child node, select the child node using the following formula: ; in, Represents a node Selected child node actions; Indicates at node The value of the action obtained by performing action a; Indicates at node The selection probability value of the next selection action a; Represents a node The cumulative number of visits; Represents a node The cumulative number of visits to the child nodes corresponding to the action 'a'; This indicates the exploration of hyperparameters.

[0066] Specifically, in the path selection phase of Monte Carlo tree search, when the current node has multiple child nodes, it is necessary to select the optimal child node to continue searching downwards. In this embodiment, the action value output by the current Q-value model is combined with the selection probability value output by the task planning model and the number of node visits. In each decision step, the exploration of unknown paths and the use of known high-value paths are weighed to guide the agent to efficiently build the search tree.

[0067] Step S23: Use a pre-trained process reward model to evaluate the candidate actions in real time.

[0068] Specifically, in the node expansion phase of Monte Carlo tree search, after the task planning model generates multiple candidate actions, it is necessary to quickly evaluate the quality of these actions in order to select high-quality actions for subsequent expansion.

[0069] Furthermore, the process reward model is a lightweight Transformer encoder that takes the task requirement text, the historical step sequence from the root node to the leaf node, and the candidate actions as input, and outputs an evaluation score for the candidate actions; wherein, the evaluation score includes a task execution time efficiency score, a task interruption count score, a resource consumption score, a task completion coverage score, and a total score.

[0070] Specifically, during the construction of the search tree, the process reward model evaluates each candidate node in a timely manner, replacing the evaluation method in traditional Monte Carlo tree search that requires complete simulation to the end point, achieving millisecond-level instant feedback and significantly reducing computational costs.

[0071] Furthermore, the process reward model is trained using the following steps S231-S235: Step S231: Obtain multiple complete state-action trajectories from the initial state to the end of the task and discretize them to obtain the state-action sequence corresponding to each trajectory.

[0072] Step S232: Score each trajectory and normalize it to obtain the trajectory score for each trajectory.

[0073] Specifically, the implementation of steps S231-S232 is the same as that of steps S2111-S2112, and will not be repeated here.

[0074] Step S233: Based on the trajectory scores of each trajectory, use the following formula to obtain the score of each action in each trajectory as the process reward value: ; in, Indicates action Process reward value; Indicates including actions The trajectory coincides; Indicates including actions trajectory Trajectory score; Indicates including actions The number of trajectories.

[0075] Specifically, the final score of the sparse trajectory is converted into a dense label for each intermediate action. If an action frequently appears in high-scoring trajectories, its process reward value is higher; conversely, if it mostly appears in low-scoring trajectories, its label is correspondingly lower.

[0076] Step S234: Construct the training dataset of the process reward model; wherein, each training sample in the training dataset is a positive and negative sample pair consisting of the task requirement text, a spliced ​​sequence of historical step sequences from the root node to each node, and the action with the highest process reward value and the action with the lowest process reward value among the actions starting from each node.

[0077] Specifically, since the process reward model described in this embodiment is used for pruning candidate actions, that is, selecting several high-quality actions from multiple candidate actions, it is only necessary to train the process reward model to distinguish the relative merits of actions, rather than accurately predicting the absolute score of each action.

[0078] Therefore, this embodiment adopts pairwise ranking loss. By constructing positive and negative sample pairs of "high-value action - low-value action" for training, the model focuses on learning the relative order relationship between actions. By pairing high-value actions and low-value actions in the same state, the model is guided to learn the key discriminative ability of "which action is better" under the same context. Compared with the regression method, this training method is more robust to data noise, has higher ranking accuracy, and can better adapt to the actual needs of pruning scenarios.

[0079] Step S235: Load the training dataset of the process reward model, train the process reward model using the pairwise ranking loss function, and obtain the trained process reward model when the training converges; wherein, the formula of the pairwise ranking loss function is: ; in, The training dataset represents the process reward model; Represents a node The corresponding positive sample action; Represents a node The corresponding negative sample action; Indicates the dataset Calculate the expectation of all positive and negative sample pairs in the given condition. Represents a node The corresponding positive sample action's process reward value; Represents a node The corresponding process reward value for the negative sample action; Represents the Sigmoid function; This represents the trainable parameters of the process reward model.

[0080] Step S24: Based on the evaluation results, create a new branch for the candidate action and add it to the search tree, including: The evaluation scores of each candidate action are sorted from high to low, and the K candidate actions with the highest scores are retained as retained actions.

[0081] Specifically, high-scoring actions represent actions of higher quality and more likely to lead to a successful path in the current state, while low-scoring actions represent actions with higher risk or weaker relevance to the task objective. Removing low-scoring actions through pruning can effectively control the branching factor of the search tree and prevent the number of nodes from expanding exponentially. More specifically, in the TOP-K pruning strategy, the value of K needs to be less than or equal to the number of actions n generated in the action generation prompts of the task planning model. For example, K can be set to 3.

[0082] It should be noted that if the number of actions generated in step S22 is less than or equal to K, then there is no need to use the process reward model for scoring and pruning; all actions can be retained directly to avoid unnecessary computational overhead.

[0083] Create a child node for each of the retained actions, with the leaf node as the parent node and the retained action as the connecting edge, and add each of the child nodes to the search tree.

[0084] Step S25: When the preset second termination condition is met, the selected path is scored, and the scoring results are backpropagated along the selected path to generate training samples for optimizing the Q-value model and to optimize and train the Q-value model.

[0085] Furthermore, the second termination condition is that the depth of the current search path has reached the preset maximum depth or the current leaf node has reached the task objective specified in the task requirement text.

[0086] Specifically, when the Monte Carlo tree search in the current round satisfies the second termination condition, the evaluation algorithm R is used to score the complete task execution result of the current search path, and the total score is then normalized to obtain the trajectory score result.

[0087] Furthermore, the step of backpropagating along the selected path based on the scoring results to generate training samples for optimizing the Q-value model and optimizing and training the Q-value model includes steps S251-S253: Step S251: Based on the scoring results, update the action value of each node on the path using the following formula: ; in, Indicates at node Next action The value of the updated action; Indicates at node Next action The value of the action obtained; Represents a node Next action The cumulative number of visits to the corresponding child nodes; R represents the scoring result.

[0088] Specifically, the scoring result R is propagated backward from the leaf node to the root node along the current search path, updating the action value of each state-action pair, and incrementing the cumulative access count of each node on the selected path and the cumulative access count of the action corresponding to each node by 1.

[0089] Step S252: Based on each state-action pair on the selected path and the corresponding updated action value, obtain the optimized dataset for optimizing the Q-value model.

[0090] Step S253: Load the optimized dataset of the Q-value model, train the Q-value model using a weighted loss function including the mean squared error loss function and the conservative Q-learning function, and obtain the optimized Q-value model when the training converges.

[0091] Specifically, after each round of search, the Q-value model is immediately trained using the newly generated optimization dataset, enabling the model to continuously learn from the latest search experience. As the number of search rounds increases, the accuracy of the Q-value model in estimating action value continuously improves.

[0092] Understandably, the search tree uses the task requirement text and initial environment information as its root node, with subsequent nodes recording the historical action sequences and current environment states layer by layer. Each edge corresponds to a specific instruction output by the task planning model. Through multiple rounds of improved Monte Carlo tree search, the search tree can systematically explore the potential results of different decision paths, thereby obtaining the complete planning scheme with the highest cumulative return under limited computing resources.

[0093] Step S3: Starting from the root node, select the path with the highest accumulated action value in the search tree as the task planning scheme.

[0094] Specifically, starting from the root node, the child node with the highest cumulative value is selected at each node, and this process is repeated layer by layer down to the leaf node to obtain an optimal path from the initial state to the final state. The action sequence consisting of all actions on the optimal path is the final task planning scheme. The final task planning scheme is then converted into structured machine instructions and sent to the unmanned cluster for execution.

[0095] In summary, the unmanned swarm task planning method based on AI intelligent agents according to embodiments of the present invention has the following beneficial effects: 1. This invention utilizes a Monte Carlo tree search framework, enabling the AI ​​agent to mentally rehearse each action in conjunction with environmental information. Compared to the one-step solution generation before reinforcement learning, this invention allows the agent to explore multiple planning paths and select the optimal solution through a process reward model (PRM) and a Q-value model, significantly improving the logical rigor of the planning scheme when the agent handles complex, long-cycle tasks.

[0096] 2. This invention introduces a process reward model (PRM) to evaluate and prune each intermediate action in task planning in real time. This solves the problem of sparse and delayed feedback signals caused by existing methods that rely solely on the final task score. This enables the agent to accurately locate decision-making errors, improves self-reflection ability and learning efficiency, and effectively controls the size of the search tree through the pruning mechanism, thereby reducing computational overhead.

[0097] 3. In the tree search selection stage of this invention, the long-term value estimate of each candidate action output by the Q-value model is used as the utilization term and exploration term of the predictive upper confidence bound tree search (PUCT) algorithm to jointly determine the path selection. This enables the agent to achieve a balance between exploring unknown paths and utilizing known high-value paths. At the same time, the Q-value model continuously iterates and optimizes by collecting real reward data through backpropagation. As the number of search rounds increases, its estimation accuracy continuously improves, thus achieving continuous optimization of task planning.

[0098] 4. In this embodiment of the invention, the real paths generated in each round of Monte Carlo tree search are scored, and optimized training samples are constructed based on the scoring results and state-action pairs to optimize the Q-value model. As the number of search rounds increases, the estimation accuracy of the Q-value model continuously improves, guiding subsequent searches to focus more on high-value areas.

[0099] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0100] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for unmanned swarm task planning based on AI intelligent agents, characterized in that, Includes the following steps: Obtain the task requirement text and initial task environment information; Using the task requirement text and the initial task environment information as the root node, a search tree is constructed by performing multiple rounds of improved Monte Carlo tree search until a preset first termination condition is met. In each round of improved Monte Carlo tree search: based on the action value estimated by the current Q-value model, a path leading to a leaf node is selected from the root node; multiple candidate actions are generated at the leaf node, and the candidate actions are evaluated in real-time using a pre-trained process reward model; a new branch is created for the candidate action based on the evaluation result and added to the search tree; when a preset second termination condition is met, the selected path is scored, and backpropagation is performed along the selected path based on the scoring result to generate training samples for optimizing the Q-value model and to optimize and train the Q-value model. Starting from the root node, select the path with the highest cumulative action value in the search tree as the task planning scheme.

2. The method according to claim 1, characterized in that, The step of generating multiple candidate actions at the leaf node includes: generating prompt word templates based on the state of the leaf node and a preset action, and using the task planning big model to generate multiple candidate actions and corresponding selection probability values; The step of creating a new branch for the candidate action based on the evaluation result and adding it to the search tree includes: sorting the evaluation scores of each candidate action from high to low, and retaining the K candidate actions with the highest scores as retained actions; Create a child node for each of the retained actions, with the leaf node as the parent node and the retained action as the connecting edge, and add each of the child nodes to the search tree.

3. The method according to claim 2, characterized in that, The step of selecting a path from the root node to a leaf node based on the action value estimated by the current Q-value model includes: starting from the root node, iteratively executing the following steps until a leaf node is reached: Based on the current node state, the action value of each child node of the current node is obtained using the current Q-value model; wherein, the node state of the root node includes the task requirement text and the task initial environment information; the node state of nodes other than the root node includes the task requirement text, the historical step sequence from the root node to the current node, and the environment information of the current node. Based on the action value of each child node, the selection probability value of the action corresponding to each child node, and the cumulative number of visits to the current node and each child node, the child node is selected using the following formula: ； in, Represents a node Selected child node actions; Indicates at node The value of the action obtained by performing action a; Indicates at node The selection probability value of the next selection action a; Represents a node The cumulative number of visits; Represents a node The cumulative number of visits to the child nodes corresponding to the action 'a'; This indicates the exploration of hyperparameters.

4. The method according to claim 2 or 3, characterized in that, The large-scale task planning model is obtained through fine-tuning the following steps: Obtain a fine-tuning dataset that includes task planning instructions and the corresponding expected output actions of the model; Based on the fine-tuned dataset, the general large language model is fine-tuned using the LoRA method to obtain the task planning large model.

5. The method according to claim 4, characterized in that, The Q-value model is constructed based on the task planning big model; wherein, a multilayer perceptron is used to replace the language modeling head in the task planning big model; the multilayer perceptron takes the hidden state of the last layer of the task planning big model as input and outputs the action value of the state-action pair.

6. The method according to claim 5, characterized in that, The Q-value model is initially trained using the following steps: Obtain multiple complete state-action trajectories from the initial state to the end of the task and discretize them to obtain the state-action sequence corresponding to each trajectory; Each trajectory is scored and normalized to obtain the trajectory score for each trajectory; For each state-action pair in the state-action sequence of each trajectory, the average score of the trajectory including the state-action pair is taken as the action value of the state-action pair. The training dataset for the Q-value model is constructed based on each state-action pair and its corresponding action value as a label. Load the training dataset of the Q-value model, train the Q-value model using a weighted loss function including a mean squared error loss function and a conservative Q-learning function, and obtain the initial Q-value model when the training converges; wherein, the formula of the conservative Q-learning function is: ； in, The training dataset represents the Q-value model; This represents the state of the state-action pair q in the training dataset; This represents the action of the state-action pair q in the training dataset; This represents the action value label for state-action relationship q; π represents the action value of the current Q-value model for the state-action pair q; π represents the learning policy. This represents the action under the learning strategy π. The expected value of the action; This represents the trainable parameters of the Q-value model.

7. The method according to claim 6, characterized in that, The step of backpropagating along the selected path based on the scoring results to generate training samples for optimizing the Q-value model and then optimizing and training the Q-value model includes: Based on the scoring results, the action value of each node on the path is updated using the following formula: ； in, Indicates at node Next action The value of the updated action; Indicates at node Next action The value of the action obtained; Represents a node Next action The cumulative number of visits to the corresponding child nodes; R represents the scoring result; Based on each state-action pair on the selected path and the corresponding updated action value, an optimized dataset is obtained to optimize the Q-value model. Load the optimized dataset of the Q-value model, train the Q-value model using a weighted loss function including the mean squared error loss function and the conservative Q-learning function, and obtain the optimized Q-value model when the training converges.

8. The method according to claim 1, characterized in that, The process reward model is a lightweight Transformer encoder that takes the task requirement text, the historical step sequence from the root node to the leaf node, and the candidate actions as input, and outputs an evaluation score for the candidate actions. The evaluation score includes a task execution time efficiency score, a task interruption count score, a resource consumption score, a task completion coverage score, and a total score.

9. The method according to claim 8, characterized in that, Train the process reward model using the following steps: Obtain multiple complete state-action trajectories from the initial state to the end of the task and discretize them to obtain the state-action sequence corresponding to each trajectory; Each trajectory is scored and normalized to obtain the trajectory score for each trajectory; Based on the trajectory scores of each trajectory, the score for each action in each trajectory is obtained as the process reward value using the following formula: ； in, Indicates action Process reward value; Indicates including actions The set of trajectories; Indicates including actions trajectory Trajectory score; Indicates including actions The number of trajectories; Construct a training dataset for the process reward model; wherein each training sample in the training dataset is a positive and negative sample pair consisting of the task requirement text, a concatenated sequence of historical steps from the root node to each node, and the action with the highest process reward value and the action with the lowest process reward value among the actions starting from each node. Load the training dataset of the process reward model, train the process reward model using the pairwise ranking loss function, and obtain the trained process reward model when the training converges; wherein, the formula of the pairwise ranking loss function is: ； in, The training dataset represents the process reward model; Represents a node The corresponding positive sample action; Represents a node The corresponding negative sample action; Indicates the dataset Calculate the expectation of all positive and negative sample pairs in the given condition. Represents a node The corresponding positive sample action's process reward value; Represents a node The corresponding process reward value for the negative sample action; Represents the Sigmoid function; This represents the trainable parameters of the process reward model.

10. The method according to claim 1, characterized in that, The first termination condition is reaching the preset maximum number of search rounds or the optimal path remains unchanged for a preset number of consecutive rounds; The second termination condition is that the depth of the current search path has reached the preset maximum depth or the current leaf node has reached the task objective specified in the task requirement text.

Images