A large model decision optimization method based on environment feedback

Abstract

The application discloses a large model decision optimization method based on environmental feedback, comprising: generating an initial decision scheme and a corresponding thinking chain according to a task target, an environmental state and an execution unit state; after completing task execution according to the initial decision scheme, performing multidimensional evaluation on the task execution result by an evaluator model to generate an evaluation result containing result analysis, problem positioning and improvement suggestions; analyzing logical deviation, target deviation and strategy deficiency in the thinking chain by a reflection model to obtain reflection content; and dynamically updating the thinking chain based on the evaluation result and the reflection content to realize optimization for a subsequent decision process. The implementation of the application can enable the large model to continuously correct and optimize the original decision process according to the environmental state in a complex dynamic task scene, the task execution result and historical experience feedback.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a large model decision optimization method based on environmental feedback. Background Technology

[0002] With the rapid development of artificial intelligence technology and large language models, the application value of intelligent decision-making systems in fields such as complex task planning, industrial scheduling, and emergency management is becoming increasingly prominent. Large language models possess strong semantic understanding, knowledge organization, and logical reasoning capabilities, enabling them to generate relatively complete decision-making solutions based on task objectives, environmental states, and historical experience. Moreover, in complex and dynamic task scenarios, traditional methods relying solely on preset rules or static models are insufficient to meet the demands for high real-time performance, high reliability, and high adaptability in decision-making. Currently, some intelligent decision-making methods have begun to introduce thought chain reasoning mechanisms, improving the interpretability and traceability of the model's decision-making process through step-by-step reasoning. These methods can decompose complex tasks into several intermediate reasoning steps and generate corresponding action plans based on task objectives, thus alleviating to some extent the problem of opaque reasoning processes in traditional end-to-end decision-making models.

[0003] Specifically, existing large-scale model decision optimization methods typically rely on manually constructed prompt templates, static knowledge bases, or offline training data for policy generation and model updates. While these methods can achieve certain results in fixed task scenarios, their evaluation mechanisms are usually quite simplistic. When the execution results deviate from expectations, it is difficult to identify key errors in the inference chain in a timely manner, leading to problems such as delayed experience updates, recurring inference errors, and insufficient policy adaptability. Furthermore, traditional model updates usually rely on retraining or large-scale parameter fine-tuning, which suffers from long training cycles, high computational resource consumption, and inflexible deployment and updates.

[0004] In view of this, the present invention is hereby proposed. Summary of the Invention

[0005] The purpose of this invention is to provide a large-scale model decision optimization method based on environmental feedback. This aims to improve the rationality, interpretability, real-time response capability, and continuous adaptive capability of large-scale models in complex task scenarios, thereby addressing the problems existing in the prior art.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] A large-scale model decision optimization method based on environmental feedback includes:

[0008] An initial decision-making scheme and a corresponding thought chain are generated based on the task objective, environmental state, and execution unit state. The thought chain is used to record the reasoning process of the initial decision-making scheme.

[0009] After the task is executed according to the initial decision-making scheme, the task execution results are evaluated from multiple dimensions through the evaluator model, generating semantic feedback that includes result analysis, problem identification, and improvement suggestions to obtain evaluation results; and, through the reflection model, the logical deviations, goal deviations, and strategy deficiencies in the thought chain are analyzed to obtain reflection content that includes problem identification, decision-making logic deviations, and feasible improvement directions.

[0010] The thought process chain is dynamically updated based on the evaluation results and the reflection content to optimize the subsequent decision-making process.

[0011] The process of obtaining the evaluation results includes:

[0012] The evaluator model acquires decision-making instructions from decision-makers, task result data packages, and execution unit role transition logs, and determines corresponding quantitative evaluation indicators. The task result data package contains information including: target achievement rate, resource consumption status, and environmental status changes. The quantitative evaluation indicators include:

[0013] Target achievement rate, used to measure task completion rate;

[0014] Timeliness is used to distinguish between three timeliness statuses of tasks: completed ahead of schedule, completed on time, and completed late.

[0015] Efficiency-resource ratio is used to measure the efficiency of task input and output.

[0016] Intent alignment is used to measure the degree to which the actual performance of the drone matches the decision-maker's original mission objectives;

[0017] Based on the aforementioned quantitative evaluation indicators, an evaluation result is generated that includes result analysis, problem identification, and improvement suggestions.

[0018] The method also includes an online reinforcement learning closed-loop training process for the evaluator model, and the training process includes:

[0019] After the evaluator obtains the decision-maker's decision instructions, the results of the first task, and the UAV role transformation, the decision-maker's decision instructions are transformed into a structured representation, which is then converted into a vectorized representation corresponding to the decision-maker's decision instructions, through a natural language parsing module; the results of the first task are fused from multiple sources of sensors to form a corresponding spatiotemporal feature matrix of environmental data; and the UAV role transformation record is modeled as a state transition diagram.

[0020] The global representation vector is obtained by fusing domain knowledge through a cross-attention mechanism using a Transformer-based self-attention encoder:

[0021] ;

[0022] Among them, ht Let t be the global representation vector of the multi-source fusion at time t, i.e., the fusion representation; For cross-attention fusion operators; This is a single-branch Transformer encoder function; d t The vectorized representation of the decision-maker's instructions; s t G is the spatiotemporal feature matrix of environmental data; t The state transition diagram obtained by modeling the drone role switching record;

[0023] The corresponding decoder is based on the fusion representation h t Autoregressive generation of evaluation text y t Its probability distribution is:

[0024] ;

[0025] in, For fusion-based characterization h t Autoregressive generation of evaluation text y t The probability distribution, w k To evaluate the k-th word in the text sequence, This refers to the sequence of all preceding words generated before the k-th word. For a given and h t At that time, w k The conditional probability of occurrence;

[0026] The generated evaluation text is written into short-term memory; a new decision is generated based on the evaluation text; after the new decision is executed, a second task result is generated, which is used as the first task result for the next processing and is also input into the reward function processing part to calculate the corresponding reward value.

[0027] The reward function outputs a reward signal, which is then fed back to the evaluator model through the Proximal Policy Optimization (PPO) algorithm. The PPO algorithm stabilizes and constrains the network parameter update magnitude, and synchronously updates all network weights of the evaluator's internal Transformer domain knowledge encoder and evaluation generator, enabling continuous iterative training of the evaluator model.

[0028] Prior to performing the online reinforcement learning closed-loop training, a pre-training phase for the evaluator model is also included, which includes:

[0029] Training of the base model is completed on a general corpus. The Transformer base model is trained using a general large-scale corpus to learn the general language and basic semantic expression capabilities.

[0030] By fine-tuning the task log data using a domain-adaptive method, an initial evaluator model with basic task understanding capabilities is obtained; during the fine-tuning process, the generated evaluation text y is... t The information is sent to the decision-maker's prompting section, forming a new decision context; and the changes in task benefits before and after adoption are based on the recorded evaluations. Construct counterfactual comparison samples To calculate the counterfactual advantage:

[0031] ;

[0032] in, For the counterfactual advantage in round t, The difference between the actual benefits before and after the evaluation is adopted. To incorporate the immediate mission rewards of the current mission round prior to this round of evaluation, To generate the task rewards for the next round after the new decision is implemented, based on the evaluation results from this round, To calculate the expected value of the counterfactual virtual gain difference;

[0033] This counterfactual advantage is used to guide the PPO optimization of the evaluation model, and its objective function is:

[0034] ;

[0035] in, Optimize the loss function for the PPO strategy. To calculate the mean expected value of the loss over all time steps, This represents the probability ratio between the old and new strategies. To obtain the minimum of the two, To be Limited to the range , This is for pruning hyperparameters.

[0036] The evaluator model includes personalized evaluation generation based on role perception and iterative updates of short-term memory, wherein:

[0037] The personalized evaluation generation based on role perception refers to the evaluator generating differentiated feedback for drones with different functions by embedding drone role description vectors and configuring a lightweight adapter to dynamically adjust the evaluation generation path.

[0038] The iterative update of the short-term memory is achieved through a dynamic memory pyramid structure, which includes an immediate memory layer, an execution mode layer, and a global principle layer, wherein:

[0039] The instant memory layer stores the most recent predetermined decision-evaluation pairs in the form of a sliding window and updates them through an iterative mechanism based on historical decision and evaluation records.

[0040] The execution mode layer is used to abstract high-frequency evaluation topics into searchable execution modes. It stores high-frequency evaluation modes for UAV scheduling in key-value pairs, using the current task context features as the search key and the corresponding evaluation template as the value. Based on the matching similarity between the current task features and the modes in the database, the execution mode memory is driven to be updated iteratively.

[0041] The global principle layer forms top-level rules through long-term suggestions in cluster evaluation. It also periodically clusters and summarizes historical evaluation suggestions to form general top-level execution rules for UAV scheduling.

[0042] During the iterative update process of the short-term memory, the corresponding memory update mechanism is implemented based on the long short-term memory of the COPPER framework. The evaluation representation vector output by the evaluator model will be synchronously written back to the dynamic memory pyramid to complete the full-level iterative update of the short-term memory.

[0043] The process of obtaining reflective content involves generating structured reflective text containing reflective content through a sequence-to-sequence Transformer model that integrates a built-in domain-specific knowledge base and a causal inference engine. This generation process includes:

[0044] The encoder of the Transformer model receives evaluation units obtained after task evaluation deconstruction. and original decision fragments And the key indicator m extracted through multi-agent association analysis i A joint representation is formed through a two-order gating fusion mechanism, the calculation method of which is as follows:

[0045] ;

[0046] ;

[0047] in, Let be the fusion gating coefficient for the t-th encoding step; W is the Sigmoid activation function. g W u W d and W m For a trainable encoding matrix; u t The encoder joint representation after fusing the three inputs at step t; Tanh(·) is the hyperbolic tangent activation function; For Hadamah accumulation;

[0048] The Transformer model decoder employs a prefix-tuned constraint generation strategy, using operational procedure constraints and equipment performance parameter constraints from the professional knowledge base as constraint prefixes P. cDuring the injection decoding process, the probability distribution of the generated sequence corresponding to the injected text is defined as follows:

[0049] ;

[0050] in, For a given In the case of generating The probability of; The final output is a complete sequence of reflective texts; The joint generation probability of the entire reflective text is obtained by multiplying the conditional probabilities over all time steps t of the text. Given historical text, joint coding features, and knowledge base constrained prefixes, the t-th term w t The conditional generation probability; This represents the entire sequence of historical lexical units generated before time t;

[0051] Based on the probability distribution of reflective text generation and causal tracing results, structured reflective texts are generated that include problem identification, decision-making logic bias, and actionable improvement directions.

[0052] The implementation methods of the task evaluation deconstruction and multi-agent association analysis include:

[0053] The implementation process of the task evaluation deconstruction includes:

[0054] The task evaluation deconstruction adopts a layered attention architecture, including: a role recognition layer and a cross-source query-key attention calculation layer; wherein,

[0055] The role recognition layer performs bidirectional long short-term memory network operations on the evaluation text, and extracts the latent features of the semantic fragments of the corresponding drone by combining the trainable role embedding matrix of the corresponding drone.

[0056] The cross-source query-key attention calculation layer uses the single-machine task allocation description in the decision text as the query vector and the word vector of the evaluation text as the key vector. It calculates the fragment importance weight of each word corresponding to the single drone through dot product similarity operation.

[0057] The implementation methods of multi-agent association analysis include:

[0058] The multi-agent association analysis module is configured with a dynamic relational graph model, and the dynamic relational graph model constructs a dynamic heterogeneous graph G. t for:

[0059] ;

[0060] Among them, nodes Represents a drone instance, edge Represents task collaboration relationships, edge type Differentiate between different interaction modes;

[0061] The multi-agent association analysis module performs a classification graph convolution operation on the dynamic heterogeneous graph, and the operation formula is as follows:

[0062] ;

[0063] Where l is the layer number of the current graph convolutional network; This is the original feature vector of neighboring drone i in the (l+1)th layer; Let be the trainable weight matrix specific to the l-th layer and corresponding to edge type r; Let be the original feature vector of neighboring drone j in the l-th layer; The core neighbor set of a dynamic heterogeneous graph; To perform a two-level summation, first iterate through all neighbors of a single edge type, then iterate through all edge types. It is a non-linear activation function;

[0064] By grouping and aggregating neighbor features according to interaction type, a drone node representation that integrates multiple types of collaborative dependencies is obtained.

[0065] The training process of the reflector model includes:

[0066] The decision-making instructions from the decision-maker, the results of the first round of mission execution, and the information on the transformation of the drone's role are obtained as initial inputs. The structured features are obtained through mission evaluation deconstruction and multi-agent correlation analysis.

[0067] Reasoning is performed based on structured features, and the original reflection text is output and written into long-term memory simultaneously. The decision-maker reads the long-term memory and the original reflection text to generate a new decision. After execution, the new task result is obtained. The original reflection text is processed by removing descriptive fragments related to a certain drone to obtain a variant reflection text with deletion.

[0068] The variant reflection text is synchronously updated to form the variant long-term memory after iteration. The decision-maker combines the variant reflection text and the variant long-term memory to generate a new decision for the comparison version, executes the new decision for the comparison version, and outputs the task results corresponding to the new decision for the comparison version.

[0069] The results of the new task and the corresponding task results of the new decision in the comparison version are fed into the reward function module. The corresponding reward value is calculated based on the difference in the performance of the two sets of tasks, which serves as a supervision signal for PPO optimization. The PPO optimization algorithm is then fed back to the reflector model, and the internal model parameters of the reflector are iteratively adjusted using the reinforcement learning loss gradient to complete one round of training.

[0070] After generating the reflection text, the process further includes calculating the semantic similarity between the newly generated reflection text and the existing content in long-term memory, and updating the corresponding bottom-level reflection instance library, mid-level micro-pattern library, and top-level macro-principle library in long-term memory according to the similarity matching results. The bottom-level reflection instance library stores the reflection text and its corresponding decision-making context features; the mid-level micro-pattern library extracts high-frequency reflection themes and solutions; and the top-level macro-principle library stores standardized system operation principles. The update process includes:

[0071] Configure a differentiated memory update mechanism based on semantic similarity judgment, perform semantic encoding on the newly generated reflection text, and calculate the semantic similarity between the features of the new reflection text and the features of historical memory entries;

[0072] Calculate its semantic similarity s with existing memories. i :

[0073] ;

[0074] in The transpose row vector of the newly reflected text encoding vector; This refers to the i-th stored historical memory entry in the long-term memory bank. This is the semantic feature vector obtained after semantically encoding the i-th historical long-term memory entry; This is a text semantic encoding function, responsible for converting text content into low-dimensional real-valued semantic vectors;

[0075] When the semantic similarity is below a set threshold, a new memory entry is created; when the semantic similarity is not below the set threshold, the existing memory entries are iteratively updated using a weighted fusion method. The memory update formula is as follows:

[0076] ;

[0077] in, This indicates assignment overwriting, meaning the updated result replaces the original semantic feature vector; Weighting coefficients are retained for historical memory; this formula represents the weighting after the update. Replace the original semantic feature vector ;

[0078] Furthermore, during the decision-making stage, a multi-scale linkage retrieval mechanism can be adopted to simultaneously call up three types of memory content: the underlying instance-level cases, the mid-level micro-execution models, and the top-level macro principles, to collaboratively participate in the reflection and auxiliary reasoning of the current decision.

[0079] The process of dynamically updating the thought chain includes:

[0080] The dynamic reasoning formula satisfied by the updating process of the thought chain is:

[0081] ;

[0082] in, This represents the state of the thought chain at time t+1; This represents the state of the thought chain at time t; Top-level target prompts provided to system schedulers; Encoding of real-time situation; It is a long short-term memory system;

[0083] Among them, real-time situation coding The calculation formula is:

[0084] ;

[0085] in, For encoder function, r t Indicates resource status, e t For non-cooperative node characteristics, u t Vector of the drone role;

[0086] Long Short-Term Memory System The built-in micro-pattern retrieval network uses an attention mechanism to match historically similar cases and completes memory retrieval by calculating matching weights. The formula for calculating the matching weights is as follows:

[0087] ;

[0088] in, q represents the matching weight between the current situation and the i-th historical memory case. T Let k be the current situation query vector. i The key vector of the memory item is the encoded feature of the i-th historical case in the memory bank. j is a general index for traversing all cases in the memory, covering all stored memory entries;

[0089] The candidate solutions and corresponding reflection texts output from the above-mentioned thought chain update process are input into the counterfactual reasoning module, and the optimal strategy is obtained by solving the three-layer computational architecture of counterfactual reasoning; wherein, the three-layer computational architecture consists of a lightweight scenario simulator, a strategy mutation engine and a decision optimization module from bottom to top;

[0090] The policy mutation engine performs semantic parsing on the input reflection text, converts the reflection text into parameter adjustment terms and generates a candidate policy set, and parameterizes each candidate policy into a corresponding policy distribution.

[0091] The lightweight scenario simulator performs virtual scenario simulations on all candidate strategies in parallel, and calculates the expected cumulative return for each candidate strategy using the expected cumulative return formula, which is as follows:

[0092] ;

[0093] in, As a discount factor, For instant rewards, Candidate strategies A parameterized representation, Represents the k-th candidate strategy The expected cumulative reward. Here, t is the mathematical expectation operator, t is the deduction time step, and T is the maximum deduction duration;

[0094] The decision optimization module employs a breadth-first search strategy to screen candidate solutions and determines the optimal strategy based on Pareto front analysis. The formula for solving the optimal strategy is as follows:

[0095] ;

[0096] in, Indicates the degree of strategy risk; These are the weighting coefficients corresponding to returns and risks; To maximize the solution operator; This is the optimal strategy obtained through final selection;

[0097] The optimal strategy is constructed as a knowledge pair of hypothesis and verification results and stored in a long-term memory knowledge base. The formula for calculation is:

[0098] ;

[0099] in, Encoding for policy features, for The policy feature vector obtained by the encoder; The optimal strategy obtained through Pareto selection ; To increase the quantity of the product, Environmental characteristics.

[0100] The thought chain update process is also configured with a real-time guarantee mechanism, including:

[0101] The thought process chain is broken down into three parallel sub-modules: goal decomposition, resource allocation, and solution generation. A timeout judgment logic is set up. When the time consumed in the inference layer is greater than or equal to the warning threshold for the inference stage, a simplified inference mode is activated. The simplified inference mode calculation formula is:

[0102] ;

[0103] in, To enable lightweight updates to the thought chain after simplifying reasoning output, This represents the state of the historical thought chain before the update. Let g represent the k most relevant memory items retrieved from long-term memory, and g is a lightweight inference function for memory enhancement.

[0104] Furthermore, the total time consumption of a single thought chain update must meet the following constraint: the sum of the input encoding time, the core reasoning layer time, and the output decoding time must not exceed the global total time limit threshold.

[0105] Compared with existing technologies, the large-scale model decision optimization method based on environmental feedback provided by this invention can continuously revise and optimize the original decision-making process by constructing a closed-loop optimization mechanism of "evaluation-reflection-thinking chain update". Specifically, an evaluator model can be introduced to evaluate the decision results from multiple dimensions to generate semantic feedback with causal analysis and improvement orientation. Furthermore, a reflection model can be used to identify logical deviations, goal deviations, and strategic deficiencies in the original decision-making thinking chain, forming reflection content that can be used for subsequent decision optimization. Then, the thinking chain can be dynamically updated based on the evaluation information and reflection results, enabling the large model to continuously improve its decision-making ability without frequent large-scale retraining. Attached Figure Description

[0106] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0107] Figure 1 A schematic diagram illustrating the implementation process of the method provided in this embodiment of the invention;

[0108] Figure 2 This is a schematic diagram of a task result evaluation and processing flow based on multimedia feedback provided in an embodiment of the present invention;

[0109] Figure 3This is a schematic diagram of the evaluator training process provided in an embodiment of the present invention;

[0110] Figure 4 A schematic diagram of the reflection processing flow based on environmental situation and evaluation semantic enhancement provided in an embodiment of the present invention;

[0111] Figure 5 This is a schematic diagram of the reflector training process provided in an embodiment of the present invention;

[0112] Figure 6 This is a schematic diagram of the decision-maker's thought chain update process provided in an embodiment of the present invention;

[0113] Figure 7 This diagram illustrates the performance comparison of UAVs under different algorithms provided in this embodiment of the invention. Detailed Implementation

[0114] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them, and do not constitute a limitation on the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention.

[0115] First, the following explanations are provided for the terms that may be used in this article:

[0116] The term "and / or" means that either or both can be achieved simultaneously. For example, X and / or Y means that it includes both "X" or "Y" as well as the three cases of "X and Y".

[0117] The terms "comprising," "including," "containing," "having," or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.) should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0118] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.

[0119] The term "parts by mass" indicates the mass ratio between multiple components. For example, if component X is described as x parts by mass and component Y as y parts by mass, then the mass ratio of component X to component Y is x:y. One part by mass can represent any mass; for example, one part by mass can be expressed as 1 kg or 3.1415926 kg, etc. The sum of the parts by mass of all components is not necessarily 100 parts; it can be greater than 100 parts, less than 100 parts, or equal to 100 parts. Unless otherwise stated, parts, proportions, and percentages mentioned herein are all measured by mass.

[0120] Unless otherwise explicitly specified or limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this document according to the specific circumstances.

[0121] When concentration, temperature, pressure, size, or other parameters are expressed as numerical ranges, such ranges should be understood to specifically disclose all ranges formed by any pairing of upper limits, lower limits, or preferred values ​​within that range, regardless of whether the range is explicitly stated; for example, if the numerical range "2 to 8" is stated, then that range should be interpreted to include ranges such as "2 to 7", "2 to 6", "5 to 7", "3 to 4 and 6 to 7", "3 to 5 and 7", "2 and 5 to 7", etc. Unless otherwise stated, the numerical ranges described herein include both their endpoints and all integers and fractions within that range.

[0122] The terms “center,” “longitudinal,” “lateral,” “length,” “width,” “thickness,” “upper,” “lower,” “front,” “back,” “left,” “right,” “vertical,” “horizontal,” “top,” “bottom,” “inner,” “outer,” “clockwise,” and “counterclockwise” indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience and simplification of description and do not imply that the device or component referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this document.

[0123] The following is a detailed description of a large-scale model decision optimization method based on environmental feedback provided by this invention. Contents not described in detail in the embodiments of this invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, they are performed according to conventional conditions in the art or conditions recommended by the manufacturer. Reagents or instruments used in the embodiments of this invention, unless otherwise specified by the manufacturer, are all commercially available conventional products.

[0124] In the implementation process of this invention, it was discovered that in complex and dynamic task scenarios, the implementation method of continuously adjusting strategies based on environmental changes, execution results, and external feedback can overcome the problem in existing technologies of difficulty in fully utilizing multi-dimensional feedback information after task execution. This allows the large model to improve the reliability of decisions through effective self-evaluation and reflection mechanisms when execution results deviate from expectations. Thus, for complex task systems requiring rapid response, an effective mechanism can be established to link and update evaluation results, reflection conclusions, and the thought chain reasoning process, thereby forming a closed-loop optimization process of "task execution—result evaluation—reasoning reflection—thought chain update."

[0125] The implementation scheme provided by the embodiments of the present invention can construct a large model decision optimization scheme in complex dynamic environments that can integrate multi-dimensional feedback information, automatically evaluate the decision effect, identify the defects of the reasoning chain, and update the thinking chain in real time. This implementation scheme can transform environmental feedback, task results, and historical experience into reusable reasoning knowledge without relying on frequent large-scale retraining, thereby improving the rationality, interpretability, real-time response capability, and continuous adaptive capability of the large model in complex task scenarios.

[0126] This invention provides a large-scale model decision optimization method based on environmental feedback, which can realize corresponding large-scale model reflective decision optimization schemes based on dynamic feedback from complex environments. This scheme is designed for complex dynamic task scenarios, and the corresponding processing methods mainly include:

[0127] First, the central decision-making model generates an initial decision scheme and a corresponding thought chain based on the task objective, environmental state, and execution unit state. The thought chain describes the reasoning process of the decision scheme.

[0128] Subsequently, referring to Figure 1 As shown, a multimodal feedback-based operational outcome (i.e. mission outcome) evaluation mechanism is implemented to evaluate the mission execution results from multiple dimensions through an evaluator model, generating semantic feedback that includes outcome analysis, problem identification, and improvement suggestions, and obtaining corresponding evaluation results. Furthermore, a thinking reflection mechanism based on evaluation semantic reinforcement is also implemented to analyze logical deviations, target deviations, and strategic deficiencies in the original decision-making thinking chain through a reflection model, and obtain corresponding reflection content.

[0129] Finally, an evaluation and reflection-driven thinking chain update mechanism is implemented to dynamically update the corresponding thinking chain based on the evaluation results and the reflection content, enabling the large model to continuously optimize the subsequent decision-making process without frequent large-scale retraining.

[0130] The specific implementation methods of each of the above processing steps will be explained in detail below.

[0131] (I) Task outcome evaluation mechanism based on multimodal feedback

[0132] In the above processing, the implementation method of the task result evaluation mechanism based on multimodal feedback is as follows: Figure 2 and Figure 3 As shown, it can specifically include:

[0133] In modern intelligent and complex task environments, the central decision-making model, as the decision-making core of a multi-execution unit cluster, directly determines task execution efficiency through its decision-making quality. However, traditional decision-making systems often lack dynamic evaluation and self-correction mechanisms for task execution results, leading to rigid decision-making patterns and insufficient adaptability. To address this issue, this invention introduces an evaluator module (i.e., an evaluator model) based on a large model during implementation. This module utilizes a combination of natural language generation and reinforcement learning to construct a closed-loop cognitive system covering "decision-evaluation-memory update." The overall system architecture for the corresponding result evaluation mechanism is as follows: Figure 2 As shown, this evaluator module can not only quantify and analyze the execution value of complex task decisions, but also provide decision-makers with actionable improvement directions through semantic feedback, ultimately achieving the continuous evolution of the decision-making system.

[0134] The core function of the corresponding evaluator module is to transform the multi-dimensional impact of complex task decisions into structured semantic feedback: (refer to...) Figure 2 As shown, the inputs and outputs of this evaluator architecture can include:

[0135] (1) Evaluator's input

[0136] It can contain three types of key data, which can be:

[0137] (11) The original decision-making instructions of the central decision-maker, also known as the initial decision-making instructions, are the decision-making instructions of the decision-maker shown in the figure, such as "dispatch the perception unit A to cover area X, and the execution unit B to cooperate in completing the goal Y". The corresponding original decision-making instructions usually exist in the form of natural language or structured commands.

[0138] (12) The task result data package may include parameter data information such as target achievement rate, resource consumption status (i.e., resource consumption amount) and environmental status changes (i.e., complex environment dynamics);

[0139] (13) Execution unit role conversion log, i.e. the UAV role conversion part shown in the figure, such as the execution unit switching to the perception unit due to insufficient resources, the support unit temporarily withdrawing from the task chain due to external interference, etc.; the corresponding execution unit role conversion data can be transmitted to the decision center in real time through the environmental perception network, and can be integrated into a timestamp-aligned multimodal input stream by the preprocessing module.

[0140] (2) Evaluator's output

[0141] The evaluation results output by the evaluator can include four types of quantitative evaluation indicators:

[0142] The target achievement rate, calculated as "actual number completed ÷ planned number", is used to measure the degree of task completion.

[0143] Timeliness is used to distinguish between three timeliness statuses of tasks: completed ahead of schedule, completed on time, and completed late.

[0144] The efficiency-resource ratio, calculated as "target value ÷ total energy consumption of the UAV", is used to measure the efficiency of mission input and output.

[0145] Intent alignment is used to measure the degree to which the actual performance of a drone matches the decision-maker's original mission objectives.

[0146] The evaluation results, including outcome analysis, problem identification, and improvement suggestions, are generated based on the aforementioned quantitative evaluation indicators. In other words, the output of the evaluator is not a simple score or binary judgment; it includes causal analysis, dynamic relationships between execution unit roles, and natural language text that integrates task analysis and experience summaries. For example, when a decision-maker underestimates the intensity of external interference, leading to a disruption of the support link, the evaluation text might be generated as: "The failure of the third support task stems from insufficient prediction of the interference intensity in region Z (task result verification). It is recommended to prioritize the deployment of environmental awareness units and activate backup communication methods (improvement strategy); the current execution cluster resource reserves have dropped to the threshold, requiring adjustment of the subsequent task allocation ratio (role change-related impact)." It is evident that the semantic density of this output far exceeds that of traditional reward functions.

[0147] The evaluation texts used as evaluation results are further distributed to a three-tier short-term memory storage system, and written to a three-tiered short-term memory architecture, with each tier having its own independent database, including:

[0148] The instant memory layer is used to store the evaluation data of the most recent k decisions, and it uses an iterative update mechanism to store and record recent continuous decision evaluation records;

[0149] The execution pattern layer is used to extract high-frequency evaluation patterns and store them in a key-value pair structure to capture the recurring typical task execution characteristics.

[0150] The global principle layer is used to perform aggregation analysis on the full evaluation data through clustering algorithms and to periodically update the global general decision constraints and optimization principles. The full evaluation data may include: all evaluation texts output in each cycle, as well as the corresponding pre-input data (i.e., the input data that generates the current evaluation result).

[0151] (3) The following will be combined with Figure 3 right Figure 2 The specific implementation method of the training process of the evaluator module in the evaluator is described.

[0152] Specifically, the training process for the corresponding evaluator module is as follows: Figure 3 As shown, it may include:

[0153] (31) After the evaluator obtains the decision-maker's input instructions, the first task result (1), and the UAV role transformation, the input data processing for the evaluator can adopt a hierarchical coding method, wherein:

[0154] (311) The original decision instructions (i.e., the decision-maker's decision instructions) are transformed into a structured representation through the natural language parsing module. The process can be represented as follows:

[0155] ;

[0156] Among them, c t Let d be the natural language instruction at time t. t Parser(.) is a vectorized representation of the decision-maker's instructions, and Parser(.) is a natural language parsing function.

[0157] (312) The environmental data (i.e., the result of the first task (1)) is fused through multi-source sensors to form the corresponding environmental data spatiotemporal feature matrix S. t ,for:

[0158] ;

[0159] in, m t r represents the task objective completion rate index. t Let e ​​be the resource state vector. t Then it encodes dynamic information about the external environment. This is an element-wise addition and fusion process;

[0160] (313) The record of UAV role changes (i.e. UAV role transitions) is modeled as a state transition graph G. t,for:

[0161] ;

[0162] Among them, nodes Represents individual drones, edges Depict the relationship between the changing roles.

[0163] (32) To enable the evaluator to achieve the corresponding output capability, based on the above-mentioned input data of the evaluator, the evaluator module can adopt a two-stage generation strategy:

[0164] First, a domain knowledge encoder based on Transformer (self-attention transformer) is used to perform deep feature extraction on multi-source input data of the task, so as to extract the task features of the input data and explore the hidden causal relationship between task elements, such as the implicit relationship between "ensuring interruption" and "interference intensity".

[0165] Subsequently, the evaluation generator uses the extracted real-time task features as the core reasoning basis, and combines the domain-specific cases and dialogue templates provided by the task dataset to generate professional comments that conform to the task context, that is, to generate corresponding evaluation texts adapted to the scheduling business scenario.

[0166] Specifically, Figure 3 The evaluator shown (i.e., the evaluation generation network) employs an encoder-decoder architecture, where:

[0167] The encoder consists of three parallel Transformer branches, which process decision instructions, environment states, and role transition graphs, respectively; the outputs of each branch are fused through a cross-attention mechanism.

[0168] ;

[0169] Among them, h t Let be the multi-source fusion global representation vector at time t, i.e., the fusion representation. For cross-attention fusion operators, This is a single-branch Transformer encoder function;

[0170] The decoder is based on the fusion representation h t Autoregressive generation of evaluation text y t Its probability distribution is:

[0171] ;

[0172] in, For fusion-based characterization h t Autoregressive generation of evaluation text y t The probability distribution, w k To evaluate the k-th word in the text sequence, This refers to the sequence of all preceding words generated before the k-th word. For a given and h t At that time, w k The conditional probability of occurrence;

[0173] (33) Based on the evaluation text generated above, realize the online reinforcement learning closed-loop training of the evaluator model;

[0174] Specifically, the generated evaluation text can be written into short-term memory storage on the one hand; on the other hand, it can be input into the decision processing part to generate a corresponding new decision; and after executing the new decision, a second task result (2) is generated. The second task result (2) can be used as the first task result (1) for the next processing. At the same time, it is also input into the reward function processing part to calculate the corresponding reward value to judge the quality of the evaluation text and the new decision in this round.

[0175] The corresponding reward function outputs a reward signal, which is then passed back to the evaluator module through the PPO (Proximal Policy Optimization) algorithm. This PPO algorithm stabilizes and constrains the network parameter update amplitude, preventing model oscillations, and synchronously updates all network weights of the evaluator's internal Transformer domain knowledge encoder and evaluation generator, enabling continuous iterative training of the evaluator model.

[0176] The iterative execution of the above process can realize the complete iterative process of online reinforcement learning closed-loop training for the evaluator.

[0177] (34) To improve the task relevance of the evaluation, a pre-training stage for the evaluator model is also included before performing the above-mentioned online reinforcement learning closed-loop training;

[0178] The corresponding evaluator model needs to be infused with relevant domain knowledge during the pre-training phase. Specifically, this pre-training phase can adopt a two-stage training strategy:

[0179] Phase 1: Complete the training of the base model on a general corpus. Use a general large-scale corpus to train the Transformer base model and learn the general language and basic semantic expression capabilities.

[0180] Phase Two involves fine-tuning the task log data using domain adaptation methods. For example, using UAV task logs, historical evaluation records, and task indicator datasets for domain adaptation fine-tuning, the model is injected with UAV domain knowledge to obtain an initial evaluator model with basic task understanding capabilities.

[0181] In stage two, the corresponding fine-tuning objective function can be:

[0182] ;

[0183] in, To fine-tune the loss function for the neighborhood, To analyze the dataset from the task Calculate the expected value of the sample (x, y) from the sampled data. For the provided task dataset, Generate the log-conditional probability of the standard evaluation y for a given input x;

[0184] During the fine-tuning process, each generated evaluation text y t The information is sent to the decision-maker's prompting section, forming a new decision-making context; the system records changes in task benefits before and after evaluation and adoption. And construct counterfactual comparison samples. To calculate the counterfactual advantage:

[0185] ;

[0186] in, For the counterfactual advantage in round t, R represents the difference in actual benefits before and after the evaluation. t Before this round of evaluation is adopted, the immediate task benefit (reward function output value) of the current task round, R t+1 To generate the task rewards for the next round after the new decision is implemented, based on the evaluation results from this round, The expected value of the counterfactual virtual return difference represents the average baseline return;

[0187] This advantage signal (i.e., counterfactual advantage) is used to guide the PPO optimization of the evaluation model, and its objective function is:

[0188] ;

[0189] in, Optimize the loss function for the PPO strategy. To calculate the mean expected value of the loss over all time steps, This represents the probability ratio between the old and new strategies. To obtain the minimum of the two, To be Limited to the range , This is for pruning hyperparameters.

[0190] (35) Implement an evaluation mechanism based on a shared evaluation system framework;

[0191] After training the evaluator module, directly using the pre-trained large model as the evaluator will face the challenge of balancing semantic generalization and task specialization. Therefore, this embodiment of the invention implements an evaluation mechanism based on a shared evaluation system framework. That is, the corresponding evaluator model can include personalized evaluation generation based on role awareness and iterative updates of short-term memory, so that it can work with the reflector to form a closed-loop optimization mechanism to improve the performance of the decision-making system; wherein:

[0192] (351) The personalized evaluation generation of role perception refers to the evaluator generating differentiated feedback for UAVs with different functions; for example, the evaluation of inspection UAVs can focus on data accuracy (such as "false alarms of infrared sensors lead to loss of search targets"), while the evaluation of core operation UAVs can focus on execution efficiency (such as "the intervention effect of conventional payloads on specific targets is insufficient"). To this end, the evaluator embeds UAV role description vectors in the input layer and dynamically adjusts the evaluation generation path through a lightweight adapter. This design method strengthens the role specificity of professional fields. For example, the evaluation template of material support UAVs will include logistics risk assessment fields by default, while the template of core operation UAVs will embed execution efficiency indicators.

[0193] (352) The iterative update of short-term memory refers to the need for the evaluator's output to be converted into short-term memory in order to provide the decision-maker's cognition with the impetus for evolution; for this purpose, a dynamic memory pyramid is constructed, which includes an immediate memory layer, an execution mode layer, and a global principle layer, wherein:

[0194] The instant memory layer stores the most recent predetermined number of decision-evaluation pairs (e.g., 3 times) in the form of a sliding window, and fuses them by weighting through an attention mechanism; for example, if the current decision involves dealing with high-frequency interference, the weight of relevant evaluations in the history (e.g., "anti-interference measures recommendations") is automatically strengthened.

[0195] The execution mode layer is used to abstract high-frequency evaluation topics into searchable execution modes; for example, if the evaluation of "supply route optimization" appears more than 5 times, a "logistics vulnerability mitigation" mode card will be automatically generated for decision-makers to call directly.

[0196] The global principle layer forms top-level rules through long-term recommendations in cluster evaluation (such as "prioritizing the detection of key nodes").

[0197] In the iterative update process of the short-term memory, the corresponding memory update mechanism is based on the design and implementation of the long short-term memory of the COPPER framework. Edge computing optimization is introduced to address the real-time requirements of the environment. For example, the immediate memory layer is deployed on the master node server, while the higher-order memory layer is synchronized to the cloud for offline mining. Since evaluators will query the task dataset as needed during the pre-training phase and during the evaluation generation process, the content of the short-term memory will include the content of the task dataset. Specifically, the update methods adopted for the immediate memory layer, execution mode layer, and global principle layer include:

[0198] Corresponding instant memory The instantaneous memory storage vector at time t can be updated iteratively based on the most recent k decision and evaluation records. for:

[0199] ;

[0200] Where, d j Let y be the structured decision instruction vector encoded by Parser in the j-th round. j The evaluation text representation vector generated for the j-th round of the model is a sequence composed of the decision vector and evaluation vector of each round from round tk to round t, where k is the sliding window length, representing the retention of the most recent k rounds of historical records. It is a gated loop unit used for sequential iterative updates of the instantaneous memory state;

[0201] Corresponding execution mode memory High-frequency evaluation patterns can be stored in key-value format, where the key represents contextual features. , value is the associated evaluation template The corresponding search formula used in the search process can be:

[0202] ;

[0203] in, The optimal evaluation template retrieved and matched based on the current context at time t. As a normalization function, feature similarity is converted into probability weights. To transpose the task environment feature vector at time t, W is a trainable similarity mapping weight matrix. For execution mode memory;

[0204] The above retrieval formula can be used to extract the formula from the above formula. The similarity probability vector output by the operation is used as the execution mode memory. The criteria for iterative updates; specifically, when calculating the retrieval formula, extracting from the formula... The output similarity probability vector is used to compare the maximum matching probability (i.e., the maximum similarity probability) with a preset threshold, and the execution mode memory is then completed based on the comparison result. Iterative update operations.

[0205] Corresponding global memory Then, the abstract task principles can be formed by periodically updating the clustering algorithm.

[0206] The above-mentioned technical processing achieves continuous and dynamic evaluation of task environment decisions by tightly coupling semantic generation and reinforcement learning mechanisms; the corresponding evaluator not only parses explicit task results, but also extracts evaluation information from implicit signals such as drone role transformation, providing a stable foundation for the subsequent generation of reflection modules.

[0207] (II) Reflection Mechanism Based on Link Status and Evaluation Semantics

[0208] In this reflection mechanism, the process of generating the corresponding reflection text may include:

[0209] The process of acquiring the original decision text, the result data package from the drone swarm feedback, and the high-order semantic feedback generated by the evaluator, and performing parallel joint parsing and processing includes:

[0210] Task evaluation fragment deconstruction is used to extract the unique evaluation fragment corresponding to a single drone from the complete evaluation semantics using a role-aware deep neural network.

[0211] Multi-agent association analysis is used to analyze the collaborative dependency relationships among swarmed drones.

[0212] Decision indicator tracing and fusion is used to correlate task quantitative indicators to complete the fusion of multi-source input features and form a comprehensive feature representation of the task.

[0213] Using the evaluation segments determined by task evaluation deconstruction, cluster collaborative association, and integrated task comprehensive features as input, and relying on the multi-head attention task decomposition module and the domain knowledge graph causal reasoning engine, the cause-and-effect tracing of decision defects is carried out.

[0214] Based on the output of causal tracing, a standardized and structured reflection text is generated, which includes specific problem identification, decision-making logic deviations, and actionable improvement directions.

[0215] Calculate the semantic similarity between the newly generated reflective text and the content in the long-term memory inventory. Based on the similarity matching results, update the immediate memory layer, execution mode layer, and global principle layer of the long-term memory layer in a hierarchical manner.

[0216] Specifically, the implementation method of the reflection mechanism based on link status and evaluation semantic reinforcement is as follows: Figure 4 and Figure 5As shown, it can specifically include:

[0217] (1) Inputs and outputs of the reflector

[0218] The reflector in the decision-making system plays a core role in inferring logical flaws in the decision-making process from the results of execution. The corresponding overall system architecture of the reflector is as follows: Figure 4 As shown, where:

[0219] (A) The reflector's input data stream comprises three key components:

[0220] (11) The original decision text of the central decision-maker, i.e. the decision-maker's decision instructions, usually describes the task objectives and task allocation scheme in structured natural language, such as "prioritize the handling of the interference node at coordinates (X,Y), with inspection unit A responsible for target location and core unit B implementing precise intervention";

[0221] (12) The result data packet fed back by the UAV swarm, i.e. the task result, may include quantitative indicators such as target achievement rate, resource consumption status (i.e. resource consumption), environmental interference intensity (i.e. complex environment dynamics), and role transformation records of each UAV during the execution process.

[0222] (13) The high-order semantic feedback generated by the evaluator, i.e. the evaluator's evaluation text, such as "Core unit B failed to obtain the data update provided by inspection unit A in a timely manner, resulting in the intervention deviating from the target core area by 3.2 meters".

[0223] (B) The output of the reflector is an in-depth analytical text for decision improvement, namely the corresponding reflective text, which can not only point out the specific problems, but also trace the logical deviations in the decision-making process, thereby providing the system with actionable directions for improvement.

[0224] In the process of generating reflective texts, the output of reflective thinkers can involve two stages:

[0225] Phase 1 involves task-level deconstruction of the evaluators' comprehensive assessments to identify evaluation segments directly relevant to each drone. This process relies on a role-aware analysis model built from deep neural networks.

[0226] Phase Two involves causal tracing within the decision-making context, conducting multi-dimensional causal analysis, and ultimately generating reflective content with execution guidance value—that is, generating reflective text. For example, when receiving feedback from evaluators regarding "communication delays leading to collaboration failures," the reflector will first isolate the criticism of the inspection unit's data transmission timeliness, and then, combined with the communication protocol parameters set during decision-making, generate reflective text such as "The communication interval threshold set during decision-making did not consider signal attenuation under strong interference environments; it is recommended to dynamically adjust it to 300ms." This not only points out the manifestation of communication delays but also traces back to the root cause: the mismatch between the communication parameters set during decision-making and the on-site environment. Technically, this process relies on a task decomposition module with a multi-head attention mechanism and a causal reasoning engine based on a professional knowledge graph. The corresponding decision-making context originates from the decision-maker's decision instructions, task result data, and the evaluator's output evaluation text.

[0227] (2) Reflective text generation and processing based on environmental situation and evaluation semantic enhancement

[0228] Based on the input and output information of the reflector mentioned above, the corresponding reflection text generation process is as follows: Figure 4 As shown, it can specifically include:

[0229] (21) After receiving three types of input information, the reflector module processes the corresponding input information through three parallel and linked analysis tasks, which may include:

[0230] Task and evaluation deconstruction are used to identify content fragments related to a specific drone mission from all inputs, which is equivalent to the first stage of the process of generating reflective text.

[0231] Multi-agent association analysis is used to identify collaborative dependencies within a drone swarm.

[0232] Decision tracing is used to connect various key indicators and integrate features to form a comprehensive understanding of this task;

[0233] The corresponding multi-agent correlation analysis and decision tracing output analysis results jointly support the multi-dimensional causal tracing processing in stage two of the reflective text generation process;

[0234] (22) Generate reflective text based on the above three parallel and interconnected analysis results;

[0235] The corresponding reflective text generation process relies on the aforementioned task, evaluation deconstruction, and the results of three types of analysis: multi-agent correlation analysis and decision attribution, to produce structured reflective texts, such as... Figure 4 As shown, the content of a reflection text can include: the specific problems, deviations in decision-making logic, and directions for future improvement.

[0236] Furthermore, still referring to Figure 4 As shown, for the newly generated reflective text, semantic similarity calculation is performed between it and the existing content in long-term memory. Then, the three levels of long-term memory are updated according to the matching results, including: immediate memory layer, execution mode layer, and global principle layer, to complete the experience accumulation.

[0237] (3) The training process of the reflector model

[0238] The core of the reflective model includes task evaluation deconstruction and multi-agent association analysis. The core goal of training the reflective model is to optimize the parameters of the corresponding task evaluation deconstruction and multi-agent association analysis processes through the loss function so that it can accurately complete the corresponding analysis tasks.

[0239] Specifically, the training process of this reflector model may include:

[0240] Step 31: Input the original task information to drive the reflector to generate basic output;

[0241] The three types of initial inputs to the reflector module include: decision-maker's decision instructions, the results of the first round of task execution (1), and drone role conversion information;

[0242] The reflector preprocesses the three types of initial inputs through task evaluation deconstruction and multi-agent association analysis, and converts them into structured features that the model can compute, which serve as the encoder input for the reflector. Then, the reflector completes reasoning based on the corresponding initial inputs and simultaneously outputs two types of products: long-term memory stored in the system, and complete original reflection text.

[0243] Step 32: First decision simulation, obtaining the first round of task feedback;

[0244] That is, the decision-maker reads long-term memory and complete reflection text to generate a new decision (1); after executing the new decision (1), the corresponding new task result (2) is obtained;

[0245] Step 33: Construct a comparative reflection sample and generate variant reflection texts;

[0246] Specifically, the original reflective text is processed to remove descriptive fragments related to a certain drone, resulting in a modified version of the text, i.e., the modified reflective text.

[0247] Step 34: Based on the variant reflection text, obtain the task results corresponding to the new decision in the comparison version (3);

[0248] Specifically, the variant reflection text can be synchronously updated to form the variant long-term memory after iteration; then the decision-maker combines the variant reflection text and the variant long-term memory to generate a new decision for the comparison version (2), executes the new decision (2), and outputs the task result corresponding to the new decision for the comparison version (3);

[0249] Through the above steps 31-34, we can obtain the new task results (2) corresponding to the original reflection sample (i.e. standard sample) and the task results (3) corresponding to the new decision of the control version obtained based on the variant reflection text (i.e. control sample).

[0250] Step 35: The reward function collects two types of task results, namely the new task result (2) and the task result corresponding to the new decision in the comparison version (3), and calculates the optimization reward value;

[0251] Specifically, the results of the new task (2) and the results of the task corresponding to the new decision in the new version (3) can all be sent to the reward function module. By comparing the differences in the execution effects of the two sets of tasks, the corresponding reward / loss values ​​can be quantitatively calculated as a monitoring signal for PPO optimization.

[0252] Step 36: Update and optimize the reflector model in reverse based on the PPO algorithm;

[0253] The reward value output by the reward function is fed back to the reflector module through the PPO optimization algorithm. The internal model parameters of the reflector are iteratively adjusted using the reinforcement learning loss gradient to complete one round of training and update.

[0254] Through this PPO reinforcement training process, a closed-loop iterative optimization of the entire chain can be achieved for task evaluation deconstruction, multi-agent correlation analysis, and reflection text generation; thereby improving the quality of reflection text, making the reflection text contain accurate single UAV defects and complete cross-unit collaborative causality, and guiding decision-makers to produce better scheduling results;

[0255] Specifically, the PPO algorithm backpropagates the reward gradient, which can simultaneously improve the encoder fusion and constraint decoding capabilities of the Transformer model for reflective text generation, as well as optimize the feature extraction model for upstream task evaluation deconstruction and multi-agent association analysis, enabling it to more accurately identify UAV semantic segments and uncover implicit cross-unit influences. This allows the deconstruction, association analysis, and reflective generation processes updated after training to produce higher-quality reflective texts in the next round of training, continuously iterating the training process until the training termination conditions are met.

[0256] The following section will describe in detail the specific implementation of the core components of the reflective model.

[0257] (31) The Reflector has deployed a dedicated task deconstruction engine.

[0258] This task deconstruction engine is used to implement corresponding task evaluation deconstruction processing. It can use an attention-based text analysis method to accurately identify semantic segments related to specific UAVs in the evaluation content. In the process of building the task deconstruction engine, the system needs to preload the functional characteristic description of each UAV so that it can automatically establish the mapping relationship between the evaluation segments and the work units when parsing the evaluation text. This allows such fine-grained evaluation deconstruction to lay the corresponding data foundation for subsequent targeted reflection.

[0259] Furthermore, the task deconstruction module within the reflector can adopt a hierarchical attention architecture, including a role recognition layer and a cross-source query-key attention calculation layer. The role recognition layer performs bidirectional long short-term memory network operations on the evaluation text, and extracts the latent features of the semantic fragments of the corresponding drone by combining the trainable role embedding matrix of the corresponding drone. The cross-source query-key attention calculation layer uses the single-machine task allocation description in the decision text as the query vector and the word vector of the evaluation text as the key vector, and calculates the fragment importance weight of each word corresponding to the single drone through dot product similarity operation.

[0260] In other words, the task deconstruction module can employ a hierarchical attention mechanism to achieve fine-grained parsing of the evaluation text; specifically, based on paragraphs output by a given evaluator... First, semantic fragments related to a specific drone are extracted through the role recognition layer:

[0261] ;

[0262] in, Let be the latent feature vector of the semantic fragment of the character corresponding to the i-th drone. It is a bidirectional long short-term memory network operator used to extract global temporal semantic features of the entire evaluation text. It is a trainable role embedding matrix, corresponding to the functional features of the i-th drone;

[0263] The importance weights of each segment can be calculated using query-key-value attention:

[0264] ;

[0265] Among them, a ij Let q be the importance weight of the i-th drone and the j-th evaluation text terminology. i Task assignment description from the decision text, k j Word vectors from the evaluation text, The dot product of query and key is used to measure the value of drone mission q. i and evaluation term k j similarity, For natural index calculation, n is the total number of terms in the evaluated text, and i and j are the term indices of the evaluated text.

[0266] Based on the semantic fragments related to the specific drone extracted above Using the importance weights of segments evaluating text words a ij The evaluation text is weighted and fused using word vectors, i.e., grouped by drone index i, and the global evaluation text is decomposed, filtered, and recombined. Then, for each drone i, all high-weight related evaluation segments are aggregated and compressed to generate a structured deconstructed evaluation unit specific to that drone. Finally, all N drones are traversed to obtain the deconstructed evaluation unit. This, along with its clear mapping relationship with drone swarms, lays the foundation for subsequent targeted reflection.

[0267] (32) Multi-agent association analysis

[0268] To further grasp the overall situation of drone swarm collaboration, Reflector also constructed a dynamic relational graph model for multi-agent association analysis. This model represents each drone as a node, with connections between nodes representing task collaboration relationships. The graph also distinguishes different types of interaction modes. Based on this dynamic relational graph model, through graph neural network analysis, the system can capture indirect impacts not explicitly stated in the evaluation text. For example, it can identify how the route selection of a certain material unit indirectly limits the effectiveness of other units. This multi-agent association analysis capability based on the dynamic relational graph model allows Reflector to examine the global impact of decisions at the system level.

[0269] The multi-agent association analysis module can model the collaborative dependencies between UAVs through graph neural networks. The specific implementation process may include:

[0270] First, construct a dynamic heterogeneous graph. , where nodes Represents a drone instance, edge Represents task collaboration relationships, edge type Different interaction modes such as inspection-intervention and replenishment-operation are distinguished.

[0271] Subsequently, graph convolution operations are performed based on the dynamic heterogeneous graph to quantify the collaborative dependencies between UAVs. The corresponding graph convolution operation formula can be:

[0272] ;

[0273] Where l is the layer number of the current graph convolutional network; This is the original feature vector of neighboring drone i in the (l+1)th layer; Let be the trainable weight matrix specific to the l-th layer and corresponding to edge type r; Let be the original feature vector of neighboring drone j in the l-th layer; The core neighbor set of a dynamic heterogeneous graph, i.e., in the dynamic heterogeneous graph G t In the context, all neighboring drones j connected to node i via an edge of type r; To perform a two-level summation, first iterate through all neighbors of a single edge type, then iterate through all edge types. It is a non-linear activation function;

[0274] Through the above processing, the multi-agent association analysis module can identify cross-unit effects that are not explicitly stated in the evaluation text, that is, determine the corresponding collaborative dependencies, such as discovering second-order effects such as "the route selection of material unit C indirectly leads to the limited field of vision of inspection unit D".

[0275] The multi-agent association analysis module described above can uncover implicit second-order constraints across units that are not explicitly stated in the evaluation text, supplementing the missing deep-seated causal information in a single evaluation text. It also integrates these cluster linkage features into the deconstructed evaluation units, providing a global collaborative dimension input for the subsequent Transformer model that generates reflection text. This allows the model to go beyond the isolated problem summary of a single drone when decoding and generating reflection content. It can output cluster task review texts that include indirect effects from multiple drones and possess complete causal logic. Furthermore, it can supplement real-time dynamic collaboration dependencies into the knowledge base constraint prefixes, ensuring that the generated reflection conclusions align with the actual collaborative working conditions of this cluster operation.

[0276] (33) Decision generation based on task evaluation deconstruction and multi-agent association analysis: causal and reflective generation

[0277] The Reflection Generation Core Module (hereinafter referred to as the Reflection Generation Module) integrates a domain-specific knowledge base (mainly from complex scenario datasets) and a causal reasoning engine. The Reflection Generation Core Module receives deconstructed evaluation units, corresponding original decision fragments, and key indicators extracted from the execution results, and forms a comprehensive understanding through a feature fusion mechanism.

[0278] When generating reflection texts, two dimensions of prior knowledge can be referenced: (a) written operating procedures and system rules to ensure that the reflection recommendations comply with the standards; and (b) the technical parameters and performance boundaries of various equipment to ensure the feasibility of the recommendations.

[0279] The process of generating reflective texts employs controlled natural language generation technology, dynamically adjusting the expression and content focus during the output of reflective texts to ensure that the texts contain both specific improvement suggestions and elucidate the underlying decision-making principles. Moreover, the content of the generated reflective texts not only explains "how to do it" but also explains "why to do it," providing comprehensive material for cognitive upgrading.

[0280] The core of reflective text generation is a sequence-to-sequence Transformer model based on a professional causal knowledge base. The specific process of generating reflective text can include:

[0281] The encoder of the Transformer model receives three inputs: the deconstructed evaluation unit. Original decision fragment d i and key metrics (m) extracted from task results. i Among them, the deconstructed evaluation unit Original decision fragment d i The key indicator m is obtained through task evaluation analysis. i The three elements obtained through multi-agent association analysis are then combined into a joint representation using a two-level gating fusion mechanism. The computation method of the two-level gating fusion mechanism can be as follows:

[0282] ;

[0283] ;

[0284] Among them, g t Let be the fusion gating coefficient for the t-th encoding step, which controls the fusion ratio of evaluation information and decision + indicator information; W is the Sigmoid activation function. g W u W d and W m Both are trainable encoding matrices; u t Let be the joint encoder representation after fusing the three inputs at step t; Tanh(·) is the hyperbolic tangent activation function, which performs nonlinear feature mapping on the three inputs respectively; This is the Hadamard product, which is the product of elements one by one.

[0285] Its decoder employs a constraint generation strategy to ensure that the output text conforms to the expression specifications of the operating procedures. Specifically, it can inject two types of prior knowledge into the decoding process through prefix-tuning: first, the scheme rules involved in the dataset, such as "at least two inspection searches must be completed before intervening in the target"; second, component performance parameters, such as "the maximum communication latency threshold of a certain type of UAV". The probability distribution of the generated reflection text is as follows:

[0286] ;

[0287] in, For a given In the case of generating The probability of; For the final output, a complete sequence of reflective texts; P c This indicates the constraint prefix retrieved from the professional knowledge base; The joint generation probability of the entire reflective text is obtained by multiplying the conditional probabilities over all time steps t of the text. Given historical text, joint encoding features, and knowledge base constrained prefixes, the conditional generation probability of the current t-th lexical unit; w t To reflect on the t-th word element of the text; This represents the entire sequence of historical lexical units generated before time t;

[0288] Similar to the evaluator module, the reflector module also employs a two-stage strategy of training and fine-tuning the pedestal model during training. The fine-tuning technique is the same as that used in the evaluator module.

[0289] The reflection generation module is equipped with a long-term memory construction and update mechanism. Reflectors manage this long-term memory through a three-tiered storage architecture, leveraging historical cases to support the generation of current reflection texts, quickly identify shortcomings in similar tasks, continuously accumulate transferable problem-solving patterns and standardized operating principles, avoid repeating similar mistakes in subsequent decisions, and continuously optimize drone swarm scheduling, task execution, and risk avoidance capabilities. The three-tiered storage architecture specifically includes:

[0290] The underlying memory bank is a reflection instance bank oriented towards specific cases. It is an immediate memory layer, also known as the underlying reflection instance bank. It completely preserves the original text of each reflection and its corresponding decision-making context characteristics, forming a traceable case archive.

[0291] The middle-level memory bank is the micro-pattern library, that is, the execution pattern layer. It automatically extracts frequently occurring reflection themes and summarizes solutions to common problems through machine learning algorithms.

[0292] The top-level memory is the top-level macro-principle library, or global principle layer, used to store system principles that have been verified by experts, providing fundamental guidance for decision-making.

[0293] To manage long-term memory using the aforementioned three-level storage architecture, a corresponding memory update mechanism is also adopted. This memory update mechanism follows the principle of intelligent filtering, that is, newly generated reflection texts are first compared with existing memories for semantic similarity. Independent memory items are created for new types of reflections, while knowledge reinforcement is performed on existing types of reflections. The system will periodically optimize and organize the memory bank, merge similar topics, and remove outdated content to maintain the timeliness and usability of the memory system.

[0294] The specific memory update process needs to follow an importance-based sampling strategy, including: for newly generated reflective texts First, calculate its semantic similarity s with existing memories. i :

[0295] ;

[0296] in The transpose row vector of the newly reflected text encoding vector; This refers to the i-th stored historical memory entry in the long-term memory bank. This is the semantic feature vector obtained after semantically encoding the i-th historical long-term memory entry; This is a text semantic encoding function, responsible for converting text content into low-dimensional real-valued semantic vectors;

[0297] According to the calculation results, when s i Below the threshold If a new memory entry is created, it is triggered automatically; otherwise, knowledge reinforcement is performed through weighted averaging. This involves using a weighted fusion method to combine old memory knowledge with the new reflection text, updating the original memory entry so that it retains both historical experience and the conclusions of the current review.

[0298] The specific memory update formula used is as follows:

[0299] ;

[0300] in, This indicates assignment overwriting, meaning the updated result replaces the original semantic feature vector; Weighting coefficients are retained for historical memory; this formula represents the weighting after the update. Replace the original semantic feature vector ;

[0301] The above memory update formula can be used to strengthen and update knowledge when semantics are similar, so as to integrate the original historical memory with the new reflective text and complete the iterative update of old memory.

[0302] After achieving the construction and updating of long-term memory for reflective texts as described above, during the decision-making stage, a retrieval mechanism with multi-scale query and multi-level linkage can be adopted for reflective memory. The system will simultaneously call memory content at three levels: instance-level specific cases, micro-patterns, and macro-principles. Instance-level memory provides similar case references, pattern-level memory prompts common execution defects, and principle-level memory ensures that decisions comply with norms. Relying on the collaborative output of the three-level memory, the system provides decision-makers with comprehensive reference information to ensure that the results of historical reflection can continue to influence the quality of subsequent decisions, thereby constructing a virtuous and complete cognitive evolution loop.

[0303] In this embodiment of the invention, the reflector module can specifically adopt a microservice architecture to work collaboratively with other components in the system. This collaborative process can include: when the evaluator generates new output, the system automatically triggers the reflection task pipeline and allocates computing resources reasonably through the task scheduler: first, the task scheduler allocates computing resources to prioritize the processing of feedback from key node units; then, through pipeline parallel technology, the three stages of evaluation deconstruction, correlation analysis, and reflection generation are executed in an overlapping manner.

[0304] In terms of hardware deployment, the Reflector module adopts a CPU-GPU (Central Processing Unit-Graphics Processing Unit) heterogeneous computing solution: lightweight tasks such as evaluation and deconstruction run on edge computing nodes, while complex inference tasks are handled by a high-performance GPU cluster in the cloud. The processing flow adopts a parallel design, enabling efficient integration of evaluation parsing, correlation analysis, and reflection generation.

[0305] Furthermore, to meet the stringent real-time requirements of dynamic environments, the system implements a dynamic resource allocation strategy. The corresponding dynamic resource allocation process may include: the monitoring component continuously tracks the performance indicators of each processing stage, and automatically activates a simplified calculation mode when the response latency is detected to be close to the critical value; in the simplified calculation mode, corresponding emergency measures may include reducing the text generation length and simplifying the model calculation path, so as to significantly improve the processing speed while ensuring the core functions are maintained, and ensure that the end-to-end latency of reflection generation is controlled within an acceptable range, thus meeting the timeliness requirements of scheduling decisions; through this dynamic resource allocation strategy, millisecond-level response capability can be maintained in most job scenarios, fully meeting the timeliness requirements of scheduling decisions.

[0306] Through the aforementioned technologies, the Reflector module constructs a complete analytical chain from surface-level execution phenomena to deep-level decision-making logic. The reflected texts it generates not only contain specific improvement solutions but also reveal systemic deficiencies in the decision-making framework, providing central decision-makers with cognitive fuel for continuous optimization. This complements the Evaluator module, jointly promoting the continuous evolution of the system. Moreover, since the Reflector queries complex scenario datasets as needed during the pre-training phase and the generation of reflected texts, the content of long-term memory will also include the content of this dataset. This enriches the knowledge coverage of the memory bank and supplements the benchmark scenario information of various complex working conditions for the generation of reflected texts and the reference of scheduling decisions.

[0307] (III) Evaluation and Reflection-Driven Mechanism for Updating the Mind Chain

[0308] In the above process, the corresponding evaluation and reflection-driven thinking chain updating mechanism is implemented as follows: Figure 6 As shown, it can specifically include:

[0309] (1) Coupling mechanism between thinking chain architecture and task decision-making

[0310] In a dynamic environment, the Chain of Thought (CoT) of the central decision-maker plays a core role in translating global goals into specific, actionable tasks. Its update process is not a simple linear extension, but rather a dynamic reasoning system that integrates multi-source information. Its specific update framework is as follows: Figure 6 As shown, the top-level goal provided by the dispatcher (such as "inspect area A") serves as the initial prompt, the real-time situation on-site (UAV status, node distribution, mission progress, etc.) constitutes the environmental input, and the historical decision-evaluation-reflection triplet in long and short-term memory provides empirical constraints; specifically, the process of updating the decision-maker's thought chain can include:

[0311] The first stage: The central decision-maker executes the original decision deduction, that is, the generation of the basic thought chain, which may include:

[0312] Step 1: Clarify the information;

[0313] The central decision-maker reads the global input information, locks in the overall task objective of this scheduling, counts the number of drones in the cluster and assigns the execution role of each drone, and establishes the basic constraints for this round of decision-making; the global input information in this process includes the top-level objective provided by the scheduler as an initial prompt;

[0314] Step 2: Retrieve memories;

[0315] Decision-makers retrieve historical experience from the storage system, namely: reading recent past plans and corresponding effect evaluations from short-term memory, and extracting historical experience of major tasks and general scheduling principles from long-term memory to provide historical reference for reasoning;

[0316] Step 3: Analysis and Reasoning;

[0317] By combining mission objectives, UAV resources, and retrieved historical data, reasoning analysis is conducted to deduce multiple feasible scheduling schemes and complete the logical design for multi-UAV task allocation. The data used in this analysis and reasoning process includes environmental input data consisting of real-time on-site situation (UAV status, node distribution, mission progress, etc.).

[0318] Step 4: Making a decision;

[0319] The final scheduling decision is made by screening and issuing task allocation instructions to each drone, which then lands and executes the decision-making scheme.

[0320] Phase 2: Task execution, outputting two types of feedback observation data;

[0321] After the drone executes the allocation decision, the system collects two types of objective feedback information as data sources for evaluation and reflection, which may specifically include:

[0322] The operational results indicators include parameters such as mission objective achievement rate, total energy consumption of UAVs, and real-time environmental dynamics of the mission area; these operational results also need to be determined based on environmental input data consisting of real-time situation on site (UAV status, node distribution, mission progress, etc.).

[0323] Drone role change refers to the dynamic adjustment record of drone payload, function, and formation role during mission execution.

[0324] Both types of data are simultaneously fed into the evaluator (large model) and the reflector (large model) sections.

[0325] The third stage: Evaluate the large model, reflect on the large model to generate revised text, that is, generate the basis for improvement and drive the corresponding thought chain update process;

[0326] Among them, the evaluator (large model) outputs evaluation text, that is, the evaluation large model receives the full feedback data of the task execution, conducts a review and analysis of the decision-making plan in this round, and outputs evaluation text, which includes the overall merits and demerits of the plan and the experience summary of this task; moreover, the evaluation text is simultaneously written into the three-layer short-term memory storage (instant memory layer, execution mode layer, and global principle layer) to complete the real-time update of short-term memory.

[0327] The corresponding reflector (large model) outputs a reflection text. That is, the reflection large model deeply mines problems based on execution feedback and evaluation conclusions, and outputs a reflection text. It identifies the specific problems in this decision, sorts out the logical deviations in the decision, and gives actionable directions for improvement. This reflection text can be used to update the three-layer storage of the long-term memory bank: top-level memory (global principles), middle-level memory (execution mode), and bottom-level memory (specific cases) to expand the historical knowledge pair of "hypothesis-verification results".

[0328] Phase 4: Memory reflux, completing the closed loop, and realizing the iterative update of the thought chain;

[0329] The evaluated and updated short-term memory, and the reflected and updated long-term memory bank, are all fed back to the central decision-maker in the second step (memory retrieval) and the following operations are performed:

[0330] In the next round of task simulation, decision-makers will read the newly updated short-term and long-term memories, combine them with the improvement directions obtained from the reflection, and re-reason the solution to generate an optimized new decision.

[0331] In this way, the entire cycle of "decision execution → feedback collection → evaluation and reflection → memory update → memory reuse and decision re-making" is repeated continuously, iterating and updating the thought chain of the central decision-maker.

[0332] Three aspects were strengthened in the updated framework of the MindChain:

[0333] First, we introduce a field situation encoder to transform data such as role changes and resource consumption into structured feature vectors;

[0334] Second, construct a micro-pattern retrieval network to quickly match similar cases from long short-term memory (including datasets);

[0335] Third, deploy a verification module to ensure that the generated solution complies with the operating rules.

[0336] The triggering conditions for updating the thought chain are divided into two categories: event-driven and cycle-driven. Event-driven means that when the evaluator generates a new evaluation or the reflector outputs an in-depth analysis, the system immediately initiates a local thought chain revision (event-driven); cycle-driven means that global thought chain optimization is performed at fixed time intervals (such as every 30 minutes) (cycle-driven).

[0337] The corresponding update process follows a three-stage workflow: "goal alignment - solution generation - risk assessment," namely:

[0338] First, verify the logical consistency between the current sub-target (such as "inspecting area B") and the overall target;

[0339] Next, candidate solutions containing node configurations and combination methods are generated;

[0340] Finally, the expected loss ratio of each scheme was evaluated through simple simulation.

[0341] This processing mechanism can introduce additional constraints specified in the dataset on the basis of the existing continuous thinking chain, such as mandatory verification of scheme rules such as "the number of relay support unmanned equipment is ≥ 1 / 3 of the number of core operation unmanned equipment".

[0342] Furthermore, the thought process of the central decision-maker can be formalized as a dynamic reasoning system:

[0343] ;

[0344] in, This represents the state of the thought chain at time t+1; This represents the state of the thought chain at time t; Top-level target prompts provided to system schedulers; Encoding of real-time situation; It is a long short-term memory system;

[0345] This architecture is enhanced to accommodate dynamic decision-making characteristics;

[0346] In the above equation, the situation encoder converts the UAV state into a feature vector to obtain the real-time situation code. The calculation formula can be:

[0347] ;

[0348] in, For encoder function, r t Indicates resource status, e t For non-cooperative node characteristics, u t Vector of the drone role;

[0349] In the long short-term memory system Internally, the micro-pattern retrieval network matches historically similar cases through an attention mechanism. The core computation of this retrieval process is the calculation of the matching rights of historical memory cases, and its calculation formula is as follows:

[0350] ;

[0351] in, q represents the matching weight between the current situation and the i-th historical memory case. T Let k be the current situation query vector. i The key vector of a memory item is the encoded feature of the i-th historical case in the memory bank, where k is the key vector of the memory item. j j is a general index for traversing all cases in the memory (for summation), covering all stored memory entries.

[0352] (2) Optimization scheme for dynamic reasoning system based on counterfactual reasoning

[0353] The core challenge of updating the mind chain lies in how to transform abstract evaluation and reflection into actionable parameter adjustments; in this embodiment of the invention, a counterfactual reasoning framework is used to verify the effectiveness of the improvement strategy by constructing a virtual work scenario;

[0354] Specifically, the candidate solutions and corresponding reflection texts output from the aforementioned mind chain update process can be input into the counterfactual reasoning framework. The optimal strategy is obtained by solving the three-layer computational architecture of counterfactual reasoning. The optimal strategy is the updated mind chain state output by the dynamic reasoning formula. The knowledge pairs generated by counterfactual reasoning are stored in the long-term memory system to provide input constraints for subsequent mind chain iteration updates.

[0355] Furthermore, when the reflector points out that "the last push was interrupted due to interference," the system will simulate two scenarios in parallel:

[0356] One option is to maintain the original plan but add anti-interference frequency bands (counterfactual group);

[0357] Another option is to completely replace it with another solution (alternative group);

[0358] By integrating the simulations of both scenarios into the same virtual environment model and comparing execution data (such as target intervention rate and equipment integrity rate), the optimal adjustment direction can be determined. This allows for the quantification of the actual value of a single reflective suggestion and avoids the "credit allocation problem" encountered in some frameworks.

[0359] Counterfactual reasoning can be implemented using a three-layer computing architecture, including:

[0360] The underlying layer is a lightweight scene simulator, and the virtual environment built based on reinforcement learning can run hundreds of simulations in parallel.

[0361] The middle layer is a strategy mutation engine that uses semantic parsing to transform reflective text into specific parameter adjustments, such as mapping "strengthening electronic shielding" to communication protocol parameter modification.

[0362] The top layer is the decision optimization module, which adopts a breadth-first search strategy similar to a mind tree (ToT) to retain the top K best candidate solutions;

[0363] It is worth noting that the system records all counterfactual reasoning process data, forming "hypothesis-verification result" knowledge pairs stored in long-term memory. When encountering similar situations in the future, the verified strategies can be directly invoked, significantly reducing computational overhead.

[0364] Based on the above three-layer computing architecture, for a given reflective text, i.e., reflective suggestions... Candidate policy sets are generated through a mid-level policy mutation engine. And parameterized as a policy distribution Subsequently, all candidate strategies are fed into a lightweight scenario simulator for parallel simulation and deduction. The long-term return of each strategy is calculated using the built-in expected cumulative return formula of the simulator, as follows:

[0365] ;

[0366] in, As a discount factor, For instant rewards, Candidate strategies A parameterized representation, Represents the k-th candidate strategy The expected cumulative reward. Here, t is the mathematical expectation operator, t is the deduction time step, and T is the maximum deduction duration;

[0367] In the top-level decision optimization module, based on the expected cumulative returns of the corresponding candidate strategies, the optimal strategy is determined using Pareto optimality frontier analysis, and the corresponding calculation formula can be:

[0368] ;

[0369] Among them, D k Indicates the degree of strategy risk; These are the weighting coefficients corresponding to returns and risks; To maximize the solution operator, i.e., to traverse all candidate strategies Find the strategy that maximizes the overall evaluation score and use it as the global optimal output; The optimal strategy obtained through final selection is equivalent to ;

[0370] The "hypothesis-verification result" knowledge pairs generated through the above counterfactual reasoning process can be stored in the following format: a collection of all historical knowledge pairs (long-term memory knowledge base). The formula for calculation is:

[0371] ;

[0372] in, Encoding for policy features, for The policy feature vector obtained by the encoder; The optimal strategy obtained through Pareto selection ; To increase the quantity of the product, Environmental characteristics.

[0373] (3) Real-time performance guarantee and security constraint mechanism for dynamic reasoning system

[0374] (31) Real-time guarantee mechanism

[0375] In complex environments, updating the thought chain based on a dynamic reasoning system has stringent real-time requirements. Therefore, in this embodiment of the invention, a "streaming processing + incremental update" strategy can be adopted to update the decision-maker's thought chain. Specifically, the thought chain can be divided into multiple parallel-computable sub-modules (goal decomposition, resource allocation, solution generation, etc.), and a pipeline architecture can be used to achieve millisecond-level response. Furthermore, when a computation timeout is detected (e.g., exceeding 500ms), it automatically degrades to a fast reasoning mode to prioritize the computation of core functional modules and simplify non-critical steps, such as report generation. Based on this processing scheme, it is expected that most updates can be completed within 1 to 10 seconds, which is significantly faster than traditional serial reasoning.

[0376] (32) Security constraint mechanism

[0377] Security constraints permeate the entire process of mind chain updating, including:

[0378] At the input layer, deploy a security term filter to block prompts containing illegal or dangerous items (such as prohibited substances);

[0379] At the inference layer, a system specification alignment module is embedded to ensure that the generated solution complies with standard security procedures;

[0380] At the output layer, an administrator review mechanism is set up to require manual confirmation for high-risk operations (such as intervention in sensitive areas).

[0381] To implement the above measures, a dedicated rule engine can be added, for example, to automatically detect whether the scheme violates equipment safety regulations.

[0382] The system response time constraint is expressed as:

[0383] ;

[0384] in, The total time taken for a single complete thought chain update, including the total computation time of the three stages of encoding, reasoning, and output decoding; The input encoding time corresponds to the input layer processing time (input layer security operations such as situational coding, security term filtering, and prompt word verification are all included in this time segment). The time consumed by the core reasoning layer, i.e. the time consumed by the reasoning stage is calculated separately, corresponds to all the calculations of the reasoning layer, such as counterfactual simulation, Pareto strategy selection, norm alignment security verification, and memory retrieval. The time consumed by decoding the output is included in the time consumed by the corresponding output layer processing (such as scheme formatting, high-risk manual review and verification, report generation, and other output layer security processes). The system defines the maximum total time allowed for a single thought update, which is the global total time limit threshold.

[0385] Check if the following formula is true:

[0386] ;

[0387] in, This is the warning threshold for the time consumed during the inference phase;

[0388] When the formula is true, the following simplified reasoning mode is activated:

[0389] ;

[0390] in, To enable lightweight updates to the thought chain after simplifying reasoning output, This represents the state of the historical thought chain before the update. Let g represent the k most relevant memory items retrieved from long-term memory, and g is a lightweight inference function for memory enhancement.

[0391] (4) Integrated deployment of the scheduling system, i.e., the underlying deployment scheme to support the implementation of the thinking chain update mechanism.

[0392] The technical solution provided by this invention can achieve modular integration in a real scheduling system; the corresponding central decision-making model can be deployed on edge computing nodes and keep data synchronized with each UAV terminal through a secure network; the evaluator and reflector run as independent services in the cloud and transmit analysis results back through an encrypted protocol; the long short-term memory adopts a "central-region" two-level storage architecture, which ensures global knowledge sharing and meets the low-latency access requirements of local scenarios.

[0393] In addition, the embodiments of the present invention can be combined with the following technologies to achieve better decision-making performance:

[0394] First, we introduce world model technology to enable the mind chain to predict possible countermeasures from interference sources.

[0395] Second, explore collaborative reasoning among multiple nodes to achieve cross-regional macro-level integration of thought chains;

[0396] Third, optimize the memory compression algorithm to solve the knowledge explosion problem caused by long-term operation;

[0397] The aforementioned technological directions enable intelligent decision-making systems to evolve from single-point breakthroughs to a systematic evolution.

[0398] In the implementation process of this invention embodiment, to verify the algorithm's advancement, the technical solution provided by this invention embodiment is compared with existing UAV cooperative algorithms, including multi-agent cooperative algorithms EI-MARL, DMTD, and the artificial potential field method APF. The comparison metrics include: transmitted data volume D̅, coverage fairness F, average energy consumption E̅, and energy utilization efficiency η. The corresponding comparison results are as follows: Figure 7 As shown, the results indicate that the multi-agent collaborative algorithm LLM-MAPPO based on a large language model provided in this embodiment of the invention has an 80% improvement in data transmission volume and coverage fairness compared to the EI-MARL and APF algorithms. Thanks to its effective global optimization capability, LLM-MAPPO has a 194% improvement in energy utilization efficiency compared to other algorithms.

[0399] Of particular note is that, thanks to the global optimization capability brought about by the reflective mechanism, LLM-MAPPO achieved an energy utilization efficiency η of 1.00, a 194% improvement compared to other algorithms. This fully demonstrates the superior effectiveness of reflective learning in resource optimization. (Refer to...) Figure 7 As shown, on the horizontal axis of the figure Four normalized performance metrics are used to evaluate the effectiveness of algorithms. The vertical axis represents the normalized performance metric, with values ​​ranging from 0 to 1. A higher value indicates better performance of the corresponding algorithm under that metric. Figure 7 As can be clearly seen, algorithms without a reflection mechanism (such as EI-MARL at 0.52 and APF at 0.00) exhibit significant fluctuations and deficiencies in performance metrics. In contrast, the LLM-MAPPO scheme provided in this embodiment of the invention maintains stable and excellent performance across all key metrics through continuous reflection and optimization, thus verifying the important value of the reflection mechanism in improving the collaborative efficiency of multi-agent systems.

[0400] In summary, this invention, through the construction of a closed-loop optimization mechanism of "evaluation-reflection-thinking chain update," enables large-scale models to continuously revise and optimize their original decision-making processes based on environmental conditions, task execution results, and historical experience feedback in complex dynamic task scenarios. The evaluator model introduced in this invention provides multi-dimensional evaluation of decision results, generating semantic feedback with causal analysis and improvement objectives. Furthermore, the reflection model identifies logical biases, goal deviations, and strategic deficiencies in the original decision-making thinking chain, forming reflective content that can be used for subsequent decision optimization. Finally, the thinking chain is dynamically updated based on evaluation information and reflection results, enabling large-scale models to continuously improve their decision-making capabilities without frequent large-scale retraining.

[0401] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. A large-scale model decision optimization method based on environmental feedback, characterized in that, include: An initial decision-making scheme and a corresponding thought chain are generated based on the task objective, environmental state, and execution unit state. The thought chain is used to record the reasoning process of the initial decision-making scheme. After the task is executed according to the initial decision-making scheme, the task execution results are evaluated from multiple dimensions through the evaluator model, generating semantic feedback that includes result analysis, problem identification, and improvement suggestions to obtain evaluation results; and, through the reflection model, the logical deviations, goal deviations, and strategy deficiencies in the thought chain are analyzed to obtain reflection content that includes problem identification, decision-making logic deviations, and feasible improvement directions. The thought process chain is dynamically updated based on the evaluation results and the reflection content to optimize the subsequent decision-making process.

2. The method according to claim 1, characterized in that, The process of obtaining the evaluation results includes: The evaluator model acquires decision-making instructions from decision-makers, task result data packages, and execution unit role transition logs, and determines corresponding quantitative evaluation indicators. The task result data package contains information including: target achievement rate, resource consumption status, and environmental status changes. The quantitative evaluation indicators include: Target achievement rate, used to measure task completion rate; Timeliness is used to distinguish between three timeliness statuses of tasks: completed ahead of schedule, completed on time, and completed late. Efficiency-resource ratio is used to measure the efficiency of task input and output. Intent alignment is used to measure the degree to which the actual performance of the drone matches the decision-maker's original mission objectives; Based on the aforementioned quantitative evaluation indicators, an evaluation result is generated that includes result analysis, problem identification, and improvement suggestions.

3. The method according to claim 1, characterized in that, The method also includes an online reinforcement learning closed-loop training process for the evaluator model, and the training process includes: After the evaluator obtains the decision-maker's decision instructions, the results of the first task, and the UAV role transformation, the decision-maker's decision instructions are transformed into a structured representation, which is then converted into a vectorized representation corresponding to the decision-maker's decision instructions, through a natural language parsing module; the results of the first task are fused from multiple sources of sensors to form a corresponding spatiotemporal feature matrix of environmental data; and the UAV role transformation record is modeled as a state transition diagram. The global representation vector is obtained by fusing domain knowledge through a cross-attention mechanism using a Transformer-based self-attention encoder: ； Among them, h t Let t be the global representation vector of the multi-source fusion at time t, i.e., the fusion representation; For cross-attention fusion operators; This is a single-branch Transformer encoder function; d t The vectorized representation of the decision-maker's instructions; s t G is the spatiotemporal feature matrix of environmental data; t The state transition diagram obtained by modeling the drone role switching record; The corresponding decoder is based on the fusion representation h t Autoregressive generation of evaluation text y t Its probability distribution is: ； in, For fusion-based characterization h t Autoregressive generation of evaluation text y t The probability distribution, w k To evaluate the k-th word in the text sequence, This refers to the sequence of all preceding words generated before the k-th word. For a given and h t At that time, w k The conditional probability of occurrence; The generated evaluation text is written into short-term memory; a new decision is generated based on the evaluation text; after the new decision is executed, a second task result is generated, which is used as the first task result for the next processing and is also input into the reward function processing part to calculate the corresponding reward value. The reward function outputs a reward signal, which is then fed back to the evaluator model through the Proximal Policy Optimization (PPO) algorithm. The PPO algorithm stabilizes and constrains the network parameter update magnitude, and synchronously updates all network weights of the evaluator's internal Transformer domain knowledge encoder and evaluation generator, enabling continuous iterative training of the evaluator model.

4. The method according to claim 3, characterized in that, Prior to performing the online reinforcement learning closed-loop training, a pre-training phase for the evaluator model is also included, which includes: Training of the base model is completed on a general corpus. The Transformer base model is trained using a general large-scale corpus to learn the general language and basic semantic expression capabilities. By fine-tuning the task log data using a domain-adaptive method, an initial evaluator model with basic task understanding capabilities is obtained; during the fine-tuning process, the generated evaluation text y is... t The information is sent to the decision-maker's prompting section, forming a new decision context; and the changes in task benefits before and after adoption are based on the recorded evaluations. Construct counterfactual comparison samples To calculate the counterfactual advantage: ； in, For the counterfactual advantage in round t, The difference between the actual benefits before and after the evaluation is adopted. To incorporate the immediate mission rewards of the current mission round prior to this round of evaluation, To generate the task rewards for the next round after the new decision is implemented, based on the evaluation results from this round, To calculate the expected value of the counterfactual virtual gain difference; This counterfactual advantage is used to guide the PPO optimization of the evaluation model, and its objective function is: ； in, Optimize the loss function for the PPO strategy. To calculate the mean expected value of the loss over all time steps, This represents the probability ratio between the old and new strategies. To obtain the minimum of the two, To be Limited to the range , This is for pruning hyperparameters.

5. The method according to claim 4, characterized in that, The evaluator model includes personalized evaluation generation based on role perception and iterative updates of short-term memory, wherein: The personalized evaluation generation based on role perception refers to the evaluator generating differentiated feedback for drones with different functions by embedding drone role description vectors and configuring a lightweight adapter to dynamically adjust the evaluation generation path. The iterative update of the short-term memory is achieved through a dynamic memory pyramid structure, which includes an immediate memory layer, an execution mode layer, and a global principle layer, wherein: The instant memory layer stores the most recent predetermined decision-evaluation pairs in the form of a sliding window and updates them through an iterative mechanism based on historical decision and evaluation records. The execution mode layer is used to abstract high-frequency evaluation topics into searchable execution modes. It stores high-frequency evaluation modes for UAV scheduling in key-value pairs, using the current task context features as the search key and the corresponding evaluation template as the value. Based on the matching similarity between the current task features and the modes in the database, the execution mode memory is driven to be updated iteratively. The global principle layer forms top-level rules through long-term suggestions in cluster evaluation. It also periodically clusters and summarizes historical evaluation suggestions to form general top-level execution rules for UAV scheduling. During the iterative update process of the short-term memory, the corresponding memory update mechanism is implemented based on the long short-term memory of the COPPER framework. The evaluation representation vector output by the evaluator model will be synchronously written back to the dynamic memory pyramid to complete the full-level iterative update of the short-term memory.

6. The method according to any one of claims 1 to 5, characterized in that, The process of obtaining reflective content involves generating structured reflective text containing reflective content through a sequence-to-sequence Transformer model that integrates a built-in domain-specific knowledge base and a causal inference engine. This generation process includes: The encoder of the Transformer model receives evaluation units obtained after task evaluation deconstruction. and original decision fragments And the key indicator m extracted through multi-agent association analysis i A joint representation is formed through a two-order gating fusion mechanism, the calculation method of which is as follows: ； ； in, Let be the fusion gating coefficient for the t-th encoding step; W is the Sigmoid activation function. g W u W d and W m For a trainable encoding matrix; u t The encoder joint representation after fusing the three inputs at step t; Tanh(·) is the hyperbolic tangent activation function; For Hadamah accumulation; The Transformer model decoder employs a prefix-tuned constraint generation strategy, using operational procedure constraints and equipment performance parameter constraints from the professional knowledge base as constraint prefixes P. c During the injection decoding process, the probability distribution of the generated sequence corresponding to the injected text is defined as follows: ； in, For a given In the case of generating The probability of; The final output is a complete sequence of reflective text; The joint generation probability of the entire reflective text is obtained by multiplying the conditional probabilities over all time steps t of the text. Given historical text, joint coding features, and knowledge base constrained prefixes, the t-th term w t The conditional generation probability; This represents the entire sequence of historical lexical units generated before time t; Based on the probability distribution of reflective text generation and causal tracing results, structured reflective texts are generated that include problem identification, decision-making logic bias, and actionable improvement directions.

7. The method according to claim 6, characterized in that, The implementation methods of the task evaluation deconstruction and multi-agent association analysis include: The implementation process of the task evaluation deconstruction includes: The task evaluation deconstruction adopts a layered attention architecture, including: a role recognition layer and a cross-source query-key attention calculation layer; wherein, The role recognition layer performs bidirectional long short-term memory network operations on the evaluation text, and extracts the latent features of the semantic fragments of the corresponding drone by combining the trainable role embedding matrix of the corresponding drone. The cross-source query-key attention calculation layer uses the single-machine task allocation description in the decision text as the query vector and the word vector of the evaluation text as the key vector. It calculates the fragment importance weight of each word corresponding to the single drone through dot product similarity operation. The implementation methods of multi-agent association analysis include: The multi-agent association analysis module is configured with a dynamic relational graph model, and the dynamic relational graph model constructs a dynamic heterogeneous graph G. t for: ； Among them, nodes Represents a drone instance, edge Represents task collaboration relationships, edge type Differentiate between different interaction modes; The multi-agent association analysis module performs a classification graph convolution operation on the dynamic heterogeneous graph, and the operation formula is as follows: ； Where l is the layer number of the current graph convolutional network; This is the original feature vector of neighboring drone i in the (l+1)th layer; Let be the trainable weight matrix specific to the l-th layer and corresponding to edge type r; Let be the original feature vector of neighboring drone j in the l-th layer; The core neighbor set of a dynamic heterogeneous graph; To perform a two-level summation, first iterate through all neighbors of a single edge type, then iterate through all edge types. It is a non-linear activation function; By grouping and aggregating neighbor features according to interaction type, a drone node representation that integrates multiple types of collaborative dependencies is obtained. The training process of the reflector model includes: The decision-making instructions from the decision-maker, the results of the first round of mission execution, and the information on the role transformation of the UAV are used as initial inputs. Structured features are obtained through mission evaluation deconstruction and multi-agent association analysis. Reasoning is performed based on structured features, and the original reflection text is output and written into long-term memory simultaneously. The decision-maker reads the long-term memory and the original reflection text to generate a new decision. After execution, the new task result is obtained. The original reflection text is processed by removing descriptive fragments related to a certain drone to obtain a variant reflection text with deletion. The variant reflection text is synchronously updated to form the variant long-term memory after iteration. The decision-maker combines the variant reflection text and the variant long-term memory to generate a new decision for the comparison version, executes the new decision for the comparison version, and outputs the task results corresponding to the new decision for the comparison version. The results of the new task and the corresponding task results of the new decision in the comparison version are fed into the reward function module. The corresponding reward value is calculated based on the difference in the performance of the two sets of tasks, which serves as a supervision signal for PPO optimization. The PPO optimization algorithm is then fed back to the reflector model, and the internal model parameters of the reflector are iteratively adjusted using the reinforcement learning loss gradient to complete one round of training.

8. The method according to claim 6, characterized in that, After generating the reflection text, the process further includes calculating the semantic similarity between the newly generated reflection text and the existing content in long-term memory, and updating the corresponding bottom-level reflection instance library, mid-level micro-pattern library, and top-level macro-principle library in long-term memory according to the similarity matching results. The bottom-level reflection instance library stores the reflection text and its corresponding decision-making context features; the mid-level micro-pattern library extracts high-frequency reflection themes and solutions; and the top-level macro-principle library stores standardized system operation principles. The update process includes: Configure a differentiated memory update mechanism based on semantic similarity judgment, perform semantic encoding on the newly generated reflection text, and calculate the semantic similarity between the features of the new reflection text and the features of historical memory entries; Calculate its semantic similarity s with existing memories. i : ； in The transpose row vector of the newly reflected text encoding vector; This refers to the i-th stored historical memory entry in the long-term memory bank. This is the semantic feature vector obtained after semantically encoding the i-th historical long-term memory entry; This is a text semantic encoding function, responsible for converting text content into low-dimensional real-valued semantic vectors; When the semantic similarity is below a set threshold, a new memory entry is created; when the semantic similarity is not below the set threshold, the existing memory entries are iteratively updated using a weighted fusion method. The memory update formula is as follows: ； in, This indicates assignment overwriting, meaning the updated result replaces the original semantic feature vector; Weighting coefficients are retained for historical memory; this formula represents the weighting after the update. Replace the original semantic feature vector ; Furthermore, during the decision-making stage, a multi-scale linkage retrieval mechanism can be adopted to simultaneously call up three types of memory content: the underlying instance-level cases, the mid-level micro-execution models, and the top-level macro principles, to collaboratively participate in the reflection and auxiliary reasoning of the current decision.

9. The method according to claim 6, characterized in that, The process of dynamically updating the thought chain includes: The dynamic reasoning formula satisfied by the updating process of the thought chain is: ； in, This represents the state of the thought chain at time t+1; This represents the state of the thought chain at time t; Top-level target prompts provided to system schedulers; Encoding of real-time situation; It is a long short-term memory system; Among them, real-time situation coding The calculation formula is: ； in, For encoder function, r t Indicates resource status, e t For non-cooperative node characteristics, u t Vector of the drone role; Long Short-Term Memory System The built-in micro-pattern retrieval network uses an attention mechanism to match historically similar cases and completes memory retrieval by calculating matching weights. The formula for calculating the matching weights is as follows: ； in, q represents the matching weight between the current situation and the i-th historical memory case. T Let k be the current situation query vector. i The key vector of the memory item is the encoded feature of the i-th historical case in the memory bank. j is a general index for traversing all cases in the memory, covering all stored memory entries; The candidate solutions and corresponding reflection texts output from the above-mentioned thought chain update process are input into the counterfactual reasoning module, and the optimal strategy is obtained by solving the three-layer counterfactual reasoning computational architecture; wherein, the three-layer computational architecture consists of a lightweight scenario simulator, a strategy mutation engine and a decision optimization module from bottom to top; The policy mutation engine performs semantic parsing on the input reflection text, converts the reflection text into parameter adjustment terms and generates a candidate policy set, and parameterizes each candidate policy into a corresponding policy distribution. The lightweight scenario simulator performs virtual scenario simulations on all candidate strategies in parallel, and calculates the expected cumulative return for each candidate strategy using the expected cumulative return formula, which is as follows: ； in, As a discount factor, For instant rewards, Candidate strategies A parameterized representation, Represents the k-th candidate strategy The expected cumulative reward. Here, t is the mathematical expectation operator, t is the deduction time step, and T is the maximum deduction duration; The decision optimization module employs a breadth-first search strategy to screen candidate solutions and determines the optimal strategy based on Pareto front analysis. The formula for solving the optimal strategy is as follows: ； in, Indicates the degree of strategy risk; These are the weighting coefficients corresponding to returns and risks; To maximize the solution operator; This is the optimal strategy obtained through final selection; The optimal strategy is constructed as a knowledge pair of hypothesis and verification results and stored in a long-term memory knowledge base. The formula for calculation is: ； in, Encoding for policy features, for The policy feature vector obtained by the encoder; The optimal strategy obtained through Pareto selection ; To increase the quantity of the product, Environmental characteristics.

10. The method according to claim 9, characterized in that, The thought chain update process is also configured with a real-time guarantee mechanism, including: The thought process chain is broken down into three parallel sub-modules: goal decomposition, resource allocation, and solution generation. A timeout judgment logic is set up. When the time consumed in the inference layer is greater than or equal to the warning threshold for the inference stage, a simplified inference mode is activated. The simplified inference mode calculation formula is: ； in, To enable lightweight updates to the thought chain after simplifying reasoning output, This represents the state of the historical thought chain before the update. Let g represent the k most relevant memory items retrieved from long-term memory, and g is a lightweight inference function for memory enhancement. Furthermore, the total time consumption of a single thought chain update must meet the following constraint: the sum of the input encoding time, the core reasoning layer time, and the output decoding time must not exceed the global total time limit threshold.

Images