An unmanned system instruction interaction and task planning method based on memory enhancement

Abstract

The application discloses a kind of based on memory enhancement unmanned system instruction interaction and task planning method, it is related to unmanned system autonomous navigation and task planning technical field, wherein the method includes: obtaining task instruction and splitting into ordered instruction unit, each unit carries environmental anchor point information;In task, environmental perception features are continuously collected and written into historical perception memory bank;According to current instruction unit, determine the search range, use environmental anchor point matching to filter task-related memory;With current planning decision features as query, filter memory as key value, get memory-enhanced decision features by means of attention mechanism fusion, send into action prediction module to generate path point, complete subtask path planning.The present application constructs memory evaluation benchmark, based on instruction anchor point search and adaptive memory fusion mechanism, evaluates the memory capacity of model, solves the problem of long-range navigation memory failure, large search noise.

Description

Technical Field

[0001] This invention relates to the field of autonomous navigation and mission planning technology for unmanned systems, and in particular to a memory-enhanced method for command interaction and mission planning in unmanned systems. Background Technology

[0002] Visual language navigation tasks require agents to understand natural language commands based on continuous visual observation and to complete goal-oriented path planning and action decisions. As embodied intelligence expands from short-range indoor navigation to long-term autonomous execution in open environments, unmanned aerial vehicles (UAVs), with their three-dimensional spatial mobility and wide-area environmental coverage capabilities, have become important application platforms for visual language navigation. Compared to ground platforms, UAV visual language navigation faces more significant challenges, including expanded motion space, dynamic changes in perspective, increased environmental scale, and visual confusion with landmarks. Its mission execution typically spans longer time periods and involves more sub-tasks, placing higher demands on the continuous maintenance of historical information and task-related retrieval.

[0003] Existing visual-language navigation methods primarily rely on outcome metrics such as overall navigation success rate, endpoint position error, or trajectory quality for model evaluation. Public datasets and evaluation frameworks such as OpenUAV and OpenVLN provide a basic platform and long-range mission scenarios for UAV visual-language navigation, but their evaluation systems still focus on whether the final navigation objective is achieved. While these metrics can reflect the model's overall performance on the complete trajectory, they struggle to distinguish whether the failure stems from visual recognition errors, path planning deviations, or memory retention and retrieval failures. In complex tasks involving multi-stage sub-targets, historical location revisits, delayed action triggering, or cumulative state maintenance, models may exhibit various memory-related failure patterns, such as recognizing key landmarks but failing to maintain them, retaining information but failing to correctly recall it at subsequent decision points, or confusing similar environmental cues at different stages. These patterns may present the same numerical results in terms of final success rate or endpoint error.

[0004] Research indicates that memory capabilities in long-range embodied missions involve multiple separable cognitive processes, including information encoding, retention, updating, retrieval, and integration. The reliance on historical information varies significantly across different mission phases. Some UAV visual-language navigation studies have attempted to organize historical observations through structured memory, semantic topology maps, or global memory banks to support long-range spatial reasoning. However, current research on memory mechanisms primarily focuses on improving overall navigation performance, lacking an analytical framework that can isolate memory capabilities from overall navigation capabilities and evaluate them independently. Furthermore, existing methods typically employ global historical matching or indiscriminate historical feature fusion during the memory retrieval phase. This makes it difficult to establish explicit correspondences with the environmental anchors of the current sub-task and easily introduces irrelevant historical observations into action decisions, leading to a continuous accumulation of prediction biases in subsequent flights. In addition, complex navigation commands often contain multiple sub-targets and environmental constraints with temporal relationships. Existing methods often treat the complete command as the overall semantic input, failing to fully utilize the inherent phased structure of the command to guide task-related retrieval of historical information, thus limiting the accuracy and interpretability of memory retrieval.

[0005] Therefore, there is an urgent need for a method that can separate memory capabilities from overall navigation capabilities and evaluate and enhance them independently. This method can be used to systematically diagnose memory capabilities such as sequence maintenance, count accumulation, historical revisiting, and delayed triggering during long-term mission execution by establishing a phased structure of mission instructions, explicit storage of historical perception information, and a correspondence between them and mission-related retrieval. Furthermore, it can achieve controllable fusion of mission-related historical information during the action decision-making stage, thereby improving the reliability and interpretability of UAV visual language navigation systems in complex long-range missions. Summary of the Invention

[0006] The main objective of this invention is to provide a method for instruction interaction and task planning in unmanned systems based on memory enhancement.

[0007] To achieve the above objectives, a first aspect of the present invention proposes a method for instruction interaction and task planning in unmanned systems based on memory enhancement, comprising: Obtain task instructions, break down the task instructions, and generate multiple instruction units arranged in the order of task execution. Each instruction unit contains environmental anchor point information associated with the corresponding sub-task stage. At the beginning of each task execution cycle, the historical perception memory bank is cleared, and during the task execution, the current environmental perception data of the unmanned system is collected, the feature information of the environmental perception data is extracted, and the feature information is written into the historical perception memory bank to update the historical perception memory bank. The current instruction unit is advanced sequentially according to the order of the instruction units, and the candidate retrieval range is determined in the historical sensory memory bank according to the progress status of the current instruction unit. The environmental anchor information in the current instruction unit is matched with each historical perception feature in the candidate retrieval range, and the historical perception features that meet the preset matching conditions are selected as task-related memory information. The current planning and decision features are used as queries, and task-related memory information is used as keys and values. These are then fused through an attention mechanism to obtain memory-enhanced decision features. These memory-enhanced decision features are then input into the action prediction module to generate subsequent task planning results and complete the path point generation for the current subtask stage.

[0008] Optionally, the task instructions are obtained, and the task instructions are split into multiple instruction units arranged in the order of task execution, including: Obtain the complete text description of the task instruction, call the large language model interface, and perform structured segmentation of the task instruction according to the predefined instruction unit template, so that each complete instruction is divided into multiple instruction units that correspond one-to-one with the sub-task stage; Each instruction unit is labeled with a corresponding subtask stage number, so that each instruction unit forms an ordered sequence according to the task execution stage, and the successor relationship between adjacent instruction units is established. The sequence of instruction units after being split is saved in a structured file format as a basis for subsequent memory retrieval.

[0009] Optionally, each instruction unit includes environmental anchor information associated with the corresponding subtask phase, wherein: The environmental anchor information is a semantic description of the environmental landmarks or objects associated with the memory points in the corresponding subtask stage of the instruction unit. There is an explicit binding relationship between the semantic description and the landmarks or objects pre-arranged in the simulation scene. The environmental anchor information is used to map local semantic units in natural language instructions into computable vector representations. The vector representations are obtained through word segmentation and encoding, and have the same feature dimension as the memory feature vectors stored in the historical perception memory bank. The environmental anchor information serves as a query condition for retrieving the historical perception memory database, and is used to filter historical perception features related to the current subtask stage.

[0010] Optionally, at the beginning of each task execution cycle, the historical perception memory bank is cleared, including: Clear all historical perception feature vectors stored in the historical perception memory bank to ensure that historical information within the current task execution cycle is not reused across cycles. Clear the index of search results corresponding to each recorded instruction unit to avoid the search records of the previous cycle from interfering with the current cycle; Reset the progress state of the current instruction unit to the initial state, so that the instruction unit sequence of the current cycle advances sequentially from the first instruction unit; Load the instruction unit sequence corresponding to the current task execution cycle as the basis for memory retrieval within the current cycle.

[0011] Optionally, the following steps are taken: collecting environmental perception data of the unmanned system at the current moment, extracting feature information from the environmental perception data, writing the feature information into the historical perception memory, and updating the historical perception memory, including: During the mission execution, the environmental perception data of the unmanned system is collected step by step according to the time step. The environmental perception data is input into the multimodal encoder to extract the frame-level feature representation of the current moment, wherein the frame-level feature includes visual perception information. Frame-level feature representations are pooled and compressed along the sequence dimension to obtain fixed-dimensional memory feature vectors, thereby reducing storage overhead and unifying feature dimensions. The memory feature vectors are concatenated along the time dimension into the historical perception memory bank, so that the historical perception memory bank accumulates and stores the memory feature vectors of each historical moment in the current task execution cycle in chronological order. When the number of memory feature vectors in the historical perception memory bank exceeds a preset capacity threshold, the latest memory feature vector with the latest time sequence is retained, and the earliest memory feature vector with the earliest time sequence is deleted.

[0012] Optionally, the candidate retrieval range in the historical perceptual memory is determined based on the progress status of the current instruction unit, including: Obtain the progress status of the current instruction unit, wherein the progress status is used to indicate the position number of the instruction unit that has been retrieved and completed in the instruction unit sequence; Using progress status as a time constraint, the historical perception features located after the index of the retrieved frame in the historical perception memory are determined as candidate retrieval ranges, so that the retrieval of the current instruction unit is only carried out in historical frames that have not yet been retrieved. After the retrieval corresponding to the current instruction unit is completed and the task-related memory information is successfully obtained, the progress status is updated so that the retrieval of the next instruction unit is executed in the updated progress status.

[0013] Optionally, the environmental anchor information in the current instruction unit is matched with each historical perception feature in the candidate retrieval range, and the historical perception features that meet the preset matching conditions are selected as task-related memory information, including: The text of the environmental anchor information in the current instruction unit is segmented and encoded, and then mapped into a query tensor. The query tensor has the same feature dimension as the memory feature vector in the historical perception memory bank. The cosine similarity between the query tensor and each historical perception feature within the candidate retrieval range is calculated sequentially, and the cosine similarity is mapped from the value range to a preset interval to obtain the matching degree between each historical perception feature and the environmental anchor information. The historical perceptual feature with the highest matching degree within the candidate retrieval range is selected as candidate memory information, and the matching degree is compared with a preset threshold. When the matching degree exceeds the preset threshold, the corresponding historical perception feature is accepted as task-related memory information, and the frame index of the historical perception feature in the historical perception memory bank is recorded. When the matching degree does not exceed the preset threshold, the current search result is abandoned and the current progress state is maintained.

[0014] Optionally, the planning and decision features at the current moment are used as the query, and task-related memory information is used as the key and value. These are then fused using an attention mechanism to obtain memory-enhanced decision features. These memory-enhanced decision features are then input into the action prediction module to generate subsequent task planning results, including: The planning and decision features at the current moment are linearly mapped to a query vector, and the task-related memory information is linearly mapped to a key vector and a value vector. By using a multi-head cross-attention mechanism, the attention weight between the query vector and the key vector is calculated, and the attention weight is applied to the value vector to obtain a memory-enhanced representation after weighted aggregation of task-related memory information. The fusion weights are calculated through a gating mechanism. The memory-enhanced representation is then weighted and summed with the planning and decision features at the current moment to obtain the memory-enhanced decision features. The fusion weights are controlled by learnable gating parameters to adaptively adjust the proportion of historical information and current information in the memory-enhanced decision features. The memory-enhanced decision features are input into the action prediction module to generate a continuous path point sequence for the current subtask stage, thus completing the path planning for the current subtask stage.

[0015] The embodiments of the present invention have the following beneficial effects: (1) By constructing a memory evaluation benchmark that includes five types of tasks: long sequence, counting, circling, location-environment-action and prospective triggering, memory ability is separated from overall navigation ability, and independent evaluation and diagnosis of memory abilities such as sequence retention, cumulative counting, historical revisiting and delayed triggering are realized, overcoming the defect that traditional outcome indicators are difficult to distinguish the sources of memory-related failures.

[0016] (2) By using the environment field obtained from instruction splitting as the retrieval anchor point, the historical memory retrieval and the environmental target of the current subtask stage are explicitly matched, avoiding the noise interference introduced by global indiscriminate historical matching, and improving the accuracy and interpretability of task-related historical information retrieval.

[0017] (3) The designed memory evaluation index describes the model performance from four dimensions: partial completion level, cumulative state error, strict task success and historical revisit ability. It can attribute model failure to specific memory failure types such as sequential state loss, counting state error, historical position revisit failure or prospective trigger failure, providing a quantifiable analytical basis for the improvement of memory enhancement methods.

[0018] (4) Through the collaborative mechanism of memory bank writing, task-related retrieval, cross-attention fusion and gating adjustment, the adaptive fusion of historical information and current decision features is realized, which significantly improves the execution reliability and action decision accuracy in long-range multi-stage mission scenarios while maintaining basic navigation capabilities. Attached Figure Description

[0019] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A flowchart illustrating a memory-enhanced unmanned system instruction interaction and task planning method provided in an embodiment of the present invention; Figure 2 A complete flowchart of the construction process of the memory benchmark dataset provided in the embodiments of the present invention; Figure 3 This is a schematic diagram of the scene mesh topology provided in an embodiment of the present invention; Figure 4 The trajectory and memory point generation pipeline diagram provided in the embodiments of the present invention; Figure 5 A flowchart illustrating the breakdown of natural language instructions and instruction units provided in this embodiment of the invention; Figure 6 This is a diagram of the overall network architecture of MemoryUAV provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the internal structure of the memory module provided in an embodiment of the present invention. Detailed Implementation

[0020] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0021] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0022] The following description, with reference to the accompanying drawings, describes a method and apparatus for instruction interaction and task planning in an unmanned system based on memory enhancement, according to an embodiment of the present invention.

[0023] Example 1 This invention provides a memory-enhanced method for instruction interaction and task planning in unmanned systems. Figure 1 This is a flowchart illustrating a memory-enhanced unmanned system command interaction and task planning method provided in an embodiment of the present invention. Figure 1 As shown, the method includes the following steps: Step S1: Obtain task instructions, split the task instructions into multiple instruction units arranged in the order of task execution, and each instruction unit contains environmental anchor point information associated with the corresponding sub-task stage.

[0024] It should be understood that existing UAV visual language navigation research mainly focuses on outcome metrics such as mission success rate, endpoint error, and trajectory quality. OpenUAV has constructed a data platform and evaluation framework that more closely resembles real-world UAV missions, providing an important foundation for UAV VLN. OpenVLN further extends the research scenario to open world and long-range navigation, emphasizing the coupling problem between semantic understanding and path planning in complex scenarios. However, such evaluations usually only reflect whether the model has achieved the final navigation goal, and it is difficult to further distinguish the sources of failure: the model may not have recognized key landmarks, or it may have recognized them but not remembered them, or it may have remembered them but not called them correctly in subsequent stages. Therefore, relying solely on outcome metrics is insufficient to analyze the boundary of the explicit memory mechanism in UAV VLN. The mission instructions in the embodiments of this application are generated based on a specially constructed UAV VLN memory capability evaluation benchmark. This benchmark aims to separate the memory capability in navigation from the overall navigation capability, forming a definable, controllable, and comparable diagnostic evaluation set. Its overall construction process is as follows: Figure 2As shown. The benchmark construction first builds a navigable map structure based on the simulation scenario, and samples the starting point, path, and task parameters on the map; then, it generates trajectories and memory points according to different memory tasks, and places landmarks or objects at key locations; then, it calls AirSim to collect multi-camera observations according to the trajectory, and maintains the data body format compatible with the existing navigation framework, so that the generated data can be directly connected to the training and evaluation process; finally, it generates corresponding natural language commands and structured decomposition results for model training and inference. Unlike directly expanding ordinary navigation samples, the benchmark in this application designs tasks around memory points, so that each trajectory has clear historical observations, delayed targets, or future triggering conditions, thereby enabling the evaluation of the model's ability to maintain, update, retrieve, and utilize historical information within a single task cycle, paying particular attention to typical problems in long-range UAV missions, including sequence loss, counting bias, revisit errors, environmental anchor confusion, and future cue triggering failures.

[0025] It is important to note that scene graph construction is fundamental to baseline generation. Its role is to constrain the UAV's flyable area, turning positions, and path connectivity, reducing collisions, boundary violations, and semantic uninterpretability issues in trajectory generation. Existing UAV VLN work shows that aerial navigation tasks are significantly more affected by scene scale, viewpoint changes, and path continuity than traditional ground-based VLN. Therefore, the data construction stage requires stronger constraints on spatial topology. The scene graph is denoted as G=(V,E), where V represents the set of passable nodes, and E represents the set of flyable paths between nodes. Each node corresponds to a key location in the scene, such as a road intersection, turning point, observable point near a building, or a suitable location for placing a landmark. Each edge connects two adjacent nodes and stores a discrete trajectory point sequence from the start to the end point. For areas with relatively straight roads and few obstacles, edges are obtained by linear interpolation between two nodes; for curved roads or areas requiring detours, trajectory points along curved edges are manually collected to avoid simple straight-line interpolation traversing impassable areas. Ultimately, each scene corresponds to a unified `scene_graph.json` file, which stores node, edge, and trajectory point information for each edge, shared by all subsequent tasks. For example... Figure 3 The image shown is a scene diagram of the BrushifyUrban scene. Hollow circles represent key nodes, solid circles represent trajectory sampling points, and lines represent passable paths. The overall structure presents a grid-like passable topology that can cover the main flyable areas within the scene.

[0026] To further clarify, after constructing the scene graph, trajectory and memory point generation are performed. When generating trajectories, starting points are first sampled from the scene graph. To avoid samples concentrating on a few paths or local areas, the generation program records the number of times different edges are selected as starting points and constrains subsequent starting point selections based on the sampling frequency, ensuring the generated trajectories have more balanced spatial coverage within the scene. Subsequently, paths are generated on the graph structure according to the task type; different types of memory tasks correspond to different path generation logics. Memory is not a single storage unit but consists of multiple collaborative functional processes. Neuroscience research shows that the human memory system includes different mechanisms for information retention, organization, binding, and retrieval, which collectively support goal-oriented behavior. In working memory theory, the episodic buffer is used to explain the integration and maintenance of information from different sources in a short period. It not only preserves fragmented information but also supports cross-modal and cross-temporal relationship binding. For UAVVLN, not all memory mechanisms are equally important. The core problem the model needs to solve is not long-term preservation of all historical observations, but rather maintaining information relevant to the current task within a single instruction execution cycle and correctly recalling it at subsequent decision points. Based on the execution characteristics of UAV tasks, this application breaks down memory requirements into six basic capabilities: online task status update, sequence memory, counting memory, context binding capability, look-ahead triggering capability, and anti-interference capability. Correspondingly, five types of memory tasks are designed: long sequence tasks, counting tasks, circling tasks, position-environment-action tasks, and look-ahead triggering tasks. The correspondence between various memory capabilities and task types is shown in Table 1.

[0027] Table 1

[0028] It should be further noted that the five types of tasks correspond to different memory evaluation dimensions. Long sequence tasks primarily evaluate sequential memory and online task state updates, requiring the model to arrive at multiple memory points sequentially according to the instructions and correctly recall intermediate target information in subsequent stages. For example, the instructions for a long sequence task can be expressed as: "Go straight, pass a red car, and reach an intersection. Turn left, pass a man, and reach an intersection with a stop sign. Turn right, fly along the road, pass a trash can, turn right at the next intersection, and reach a fire truck." In this example, the model not only needs to recognize environmental cues such as "red car," "person," "stop sign," "trash can," and "fire truck," but also needs to maintain the sequential relationship between these cues in the trajectory. If the model only relies on the current visual input, it cannot know where it has already arrived and where it needs to go next, and may repeat actions already completed. Therefore, the focus of long sequence tasks is not single-frame recognition, but rather the preservation of sequence and state updates in multi-stage tasks.

[0029] The counting task primarily evaluates counting memory, requiring the model to count the number of times a certain type of target appears during flight and perform an action when a specified number is reached. For example, the instruction for the counting task could be: "Fly forward along the road and stop at the fourth stop sign." This task differs from long sequence tasks in that the model does not need to remember the complete identity and sequence details of each stop sign, but rather needs to continuously maintain a cumulative count state. Even if the model can identify a stop sign in the current frame, it may not know which stop sign it is. Therefore, the counting task distinguishes target recognition ability from cross-time accumulation ability. Typical errors include undercounting, overcounting, double-counting, and mistakenly including interfering targets in the target count.

[0030] The circling task evaluates the cyclical state memory during continuous motion, requiring the model to continuously determine whether it is still moving around the correct target, how many laps it has completed, and whether it has reached the specified number of laps. For example, the instruction for the circling task can be expressed as: "Fly clockwise around the building on the right three times." In this task, the model needs to continuously determine whether it is still moving around the correct target, how many laps it has completed, and whether it has reached the specified number of laps. Unlike counting tasks where counting is triggered by discrete targets such as parking signs, the state update in the circling task comes from the continuous trajectory process. If the model lacks retention of historical trajectories, it is prone to errors such as insufficient lap count, excessive lap count, or misjudging a partial turn as a complete circle. This task is more similar to real-world scenarios such as UAV inspection, hovering observation, and area search.

[0031] The location-environment-action task is used to evaluate context binding ability, that is, whether the model can bind location, environmental cues, and actions into a unified memory unit. This task can be broken down into three typical forms: location, environment, and action. For example, the instruction examples for the three forms are as follows: Location task example: "Go straight, pass a red car, and reach the intersection. Turn left, pass a person, and reach the intersection with the stop sign. Turn right, drive along the road, and reach the trash can. Then return along the same road to the coordinates (-83.0, 43.7)." Environment task example: "Go straight, pass a red car, and reach the intersection. Turn left, pass a person, and reach the intersection with the stop sign. Turn right, drive along the road, and reach the trash can. Then return along the same road to the location you just..." The stop sign you just passed. Example of an action task: "Go straight, pass a red car, and reach an intersection. Turn left, pass a person, and reach an intersection with a stop sign. Turn right, drive along the road, and reach a trash can. Then return along the same route to the location where you last turned right." These three forms point to historical memory points from three perspectives: coordinate location, environmental cues, and historical actions. This task requires the model not only to know "what it currently sees," but also whether the cue belongs to a previously passed location, and the relationship between that location and previously performed actions.

[0032] The look-ahead triggering task primarily evaluates look-ahead triggering capability, requiring the model to maintain a delayed intention and trigger an action when a specified cue appears in the future. For example, the instruction for a look-ahead triggering task can be expressed as: "Go straight, pass a red car, and reach an intersection. Turn left, pass a man, and reach an intersection with a stop sign. Turn right, drive along the road, and reach a trash can. When you approach a puppy, stop immediately and end this task." This task differs from long-sequence tasks in that the action trigger is not determined by a fixed order of the next target, but by future environmental events. The model needs to maintain the condition "when approaching a puppy" throughout continuous flight and stop immediately when the cue appears. If the model stops prematurely, stops late, stops at similar interfering targets, or fails to stop when the puppy appears, it indicates a deficiency in look-ahead triggering capability.

[0033] It is worth noting that memory points are the core annotation units of the memory benchmark. Each memory point stores information such as its category, index, coordinates, and orientation, and corresponds to a local semantic unit in the task instruction. For long sequence tasks, memory points represent sub-targets that need to be traversed in sequence; for counting tasks, memory points represent a counting event or the location of a counting target; for circling tasks, memory points represent key state points during the circling process; for position-environment-action tasks, memory points represent historical events that need to be bound to position, environment, and action; for look-ahead triggering tasks, memory points represent the location of future triggering cues. To maintain compatibility with the data format of existing navigation frameworks, this application does not change the original image and trajectory main format, but instead saves an additional memory point folder in the sample directory to record the coordinate information of each memory point and the corresponding image observation, facilitating subsequent visualization, index calculation, and memory retrieval analysis.

[0034] It should be further noted that the trajectory file uses a step-by-step pose representation, recording the UAV's position and attitude at each time step, including x, y, z, pitch, roll, and yaw. The attitude is calculated from the directions of adjacent trajectory points to ensure that the UAV's orientation is basically consistent with its direction of motion. For trajectory points near the memory point, this application further employs an orientation locking and secondary correction strategy to ensure that key landmarks or objects appear in the effective field of view as much as possible. This processing can reduce invalid samples where "annotated points exist but are not visible in the image," improving the learnability and evaluation stability of the dataset. If a trajectory cannot be generated, the memory point is not visible, key landmarks are missing, or subsequent instructions cannot be generated, the sample is deleted or regenerated to ensure that each sample in the benchmark has a clear memory supervision signal.

[0035] It is worth mentioning that after generating the trajectory and memory points, landmarks or objects are further placed in the simulation scene, and AirSim is used to collect multi-camera observations according to the trajectory. The overall acquisition process is as follows: Figure 5 As shown. The data acquisition process is initiated by MemoryTrajGenerator. First, it reads the scene_graph.json file of the corresponding scene, randomly selects the starting point from the edge with the lowest sampling ratio, selects the generation function corresponding to the task scene, samples memory points at edge-node-edge intervals and generates trajectories, and outputs memory_points.json and trajectory.json files. Then, the above results are used as input to the PrepareScene module, which sets landmarks near the memory points according to probability, adjusts the positions of the memory points to face the landmarks, controls the UAV to fly along the trajectory to take pictures, and finally outputs a data point with an image, completing the single sample acquisition. The role of landmarks is to provide perceptible environmental anchors for memory points, enabling the model to establish a correspondence between instructions, images and memory annotations. Since the memory module of this application mainly uses the environmental field in the instruction unit for memory filtering, the placement of landmarks not only serves visual recognition, but also directly affects the reliability of subsequent instruction splitting, memory retrieval and task evaluation. Compared with ordinary UAV VLN samples, the landmarks and memory points in the data of this application have an explicit binding relationship, which is also the basis for the benchmark to diagnose memory capabilities.

[0036] It should be noted that different tasks employ different landmark or object placement strategies. In long sequence tasks, landmarks are used to distinguish targets at different stages, such as environmental cues like red cars, parking signs, and trash cans. In counting tasks, the system controls the category, quantity, and location of target objects based on object information files, ensuring a controllable number of targets. In circling tasks, landmarks or region centers are used to define the objects being circled. In location-environment-action tasks, landmarks are used to bind historical locations to environmental cues. In look-ahead triggering tasks, landmarks serve as future triggering cues, and similarity interference terms can be set to test the model's robustness. All placement results are consistent with the trajectory and memory points and recorded in the sample metadata to support subsequent instruction generation and metric calculation.

[0037] Furthermore, during the data acquisition phase, the UAV pose is progressively set according to the trajectory file, and multi-camera images are acquired simultaneously. This application maintains the original OpenUAV data format, enabling the generated data to be directly integrated into the OpenUAV training and evaluation process; it also additionally saves the local observation and coordinate information corresponding to the memory points. Acquisition perspectives include front-view, left-view, right-view, rear-view, and downward-view cameras, and RGB and depth images are saved simultaneously when necessary. Multi-view observation can reduce the instability caused by single-view occlusion and also provides a visual basis for subsequent analysis of whether the model retrieves the relevant memory points in the correct location. To improve the stability of batch generation, this application incorporates several engineering constraints during the acquisition process: First, the simulated physical state is frozen during acquisition to prevent the UAV from drifting due to gravity or collisions during the progressive pose setting process; second, the placed objects are cleaned up after each task cycle to prevent residual objects from contaminating subsequent samples; finally, if image acquisition fails, landmarks are not visible, key files are missing, or instructions cannot be aligned with memory points, the sample is deleted or regenerated. Through the above processing, the data constructed in this application not only maintains compatibility with existing navigation frameworks, but also provides additional environmental anchors, key observations, and structured annotations required for memory tasks.

[0038] Optionally, the task instructions are obtained, and then split into multiple instruction units arranged in the order of task execution. This process includes: first, obtaining the complete text description of the task instructions; then, calling the multimodal large model and the large language model to generate and split the instructions. The overall process is as follows: Figure 4 As shown. When generating natural language commands, the trajectory information, memory points, landmarks, or object placement information of each sample are input into the Qwen3-VL multimodal large model to generate UAV navigation commands that conform to human expression habits. Three types of information are carefully ensured to be complete during command generation: First, the command should cover the task objective, such as sequentially passing multiple targets, counting the number of targets, completing an orbit, revisiting historical locations, or waiting for future cues; second, the command should include environmental anchors aligned with memory points, such as "intersection with a red car," "stop sign," or "trash can"; third, the action description in the command should be consistent with the trajectory and task type, such as "turn left," "stop immediately," or "return along the original route." If the generated result lacks key anchors, is inconsistent with the trajectory semantics, or cannot be mapped to an labeled memory point, the sample is deleted or regenerated.

[0039] After obtaining the natural language instructions, the Qwen3 large language model interface is further invoked to structurally segment the task instructions according to the predefined instruction_unit_template.json instruction unit template. This divides each complete instruction into multiple instruction units that correspond one-to-one with subtask stages. Each instruction unit is labeled with its corresponding subtask stage number, forming an ordered sequence according to the task execution stage, and establishing the successor-successor relationship between adjacent instruction units. The segmented instruction unit sequence is saved in a structured file format as a condition for subsequent memory retrieval. Finally, an additional instruction_units.json file is saved for each sample to record the structured segmentation results corresponding to the complete instruction.

[0040] For example, the instruction unit template includes fields such as environment, location, and action. In this embodiment, the environment field is mainly used to carry the landmark anchor information corresponding to the memory point. For example, for the instruction fragment "turn left at an intersection with a red car", the environment field of the corresponding instruction unit in the split result can be represented as "intersection with a red car" or "intersection, red car". This splitting method transforms the originally continuous and freely expressed natural language into computable local task units, providing more stable retrieval conditions for the memory module. During the model training or evaluation phase, the environment field is segmented and embedded as a tensor, serving as a task-related query for the memory module to retrieve historical memory. Compared with directly using the entire instruction for retrieval, retrieval based on instruction units can reduce the interference of irrelevant language fragments on memory selection, making the model more inclined to integrate historical observations related to the current memory point's environment anchor.

[0041] It should be noted that each instruction unit contains environmental anchor information associated with the corresponding subtask stage. The environmental anchor information is a semantic description of the environmental landmarks or objects associated with the memory points in the corresponding subtask stage of the instruction unit. There is an explicit binding relationship between the semantic description and the landmarks or objects pre-arranged in the simulation scene. The environmental anchor information is used to map the local semantic units in the natural language instruction into computable vector representations. The vector representations are obtained through word segmentation and encoding, and have the same feature dimension as the memory feature vectors stored in the historical perception memory bank. The environmental anchor information serves as a query condition for retrieving the historical perception memory bank and is used to filter historical perception features related to the current subtask stage.

[0042] Step S2: At the beginning of each task execution cycle, clear the historical perception memory bank, and during the task execution, collect the current environmental perception data of the unmanned system, extract the feature information of the environmental perception data, write the feature information into the historical perception memory bank, and update the historical perception memory bank.

[0043] It should be understood that existing baseline models primarily rely on current input, contextual latent states, and waypoint lexical features for prediction, with historical information largely stored implicitly within the model context. While this mechanism can achieve basic navigation in typical short-range tasks, it has limitations in explicit memory tasks: long-range historical observations are constrained by context windows and computational overhead; the model lacks an interpretable historical retrieval process; and indiscriminate fusion of historical features may introduce noise irrelevant to the current subtask. Existing research on embodied intelligent memory and long-video understanding shows that long-term tasks cannot simply rely on a complete historical stack but require filtering, retrieval, and fusion. Therefore, this application introduces a lightweight, pluggable memory enhancement module into the baseline model. This module does not reconstruct the original multimodal backbone but adds an explicit historical retrieval path before action prediction, enabling the model to write historical frame features within a cycle and retrieve relevant memories based on the current task stage. The core assumption of this design is that in UAV memory-based navigation, the effectiveness of memory depends not only on whether historical information is preserved but also on whether correct historical evidence can be retrieved based on the current subtask. The memory bank provides explicit storage of historical observations within a cycle, the environment field unit provides task-related retrieval targets, and the cross-attention and gating mechanism injects the retrieved memories into waypoint features, enabling historical information to influence action prediction while suppressing noisy memories.

[0044] It should be noted that the memory enhancement module in this embodiment is pluggable into the original navigation model, and its access position is as follows: Figure 6 As shown. The model as a whole includes a ViT visual encoder, a Proj projection layer, an LLM large language model, a Memory Module, and an Action Head. The ViT visual encoder extracts features from the visual input, which are then mapped by the Proj projection layer and input into the LLM large language model. The features output by the LLM are written into the MemoryBank of the Memory Module as History Memory, and also output Current Memory, i.e., the planning and decision features at the current moment. The Memory Module integrates the historical memory and the current features to output Integrated fused features, which are input into the Action Head to generate the final action planning result. This access method requires minimal modification and does not damage the original model's navigation capabilities. Furthermore, the Memory Module can be controlled by a switch, facilitating comparisons of the performance differences between the memoryless model and the memory-enhanced model under the same data and backbone. Let the waypoint lexical features output by the OpenUAV backbone be... The memory bank is The current environment field unit is The waypoint lexical features output after fusion by the memory module are: Then, the motor characteristics after memory enhancement can be represented as:

[0045] in, This represents the current waypoint lexical features output by the original model, i.e., the planning and decision features at the current moment; This represents the historical perception memory bank at the current moment; The environment field representation represents the current instruction unit; MemoryModule represents the overall processing function of the memory module; This represents the memory-enhanced decision feature output after fusion by the memory module. This feature will be input into the action head for subsequent path point prediction.

[0046] Optionally, at the beginning of each task execution cycle, the historical perception memory is cleared. Specifically, this includes: clearing all historical perception feature vectors stored in the historical perception memory to ensure that historical information within the current task execution cycle is not reused across cycles; clearing the retrieval result indexes corresponding to each recorded instruction unit to avoid interference from retrieval records of the previous cycle to the current cycle; resetting the progress state of the current instruction unit to the initial state so that the instruction unit sequence of the current cycle progresses sequentially from the first instruction unit; and loading the instruction unit sequence corresponding to the current task execution cycle as the basis for memory retrieval within the current cycle.

[0047] It should be further noted that the `reset_memory` operation is called at the beginning of each task cycle to perform the above-mentioned clearing and resetting process. The independence of the task execution cycle is the basis for the establishment of the memory evaluation benchmark. The memory points, trajectories, and instruction units designed in this application are all constructed around a single trajectory. The evaluation index also judges whether the model uses historical evidence that has appeared in the current cycle. If the memory bank is not reset, the model may use other sample information, which may lead to an overestimation of the retrieval hit rate and task success rate.

[0048] To further explain, the process of collecting environmental perception data from the unmanned system at the current moment, extracting feature information from the environmental perception data, writing the feature information into the historical perception memory, and updating the historical perception memory is as follows: During task execution, the environmental perception data of the unmanned system at the current moment is collected step by step according to time steps. The environmental perception data is input into a multimodal encoder to extract the frame-level feature representation of the current moment, where the frame-level features include visual perception information. The frame-level feature representation is pooled and compressed in the sequence dimension to obtain a fixed-dimensional memory feature vector to reduce storage overhead and unify the feature dimension. The memory feature vector is concatenated into the historical perception memory along the time dimension, so that the historical perception memory accumulates and stores the memory feature vectors of each historical moment in the current task execution cycle in chronological order. When the number of memory feature vectors in the historical perception memory exceeds a preset capacity threshold, the latest memory feature vector with the latest time sequence is retained, and the earliest memory feature vector with the earliest time sequence is deleted.

[0049] It is worth mentioning that the memory writing process can be represented by the following formula:

[0050] in, Indicates the first The frame-level features input at each time step have dimensions B×S×D, where B is the batch size, S is the feature sequence length, and D is the feature dimension. This represents the mean pooling operation along the sequence dimension; Let represent the compressed memory feature vector at time t, with dimensions B×1×D; Indicates the first A constantly updated historical perception memory bank; Indicates the first The constantly updated historical perception memory bank is obtained by splicing historical memories with current memories along the time dimension. The memory bank is not reused across task execution cycles to ensure that the model can only use information already observed in the current trajectory.

[0051] Step S3: Advance the current instruction unit sequentially according to the order of the instruction units, and determine the candidate retrieval range in the historical perception memory bank based on the progress status of the current instruction unit.

[0052] It should be understood that memory retrieval for UAV visual language navigation also requires maintaining the stage order. In long sequence tasks, circling tasks, and look-ahead triggering tasks, if global similarity matching is only performed in the entire memory, the model may mismatch subsequent units to earlier similar environments. To address this, this application introduces a mem_progress memory progress and time constraint mechanism, which limits the time range of retrieval through progress status to avoid stage misalignment issues.

[0053] Optionally, the candidate retrieval range is determined in the historical perceptual memory bank based on the progress status of the current instruction unit. Specifically, this includes: obtaining the progress status of the current instruction unit, where the progress status indicates the position number of the instruction unit that has been retrieved and completed in the instruction unit sequence; using the progress status as a time constraint, determining the historical perceptual features located after the index of the retrieved frame in the historical perceptual memory bank as the candidate retrieval range, so that the retrieval of the current instruction unit is only carried out in historical frames that have not yet been retrieved; after the retrieval corresponding to the current instruction unit is completed and the task-related memory information is successfully obtained, updating the progress status so that the retrieval of the next instruction unit is executed in the updated progress status.

[0054] It should be noted that this mechanism maps the linguistic order of instruction units to the temporal order of the memory bank, avoiding stage misalignment and corresponding to the process of "future cues triggering predetermined behaviors" in prospective memory. The module first retrieves the environment field unit corresponding to the current `mem_progress`; if the current unit is successfully retrieved, the frame index is recorded, and the module attempts to retrieve the next unit. The retrieval of the next unit must be executed after the current unit is successfully retrieved, and the retrieved frame must be later than the already recorded frame. The progress status advances gradually as instruction units are successfully retrieved, ensuring that the memory events corresponding to each instruction unit are retrieved sequentially according to the task execution order, preventing errors such as later instruction units prematurely matching earlier similar scenarios.

[0055] Step S4: Match the environmental anchor information in the current instruction unit with each historical perception feature in the candidate retrieval range, and select the historical perception features that meet the preset matching conditions as task-related memory information.

[0056] It should be understood that this step is the core retrieval process of the memory module. Its function is to filter out memory information related to the current subtask from the historical perceptual memory bank, avoiding noise interference caused by indiscriminately fusing all historical frames. The overall structure of the memory module is as follows: Figure 7 As shown, the module receives the sequence of embedded instruction units (Embedded Instruction Units) and historical memories from the Memory Bank. The Retrieve unit matches the corresponding retrieved memory for each instruction unit. The retrieved retrieved memory sets are used as keys and values, and the Current Memory is used as the query. The Cross-Attn cross-attention mechanism is used to obtain the retrieved and fused features. These features are then weighted and fused with the current memory through the Gate unit, ultimately outputting the Integrated features.

[0057] Optionally, the environmental anchor information in the current instruction unit is matched with each historical perception feature in the candidate retrieval range, and the historical perception feature whose matching degree meets the preset condition is selected as task-related memory information. Specifically, the process includes the following steps: the text of the environmental anchor information in the current instruction unit is segmented and encoded, and then mapped into a query tensor. The query tensor and the memory feature vector in the historical perception memory bank have the same feature dimension. The cosine similarity between the query tensor and each historical perception feature in the candidate retrieval range is calculated in turn, and the cosine similarity is mapped from the value range to a preset interval to obtain the matching degree between each historical perception feature and the environmental anchor information. The historical perception feature with the highest matching degree in the candidate retrieval range is selected as candidate memory information, and the matching degree is compared with a preset threshold. When the matching degree exceeds the preset threshold, the corresponding historical perception feature is accepted as task-related memory information, and the frame index of the historical perception feature in the historical perception memory bank is recorded. When the matching degree does not exceed the preset threshold, the current retrieval result is abandoned and the current progress state is maintained.

[0058] Furthermore, the above matching and retrieval process can be represented by the following formula:

[0059] when The search results will be received in a timely manner.

[0060] in, Indicates the first The query tensor corresponding to the environment anchor point information of the current instruction unit; Indicates the first candidate in the search range Each historical perceptual feature vector; cos represents the cosine similarity calculation function; Indicates the first The degree of matching between a historical perception feature and the current environmental anchor point, with a value ranging from 0 to 1; This represents the index corresponding to the historical perceptual feature with the highest matching degree. This indicates the preset search threshold, used to control the acceptance criteria for search results; This represents the highest matching degree within the candidate range. When this value is greater than the threshold τ, the historical features of the corresponding index will be used as task-related memory information; otherwise, the search results will not be accepted.

[0061] It is worth noting that, compared to directly using entire instructions for retrieval, instruction-unit-based retrieval reduces the interference of irrelevant language fragments on memory selection, making the model more inclined to integrate historical observations related to the current memory point's environmental anchor. The model does not indiscriminately integrate all historical frames, but rather selects the most relevant few memories based on environmental field units, thereby reducing the impact of noisy memory injection on action prediction.

[0062] Step S5: Use the current planning and decision features as the query and the task-related memory information as the key and value, and fuse them through an attention mechanism to obtain memory-enhanced decision features. Input the memory-enhanced decision features into the action prediction module to generate subsequent task planning results and complete the path point generation for the current sub-task stage.

[0063] It should be understood that the core of memory enhancement is to effectively integrate relevant historical information retrieved into current decision features, thereby improving the model's performance in memory-related tasks. This application does not directly input all historical frames from the memory bank into the model, but only uses selected relevant memory frames as fusion objects. This is because UAV VLN tasks typically have long cycles, and the memory bank contains both key environmental evidence and a large number of transitional frames; indiscriminate fusion would increase noise. Existing VLA and long video understanding work shows that the effectiveness of long-term memory depends not only on storage capacity but also on whether the retrieval results can be aligned with the current task.

[0064] Optionally, the current planning and decision features are used as the query, and task-related memory information is used as the key and value. These are then fused using an attention mechanism to obtain memory-enhanced decision features. These memory-enhanced decision features are input into the action prediction module to generate subsequent task planning results. Specifically, this includes: linearly mapping the current planning and decision features to a query vector, and linearly mapping the task-related memory information to a key vector and a value vector; calculating the attention weight between the query vector and the key vector using a multi-head cross-attention mechanism, and applying the attention weight to the value vector to obtain a weighted aggregated memory-enhanced representation of the task-related memory information; calculating the fusion weight using a gating mechanism, and using the fusion weight to perform a weighted summation of the memory-enhanced representation and the current planning and decision features to obtain the memory-enhanced decision features. The fusion weight is controlled by a learnable gating parameter to adaptively adjust the proportion of historical and current information in the memory-enhanced decision features; and inputting the memory-enhanced decision features into the action prediction module to generate a continuous path point sequence for the current sub-task stage, completing the path planning for the current sub-task stage.

[0065] Furthermore, the above fusion process can be represented by the following formula:

[0066] in, This represents the planning and decision-making characteristics at the current moment, i.e., waypoint lexical features; This represents the set of relevant historical memories retrieved. This represents the multi-head cross-attention calculation function; This represents the query vector, which is obtained by mapping the current planning and decision features. The key vector is obtained by mapping from the historical memory set; The value vector is obtained by mapping from the historical memory set; This represents the memory enhancement representation obtained through cross-attention; This represents the learnable gating parameters used to control the fusion weights of the memory-enhanced representation; This represents the memory-enhanced decision-making features of the final output.

[0067] It should be further noted that the memory-enhanced decision features after gating fusion are input into the original waypoint action head, which outputs subsequent waypoint predictions. Since the action head, output format, and loss function are consistent with the baseline model, the main difference between this method and the baseline model lies in whether or not a memory retrieval and fusion mechanism based on instruction units is introduced, which facilitates the analysis of the actual role of the memory module through ablation experiments.

[0068] It is worth mentioning that the training strategy in this application embodiment can be selected according to actual needs, including two optional schemes: The first is to train only the memory module. In this setting, both the model backbone and the waypoint action head remain frozen, and only the parameters related to retrieval, cross-attention, and gating in the memory module are updated. This strategy is used to isolate the role of the memory module itself, so that the performance change mainly comes from the explicit memory path, rather than the backbone representation or the action head re-adaptation. Its advantages are low training cost and few variables. The limitation is that the original action head receives the waypoint feature distribution formed during the training of the baseline model, while the features after memory fusion may be shifted, and freezing the action head may not be able to fully utilize the enhanced representation. The second is to train both the memory module and the action head. In this setting, the main body of the multimodal large model remains frozen, but the memory module and the waypoint action head participate in the training together. This strategy is based on a more realistic assumption: the memory module changes the waypoint lexical feature distribution, and the action head needs to re-adapt to this memory-enhanced representation. Compared with the first strategy, this setting increases the learnability of the action prediction layer, enabling the model to learn how to map the features after memory fusion to continuous waypoints while maintaining the stability of the original visual language representation. Similar ideas of local fine-tuning are more common in large-scale model applications. Their advantage lies in avoiding retraining of a large-scale backbone while allowing task-related modules to be fully optimized.

[0069] It should be noted that, regarding the loss function, this application maintains the same waypoint supervision form as the baseline model, ensuring comparability between the memory-enhanced model and the baseline in terms of training objectives and evaluation protocols. The expression for the loss function is as follows:

[0070] in, Indicates the total loss; This represents the path point loss, used to constrain the positional deviation between the predicted waypoints and the actual waypoints; This represents the orientation loss, used to constrain the orientation deviation of the drone; The weighting coefficients represent the orientation loss and are used to balance the proportions of the two types of losses. This application does not currently introduce additional supervised loss for the retrieval memory frame, but this can be expanded to include a loss term for the retrieval memory frame in the future.

[0071] Furthermore, to comprehensively evaluate the memory enhancement effect of this method, the embodiments of this application design four types of diagnostic indicators to measure the model's memory behavior at different granularities. Traditional VLN and UAV VLN evaluations typically focus on outcome indicators such as final success rate, destination distance, or trajectory deviation. These indicators can reflect the overall navigation capability of the model, but they are difficult to distinguish whether the model failure is caused by visual recognition errors, path planning errors, or memory retention and retrieval errors. For the memory task designed in this application, the model may fail near the final position, or deviations may have already occurred in the order, counting, or revisiting stages of intermediate memory points. Therefore, more granular diagnostic indicators are needed.

[0072] The first type is the Reach-to-Memory Ratio (RMPR), which measures the proportion of the model that reaches a memory point during execution. Its calculation formula is as follows:

[0073] in, This represents the total number of memory points in a sample, arranged according to the task definition. This metric represents the number of memory points actually reached by the model in the valid order. It is primarily used for long sequence tasks and location-environment-action tasks. For long sequence tasks, RMPR calculation emphasizes order constraints: the model only reaches memory points if it follows the order of instructions. After the first memory point, the second A model is considered valid only if all memory points are reached. If the model reaches a subsequent target before returning to the previous target, it cannot be simply counted as completing both targets. This design prevents the model from obtaining excessively high scores due to out-of-order access, thus more accurately reflecting the ability to remember sequence order and update the state of the online task. For location-environment-action tasks, RMPR can be used to measure the model's phased completion of target memory points during forward or backward visits.

[0074] The second category is counting error. This is used to measure the cumulative state error of the model in counting and circling tasks, and its calculation formula is:

[0075] in, Indicates the target count or target number of laps required by the task; This represents the count predicted by the model or actually completed. This metric directly reflects the model's deviation in the accumulation of states over time. For counting tasks, The evaluation measures whether the model correctly counts the occurrences of target objects or landmarks; for looping tasks, This measures whether the model has completed the specified number of revolutions. The smaller the value, the more accurately the model preserves the cumulative state. It's important to note that... It does not require the final trajectory of the model to be completely correct; its main focus is on whether the "quantity state" has been correctly maintained. It can be used in conjunction with task success rate: the former reveals the magnitude of the error, while the latter determines whether the sample was strictly successful.

[0076] The third category is the task success rate (SR), which serves as a sample-level strict success rate. Its calculation formula is as follows:

[0077] in, This indicates the total number of samples in the test set; This represents the number of successful samples in the test set. The definition of SR varies depending on the task type, but all follow the principle of "success is determined only if all task requirements are fully met": For long sequence tasks, the model must reach all memory points in the correct order; for counting tasks, the model must complete the action under the condition of correct counting; for circling tasks, the model must complete the specified number of circlings; for position-environment-action tasks, the model needs to execute the corresponding action at the correct historical position; for look-ahead triggering tasks, the model must trigger the action when the correct cue appears. SR is the most stringent overall evaluation metric, directly reflecting the model's end-to-end completion capability for a certain task type. Unlike RMPR and CE, SR does not measure partial completion; if any key constraint among sequence, counting, revisiting, or triggering conditions is not met, the sample is considered a failure. For look-ahead triggering tasks, only SR is used as the evaluation metric because the core objective of this task is to determine whether the model executes the specified action when the correct future cue appears, and the boundary between success and failure is relatively clear.

[0078] The fourth category is Successful Revisit Rate (SRR), specifically used to evaluate historical revisit capability in location-environment-action tasks. Its calculation formula is:

[0079] in, This represents the total number of samples in the location-environment-action task; This indicates the number of samples in which the model successfully revisits the target memory point. This task requires the model to return to a previously visited specified memory point based on location, environment, or action cues; therefore, the final success rate alone is insufficient to determine whether the model has truly completed the "revisit." The difference between SRR and SR is that SRR only focuses on whether the model returns to the correct historical location, while SR further requires the correct action to be performed at that location. Therefore, when the model returns to the correct location but performs the wrong action, SRR can be considered a success while SR is considered a failure; when the model performs the correct action but the location is incorrect, both should be considered failures. This metric can distinguish between "location memory errors" and "action binding errors," thus allowing for a more detailed analysis of the sources of failure in context binding ability.

[0080] In general, RMPR, CE, SR, and SRR characterize model performance from four perspectives: partial completion, cumulative state error, rigorous task success, and historical revisit capability, respectively. RMPR is suitable for analyzing the staged completion status in multi-memory-point tasks; CE is suitable for analyzing the cumulative error in counting and circling tasks; SR provides a unified end-to-end success determination; and SRR supplements the analysis of revisit capability in location-environment-action tasks. Through these metrics, the benchmark of this application can further attribute model errors to different types such as sequential state loss, counting state error, historical position revisit failure, action binding error, or look-ahead triggering failure, thereby providing interpretable evaluation criteria for the optimization and analysis of memory-enhanced models.

[0081] This application embodiment experimentally verifies the above-mentioned memory-enhanced unmanned system instruction interaction and task planning method, namely MemoryUAV.

[0082] This embodiment's experiments revolve around three core questions: First, whether different training strategies affect the flight stability of the memory-enhanced model; second, whether MemoryUAV can maintain or improve baseline navigation capabilities on the original OpenUAV dataset; and third, whether the explicit memory mechanism can improve the performance of UAV VLN tasks requiring historical information on the memory benchmark constructed in this application. It should be understood that this embodiment uses OpenUAV as the primary baseline model, whose task is to predict subsequent waypoints based on natural language instructions and visual observations. MemoryUAV adds a memory module based on instruction unit filtering, uses environmental field representations to retrieve relevant historical frames, and injects memory into waypoint features through cross-attention and gating mechanisms.

[0083] It should be noted that the experimental data in this embodiment includes two parts: first, the original OpenUAV dataset, used to evaluate the overall performance of the model on conventional UAV VLN tasks; and second, the memory benchmark constructed in this application, used to evaluate the model's performance on five types of memory tasks: long sequence tasks, counting tasks, circling tasks, position-environment-action tasks, and look-ahead triggering tasks. The evaluation metrics for each task have been defined in the preceding embodiments; this embodiment will not repeat the metric formulas, but will only analyze the experimental results.

[0084] To further clarify, this embodiment first compares the impact of two training strategies on model performance: one trains only the memory module, and the other trains both the memory module and the action head. The first strategy freezes the OpenUAV backbone and waypoint action heads, updating only the retrieval, cross-attention, and gating parameters in the memory module. The second strategy freezes the multimodal backbone and trains both the memory module and the waypoint action heads simultaneously. The core difference between the two strategies lies in whether the action head can adapt to the feature distribution after memory fusion.

[0085] Experimental results show that the model's stability is poor when only the memory module is trained. Once memory fusion is enabled, the waypoint feature distribution changes relative to the baseline, while the frozen action head still predicts according to the original feature distribution, leading to easy deviations from the flight trajectory and frequent crashes during experiments. Further removal of instruction unit filtering, i.e., inputting all historical memories from the memory bank into cross-attention at each step, results in even worse model performance, typically failing within ten steps. This phenomenon indicates that unfiltered historical memories introduce a large amount of information irrelevant to the current action, and the frozen action head cannot digest this noise. In contrast, training both the memory module and the action head significantly improves stability. After the action head participates in training, it can re-adapt the memory-enhanced waypoint representation, making the impact of memory retrieval on continuous waypoint prediction more controllable. Considering that the goal of this application is to verify the role of explicit memory mechanisms in memory-based tasks, rather than large-scale retraining of a multimodal backbone, subsequent experiments all adopt a training strategy combining memory modules and action heads.

[0086] It should be noted that, to evaluate the impact of MemoryUAV on conventional navigation capabilities, this embodiment compares the evaluation results of CMA, OpenUAV, OpenUAV-DA, and MemoryUAV on the OpenUAV dataset. The significance of this experiment is to confirm that the memory module does not impair the basic flight capabilities in the original UAV VLN task. If the memory-enhanced model significantly degrades on ordinary tasks, its performance on memory tasks cannot demonstrate the effectiveness of the method. The evaluation results of each model are shown in Table 2.

[0087] Table 2

[0088] The experimental results show that MemoryUAV outperforms the OpenUAV baseline in all three metrics: NE, SR, and OSR. This indicates that adding the instruction unit-based memory module not only does not compromise the original navigation capabilities but also improves overall navigation success rate and reachability. The SPL metric is close to that of OpenUAV-DA, demonstrating that MemoryUAV improves success rate without significant degradation in path efficiency. These results demonstrate that the instruction unit-based filtering mechanism effectively controls memory noise, maintaining compatibility of the memory module in typical OpenUAV tasks.

[0089] It is worth mentioning that the memory benchmark constructed in this application is used to test whether the model possesses explicit memory capabilities. Compared with the original OpenUAV dataset, each task in this dataset requires the model to make decisions using historical observations, environmental anchors, or task stage information, thus making it more suitable for evaluating the role of memory banks and instruction retrieval. This embodiment compares the evaluation results of OpenUAV and MemoryUAV on five task categories, and the experimental results for each task are as follows.

[0090] First, the experimental results for long sequence tasks are shown in Table 3.

[0091] Table 3

[0092] Experimental results show that MemoryUAV significantly improves both RMPR and SR in long sequence tasks, indicating that the model can better maintain task stages according to historical memory points and effectively improve the ability to remember sequence order and update online task state.

[0093] Secondly, the experimental results of the counting task are shown in Table 4.

[0094] Table 4

[0095] In counting tasks, MemoryUAV significantly reduced CE and significantly improved SR, indicating that explicit memory banks are helpful in maintaining the frequency of target occurrences or evidence of key objects, and can effectively improve the accuracy of counting memory.

[0096] Furthermore, the experimental results for the circling task are shown in Table 5.

[0097] Table 5

[0098] In the circling task, the difference between the two is relatively small, but MemoryUAV still achieves a higher SR, indicating that the task is affected by the stability of motion control in addition to memory, and the explicit memory mechanism can still bring certain performance gains.

[0099] It should be noted that the experimental results for the location-environment-action task are shown in Table 6.

[0100] Table 6

[0101] In the location-environment-action task, MemoryUAV significantly outperformed the baseline in RMPR, SRR, and SR, indicating that the environment anchors in the instruction unit can effectively help the model locate key environments that have appeared before and link them with subsequent actions, significantly improving context binding and historical revisit capabilities.

[0102] Finally, the experimental results of the look-ahead triggering task are shown in Table 7.

[0103] Table 7

[0104] In look-ahead triggering tasks, MemoryUAV's SR is significantly higher than OpenUAV's, indicating that the model has a stronger conditional response capability in future cue-triggered tasks, effectively maintaining delayed intent and accurately triggering actions when the corresponding cue appears.

[0105] Overall, the results demonstrate that the memory benchmark effectively amplifies the shortcomings of ordinary UAV VLN models in explicit memory capabilities. OpenUAV relies on implicit context, making it difficult to stably preserve crucial evidence over long task cycles; MemoryUAV, on the other hand, retrieves historical frames through context field units, enabling current action prediction to access memories relevant to subtasks. This conclusion aligns with fundamental viewpoints in long-term memory research: the key to long-term tasks lies not only in preserving history but also in retrieving information relevant to the current goal from that history.

[0106] Optionally, to verify the necessity of the instruction unit filtering mechanism, this embodiment conducts ablation experiments on the OpenUAV dataset. The ablation version removes environment field memory retrieval and no longer filters relevant frames based on the current instruction unit. Instead, at each step, all historical memories in the memory bank are used as keys and values ​​for cross-attention. This setting preserves the memory bank and memory fusion structure but removes task-related retrieval constraints. The results of the ablation experiments are shown in Table 8.

[0107] Table 8

[0108] Experimental results show that removing instruction unit filtering significantly degrades model performance. Specifically, SR drops to 0, and OSR and SPL also decrease substantially, indicating that while the model can access more historical information, it does not gain stronger capabilities as a result. Conversely, the complete memory bank contains a large number of transition frames and irrelevant environment frames, and cross-attention of all inputs can cause waypoint features to be disturbed by noise, thus disrupting continuous action prediction. This phenomenon is consistent with the crash results in the training strategy experiment: in UAV VLN, the memory module cannot be simply equated with a longer context; unfiltered historical information may be less stable than no memory at all.

[0109] The complete MemoryUAV uses the environment field representation in the instruction unit for retrieval, enabling the model to prioritize the fusion of historical frames relevant to the current subtask. This mechanism maps the environment anchors of memory points in the aforementioned embodiments to the memory retrieval process within the model, thereby reducing irrelevant memories from entering action prediction. Ablation results show that the performance improvement of MemoryUAV does not come from simply increasing the memory bank, but from the combined design of task-related filtering and memory fusion.

[0110] It should be understood that, based on the above experimental results, the boundaries of MemoryUAV's role are relatively clear: on the OpenUAV raw dataset, it maintains baseline navigation capabilities and improves performance on multiple metrics; on the memory benchmark, it effectively improves the performance of various memory tasks through explicit history retrieval. Experimental results show that the memory benchmark proposed in this application can effectively detect differences in model memory capabilities, and the instruction unit-based memory retrieval mechanism can also provide effective historical evidence support for UAV VLN, achieving simultaneous improvement in navigation performance and memory capabilities.

[0111] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0112] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0113] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A method for instruction interaction and task planning in unmanned systems based on memory enhancement, characterized in that, include: Obtain task instructions, break down the task instructions, and generate multiple instruction units arranged in the order of task execution. Each instruction unit contains environmental anchor point information associated with the corresponding sub-task stage. At the beginning of each task execution cycle, the historical perception memory bank is cleared, and during the task execution, the current environmental perception data of the unmanned system is collected, the feature information of the environmental perception data is extracted, and the feature information is written into the historical perception memory bank to update the historical perception memory bank. The current instruction unit is advanced sequentially according to the order of the instruction units, and the candidate retrieval range is determined in the historical sensory memory bank according to the progress status of the current instruction unit. The environmental anchor information in the current instruction unit is matched with each historical perception feature in the candidate retrieval range, and the historical perception features that meet the preset matching conditions are selected as task-related memory information. The current planning and decision features are used as queries, and task-related memory information is used as keys and values. These are then fused through an attention mechanism to obtain memory-enhanced decision features. These memory-enhanced decision features are then input into the action prediction module to generate subsequent task planning results and complete the path point generation for the current subtask stage.

2. The method according to claim 1, characterized in that, Obtain task instructions, break down the task instructions, and generate multiple instruction units arranged in the order of task execution, including: Obtain the complete text description of the task instruction, call the large language model interface, and perform structured segmentation of the task instruction according to the predefined instruction unit template, so that each complete instruction is divided into multiple instruction units that correspond one-to-one with the sub-task stage; Each instruction unit is labeled with a corresponding subtask stage number, so that each instruction unit forms an ordered sequence according to the task execution stage, and the successor relationship between adjacent instruction units is established. The sequence of instruction units after being split is saved in a structured file format as a basis for subsequent memory retrieval.

3. The method according to claim 2, characterized in that, Each instruction unit contains environmental anchor information associated with the corresponding subtask phase, where: The environmental anchor information is a semantic description of the environmental landmarks or objects associated with the memory points in the corresponding subtask stage of the instruction unit. There is an explicit binding relationship between the semantic description and the landmarks or objects pre-arranged in the simulation scene. The environmental anchor information is used to map local semantic units in natural language instructions into computable vector representations. The vector representations are obtained through word segmentation and encoding, and have the same feature dimension as the memory feature vectors stored in the historical perception memory bank. The environmental anchor information serves as a query condition for retrieving the historical perception memory database, and is used to filter historical perception features related to the current subtask stage.

4. The method according to claim 3, characterized in that, At the beginning of each task execution cycle, the historical perception memory bank is cleared, including: Clear all historical perception feature vectors stored in the historical perception memory bank to ensure that historical information within the current task execution cycle is not reused across cycles. Clear the index of search results corresponding to each recorded instruction unit to avoid the search records of the previous cycle from interfering with the current cycle; Reset the progress state of the current instruction unit to the initial state, so that the instruction unit sequence of the current cycle advances sequentially from the first instruction unit; Load the instruction unit sequence corresponding to the current task execution cycle as the basis for memory retrieval within the current cycle.

5. The method according to claim 4, characterized in that, Collecting environmental perception data of the unmanned system at the current moment, extracting feature information from the environmental perception data, writing the feature information into the historical perception memory bank, and updating the historical perception memory bank include: During the mission execution, the environmental perception data of the unmanned system is collected step by step according to the time step. The environmental perception data is input into the multimodal encoder to extract the frame-level feature representation of the current moment, wherein the frame-level feature includes visual perception information. Frame-level feature representations are pooled and compressed along the sequence dimension to obtain fixed-dimensional memory feature vectors, thereby reducing storage overhead and unifying feature dimensions. The memory feature vectors are concatenated along the time dimension into the historical perception memory bank, so that the historical perception memory bank accumulates and stores the memory feature vectors of each historical moment in the current task execution cycle in chronological order. When the number of memory feature vectors in the historical perception memory bank exceeds a preset capacity threshold, the latest memory feature vector with the latest time sequence is retained, and the earliest memory feature vector with the earliest time sequence is deleted.

6. The method according to claim 5, characterized in that, Based on the progress status of the current instruction unit, a candidate retrieval range is determined in the historical perceptual memory, including: Obtain the progress status of the current instruction unit, wherein the progress status is used to indicate the position number of the instruction unit that has been retrieved and completed in the instruction unit sequence; Using progress status as a time constraint, the historical perception features located after the index of the retrieved frame in the historical perception memory are determined as candidate retrieval ranges, so that the retrieval of the current instruction unit is only carried out in historical frames that have not yet been retrieved. After the retrieval corresponding to the current instruction unit is completed and the task-related memory information is successfully obtained, the progress status is updated so that the retrieval of the next instruction unit is executed in the updated progress status.

7. The method according to claim 6, characterized in that, The environmental anchor information in the current instruction unit is matched with each historical perception feature in the candidate retrieval range, and the historical perception features that meet the preset matching conditions are selected as task-related memory information, including: The text of the environmental anchor information in the current instruction unit is segmented and encoded, and then mapped into a query tensor. The query tensor has the same feature dimension as the memory feature vector in the historical perception memory bank. The cosine similarity between the query tensor and each historical perception feature within the candidate retrieval range is calculated sequentially, and the cosine similarity is mapped from the value range to a preset interval to obtain the matching degree between each historical perception feature and the environmental anchor information. The historical perceptual feature with the highest matching degree within the candidate retrieval range is selected as candidate memory information, and the matching degree is compared with a preset threshold. When the matching degree exceeds the preset threshold, the corresponding historical perception feature is accepted as task-related memory information, and the frame index of the historical perception feature in the historical perception memory bank is recorded. When the matching degree does not exceed the preset threshold, the current search result is abandoned and the current progress state is maintained.

8. The method according to claim 7, characterized in that, Using the current planning and decision features as the query and task-related memory information as the key and value, the data is fused through an attention mechanism to obtain memory-enhanced decision features. These memory-enhanced decision features are then input into the action prediction module to generate subsequent task planning results, including: The planning and decision features at the current moment are linearly mapped to a query vector, and the task-related memory information is linearly mapped to a key vector and a value vector. By using a multi-head cross-attention mechanism, the attention weight between the query vector and the key vector is calculated, and the attention weight is applied to the value vector to obtain a memory-enhanced representation after weighted aggregation of task-related memory information. The fusion weights are calculated through a gating mechanism. The memory-enhanced representation is then weighted and summed with the planning and decision features at the current moment to obtain the memory-enhanced decision features. The fusion weights are controlled by learnable gating parameters to adaptively adjust the proportion of historical information and current information in the memory-enhanced decision features. The memory-enhanced decision features are input into the action prediction module to generate a continuous path point sequence for the current subtask stage, thus completing the path planning for the current subtask stage.

Images