A data augmentation method for UAV vision-language navigation tasks

Abstract

The present invention belongs to the technical field of deep learning, and specifically discloses a data augmentation method for unmanned aerial vehicle visual language navigation tasks, including: determining a path-action sequence-visual observation pair; obtaining a path-image feature-text feature pair; obtaining a merged path-image feature-text feature pair; constructing a training data set for training a projection layer according to visual observations and the merged path-image feature-text feature pair; training the projection layer using the training data set; obtaining a path-image feature embedding-text feature embedding-instruction description pair; screening the path-image feature embedding-text feature embedding-instruction description pair to obtain augmented data. The present invention can simultaneously utilize the dual-scale visual representation capability and large language model of a dual-scale graph Transformer model to improve the quality of generated instructions. The problem of poor quality of natural language navigation instructions generated in the prior art is solved.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology, and in particular relates to a data augmentation method for unmanned aerial vehicle (UAV) visual language navigation tasks. Background Art

[0002] Vision and Language Navigation (VLN) is a challenging task that requires an agent to autonomously navigate in unseen real-world environments following human natural language instructions, such as “go downstairs, walk to the dining table, turn left to the kitchen, and stop in front of the refrigerator.”

[0003] Solving VLN tasks relies heavily on correctly interpreting instructions, perceiving the environment, and learning from interactions, which requires a large amount of diverse visual-language data to train the model. However, due to the expensive process of collecting large-scale navigation data, visual-language navigation learning usually faces the problem of data scarcity. Currently, there are many methods to solve the problem of visual-language navigation data scarcity, including augmenting data by collecting more human annotations or creating new environments, however, this method is too costly. In addition, recent methods tend to use a large amount of automatically generated data to push the limits of agent performance, or introduce large-scale pre-training methods to improve generalization ability, however, automatically generated data faces problems such as low data quality.

[0004] The visual language navigation task (AerialVLN) based on urban drones is a more challenging task. Compared with indoor or ground VLN tasks, AerialVLN has a larger action space, a larger and more complex outdoor environment, and a longer path and instruction sequence. These characteristics together determine the complexity of the AerialVLN task. Since the average navigation path length of the AerialVLN task is more than 600 meters, this determines that its corresponding navigation instruction sequence will also have more and more complex descriptions, so it is more difficult to augment the dataset for the AerialVLN task. Most of the existing work is completed by manually collecting paths and generating corresponding instruction descriptions. This method is costly and inefficient, and obviously cannot meet the needs of large-scale data augmentation. There are also some works that aim to achieve large-scale data augmentation by automatically generating data. However, most of the work is carried out for indoor or ground VLN tasks. Directly migrating it to the data augmentation task of the AerialVLN task will result in the "long flat" problem where the key actions are sparse and irrelevant semantic redundancy as the path length becomes longer.

[0005] The visual language navigation task dataset mainly consists of two parts: path sequence and navigation instructions. Among them, generating navigation instructions with high richness, conforming to human language habits, and rich navigation semantics is the main challenge faced by data augmentation, which largely determines the performance of the intelligent agent. Most existing methods use existing artificial datasets of specific scenes to train a local small model to solve the navigation instruction generation task. However, this method often faces defects such as insufficient generalization ability, strong data dependence, and poor transferability. Typical methods are:

[0006] 1) Speaker-follower

[0007] The Speaker-Follower method consists of two main components: the Speaker module and the Follower module. The Follower module receives visual input and language instructions and generates navigation decisions through a neural network; the Speaker module receives the navigation path and generates natural language instructions that describe the path. Through loop training of the existing data set, the Speaker module generates new instruction-path pairs to increase data diversity, and the Follower module uses this data to learn navigation strategies. However, the Speaker-Follower method has the defects of high dependence on data and model complexity. The quality and diversity of the generated language instructions are insufficient, which affects the accuracy of the navigation strategy. In addition, two complex modules need to be optimized simultaneously during the training process, which increases the demand for computing resources and the difficulty of model training. The decoupling of instruction understanding and path execution and the limitations of evaluation indicators may also affect the performance of the model in the actual environment.

[0008] 2) CCC-Speaker

[0009] The CCC-Speaker method adopts a counterfactual cycle-consistent learning framework to jointly train the "Speaker" and "Follower" models and introduce a "Creator" module. Specifically, the Speaker generates instructions describing the navigation path, the Follower navigates according to the instructions, and the Creator is responsible for generating the counterfactual environment. Through cycle-consistency learning, the Follower generates the path and the Speaker verifies its correctness, and vice versa. This method not only uses labeled data for training, but can also be applied on unlabeled paths to enhance the generalization ability of the model. Although the CCC-Speaker method has a significant effect in improving the performance of the navigation model, it also has some defects. First, this method increases the complexity of training and computing resource requirements because three complex modules (Speaker, Follower, and Creator) need to be trained simultaneously. Second, although the generation of counterfactual environments can enhance the robustness of the model, the quality of the generated counterfactual environments is difficult to guarantee, which may affect the training effect of the model. In addition, the difference between the creation of counterfactual environments and the real environment may cause the model to perform unstable in practical applications.

[0010] 3) AIGeN

[0011] The AIGeN (Adversarial Instruction Generation for Vision-and-Language Navigation) method is an architecture based on Generative Adversarial Networks (GANs) that aims to generate meaningful and well-structured synthetic instructions to improve the performance of navigation agents. The model consists of a Transformer decoder (GPT-2) and a Transformer encoder (BERT). During the training phase, the decoder generates sentences describing the agent's path, while the encoder is responsible for distinguishing true from false instructions. By generating synthetic instructions for unlabeled navigation paths, AIGeN can significantly improve the performance of off-the-shelf VLN methods, especially instructions for generating 217K trajectories on the Habitat-Matterport 3D (HM3D) dataset, and has been verified on the REVERIE and R2R datasets, showing superior performance. Although the AIGeN method performs well in improving the performance of navigation models, it also has some drawbacks. First, the training process of this method is complex and requires a lot of computing resources and time, because both the generator and the discriminator modules need to be trained simultaneously. In addition, AIGeN relies on high-quality object detection models (such as Mask2Former) to extract visual features, which is not only time-consuming, but also may contain errors in the visual feature extraction process, affecting the quality of the generated instructions. Finally, although the instructions generated by AIGeN perform well in experiments, in real environments, its performance may be limited by the consistency of the training dataset and generative adversarial training, resulting in poor stability and generalization of the model in practical applications.

[0012] 4) EDrop-Speaker

[0013] The EDrop-Speaker method improves the generalization ability of the visual language navigation (VLN) model in unknown environments by introducing an environmental dropout strategy. The method adopts a two-stage strategy in the training process: first, it combines imitation learning and reinforcement learning for training, and then generates new unseen environmental data through environmental dropout. During the environmental dropout process, the model randomly discards visual features to simulate the loss and change of perspective in the real world, and uses back translation to generate new paths and instructions to enhance the generalization ability of the model. Experiments show that by fine-tuning on these newly generated environmental data, the navigation performance of the model in unknown environments is significantly improved. Although the EDrop-Speaker method performs well in improving the generalization ability of the model, it also has some defects. First, the training process of this method is complex, especially the environmental dropout and back translation steps, which require a lot of computing resources and time. Second, the new environmental data generated by environmental dropout may not fully represent the complex changes in the actual unknown environment, resulting in the limited stability and reliability of the model in practical applications. In addition, the quality and diversity of the generated new data depend to a certain extent on the quality of the existing data. Insufficient or low-quality data may limit the performance improvement of the model. Summary of the invention

[0014] In view of the above-mentioned deficiencies in the prior art, the present invention provides a data augmentation method for unmanned aerial vehicle visual language navigation tasks, which solves the problem of poor quality of natural language navigation instructions generated in the prior art.

[0015] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is: a data augmentation method for unmanned aerial vehicle visual language navigation task, comprising the following steps:

[0016] S1, obtaining a navigation map, searching the navigation map using a heuristic search algorithm, obtaining a path and an action sequence, calling a simulator to generate a visual observation corresponding to the path, and determining a path-action sequence-visual observation pair based on the path, the action sequence and the visual observation;

[0017] Wherein, each path includes at least two path points;

[0018] S2, using the pre-trained visual feature encoder and the pre-trained language feature encoder to pre-process the existing visual language navigation task dataset and the path-action sequence-visual observation pair respectively to obtain the path-image feature-text feature pair;

[0019] The existing visual language navigation task dataset includes at least one path-action sequence-visual observation-instruction description pair data;

[0020] S3, using a dynamic processing strategy based on equality judgment to dynamically merge the path-image feature-text feature pair to obtain a merged path-image feature-text feature pair;

[0021] S4, constructing a training dataset for training the projection layer based on visual observations and the merged path-image feature-text feature pairs;

[0022] S5. Use the training dataset to train the projection layer, and use the model fine-tuning method to fine-tune the large language model based on the existing visual language navigation task dataset;

[0023] S6. Use the visual space representation module of the DUET model to process the merged path-image feature-text feature pair to obtain a path-image feature embedding-text feature embedding pair, and use the trained projection layer to project the path-image feature embedding-text feature embedding pair to the semantic space of the fine-tuned large language model, and output the path-image feature embedding-text feature embedding-instruction description pair through the fine-tuned large language model;

[0024] S7. Use language evaluation indicators to filter the path-image feature embedding-text feature embedding-instruction description pairs to obtain augmented data.

[0025] The beneficial effects of the above scheme are:

[0026] (1) The present invention utilizes the dual-scale visual representation capability of the DUET model and the contextual learning and reasoning capability of the large language model to construct a hierarchical instruction generation architecture. At the same time, prompt templates or thought chains can be used to make the generation effect better, the language description more precise, and more in line with human style habits. By training it with existing datasets and fine-tuning it with LoRA, the generated instructions are more in line with the AerialVLN style, thereby improving the quality of the generated instructions.

[0027] (2) The DUET model in the present invention can globally and locally encode the spatial visual information on the navigation path, that is, it not only pays attention to the current spatial visual information, but also pays attention to the global historical information. Therefore, the representation of the spatial vision on the path by the DUET model will not decline as the path sequence becomes longer. In addition, the dynamic processing strategy of the present invention can reduce redundant features. The instructions finally generated are concise and contain key actions, which is crucial for solving the "long flat" problem of the AerialVLN task.

[0028] (3) Compared with typical data augmentation methods such as data interpolation and mixing, noise injection, and expert models designed for specific scenarios, the present invention utilizes a large language model and benefits from its large amount of rich corpus for pre-training, so that the generated data has rich navigation semantic information and is more in line with human language habits.

[0029] Further, in step S2, the pre-trained visual feature encoder is ViT-B / 16, and the pre-trained language feature encoder is BERT;

[0030] Step S2 specifically includes:

[0031] S21, using ViT-B / 16 to extract features from the visual observation in the path-action sequence-visual observation-instruction description pair to obtain a first image feature;

[0032] S22, using BERT to extract features from the action sequence and instruction description in the path-action sequence-visual observation-instruction description pair to obtain a first text feature;

[0033] S23, determining a first path-image feature-text feature pair according to the path-action sequence-visual observation-instruction description pair, the first image feature and the first text feature;

[0034] S24, extracting features of the visual observations in the path-action sequence-visual observation pair using ViT-B / 16 to obtain second image features;

[0035] S25, using BERT to extract features of the action sequence in the path-action sequence-visual observation pair to obtain a second text feature;

[0036] S26, obtaining a second path-image feature-text feature pair according to the path-action sequence-visual observation pair, the second image feature and the second text feature;

[0037] S27: Merge the first path-image feature-text feature pair and the second path-image feature-text feature pair to obtain a path-image feature-text feature pair.

[0038] The beneficial effects of the above further scheme are: the visual feature encoder can automatically extract meaningful feature representations from images or videos. These features are usually highly generalized and task-related abstract descriptions of the image content, which improves the efficiency and effectiveness of subsequent task processing. The language feature encoder can capture the semantic information of words, phrases and even entire sentences by converting text data into high-dimensional feature vectors, thereby deepening the understanding of the meaning of the text.

[0039] Furthermore, in step S3, a dynamic processing strategy based on equality judgment is used to dynamically merge the path-image feature-text feature pair, specifically including:

[0040] Using a dynamic processing strategy based on equality judgment, the text features in the path-image feature-text feature pair are traversed to determine whether adjacent text features are the same. If so, the path-image feature-text feature pairs corresponding to the adjacent text features are merged. If not, the path-image feature-text feature pairs are continued to be traversed.

[0041] The beneficial effect of the above further scheme is that through equality judgment, data pairs with the same or related paths, image features and text features can be quickly identified, and only these matching data are merged, which reduces unnecessary traversal and processing operations and improves processing speed. Reducing redundant information helps solve the problem of sparse key signals in AerialVLN instructions, making the generated instructions concise while containing key actions.

[0042] Further, step S4 specifically includes:

[0043] S41, inputting the merged path-image feature-text feature pair into the coarse-scale encoding module of the DUET model for scene graph encoding, and outputting the path-scene graph encoding embedding-text feature pair through the DUET model;

[0044] S42, using the image description model to process the visual observation to obtain a natural language description corresponding to the visual observation;

[0045] S43, designing a natural language generation template, and using the natural language generation template to train a large language model;

[0046] S44, inputting the path-scene graph encoding embedding-text feature pair and the natural language description into the trained large language model, and outputting the structured natural language description through the trained large language model;

[0047] S45. Input the structured natural language description into the language encoding module, and output the scene graph encoding embedding-natural language encoding embedding through the language encoding module as a training data set for training the projection layer.

[0048] The beneficial effect of the above further scheme is that by combining image features, text features and scene graph encoding, a deep integration of visual and language information is achieved, which promotes the comprehensive understanding and expression of complex scenes. Customized natural language generation templates combined with large language model training can generate more accurate, structured and natural language descriptions that meet specific application scenarios, improving the practicality and readability of training data sets.

[0049] Furthermore, step S5 specifically includes:

[0050] S51, using a randomly initialized projection layer to connect the global space visual representation module of the DUET model with the semantic space of the trained large language model, using a Q-former initialized by InstructBLIP to connect the local space visual representation module of the DUET model with the semantic space of the trained large language model to form a projection layer, and using the training data set to train the projection layer to obtain a trained projection layer;

[0051] S52, freezing the parameters of the visual representation module and the trained projection layer of the DUET model, and fine-tuning the trained large language model using a model fine-tuning method based on an existing visual language navigation task dataset;

[0052] Among them, the visual representation module of the DUET model includes a global space visual representation module and a local space visual representation module.

[0053] The beneficial effects of the above further scheme are: global representation covers the overall semantics, local representation focuses on detailed information, and the fusion of the two allows the model to understand the macro background and micro elements of the image at the same time, improving the ability to understand complex scenes. The effective integration of global and local visual features and language features can enhance the consistency between visual information and text descriptions, and improve the accuracy and richness of image content descriptions.

[0054] Furthermore, in step S7, the language evaluation indicators include BLEU, METEOR, ROUGE, CIDEr, SPICE and NIST.

[0055] The beneficial effect of the above further solution is that the quality of generative tasks such as machine translation, text summarization, and dialogue systems can be objectively quantified through standardized language evaluation indicators. It helps developers and researchers understand the strengths and weaknesses of the model and guides the iterative optimization of the model.

[0056] Further, in step S52, the model fine-tuning method includes LoRA, QLoRA, Adapter-tuning and Prefix-tuning.

[0057] The beneficial effect of the above further scheme is that the fine-tuning method reduces the number of required parameters and improves computational efficiency and flexibility, so that the model can be applied to specific tasks more efficiently and specifically, while also lowering the resource threshold in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 Shown is a flowchart of a data augmentation method for UAV vision-language navigation tasks.

[0059] Figure 2Shown is a model architecture diagram of a data augmentation method for drone vision-language navigation tasks. DETAILED DESCRIPTION

[0060] Now, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be understood that the embodiments shown and described in the accompanying drawings are only exemplary and are intended to explain the principles and spirit of the present invention, rather than to limit the scope of the present invention.

[0061] like Figure 1 As shown, a data augmentation method for UAV visual language navigation task includes the following steps:

[0062] S1, obtaining a navigation map, searching the navigation map using a heuristic search algorithm, obtaining a path and an action sequence, calling a simulator to generate a visual observation corresponding to the path, and determining a path-action sequence-visual observation pair based on the path, the action sequence and the visual observation;

[0063] Each path includes at least two path points.

[0064] Optionally, the simulators in this embodiment may include AirSim Simulator, Habitat Simulator, Matterport3D Simulator, etc., and no specific limitation is made here, as long as the corresponding functions of the simulator can be realized.

[0065] For example, AirSim Simulator is an advanced open source simulation platform, mainly used for algorithm development and testing of unmanned systems such as drones and cars. Habitat Simulator is a platform focused on studying the learning and decision-making of intelligent agents in virtual environments, suitable for robot navigation and manipulation tasks in home or indoor scenes. Matterport3D Simulator is based on the Matterport3D dataset and provides a detailed 3D indoor environment platform for research and development of AI applications involving spatial understanding, navigation and interaction.

[0066] S2, using the pre-trained visual feature encoder and the pre-trained language feature encoder to pre-process the existing visual language navigation task dataset and the path-action sequence-visual observation pair respectively to obtain the path-image feature-text feature pair;

[0067] Existing visual language navigation task datasets include at least one path-action sequence-visual observation-instruction description pair data.

[0068] In this embodiment, in step S2, the pre-trained visual feature encoder is ViT-B / 16, and the pre-trained language feature encoder is BERT;

[0069] Step S2 specifically includes:

[0070] S21, using ViT-B / 16 to extract features from the visual observation in the path-action sequence-visual observation-instruction description pair to obtain a first image feature;

[0071] S22, using BERT to extract features from the action sequence and instruction description in the path-action sequence-visual observation-instruction description pair to obtain a first text feature;

[0072] S23, determining a first path-image feature-text feature pair according to the path-action sequence-visual observation-instruction description pair, the first image feature and the first text feature;

[0073] S24, extracting features of the visual observations in the path-action sequence-visual observation pair using ViT-B / 16 to obtain second image features;

[0074] S25, using BERT to extract features of the path-action sequence-visual observation pair to obtain a second text feature;

[0075] S26, obtaining a second path-image feature-text feature pair according to the path-action sequence-visual observation pair, the second image feature and the second text feature;

[0076] S27: Merge the first path-image feature-text feature pair and the second path-image feature-text feature pair to obtain a path-image feature-text feature pair.

[0077] For example, ViT-B / 16 is a variant of the Vision Transformer, where "B" stands for "Base", indicating the basic version, and "16" means that the input image is divided into 16x16 patches. ViT (Vision Transformer) is a model based on the Transformer architecture, which can treat image processing problems as sequence processing problems. The natural language processing model based on the Transformer model (Bidirectional Encoder Representations from Transformers, BERT) is a pre-trained model in the field of natural language processing. By learning language patterns in large amounts of text data, the model can provide high-quality text feature representations for downstream natural language processing tasks.

[0078] Optionally, the pre-trained visual feature encoder in this embodiment may also be ViT-H / 14, etc., which is not specifically limited here.

[0079] Optionally, the pre-trained language feature encoder in this embodiment may also be a multi-layer Transformer, which is not specifically limited here.

[0080] For example, ViT-H / 14 is a variant of the Vision Transformer, where "H" stands for "Huge", indicating that this is a larger version of the model with more parameters than the basic version (such as ViT-B) in order to achieve better performance. "14" means that the image is cut into 14x14 patches for processing.

[0081] S3. Use a dynamic processing strategy based on equality judgment to dynamically merge the path-image feature-text feature pair to obtain a merged path-image feature-text feature pair.

[0082] In this embodiment, a dynamic processing strategy based on equality judgment is used to dynamically merge the path-image feature-text feature pair, specifically including:

[0083] Using a dynamic processing strategy based on equality judgment, the text features in the path-image feature-text feature pair are traversed to determine whether adjacent text features are the same. If so, the path-image feature-text feature pairs corresponding to the adjacent text features are merged. If not, the path-image feature-text feature pairs are continued to be traversed.

[0084] S4. Construct a training dataset for training the projection layer based on visual observations and the merged path-image feature-text feature pairs.

[0085] In this embodiment, step S4 specifically includes:

[0086] S41, inputting the merged path-image feature-text feature pair into the coarse-scale encoding module of the DUET model for scene graph encoding, and outputting the path-scene graph encoding embedding-text feature pair through the DUET model;

[0087] S42, using the image description model to process the visual observation to obtain a natural language description corresponding to the visual observation;

[0088] S43, designing a natural language generation template, and using the natural language generation template to train a large language model;

[0089] S44, inputting the path-scene graph encoding embedding-text feature pair and the natural language description into the trained large language model, and outputting the structured natural language description through the trained large language model;

[0090] S45. Input the structured natural language description into the language encoding module, and output the scene graph encoding embedding-natural language encoding embedding through the language encoding module as a training data set for training the projection layer.

[0091] For example, the DUET (Dual-scale Graph Transformer) model achieves long-range action planning and fine-grained cross-modal understanding in the visual language navigation (VLN) task through a dual-scale graph transformer architecture. The DUET model uses the graph structure constructed by the topological graph during navigation for two-step training: first pre-training on four auxiliary tasks, and then fine-tuning on the VLN task. The purpose is to improve navigation performance by paying attention to both global and local information when processing navigation instructions. The DUET model has dual-scale spatial visual representation capabilities through its dual-scale graph transformer architecture. During the navigation process, the DUET model uses the topological graph structure to capture the global and local features of the environment. This approach enables the model to better understand and utilize the structural information in the navigation environment.

[0092] Optionally, in step S42, the image description model may be a multimodal model such as BLIP, BLIP2, InstructBLIP, and LLaVA, which is not specifically limited herein.

[0093] For example, BLIP is a pre-trained model that combines images and text, focusing on image description and cross-modal understanding tasks. It is pre-trained on large-scale image-text pairing data through self-supervised learning methods. BLIP uses two main modules: a visual encoder (such as ViT) to process images, and a language model to process text. Through cross-modal alignment and self-supervised learning, BLIP can efficiently understand image content and generate natural language descriptions. Its training strategy enables the model to cope with a variety of downstream tasks, such as image description generation, visual question answering, and image-text retrieval.

[0094] BLIP-2 is an improved version of BLIP, designed to improve the model's cross-modal reasoning capabilities. Compared to BLIP, BLIP-2 introduces a stronger visual-language alignment mechanism and further optimizes the interaction between the visual encoder and the language model. By combining a pre-trained visual encoder and a large-scale pre-trained language model, BLIP-2 is able to better capture the deep semantic associations between images and text. This makes it perform well in generating more precise image descriptions and coping with complex visual question-answering tasks.

[0095] InstructBLIP is a multimodal model specifically designed to generate instruction-driven outputs combining images and text. Unlike BLIP and BLIP-2, which are mainly used for image description, InstructBLIP combines image encoders and language models to generate text output that meets the requirements based on the instructions entered by the user. The core of the model is the use of a module called Q-former, which is used to extract key information from the image and input it into the language model together with the text instructions. This makes InstructBLIP particularly suitable for tasks that require generating complex instruction responses based on images, such as visual question answering, complex image description, and multimodal reasoning tasks.

[0096] LLaVA is a multimodal assistant that combines large language models (such as GPT) and visual models, aiming to provide an efficient combination of visual understanding and natural language generation. LLaVA uses a pre-trained visual encoder to process images, while fusing image features with a powerful language model to generate detailed image descriptions or answer image-related questions. LLaVA is designed to provide capabilities similar to human assistants, not only describing images, but also interacting and reasoning with users through natural language. It is suitable for a variety of complex multimodal tasks, such as visual dialogue and image generation guidance.

[0097] Optionally, the large language model in this embodiment may be Vicuna-7B, Vicuna-15B, Llama, ChatGPT, etc., which is not specifically limited here.

[0098] Exemplarily, the Vicuna model is an advanced large language model built on the latest natural language processing technology, designed to provide human-level conversational understanding and generation capabilities. It uses large-scale pre-training and fine-tuning techniques to learn language patterns and contextual associations from large amounts of conversational data. Vicuna-7B and Vicuna-15B can be understood as Vicuna models with 7 billion parameters and 15 billion parameters, respectively. LLaMA (Leveraging LAnguage Models for Alignment) is a high-performance language model. ChatGPT is an artificial intelligence technology-driven language model designed for conversational interactions that can provide human-like conversational responses.

[0099] S5. Use the training dataset to train the projection layer, and use the model fine-tuning method to fine-tune the large language model based on the existing visual language navigation task dataset.

[0100] In this embodiment, step S5 specifically includes:

[0101] S51, using a randomly initialized projection layer to connect the global space visual representation module of the DUET model with the semantic space of the trained large language model, using a Q-former initialized by InstructBLIP to connect the local space visual representation module of the DUET model with the semantic space of the trained large language model to form a projection layer, and using the training data set to train the projection layer to obtain a trained projection layer;

[0102] S52, freezing the parameters of the visual representation module and the trained projection layer of the DUET model, and fine-tuning the trained large language model using a model fine-tuning method based on an existing visual language navigation task dataset;

[0103] Among them, the visual representation module of the DUET model includes a global space visual representation module and a local space visual representation module.

[0104] For example, the InstructBLIP model is a visual language model trained by fine-tuning instructions based on the BLIP-2 model. It combines the techniques of natural language processing (NLP) and computer vision (CV) to process and understand the association between images and text. InstructBLIP introduces instruction tuning technology, which enables the model to understand and follow natural language instructions, making it more flexible and accurate in processing multimodal tasks.

[0105] Optionally, in step S52, the model fine-tuning method includes LoRA, QLoRA, Adapter-tuning and Prefix-tuning.

[0106] For example, Low-Rank Adaptation (LoRA) is a low-rank adjustment method that achieves fine-tuning by superimposing a low-rank matrix on the weight matrix of a large model instead of directly updating the original weights. This greatly reduces the storage space complexity and computational cost required for fine-tuning.

[0107] Quantized Low Rank Adaptation (QLoRA) is an extension of LoRA. It further compresses the low-rank matrix through quantization technology, reduces memory usage and computing requirements, and enables the model to run efficiently on devices with limited resources.

[0108] Adapter-Tuning methods insert small trainable adapter modules into the model. By optimizing these adapters individually, task-specific tuning can be performed without affecting the main model.

[0109] Prefix-tuning technology guides the model to generate task-specific outputs by optimizing only the prefix of the model input sequence (i.e., a learnable vector sequence added before the actual input) without updating the weights of the model body, thus reducing the burden of fine-tuning.

[0110] Optionally, in this embodiment, the projection layer can be a linear layer, a Q-former, a multi-layer Transformer, a convolutional layer, etc.

[0111] For example, Q-former is an improved Transformer model specially designed to bridge the visual image encoder (such as the visual part of BLIP) and the language model. It extracts key information related to the instruction from the image encoder through the query mechanism (QueryMechanism), and inputs this information into the language model together with the text instruction to generate an output that conforms to the context. The advantage of Q-former is that it efficiently integrates image and text information, ensuring that the model can understand complex multimodal instructions, thereby improving the performance of tasks such as visual question answering and image description.

[0112] S6. Use the visual space representation module of the DUET model to process the merged path-image feature-text feature pair to obtain a path-image feature embedding-text feature embedding pair, and use the trained projection layer to project the path-image feature embedding-text feature embedding pair to the semantic space of the fine-tuned large language model, and output the path-image feature embedding-text feature embedding-instruction description pair through the fine-tuned large language model.

[0113] Optionally, the large language model in this embodiment may be Vicuna-7B, Vicuna-15B, Llama, ChatGPT, etc., which is not specifically limited here.

[0114] like Figure 2 As shown, Figure 2 The following is a model architecture diagram of a data augmentation method for UAV visual language navigation tasks. Figure 2 In the left part of the model architecture, the panoramic image encoding and topology map construction are used. The first image feature after ViT-B / 16 encoding is sent to the dynamic processing module for dynamic merging. The merged path-image feature-text feature pair is sent to the topology map construction part and the fine-grained encoding module (the fine-grained encoding module, i.e. Figure 2 The right half of the model is the dual-scale visual encoding and large language model of the DUET model. The encoded and dynamically processed path-image feature-text feature pairs are sent to the coarse-scale encoding module (coarse-scale encoding module, i.e. Figure 2The global coarse-scale encoder in ) and the fine-grained encoding module (the fine-grained encoding module, i.e. Figure 2 The local fine-scale encoder in the neural network performs cross-attention calculations with node features and image features, and then generates DUET-encoded node embeddings and panoramic image embeddings through self-attention and feedforward networks. The node embeddings are projected into the semantic space of the large language model through a trainable projection layer, and the panoramic image embeddings are projected into the semantic space of the large language model through the pre-trained Q-former. Finally, the large language model generates path-image feature embedding-text feature embedding-instruction description pairs (i.e., navigation instructions).

[0115] S7. Use language evaluation indicators to filter the path-image feature embedding-text feature embedding-instruction description pairs to obtain augmented data.

[0116] In this embodiment, in step S7, the language evaluation indicators include BLEU, METEOR, ROUGE, CIDEr, SPICE and NIST.

[0117] For example, the Bilingual Evaluation Understudy (BLEU) evaluates translation quality by comparing the overlap between machine-generated text and manual reference text. The Metric for Evaluation of Translation with Explicit Ordering (METEOR) takes into account word order, part-of-speech matching, and exact and fuzzy matching, and can therefore evaluate translation quality in a more detailed manner. The Recall-Oriented Understudy for Gisting Evaluation (ROUGE) can be used for quality assessment of automatic summarization tasks, measuring the degree of overlap between system-generated summaries and reference summaries. The Consensus-Based Image Description Evaluation (CIDEr) evaluates by calculating the mean similarity between candidate descriptions and manual descriptions written by multiple people. The Semantic Propositional Image Caption Evaluation (SPICE) focuses on evaluating the semantic accuracy of generated content for image caption generation tasks, measured by comparing the structural consistency of scene graphs. NIST (National Institute of Standards and Technology) is an evaluation system that uses different weighting methods to screen data.

[0118] In the present invention, the DUET model performs extremely well in visual language navigation tasks due to its dual-scale visual representation ability, and the large language model is well-known for its ability to learn knowledge from a large corpus and achieve state-of-the-art performance in various NLP tasks. Therefore, the present invention constructs an instruction generation architecture, which utilizes the dual-scale spatial visual representation ability of the DUET model and the contextual learning and reasoning ability of the large language model, and organically integrates the advantages of the two through hierarchical model construction, thereby overcoming the defects of existing methods such as strong data dependence, poor generalization ability, and low instruction quality. As the best performing model in the current VLN task, the DUET model's dual-scale spatial visual representation capability prevents its performance from degrading as the path sequence becomes longer. This determines that the DUET model can provide the large language model with a spatial visual representation that matches the fine-grained path. As a fairly well-developed large language model, it can efficiently implement contextual learning and reasoning. The spatial visual representation extracted by the DUET model can generate rich, diverse, and high-quality natural language texts, and its performance will not degrade as the path sequence becomes longer. Therefore, it can ensure the high quality of the augmented data and also help solve the "long flat" problem in the AerialVLN task.

[0119] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A data augmentation method for unmanned aerial vehicle visual language navigation tasks, characterized in that: The method comprises: S1, obtaining a navigation map, searching the navigation map using a heuristic search algorithm to obtain a path and an action sequence, calling a simulator to generate a visual observation corresponding to the path, and determining a path-action sequence-visual observation pair according to the path, the action sequence and the visual observation; wherein each of the paths comprises at least two path points; S2, using a pre-trained visual feature encoder and a pre-trained language feature encoder to pre-process the existing visual language navigation task dataset and the path-action sequence-visual observation pair respectively to obtain a path-image feature-text feature pair; The existing visual language navigation task dataset includes at least one path-action sequence-visual observation-instruction description pair data; S3, dynamically merging the path-image feature-text feature pair using a dynamic processing strategy based on equality judgment to obtain a merged path-image feature-text feature pair; S4, constructing a training data set for training a projection layer according to the visual observation and the merged path-image feature-text feature pair; The step S4 specifically includes: S41, inputting the merged path-image feature-text feature pair into the coarse-scale encoding module of the DUET model for scene graph encoding, and outputting the path-scene graph encoding embedding-text feature pair through the DUET model; S42, using an image description model to process the visual observation to obtain a natural language description corresponding to the visual observation; S43, designing a natural language generation template, and using the natural language generation template to train a large language model; S44, inputting the path-scene graph encoding embedding-text feature pair and the natural language description into a trained large language model, and outputting a structured natural language description through the trained large language model; S45, inputting the structured natural language description into a language encoding module, and outputting the scene graph encoding embedding-natural language encoding embedding through the language encoding module as the training data set for training the projection layer; S5. Using the training data set to train the projection layer, and using a model fine-tuning method to fine-tune the large language model based on the existing visual language navigation task data set; The step S5 specifically includes: S51, using a randomly initialized projection layer to connect the global space visual representation module of the DUET model with the semantic space of the trained large language model, using a pre-trained Q-former to connect the local space visual representation module of the DUET model with the semantic space of the trained large language model to form the projection layer, and using the training data set to train the projection layer to obtain a trained projection layer; S52, freezing the parameters of the visual representation module and the trained projection layer of the DUET model, and fine-tuning the trained large language model using the model fine-tuning method based on the existing visual language navigation task dataset; Wherein, the visual representation module of the DUET model includes a global space visual representation module and a local space visual representation module; S6, using the visual space representation module of the DUET model to process the merged path-image feature-text feature pair to obtain a path-image feature embedding-text feature embedding pair, and using the trained projection layer to project the path-image feature embedding-text feature embedding pair to the semantic space of the fine-tuned large language model, and outputting the path-image feature embedding-text feature embedding-instruction description pair through the fine-tuned large language model; S7. Use language evaluation indicators to screen the path-image feature embedding-text feature embedding-instruction description pairs to obtain augmented data.

2. The method according to claim 1, characterized in that In step S2, the pre-trained visual feature encoder is ViT-B / 16, and the pre-trained language feature encoder is BERT; The step S2 specifically includes: S21, using the ViT-B / 16 to extract features from the visual observation in the path-action sequence-visual observation-instruction description pair to obtain a first image feature; S22, using the BERT to extract features from the action sequence and the instruction description in the path-action sequence-visual observation-instruction description pair to obtain a first text feature; S23, determining a first path-image feature-text feature pair according to the path-action sequence-visual observation-instruction description pair, the first image feature and the first text feature; S24, using the ViT-B / 16 to extract features from the visual observations in the path-action sequence-visual observation pair to obtain a second image feature; S25, using the BERT to extract features of the path-action sequence-visual observation pair to obtain a second text feature; S26, obtaining a second path-image feature-text feature pair according to the path-action sequence-visual observation pair, the second image feature and the second text feature; S27: Merge the first path-image feature-text feature pair and the second path-image feature-text feature pair to obtain the path-image feature-text feature pair.

3. The method according to claim 1, characterized in that In step S3, the dynamic processing strategy based on equality judgment is used to dynamically merge the path-image feature-text feature pair, specifically including: Using the dynamic processing strategy based on equality judgment, traverse the text features in the path-image feature-text feature pair to determine whether adjacent text features are the same. If so, merge the path-image feature-text feature pairs corresponding to the adjacent text features. If not, continue to traverse the path-image feature-text feature pairs.

4. The method according to claim 1, characterized in that: In step S7, the language evaluation indicators include BLEU, METEOR, ROUGE, CIDEr, SPICE and NIST.

5. The method according to claim 1, characterized in that In step S52, the model fine-tuning method includes LoRA, QLoRA, Adapter-tuning and Prefix-tuning.

Images