A visual language navigation method and system based on generative data driven learning
By rewriting visual observation and rewriting of trajectory-instruction pairs, using large language models and literary graph generation models, a diverse enhanced data set is generated, which solves the scarcity and noise of visual language navigation data, improves navigation performance and Reduces processing costs and is suitable for unknown scenarios.
Patent Information
- Application Number
- CN202410897028.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-05
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-07-05
AI Technical Summary
Due to the scarcity of data, the existing visual language navigation technology has insufficient generalization capabilities in unknown environments. The existing data augmentation methods have noise problems and high labor costs, which cannot effectively improve the visual language navigation performance.
A visual language navigation method based on generated data-driven learning is proposed. By rewriting visual observation and rewriting trajectory-instruction pairs, a new trajectory visual observation image and new sequence observation description information are generated, combining large language models and literary students. Graph generation models, creating diverse augmented datasets for training visual language navigation models.
It effectively improves the visual language navigation performance, can be applied to unknown scenarios, reduces the cost of unlabeled data processing, and generates a large amount of data information, similar to human instructions, and is suitable for diversified environments.
Smart Images

Figure CN118424301B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of deep learning and computer vision, and in particular to a visual language navigation method and system based on generated data driven learning. Background Art
[0002] Vision-Language Navigation (VLN) is a core task in the field of embodied intelligence. Its research focuses on an embodied robotic agent that can receive natural language instructions provided by humans and navigate in a complex 3D visual environment. In order to successfully navigate to the target location, a vision-language navigation agent needs to have the ability to understand the dynamic information of the environment and accurately align cross-modal information in serialized instruction information and observation information.
[0003] In recent years, in visual-language navigation tasks, the extreme scarcity of real data (i.e., high-quality visual-language navigation data with manual annotations) has greatly restricted the effective learning of cross-modal alignment knowledge and navigation skills by intelligent agents, resulting in the obvious lack of generalization ability of existing intelligent agents in unknown environments.
[0004] In order to solve the data scarcity problem in the field of visual language navigation research, many scientists have studied various data enhancement strategies. For example, traditional methods use speaker-follower models, environment dropout schemes, or confusion strategies of known environments to synthesize new instructions and environments. However, the enhanced data of these works still comes from the original restricted 3D environment in the Matterport3D dataset. Some works even try to introduce image / video data from the Internet or data from other simulators to further enhance the diversity of the environment. However, these working methods still have two disadvantages. On the one hand, the data on the Internet usually contains a lot of noise, which may seriously hinder the training of navigation and lead to poor visual language navigation performance. On the other hand, simply stacking the available unlabeled data may still be limited to the data style of a specific simulation environment and may not be extended to various other environments and spatial layouts. It is also worth noting that the process of processing such a large amount of unlabeled data may be very labor-intensive and time-consuming. Summary of the invention
[0005] In order to solve the problems of poor visual language navigation performance, inability to be applied to unknown scenarios and high cost of processing unlabeled data in the above-mentioned prior art, the present invention proposes a visual language navigation method and system based on generated data-driven learning, which is not limited to visual language navigation data of a specific field obtained from a limited simulator environment, effectively improves the visual language navigation performance, can be applied to unknown scenarios, and reduces the cost of processing unlabeled data.
[0006] A visual language navigation method based on generative data driven learning, comprising the following steps:
[0007] S1. Acquire a trajectory-command pair, wherein the trajectory-command pair includes an original trajectory and an original command;
[0008] S2. rewrite the original trajectory by visual observation to generate a new trajectory visual observation image;
[0009] S3. rewrite the original instruction and the new trajectory visual observation image to generate original alignment sequence landmark information and new sequence observation description information respectively;
[0010] S4. Preprocessing the original alignment sequence landmark information and the new sequence observation description information to generate new trajectory pairing instructions;
[0011] S5. Mix the new trajectory pairing instruction and the trajectory-instruction pair to obtain a mixed data set, and use the mixed data set to train a preset visual language navigation model to obtain a trained visual language navigation model, which is used to perform the visual language navigation task.
[0012] Preferably, the visual observation and rewriting of the original trajectory comprises:
[0013] S201. Collecting panoramic description texts during different navigation time steps in the original trajectory;
[0014] S202. Rewriting the panoramic description text based on the coexistence knowledge using the large language model to obtain a rewritten description text;
[0015] S203. Execute a panorama-to-view generation strategy on the rewritten description text to obtain the new trajectory visual observation image.
[0016] Preferably, the collecting of panoramic description texts during different navigation time steps in the original trajectory includes:
[0017] Define the original trajectory t The panoramic image at the time step is , the original trajectory t Panoramic image at time step Input the preset visual language model and output different navigations in the original trajectory t Panorama description text during the time step as follows:
[0018]
[0019] in, Represents a visual language model.
[0020] Preferably, the step of rewriting the panoramic description text based on the co-existence knowledge by using the large language model comprises:
[0021] Get the description text of the panorama and description rewrite hints , the panorama description text and the description rewrites the prompt word The large language model is input, and the added co-existence and the rewritten description text corresponding to the co-existence are output. The calculation expression of the rewritten description text is as follows:
[0022]
[0023] in, Indicates rewriting the description text. N Indicates the number of rewritten description texts. Represents a large language model.
[0024] Preferably, the panorama-to-view generation strategy includes:
[0025] Rewrite the description text Input into the preset Wensheng image generation model and output N new panoramic images ;
[0026] Each new panoramic image Perform panorama discretization to obtain a new trajectory visual observation image, which is a single view image , The calculation expression is as follows:
[0027]
[0028] in, Equirec2Perspec represents the synthesis of a series of coordinate transformation functions, represents the camera's projection matrix, Represents the rotation matrix of the camera.
[0029] Preferably, the original instruction is rewritten, including:
[0030] S301. Extract the landmark information list in the original instruction ,in is the size of the landmark information list, List of landmark information The k elements;
[0031] S302. Use the landmark information list as the input of the pre-trained visual language model and output the original trajectory t The first benchmark action at the time step and Best matching landmarks , The calculation expression is as follows:
[0032]
[0033] in, represents the similarity calculation function, Indicates that a similarity calculation function has been found Get the maximum value of the parameter. A text encoder representing a visual language model, An image encoder representing a visual language model;
[0034] S303. Collect each of the original trajectories Collect the best matching landmarks , get the original aligned sequence landmark information , The number of navigation steps.
[0035] Preferably, rewriting the new trajectory visual observation image with instructions comprises:
[0036] S311. Extracting the second benchmark action from the generated new trajectory visual observation image ;
[0037] S312. The second reference action Describe and get the new observation description text ,
[0038] S313. Collect the new observation description text , get the new sequence observation description information .
[0039] Preferably, preprocessing the original aligned sequence landmark information and the new sequence observation description information comprises: and the new sequence observation description information Input to the large language model, prompting the large language model to compare the landmark information of the original alignment sequence and the new sequence observation description information , output new trajectory pairing instructions.
[0040] Preferably, the navigation loss function is used Train the preset visual language navigation model and the navigation loss function The calculation expression is as follows:
[0041]
[0042] in, represents the navigation agent, represents the original trajectory-command pair, Represents the original instruction, Indicates the rewritten new track pairing instructions, Indicates a rewrite instruction.
[0043] The present invention also proposes a visual language navigation system based on generated data driven learning, the system comprising:
[0044] An acquisition module, used for acquiring a trajectory-instruction pair, wherein the trajectory-instruction pair includes an original trajectory and an original instruction;
[0045] A visual observation rewriting module, used for rewriting the original trajectory by visual observation to generate a new trajectory visual observation image;
[0046] An instruction rewriting module, used for rewriting the original instruction and the new trajectory visual observation image to generate original alignment sequence landmark information and new sequence observation description information respectively;
[0047] A preprocessing module, used to preprocess the original alignment sequence landmark information and the new sequence observation description information to generate new trajectory pairing instructions;
[0048] The visual language navigation module is used to mix the new trajectory pairing instruction and the trajectory-instruction pair to obtain a mixed data set, and use the mixed data set to train a preset visual language navigation model to obtain a trained visual language navigation model, and the visual language navigation model is used to perform the visual language navigation task.
[0049] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:
[0050] The present invention proposes a visual language navigation method and system based on generative data driven learning. Firstly, for manually annotated trajectory-instruction pairs, the trajectory-instruction pairs are rewritten based on two key aspects, namely observation rewriting and instruction rewriting. On the one hand, the original trajectory is visually observed and rewritten to obtain a new trajectory visual observation image; on the other hand, the original instruction and the new trajectory visual observation image are instruction rewritten to generate original alignment sequence landmark information and new sequence observation description information; then, new trajectory pairing instructions are generated by preprocessing the original alignment sequence landmark information and the new sequence observation description information; after obtaining the new trajectory pairing instructions, the new trajectory pairing instructions are mixed with the trajectory-instruction pair to promote the training of the visual language navigation model. The present invention proposes a new cross-modal data augmentation framework for visual language navigation, which has excellent visual language navigation performance on benchmark datasets of multiple visual language navigation tasks. The framework can create different enhanced data simply by rewriting available manually annotated data. It is not limited to visual language navigation data in specific fields obtained from limited simulator environments. It reduces the processing cost of unlabeled data, effectively improves the visual language navigation performance, can be applied to unknown scenarios, and provides a meaningful reference for how to make full use of existing manually annotated data. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] Figure 1 A flow chart showing a visual language navigation method based on generated data driven learning proposed in an embodiment of the present invention;
[0052] Figure 2 A new cross-modal data enhancement framework diagram proposed in an embodiment of the present invention is shown;
[0053] Figure 3 A flow chart showing a method for data enhancement from an original / additional simulator according to an embodiment of the present invention;
[0054] Figure 4 A flow chart showing a method for enhancing data from network images or videos proposed in an embodiment of the present invention;
[0055] Figure 5 A flow chart of a VLN-Rewriter cross-modal data enhancement method proposed in an embodiment of the present invention is shown;
[0056] Figure 6 A visualization diagram showing data generated by the VLN-Rewriter proposed in an embodiment of the present invention;
[0057] Figure 7 A visualization diagram showing the rewriting of the panoramic description text and the corresponding generation of the panoramic image proposed in the embodiment of the present invention;
[0058] Figure 8A visual comparison diagram showing the instructions generated by the Speaker model and the VLN-Rewriter proposed in the embodiment of the present invention;
[0059] Fig. 9 A block diagram of a visual language navigation system based on generated data driven learning proposed in an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0060] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;
[0061] It is understandable to those skilled in the art that some well-known contents may be omitted in the drawings;
[0062] In order to facilitate understanding of this embodiment, first, the prior art information of this embodiment is introduced as follows:
[0063] Vision-and-Language Navigation (VLN) is a challenging task that requires an intelligent agent to navigate to a target location according to human instructions. As a core task in the field of embodied intelligence, VLN has attracted extensive research interest from researchers in recent years. Early methods used Seq2Seq architecture to build VLN models with data augmentation strategies, or constructed cross-modal alignment modules and effective training mechanisms. Inspired by the great success of vision-language pre-training, recent methods use Transformer-based architectures and design domain-specific proxy tasks to improve the performance of VLN models. For example, the method "Towards Learning a Generic Agent for Vision-and-Language Navigation via Pretraining" constructs large-scale image-text-action triplets to pre-train the agent; the method "VLN BERT: A Recurrent Vision-and-Language BERTfor Navigation" enables the agent to recognize time-related inputs by introducing a loop function; the method "HistoryAware Multimodal Transformer for Vision-and-Language Navigation" constructs a history-aware multimodal Transformer architecture for navigation history encoding with long viewing distances; the method "Think Global, ActLocal: Dual-scale Graph Transformer for Visionand-Language Navigation" introduces a dual-scale graph Transformer to perform joint long-term action planning and fine-grained cross-modal understanding; the method "Bevbert: Multimodal map pre-training for language-guided navigation" develops a graph-based pre-training paradigm to enhance the agent's spatial understanding ability. However, most existing methods rely on domain-specific vision-language navigation data obtained from limited simulator environments, resulting in poor vision-language navigation performance and inability to be applied to unknown scenarios;
[0064] The present invention proposes a new rewriting-based cross-modal data augmentation paradigm to help visual language navigation agents get rid of limited simulation environments and make full use of limited human annotated data. Driven by the large language model and the text graph generation model, the method proposed in the embodiment of the present invention can effectively integrate various information-rich data, thereby improving the performance of visual language navigation.
[0065] For data augmentation of vision-and-language navigation, data scarcity is a long-standing problem in the field of vision-and-language navigation. In addition to the tedious simulation of real environments, collecting large-scale manually annotated instructions is also a very labor-intensive task. To alleviate the problem of data scarcity, early works used environment augmentation strategies to sample new trajectories in simulator environments and train speaker models to obtain paired instructions. However, these methods are limited by the limited diversity of the original simulator environment and the unsatisfactory quality of augmented instruction data. Another research direction of data augmentation methods is to introduce image data or video data from the Internet to increase the diversity of training data. However, the noise of these network resources brings a great burden to data cleaning and also causes certain harm to training. Some recent works have attempted to introduce large-scale unannotated environments from additional simulators. For example, the "Scaling data generation in vision-and-language navigation" method applies real environments from the HM3D dataset and the Gibson dataset and synthesizes 4.9 million trajectory-instruction pairs for training. However, obtaining large-scale simulated environments is not practical in real-world applications, and it is also time-consuming to annotate such large-scale data. A recent work, “Panogen: Text-conditioned panoramic environment generation for vision-and-language navigation”, adopts a text-to-image generation model to synthesize viewpoint images and designs an image stitching scheme to convert viewpoint images into panoramic images.
[0066] Different from the above work, the embodiment of the present invention proposes a new paradigm of VLN cross-modal data enhancement, called VLN-Rewriter. VLN-Rewriter attempts for the first time to release the rich world knowledge and powerful language ability of the large language model by rewriting the human-annotated trajectory instruction data, and performs data enhancement in the visual language navigation task. In addition, unlike the "view to panorama" scheme in the "Panogen: Text-conditioned panoramic environment generation for vision-and-language navigation" method, the present invention application develops a reverse "panorama to view" generation strategy, which effectively ensures efficiency and consistency between adjacent views.
[0067] For robotic data generation driven by large language models, since large language models have rich common sense knowledge and powerful reasoning capabilities, some recent studies have shown that they can help generate customized high-quality data to alleviate the problem of data scarcity in robotic tasks. For example, "Robotic skill acquisition via instruction augmentation with vision-language models" uses GPT-3 to perform instruction augmentation by generating rephrased instructions under the guidance of world knowledge; "GenSim: Generating Robotic Simulation Tasks via Large Language Models" enables GPT-4 to generate task courses, enriching the existing benchmark dataset tenfold; "Holodeck: Language Guided Generation of 3D Embodied AI Environments" uses GPT-4 to generate spatial relationship constraints between objects to create diverse scenes; "Robogen: Towards unleashing infinite data for automated robot learning via generative simulation" uses GPT-4 to generate task proposals and scene configurations to achieve generative simulation; "EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agents" creates new training environments by prompting GPT-4 to generate a set of environment configurations that can be parsed by the simulator.
[0068] In view of the problems existing in the above-mentioned prior art, a visual language navigation method and system based on generated data-driven learning is proposed in the embodiment of the present invention, which utilizes the object coexistence knowledge, reasoning ability and language skills of the large language model to generate the environment and instructions for the VLN task. Therefore, the synthetic environment will have reasonable and diverse objects and conform to the scenes of the real world. In addition, compared with the instructions generated by the traditional Speaker model, the synthetic instruction data obtained in the embodiment of the present invention will be more human-like and have a larger amount of information. The technical solution of the present invention is further explained below in conjunction with the accompanying drawings and embodiments.
[0069] Example 1
[0070] like Figure 1 and Figure 2 As shown, this embodiment proposes a visual language navigation method based on generated data driven learning, including the following steps:
[0071] S1. Acquire a trajectory-command pair, wherein the trajectory-command pair includes an original trajectory and an original command;
[0072] S2. rewrite the original trajectory by visual observation to generate a new trajectory visual observation image;
[0073] In S2, many studies on visual language navigation have explored how to generate or introduce more environments to increase the diversity of visual observations. These methods edit existing environments by mixing environments or changing room styles and object appearances, or generate panoramic images by recursive drawing based on redundant single-view text descriptions. However, these methods are still limited by the Matterport3D environment. In addition, they may not essentially change the spatial layout between different objects, nor introduce objects with new categories. In an embodiment of the present invention, the object co-occurrence knowledge of a large language model and a Vincent graph generation model is used to perform observation rewriting on existing trajectory-instruction pairs, so that the generated observation image introduces a diverse spatial layout and enriches the object categories. In specific scenarios, this method effectively alleviates the problem of scarce diversity in the original simulator environment. The visual observation rewriting of the original trajectory includes:
[0074] S201. Collecting panoramic description texts during different navigation time steps in the original trajectory;
[0075] In S201, the collecting of panoramic description texts during different navigation time steps in the original trajectory includes:
[0076] Define the original trajectory t The panoramic image at the time step is , the original trajectory t Panoramic image at time step Input the preset visual language model and output different navigations in the original trajectory t Panorama description text during the time step as follows:
[0077]
[0078] in, Represents a visual language model.
[0079] Since panoramic images may contain rich object information, the preset visual language model used is the visual language model Tag2Text. The visual language model Tag2Text is an efficient visual language model with tag guidance, which is used to obtain description text. The visual language model Tag2Text can integrate the recognized image tags (i.e., object tags) as guiding elements into text generation, thereby generating more informative panoramic description text.
[0080] S202. Rewriting the panoramic description text based on the coexistence knowledge using the large language model to obtain a rewritten description text;
[0081] In S202, benefiting from training on a super-large corpus, large language models such as GPT-3 and Chat-GPT have rich knowledge of the physical world. Since a specific scene may contain a variety of possible co-existing objects, a large language model is used to obtain rewritten description texts that are based on the original panoramic description texts and clearly indicate different co-existing objects. Therefore, this step can expand the original scene to various possible situations to increase the diversity of training data.
[0082] The step of rewriting the panoramic description text based on the co-existence knowledge by using the large language model includes:
[0083] Get the panorama description text and description rewrite hints , the panorama description text and the description rewrites the prompt word The large language model is input, and the added co-existence and the rewritten description text corresponding to the co-existence are output. The calculation expression of the rewritten description text is as follows:
[0084]
[0085] in, Indicates rewriting the description text. N Indicates the number of rewritten description texts. Represents a large language model. In order to constrain the large language model to explicitly indicate the co-existing object in the rewritten description text it generates, the large language model is required to generate the added co-existing object and the corresponding rewritten description text. The task example of rewriting the description text is as follows:
[0086] Caption: a bathroom that has mirrors on the walls
[0087] Added objects: vanity, sink, faucet
[0088] Rewritten captions: A spacious bathroom featuring a large vanity with a marble countertop and double sinks, reflecting off the expansive mirrors on the walls, complemented by chrome faucets and a tiled backsplash.
[0089] The corresponding Chinese meaning is:
[0090] Description: A bathroom with mirrors on the walls
[0091] Added coexisting objects: Dresser, Sink, Faucet
[0092] Rewritten description text: A spacious bathroom with a large vanity, marble countertop and double sinks that reflect off the expansive mirror on the wall, complemented by chrome faucets and tile backsplash.
[0093] In addition, in order to encourage observations based on rewriting description text generation to indicate different salient objects, which may further enhance diversity, the large language model is required to explicitly output the representation of the original description to highlight different objects in the prompt words. The prompt words of the rewritten description text are as follows:
[0094] Please generate 3 rewritten captions for the caption I provide to you. The principle is
[0095] 1) You should rewrite the sentence pattern of caption,
[0096] 2) You should add the possible objects that may exist in the scenedescribed in the given caption.
[0097] The sentence pattern and added objects should not be the same among 3rewritten captions. You have to make sure that the added objects can be seen in the indoor environment of a building.
[0098] Your output format should be: (You must output 3 times in the following format)
[0099] Added objects: XXX. (at least 3 new objects)
[0100] Rewritten captions: XXX. (20 to 40 words)
[0101] Here is an example.
[0102] Caption: a bathroom that has mirrors on the walls
[0103] Added objects: vanity, sink, faucet
[0104] Rewritten captions: <1> A spacious bathroom featuring a large vanity with a marble countertop and double sinks, reflecting off the expansive mirrors on the walls, complemented by chrome faucets and a tiled backsplash.
[0105] The corresponding Chinese meaning is: Please generate 3 rewritten description texts based on the description text I provide you. The principle of generation is
[0106] 1) You should rewrite the sentence structure of the description text,
[0107] 2) You should add the co-existent objects that may exist in the scene described by the given description text.
[0108] In the 3 rewritten description texts, the sentence structure and the added coexisting objects should not be the same. You must make sure that the added coexisting objects can be seen in the interior environment of the building.
[0109] Your output should be in the following format: (You must output the following format 3 times)
[0110] Objects added: XXX. (There are at least 3 new objects)
[0111] Rewritten description text: XXX. (20 to 40 characters)
[0112] Here is an example.
[0113] Description: A bathroom with mirrors on the walls
[0114] Objects added: Dresser, Sink, Faucet
[0115] Rewritten description text: <1> A spacious bathroom with a large vanity with marble countertop and double sinks reflected from the spacious mirror on the wall, paired with a chrome faucet and tile backsplash.
[0116] S203. Execute a panorama-to-view generation strategy on the rewritten description text to obtain the new trajectory visual observation image;
[0117] It should be noted that for visual observation rewriting, panoramic description texts are collected and integrated into the input to the large language model to generate rewritten description texts based on co-existence knowledge. These rewritten description texts are used to obtain new trajectory visual observation images through the text graph generation model.
[0118] In S203, for the panoramic generation step, the MultiDiffusion model is selected as the Wensheng image generation model. The MultiDiffusion model can generate images with the required aspect ratio and information content to generate new panoramic images. By generating new panoramic images, the perspective consistency between adjacent single-view observation images can be effectively guaranteed.
[0119] The panorama-to-view generation strategy includes
[0120] Rewrite the description text Input into the preset Wensheng image generation model and output N new panoramic images ;
[0121] Each new panoramic image Perform panorama discretization to obtain a new trajectory visual observation image, which is a single view image , for single view image generation , the new panoramic image is discretized into 36 perspective images using the Equirec2Perspec algorithm. This algorithm first uses the camera parameters, including field of view (FOV), heading ( ) and elevation angle ( ), to calculate the camera projection matrix (denoted as ) and the rotation matrix (denoted as ) is the inverse matrix:
[0122]
[0123] in, is the camera's projection matrix calculation function, is the rotation matrix calculation function. Then the algorithm takes , And multiple processing functions, convert the pixel coordinates on the image into 3D point coordinates on the sphere, and then convert them into two-dimensional coordinates on the image plane to generate a single-view image : The calculation expression is as follows:
[0124]
[0125] in, Equirec2Perspec represents the synthesis of a series of coordinate transformation functions, represents the camera's projection matrix, Represents the rotation matrix of the camera. Figure 2 The generated single-view image will have a similar field of view as the original single-view image in the Matterport3D dataset.
[0126] S3. rewrite the original instruction and the new trajectory visual observation image to generate original alignment sequence landmark information and new sequence observation description information respectively;
[0127] In S3, since the new trajectory visual observation image is different from the panoramic image of the original trajectory, the original paired instructions will not be well aligned with the new trajectory visual observation image. Previous work mainly trained the LSTM-based Speaker model from scratch on the VLN dataset to generate instructions for unannotated paths in the Matterport3D environment; however, the instructions generated by the Speaker model are usually uninformative and noisy, and the sentence patterns and verbs of these generated instructions are very homogeneous. Therefore, with the help of the powerful reasoning ability and language skills of the large language model, the instruction rewriting is performed in a way that does not require model training, rather than the traditional training-based Speaker model data enhancement paradigm. To this end, this embodiment first extracts the original alignment sequence landmark information in the original instruction for the original instruction, and collects the new sequence observation description information in the generated trajectory, and then provides them to the large language model to generate new trajectory paired instructions with new modality alignment. By constraining the large language model to rewrite the instructions to include more diverse sentence patterns and modality alignment information of verbs, the generated new trajectory paired instructions can effectively enhance the text understanding and cross-modal alignment capabilities of the VLN agent. The original instruction is rewritten, including
[0128] S301. Extract the landmark information list in the original instruction ,in is the size of the landmark information list, List of landmark information The k elements;
[0129] S302. Use the landmark information list as the input of the pre-trained visual language model and output the original trajectory t The first benchmark action at the time step and Best matching landmarks , The calculation expression is as follows:
[0130]
[0131] in, represents the similarity calculation function, Indicates that a similarity calculation function has been found Get the maximum value of the parameter. A text encoder representing a visual language model, An image encoder representing a visual language model;
[0132] S303. Collect each of the original trajectories Collect the best matching landmarks , get the original aligned sequence landmark information , The number of navigation steps.
[0133] After aligning the original trajectory with the sequential landmarks, the visual language model is then used Obtain new sequence observation description information for generating new trajectory visual observation images.
[0134] The instruction rewriting of the new trajectory visual observation image includes:
[0135] S311. Extracting the second benchmark action from the generated new trajectory visual observation image ;
[0136] S312. The second reference action Describe and get the new observation description text ,
[0137] Obtained by the following formula New observation description text :
[0138]
[0139] in It is a visual language model;
[0140] S313. Collect the new observation description text , get the new sequence observation description information .
[0141] It should be noted that for instruction rewriting, the original alignment sequence landmark information and the new sequence observation description information are first obtained, and then, by comparing the original alignment sequence landmark information and the new sequence observation description information, the large language model is prompted to rewrite the original instruction to obtain a new trajectory pairing instruction.
[0142] S4. Preprocessing the original alignment sequence landmark information and the new sequence observation description information to generate new trajectory pairing instructions;
[0143] In S4, the original alignment sequence landmark information and the new sequence observation description information are pre-processed for instruction rewriting, including: and the new sequence observation description information Input to the large language model, prompting the large language model to compare the landmark information of the original alignment sequence and the new sequence observation description information , to replace the sequential landmark information mentioned in the instructions, and to change the representation of the interaction description to output new trajectory pairing instructions.
[0144] Examples of contextual tasks that are input into the large language model for instruction rewriting are as follows:
[0145] Objects in original trajectory: Step1: workshop; Step2: workshop; Step3: black leather chair; Step4: black leather chair.
[0146] New trajectory information: Step1: a modern kitchen with stainless steel appliances and a large island. Step2: the foyer of a large house with high ceilings, windows, doors and a fireplace. Step3: the interior of a modern house with large windows, a living room and kitchen. Step4: a dining table and chairs in a modern kitchen and dining area of an apartment.
[0147] Original Instruction: Leave the workshop and head to the blackleather chair. Stop next the chair and the red wicker chairs.
[0148] Rewritten instruction: Go out of the kitchen and walk forward to the large windows. Stand next to a dining table with chairs.
[0149] The corresponding Chinese meaning is:
[0150] Objects in the original trajectory: Step 1: workshop; Step 2: workshop; Step 3: black leather chair; Step 4: black leather chair.
[0151] New Track Information: Step 1: A modern kitchen with stainless steel appliances and a large kitchen island. Step 2: A large house foyer with high ceilings, windows, doors and a fireplace. Step 3: The interior of a modern house with large windows, living room and kitchen. Step 4: Dining table and chairs in the modern kitchen and dining room of the apartment.
[0152] Original description text: Leave the workshop and go to the black leather chair. Stop next to the chair and the red wicker chair.
[0153] Rewritten description text: Walk out of the kitchen and over to the large window. Stand next to a dining table with chairs.
[0154] Based on the given example, the large language model is asked to rewrite the original instruction with the original aligned sequence landmark information and the new sequence observation description information. The large language model is also required to modify the action representation in the original instruction, such as changing "Leave" to "Go out". Based on the above task example, the large language model is prompted to generate new trajectory pairing instructions as follows:
[0155] Objects in original trajectory (objects in the original trajectory, i.e. original alignment sequence landmark information):
[0156] New trajectory information:
[0157] Original Instruction:
[0158] Rewritten Instruction: '
[0159] Therefore, for a given trajectory-command pair with a panoramic observation sequence and instructions , through visual observation rewriting and instruction rewriting, multiple generated trajectory-instruction pairs with new panoramic observation sequences are obtained and rewrite instructions The instruction rewrite prompt words are as follows.
[0160] Rewrite the original instruction according to the trajectoryinformation given. The rewritten instruction should satisfy the following principles:
[0161] 1) The rewritten instruction should replace the object (scene) described in the given instruction with a specific object (scene) mentioned in the new trajectory information according to the matched step information. The rewritten instruction should not contain the objects mentioned in the given instruction.
[0162] 2) The rewritten instruction must have similar length with the given instruction.
[0163] 3) The rewritten instruction must not contain the word "Step".
[0164] 4) The rewritten instruction should change synonyms of verbs in the original instruction, like changing "turn left" to "take a left to".
[0165] The corresponding Chinese meaning is:
[0166] Rewrite the original instructions according to the given trajectory information. The rewritten instructions should meet the following principles:
[0167] 1) The rewritten instructions should replace the objects (scenes) described in the given instructions with the specific objects (scenes) mentioned in the new trajectory information according to the matching step information. The rewritten instructions should not contain the objects mentioned in the given instructions.
[0168] 2) The length of the rewritten instruction must be comparable to the length of the given instruction.
[0169] 3) The rewritten instruction cannot contain the word "Step".
[0170] 4) The rewritten instructions should change the synonyms of the verbs in the original instructions, such as changing "turn left" to "left" (the English expression is different).
[0171] Here is an example.
[0172] Objects in original trajectory: Step1: workshop; Step2: workshop; Step3: black leather chair; Step4: black leather chair.
[0173] New trajectory information: Step1: a modern kitchen with stainless steel appliances and a large island. Step2: the foyer of a large house with high ceilings, windows, doors and a fireplace. Step3: the interior of a modern house with large windows, a living room and kitchen. Step4: a dining table and chairs in a modern kitchen and dining area of an apartment.
[0174] Original Instruction: Leave the workshop and head to the blackleather chair. Stop next the chair and the red wicker chairs.
[0175] Objects in new trajectory information: Step1: kitchen. Step2: highceilings, windows, doors and a fireplace. Step3: large windows. Step4: adining table and chairs.
[0176] Rewritten instruction: Go out of the kitchen and walk forward to thelarge windows. Stand next a dining table and chairs.
[0177] The corresponding Chinese meaning is:
[0178] Here is an example.
[0179] Objects in the original trajectory: Step 1: workshop; Step 2: workshop; Step 3: black leather chair; Step 4: black leather chair.
[0180] New Track Information: Step 1: A modern kitchen with stainless steel appliances and a large island. Step 2: A large house foyer with high ceilings, windows, doors and a fireplace. Step 3: The interior of a modern house with large windows, living room and kitchen. Step 4: Dining table and chairs in the modern kitchen and dining room of the apartment.
[0181] Original description text: Leave the workshop and go to the black leather chair. Stop by the chair and the red wicker chair. Side.
[0182] Objects in the new track information: Step 1: Kitchen. Step 2: High ceilings, windows, doors, and fireplace. Step 3: Large window. Step 4: A dining table and chairs.
[0183] Rewritten description text: Walk out of the kitchen and over to the large window. Stand next to a dining table and a chair.
[0184] Here is the instruction you need to rewrite. You should generate the objects in new trajectory information and the rewritten instruction with the same format of the given example.
[0185] Objects in original trajectory:
[0186] New trajectory information:
[0187] Original Instructions:
[0188] The corresponding Chinese meaning is:
[0189] This is the instruction you need to rewrite. You should generate new trajectory information and objects in the rewritten instruction using the same format as the given example.
[0190] Objects in the original trajectory:
[0191] New track information:
[0192] Original instruction:
[0193] S5. Mix the new trajectory pairing instruction and the trajectory-instruction pair to obtain a mixed data set, and use the mixed data set to train a preset visual language navigation model to obtain a trained visual language navigation model, which is used to perform the visual language navigation task.
[0194] In S5, the embodiment of the present invention proposes a new paradigm of VLN cross-modal data enhancement, called VLN-Rewriter. VLN-Rewriter is connected to a powerful pre-trained baseline model, namely the visual language navigation model. In the pre-training stage, the navigation agent is trained with three proxy tasks: masked language modeling (MLM), instruction and trajectory matching (ITM), and single action prediction (SAP). In the fine-tuning stage, the visual language navigation model uses the DAGGER method to train the navigation agent. Rewriting data is added in both the pre-training and fine-tuning stages to enhance the training of the visual language navigation model. The navigation loss function is used Train the preset visual language navigation model and the navigation loss function The calculation expression is as follows:
[0195]
[0196] in, represents the navigation agent, represents the original trajectory-command pair, Represents the original instruction, Indicates the rewritten new track pairing instructions, Indicates a rewrite instruction.
[0197] In this embodiment, see Figure 2 , for human-annotated trajectory-instruction pairs, visual observation rewriting and instruction rewriting are performed to generate enhanced cross-modal data. In observation rewriting, panoramic description texts are first collected, and co-occurrence-based description text rewriting operations are performed to generate diverse environment descriptions. Then, an effective panoramic-to-view generation strategy is developed to generate new trajectory visual observation images with different sequential observation information. In instruction rewriting, the original aligned sequence landmark information in the original instructions is extracted, and the new sequence observation description information of the generated trajectory is obtained. The large language model is further prompted to generate new trajectory pairing instructions by comparing the original aligned sequence landmark information and the new sequence observation description information. After obtaining the augmented new trajectory pairing instructions, they are mixed with the original data in the pre-training and training fine-tuning stages to train the visual language navigation model, and the trained visual language navigation model is used to perform the visual language navigation task. Among them, the visual language navigation task of the visual language navigation model is set as follows. In an embodiment of the present invention, it focuses on the visual language navigation task in a discrete environment, in which the navigation agent needs to explore the navigation connection graph , according to the original instructions given , moving from the starting node to the target node, and Represent the nodes and edges in the navigation connection graph respectively. , the navigation agent will receive a message containing Single-view observation Panoramic observation ,Right now .in, In the view N The action space consists of navigable views and “stop” actions. The navigation agent navigates from different time steps. In the action space, select one of them as the action prediction ,During the training process of the vision-language navigation model, the navigation agent receives ,trajectory instruction pairs to learn the ability of cross-modal alignment to ,successfully navigate under a specific trajectory.
[0198] Example 2
[0199] The visual language navigation method based on generative data-driven learning proposed in this embodiment is a new paradigm for VLN cross-modal data enhancement, called VLN-Rewriter. Thanks to the rich world knowledge in the large language model LLM and the literary graph generation model, it rewrites limited manually annotated data (trajectory-instruction pairs) to achieve an economical and efficient way to create diverse and high-information enhanced data.
[0200] Specifically, firstly, for a given manually annotated trajectory-instruction pair, two main rewriting processes are performed, namely “visual observation rewriting” and “instruction rewriting”.
[0201] In the "visual observation rewriting" process, this embodiment collects panoramic description texts of trajectories through a pre-trained visual language model; then, the large language model is prompted to generate rewritten description texts to allow it to explicitly point out various possible co-existing objects, thereby achieving controllable diversity enhancement of the environment; and based on the rewritten description texts, a panoramic-to-view generation strategy based on the text-generated graph generation model is developed to obtain new trajectory visual observation images while ensuring high efficiency and consistency between adjacent views.
[0202] For the "instruction rewriting" process, the sequential landmark information mentioned in the original instruction is first obtained and aligned to different navigation time steps in the original trajectory to obtain the original aligned sequence landmark information; then, the new sequence observation description information at different time steps is collected in the generated trajectory.
[0203] Considering the powerful reasoning and language capabilities of the large language model, the large language model is required to replace the sequential landmark information mentioned in the instruction and change the representation of the behavior description by comparing the original aligned sequence landmark information with the new sequence observation description information to generate a new trajectory pairing instruction.
[0204] After obtaining the new track pairing instructions, they are mixed with the original data during the pre-training and fine-tuning stages for training vision-language navigation.
[0205] See also Figure 3-Figure 5 , compare the VLN-Rewriter proposed in this embodiment with the typical VLN data enhancement method; wherein, Figure 3 represents data augmentation from the original / additional simulator, Figure 4 represents data enhancement from network images or videos. These two methods may be affected by limited environmental diversity and a large amount of noise in the data. This embodiment proposes Figure 5 The VLN-Rewriter achieves customized diversity data augmentation in an economical and efficient way by leveraging a powerful large-scale language model and a vincent graph generation model. The co-occurring objects generated in step 1 in the new trajectory are represented by hangers, vases, bookcases, and pendants, and the newly introduced objects mentioned in the pairing instructions of the new trajectory are represented by bathtubs, windows, green plants, walls, white toilets, sinks, gray walls, pendant lights, and glass shower doors.
[0206] It should be particularly noted that the VLN-Rewriter proposed in the embodiment of the present invention has the following advantages.
[0207] First, by leveraging the rich physical world knowledge of the large language model and the cultural graph generation model, the present invention achieves customized diversity data enhancement by combining various co-existing objects with reasonable spatial relationships.
[0208] Secondly, unlike the previous instruction enhancement strategy based on the Speaker model (which relies on training and tends to generate low-quality instructions), the enhanced instructions generated by the VLN-Rewriter proposed in the embodiment of the present invention are more informative and more similar to manually annotated instructions in a way that does not require training.
[0209] In addition, the VLN-Rewriter proposed in the embodiment of the present invention fully utilizes limited manually annotated data by completely "rewriting" the original trajectory-instruction pairs, thereby achieving a cost-effective way to create diversified data.
[0210] The visual language navigation method based on generative data-driven learning proposed in the embodiment of the present invention is experimented on three common VLN benchmark datasets: R2R, REVERIE, and R4R. The experimental results show that, combined with a simple pre-training-based baseline model, the VLN-Rewriter proposed in the embodiment of the present invention significantly outperforms the baseline model and previous advanced methods. In addition, with only small-scale data augmentation, the VLN-Rewriter proposed in the embodiment of the present invention achieves comparable or even better performance than recent methods that rely on large-scale additional simulator data.
[0211] Example 3
[0212] This example tests the proposed visual language navigation method based on generated data driven learning. Figure 6 , Figure 6 The main visualizations include the rewritten panorama description text, the generated panorama, the baseline viewpoint sequence extracted from the generated panorama, and examples of the rewritten instructions.
[0213] The original description text is: a photo of a bedroom and living room from the balcony of a home. The corresponding Chinese text of the original description text is: a photo of a bedroom and living room taken from the balcony of a home;
[0214] The rewritten description text is: Looking from the balcony of a home, you can see a cozy bedroom with a plush armchair and a sleek coffee table in the living room, illuminated by the warm glow of a contemporary floor lamp. The rewritten description text corresponds to the Chinese: Looking from the balcony of a home, you can see a cozy bedroom with a plush armchair and a sleek coffee table in the living room, illuminated by the warm glow of a contemporary floor lamp.
[0215] The original instruction is: Enter the bedroom by the door to the left of the large red painting. Once inside the bedroom, take the first door on the left to stepout onto the balcony. Stop as soon as you're outside.
[0216] Rewrite the command to: Walk into the hallway and turn left at the plants and flowers. Go
[0217] Walk into the hallway and turn left in front of the plants and flowers. Go through the first door on the left to step out onto the balcony. Stop as soon as you're outside.
[0218] The new co-existing objects and modally aligned objects generated in the instruction are represented by armchair, coffee table, floor lamp, painting, balcony, hallway, plants and flowers respectively.
[0219] Figure 6A visualization example of the data generated by VLN-Rewriter is shown. From the fine-grained description text rewritten by the large language model, it can be found that multiple co-occurrences, such as armchair and coffee table, are explicitly pointed out. Benefiting from the fine-grained rewritten description text, the generated panorama contains these co-occurrences and also introduces new layouts with respect to the rewritten description text, resulting in enhanced diversity of the environment. In addition, it can be observed that the extracted single-view images are also of high quality compared to the images taken by the camera in the Matterport3D dataset. It can also be found that the rewritten instructions successfully indicate modality-aligned objects, such as hallway corridors and plants plants, which are well aligned in the benchmark action sequences.
[0220] Figure 7 More visualization examples of panoramas generated by the VLN-Rewriter method of the present invention and their corresponding rewritten description texts are shown. Figure 7 The following can be observed:
[0221] The original description text 1 is: a picture of a living room in a house or apartment. The Chinese translation of the original description text 1 is: a picture of a living room in a house or apartment.
[0222] Rewrite description text 1 as follows: A stylishly designed living room in an urban apartment, featuring a large audio player, a vintage camera, and captivating wall art, providing a sophisticated and comfortable setting for entertainment and relaxation. The Chinese translation of rewrite description text 1 is: A stylishly designed living room in an urban apartment, featuring a large audio player, a vintage camera, and captivating wall art, providing a sophisticated and comfortable setting for entertainment and relaxation. The new coexistence objects generated by rewriting description text 1 are represented by audio player, vintage camera, and wall art.
[0223] The original description text 2 is: the lobby of a building with a kitchen and livingroom. The corresponding Chinese translation of the original description text 2 is: the lobby of a building with a kitchen and livingroom.
[0224] Rewrite description text 2 as follows: The expansive lobby boasts an array of lush potted plants, showcasing vibrant greenery against the backdrop of captivating artwork, while a LCD television hangs on the wall, adding an elegant touch to the space. The Chinese translation of rewrite description text 2 is: The expansive lobby boasts an array of lush potted plants, showcasing vibrant greenery against the backdrop of captivating artwork, while a LCD television hangs on the wall, adding an elegant touch to the space. The new coexistent objects generated by rewriting description text 2 are represented by greenery, artwork, and LCD television.
[0225] The original description text 3 is: a home office in a house with curved walls and a hallway. The corresponding Chinese translation of the original description text 3 is: a home office in a house with curved walls and a hallway.
[0226] Rewrite description text 3 as follows: Amongst the residence's unique curved architecture, there lies captivating home office space, adorned with a comfortable chair, striking artwork, and a sturdy filing cabinet standing proudly in the curved hallway. The corresponding Chinese translation of rewrite description text 3 is: Amongst the residence's unique curved architecture, there lies captivating home office space, adorned with a comfortable chair, striking artwork, and a sturdy filing cabinet standing proudly in the curved hallway. The new coexisting objects generated by rewriting description text 3 are represented by comfortable chair, filing cabinet, and curved hallway;
[0227] Through the customized description text rewriting prompt words of the present invention, the rewritten description text generated by the large language model can indicate reasonable and informative coexisting objects in different scenes. For example, the rewritten description text contains audio players and wall art that may appear in the living room. In addition, Figure 7 It can be found that the added co-occurrences can be successfully included in the generated panorama, and the room layout of the new panorama is very different from the original panorama. Therefore, the proposed panorama generation method can controllably enhance the diversity of the environment to achieve information data enhancement, which is beneficial to improve the generalization ability of the VLN agent.
[0228] See also Figure 8 The original instruction is: Enter the bedroom by the door to the left of the large red painting. Once inside the bedroom, take the first door on the left to step out onto the balcony. Stop as soon as you're outside. The Chinese translation of this original instruction is: Enter the bedroom by the door to the left of the large red painting. Once inside the bedroom, take the first door on the left to step out onto the balcony. Stop as soon as you're outside. The modal alignment objects in the instruction are represented by painting and balcony.
[0229] The command generated by the Speaker model is: walk out of the bedroom and turn right. walk past
[0230] The generated instruction is translated into Chinese as: Walk out of the bedroom and turn right. Walk past the bed and stop in the doorway. The modal alignment object in the instruction is represented by bed.
[0231] The rewritten instruction is: Walk into the hallway and turn left at the plants and flowers. Go through the first door on the left to step out onto the balcony. Stop as soon as you're outside. The Chinese translation of this rewritten instruction is: Walk into the hallway and turn left at the plants and flowers. Go through the first door on the left to step out onto the balcony. Stop as soon as you're outside. The modal alignment objects in the instruction are represented by hallway, plants and flowers, and balcony.
[0232] For Speaker model and Figure 8 The VLN-Rewriter generation instructions in Figure 8 It can be found that the instructions generated by the Speaker model contain less information than the instructions rewritten by the present invention, and may also contain incorrect action information, such as changing the original "enter the bedroom" to "walk out of the bedroom". The rewritten instructions tend to introduce reasonable behavior representations and indicate multiple modal alignment objects that are different from the original instructions, which can effectively improve the cross-modal alignment ability of the agent.
[0233] Example 4
[0234] See also Fig. 9 This embodiment also proposes a visual language navigation system based on generated data driven learning, the system comprising:
[0235] An acquisition module, used for acquiring a trajectory-instruction pair, wherein the trajectory-instruction pair includes an original trajectory and an original instruction;
[0236] A visual observation rewriting module, used for rewriting the original trajectory by visual observation to generate a new trajectory visual observation image;
[0237] An instruction rewriting module, used for rewriting the original instruction and the new trajectory visual observation image to generate original alignment sequence landmark information and new sequence observation description information respectively;
[0238] A preprocessing module, used to preprocess the original alignment sequence landmark information and the new sequence observation description information to generate new trajectory pairing instructions;
[0239] The visual language navigation module is used to mix the new trajectory pairing instruction and the trajectory-instruction pair to obtain a mixed data set, and use the mixed data set to train a preset visual language navigation model to obtain a trained visual language navigation model, and the visual language navigation model is used to perform the visual language navigation task.
[0240] In this embodiment, firstly, for the manually annotated trajectory-instruction pairs, the trajectory-instruction pairs are rewritten based on two key aspects, namely observation rewriting and instruction rewriting. On the one hand, the original trajectory is rewritten by visual observation to obtain a new trajectory visual observation image; on the other hand, the original instruction and the new trajectory visual observation image are rewritten by instruction to generate original alignment sequence landmark information and new sequence observation description information; then the new trajectory pairing instruction is generated by preprocessing the original alignment sequence landmark information and the new sequence observation description information; after obtaining the new trajectory pairing instruction, the new trajectory pairing instruction and the trajectory-instruction pair are mixed to promote the training of the visual language navigation model. The present invention proposes a new cross-modal data enhancement framework for visual language navigation, which has excellent visual language navigation performance on benchmark datasets of multiple visual language navigation tasks. The framework can create different enhanced data only by rewriting available manually annotated data, and is not limited to visual language navigation data in a specific field obtained from a limited simulator environment. It reduces the processing cost of unlabeled data, effectively improves the visual language navigation performance, can be applied to unknown scenes, and provides a meaningful reference for how to make full use of existing manually annotated data.
[0241] Obviously, the above embodiments of the present invention are only examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.
Claims
1. A visual language navigation method based on generative data driven learning, characterized in that: The following steps are involved: S1. Acquire a trajectory-command pair, wherein the trajectory-command pair includes an original trajectory and an original command; S2. rewrite the original trajectory by visual observation to generate a new trajectory visual observation image; S3. rewrite the original instruction and the new trajectory visual observation image to generate original alignment sequence landmark information and new sequence observation description information respectively; S4. Preprocessing the original alignment sequence landmark information and the new sequence observation description information to generate new trajectory pairing instructions; S5. Mix the new trajectory pairing instruction and the trajectory-instruction pair to obtain a mixed data set, and use the mixed data set to train a preset visual language navigation model to obtain a trained visual language navigation model, which is used to perform the visual language navigation task.
2. The visual language navigation method based on generative data driven learning according to claim 1, characterized in that: The visual observation and rewriting of the original trajectory comprises: S201. Collecting panoramic description texts during different navigation time steps in the original trajectory; S202. Rewriting the panoramic description text based on the coexistence knowledge using the large language model to obtain a rewritten description text; S203. Execute a panorama-to-view generation strategy on the rewritten description text to obtain the new trajectory visual observation image.
3. The visual language navigation method based on generative data driven learning according to claim 2, characterized in that: The panorama description text collected during different navigation time steps in the original trajectory includes: Define the original trajectory t The panoramic image at the time step is , the original trajectory t Panoramic image at time step Input the preset visual language model and output different navigations in the original trajectory t Panorama description text during the time step as follows: in, Represents a visual language model.
4. The visual language navigation method based on generative data driven learning according to claim 3 is characterized in that: The step of rewriting the panoramic description text based on the co-existence knowledge by using the large language model includes: Get the description text of the panorama and description rewrite hints , the panorama description text and the description rewrites the prompt word The large language model is input, and the added co-existence and the rewritten description text corresponding to the co-existence are output. The calculation expression of the rewritten description text is as follows: in, Indicates rewriting the description text. N Indicates the number of rewritten description texts. Represents a large language model.
5. The visual language navigation method based on generated data driven learning according to claim 4, characterized in that: The panorama-to-view generation strategy includes Rewrite the description text Input into the preset Wensheng image generation model and output N new panoramic images ; Each new panoramic image Perform panorama discretization to obtain a new trajectory visual observation image, which is a single view image , The calculation expression is as follows: in, Equirec2Perspec Represents the synthesis of a series of coordinate transformation functions, represents the camera's projection matrix, Represents the rotation matrix of the camera.
6. The visual language navigation method based on generative data driven learning according to claim 1, characterized in that: Rewriting the original instruction includes: S301. Extract the landmark information list in the original instruction ,in is the size of the landmark information list, List of landmark information The k elements; S302. Use the landmark information list as the input of the pre-trained visual language model and output the original trajectory t The first benchmark action at the time step and Best matching landmarks , The calculation expression is as follows: in, represents the similarity calculation function, Indicates that a similarity calculation function has been found Get the maximum value of the parameter. A text encoder representing a visual language model, An image encoder representing a visual language model; S303. Collect each of the original trajectories Collect the best matching landmarks , get the original aligned sequence landmark information , The number of navigation steps.
7. The visual language navigation method based on generated data driven learning according to claim 6, characterized in that: The new trajectory visual observation image is rewritten by instructions, including: S311. Extracting the second benchmark action from the generated new trajectory visual observation image ; S312. The second reference action Describe and get the new observation description text , S313. Collect the new observation description text , get the new sequence observation description information .
8. The visual language navigation method based on generated data driven learning according to claim 7, characterized in that: Preprocessing the original aligned sequence landmark information and the new sequence observation description information includes: and the new sequence observation description information Input to the large language model, prompting the large language model to compare the original aligned sequence landmark information and the new sequence observation description information , output new trajectory pairing instructions.
9. The visual language navigation method based on generated data driven learning according to any one of claims 1 to 8, characterized in that: Using the navigation loss function Train the preset visual language navigation model and the navigation loss function The calculation expression is as follows: in, represents the navigation agent, represents the original trajectory-command pair, Represents the original instruction, Indicates the rewritten new track pairing instructions, Indicates a rewrite instruction.
10. A visual language navigation system based on generative data driven learning, characterized in that: The system comprises: An acquisition module, used for acquiring a trajectory-instruction pair, wherein the trajectory-instruction pair includes an original trajectory and an original instruction; A visual observation rewriting module, used for rewriting the original trajectory by visual observation to generate a new trajectory visual observation image; An instruction rewriting module, used for rewriting the original instruction and the new trajectory visual observation image to generate original alignment sequence landmark information and new sequence observation description information respectively; A preprocessing module, used to preprocess the original alignment sequence landmark information and the new sequence observation description information to generate new trajectory pairing instructions; The visual language navigation module is used to mix the new trajectory pairing instruction and the trajectory-instruction pair to obtain a mixed data set, and use the mixed data set to train a preset visual language navigation model to obtain a trained visual language navigation model, and the visual language navigation model is used to perform the visual language navigation task.
Citation Information
Patent Citations
Visual language navigation pre-training method based on prompt and automatic environment exploration
CN114970457A
Visual language navigation method combining image description and text generation image
CN117571014A
System and method for navigating a vehicle using language instructions
WO2021058090A1
