An embodied agent indoor navigation method based on visual and textual common sense enhancement

Abstract

The present invention discloses an embodied intelligent indoor navigation method based on visual and textual common sense enhancement. The method comprises providing environmental common sense information using ChatGPT and BLIP-2; generating common sense visual images using Stable Diffusion; extracting visual features and textual features using a visual encoder and a text encoder respectively; constructing topological map features through topological mapping; helping a VLN agent learn environmental common sense from the language and visual dimensions respectively using a textual common sense layout learning module and a visual common sense layout learning module; fusing the learned common sense text features with the learned common sense visual-topological features to obtain multimodal fusion features, which are used as input to a classification regression head for action prediction; and finally performing online navigation and loading a trained model to test the navigation effect. The present invention enhances the expressive power of visual features by using semantic and visual information such as room types and common objects in room types contained in text and visual common sense, thereby improving navigation accuracy and robustness.

Description

Technical Field

[0001] The present invention relates to the field of visual language navigation technology, and in particular to an embodied intelligent indoor navigation method based on visual and textual common sense enhancement. Background Art

[0002] With the rapid development of embodied intelligence technology, vision-and-language navigation (VLN), a key AI navigation task that combines technologies from multiple fields, including computer vision, natural language processing, and robotics, has gradually become a research hotspot in the field of embodied intelligence. VLN tasks require intelligent robots to understand and execute natural language commands and effectively navigate using visual information from the environment. This task not only places high demands on the perception and decision-making capabilities of intelligent robots but also involves complex multimodal information fusion, a key capability for intelligent robots to perform tasks in the real world. In recent years, VLN tasks have garnered widespread attention and achieved considerable research progress, particularly in areas such as indoor navigation, autonomous driving, and augmented reality.

[0003] Early VLN methods primarily built models based on LSTM architectures. However, these methods have significant shortcomings in leveraging long-term historical information, making it difficult to effectively capture and understand complex spatiotemporal structures. To improve model performance, some VLN methods have introduced data augmentation strategies to enhance the agent's understanding of environmental information and promote multimodal alignment and fusion. With the development of Transformers, cross-modal pre-training models have made significant progress, and VLN methods based on cross-modal pre-training models have also flourished. These methods have greatly improved the modal alignment capabilities of VLN agents. However, due to limited navigation data, methods based on cross-modal pre-training models still struggle to fully understand the common sense relationships between rooms and landmarks in a scene. This common sense often relies on rich open-world knowledge, such as the basic understanding that showerheads are typically located in bathrooms.

[0004] Recent work has introduced large language models (LLMs) and large visual language models (VLMs) to VLN tasks. Leveraging the powerful reasoning capabilities of large language models, these models can dynamically parse natural language instructions in complex navigation tasks. By correlating multimodal information, they can infer implicit relationships within the scene, providing a potential path to zero-shot navigation. However, these models do not fully exploit the rich open-world knowledge embedded in large language models. This rich open-world knowledge, including commonsense information such as room functions, landmark attributes, and their relationships, can provide navigation agents with a deep understanding of the environment. For example, "a sofa is typically in the living room," "a kitchen typically has a refrigerator," or "a bathroom might have a shower and a towel rack." This implicit commonsense knowledge not only helps agents more accurately identify and localize landmarks but also guides them to make more informed decisions in complex environments, thereby improving navigation performance. By incorporating this rich commonsense information, agents can better understand the layout of the environment and reason about underlying spatial relationships, thereby achieving greater robustness in unknown scenarios.

[0005] Existing visual-language navigation (VLN) approaches using large-scale language models (LLMs) primarily focus on using LLMs to reason about navigation paths. This involves using the language model to parse instructions, infer target locations, and plan reasonable routes within the environment. These approaches typically rely on the powerful language understanding and reasoning capabilities of LLMs to help agents more accurately understand complex instructions and make decisions. However, they often neglect deep modeling of environmental layout and scene common sense, potentially limiting the agent's navigation effectiveness in complex or unseen environments. Summary of the Invention

[0006] In response to the problem that the existing technology often ignores the deep modeling of environmental layout and scene common sense, and the visual language navigation method based on LLMs is ineffective, the present invention provides an embodied intelligent indoor navigation method based on visual and textual common sense enhancement. By fully leveraging the open-world knowledge contained in the large model, ChatGPT and BLIP-2 are used to provide environmental common sense information, where BLIP-2 is used to parse the room type corresponding to each panorama, while ChatGPT is used for reasoning and providing the most common landmark knowledge in each room type; Stable Diffusion is used to generate common sense visual images to enhance the visual cognition ability of the VLN agent; a text common sense layout learning module and a visual common sense layout learning module are designed to help the VLN agent learn environmental common sense from the two dimensions of language and vision, respectively, so that the VLN agent can fully utilize text and visual common sense information during the navigation process.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0008] A method for indoor navigation of an embodied intelligent agent based on visual and textual common sense enhancement, characterized by comprising the following steps:

[0009] S10, based on the panoramic images of indoor environments generated by the Matterport3D simulator, uses the BLIP-2 model and the ChatGPT model to provide common sense information about the environment. The BLIP-2 model is used to parse the room type corresponding to each panorama, and the ChatGPT model is used to infer and provide common sense about the landmarks in each room type.

[0010] S20, using the Stable Diffusion model to generate common sense visual images of room types with common sense landmarks based on room types and landmark common sense;

[0011] S30, obtain the panoramic image and instruction dataset from the Matterport3D simulator;

[0012] S40, extracting visual features of the view from the panoramic image through a visual encoder, and extracting text features of the instruction from the instruction dataset through a text encoder;

[0013] S50, constructing a topological map feature by topological mapping using the acquired visual features;

[0014] S60, extract text common sense features and visual common sense features respectively through the CLIP model;

[0015] S70, based on the acquired topological map features, textual common sense features, and visual common sense features, the constructed textual common sense layout learning module and visual common sense layout learning module are used to help the VLN agent learn environmental common sense from the language and visual dimensions, respectively, and output visual-topological features and textual instruction features;

[0016] S80, fusing the visual-topological features and the text instruction features output in step S70 to obtain a multimodal fusion feature;

[0017] S90, inputting the multimodal fusion features into the action prediction classification head to perform action prediction;

[0018] S100. Train in a pre-training-fine-tuning mode, and use reinforcement learning and imitation learning to perform online navigation. Load and train a layout learning fine-tuning model based on visual and textual common sense enhancement to test the navigation effect, wherein the layout learning fine-tuning model based on visual and textual common sense enhancement is implemented by the process of the above steps S10-S90.

[0019] Specifically, in step S10, the panoramic image in the Matterport3D simulator is input into the BLIP-2 model, and the content scope of the produced room type text description is limited by the set prompt words used to describe the characteristics of the room type; then the obtained room type text description is used to ask questions to the ChatGPT model, and through the preset prompt words, the corresponding landmark knowledge is obtained for the specified number of the most common objects in each room type, thereby forming landmark common sense for each room type.

[0020] Specifically, in step S20 , different common sense visual images are generated for each different panorama of the same room type to cover different layouts and landmark configurations of the room type.

[0021] Specifically, in step S40, the panoramic image obtained in step S30 is input into a visual encoder to extract visual features of the view; and the instruction data set obtained in step S30 is input into a text encoder to extract text features of the instructions.

[0022] Specifically, in step S50, the VLN agent gradually constructs its own map by observing along the path, and adds the current node and the adjacent visited nodes to the topology map at each time step to construct the topology map features.

[0023] Specifically, in step S60, the text features of the instruction and the visual features of the view are respectively input into the text encoder and the visual encoder of the CLIP model to obtain text common sense features and visual common sense features.

[0024] Specifically, in step S70, the process of the text common sense layout learning module is expressed as follows:

[0025]

[0026]

[0027]

[0028]

[0029]

[0030]

[0031]

[0032] In the above formula, represents visual common sense features, Represents the visual features enhanced with room type knowledge, Represents the room type text description in the text common sense feature, Represents the topological map feature, Trans represents a 4-layer cross-modal Transformer, represents the landmark common sense in the text common sense feature, Softmax represents the Softmax activation function, d represents the dimension size of the feature vector, and Represents landmark common sense Two cosine similarity matrices with the current view and the topology graph respectively, and Represents the two landmark common senses that best match the current view and topology map respectively. express and The weighted weight of , Sigmoid represents the Sigmoid activation function, represents the landmark common sense features after dynamic fusion, [;] represents the splicing operation, Represents the visual-topological features output by the text commonsense layout learning module.

[0033] Specifically, in step S70, the process of the visual common sense layout learning module is represented as follows:

[0034]

[0035]

[0036]

[0037] In the above formula, The text features representing the instruction, represents visual common sense features, Represents the instruction features after learning visual common sense, express and The weighted weight of Represents the text instruction features output by the visual commonsense layout learning module.

[0038] Specifically, in step S80, the process of obtaining the multimodal fusion feature is expressed as follows:

[0039]

[0040] In the above formula, Represents the output multimodal fusion features.

[0041] Specifically, in step S90, the process of action prediction is expressed as follows:

[0042]

[0043]

[0044]

[0045]

[0046] In the above formula, Represents the visual features in the output multimodal fusion features, Represents the topological features in the output multimodal fusion features, FFN represents a two-layer feedforward network, and Represent the local navigation score and global navigation score of the candidate view respectively, express The weighted weight of , S represents the final navigation score.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] (1) This paper makes full use of the open-world knowledge contained in the large model, using the ChatGPT model and the BLIP-2 model to obtain common-sense environmental information, providing the VLN agent with common-sense landmarks about room layouts and common scenes. It also uses Stable Diffusion to generate common-sense visual images and simulate the visual features of common landmarks in the scene, making up for the lack of visual knowledge. This cross-modal knowledge acquisition method provides the VLN agent with more comprehensive world knowledge, helping it to more accurately understand and reason about different room types and their layouts, thereby improving its navigation capabilities. It effectively solves the problem of neglecting deep modeling of environmental layout and scene common sense, which leads to reduced navigation accuracy and limited path planning capabilities of the VLN agent when facing unseen environments or complex layouts.

[0049] (2) This paper designs a layout learning model enhanced by visual and textual common sense. Through the textual common sense layout learning module and the visual common sense layout learning module, it helps the agent to more accurately understand the relationship between the landmarks in the instructions and the room type, as well as the visual characteristics and layout rules of common landmarks in the room, thereby improving the navigation decision-making and reasoning capabilities of the VLN agent in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 Schematic diagram of a process of an embodiment of the present invention.

[0051] Figure 2 Schematic diagram of the network structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The present invention will be further described below with reference to the accompanying drawings and examples. The embodiments of the present invention include but are not limited to the following examples.

[0053] like Figures 1 to 2 As shown, the embodied intelligent indoor navigation method based on visual and textual common sense enhancement includes the following steps:

[0054] S10. Based on panoramic images of indoor environments generated by the Matterport 3D simulator, the BLIP-2 model and ChatGPT model are used to provide common-sense information about the environment. The BLIP-2 model is used to parse the room type corresponding to each panoramic image, while the ChatGPT model is used to infer and provide landmark common sense for each room type. Specifically, panoramic images from the Matterport 3D simulator are input into the BLIP-2 model. Prompt words used to describe the characteristics of the room type are used to limit the content scope of the generated text description of the room type. For example, after the BLIP-2 model conducts an in-depth analysis of each panoramic image, it is asked "What type of room is this?" to obtain the corresponding room type for each panoramic image. A total of 40 different room types are obtained. To enrich the landmark knowledge of each room type, the ChatGPT model is asked questions about the five most common objects in each room type to obtain corresponding landmark knowledge and form landmark common sense for each room type. Ultimately, five common landmark common senses are generated for each room type.

[0055] S20. To better adapt to embodied recognition tasks, this invention uses generated textual common sense to build visual common sense, helping the VLN agent understand the layout patterns of common landmarks in different room types. Specifically, the StableDiffusion model is used to generate common-sense visual images of room types, incorporating common-sense landmarks, based on the room type textual descriptions and landmark common sense. To ensure sufficient visual diversity for each room type, different images are generated for each panorama of the same room type to cover different room layouts and landmark configurations, thereby enabling the VLN agent to learn room layout characteristics from multiple angles and scenarios. Ultimately, a total of 52,835 images were generated.

[0056] S30. Obtain the panoramic image and instruction dataset from the Matterport3D simulator.

[0057] S40: Extract visual features of the view from the panoramic image using a visual encoder, and extract text features of the instructions from the instruction dataset using a text encoder. Specifically, the panoramic image obtained in step S30 is input into a visual encoder (e.g., VIT) to extract visual features of the view; the instruction dataset obtained in step S30 is input into a text encoder (e.g., BERT) to extract text features of the instructions. The visual encoder and text encoder have 9 and 4 Transformer layers, respectively, and are both initialized using LXMERT.

[0058] S50: Input the visual features acquired in step S40 into the topology mapping module to construct topology map features. Specifically, the topology map is initially unknown to the agent. The VLN agent gradually constructs its own map by observing along the path. At each time step t, the current node and the adjacent visited nodes are added to the topology map to construct the topology map features.

[0059] S60: Extract textual common sense features and visual common sense features using the CLIP model. The CLIP model used is configured as the CLIP-ViT-B / 16 model. The textual features of the instruction and the visual features of the view are input into the CLIP text encoder and visual encoder, respectively, for encoding to obtain textual common sense features and visual common sense features.

[0060] S70. Based on the acquired topological map features, text common sense features, and visual common sense features, the constructed text common sense layout learning module and visual common sense layout learning module are used to help the VLN agent learn environmental common sense from the language and vision dimensions respectively, and output visual-topological features and text instruction features.

[0061] Specifically, topological features, textual common sense features, and visual common sense features are input into the textual common sense layout learning module to help the VLN agent identify the characteristics of different rooms and the room types corresponding to landmarks, thereby effectively inferring the location of the task target. The process of the textual common sense layout learning module is expressed as follows:

[0062]

[0063]

[0064]

[0065]

[0066]

[0067]

[0068]

[0069] In the above formula, represents visual common sense features, Represents the visual features enhanced with room type knowledge, Represents the room type text description in the text common sense feature, Represents the topological map feature, Trans represents a 4-layer cross-modal Transformer, represents the landmark common sense in the text common sense feature, Softmax represents the Softmax activation function, d represents the dimension size of the feature vector, and Represents landmark common sense Two cosine similarity matrices with the current view and the topology graph respectively, and Represents the two landmark common senses that best match the current view and topology map respectively. express and The weighted weight of , Sigmoid represents the Sigmoid activation function, represents the landmark common sense features after dynamic fusion, [;] represents the splicing operation, Represents the visual-topological features output by the text commonsense layout learning module.

[0070] The text features and visual common sense features of the instructions are input into the visual common sense layout learning module. By learning these common sense visual images, the VLN agent can recognize common sense landmarks in the environment, such as furniture and decorations, and infer the room type and layout based on visual clues. The process of the visual common sense layout learning module is expressed as follows:

[0071]

[0072]

[0073]

[0074] In the above formula, The text features representing the instruction, represents visual common sense features, Represents the instruction features after learning visual common sense, express and The weighted weight of Represents the text instruction feature output by the visual common sense layout learning module. The text instruction feature is the dynamic fusion of the instruction feature after learning visual common sense and the text feature of the instruction.

[0075] S80: Fusing the visual-topological features and textual instruction features outputted in step S70 to obtain a multimodal fusion feature. The specific process is: aligning the visual-topological features and textual instruction features with text and vision, and fusing them to obtain a multimodal fusion feature. The process is expressed as follows:

[0076]

[0077] In the above formula, Represents the output multimodal fusion features.

[0078] S90, input the multimodal fusion features into the action prediction classification head to perform action prediction. and topological features To predict the next action; the process of action prediction is expressed as:

[0079]

[0080]

[0081]

[0082]

[0083] In the above formula, Represents the visual features in the output multimodal fusion features, Represents the topological features in the output multimodal fusion features, FFN represents a two-layer feedforward network, and Represent the local navigation score and global navigation score of the candidate view respectively, express The weighted weight of , S represents the final navigation score.

[0084] S100. Train in a pre-training-fine-tuning mode, and use reinforcement learning and imitation learning to perform online navigation. Load and train a layout learning fine-tuning model enhanced by visual and textual common sense to test the navigation effect. The layout learning fine-tuning model enhanced by visual and textual common sense is implemented by the process of the above steps S10-S90.

[0085] In this embodiment, the layout learning fine-tuning model designed based on visual and textual common sense enhancement is trained for 100k iters in the training phase, with a learning rate of 5×10 -5 , trained for 100k iters in the fine-tuning phase with a learning rate of 1×10 -5 .

[0086] Tables 1 to 7 show the comparison results of the proposed method (ViTeC) with some existing vision-language navigation methods on the R2R, REVERIE, and SOON datasets, respectively, where Val Seen, Val Unseen, and Test Unseen represent the validation set in the environment, the validation set in the unseen environment, and the test set, respectively. The existing visual-language navigation methods compared include: sequence-to-sequence model (Seq2Seq), visual-language navigation method with environment discarding (EnvDrop), visual-language navigation based on random environment mixing (REM), in-domain pre-training method for visual-language navigation (AirBERT), history and order-aware pre-training method for visual-language navigation (HOP), history-aware multimodal transformer (HAMT) for visual-language navigation, dual-scale graph transformer (DUET) for visual-language navigation, visual-language navigation method with explicit reasoning combined with large language model (NavGPT), map-guided hint mechanism for unified visual-language navigation (MapGPT), exploratory learning for general model of embodied navigation (NaviLLM), unleashing the navigation reasoning capability of large-scale visual-language model (NavGPT-2), learning and adaptation of visual-language navigation strategy (RCM), room and object-aware knowledge reasoning for long-range embodied reference resolution (CKR), augmented commonsense knowledge for long-range object localization (ACK), visual-language navigation method with knowledge-augmented reasoning (KERM), and visual-language navigation method with correctable landmark discovery (CONSOLE).

[0087] The performance evaluation of the R2R dataset uses the following metrics: (1) trajectory length (TL): the average length of the agent's path; (2) navigation error (NE): the average distance between the agent's final position and the target position; (3) success rate (SR): the proportion of paths where the agent stops within 3 meters of the target point; and (4) SPL: the success rate weighted by path length. For the REVERIE and SOON datasets that include the object grounding task, three additional metrics are required to evaluate the object grounding performance: (5) oracle success rate (OSR): the success rate calculated under the ideal stopping policy; (6) long-range alignment success rate (RGS): the proportion of successfully executed instructions; and (7) RGSPL: the long-range alignment success rate weighted by the navigation path length. Except for NE and TL, the higher the values ​​of the remaining metrics, the better.

[0088]

[0089] Table 1 Val Seen comparison of the present invention and similar methods on the R2R dataset

[0090]

[0091] Table 2 Comparison of Val Unseen between the present invention and similar methods on the R2R dataset

[0092]

[0093] Table 3 Test Unseen comparison of the present invention and similar methods on the R2R dataset

[0094]

[0095] Table 4 Comparison of the Val Seen results of the present invention and similar methods on the REVERIE dataset

[0096]

[0097] Table 5 Val Unseen comparison of the present invention and similar methods on the REVERIE dataset

[0098]

[0099] Table 6 Test Unseen comparison of the present invention and similar methods on the REVERIE dataset

[0100]

[0101] Table 7 Comparison of the present invention with similar methods on the SOON dataset

[0102] From the above comparison, we can see that the method of the present invention (ViTeC) successfully learns textual common sense and visual common sense, enabling the intelligent agent to accurately identify key landmarks in the environment and infer the type and layout of the room based on visual clues, thereby making more reasonable decisions in complex navigation tasks.

[0103] The above embodiments are only preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any changes that adopt the design principles of the present invention and any changes made through non-creative work on this basis should fall within the scope of protection of the present invention.

Claims

1. A method for indoor navigation of an embodied intelligent agent based on visual and textual common sense enhancement, characterized in that: The following steps are involved: S10, based on the panoramic images of indoor environments generated by the Matterport3D simulator, uses the BLIP-2 model and the ChatGPT model to provide common sense information about the environment. The BLIP-2 model is used to parse the room type corresponding to each panorama, and the ChatGPT model is used to infer and provide common sense about the landmarks in each room type. S20, using the Stable Diffusion model to generate common sense visual images of room types with common sense landmarks based on room types and landmark common sense; S30, obtain the panoramic image and instruction dataset from the Matterport3D simulator; S40, extracting visual features of the view from the panoramic image through a visual encoder, and extracting text features of the instruction from the instruction dataset through a text encoder; S50, constructing a topological map feature by topological mapping using the acquired visual features; S60, extract text common sense features and visual common sense features respectively through the CLIP model; S70, based on the acquired topological map features, textual common sense features, and visual common sense features, the constructed textual common sense layout learning module and visual common sense layout learning module are used to help the VLN agent learn environmental common sense from the language and visual dimensions, respectively, and output visual-topological features and textual instruction features; S80, fusing the visual-topological features and the text instruction features output in step S70 to obtain a multimodal fusion feature; S90, inputting the multimodal fusion features into the action prediction classification head to perform action prediction; S100. Train in a pre-training-fine-tuning mode, and use reinforcement learning and imitation learning to perform online navigation. Load and train a layout learning fine-tuning model based on visual and textual common sense enhancement to test the navigation effect, wherein the layout learning fine-tuning model based on visual and textual common sense enhancement is implemented by the process of the above steps S10-S90.

2. The method for indoor navigation of an embodied intelligent agent based on visual and textual common sense enhancement according to claim 1, characterized in that: In step S10, the panoramic image in the Matterport3D simulator is input into the BLIP-2 model, and the content scope of the generated room type text description is limited by the set prompt words used to describe the characteristics of the room type; then the obtained room type text description is used to ask questions to the ChatGPT model, and the preset prompt words are used to obtain corresponding landmark knowledge for the most common specified number of objects in each room type, thereby forming landmark common sense for each room type.

3. The method for indoor navigation of an embodied intelligent agent based on enhanced visual and textual common sense according to claim 2, characterized in that: In step S20 , different common sense visual images are generated for each different panorama of the same room type to cover different layouts and landmark configurations of the room type.

4. The method for indoor navigation of an embodied intelligent agent based on enhanced visual and textual common sense according to claim 3, characterized in that: In step S40, the panoramic image obtained in step S30 is input into a visual encoder to extract visual features of the view; The instruction data set obtained in step S30 is input into a text encoder to extract text features of the instructions.

5. The method for indoor navigation of an embodied intelligent agent based on enhanced visual and textual common sense according to claim 4, characterized in that: In step S50 , the VLN agent gradually constructs its own map by observing along the path, and adds the current node and the adjacent visited nodes to the topology map at each time step to construct the topology map features.

6. The method for indoor navigation of an embodied intelligent agent based on visual and textual common sense enhancement according to claim 5, characterized in that: In step S60 , the text features of the instruction and the visual features of the view are respectively input into the text encoder and the visual encoder of the CLIP model to obtain text common sense features and visual common sense features.

7. The method for indoor navigation of an embodied intelligent agent based on enhanced visual and textual common sense according to claim 6, characterized in that: In step S70, the process of the text common sense layout learning module is represented as follows: In the above formula, represents visual common sense features, Represents the visual features enhanced with room type knowledge, Represents the room type text description in the text common sense feature, Represents the topological map feature, Trans represents a 4-layer cross-modal Transformer, represents the landmark common sense in the text common sense feature, Softmax represents the Softmax activation function, d represents the dimension size of the feature vector, and Represents landmark common sense Two cosine similarity matrices with the current view and the topology graph respectively, and Represents the two landmark common senses that best match the current view and topology map respectively. express and The weighted weight of , Sigmoid represents the Sigmoid activation function, represents the landmark common sense features after dynamic fusion, [;] represents the splicing operation, Represents the visual-topological features output by the text commonsense layout learning module.

8. The method for indoor navigation of an embodied intelligent agent based on enhanced visual and textual common sense according to claim 7, characterized in that: In step S70, the process of the visual common sense layout learning module is represented as follows: In the above formula, The text features representing the instruction, represents visual common sense features, Represents the instruction features after learning visual common sense, express and The weighted weight of Represents the text instruction features output by the visual commonsense layout learning module.

9. The method for indoor navigation of an embodied intelligent agent based on visual and textual common sense enhancement according to claim 8, characterized in that: In step S80, the process of obtaining the multimodal fusion feature is expressed as follows: In the above formula, Represents the output multimodal fusion features.

10. The method for indoor navigation of an embodied intelligent agent based on visual and textual common sense enhancement according to claim 9, characterized in that: In step S90, the process of action prediction is expressed as follows: In the above formula, Represents the visual features in the output multimodal fusion features, Represents the topological features in the output multimodal fusion features, FFN represents a two-layer feedforward network, and Represent the local navigation score and global navigation score of the candidate view respectively, express The weighted weight of , S represents the final navigation score.

Images