A Visual-Language Navigation Method Based on Dual Semantic Graphs and Modal Alignment

Abstract

The present invention provides a visual language navigation method based on dual semantic graphs and modality alignment. The agent is initialized at a random navigable point. According to the visual environment information of the current navigable point, the visual features of the current navigable point and the corresponding text semantic features are extracted. Then, topological graphs are constructed respectively using the features of these two modalities. At the same time, a grammar parser is used to extract the key noun features in the instruction. The text instruction features for enhancing the text features are respectively subjected to cross-modal attention fusion with the visual semantic topological graph, the text semantic topological graph, and the visual features of the current navigable point, so as to make correct navigation decisions. Based on the text instruction features and visual features, the present invention enhances the representation of key information for each navigable point by constructing a visual topological graph and a text semantic topological graph, reduces the semantic gap between the text instruction and the visual environment, and improves the accuracy and efficiency of visual language navigation.

Description

Technical Field

[0001] The present invention relates to the technical field of visual language navigation, and particularly to a visual language navigation method based on dual semantic graphs and modality alignment. Background Art

[0002] A long-term goal of artificial intelligence is to develop an agent that can interact with humans using natural language to complete real-world tasks; the visual language navigation task requires the agent to be able to complete navigation and other instruction-related tasks according to natural language instructions provided by humans. For example, a detailed instruction describing a navigation trajectory: "Walk along the corridor, then turn right into the kitchen and stop next to the microwave oven", and another is a navigation instruction that conforms to the human language expression habit: "Enter the kitchen and get me an egg". This VLN task requires the agent to be able to understand the semantic relationship between the two modalities in order to effectively explore and plan the navigation path.

[0003] Currently, the methods for aligning natural language instructions and visual environments include two categories: one is to align sub-instructions and sub-paths. The navigation instructions of the original visual language navigation task usually consist of multiple sub-instructions, and the correct navigation path can also be split into multiple sub-paths accordingly. However, the instructions provided by humans are usually concise and diverse, so this method relies on manual annotation of sub-instructions and sub-paths and is not applicable to other high-level navigation tasks. The other method is to extract and align the landmark objects in the visual environment and the key nouns in the instructions. But this method can only mechanically align the key nouns mentioned in the language instructions, and this method still directly aligns the two modalities without reducing the semantic differences between different modalities. Summary of the Invention

[0004] The purpose of the present invention is to provide a visual language navigation method based on dual semantic graphs and modality alignment to solve the deficiencies of the above-mentioned prior art.

[0005] To achieve the above purpose, the present invention adopts the following technical solutions:

[0006] A visual language navigation method based on dual semantic graphs and modality alignment specifically includes the following steps:

[0007] S1. Initialize the navigation of the agent, and obtain a panoramic visual image including 36 perspectives and an instruction based on natural language description according to the position coordinates of the agent;

[0008] S2. Extract instruction features and instruction key noun features based on the instruction, encode the instruction features and instruction key noun features respectively, and perform attention fusion to generate a text instruction feature with enhanced features;

[0009] S3, extracting visual features from the panoramic visual image;

[0010] S4. Modally align the text instruction features and the visual features of the current navigable point through a cross-modal attention mechanism to obtain cross-modal fusion features. The agent then makes a local navigation decision at the current navigable point through an action decision module.

[0011] S5. Build and save a global visual semantic topology map based on the visual features of the current and previously visited navigable points. Use a cross-modal attention mechanism to align the text instruction features with the global visual semantic topology map to obtain cross-modal fusion features. The agent then makes global navigation decisions based on the visual features through an action decision module.

[0012] S6. Based on the visual features of the current navigable point, the corresponding text semantic features are obtained, a global text semantic topology map is constructed and saved, and the text instruction features and the global text semantic topology map are modally aligned through a cross-modal attention mechanism to obtain cross-modal fusion features. The intelligent agent makes a global navigation decision based on text semantics through the action decision module.

[0013] S7, dynamically integrate the local navigation decision made by the agent and the global navigation decision based on visual features and text semantics, so that the agent can obtain the forward direction or stop the action;

[0014] S8. Save the visual environment information of the visited navigable points, update the panoramic visual image, and repeat steps S4 to S8 until the agent reaches the destination, stops moving forward, and completes navigation.

[0015] Furthermore, the method for extracting the instruction features and the instruction key noun features includes using a grammar parser Stanford Parser, Berkeley Parser, or NLTK; specifically, they are expressed by formulas (1) and (2):

[0016] I feat =f command (I NL ) (1);

[0017] I kn =f key_noun (p(I NL )) (2);

[0018] In formulas (1) and (2): I NL Indicates instructions based on natural language description; f command Indicates the extraction of the features of the entire instruction; I feat Indicates the instruction features; p indicates the use of a grammar parser to extract key nouns in the instruction; f key_noun Represents the characteristics of the key nouns in the extraction instruction; Ikn Indicates the key noun features of the instruction.

[0019] Furthermore, the encoding methods for the instruction features and key noun features include using text encoders BERT and CLIP; the attention fusion of the encoded instruction features and instruction key noun features to generate enhanced text instruction features is specifically represented by formula (3):

[0020] I enhanced = MHA(I feat , I kn ) (3);

[0021] In formula (3): MHA represents the multi-head attention mechanism in the Transformer encoder; I enhanced represents the enhanced text instruction features.

[0022] Furthermore, the extraction method of the visual features includes using visual encoders CLIP, VIT, and Res Net; it is specifically represented by formula (4):

[0023] V feat = f visual (V); V = {v1, v2,..., v c} c＝36 (4);

[0024] In formula (4): V represents the panoramic visual image; c represents the serial number of the viewing angle; f visual represents using the visual encoder to extract visual features; V feat represents the visual features.

[0025] Furthermore, step S4 is specifically implemented by the following method:

[0026] First, perform modality alignment on the text instruction features and visual features through the cross-modal attention framework CA based on the LXMERT model to obtain cross-modal fusion features;

[0027] Then, based on the action decision module, the agent classifies and makes decisions on the obtained cross-modal fusion features through the feed-forward neural network FFN to obtain the probability distribution of local navigable points, that is, the probability distribution of navigable points adjacent to the agent's current point, and the one with the highest probability value is the navigable point to be selected;

[0028] It is specifically represented by formulas (5) and (6):

[0029]

[0030]

[0031] In formulas (5) and (6), CA represents the cross-modal attention framework based on the LXMERT model; Represents cross-modal fusion features of text instruction features and visual features; D local It indicates that the local navigable point prediction is performed on the obtained cross-modal fusion features through the feed-forward neural network FFN; Represents the predicted distribution of the current local navigable points, that is, the agent is based on cross-modal fusion features Local navigation decisions made.

[0032] Furthermore, step S5 is specifically implemented by the following method:

[0033] First, perform an average pooling operation on the visual features of the current navigable point to obtain the visual features of the current navigable point on the global topology map;

[0034] Then the historical global visual semantic topology map is spliced with the visual features of the current navigable point on the global topology map to construct the current global visual semantic topology map;

[0035] Then, the cross-modal attention framework CA based on the LXMERT model is used to align the text instruction features and the global visual semantic topology map to obtain cross-modal fusion features.

[0036] Then, based on the action decision module, the intelligent agent uses the feedforward neural network (FFN) to classify and decide on the obtained cross-modal fusion features to obtain the probability distribution of global navigable points.

[0037] Specifically expressed by formulas (7) to (10):

[0038] V c_node =f node (V current ) (7);

[0039]

[0040]

[0041]

[0042] In formulas (7) to (10): V current Indicates the visual features of the current navigable point; f node represents the average pooling operation; V c_node Represents the visual features of the current navigable point on the global topology map; Represents the topological map of the global visual features of the history; Concat represents the concatenation of the topological map of the global visual features of the history with the visual features of the current navigable point on the global topological map; G visual_semanticA topological map representing the global visual features of the current navigable point; Represents the cross-modal fusion features of the text instruction features and the global visual semantic topological map; D global Represents the global navigable point prediction by performing global navigable point prediction on the obtained cross-modal fusion features through a feed-forward neural network FFN; Represents the prediction distribution of the global navigable point, that is, the global navigation decision made by the agent based on the cross-modal fusion features Made global navigation decisions.

[0043] Furthermore, the step S6 is specifically implemented by the following method:

[0044] Pre-obtain a number of object noun and room scene type nouns according to the R2R and REVERIE training sets, and perform feature extraction respectively;

[0045] Then, through cosine similarity calculation, match the visual features of each perspective in the panoramic visual image of the current navigable point with a number of most similar object noun features and room scene type noun features respectively, and randomly initialize a token symbol with the same dimension as a single noun and splice it with them as the text semantic features corresponding to the visual features of each perspective, and then obtain the text semantics corresponding to each navigation node through average pooling operation;

[0046] Then splice the historical global text semantic topological map with the text semantic features corresponding to the visual features of the current navigable point to construct the current global text semantic topological map;

[0047] Then, through the cross-modal attention framework CA based on the LXMERT model, perform modal alignment on the text instruction features and the global text semantic topological map to obtain cross-modal fusion features;

[0048] Then the agent, based on the action decision module, makes a classification decision on the obtained cross-modal fusion features through a feed-forward neural network FFN to obtain the prediction distribution of the current global navigable point;

[0049] Specifically represented by formulas (11) to (16):

[0050] ON i =sim topk (v c ,N obj ) (11);

[0051] OR i =sim topk (v c ,N rt ) (12);

[0052] V text_semantic=concat(ON i , token, OR i ); (13)

[0053]

[0054]

[0055]

[0056] In formulas (11) to (16): N obj represents the object noun; N rt represents the room scene type noun; sim topk represents selecting the k object nouns or room scene type nouns most similar to each perspective according to the cosine similarity; ON i represents the object noun most similar to the visual feature of perspective c; OR i represents the room scene type noun most similar to the visual feature of perspective c; token represents a randomly initialized tensor; V text_semantic represents the text semantic feature corresponding to the visual feature of each navigable point; represents the cross-modal fusion feature of the text instruction feature and the global text semantic topology map; D gkobal represents globally predicting the navigable points by performing a feed-forward neural network FFN on the obtained cross-modal fusion feature; represents the prediction distribution of the current global navigable points, that is, the text-based global navigation decision made by the agent based on the cross-modal fusion feature

[0057] Furthermore, the specific implementation of step S7 is as follows:

[0058] Weighted sum the local navigation decision the global navigation decision based on the visual feature and the global navigation decision based on the text semantics to obtain the prediction distribution of the global navigable points through dynamic fusion, and the navigable point with the largest predicted value is the navigation action selected by the agent; specifically, it is represented by formula (17):

[0059]

[0060] In formula (17): DynamicFusion represents weighted summing of ; P represents the prediction distribution of the global navigable points obtained through dynamic fusion, and the navigable point with the largest predicted value is the navigation action selected by the agent.

[0061] ​As can be seen from the above technical solutions, compared with the prior art, the present invention has the following technical advantages:

[0062] 1. By augmenting the data of text semantics, the diversity of input data is significantly improved, and by aligning the modalities of the extracted text semantics and text instructions, the semantic differences between modalities are effectively reduced;

[0063] 2. By introducing random token symbols, the agent can capture potential features that are difficult to express, significantly improving the generalization ability of the agent; increasing the depth of the agent's understanding of text information, making it more flexible and robust when processing inputs with diverse semantic representations, and enriching the agent's learning of text semantics;

[0064] 3. By extracting the key noun features in the text instructions and enhancing them, the agent can more effectively capture the objects to be recognized, more accurately understand the requirements of the instructions for the room type, thereby improving the performance of the overall system in target recognition and task understanding; making the agent more intelligent and adaptive, and at the same time improving the accuracy and efficiency of the agent when processing specific navigation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is a schematic diagram of the principle framework of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0066] A preferred embodiment of the present invention will be described in detail below with reference to the accompanying drawings.

[0067] As Figure 1 shown, a visual language navigation method based on a dual semantic graph and modality alignment specifically includes the following steps:

[0068] S1. Initialize the navigation of the agent, and obtain a panoramic visual image including 36 perspectives and an instruction based on natural language description according to the position coordinates of the agent.

[0069] The navigation initialization provides the starting point of the navigation task; the panoramic visual image in this preferred embodiment is an RGB image, which is evenly divided into 12 segments in the horizontal direction and 3 segments in the vertical direction, and a total of 36 perspective sub-images are obtained, denoted as V = {v1, v2,..., v c} c＝36 ; the guiding instruction is a continuous and complete sentence describing the navigable trajectory, denoted as I NL .

[0070] S2. Extract instruction features and instruction key noun features based on the instruction, encode the instruction features and instruction key noun features respectively, and perform attention fusion to generate text instruction features with enhanced features.

[0071] Specifically, first, based on the instruction, instruction features and key noun features of the instruction are extracted through a syntax parser such as Stanford Parser, Berkeley Parser or NLTK (Natural Language Toolkit). The instruction features and key noun features of the instruction are denoted as I feat and I kn ;

[0072] Then, the instruction features and key noun features are encoded through a text encoder such as BERT or CLIP; the extraction of the instruction features and key noun features of the instruction is represented by formulas (1) and (2):

[0073] I feat = f command (I NL ) (1);

[0074] I kn = f key_noun (p(I NL )) (2);

[0075] In formulas (1) and (2): I NL represents the instruction described based on natural language; f command represents extracting the features of the entire instruction; I feat represents the instruction features; p represents using a syntax parser to extract the key nouns in the instruction; f key_noun represents extracting the features of the key nouns in the instruction; I kn represents the key noun features of the instruction;

[0076] Then, the multi-head attention mechanism MHA in the Transformer encoder is adopted for attention fusion. The multi-head attention mechanism MHA can better capture the relationships between input features, improve the representation ability of text instruction features, and provide more effective information input for subsequent modality alignment; the attention fusion of the encoded instruction features and key noun features of the instruction to generate text instruction features with enhanced features is specifically represented by formula (3):

[0077] I enhanced = MHA(I feat , I kn ) (3);

[0078] In formula (3): MHA represents the multi-head attention mechanism in the Transformer encoder; I enhanced represents the text instruction features with enhanced features.

[0079] By extracting the key noun features in the text instructions and enhancing them, the present invention enables the intelligent agent to more effectively capture the objects to be recognized and more accurately understand the requirements of the instructions for the room type, thereby improving the performance of the overall system in target recognition and task understanding; making the intelligent agent more intelligent and adaptive, and also improving the accuracy and efficiency of the intelligent agent in dealing with specific navigation tasks.

[0080] S3. Extract visual features from the panoramic visual image.

[0081] Specifically, extract visual features from the panoramic visual image through visual encoders such as CLIP, VIT, and Res Net; specifically represented by formula (4):

[0082] V feat =f visual (V); V = {v1, v2, …, v c} c＝36 (4);

[0083] In formula (4): V represents the panoramic visual image; c represents the serial number of the viewing angle; f visual represents extracting visual features using the visual encoder; V feat represents the visual features.

[0084] S4. Align the text instruction features and visual features through a cross-modal attention mechanism to obtain cross-modal fusion features, and the intelligent agent makes local navigation decisions at specific navigable points through an action decision module.

[0085] Specifically implemented through the following method:

[0086] First, align the text instruction features and visual features through the cross-modal attention framework CA based on the LXMERT model to obtain cross-modal fusion features;

[0087] Then, based on the action decision module, the intelligent agent classifies and makes decisions on the obtained cross-modal fusion features through a feed-forward neural network FFN to obtain the probability distribution of local navigable points, that is, the navigable points adjacent to the current point of the intelligent agent, and the one with the highest probability value is the navigable point to be selected;

[0088] Specifically represented by formulas (5) and (6):

[0089]

[0090]

[0091] In formulas (5) and (6): CA represents the cross-modal attention framework based on the LXMERT model; Represents the cross-modal fusion feature of text instruction features and visual features; D local Represents predicting local navigable points for the obtained cross-modal fusion feature through the feed-forward neural network FFN; Represents the prediction distribution of the current local navigable point, that is, the local navigation decision made by the agent based on the cross-modal fusion feature

[0092] S5. Construct and save the global visual semantic topology map based on the visual features of the current navigable point. Align the text instruction features and the global visual semantic topology map through the cross-modal attention mechanism to obtain the cross-modal fusion feature. The agent makes a global navigation decision through the action decision module.

[0093] The construction of the visual semantic topology map helps the agent understand the environmental structure during navigation and provides support for global navigation decisions. It is specifically implemented through the following methods:

[0094] First, perform average pooling operation on the visual features of the current navigable point to obtain the visual features of the current navigable point on the global topology map;

[0095] Then, splice the historical global visual semantic topology map with the visual features of the current navigable point on the global topology map to construct the global visual semantic topology map of the current navigable point;

[0096] Then, through the cross-modal attention framework CA based on the LXMERT model, align the text instruction features and the global visual semantic topology map to obtain the cross-modal fusion feature;

[0097] Then, based on the action decision module, the agent classifies and makes a decision on the obtained cross-modal fusion feature through the feed-forward neural network FFN to obtain the probability distribution of the global navigable point;

[0098] Specifically, it is represented by formulas (7) to (10):

[0099] V c_node = f node (V current ) (7);

[0100]

[0101]

[0102]

[0103] In formulas (7) to (10): V current Represents the visual features of the current navigable point; f node Represents the average pooling operation; V c_node ​Represents the visual features of the current navigable point on the global topological map; The topological map representing the historical global visual features; Concat means concatenating the topological map of the historical global visual features with the visual features of the current navigable point on the global topological map; G visual_semantic The topological map representing the global visual features of the current navigable point; Represents the cross-modal fusion features of the text instruction features and the global visual semantic topological map; D global Represents predicting the global navigable point through the feed-forward neural network FFN for the obtained cross-modal fusion features; Represents the predicted distribution of the global navigable point, that is, the agent makes a global navigation decision based on the visual features based on the cross-modal fusion features Makes a global navigation decision based on visual features.

[0104] S6. Obtain the corresponding text semantic features based on the visual features of the current navigable point, construct the global text semantic topological map, perform modal alignment on the text instruction features and the global text semantic topological map through the cross-modal attention mechanism to obtain the cross-modal fusion features, and the agent makes a secondary local navigation decision through the action decision module.

[0105] Specifically implemented through the following method:

[0106] Pre-obtain a number of object noun and room scene type nouns according to the R2R and REVERIE training sets, and perform feature extraction respectively;

[0107] Then, through cosine similarity calculation, match the visual features of each perspective in the panoramic visual image of the current navigable point with a number of most similar object noun features and room scene type noun features respectively, and randomly initialize a token symbol with the same dimension as a single noun and concatenate it with them as the text semantic features corresponding to the visual features, and then obtain the text semantic features corresponding to each navigation node through the same average pooling operation as in step S5;

[0108] Then concatenate the historical global text semantic topological map with the text semantic features corresponding to the visual features of the current navigable point to construct the global text semantic topological map of the current navigable point;

[0109] Then perform modal alignment on the text instruction features and the global text semantic topological map through the cross-modal attention framework CA based on the LXMERT model to obtain the cross-modal fusion features;

[0110] Then the agent, based on the action decision module, makes a classification decision on the obtained cross-modal fusion features through the feed-forward neural network FFN to obtain the predicted distribution of the current global navigable point;

[0111] Specifically, it is represented by formulas (11) to (16):

[0112] ON i = sim topk (v c , N obj )(11);

[0113] OR i = sim topk (v c , N rt )(12);

[0114] V text_semantic = concat(ON i , token, OR i )(13);

[0115]

[0116]

[0117]

[0118] In formulas (11) to (16): N obj represents the object noun; N rt represents the room scene type noun; sim topk represents selecting the k object nouns or room scene type nouns most similar to each perspective according to the cosine similarity; ON i represents the object noun most similar to the visual feature of perspective c; OR i represents the room scene type noun most similar to the visual feature of perspective c; token represents a randomly initialized 768-dimensional tensor; V text_semantic represents the text semantic feature corresponding to the visual feature of each navigable point; represents the cross-modal fusion feature between the text semantic feature and the global visual semantic topology map; D global represents making a local navigable point prediction on the obtained cross-modal fusion feature through the feed-forward neural network FFN; represents the prediction distribution of the current local navigable point, that is, the global navigation decision based on text made by the agent based on the cross-modal fusion feature

[0119] ​By introducing random token symbols, the present invention enables the agent to capture latent features that are difficult to express, significantly improving the generalization ability of the agent; increasing the depth of the agent's understanding of text information, making it more flexible and robust when processing inputs with diverse semantic representations, and enriching the agent's learning of text semantics; at the same time, through data augmentation of text semantics, the diversity of input data is significantly enhanced, and by aligning the extracted text semantics with the modality of text instructions, the semantic differences between modalities are effectively reduced.

[0120] S7. Dynamically fuse the local navigation decisions made by the agent and the global navigation decisions based on visual features and text semantics, so that the agent obtains the forward direction or stop action.

[0121] Specifically, it is achieved through the following method:

[0122] Perform weighted summation on the local navigation decisions the global navigation decisions based on visual features and the global navigation decisions based on text semantics to obtain the predicted distribution of the globally navigable points through dynamic fusion. The navigable point with the largest predicted value is the navigation action selected by the agent; specifically, it is represented by formula (17):

[0123]

[0124] In formula (17): DynamicFusion represents performing weighted summation on [[ID=...]] P represents the predicted distribution of the globally navigable points obtained through dynamic fusion. The navigable point with the largest predicted value is the navigation action selected by the agent.

[0125] S8. Save the visual environment information of the visited navigable points, update the panoramic visual image, and repeat steps S4 to S8 until the agent reaches the destination, stops moving forward, and completes the navigation.

[0126] The above-described embodiments are merely descriptions of the preferred embodiments of the present invention, and do not limit the scope of the present invention. Without departing from the design spirit of the present invention, various deformations and improvements made by those of ordinary skill in the art to the technical solutions of the present invention shall fall within the protection scope determined by the claims of the present invention.

Claims

1. A visual language navigation method based on dual semantic graph and modality alignment, characterized by: The specific steps include: S1. Initialize navigation for the intelligent agent and obtain panoramic visual images including 36 perspectives and instructions based on natural language descriptions based on the position coordinates of the intelligent agent; S2. Extract instruction features and instruction key noun features based on the instruction, encode the instruction features and instruction key noun features separately, and then perform attention fusion to generate feature-enhanced text instruction features; S3, extracting visual features from the panoramic visual image; S4. Modally align the text instruction features and the visual features of the current navigable point through a cross-modal attention mechanism to obtain cross-modal fusion features. The agent then makes a local navigation decision at the current navigable point through an action decision module. S5. Build and save a global visual semantic topology map based on the visual features of the current and previously visited navigable points. Use a cross-modal attention mechanism to align the text instruction features with the global visual semantic topology map to obtain cross-modal fusion features. The agent then makes global navigation decisions based on the visual features through an action decision module. S6. Based on the visual features of the current navigable point, the corresponding text semantic features are obtained, a global text semantic topology map is constructed and saved, and the text instruction features and the global text semantic topology map are modally aligned through a cross-modal attention mechanism to obtain cross-modal fusion features. The intelligent agent makes a global navigation decision based on text semantics through the action decision module. S7, dynamically integrates the local navigation decision made by the agent and the global navigation decision based on visual features and text semantics, and the agent obtains the forward direction or stops the action; S8. Save the visual environment information of the visited navigable points, update the panoramic visual image, and repeat steps S4 to S8 until the agent reaches the destination, stops moving forward, and completes navigation.

2. The visual language navigation method based on dual semantic graph and modality alignment according to claim 1 is characterized in that: The method for extracting the instruction features and the instruction key noun features includes using a grammar parser Stanford Parser, Berkeley Parser, or NLTK; specifically, it is expressed by formulas (1) and (2): I feat =f command (I NL ) (1); I kn =f key_noun (p(I NL )) (2); In formulas (1) and (2): I NL Indicates instructions based on natural language description; f command Indicates the extraction of the features of the entire instruction; I feat Indicates the instruction features; p indicates the use of a grammar parser to extract key nouns in the instruction; f key_noun Represents the features of key nouns in the extraction instruction; I kn Indicates the key noun features of the instruction.

3. The visual language navigation method based on dual semantic graph and modality alignment according to claim 2 is characterized in that: The encoding method of the instruction features and key noun features includes using text encoders BERT and CLIP; the encoded instruction features and instruction key noun features are subjected to attention fusion to generate feature-enhanced text instruction features, which are specifically expressed by formula (3): I enhanced =MHA(I feat ,I kn ) (3); In formula (3): MHA represents the multi-head attention mechanism in the Transformer encoder; I enhanced Represents the text instruction features after feature enhancement.

4. The visual language navigation method based on dual semantic graph and modality alignment according to claim 1 is characterized in that: The visual feature extraction method includes using visual encoders CLIP, VIT, and Res Net; specifically, it is expressed by formula (4): V feat =f visual (V);V={v1,v2,…,v c } c＝36 (4); In formula (4), V represents the panoramic visual image; c represents the number of the viewing angle; f visual Indicates the use of visual encoder to extract visual features; V feat Represents visual features.

5. The visual language navigation method based on dual semantic graph and modality alignment according to claim 1 is characterized in that: The step S4 is specifically implemented by the following method: First, the cross-modal attention framework CA based on the LXMERT model is used to align the text instruction features and visual features to obtain cross-modal fusion features. Then, based on the action decision module, the agent uses the feedforward neural network (FFN) to classify and decide on the obtained cross-modal fusion features, and obtains the probability distribution of local navigable points, that is, the navigable points adjacent to the current point of the agent. The navigable point with the highest probability value is the navigable point to be selected. Specifically, it is expressed by formulas (5) and (6): In formulas (5) and (6), CA represents the cross-modal attention framework based on the LXMERT model; Cross-modal fusion features representing textual instruction features and visual features; D local It indicates that the local navigable point prediction is performed on the obtained cross-modal fusion features through the feed-forward neural network FFN; Represents the predicted distribution of the current local navigable points, that is, the agent is based on cross-modal fusion features Local navigation decisions made.

6. The visual language navigation method based on dual semantic graph and modality alignment according to claim 1, characterized in that: The step S5 is specifically implemented by the following method: First, perform an average pooling operation on the visual features of the current navigable point to obtain the visual features of the current navigable point on the global topology map; Then the historical global visual semantic topology map is spliced with the visual features of the current navigable point on the global topology map to construct the current global visual semantic topology map; Then, the cross-modal attention framework CA based on the LXMERT model is used to align the text instruction features and the global visual semantic topology map to obtain cross-modal fusion features. Then, based on the action decision module, the intelligent agent uses the feedforward neural network (FFN) to classify and decide on the obtained cross-modal fusion features to obtain the probability distribution of global navigable points. Specifically expressed by formulas (7) to (10): V c_node =f node (V current ) (7); In formulas (7) to (10): V current A visual feature representing the current navigable point; f node represents the average pooling operation; V c_node Represents the visual features of the current navigable point on the global topology map; A topological graph representing the global visual features of history; Concat means concatenating the topological graph of the historical global visual features with the visual features of the current navigable point on the global topological graph; G visual_semantic A topological map representing the global visual features of the current navigable point; Cross-modal fusion features representing text instruction features and global visual semantic topology maps; D global It indicates that the global navigable point prediction is performed on the obtained cross-modal fusion features through the feed-forward neural network FFN; Represents the predicted distribution of global navigable points, that is, the agent is based on cross-modal fusion features Global navigation decisions made.

7. The visual language navigation method based on dual semantic graph and modality alignment according to claim 1 is characterized in that: The step S6 is specifically implemented by the following method: Based on the R2R and REVERIE training sets, several object nouns and room scene type nouns are obtained in advance and feature extraction is performed on them respectively; Then, through cosine similarity calculation, the visual features of each perspective in the panoramic visual image of the current navigable point are matched with several most similar object noun features and room scene type noun features. A token symbol with the same dimension as a single noun is randomly initialized and spliced with it as the textual semantic feature corresponding to the visual features of each perspective. Then, the textual semantics corresponding to each navigation node is obtained through average pooling operation. Then, the historical global text semantic topology map is spliced with the text semantic features corresponding to the visual features of the current navigable point to construct the current global text semantic topology map; Then, the cross-modal attention framework CA based on the LXMERT model is used to align the text instruction features and the global text semantic topology map to obtain cross-modal fusion features. Then, based on the action decision module, the intelligent agent uses the feedforward neural network (FFN) to classify and decide on the obtained cross-modal fusion features to obtain the predicted distribution of the current global navigable points. Specifically expressed by formulas (11) to (16): ON i =sim topk (v c ,N obj ) (11); OR i =sim topk (v c ,N rt ) (12); V text_semantic =concat(ON i ,token,OR i ) (13); In formulas (11) to (16): N obj Noun indicating an object; N rt Indicates the type of room scene noun; sim topk Indicates selecting the k object nouns or room scene type nouns that are most similar to each perspective based on cosine similarity; ON i The noun representing the object that is most similar to the visual features of viewpoint c; OR i Represents the room scene type noun that is most similar to the visual features of view c; token represents a randomly initialized tensor; V text_semanyic Represents the text semantic features corresponding to the visual features of each navigable point; Cross-modal fusion features representing text instruction features and global text semantic topology graph; D global It indicates that the global navigable point prediction is performed on the obtained cross-modal fusion features through the feed-forward neural network FFN; Represents the predicted distribution of the current global navigable points, that is, the agent is based on cross-modal fusion features Make global text-based navigation decisions.

8. The visual language navigation method based on dual semantic graph and modality alignment according to claim 1 is characterized in that: The step S7 is specifically implemented by the following method: Local navigation decisions Global navigation decision based on visual features and global navigation decisions based on text semantics The predicted distribution of global navigable points is obtained by weighted summation through dynamic fusion. The navigable point with the largest predicted value is the navigation action selected by the agent. It is specifically expressed by formula (17): In formula (17): DynamicFusion represents Perform weighted summation; P represents the predicted distribution of global navigable points obtained through dynamic fusion, and the navigable point with the largest predicted value is the navigation action selected by the agent.

Images