Visual language navigation method and system based on collaborative alignment and adaptive fusion

Abstract

The present invention proposes a visual language navigation method and system based on collaborative alignment and adaptive fusion, which belongs to the field of artificial intelligence and computer vision. The method includes: obtaining natural language instructions, panoramic views, navigation history and dynamic topological map data, extracting instruction features, visual features, navigation history features and map node features; pre-aligning the instruction features and visual features with feature enhancement and inputting them into the encoder for deep fusion to generate global multimodal fusion features and auxiliary decision signals; inputting the global multimodal fusion features and auxiliary decision signals into a coarse and fine granularity combined expert network to output the final predicted navigation action; wherein the expert network is used to obtain the predicted value; constructing a multi-source uncertainty vector to generate fusion weights, fusing the predicted values, and obtaining the final predicted navigation action. It effectively solves the modal gap problem in traditional methods, and improves the accuracy of complex instruction understanding, navigation stability and success rate.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and computer vision technology, and in particular relates to a visual language navigation method and system based on collaborative alignment and adaptive fusion. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Vision-and-language navigation (VLN), a key research direction in embodied intelligence, aims to enable intelligent agents to autonomously navigate complex three-dimensional environments based on natural language instructions. This technology holds broad application prospects in areas such as human-computer interaction, service robotics, and virtual reality. However, existing VLN technologies still face numerous challenges in achieving efficient and robust navigation.

[0004] Early VLN research primarily employed end-to-end deep learning models, achieving navigation decisions by simply concatenating or integrating visual and language features. While these approaches have achieved some progress, their lack of deep understanding and alignment of multimodal information has limited navigation performance in complex scenarios. Specifically, existing technologies suffer from the following key issues:

[0005] First, the modality gap seriously hinders the effective fusion of cross-modal information. Significant semantic differences exist between the abstract symbolic nature of language instructions and the pixel-level features of visual scenes, making it difficult for models to establish accurate cross-modal associations. Existing methods typically directly fuse raw visual and language features, neglecting the importance of semantic alignment before feature fusion, resulting in inefficient subsequent cross-modal interaction.

[0006] Secondly, current multimodal fusion modules are still insufficiently able to integrate complex linguistic structures with visual details, preventing fine-grained phrase-level integration. Existing fusion mechanisms typically focus on global scene-command matching, ignoring the multi-level, multi-stage correspondence between different clauses or landmark components in a command and the visual regions. This coarse-grained aggregation strategy not only reduces the model's accuracy in localizing key landmarks but also limits its ability to model the complex dependencies between dynamic historical information and map objects. Summary of the Invention

[0007] To overcome the shortcomings of the above-mentioned prior art, the present invention provides a visual language navigation method and system based on collaborative alignment and adaptive fusion. The pre-fusion alignment module is used to alleviate modal differences and enhance the structured representation of features at an early stage. A unified Transformer encoder is introduced to deeply fuse multi-source information and promote fine-grained grounding. Finally, an adaptive decision and fusion module is used to dynamically adjust the fusion weights of coarse and fine-grained experts by constructing a multi-source uncertainty vector, thereby achieving a more robust navigation strategy.

[0008] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0009] A first aspect of the present invention provides a visual language navigation method based on collaborative alignment and adaptive fusion;

[0010] The vision-language navigation method based on collaborative alignment and adaptive fusion includes:

[0011] Obtain natural language instructions, panoramic views, navigation history, and dynamic topological map data, and extract instruction features, visual features, navigation history features, and map node features;

[0012] Performing feature enhancement and pre-alignment on the command features and the visual features to obtain enhanced command features and visual features;

[0013] The enhanced command features and visual features are input into the unified Transformer encoder for deep fusion to generate global multimodal fusion features and auxiliary decision signals;

[0014] The global multimodal fusion features and auxiliary decision signals are input into a coarse- and fine-grained expert network to output a final predicted navigation action; the global multimodal fusion features and auxiliary decision signals are processed by a coarse-grained expert unit and a fine-grained expert unit respectively to obtain a first comprehensive prediction value and a second comprehensive prediction value; a multi-source uncertainty vector is constructed to dynamically generate weights of the coarse- and fine-grained expert network prediction results, and the first comprehensive prediction value and the second comprehensive prediction value are fused based on the weights to obtain the final predicted navigation action.

[0015] As a further technical solution, the extraction of instruction features, visual features, navigation history features and map node features includes:

[0016] Using a two-layer instruction encoding method, instruction words and landmark words are extracted from natural language instructions, and the instruction words and landmark words are concatenated into an extended sequence containing special tokens; structured instruction features are obtained through encoding;

[0017] The visual features include panoramic region features and object features; the panoramic region features with direction embedding and the object features with direction embedding are extracted from the panoramic view;

[0018] The visual embedding and the corresponding action embedding are encoded into navigation history features through Transformer; a topological map is dynamically constructed and map node features are extracted.

[0019] As a further technical solution, the feature enhancement pre-alignment includes instruction-history alignment and landmark-observation alignment.

[0020] As a further technical solution, the instruction-history alignment includes:

[0021] Aggregating the navigation history features to obtain a history context summary;

[0022] Using the historical context summary as a query vector, performing attention calculation on the instruction feature to obtain the instruction feature attention weight;

[0023] The attention weight is used to enhance the instruction feature to obtain a context-aware instruction feature.

[0024] As a further technical solution, the landmark-observation alignment includes:

[0025] Extracting candidate landmark phrases and their representations from the context-aware instruction features, using the candidate landmark phrases and their representations as query vectors, using the panoramic region features and the object features with direction embedding as key vectors and value vectors, respectively, and calculating an alignment score between the landmarks and the visual entities;

[0026] For a salient visual object, calculate its attention distribution to all landmark phrases to obtain the textual landmark context most relevant to the visual object;

[0027] By utilizing the aligned context information and matching scores, enhanced panoramic region features and object features are obtained by fusing the original visual features and their corresponding textual contexts.

[0028] The enhanced instruction features are obtained by fusing the instruction features and their corresponding visual contexts.

[0029] As a further technical solution, the enhanced command features and visual features are input into a unified Transformer encoder for deep fusion to generate global multimodal fusion features and auxiliary decision signals, including the following process:

[0030] The enhanced panoramic area features, object features, command features, navigation history features, map node features and special tokens are combined to form a unified input sequence.

[0031] Input the sequence into the Transformer encoder for deep cross-modal interaction to obtain global multimodal fusion features and auxiliary decision signals;

[0032] Among them, the auxiliary decision-making signals include map node target probability distribution and uncertainty signals.

[0033] As a further technical solution, the process of using coarse-grained expert units and fine-grained expert units to process the global multimodal fusion features and auxiliary decision signals to obtain the first comprehensive prediction value and the second comprehensive prediction value is as follows:

[0034] The coarse-grained expert unit combines the global multimodal fusion features and the map node target probability distribution to predict the next target node or stop on the global topology map and output the first comprehensive prediction value:

[0035] The fine-grained expert unit combines the global multimodal fusion features and regional matching information, selects an action or judges to stop in the current local action space, and outputs a second comprehensive prediction value.

[0036] A second aspect of the present invention provides a visual language navigation system based on collaborative alignment and adaptive fusion.

[0037] A visual language navigation system based on collaborative alignment and adaptive fusion, including:

[0038] A feature extraction module is configured to: obtain natural language instructions, panoramic views, navigation history, and dynamic topology map data, and extract instruction features, visual features, navigation history features, and map node features;

[0039] A pre-fusion alignment module is configured to: perform feature enhancement pre-alignment on the command features and the visual features to obtain enhanced command features and visual features;

[0040] The feature fusion module is configured to: input the enhanced command features and visual features into the unified Transformer encoder for deep fusion to generate global multimodal fusion features and auxiliary decision signals;

[0041] The navigation action prediction module is configured to: input the global multimodal fusion features and auxiliary decision signals into a coarse- and fine-grained expert network to output the final predicted navigation action; wherein the global multimodal fusion features and auxiliary decision signals are processed by coarse-grained expert units and fine-grained expert units respectively to obtain a first comprehensive prediction value and a second comprehensive prediction value; construct a multi-source uncertainty vector to dynamically generate the weights of the coarse- and fine-grained expert network prediction results, and fuse the first comprehensive prediction value and the second comprehensive prediction value based on the weights to obtain the final predicted navigation action.

[0042] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of the visual language navigation method based on collaborative alignment and adaptive fusion as described in the first aspect of the present invention.

[0043] The fourth aspect of the present invention provides an electronic device, comprising a memory, a processor, and a program stored in the memory and runnable on the processor, wherein when the processor executes the program, the steps in the visual language navigation method based on collaborative alignment and adaptive fusion as described in the first aspect of the present invention are implemented.

[0044] One or more of the above technical solutions have the following beneficial effects:

[0045] (1) This invention uses an innovative two-level pre-alignment mechanism to explicitly establish a fine-grained correspondence between language instructions and visual observations before feature fusion, effectively solving the modality gap problem in traditional methods. It improves the accuracy of understanding complex instructions and enhances the ability to handle instructions containing multiple landmarks and complex spatial relationships.

[0046] (2) The present invention constructs a multi-source uncertainty vector and dynamically adjusts the fusion weights of coarse-grained and fine-grained navigation experts based on it, so that the model can make adaptive decisions based on the reliability of different information sources and show stronger fault tolerance in uncertain scenarios such as environmental occlusion and command ambiguity, thereby improving the stability and success rate of navigation.

[0047] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0049] Figure 1 This is a flow chart of the method of the first embodiment.

[0050] Figure 2 This is a diagram of the overall method architecture in the first embodiment.

[0051] Figure 3 Schematic diagram of the feature enhancement pre-alignment process in the first embodiment.

[0052] Figure 4 This is a schematic diagram of the expert network structure combining coarse and fine granularity in the first embodiment.

[0053] Figure 5 This is a system structure diagram of the second embodiment. DETAILED DESCRIPTION

[0054] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0055] It should be noted that the terms used herein are for describing particular embodiments only and are not intended to limit the exemplary embodiments according to the present invention.

[0056] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0057] Example 1

[0058] This embodiment discloses a visual language navigation method based on collaborative alignment and adaptive fusion. The visual language navigation (VLN) task requires the agent to follow natural language instructions. , in discrete, can be represented as a graph Navigate to the target location in the environment and locate the target object described in the instruction when necessary. , the agent receives instructions and panoramic views (Extract panoramic area features from and object features ), navigation history features and dynamic topological map data (Node Features ). The agent’s action space includes moving to a neighboring connected node or performing a stop action.

[0059] Combining the above content, such as Figure 1 and Figure 2 As shown in the figure, the visual language navigation method based on collaborative alignment and adaptive fusion includes:

[0060] Step S1, obtaining natural language instructions, panoramic views, navigation history and dynamic topological map data, and extracting instruction features, visual features, navigation history features and map node features;

[0061] Step S2, performing feature enhancement and pre-alignment on the command features and visual features to obtain enhanced command features and visual features;

[0062] Step S3: Input the enhanced command features and visual features into the unified Transformer encoder for deep fusion to generate global multimodal fusion features and auxiliary decision signals;

[0063] Step S4, input the global multimodal fusion features and auxiliary decision signals into the coarse-grained and fine-grained expert network, and output the final predicted navigation action; wherein the global multimodal fusion features and auxiliary decision signals are processed by the coarse-grained expert unit and the fine-grained expert unit respectively to obtain the first comprehensive prediction value and the second comprehensive prediction value; construct a multi-source uncertainty vector to dynamically generate the weights of the coarse-grained and fine-grained expert network prediction results, and fuse the first comprehensive prediction value and the second comprehensive prediction value based on the weights to obtain the final predicted navigation action.

[0064] Furthermore, it also includes the following:

[0065] In step S1, natural language instructions, panoramic views, navigation history and dynamic topological map data are obtained, and instruction features, visual features, navigation history features and map node features are extracted through multimodal coding and mapping. In order to better distinguish global intent from local landmarks, a two-layer instruction encoding is used to extract the natural language instructions. Extract instruction tokens and landmark words , and concatenate them into an extended sequence containing special tokens The sequence is fed into the pre-trained language model BERT (Bidirectional Encoder Representations from Transformers) for encoding to obtain structured instruction features. , where the structured instruction features Including global representation , original instruction embedding and landmark embedding . This can provide more focused text input for subsequent specific alignment tasks.

[0066] Furthermore, from the current panoramic view In the paper, a pre-trained neural network model CLIP (Contrastive Language-Image Pre-Training) for matching images and text is used to extract visual features, where the visual features include panoramic area features with directional embeddings. , such as 36 fixed viewpoints that are evenly divided, and the DETR (Detection Transformer) model for target detection and panoptic segmentation is used to extract object features with directional embeddings At the same time, a special "stop" visual correspondence is introduced, that is, a stop option is set in the action, and a corresponding stop visual correspondence is introduced.

[0067] will be recently Step visual embedding and corresponding action embedding Encoded into navigation history features and . Among them, the visual embedding is from 、 The pooled representation of .

[0068] At the same time, dynamically build a topological map , extract node features , node features It integrates the visual summary, relative position and navigation state encoding of nodes to provide support for global planning.

[0069] Combine Figure 3 In step S2, in order to alleviate the modality gap, targeted pre-alignment is performed before the feature cores of different modalities interact; specifically, feature enhancement pre-alignment of instruction features and visual features is performed through instruction-history alignment and landmark-observation alignment respectively.

[0070] Among them, in the instruction-history alignment, first, from the navigation history features Aggregate to get historical context summary . Then, a summary with historical context is the query vector (Query), for the instruction feature Perform attention calculation. Specifically, first calculate the attention score :

[0071]

[0072] in is a learnable linear projection matrix, which takes instruction features as key vectors or value vectors, and historical context summaries As the query vector, projected onto dimensional space.

[0073] Get the normalized attention weight through the Softmax function , which indicates the importance of each token in the current historical context instruction:

[0074]

[0075] Finally, using this attention distribution To enhance the command features , output context-aware instruction features Specifically, the computation outputs context-aware instruction features The process includes:

[0076] First, calculate the attention-weighted instruction context representation:

[0077]

[0078] in, The focused instruction context; Structured instruction features In , the feature vector corresponding to the i-th token; is the length of the instruction sequence.

[0079] It is then fused with the original instruction features, and the output context-aware instruction features are obtained by splicing followed by a feed-forward network (FFN) and applying residual connections and layer normalization. :

[0080]

[0081] in, An enhancement function that uses historical attention to enhance instruction features.

[0082] Furthermore, the explicit association between key landmarks in the instruction and the current visual observation is achieved through landmark-observation alignment. Specifically:

[0083] First, from the enhanced instructions Extract candidate landmark phrases and their representations Then, through the text-to-visual attention mechanism, each landmark As the query vector, the panoramic region feature and object features with orientation embedding As key vector and value vector, calculate the alignment score between landmark and visual entity (as a key component of the multi-source uncertainty vector). Specifically,

[0084]

[0085]

[0086] in is a learnable projection matrix; It is a landmark Attention distribution on all objects. With landmarks The most relevant object visual context is:

[0087]

[0088] Landmark The highest matching score (or attention weight) with the object:

[0089]

[0090] will serve as Secondly, we also perform visual-to-text auxiliary alignment to enhance bidirectional understanding, and for salient visual objects , calculate its Attention distribution , get the same as the visual object Most relevant text landmark context .

[0091] Finally, we use these aligned context information and matching scores to fuse the original visual features with their corresponding textual context:

[0092]

[0093] Furthermore, all the calculated Put together, we get . is the set of all object features observed at the current time step t.

[0094] Similarly, repeat the above method to obtain .

[0095] Furthermore, the instruction features and their corresponding visual context are fused:

[0096] These enhanced landmark features can be used to update context-aware instruction features The corresponding part in the form of the final enhanced instruction features .

[0097] At the same time, the alignment loss is designed , based on the matching information here and Supervision. It can provide an additional, more direct supervisory signal to the model to "encourage" or "teach" the model to explicitly match landmark phrases in instructions with objects or regions in the visual scene during the pre-alignment stage.

[0098] In step S3, the enhanced instruction features , object features and instruction characteristics and navigation history features and map nodes Features and special tokens are concatenated to form a unified input sequence;

[0099] Add modality type embed for each token and local position embedding , forming a unified input sequence The sequence is fed into a layer Transformer encoder (e.g. ) for deep cross-modal interaction. The vector corresponding to the first token of the encoder output sequence As a global multimodal fusion feature .

[0100] In addition, in step S3, an auxiliary decision signal is generated, wherein the auxiliary decision signal includes the map node target probability distribution and uncertainty signals . Map node target probability distribution for:

[0101]

[0102] in, is a feature vector derived from the global features output by the core multimodal Transformer or other instruction-related output features; 、 is a learnable linear projection matrix; It is the map node feature after the core Transformer fusion;

[0103] The distribution For supervised auxiliary tasks And rough expert guidance.

[0104] In step S4, based on the global multimodal fusion feature And auxiliary decision signals to predict the final navigation action. Specifically:

[0105] Step S41, using a coarse-grained and fine-grained expert network to process the global multimodal fusion features and the auxiliary decision signal to obtain a first comprehensive prediction value and a second comprehensive prediction value;

[0106] Combine Figure 4 In the coarse-grained and fine-grained expert network, the coarse-grained expert unit is combined with the global multimodal fusion feature and map node target probability distribution , predict the next target node on the global topology graph or stop, and output the first comprehensive prediction value :

[0107]

[0108] Fine-grained expert unit combined with global multimodal fusion features And the regional matching information obtained from step S3, that is, the alignment score obtained by landmark-observation alignment, selects an action or judges to stop in the current local action space, and outputs the second comprehensive prediction value :

[0109]

[0110] Optional gate selection mechanism ( ) allows different types of experts to intelligently extract and focus on the subset of information most relevant to their task.

[0111] Step S42: construct a multi-source uncertainty vector to dynamically generate weights for prediction results of coarse and fine granularity expert networks.

[0112] In order to achieve more robust decision fusion, a multi-source uncertainty vector is constructed , which systematically aggregates the alignment scores between landmarks and visual entities , and introduce the statistical indicator attention entropy and map target distribution kurtosis Multiple upstream uncertainty indicators, such as CNN, provide comprehensive state awareness for fusion decision-making:

[0113]

[0114] Step S43: The first comprehensive prediction value and the second comprehensive prediction value are fused based on the weights to obtain a final predicted navigation action.

[0115] The fusion weight of coarse-grained and fine-grained expert prediction results no longer depends solely on the post-evaluation of expert output, but is constructed by the multi-source uncertainty vector Dynamically generated via a learnable gating network:

[0116]

[0117] In the first comprehensive prediction value and the second comprehensive prediction value The predictions of align to the common executable action space (get ), the final action probability is obtained by weighted fusion :

[0118]

[0119] This mechanism enables fusion decision-making to proactively perceive and adapt to the information quality and uncertainty status throughout the entire processing flow.

[0120] Finally, output the final action probability (including stopping probability ), the agent will choose the action with the highest probability implement.

[0121] Furthermore, in this embodiment, an end-to-end training method is adopted, and the overall model parameters are optimized by setting a loss function.

[0122] Among them, the behavior cloning method is used to maximize the real actions in the expert demonstration path. The probability of the final action output is used to train the model. , the navigation loss is defined as the negative log-likelihood:

[0123] Navigation loss For tasks such as REVERIE, add object recognition cross entropy loss , at this time the main navigation loss is .

[0124] To provide richer supervision signals, the landmark-observation alignment loss is introduced , acts on landmark-observation alignment, and encourages clear matching of landmarks and visual areas based on attention score / entropy regularization terms. Introducing map target loss , acting on If the true sub-target map node corresponding to the current navigation step can be inferred from the expert path , then cross entropy loss can be used for supervision:

[0125]

[0126] Therefore, the total loss after adjustment is for:

[0127]

[0128] in, and is the weight hyperparameter of each auxiliary loss, and its initial value will be adjusted through experiments to achieve the best performance. The gating network parameters that generate the fusion weights will be learned end-to-end through the total loss.

[0129] The training process involves initializing the visual and text encoder backbones using pre-trained weights and fine-tuning them on VLN task data. The remaining module parameters are trained from scratch. The optimization strategy uses the AdamW optimizer, combined with a cosine learning rate decay strategy with a linear warmup phase and gradient clipping to ensure training stability and effectiveness.

[0130] Example 2

[0131] This embodiment discloses a visual language navigation system based on collaborative alignment and adaptive fusion;

[0132] like Figure 5 As shown in the figure, the visual language navigation system based on collaborative alignment and adaptive fusion includes:

[0133] A feature extraction module is configured to: obtain natural language instructions, panoramic views, navigation history, and dynamic topology map data, and extract instruction features, visual features, navigation history features, and map node features;

[0134] A pre-fusion alignment module is configured to: perform feature enhancement pre-alignment on the command features and the visual features to obtain enhanced command features and visual features;

[0135] The feature fusion module is configured to: input the enhanced command features and visual features into the unified Transformer encoder for deep fusion to generate global multimodal fusion features and auxiliary decision signals;

[0136] The navigation action prediction module is configured to: input the global multimodal fusion features and auxiliary decision signals into a coarse- and fine-grained expert network to output the final predicted navigation action; wherein the global multimodal fusion features and auxiliary decision signals are processed by coarse-grained expert units and fine-grained expert units respectively to obtain a first comprehensive prediction value and a second comprehensive prediction value; construct a multi-source uncertainty vector to dynamically generate the weights of the coarse- and fine-grained expert network prediction results, and fuse the first comprehensive prediction value and the second comprehensive prediction value based on the weights to obtain the final predicted navigation action.

[0137] Example 3

[0138] The purpose of this embodiment is to provide a computer-readable storage medium.

[0139] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the visual language navigation method based on collaborative alignment and adaptive fusion as described in Example 1.

[0140] Example 4

[0141] The purpose of this embodiment is to provide an electronic device.

[0142] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps of the visual language navigation method based on collaborative alignment and adaptive fusion as described in Example 1 are implemented.

[0143] The steps involved in the apparatuses of Examples 2, 3, and 4 above correspond to those of Method Example 1. For detailed implementations, please refer to the relevant description of Example 1. The term "computer-readable storage medium" should be understood to mean a single medium or multiple media containing one or more instruction sets; it should also be understood to include any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and causing the processor to perform any method of the present invention.

[0144] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computer device. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0145] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. A visual language navigation method based on collaborative alignment and adaptive fusion, characterized by: include: Obtain natural language instructions, panoramic views, navigation history, and dynamic topological map data, and extract instruction features, visual features, navigation history features, and map node features; Performing feature enhancement and pre-alignment on the command features and the visual features to obtain enhanced command features and visual features; The enhanced command features and visual features are input into the unified Transformer encoder for deep fusion to generate global multimodal fusion features and auxiliary decision signals; The global multimodal fusion features and auxiliary decision signals are input into a coarse-grained and fine-grained expert network to output the final predicted navigation action; wherein the global multimodal fusion features and auxiliary decision signals are processed by a coarse-grained expert unit and a fine-grained expert unit respectively to obtain a first comprehensive prediction value and a second comprehensive prediction value, and the process is as follows: The coarse-grained expert unit combines the global multimodal fusion features and the target probability distribution of the map nodes to predict the next target node or stop on the global topology map and output the first comprehensive prediction value; The fine-grained expert unit combines the global multimodal fusion features and regional matching information, selects an action or determines to stop in the current local action space, and outputs a second comprehensive prediction value; A multi-source uncertainty vector is constructed to dynamically generate weights of the prediction results of the coarse-grained and fine-grained expert networks, and the first comprehensive prediction value and the second comprehensive prediction value are fused based on the weights to obtain the final predicted navigation action.

2. The visual language navigation method based on collaborative alignment and adaptive fusion according to claim 1, characterized in that: The extracting of instruction features, visual features, navigation history features and map node features includes: Using a two-layer instruction encoding method, instruction words and landmark words are extracted from natural language instructions, and the instruction words and landmark words are concatenated into an extended sequence containing special tokens; structured instruction features are obtained through encoding; The visual features include panoramic region features and object features; the panoramic region features with direction embedding and the object features with direction embedding are extracted from the panoramic view; The visual embedding and the corresponding action embedding are encoded into navigation history features through Transformer; a topological map is dynamically constructed and map node features are extracted.

3. The visual language navigation method based on collaborative alignment and adaptive fusion according to claim 2, characterized in that: The feature enhancement pre-alignment includes instruction-history alignment and landmark-observation alignment.

4. The visual language navigation method based on collaborative alignment and adaptive fusion according to claim 3, characterized in that: The instruction-history alignment includes: Aggregating the navigation history features to obtain a history context summary; Using the historical context summary as a query vector, performing attention calculation on the instruction feature to obtain the instruction feature attention weight; The attention weight is used to enhance the instruction feature to obtain a context-aware instruction feature.

5. The visual language navigation method based on collaborative alignment and adaptive fusion according to claim 3, characterized in that: The landmark-observation alignment includes: Extracting candidate landmark phrases and their representations from the context-aware instruction features, using the candidate landmark phrases and their representations as query vectors, using the panoramic region features and the object features with direction embedding as key vectors and value vectors, respectively, and calculating an alignment score between the landmarks and the visual entities; For a salient visual object, calculate its attention distribution to all landmark phrases to obtain the textual landmark context most relevant to the visual object; By utilizing the aligned context information and matching scores, enhanced panoramic region features and object features are obtained by fusing the original visual features and their corresponding textual contexts. The enhanced instruction features are obtained by fusing the instruction features and their corresponding visual contexts.

6. The visual language navigation method based on collaborative alignment and adaptive fusion according to claim 1, characterized in that: The process of inputting the enhanced command features and visual features into the unified Transformer encoder for deep fusion to generate global multimodal fusion features and auxiliary decision signals includes: The enhanced panoramic area features, object features, command features, navigation history features, map node features and special tokens are combined to form a unified input sequence. Input the sequence into the Transformer encoder for deep cross-modal interaction to obtain global multimodal fusion features and auxiliary decision signals; Among them, the auxiliary decision-making signals include map node target probability distribution and uncertainty signals.

7. A visual language navigation system based on collaborative alignment and adaptive fusion, characterized by: include: A feature extraction module is configured to: obtain natural language instructions, panoramic views, navigation history, and dynamic topology map data, and extract instruction features, visual features, navigation history features, and map node features; A pre-fusion alignment module is configured to: perform feature enhancement pre-alignment on the command features and the visual features to obtain enhanced command features and visual features; The feature fusion module is configured to: input the enhanced command features and visual features into the unified Transformer encoder for deep fusion to generate global multimodal fusion features and auxiliary decision signals; The navigation action prediction module is configured to: input the global multimodal fusion features and auxiliary decision signals into a coarse-grained and fine-grained expert network, and output the final predicted navigation action; wherein the global multimodal fusion features and auxiliary decision signals are processed by a coarse-grained expert unit and a fine-grained expert unit respectively to obtain a first comprehensive prediction value and a second comprehensive prediction value, and the process is as follows: The coarse-grained expert unit combines the global multimodal fusion features and the target probability distribution of the map nodes to predict the next target node or stop on the global topology map and output the first comprehensive prediction value; The fine-grained expert unit combines the global multimodal fusion features and regional matching information, selects an action or determines to stop in the current local action space, and outputs a second comprehensive prediction value; A multi-source uncertainty vector is constructed to dynamically generate weights of the prediction results of the coarse-grained and fine-grained expert networks, and the first comprehensive prediction value and the second comprehensive prediction value are fused based on the weights to obtain the final predicted navigation action.

8. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps of the visual language navigation method based on collaborative alignment and adaptive fusion as described in any one of claims 1 to 6 are implemented.

9. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the visual language navigation method based on collaborative alignment and adaptive fusion as described in any one of claims 1 to 6 are implemented.

Images