Visual language navigation continuous learning method based on macroscopic and microscopic double-routing experts

Abstract

The invention discloses a visual language navigation continuous learning method based on macroscopic and microscopic double-routing experts, and the method comprises the steps: combining macroscopic routing based on a global map with microscopic routing based on a hidden state of a navigation language model through constructing a macroscopic and microscopic double-routing expert model; dynamic expert routing weights are generated to activate respective expert modules. After the task is completed, a dynamic expert momentum updating mechanism is adopted to calculate the contribution degree of each expert, the experts are divided into a key category and a non-key category, and then through differentiated updating momentum merging parameters, the model can efficiently learn new knowledge while keeping old knowledge. The method has the characteristics of deep learning and continuous learning technologies, the disastrous forgetting is effectively relieved, and the stability and adaptability of the navigation agent in the continuous learning sequence are remarkably improved.

Description

Technical Field

[0001] This invention relates to the technical field of visual language navigation, and in particular to a continuous learning method for visual language navigation based on macroscopic and microscopic dual-route experts. Background Technology

[0002] In the field of visual language navigation, with the rapid growth of indoor environmental image data and the increasing complexity of practical application scenarios, how to enable navigation agents to navigate efficiently in dynamic environments based on natural language instructions has become a pressing technical challenge. Traditional navigation methods typically rely on multimodal fusion and sequential decision-making techniques. However, while navigation agents have made significant progress in instruction understanding and decision-making capabilities with the rapid development of large-scale language models, they face a catastrophic forgetting problem when dealing with new environments, leading to a decline in generalization performance in unseen scenarios.

[0003] In certain application scenarios, such as robot navigation or augmented reality, frequent environmental changes require navigation agents to continuously adapt to new domains. However, the inability to replay previous data raises issues of privacy protection and storage overhead. Therefore, the field of visual language navigation has proposed the concept of "continuous learning," aiming to enable navigation agents to sequentially learn non-overlapping sets of scenarios with a domain-incremental setting. The main goal of this approach is to reduce retraining costs and improve efficiency and feasibility in practical applications.

[0004] However, in continuous learning systems for visual language navigation, the navigation agent not only needs to handle global scene reasoning but also requires precise alignment of local perception. Therefore, how to efficiently adapt to new domains while avoiding forgetting old knowledge has become a crucial research challenge. Existing techniques, such as replay buffer-based or regularization-based strategies, are typically inefficient during knowledge transfer, leading to a decrease in navigation success rate. Furthermore, simple fine-tuning processes may lose the ability to specialize in topology and instruction alignment, further affecting the system's robustness and generalization ability.

[0005] Given the aforementioned problems, there is an urgent need for a new technical solution to effectively address the forgetting and adaptation bottlenecks faced by navigation agents during continuous learning, and to provide an effective solution for domain-incremental visual-language navigation tasks while ensuring efficient parameter usage. Therefore, developing a novel continuous learning method for visual-language navigation that optimizes expert selection and knowledge updating processes to improve the adaptability and knowledge retention of navigation agents has become a key research direction in this field. Summary of the Invention

[0006] The purpose of this invention is to solve the catastrophic forgetting problem caused by indiscriminate parameter updates during the continuous learning process of navigation agents. It provides a visual language navigation continuous learning method based on macro- and micro dual-route experts. By constructing a macro- and micro dual-route expert model to optimize single-step navigation decisions, and combining it with a dynamic expert momentum update mechanism, the parameters are merged differently according to the expert contribution after the task is completed. This allows for efficient learning of new domain knowledge while maximizing the retention of core navigation capabilities, effectively balancing the model's adaptability and knowledge retention ability, and significantly improving the long-term generalization performance of the navigation agent.

[0007] To achieve the above objectives, the technical solution provided by this invention is: a continuous learning method for visual language navigation based on macroscopic and microscopic dual-route experts, comprising the following steps: 1) Task sequence preparation and model initialization: Obtain a task sequence divided into domain incremental tasks; construct a navigation agent, which includes a navigation language model, a navigation action generation module, and a macro-micro dual-routing expert model; the macro-micro dual-routing expert model consists of macro-routing for handling scene topology relationships, micro-routing for handling fine-grained semantics of instructions, and several shared expert modules; and initialize the merging parameters of the navigation agent. The merging parameters Used as the first domain increment task The initial parameters; In processing the first Incremental tasks in individual domains At that time, the merge parameters from the previous task are used. As initial parameters, repeat steps 2) through 4). 2) Macro-level routing selection: During the navigation process of the current task, the visual features and topological adjacency matrix of the current navigation node are obtained; using the macro-routing, feature propagation based on topological relationships is performed on the visual features of the current navigation node, and task-related features are aggregated in combination with natural language navigation instructions, thereby generating macro-expert weights that reflect the macro-structure of the environment. 3) Micro-routing selection: Using the micro-routing, the hidden state reflecting the timing information of the instructions, generated based on the navigation language model, is directly mapped to generate micro-expert weights that reflect the micro-semantics of the instructions. 4) Expert activation and action generation: A dual-route fusion module is used, which is configured to adjust the decision contribution ratio of macro and micro routes through a preset balance parameter; the dual-route fusion module performs a linear weighted summation calculation on the macro expert weights obtained in step 2) and the micro expert weights obtained in step 3) to generate the final expert route weights; the final expert route weights are used to activate one or more of the several shared expert modules, and based on the output of the activated expert modules, the navigation action generation module generates the next navigation action; Complete the current domain incremental task After training, perform steps 5) and 6). 5) Calculation of expert contribution: After completing the training of the current domain incremental task, a dynamic expert momentum update mechanism is executed. That is, firstly, the final expert routing weights generated in step 4) are counted, the expert usage of each expert module in the current task is calculated, and the expert contribution distribution is obtained by normalization based on the expert usage. 6) Parameter merging and updating: Based on the expert contribution distribution described in step 5), the shared expert modules are divided into a task-critical expert set and a non-critical expert set; then, the merging parameters from the previous task are obtained. and incremental tasks in the current domain The task-specific parameters obtained by fine-tuning Different update momentum is applied to the set of key experts and the set of non-key experts for the task, and momentum-based parameter merging calculation is performed to generate new merged parameters for the next task. ; 7) Continuous learning and iteration: The new merge parameters generated in step 6) As the first Incremental tasks in individual domains The initial parameters for the next domain incremental task in the task sequence. Repeat steps 2) through 6.

[0008] Furthermore, in step 1), the task sequence is defined as follows: Each of these domain incremental tasks Each contains at least one navigation map. and at least one natural language navigation instruction , Indicates the total number of incremental tasks in the domain; Get task-specific parameters The process is defined as: incremental tasks in the current domain Training is performed on the target to obtain task-specific parameters. The mathematical expression is as follows: ; In the formula, Indicates the first Incremental tasks for each domain; This represents a navigation graph, used to define the structure of the navigation environment; This represents a natural language navigation instruction; Indicates in the task The task-specific parameters obtained through fine-tuning are then used. This represents an optimization operation used to find parameters that minimize the objective function. ; Indicates the task All navigation maps sampled in the middle and natural language navigation instructions Take the expected value; Indicates the parameter Navigation strategy All generated navigation tracks Take the expected value; Indicates navigation trajectory All navigation time steps Perform summation; Indicates the navigation time step Navigation agent state and the actions performed The loss function value is calculated below; Generate new merge parameters The process uses a merge operator. The task-specific parameters Merging parameters from the previous task The merging and calculation formula is as follows: ; In the formula, Indicates completion of the task The newly generated merge parameters; This indicates the merge parameters from the previous task; This represents a merge operator used to merge two sets of parameters.

[0009] Furthermore, in step 2), the topology-based feature propagation of the visual features of the current navigation node is performed using a graph neural network function. Based on the topological adjacency matrix of the navigation nodes and visual features A topology-sensing representation is calculated. The mathematical expression is as follows: ; In the formula, Represents a graph neural network function; This represents the navigation time step.

[0010] Furthermore, in step 2), the aggregation of task-related features first involves calculating the attention score of each navigation node feature in the topology-aware representation relative to the instruction. The mathematical expression is as follows: ; In the formula, Represents the set of navigation nodes One of the nodes; Represents a set of navigation nodes Up-normalization function; Represents topological sensing representation Corresponding node The representation; An embedding vector representing the natural language navigation instructions; This represents the matrix transpose operation; Based on the attention score A composite scene representation is calculated. The mathematical expression is as follows: .

[0011] Furthermore, in step 2), the process of generating macro-expert weights is based on the aggregation scenario representation. The generation, mathematically expressed as follows: ; In the formula, Represents the weight of macroeconomic experts; Represents a normalization function; This represents a multilayer perceptron function; This represents the aggregated scene representation.

[0012] Furthermore, in step 3), the process of generating micro-expert weights is based on the hidden states output by the navigation language model. The calculation yields the following mathematical expression: ; In the formula, Represents the weight of micro-level experts; Represents a normalization function; This represents a multilayer perceptron function.

[0013] Furthermore, in step 4), the process of generating the final expert routing weights combines macro-level expert weights. and micro-expert weight The result of fusion is as follows, expressed mathematically: ; In the formula, Represents the final expert routing weight; This represents a hyperparameter used to balance macro-expert weights and micro-expert weights.

[0014] Furthermore, in step 5), the calculations for each expert module are performed. Expert usage The process is based on all the final expert routing weights accumulated in step 4). The calculation yields the following mathematical expression: ; In the formula, Indicates the first The usage of each expert module; Indicates an incremental task from the current domain. The input corresponding to a sample; Indicates in the sample The final expert routing weight vector Chinese corresponding expert module The weighted components; Indicates the index of the expert module; Distribution of Expert Contributions The calculation process is based on the amount of expert usage. The result after normalization is expressed mathematically as follows: ; In the formula, The expert module Contribution to the current domain incremental tasks; Indicates the first The usage of each expert module; This represents the summation of usage across all experts; Indicates the number of expert modules; Indicates the index of the expert module, ranging from 1 to .

[0015] Furthermore, in step 6), the momentum-based parameter merging and updating process first utilizes the expert contribution distribution calculated in step 5). Multiple expert modules Divided into task-critical expert sets With non-critical expert group The mathematical expression is as follows: ; In the formula, This indicates the distribution of expert contributions. Select before The set of key experts for the mission is obtained by selecting the expert who made the highest contribution. Indicates before selection The operation of the largest component; This indicates the preset number of key experts for the task; The non-critical expert set The mathematical expression for the task-critical expert set obtained by excluding the entire expert set is as follows: ; In the formula, Represents the set of all expert modules, where This indicates the first expert module. This indicates the second expert module, and so on up to the third. One expert module; Indicates the number of expert modules; \ indicates set difference operation; Subsequently, different update momentum coefficients were assigned to the task-critical expert set and the non-critical expert set, respectively. The mathematical expression is as follows: ; In the formula, Representation of expert modules The updated momentum coefficient; This represents the preset momentum control coefficient and satisfies... ; Based on the updated momentum coefficient Merging parameters from the previous task and the task-specific parameters of the current domain incremental task Perform momentum-based parameter merging to generate new merged parameters for the next task. The mathematical expression is as follows: ; In the formula, Indicates based on the first New merge parameters generated by the incremental task in each domain; This represents the task-specific parameters obtained through fine-tuning in the current domain incremental task; This represents a vector consisting of the updated momentum coefficients of each expert module and is broadcast along the parameter dimension; 1 indicates that... Vectors of all 1s with the same shape; This indicates element-wise multiplication.

[0016] Compared with the prior art, the present invention has the following advantages and beneficial effects: This invention introduces a macro- and micro-level dual-routing expert model. In this model, macro-routing is based on topological reasoning using a global map, providing long-term planning for decision-making; micro-routing focuses on the current local perception state, ensuring immediate responsiveness and scenario fit in decision-making. Compared to traditional methods that rely on a single information source for decision-making, this invention's macro- and micro-level dual-routing expert model achieves dynamic coordination between global planning and local perception, enabling the navigation agent to make more reasonable navigation actions that align with long-term mission objectives, significantly reducing the navigation failure rate caused by decision-making errors.

[0017] This invention proposes a dynamic expert momentum update mechanism. After completing a domain incremental task, this mechanism can quantitatively evaluate the contribution of each expert module in the current task and perform differentiated parameter merging based on the contribution. By assigning different update momentum to key and non-key experts, this invention preserves the historical parameters of key experts to a greater extent, while allowing non-key experts to absorb new knowledge more fully. Compared with traditional fully fine-tuned or fixed regularization methods, this differentiated parameter merging strategy achieves a better balance between adaptability and stability. Without relying on historical data replay, it maximizes the preservation of core navigation capabilities and significantly improves the long-term learning performance and cross-domain generalization ability of the navigation agent.

[0018] In summary, this invention possesses the characteristics of visual language navigation and continuous learning technology. By combining a unique macro- and micro dual-route expert model with dynamic parameters, it effectively solves the catastrophic forgetting problem in domain incremental learning of navigation agents, and simultaneously improves the accuracy of navigation decisions and knowledge retention capabilities. It has practical application value and is worthy of promotion. Attached Figure Description

[0019] Figure 1 This is a flowchart of the method of the present invention.

[0020] Figure 2 This is a schematic diagram of the principle framework of the method of the present invention. Detailed Implementation

[0021] The present invention will be further described below with reference to specific embodiments.

[0022] This embodiment discloses a visual language navigation continuous learning method based on macro- and micro-level dual-route experts. This method constructs a macro- and micro-level dual-route expert model to optimize single-step navigation decisions and combines it with a dynamic expert momentum update mechanism. After the task is completed, differentiated parameters are merged based on expert contributions, thereby maximizing the retention of core navigation capabilities while efficiently learning new domain knowledge. Figure 1 As shown, the method of this invention, on the R2R (Room-to-Room) visual language navigation dataset, first performs task sequence preparation and model initialization; in each navigation step of the task processing, macro-routing selection, micro-routing selection, and expert activation and action generation are performed sequentially; after completing the training of the current task, the expert contribution is calculated based on the accumulated weights, and parameter merging and updating are performed accordingly, finally entering the continuous learning iteration of the next task. Its specific steps include: 1) Task sequence preparation and model initialization: This embodiment uses the R2R (Room-to-Room) visual-language navigation dataset to construct continuous learning tasks and initialize the model. The R2R dataset is a typical indoor visual-language navigation dataset, including 72 real-world indoor scenes and a total of 21,567 path-natural language navigation command pairs. The training set contains 14,025 paths from 61 scenes, the validation set contains 1,017 paths from 11 scenes, and the unseen validation set contains 3,400 paths from 18 novel scenes. Let's denote the R2R (Room-to-Room) visual-language navigation dataset as... To simulate the cross-scenario domain incremental learning process, in The scene set is divided into disjoint parts to form Each subset of scenarios is a non-overlapping set, and each subset corresponds to a domain incremental task. ,in The navigation agent learns incremental tasks in each domain strictly according to the task sequence. When learning the current task, it can only access the data of the current task and cannot access the scene and natural language navigation instruction data belonging to the previous task, thus ensuring a strict continuous learning setup.

[0023] After constructing the task sequence, the basic model structure of the navigation agent is initialized. The navigation agent includes: a navigation language model for processing natural language navigation commands, a navigation action generation module for generating navigation actions, and a macro- and micro-level dual-route expert model. The macro- and micro-level dual-route expert model consists of multiple expert modules. The system is structured so that each expert can be activated on demand in subsequent steps, enabling hierarchical decision-making based on a dual-routing mechanism. The initial parameters of the navigation agent are set to merged parameters. This parameter serves as the first domain incremental task. The initial parameters are used to initialize the entire navigation agent before the first training.

[0024] In the process of learning task sequences, for task sequences Each domain incremental task The task data must contain at least one navigation map. and a natural language navigation command , Indicates the total number of domain incremental tasks. (Navigation graph) Structured topological relationships representing task scenarios, natural language instructions Used to specify the current navigation target. For learning tasks. Its unique ability to acquire task-specific parameters during its training phase. , The navigation agent is fine-tuned using the current task data, with the training objective being to minimize the cumulative loss of the navigation trajectory. The mathematical expression is as follows: ; In the formula, Indicates the first Incremental tasks for each domain; This represents a navigation map; Represents natural language navigation instructions; Indicates in the task The task-specific parameters obtained above; Indicates by parameters Defined navigation strategy; Indicates the navigation trajectory; For the first Navigation time step status With the execution of actions The loss value below; This represents the cumulative summation of all navigation steps in the trajectory; Indicates the task All navigation maps sampled in the middle and natural language navigation instructions Take the expected value; Indicates the parameter Navigation strategy All generated navigation tracks Take the expected value.

[0025] Obtaining task-specific parameters Then, through the merging operator Combine its parameters with those of the previous task Perform parameter merging to generate new merged parameters for use in the next task. The calculation is as follows: ; In the formula, This indicates the new merge parameters after the current task is completed; This indicates the merge parameters retained from the previous task; Indicates task-based Task-specific parameters learned; An operator for merging two types of parameters is used to provide parameter initialization that incorporates both historical and new knowledge for subsequent tasks.

[0026] After completing the above steps, the navigation agent is ready to enter the subsequent task processing flow.

[0027] 2) Macro-level routing selection: In processing incremental tasks in the current domain At each navigation time step, the navigation agent uses merged parameters from the previous task. (For the first task, these are the initial parameters) ) are used as basic parameters for reasoning. For example... Figure 2 As shown, this step aims to capture environmental structure information and perform long-term planning using a global map.

[0028] First, in the navigation time step The navigation agent obtains the visual features of the navigation node at its current location. Adjacency matrix of navigation nodes The topological adjacency matrix of this navigation node. This reflects the connectivity between currently known nodes (including visited nodes and border nodes that have been probed but not visited).

[0029] Subsequently, the propagation of topological relationship features of navigation nodes is performed. Specifically, this is done using a graph neural network function. (This embodiment uses a graph convolutional network) to generate the topological adjacency matrix of the navigation nodes. and the visual features of the navigation nodes As input, global structural information of the environment is fused through a message passing mechanism to calculate a topology-aware representation. The mathematical expression is as follows: ; In the formula, This represents the node representation matrix after topological propagation; Represents a graph neural network function; Indicates the first The topological adjacency matrix of navigation nodes at each navigation time step; This represents the visual characteristics of navigation nodes.

[0030] Next, by aggregating features relevant to the execution of natural language navigation instructions, the most relevant parts to the navigation target in the current observations are determined. This process first calculates the attention score of each navigation node feature in the topology-aware representation relative to the natural language navigation instructions. The mathematical expression is as follows: ; In the formula, Represents the attention score; Represents the set of navigation nodes; Represents the set of navigation nodes One of the nodes; Represents one in the set of navigation nodes Up-normalization function; Represents the topology-aware representation Corresponding node The representation; An embedding vector representing the natural language navigation instructions; This represents the matrix transpose operation; Furthermore, based on the calculated attention score The features of each node are weighted and aggregated to obtain an aggregated scene representation that reflects the current global environment state and focuses on the task objective. The mathematical expression is as follows: ; In the formula, This represents the aggregated scenario representation; Represents the set of navigation nodes; Represents the set of navigation nodes One of the nodes; Represents a set All nodes Perform summation; Representing the nodes The representation of.

[0031] Finally, based on the aggregated scene representation Generate macro expert weights This weight determines the priority of participation of each expert module at the macro-planning level, and its mathematical expression is as follows: ; In the formula, Represents the aforementioned macroeconomic expert weights; Represents a normalization function; This represents a multilayer perceptron function; This represents the aggregated scenario.

[0032] 3) Micro-routing selection: While macro-routing focuses on the global topology, a micro-routing approach is used to capture local, fine-grained semantics and immediate state information during navigation. This step aims to leverage the hidden states output by the navigation language model. Enables context-sensitive selection of experts.

[0033] Specifically, the generation of micro-expert weights The process is based on the hidden state generated by the navigation language model. The calculation is as follows: For each step in the navigation process, the hidden state output by the language model is input into the micro-routes, and a gating distribution for each expert is generated through mapping. The mathematical expression is as follows: ; In the formula, This represents the aforementioned micro-expert weights; Represents a normalization function; This represents a multilayer perceptron function.

[0034] 4) Expert activation and action generation: Using a dual-route fusion module, combined with the macro-expert weights described in step 2), And the micro-expert weights described in step 3) Generate a final expert routing weight. and using the final expert routing weights Activate one or more expert modules Then, using the activated expert module... The output is used to generate the next navigation action through the navigation action generation module.

[0035] Generate final expert route weights The process involves combining the aforementioned macro-expert weights. and the aforementioned micro-expert weights The result of the fusion is expressed mathematically as follows: ; In the formula, This represents the final expert routing weight; This represents a hyperparameter used to balance the macro-expert weights and the micro-expert weights; Represents the macroeconomic expert weights mentioned in step 2); This represents the micro-expert weights described in step 3).

[0036] Complete the current domain incremental task After training, perform steps 5) and 6). 5) Calculation of expert contribution: A dynamic expert momentum update mechanism is implemented, which is first based on all the final expert route weights accumulated in step 4). Calculate each expert module Expert usage And based on the expert usage A distribution of expert contributions was calculated. .

[0037] Calculate each expert module Expert usage The process is based on all the final expert routing weights accumulated in step 4). The calculated result is expressed mathematically as follows: ; In the formula, Indicates the first The usage of each expert module; Indicates an incremental task from the current domain. The input corresponding to a sample; Indicates in the sample The final expert routing weight vector Chinese corresponding expert module The weighted components; Indicates the task Summing all samples in the dataset; Indicates the index of the expert module.

[0038] Distribution of Expert Contributions The calculation process is based on the amount of expert usage. The result after normalization is expressed mathematically as follows: ; In the formula, The expert module Contribution to the current domain incremental tasks; Indicates the first The usage of each expert module; This represents the summation of usage across all experts; Indicates the number of expert modules; Indicates the index of the expert module, ranging from 1 to .

[0039] 6) Parameter merging and updating: Based on the expert contribution distribution described in step 5) Multiple expert modules Divided into a task-critical expert set A non-critical expert group Then, obtain the merge parameters from the previous task. and incremental tasks in the current domain The task-specific parameters obtained by fine-tuning , for the task-critical expert group Non-critical expert set Applying different update momentum Perform momentum-based parameter merging calculations to generate new merged parameters for the next task. .

[0040] The momentum-based parameter merging and updating process first utilizes the expert contribution distribution calculated in step 5). Multiple expert modules Divided into task-critical expert sets With non-critical expert group The mathematical expression is as follows: ; In the formula, This indicates the distribution of expert contributions. Select before The set of key experts for the mission is obtained by selecting the expert who made the highest contribution. Indicates before selection The operation of the largest component; This represents the distribution of expert contributions obtained in step 5). This indicates the preset number of key experts for the task.

[0041] Non-critical expert set The mathematical expression for the task-critical expert set obtained by excluding the entire expert set is as follows: ; In the formula, This refers to the set of non-critical experts; Represents the set of all expert modules, where This indicates the first expert module. This indicates the second expert module, and so on up to the third. One expert module; Indicates the number of expert modules; \ indicates set difference operation.

[0042] Subsequently, different update momentum coefficients were assigned to the set of key experts and the set of non-key experts. The mathematical expression is as follows: ; In the formula, Representation of expert modules The updated momentum coefficient; This represents the preset momentum control coefficient and satisfies... .

[0043] Based on the updated momentum coefficient Merging parameters from the previous task and the task-specific parameters of the current domain incremental task Perform momentum-based parameter merging to generate new merged parameters for the next task. The mathematical expression is as follows: ; In the formula, Indicates based on the first New merge parameters generated by the incremental task in each domain; Indicates the merging parameters of the previous task; This represents the task-specific parameters obtained through fine-tuning in the current domain incremental task; This represents a vector consisting of the updated momentum coefficients of each expert module and is broadcast along the parameter dimension. Indicates the number of expert modules; 1 indicates the number of modules. Vectors of all 1s with the same shape; This indicates element-wise multiplication.

[0044] 7) Continuous learning and iteration: The new merging parameters generated in step 6) As the first Incremental tasks in individual domains The initial parameters. The next domain incremental task in the task sequence. Repeat steps 2) through 6). Continue this cycle until the task sequence is complete. middle Learning from each task. Through this iterative mechanism, the navigation agent can effectively transfer the knowledge accumulated in previous tasks to subsequent tasks while continuously adapting to new domain environments, achieving continuous performance improvement and resistance to forgetting.

[0045] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A visual-language navigation continuous learning method based on macro- and micro-level dual-route experts, characterized in that: Includes the following steps: 1) Task sequence preparation and model initialization: Obtain a task sequence divided into domain incremental tasks; construct a navigation agent, which includes a navigation language model, a navigation action generation module, and a macro-micro dual-routing expert model; the macro-micro dual-routing expert model consists of macro-routing for handling scene topology relationships, micro-routing for handling fine-grained semantics of instructions, and several shared expert modules; and initialize the merging parameters of the navigation agent. The merging parameters Used as the first domain increment task The initial parameters; In processing the first Incremental tasks in individual domains At that time, the merge parameters from the previous task are used. As initial parameters, repeat steps 2) through 4). 2) Macro-level routing selection: During the navigation process of the current task, the visual features and topological adjacency matrix of the current navigation node are obtained; using the macro-routing, feature propagation based on topological relationships is performed on the visual features of the current navigation node, and task-related features are aggregated in combination with natural language navigation instructions, thereby generating macro-expert weights that reflect the macro-structure of the environment. 3) Micro-routing selection: Using the micro-routing, the hidden state reflecting the timing information of the instructions, generated based on the navigation language model, is directly mapped to generate micro-expert weights that reflect the micro-semantics of the instructions. 4) Expert activation and action generation: A dual-route fusion module is used, which is configured to adjust the decision contribution ratio of macro and micro routes through a preset balance parameter; the dual-route fusion module performs a linear weighted summation calculation on the macro expert weights obtained in step 2) and the micro expert weights obtained in step 3) to generate the final expert route weights; the final expert route weights are used to activate one or more of the several shared expert modules, and based on the output of the activated expert modules, the navigation action generation module generates the next navigation action; To complete the current domain incremental task After training, perform steps 5) and 6). 5) Calculation of expert contribution: After completing the training of the current domain incremental task, a dynamic expert momentum update mechanism is executed. That is, firstly, the final expert routing weights generated in step 4) are counted, the expert usage of each expert module in the current task is calculated, and the expert contribution distribution is obtained by normalization based on the expert usage. 6) Parameter merging and updating: Based on the expert contribution distribution described in step 5), the shared expert modules are divided into a task-critical expert set and a non-critical expert set; then, the merging parameters from the previous task are obtained. and incremental tasks in the current domain The task-specific parameters obtained by fine-tuning Different update momentum is applied to the set of key experts and the set of non-key experts for the task, and momentum-based parameter merging calculation is performed to generate new merged parameters for the next task. ; 7) Continuous learning and iteration: The new merge parameters generated in step 6) As the first Incremental tasks in individual domains The initial parameters for the next domain incremental task in the task sequence. Repeat steps 2) through 6.

2. The visual language navigation continuous learning method based on macro- and micro-level dual-route experts according to claim 1, characterized in that: In step 1), the task sequence is defined as follows: Each of these domain incremental tasks Each contains at least one navigation map. and at least one natural language navigation instruction , Indicates the total number of incremental tasks in the domain; Get task-specific parameters The process is defined as: incremental tasks in the current domain Training is performed on the target to obtain task-specific parameters. The mathematical expression is as follows: ； In the formula, Indicates the first Incremental tasks for each domain; This represents a navigation graph, used to define the structure of the navigation environment; This represents a natural language navigation instruction; Indicates in the task The task-specific parameters obtained through fine-tuning are then used. This represents an optimization operation used to find parameters that minimize the objective function. ; Indicates the task All navigation maps sampled in the middle and natural language navigation instructions Take the expected value; Indicates the parameter Navigation strategy All generated navigation tracks Take the expected value; Indicates navigation trajectory All navigation time steps Perform summation; Indicates the navigation time step Navigation agent state and the actions performed The loss function value is calculated below; Generate new merge parameters The process uses a merge operator. The task-specific parameters Merging parameters from the previous task The merging and calculation formula is as follows: ； In the formula, Indicates completion of the task The newly generated merge parameters; This indicates the merge parameters from the previous task; This represents a merge operator used to merge two sets of parameters.

3. The visual language navigation continuous learning method based on macro- and micro-level dual-route experts according to claim 1, characterized in that: In step 2), the topology-based feature propagation of the visual features of the current navigation node is performed using a graph neural network function. Based on the topological adjacency matrix of the navigation nodes and visual features A topology-sensing representation is calculated. The mathematical expression is as follows: ； In the formula, Represents a graph neural network function; This represents the navigation time step.

4. The visual language navigation continuous learning method based on macro- and micro-level dual-route experts according to claim 1, characterized in that: In step 2), the aggregation of task-related features first involves calculating the attention score of each navigation node feature in the topology-aware representation relative to the instruction. The mathematical expression is as follows: ； In the formula, Represents the set of navigation nodes One of the nodes; Represents a set of navigation nodes Up-normalization function; Represents topological sensing representation Corresponding node The representation; An embedding vector representing the natural language navigation instructions; This represents the matrix transpose operation; Based on the attention score A composite scene representation is calculated. The mathematical expression is as follows: 。 5. The visual language navigation continuous learning method based on macro- and micro-level dual-route experts according to claim 1, characterized in that: In step 2), the process of generating macro-expert weights is based on the aggregated scenario representation. The generation, mathematically expressed as follows: ； In the formula, Represents the weight of macroeconomic experts; Represents a normalization function; This represents a multilayer perceptron function; This represents the aggregated scene representation.

6. The visual language navigation continuous learning method based on macro- and micro-level dual-route experts according to claim 1, characterized in that: In step 3), the process of generating micro-expert weights is based on the hidden states output by the navigation language model. The calculation yields the following mathematical expression: ； In the formula, Represents the weight of micro-level experts; Represents a normalization function; This represents a multilayer perceptron function.

7. The visual language navigation continuous learning method based on macro- and micro-level dual-route experts according to claim 1, characterized in that: In step 4), the process of generating the final expert routing weights combines macro-level expert weights. and micro-expert weight The result of fusion is as follows, expressed mathematically: ； In the formula, Represents the final expert routing weight; This represents a hyperparameter used to balance macro-expert weights and micro-expert weights.

8. The visual language navigation continuous learning method based on macro- and micro-level dual-route experts according to claim 1, characterized in that: In step 5), the calculations for each expert module are performed. Expert usage The process is based on all the final expert routing weights accumulated in step 4). The calculation yields the following mathematical expression: ； In the formula, Indicates the first The usage of each expert module; Indicates an incremental task from the current domain. The input corresponding to a sample; Indicates in the sample The final expert routing weight vector Chinese corresponding expert module The weighted components; Indicates the index of the expert module; Distribution of Expert Contributions The calculation process is based on the amount of expert usage. The result after normalization is expressed mathematically as follows: ； In the formula, The expert module Contribution to the current domain incremental tasks; Indicates the first The usage of each expert module; This represents the summation of usage across all experts; Indicates the number of expert modules; Indicates the index of the expert module, ranging from 1 to .

9. The visual language navigation continuous learning method based on macro- and micro-level dual-route experts according to claim 1, characterized in that: In step 6), the momentum-based parameter merging and updating process first utilizes the expert contribution distribution calculated in step 5). Multiple expert modules Divided into task-critical expert sets With non-critical expert group The mathematical expression is as follows: ； In the formula, This indicates the distribution of expert contributions. Select before The set of key experts for the mission is obtained by selecting the expert who made the highest contribution. Indicates before selection The operation of the largest component; This indicates the preset number of key experts for the task; The non-critical expert set The mathematical expression for the task-critical expert set obtained by excluding the entire expert set is as follows: ； In the formula, Represents the set of all expert modules, where This indicates the first expert module. This indicates the second expert module, and so on up to the third. One expert module; Indicates the number of expert modules; \ indicates set difference operation; Subsequently, different update momentum coefficients were assigned to the task-critical expert set and the non-critical expert set, respectively. The mathematical expression is as follows: ； In the formula, Representation of expert modules The updated momentum coefficient; This represents the preset momentum control coefficient and satisfies... ; Based on the updated momentum coefficient Merging parameters from the previous task and the task-specific parameters of the current domain incremental task Perform momentum-based parameter merging to generate new merged parameters for the next task. The mathematical expression is as follows: ； In the formula, Indicates based on the first New merge parameters generated by the incremental task in each domain; This represents the task-specific parameters obtained through fine-tuning in the current domain incremental task; This represents a vector consisting of the updated momentum coefficients of each expert module and is broadcast along the parameter dimension; 1 indicates that... Vectors of all 1s with the same shape; This indicates element-wise multiplication.

Images