Prototype-evolution-based lifelong visual language navigation method and system

Abstract

The application provides a lifelong visual language navigation method and system based on prototype evolution, and relates to the technical field of robot navigation. The method first constructs a dynamic prototype library, dynamically extends and gradient updates the prototype library, fuses a plurality of related prototypes in the prototype library to obtain an aggregated prototype, designs a prototype conditional hypernetwork, dynamically generates step-level adaptive LoRA adapter parameters based on the aggregated prototype, injects a large language model to realize task-specific reasoning, designs a complete training loss function through a multi-level distillation strategy and diversity regularization, and finally designs a task-independent prototype reasoning mechanism based on the trained prototype conditional hypernetwork, realizes adaptive navigation of a new scene through Top-K retrieval and temperature scaling soft aggregation. The method can enable a navigation intelligent agent to continuously learn new scenes while maintaining the performance of learned scenes and improving the generalization ability to unseen scenes.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, intelligent robot navigation, and continuous learning, and in particular to a lifelong visual language navigation method and system based on prototype evolution. Background Technology

[0002] Vision-and-Language Navigation (VLN) has become a fundamental problem in embodied intelligence, aiming to enable agents to navigate to target locations in a visual 3D environment by following user language instructions. Based on its intuitive and practical paradigm, VLN has demonstrated broad application value in fields such as autonomous driving, home robots, and smart factories. Although large-scale datasets and pre-trained vision-language models have driven significant progress in VLN, deploying VLN agents in real-world scenarios still faces challenges. In practice, navigation agents typically need to be adapted or fine-tuned for specific scenarios to achieve satisfactory performance. However, adapting pre-trained agents to new scenarios often leads to the following problems: 1. Catastrophic forgetting: When learning new scenarios, agents lose navigational abilities from previously learned scenarios, exhibiting severe performance degradation. Existing research shows that simply fine-tuning VLN agents sequentially across a series of navigation scenarios leads to significant catastrophic forgetting of learned navigation scenarios. 2. Decreased generalization ability: Sequential fine-tuning significantly reduces the generalization performance of VLN agents in unseen scenarios. The agent cannot utilize shared navigation knowledge across different navigation contexts, resulting in limited transfer capabilities to unseen environments and low learning efficiency. 3. Linear parameter growth: A straightforward solution is to maintain a task-specific parameter-efficient adapter (such as a LoRA module) for each navigation scenario. While this strategy can mitigate catastrophic forgetting by isolating scenario-specific parameters, it has two fundamental limitations: (1) independently learned adapters cannot capture and utilize shared navigation knowledge across different scenarios; (2) storing a dedicated adapter for each scenario leads to linear parameter growth as the number of navigation scenarios increases. 4. Lack of lifelong learning capability: Existing multi-expert or hybrid-expert LoRA methods (such as MoE-LoRA, HydraLoRA, Branch-LoRA, etc.) attempt to represent shared knowledge across sequential navigation tasks, but they typically assume a fixed number of experts, implicitly imposing an upper limit on the number of tasks, thus limiting scalability in lifelong settings.

[0003] Therefore, designing a method that enables navigation agents to continuously learn new scenarios while maintaining performance on already learned scenarios and improving generalization ability to unseen scenarios has become a research challenge in the current technology field. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to address the shortcomings of the prior art by providing a lifelong visual language navigation method and system based on prototype evolution. Through the design of a dynamic prototype library and a prototype conditional hypernetwork, a lifelong VLN agent is realized that can evolve its navigation knowledge through prototype-driven journeys in a constantly changing world.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0006] On the one hand, this invention provides a lifelong visual language navigation method based on prototype evolution, comprising the following steps:

[0007] Step S1: Build a dynamic prototype library. The prototype library is dynamically expanded and its gradient is updated through a distance threshold adaptive mechanism. Multiple related prototypes in the prototype library are merged to obtain an aggregated prototype to capture shared navigation knowledge across scenes.

[0008] Step S2: Design a prototype conditional hypernetwork and dynamically generate step-level adaptive networks based on the aggregated prototype. Adapter parameters are injected into a large language model to achieve task-specific reasoning; the prototype conditional hypernetwork Learning general transformations : Mapping aggregate prototype representation to Adapter parameters, where, The dimension is The prototype feature vector space, Embedded dimension for prototype; express The space where the projection matrix of the adapter resides, for Adapter input dimensions; express The space where the projection matrix on the adapter is located. for Adapter output dimensions; for The rank of the adapter;

[0009] Step S3: By designing a complete training loss function through a multi-level distillation strategy and diversity regularization, the training of the prototype conditional supernetwork and the continuous updating of the prototype library are realized.

[0010] Step S4: Based on the trained prototype conditional hypernetwork, design a task-independent prototype reasoning mechanism, through... Retrieval and temperature scaling soft aggregation enable adaptive navigation for new scenes.

[0011] Furthermore, the specific method of step S1 is as follows:

[0012] S11: Define a learnable prototype library Where K is the maximum prototype capacity of the prototype library, and each prototype by 3D representation spatial encoding step-level navigation knowledge;

[0013] S12: In each navigation step Using a scene encoder Based on current observations Extracting scene features ;

[0014] S13: Calculate the minimum Euclidean distance between the current scene feature and all existing prototypes in the prototype library. If the minimum Euclidean distance between the scene feature and the existing prototypes in the prototype library exceeds a threshold, initialize the scene feature as a new prototype and add it to the prototype library. ;

[0015] S14: Update the prototypes in the prototype library using gradient descent;

[0016] S15: Use a soft route aggregation strategy to merge multiple related prototypes in the prototype library to obtain an aggregated prototype in order to capture shared navigation knowledge across scenarios.

[0017] Furthermore, the specific method for using a soft route aggregation strategy to fuse multiple related prototypes in the prototype library to obtain an aggregated prototype in order to capture cross-scene shared navigation knowledge is as follows:

[0018] Given scene features Calculate its relationship with the prototype library Cosine similarity of all K prototypes in the dataset: ; via temperature scaling Function to calculate route weight ,in The temperature parameter is used; scene features are obtained through weighted summation. Corresponding aggregate prototype: .

[0019] Furthermore, the prototype conditional hypernetwork adopts a hierarchical dual-branch converter architecture, which is relevant to the large language model backbone network LLM. indivual Layers, each layer has an independent converter Mapping aggregate prototypes to layer-specific matrix;

[0020] The hierarchical dual-branch converter includes a shared coding layer, an A-branch output layer, and a B-branch output layer;

[0021] The shared coding layer will Dimensional Aggregation Prototype After linear transformation and nonlinear activation, the mapping is as follows: Dimensional intermediate feature representation ;

[0022] The A-branch output layer will share the intermediate features output by the coding layer. Generated through tensor reshaping operations Projection matrix under adapter ,in, Here is the weight matrix of the output layer of branch A. To and The corresponding bias vector, For tensor reshaping operations, For the first layer The input dimensions of the adapter;

[0023] The B-branch output layer generates the intermediate features shared with the coding layer output through a tensor reshaping operation. Projection matrix on adapter ,in, This is the weight matrix of the B-branch output layer. for The corresponding bias vector, For the first layer The output dimension of the adapter.

[0024] Furthermore, the specific method of step S3 is as follows:

[0025] S31: Starting the training task Previously, a reference snapshot of the creation of the prototype library. ;

[0026] S32: Reference snapshot based on the prototype library Design single-prototype distillation loss Maintain consistent output distribution across individual prototypes;

[0027] S33: Consistency Loss in Design Prototypes Maintain the relative geometric relationships between prototype pairs;

[0028] S34: Designing Cross-Prototype Drift Regularization Loss Anchor the prototype's position to constrain its stability;

[0029] S35: Designing a supernetwork to output distillation losses Ensure that the mapping of the prototype conditional hypernet to the old prototype remains consistent;

[0030] S36: Design Diversity Regularization Loss Prototype collapse is prevented by minimizing the cosine similarity between prototypes;

[0031] S37: The training loss function is determined as follows: ,in, For the cross-entropy loss of the navigation task, For the complete prototype distillation loss, These are the weights of each loss.

[0032] Furthermore, the specific method of step S4 is as follows:

[0033] S41: For the query prototype q, given observations from the new environment Extract the scene features of the current step. And retrieve the Top-K most similar prototypes in the prototype library to obtain Prototype subset;

[0034] S42: In Temperature scaling on a subset of the prototype Calculate the aggregate weights, and then calculate the scene features for the current step. Corresponding aggregate prototype ;

[0035] S43: For aggregate prototypes Through conditional hypernetwork Dynamically generated step specific Adapter parameters;

[0036] S44: Inject the LoRA adapter parameters generated in step S43 into the frozen base model of the large language model to obtain the injected parameters. Adaptation model after adapter parameters ;

[0037] S45: After the agent adapts to the new environment, it injects... The adapter parameterized adaptive model predicts the agent's performance in the first... Step navigation actions Perform navigation, where, Natural language navigation instructions; This refers to real-time environmental observations at step t of the navigation execution phase.

[0038] On the other hand, the present invention also provides a lifelong visual language navigation system based on prototype evolution, including a prototype library construction and update module, a prototype aggregation module, a task reasoning module, a loss function construction module, and an adaptive navigation module.

[0039] The prototype library construction and update module constructs a dynamic prototype library and achieves dynamic expansion and gradient update of the prototype library through a distance threshold adaptive mechanism.

[0040] The prototype aggregation module merges multiple related prototypes in the prototype library to obtain an aggregated prototype in order to capture cross-scene shared navigation knowledge.

[0041] The task reasoning module is designed with a prototype conditional hypernetwork, which dynamically generates step-level adaptive networks based on aggregate prototypes. Adapter parameters are injected into a large language model to achieve task-specific reasoning.

[0042] The loss function construction module uses a multi-level distillation strategy to train the prototype conditional supernetwork and update the prototype library, thereby determining the complete training loss function.

[0043] The adaptive navigation module, based on a pre-trained prototype conditional hypernetwork, designs a task-independent prototype reasoning mechanism, through... Retrieval and temperature scaling soft aggregation enable adaptive navigation for new scenes.

[0044] Thirdly, this application proposes an electronic device comprising: one or more processors, and a memory for storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the prototyping-based lifelong visual language navigation method.

[0045] Fourthly, this application proposes a computer-readable storage medium storing executable instructions that, when executed, cause a processor to perform the lifelong visual language navigation method based on prototype evolution.

[0046] Fifthly, this application proposes a computer program product, including a computer program or instructions that, when executed by a processor, implement the aforementioned prototype-based lifelong visual language navigation method.

[0047] The beneficial effects of adopting the above technical solution are as follows: The lifelong visual language navigation method and system based on prototype evolution provided by this invention (1) Innovative design of dynamic prototype library: The dynamic prototype library realizes compact and efficient storage of navigation knowledge, and the prototype evolves through adaptive expansion and gradient update mechanism. This is different from storing information independently for each scene. Different modules, prototype library achieves sublinear parameter growth, significantly improving parameter efficiency. (2) Efficient knowledge sharing and transfer: The soft route aggregation mechanism enables the agent to combine complementary knowledge of multiple related prototypes, realizing cross-scenario knowledge sharing and smooth behavior transition. This makes the generalization performance continuously improve rather than decline as more scenarios are learned. (3) Strong anti-forgetting ability: Through the two-layer anti-forgetting mechanism of three-level prototype distillation (single prototype, prototype pair, cross prototype) and super network output distillation, this invention effectively prevents catastrophic forgetting in the continuous learning process, with an average forgetting rate of only 14.9%, which is significantly better than existing methods. (4) Task-independent adaptive reasoning: Prototype retrieval and temperature scaling soft aggregation mechanisms enable the agent to autonomously adapt to unseen scenarios during inference, without the need for explicit task identifiers or gradient updates, ensuring efficient real-time navigation. (5) Superior generalization performance: Unlike other methods, the generalization performance decreases as more tasks are learned, The zero-sample success rate of the method steadily increased from 52.8% to 59.3%, proving the effectiveness of incremental knowledge accumulation. (6) Real robot verification: Deployment tests were conducted on a real quadruped robot platform to verify the practicality of the method in real-world scenarios. Attached Figure Description

[0048] Figure 1 The diagram below is an overview of the lifelong visual language navigation task definition and EverWalker method provided in Embodiment 1 of the present invention. (a) is a schematic diagram of the lifelong visual language navigation task sequence and prototype-driven adaptation, (b) is a comparison curve of the forgetting rate of each method in the continuous learning stage, and (c) is an evolution curve of the zero-sample generalization success rate of each method in the continuous learning stage.

[0049] Figure 2 The above is a diagram of the overall system architecture of the EverWalker method provided in Embodiment 1 of the present invention, including a dynamic prototype library, soft prototype aggregation, and step-specific prototype reasoning.

[0050] Figure 3 This is a schematic diagram of the task sequence for the lifetime VLN benchmark test set provided in Embodiment 1 of the present invention;

[0051] Figure 4 The following are examples of VLN navigation visualization provided in Embodiment 1 of the present invention: (a) is an example of indoor scene navigation execution in Task 4 simulator, (b) is an example of indoor scene navigation execution in Task 12 simulator, (c) is an example of outdoor cone obstacle scene navigation execution in Task 2 simulator, and (d) is an example of outdoor cone obstacle scene navigation execution in Task 5 simulator.

[0052] Figure 5 The radar charts of the various methods provided in Embodiment 1 of the present invention are compared on the SPL, FSPL, OSR and FOSR indicators. Among them, (a) is the radar chart of each method on the SPL indicator, (b) is the radar chart of each method on the FSPL indicator, (c) is the radar chart of each method on the OSR indicator, and (d) is the radar chart of each method on the FOSR indicator.

[0053] Figure 6The visualization diagram of the evolution of zero-shot generalization ability with training progress provided in Embodiment 1 of the present invention is shown in (a) the surface diagram of the success rate SR corresponding to the zero-shot test after training TaskID, and (b) the surface diagram of the forgetting rate FSR corresponding to the zero-shot test after training TaskID.

[0054] Figure 7 The t-SNE visualization of the prototype space provided in Embodiment 1 of the present invention includes (a) the prototype distribution after training for Task 6, (b) the prototype distribution after training for Task 18, (c) the zero-sample generalization prototype selection, and (d) the prototype collapse that occurs without diversity constraints.

[0055] Figure 8 This is a heatmap showing the similarity between prototypes provided in Embodiment 1 of the present invention.

[0056] Figure 9 This is a schematic diagram of a real robot navigation system platform provided in Embodiment 1 of the present invention. Detailed Implementation

[0057] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0058] Example 1:

[0059] In this embodiment, the prototype-evolution-based lifetime visual language navigation method (EverWalker) is used, such as Figure 1 , Figure 2 As shown, it involves a continuous learning phase and a reasoning phase, specifically including the following steps:

[0060] S1: Build a dynamic prototype library, realize the dynamic expansion and gradient update of the prototype library through the distance threshold adaptive mechanism, and use the soft route aggregation strategy to merge multiple related prototypes in the prototype library to obtain an aggregated prototype to capture cross-scene shared navigation knowledge.

[0061] S11: Define a learnable prototype library Where K is the maximum prototype capacity of the prototype library, and each prototype With compact Dimensional representation spatial encoding step-level navigation knowledge; in this embodiment, setting , ;

[0062] S12: In each navigation step Using a scene encoder Based on current observations Extracting scene features ;in, This is the current navigation step number. For scene encoder, For the agent in the first Real-time environmental observation input (including multimodal information such as visual images and verbal commands). For the extracted 3D scene feature vector, This represents the spatial dimension of scene features.

[0063] In this embodiment, pre-trained... The visual encoder extracts visual features as a scene encoder; a learnable linear projection layer is added to project the visual features onto the same surface as the prototype. The scene features are obtained in 3D space; the projection layer is designed as a fully connected layer with an input dimension of 1. The output dimension of the visual encoder is 512.

[0064] S13: Regarding the expansion of the prototype library, when the agent encounters a new scenario At that time, calculate the minimum Euclidean distance between the current scene features and all existing prototypes in the prototype library. If the minimum Euclidean distance between the scene features and existing prototypes in the prototype library exceeds a threshold... ,Right now Then the scene features will be initialized as a new prototype. And add to the prototype library ;

[0065] S14: For prototype updates in the prototype library, unlike relying on exponential moving averages which can lead to unstable updates, existing prototypes are updated via gradient descent: ,in For learning rate, For the k-th prototype in the prototype library, Let be the total loss function of the system. It is the total loss function Regarding prototypes The gradient represents the rate of change of the loss in the direction of that parameter. This update mechanism is represented by a backpropagation natural evolution prototype.

[0066] S15: Use a soft route aggregation strategy to merge multiple related prototypes in the prototype library to obtain an aggregated prototype in order to capture cross-scene shared navigation knowledge;

[0067] Given scene features Calculate its relationship with the prototype library Cosine similarity of all K prototypes in the dataset: ; via temperature scaling Function to calculate route weight ,in Temperature parameter (in this embodiment) Scene features are obtained through weighted summation. Corresponding aggregate prototype: .

[0068] S2: Design a prototype conditional hypernetwork to dynamically generate step-level adaptive networks based on aggregate prototypes. Adapter parameters are injected into a large language model to achieve task-specific reasoning.

[0069] Setting prototype conditions for hypernetic networks Learning general transformations : Mapping aggregate prototype representation to Adapter parameters, where, The dimension is The prototype feature vector space, Embedded dimension for prototype; express The space where the projection matrix of the adapter resides, for Adapter input dimensions; express The space where the projection matrix on the adapter is located. for Adapter output dimensions; for The rank of the adapter satisfies Used for balance In this embodiment, the adapter parameter efficiency and expressive power are set... ;

[0070] The prototype conditional hypernetwork adopts a hierarchical dual-branch converter architecture, which is relevant to the large language model backbone network LLM. indivual Layer (in this embodiment) =32), each layer l has an independent converter Mapping aggregate prototypes to layer-specific matrix ;

[0071] The hierarchical dual-branch converter includes a shared coding layer, an A-branch output layer, and a B-branch output layer;

[0072] The shared coding layer will Dimensional Aggregation Prototype After linear transformation and nonlinear activation, the mapping is as follows: Dimensional intermediate feature representation The calculation process is as follows: ,in This is the weight matrix. To and The corresponding bias vector, The intermediate feature dimension output by the shared coding layer (in this embodiment) ).

[0073] The A-branch output layer will share the intermediate features output by the coding layer. Generated through tensor reshaping operations Projection matrix under adapter ,in, Here is the weight matrix of the output layer of branch A. To and The corresponding bias vector, This is a tensor reshaping operation, which reshapes a one-dimensional output vector into a tensor. In matrix form, For the first layer The input dimensions of the adapter;

[0074] The B-branch output layer generates the intermediate features shared with the coding layer output through a tensor reshaping operation. Projection matrix on adapter ,in, This is the weight matrix of the B-branch output layer. for The corresponding bias vector, For the first layer The output dimension of the adapter.

[0075] S3: Through a multi-level distillation strategy and diverse regularization design, a complete training loss function is designed to achieve the training of the prototype conditional supernetwork and the continuous updating of the prototype library. It includes three-level prototype distillation and supernetwork output distillation to prevent catastrophic forgetting during the continuous learning process.

[0076] S31: Starting the training task Previously, a reference snapshot of the creation of the prototype library. ,in, This refers to the set of historical prototypes, that is, the set of all prototypes of historical tasks. As prototype Reference snapshot;

[0077] S32: Designing single-prototype distillation losses to maintain consistent output distribution across individual prototypes: ,in, This represents the loss from single-prototype distillation. This is the KL divergence function, used to measure the difference between two probability distributions; For student models in prototype Output probability distribution under the given conditions; For the teacher model in the reference prototype Output probability distribution under the given conditions; Actions or decision variables used to predict output; For the current student model, the first One prototype vector; For the teacher model corresponding to the first One reference prototype vector;

[0078] S33: Design prototype pair consistency loss to maintain the relative geometric relationship between prototype pairs: ,in These are interpolation coefficients to prevent distortion of the prototype manifold;

[0079] S34: Cross-prototype drift regularization anchors the prototype position, directly constraining the stability of the prototype position: ;

[0080] S35: The supernetwork output distillation loss is designed to ensure that the mapping of the prototype-conditional supernetwork to the old prototype remains consistent; during the training task... Previously, a reference adapter was cached for all legacy prototypes: ,in, This is the prototype conditional supernetwork before the current task begins; during training, the distillation loss is output through the supernetwork. To constrain the output to match the reference: ;in, and For reference adapter matrix, i.e. in task Before starting, the prototype conditional hypernet with updated parameters is used. Based on the old prototype The generated and cached fixed reference values ​​remain unchanged during the current task training. and This is the current adapter matrix, i.e., the hypernetwork. In the mission The output values ​​are dynamically generated during training and change as the parameters are updated. By constraining the Frobenius norm distance between the current adapter matrix and the reference adapter matrix. This ensures that the hypernetwork maintains consistent mapping outputs of old prototypes when learning new tasks, thereby suppressing catastrophic forgetting. The number of prototypes in the historical prototype set. This represents the number of adapter layers.

[0081] S36: Design diversity regularization loss to prevent prototype collapse by minimizing the cosine similarity between prototypes;

[0082] The diversity regularization loss is defined as:

[0083] ;

[0084] The loss minimizes the cosine similarity between all prototype pairs, ensuring that each prototype captures distinct scene features.

[0085] S37: The training loss function is designed as follows: ,in, For the cross-entropy loss of the navigation task, For the complete prototype distillation loss, These are the weights of each loss; in this embodiment, the loss weights are set as follows: For the first task, set... Distillation is disabled.

[0086] Steps S1-S3 above constitute the continuous learning phase;

[0087] S4: Enter the reasoning phase, design a task-independent prototype reasoning mechanism, through... Retrieval and temperature-scaled soft aggregation enable adaptive navigation for new scenes;

[0088] S41: For the query prototype q, given observations from the new environment Extract the scene features of the current step. ,in It is a scene encoder, and it retrieves the Top-K most similar prototypes in the prototype library to obtain... Prototype subset: In this embodiment =5.

[0089] S42: In Temperature scaling on a subset of the prototype Calculate aggregate weights (inference temperature) ), and then calculate the scene features of the current step. Corresponding aggregate prototype ;

[0090] S43: For aggregate prototypes Through conditional hypernetwork Dynamically generated step specific Adapter parameters ,in, For the aggregate prototype The corresponding number layer Down projection matrix, For the aggregate prototype The corresponding number layer Up projection matrix, For injection Total number of layers in the adapter;

[0091] S44: Inject the LoRA adapter parameters generated in step S43 into the frozen base model of the large language model to obtain the injected parameters. Adaptation model after adapter parameters: ,in, To freeze the base model, To freeze the fixed parameters of the base model; For query prototype generated The adapter parameter increment consists of the product of the lower projection matrix and the upper projection matrix at each layer, i.e. ; For injection The adaptation model following the adapter parameters;

[0092] S45: After the agent adapts to the new environment, it injects... The adapter parameterized adaptive model predicts the agent's performance in the first... Step navigation actions Perform navigation, where Natural language navigation instructions; This is a real-time environmental observation for step t of the navigation execution phase, which differs from the query observation used for prototype retrieval in S41. The Fadapted model is already based on query observations. Once the adaptation is complete, only forward propagation is performed during the inference phase, eliminating the need to re-query the prototype library, perform backpropagation, and calculate gradients.

[0093] This embodiment also constructs a lifetime visual language navigation benchmark dataset and preprocesses the dataset; the lifetime VLN benchmark dataset constructed in this embodiment is as follows: Figure 3 As shown, it contains 24 sequential navigation tasks; among them, tasks 1-18 are the continuous learning phase, including 14 tasks from... Dataset The simulation scenarios and four real-world space scans; Tasks 19-24 are the zero-sample generalization phase, including six completely unseen test environments.

[0094] Each task consists of 250-350 navigation rounds, each containing a navigation trajectory and manually annotated natural language instruction pairs. ,in, Represents natural language navigation instructions. This indicates the corresponding navigation trajectory.

[0095] The navigation data is then preprocessed, converting the panoramic visual observation data into discrete viewpoint representations, with the motion space fixed as... .

[0096] To fully verify the effectiveness of the method of this invention, this embodiment conducted extensive experiments on the lifelong visual language navigation benchmark dataset, the specific details of which include:

[0097] I. Experimental Environment and Configuration:

[0098] 1) Software environment: Based on PyTorch 2.1.2 + CUDA 11.8 + DeepSpeed ​​ZeRO-2 deep learning framework.

[0099] 2) Hardware configuration: 8 NVIDIA RTX A6000 GPUs (48GB VRAM), CPU is AMD EPYC 7763.

[0100] 3) Backbone network: The Qwen-7B large language model is used as the frozen base model. As a visual encoder.

[0101] 4) Optimizer configuration: Use the AdamW optimizer with a learning rate of [missing information]. The weight decay is 0.01, and cosine learning rate scheduling is used. Cosine learning rate scheduling is a strategy that dynamically adjusts the learning rate, which gradually decays from the initial value to near 0 according to a cosine function curve. The warm-up ratio is 0.075. Each task is trained for 1 epoch, and the effective batch size is 32.

[0102] 5) Hyperparameter settings are shown in Table 1:

[0103] Table 1 Hyperparameter Settings

[0104]

[0105] II. Evaluation Indicators;

[0106] This embodiment uses six quantitative indicators to evaluate the method of the present invention:

[0107] 1) Success Rate (SR): The proportion of rounds in which the agent's final position is ≤3 meters from the target. The higher the value, the better.

[0108] 2) Path Length Weighted Success Rate (SPL): Success rate multiplied by the ratio of the shortest path to the actual path length; the higher the value, the better.

[0109] 3) Prediction Success Rate (OSR): The percentage of rounds in which any point in the trajectory is ≤3 meters from the target. The higher the value, the better.

[0110] 4) Forgetting Rate (FSR / FSPL / FOSR): The percentage decrease in performance compared to peak performance after learning all tasks; the smaller the value, the better.

[0111] The forgetting rate corresponds to three dimensions of indicators, calculated based on SR, SPL, and OSR, and is defined as follows:

[0112] (1) FSR (Forgetting Rate Based on Success Rate):

[0113] ;

[0114] (2) FSPL (Forgetting Rate Based on Path Length Weighted Success Rate):

[0115] ;

[0116] (3) FOSR (Forgetting Rate Based on Prediction Success Rate):

[0117] ;

[0118] in, For the total number of tasks, The core model trained by the method of this invention learns the first... After the first task Success rate, path length weighted success rate, and prediction success rate for each historical task; This represents the historical best performance (peak performance) of the task; the three metric balancing measures the decrease in historical task performance relative to the peak after learning a new task, with smaller values ​​indicating less forgetting.

[0119] III. Controlled Experiment;

[0120] To verify the effectiveness of the method of the present invention, the EverWalker method of the present invention was compared with 11 state-of-the-art continuous learning methods:

[0121] 1) Baseline method: Seq-FT (sequential fine-tuning of individual LoRAs);

[0122] 2) Regularization methods: LwF (Knowledge Distillation), EWC (Elastic Weights Solidification);

[0123] 3) Playback method: iCaRL (Example Playback);

[0124] 4) Hybrid expert methods: Dense MoLE, Sparse MoLE, MoLA;

[0125] 5) Multi-adaptor methods: HydraLoRA, BranchLoRA, O-LoRA, SD-LoRA;

[0126] All methods are implemented based on the same StreamVLN framework, ensuring fair comparison.

[0127] IV. Experimental Results;

[0128] 4.1: Performance comparison during the continuous learning phase;

[0129] The navigation performance of each method was tested on tasks 1-18, and the quantitative results are shown in Table 2. The navigation effects are as follows: Figure 4 As shown.

[0130] Table 2 Performance Comparison During Continuous Learning Phase (SR ↑ %)

[0131]

[0132] Experimental conclusions: The EverWalker method of this invention achieves an average success rate of 61.9% during the continuous learning phase, which is 4.4 percentage points higher than the best comparison method SD-LoRA (57.5%). Particularly on real-world tasks (T2, T5, T10, T13), the EverWalker method achieves an average success rate of 47.5%, significantly outperforming SD-LoRA's 43.3%.

[0133] 4.2: Comparison of anti-forgetting performance;

[0134] The forgetting rate of each method during continuous learning was tested, and the quantitative results are shown in Table 3. A comprehensive comparison of each method across the four metrics (SPL, FSPL, OSR, and FOSR) is also presented. Figure 5 As shown.

[0135] Table 3 Comparison of Forgetting Rates (F-SR ↓ %)

[0136]

[0137] Experimental conclusions: The average forgetting rate of the EverWalker method was only 28.4%, which was 4.9 percentage points lower than SD-LoRA (33.3%) and 8.0 percentage points lower than HydraLoRA (36.4%). The average forgetting rate of the early tasks (T1-T5) was 34.0%, significantly lower than SD-LoRA's 39.4%, demonstrating the effectiveness of the two-layer anti-forgetting mechanism of three-stage distillation and hypernetwork output distillation.

[0138] 4.3: Comparison of zero-shot generalization performance;

[0139] The zero-shot generalization performance of each method was tested on tasks 19-24, where no testing environment was available. The quantitative results are shown in Table 4, and the generalization ability evolution curves are shown in the figure. Figure 6 As shown.

[0140] Table 4 Comparison of zero-shot generalization performance (SR ↑ %)

[0141]

[0142] Experimental conclusions: The EverWalker method achieved an average success rate of 59.3% in the zero-shot generalization phase, which is 7.6 percentage points higher than SD-LoRA (51.7%). In real-world test scenarios (T20, T23), EverWalker achieved an average success rate of 48.0%, which is better than SD-LoRA's 43.5%.

[0143] 4.4: Generalization ability evolves with training progress;

[0144] The trend of zero-shot generalization performance with the progress of continuous learning was tested, and the quantitative results are shown in Table 5.

[0145] Table 5. Evolution of generalization performance with training progress (average SR ↑ %) for zero-shot tasks

[0146]

[0147] Experimental conclusion: Unlike other methods whose generalization performance decreases with the increase of learning tasks, the generalization performance of the EverWalker method steadily increased from 52.8% to 59.3% (an increase of 6.5 percentage points), proving that dynamic prototype libraries can effectively accumulate transferable navigation knowledge.

[0148] V. Ablation Experiment;

[0149] To verify the effectiveness of dynamic prototype library, soft route aggregation, three-level distillation, diversity loss, and hypernetwork distillation, this embodiment designed an ablation experiment, and the quantitative results are shown in Table 6.

[0150] Table 6 Ablation Experiments of Core Components

[0151]

[0152] Ablation analysis:

[0153] 1) Removing the dynamic prototype library resulted in a 9.6 percentage point decrease in SR and a doubling of the forgetting rate (14.9% → 28.5%), proving that the prototype library is crucial for knowledge storage.

[0154] 2) Replacing soft aggregation with hard Top-1 selection reduced SR by 5.4 percentage points, verifying the effectiveness of soft route aggregation.

[0155] 3) Removing tertiary distillation caused the FSR to increase from 14.9% to 30.2%, demonstrating the important role of proto-distillation in combating forgetting.

[0156] 4) Removing the supernetwork distillation caused the FSR to increase from 14.9% to 35.2%, proving that both anti-forgetting mechanisms are indispensable.

[0157] 5) The lack of diversity loss led to prototype collapse, resulting in a 6.1 percentage point decrease in SR.

[0158] VI. Comparison of parameter efficiency;

[0159] The number of trainable parameters for each method was tested, and the quantitative results are shown in Table 7.

[0160] Table 7 Comparison of Trainable Parameters

[0161]

[0162] Experimental conclusions: The EverWalker method requires only 18.6M trainable parameters (approximately 55% of the number required by fixed-parameter methods) to achieve optimal performance. For 24 tasks, the SD-LoRA method requires 201.6M parameters (T×8.4M, T=24), while the EverWalker method maintains a constant 18.6M parameters, achieving sublinear parameter growth.

[0163] VII. Prototype Space Visualization Analysis;

[0164] The prototype space was visualized and analyzed using t-SNE, and the results are as follows: Figure 7 As shown:

[0165] 1) After Task 6 (early stage): Prototypes are naturally clustered by scenario type, using loss... Forced clear boundaries, while losses through three-stage distillation. Maintain stability.

[0166] 2) After Task 18 (Complete Training): The prototype space presents a hierarchical structure, including fine-grained prototypes for different scenarios and bridging prototypes for transitional environments.

[0167] 3) Zero-shot testing: The features of the test scenario fall within the convex hull of the training prototype, verifying that the knowledge coverage is sufficient.

[0168] 4) Without diversity constraints: The prototypes collapse into overlapping clusters, and the average spacing decreases from 0.67 to 0.23. The semantic similarity distribution between different prototypes is as follows: Figure 8 As shown, the diversity regularization loss is further verified. To maintain the effectiveness of prototype discriminativeness, the mean cosine similarity between prototypes remains at a low level, demonstrating that the prototype library possesses sufficient diversity.

[0169] 8. Real robot verification;

[0170] To verify the effectiveness of the EverWalker method in real-world scenarios, this embodiment constructs a physical robot navigation platform. The system architecture is as follows: Figure 9 As shown.

[0171] Hardware configuration includes:

[0172] 1) Mobile platform: DeepRobotDogLite2 quadruped robot;

[0173] 2) Visual perception: Hikvision DS-E12 high-definition RGB camera;

[0174] 3) Communication: High-bandwidth portable WiFi communication module;

[0175] 4) Computing: A remote computing server equipped with 8 NVIDIA RTX A6000 GPUs;

[0176] Experimental results show that the EverWalker method achieves a success rate of 47.6% in real-world navigation tasks, with an inference latency of approximately 85ms / step, meeting the requirements for real-time navigation. Compared to SD-LoRA (41.0%), this represents an improvement of 6.6 percentage points, demonstrating the practicality of the method in real-world scenarios.

[0177] Example 2:

[0178] A lifelong visual language navigation system based on prototype evolution includes a prototype library construction and update module, a prototype aggregation module, a task reasoning module, a loss function construction module, and an adaptive navigation module;

[0179] The prototype library construction and update module constructs a dynamic prototype library and achieves dynamic expansion and gradient update of the prototype library through a distance threshold adaptive mechanism.

[0180] The prototype aggregation module merges multiple related prototypes in the prototype library to obtain an aggregated prototype in order to capture cross-scene shared navigation knowledge.

[0181] The task reasoning module is designed with a prototype conditional hypernetwork, which dynamically generates step-level adaptive networks based on aggregate prototypes. Adapter parameters are injected into a large language model to achieve task-specific reasoning.

[0182] The loss function construction module uses a multi-level distillation strategy to train the prototype conditional supernetwork and update the prototype library, thereby determining the complete training loss function.

[0183] The adaptive navigation module, based on a pre-trained prototype conditional hypernetwork, designs a task-independent prototype reasoning mechanism, through... Retrieval and temperature scaling soft aggregation enable adaptive navigation for new scenes.

[0184] Example 3:

[0185] This embodiment proposes an electronic device, including: one or more processors, and a memory for storing instructions, which, when executed by the one or more processors, cause the one or more processors to execute the lifelong visual language navigation method based on prototype evolution.

[0186] The electronic device may be a mobile phone, computer, or tablet computer, etc., and includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements the lifelong visual language navigation method based on prototype evolution as described in the embodiments. It is understood that the electronic device may also include an input / output (I / O) interface and communication components.

[0187] The processor is used to execute all or part of the steps in the lifelong visual language navigation method based on prototype evolution as described in the above embodiments. The memory is used to store various types of data, which may include, for example, instructions for any application or method in the electronic device, as well as application-related data.

[0188] The processor can be implemented as an Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), controller, microcontroller, microprocessor, or other electronic components, and is used to execute the lifelong visual language navigation method based on prototype evolution described in the above embodiments.

[0189] Example 4:

[0190] This embodiment proposes a computer-readable storage medium that stores executable instructions. When these instructions are executed, if they are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0191] The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the lifelong visual language navigation method based on prototype evolution described in the various embodiments of this application.

[0192] The aforementioned storage media include: flash memory, hard disks, multimedia cards, card-type memory (e.g., SD (Secure Digital Memory Card) or DX (Memory Data Register, MDR) memory), random access memory (RAM), static random-access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, disks, optical discs, servers, APP (Application) application stores, and other media capable of storing program verification codes. These media store computer programs, which, when executed by a processor, can implement the various steps of the prototype-evolution-based lifelong visual language navigation method described above.

[0193] Example 5:

[0194] This embodiment proposes a computer program product, including a computer program or instructions, which, when executed by a processor, implements the aforementioned lifelong visual language navigation method based on prototype evolution.

[0195] Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a computer program product.

[0196] The various embodiments in this application are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0197] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the present invention.

Claims

1. A lifelong visual language navigation method based on prototype evolution, characterized in that, Includes the following steps: Step S1: Build a dynamic prototype library. The prototype library is dynamically expanded and its gradient is updated through a distance threshold adaptive mechanism. Multiple related prototypes in the prototype library are merged to obtain an aggregated prototype to capture shared navigation knowledge across scenes. Step S2: Design a prototype conditional hypernetwork and dynamically generate step-level adaptive networks based on the aggregated prototype. Adapter parameters are injected into a large language model to achieve task-specific reasoning; the prototype conditional hypernetwork Learning general transformations : Mapping aggregate prototype representation to Adapter parameters, where, The dimension is The prototype feature vector space, Embedded dimension for prototype; express The space where the projection matrix of the adapter resides, for Adapter input dimensions; express The space where the projection matrix on the adapter is located. for Adapter output dimensions; for The rank of the adapter; Step S3: By designing a complete training loss function through a multi-level distillation strategy and diversity regularization, the training of the prototype conditional supernetwork and the continuous updating of the prototype library are realized. Step S4: Based on the trained prototype conditional hypernetwork, design a task-independent prototype reasoning mechanism, through... Retrieval and temperature scaling soft aggregation enable adaptive navigation for new scenes.

2. The lifelong visual language navigation method based on prototype evolution according to claim 1, characterized in that, The specific method for step S1 is as follows: S11: Define a learnable prototype library Where K is the maximum prototype capacity of the prototype library, and each prototype by 3D representation spatial encoding step-level navigation knowledge; S12: In each navigation step Using a scene encoder Based on current observations Extracting scene features ; S13: Calculate the minimum Euclidean distance between the current scene feature and all existing prototypes in the prototype library. If the minimum Euclidean distance between the scene feature and the existing prototypes in the prototype library exceeds a threshold, initialize the scene feature as a new prototype and add it to the prototype library. ; S14: Update the prototypes in the prototype library using gradient descent; S15: Use a soft route aggregation strategy to merge multiple related prototypes in the prototype library to obtain an aggregated prototype in order to capture shared navigation knowledge across scenarios.

3. The lifelong visual language navigation method based on prototype evolution according to claim 2, characterized in that, The specific method for using a soft route aggregation strategy to fuse multiple related prototypes in the prototype library to obtain an aggregated prototype in order to capture cross-scene shared navigation knowledge is as follows: Given scene features Calculate its relationship with the prototype library Cosine similarity of all K prototypes in the dataset: ; via temperature scaling Function to calculate route weight ,in The temperature parameter is used; scene features are obtained through weighted summation. Corresponding aggregate prototype: .

4. The lifelong visual language navigation method based on prototype evolution according to claim 3, characterized in that, The prototype conditional hypernetwork adopts a hierarchical dual-branch converter architecture, which is relevant to the large language model backbone network LLM. indivual Layers, each layer has an independent converter Mapping aggregate prototypes to layer-specific matrix; The hierarchical dual-branch converter includes a shared coding layer, an A-branch output layer, and a B-branch output layer; The shared coding layer will Dimensional Aggregation Prototype After linear transformation and nonlinear activation, the mapping is as follows: Dimensional intermediate feature representation ; The A-branch output layer will share the intermediate features output by the coding layer. Generated through tensor reshaping operations Projection matrix under adapter ,in, Here is the weight matrix of the output layer of branch A. To and The corresponding bias vector, For tensor reshaping operations, For the first layer The input dimensions of the adapter; The B-branch output layer generates the intermediate features shared with the coding layer output through a tensor reshaping operation. Projection matrix on adapter ,in, This is the weight matrix of the B-branch output layer. for The corresponding bias vector, For the first layer The output dimension of the adapter.

5. The lifelong visual language navigation method based on prototype evolution according to claim 4, characterized in that, The specific method for step S3 is as follows: S31: Starting the training task Previously, a reference snapshot of the creation of the prototype library. ; S32: Reference snapshot based on the prototype library Design single-prototype distillation loss Maintain consistent output distribution across individual prototypes; S33: Consistency Loss in Design Prototypes Maintain the relative geometric relationships between prototype pairs; S34: Designing Cross-Prototype Drift Regularization Loss Anchor the prototype's position to constrain its stability; S35: Designing a Hypernetwork Output for Distillation Losses Ensure that the mapping of the prototype conditional hypernet to the old prototype remains consistent; S36: Design Diversity Regularization Loss Prototype collapse is prevented by minimizing the cosine similarity between prototypes; S37: The training loss function is determined as follows: ,in, For the cross-entropy loss of the navigation task, For the complete prototype distillation loss, These are the weights of each loss.

6. The lifelong visual language navigation method based on prototype evolution according to claim 5, characterized in that, The specific method for step S4 is as follows: S41: For the query prototype q, given observations from the new environment Extract the scene features of the current step. And retrieve the Top-K most similar prototypes in the prototype library to obtain Prototype subset; S42: In Temperature scaling on a subset of the prototype Calculate the aggregate weights, and then calculate the scene features for the current step. Corresponding aggregate prototype ; S43: For aggregate prototypes Through conditional hypernetwork Dynamically generated step specific Adapter parameters; S44: Inject the LoRA adapter parameters generated in step S43 into the frozen base model of the large language model to obtain the injected parameters. Adaptation model after adapter parameters ; S45: After the agent adapts to the new environment, it injects... The adapter parameterized adaptive model predicts the agent's performance in the first... Step navigation actions Perform navigation, where, Natural language navigation instructions; This refers to real-time environmental observations at step t of the navigation execution phase.

7. A lifelong visual language navigation system based on prototype evolution, implemented based on the lifelong visual language navigation method based on prototype evolution as described in claim 1, characterized in that, It includes a prototype library building and updating module, a prototype aggregation module, a task reasoning module, a loss function construction module, and an adaptive navigation module; The prototype library construction and update module constructs a dynamic prototype library and achieves dynamic expansion and gradient update of the prototype library through a distance threshold adaptive mechanism. The prototype aggregation module merges multiple related prototypes in the prototype library to obtain an aggregated prototype in order to capture cross-scene shared navigation knowledge. The task reasoning module is designed with a prototype conditional hypernetwork, which dynamically generates step-level adaptive networks based on aggregate prototypes. Adapter parameters are injected into a large language model to achieve task-specific reasoning. The loss function construction module uses a multi-level distillation strategy to train the prototype conditional supernetwork and update the prototype library, thereby determining the complete training loss function. The adaptive navigation module, based on a pre-trained prototype conditional hypernetwork, designs a task-independent prototype reasoning mechanism, through... Retrieval and temperature scaling soft aggregation enable adaptive navigation for new scenes.

8. An electronic device for executing the lifelong visual language navigation method based on prototyping as described in any one of claims 1-6, characterized in that, include: One or more processors, and a memory for storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the prototyping-based lifelong visual language navigation method.

9. A computer-readable storage medium storing executable instructions for performing the lifelong visual language navigation method based on prototyping as described in any one of claims 1-6, characterized in that, When executed, the instructions cause the processor to perform the lifetime visual language navigation method based on prototype evolution.

10. A computer program product for executing the lifelong visual language navigation method based on prototyping as described in any one of claims 1-6, characterized in that, This includes a computer program or instructions that, when executed by a processor, implement the aforementioned prototype-evolution-based lifelong visual language navigation method.

Images