A zero-shot continuous visual language navigation method based on visual semantics and map integration

Abstract

The application discloses a zero-shot continuous visual language navigation method based on visual semantics and map integration, comprising the following steps: S1. accepting a natural language instruction of a current navigation task, and encoding and extracting features of the instruction through a text encoder to obtain instruction features; S2. capturing an RGB view during task execution, decoupling environment knowledge in the RGB view to obtain environment decoupled knowledge; S3. capturing a Depth view during task execution, combining the RGB view and the instruction features, and obtaining a self-centered semantic map through a cross-modal attention network; S4. combining the semantic map and the instruction features, and generating a series of navigable waypoints leading to a target according to a waypoint generation network; S5. predicting a current navigation action of a navigation agent by using the navigable waypoints and the environment decoupled knowledge; and S6. repeating the above steps until the navigation task is completed or a maximum number of movement steps is reached.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of visual language navigation cross-modal technology, and particularly relates to a zero-shot continuous visual language navigation method based on visual semantics and map integration. BACKGROUND

[0002] Visual language navigation (VLN) [1-2] requires navigation agents to understand natural language instructions and complete the navigation task from the starting point to the target point according to the instructions. It is mainly divided into two categories: one is discrete VLN executed in discrete environment, which mainly relies on a predefined connectivity graph for navigation, and can be regarded as an abstract version of the real situation; the other is continuous VLN (VLN-CE) for navigation in continuous environment, which can reflect the complexity of navigation in real environment compared with discrete VLN, and has a positive significance for promoting the practical application of the model.

[0003] Discrete VLN is an early version of VLN, and the research work therein mainly focuses on data augmentation [3-7], auxiliary measures [8-11], pretraining [12-16], map learning [17-20] and the like. Continuous VLN [21-28] is a new research paradigm proposed in recent years, wherein Krantz et al.

[21] first proposed the VLN task in continuous environment and solved it using WPN model in

[27] ; Hong et al.

[22] bridged the gap between discrete VLN and continuous VLN to facilitate many models working on discrete VLN to transition to continuous VLN; ERG

[24] promoted the understanding of the environment by the navigation agent through graph-based environment representation; other methods such as map prediction

[23] and mental planning

[26] are proposed to further improve the navigation performance.

[0004] Zero-shot learning (ZSL) is one of the ways for artificial intelligence models to transition to real-world applications today, as the data categories in the real world are diverse, and models cannot learn every data category during the training process. Early zero-shot learning was mainly applied in classification tasks [29-30], and with the rise and development of other computer vision tasks, zero-shot learning has been gradually applied to tasks such as object detection [31-33], image segmentation [34-35], etc. In recent years, the difficulties of some navigation tasks such as Object-Goal Navigation [36-39], Semantic Visual Navigation

[40] , Instance Navigation

[41] , Vision-and-Language Navigation [42-47] have also been solved through zero-shot learning. Among them, the way of using CLIP [36, 38, 41, 42] to participate in the navigation process has achieved good results; since large language models have strong cognitive and reasoning capabilities, using GPT to help navigation [44-47] is a direction worth exploring, and many works have proven the feasibility of this approach and the excellent performance of the model after integrating GPT.

[0005] From the above research, it can be seen that a large number of VLN researches mainly focus on supervised learning, which mainly relies on a large amount of training data, which will cause the model to be unable to generalize to new, unseen objects and environments, and real-world navigation contains various complex and diverse data categories, which cannot be completely contained in the training data. In order to deal with this challenge, it is a very effective solution to enable the navigation agent to effectively generalize and navigate on unseen data categories after learning limited knowledge, which is called "zero-shot navigation", and this VLN task is also called "zero-shot VLN".

[0006] Recently, a small number of new researches [42-47] have focused on the zero-shot VLN task, but it is undeniable that these researches mainly rely on the rich internal knowledge and strong reasoning ability of large models to achieve navigation planning, and the navigation ability of the navigation agent itself has not been greatly improved, and since the size of the large model is usually large, it leads to low efficiency of relying on large models for navigation, and the computational cost is also not low. SUMMARY

[0007] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a zero-sample continuous visual-language navigation method based on visual semantics and map ensemble. This method offers a navigation planning approach that does not rely on large models (including large language models or visual-language models). This approach can learn navigation experience from limited visible data and flexibly utilize spatial semantic information to compensate for missing information in unseen data classes, ultimately achieving generalized navigation on unseen data classes. Compared to previous methods, large models are not familiar with VLN tasks and lack the ability to directly learn knowledge from the navigation environment; the method of this invention precisely overcomes this deficiency.

[0008] The objective of this invention is achieved through the following technical solution:

[0009] A zero-shot continuous visual-language navigation method based on visual semantics and map ensemble includes the following steps:

[0010] S1. Receive the natural language instructions for the current navigation task, encode and extract features from them using a text encoder to obtain instruction features;

[0011] S2. Capture the RGB view of the current location and decouple the environmental knowledge in the RGB view to obtain environmental decoupling knowledge;

[0012] S3. Capture the Depth view of the current location, combine the instruction features in S1 and the RGB view in S2, and obtain an egocentric semantic map through a cross-modal attention network;

[0013] S4. Combining the instruction features in S1 and the egocentric semantic map obtained in S3, generate several navigable waypoints leading to the target;

[0014] S5. Combining the environmental decoupling knowledge obtained in S2 and the navigable waypoints obtained in S4, predict the navigation action that the navigation agent should perform;

[0015] S6. Repeat the above steps until the navigation agent determines that the navigation task is completed or the maximum number of steps has been reached.

[0016] Furthermore, in step S1, the BERT model is used as a text encoder to extract features from the input natural language instructions, and its bidirectional encoding capability is used to capture semantic information.

[0017] Furthermore, the specific workflow for decoupling environmental knowledge using the environmental knowledge decoupling module in step S2 is as follows:

[0018] First, the SAM model is used to perform zero-shot segmentation on the acquired RGB view with environmental information to obtain the corresponding object semantic class;

[0019] Then, the scene information in the RGB view is decoupled. For the layout information of object category i, it is represented as S. i =[C x C y A r ], where C x and C y Indicate the coordinates of its centroid, A r This represents the area percentage of the object relative to the current RGB view scene; ultimately, all J values ​​are represented. i These are stacked together to form a scene decoupling matrix, which stores navigation information in the current RGB view scene;

[0020] Next, the object information in the RGB view is decoupled, and for object category i, it is represented as O. i = [V, K, S], where V represents the visibility of the object; if the object is observable in the current view, its value is 1, otherwise it is 0. Several key objects that help the navigation agent understand the environment are extracted sequentially from the instructions; these objects reflect the navigation agent's navigation path. K is used to indicate whether the current object is a key object; if it is, its value is 1, otherwise it is 0. S represents the weighted average of the cosine similarity between the semantic embedding of the current object and the semantic embedding of the key objects. Key objects appearing in the already executed instruction portion have a weight of 0; as the instruction execution progresses, the weight of subsequent key objects gradually increases. Finally, all O's are... i Stacked together, they form an object decoupling matrix;

[0021] Finally, the scene decoupling matrix and the object decoupling matrix are concatenated row by row to form a view-observation decoupling matrix that is independent of specific data categories. The view-observation decoupling matrix is ​​then input into a variational autoencoder to capture the latent structure and correlations between features of the input data and extract environmental decoupling knowledge.

[0022] Furthermore, in step S3, the specific process of obtaining the egocentric semantic map through a cross-modal attention network is as follows:

[0023] First, for the obtained depth view of the current location, project its ground onto a grid map φ. t The grid map represents the current space occupancy, where t represents the current time step; for ground projection operations, inverse projection is first performed using camera intrinsic parameters to convert depth pixels into point cloud data in three-dimensional space, and then the point cloud data is projected onto a two-dimensional plane parallel to the ground.

[0024] Then, a truncated ResNet18 network was used to process this grid map φ t Feature extraction is performed to generate a grid map feature representation. For the input instruction, the encoded instruction features Λ are obtained through the BERT model;

[0025] Next, use grid map features The query Q is used as the feature, the instruction feature Λ is used as the key K and the value V, and the attention representation is obtained through a scaled dot product attention mechanism. The calculation method is as follows:

[0026]

[0027] Among them W q W k W v They are all learnable square matrices of parameters with dimension d. It is a grid map φ t Attention representation on;

[0028] Further calculations were performed using command features Λ and grid map features. The degree of semantic similarity between them, i.e., cosine similarity By combining cosine similarity with the original attention score, the navigation agent can more accurately identify the map region most relevant to the semantics of the instruction feature Λ, thus encouraging the navigation agent to act towards the most relevant region, as shown below:

[0029]

[0030] Among them, W R and W Λ It is a learnable parameter matrix; in computation When calculating the numerator, matrix multiplication is used, and element-wise multiplication is used in the denominator. ||·|| represents the L2 norm; cosine similarity. The value ranges from -1 to 1. Scaled to between 0 and 1 To ensure and The range of values ​​is the same; ultimately, it will... The weighted calculation is performed by combining the original attention score with the original attention score. This indicates the corrected grid map φ t The comprehensive attention representation on the graph, where λ1 and λ2 represent the weights used to balance the two attention scores, are both greater than 0, and sum to 1;

[0031] Finally, semantic information was inferred and completed using two Unet networks based on the ResNet18 architecture, namely Res-Unet, and semantic information outside the observation area of ​​the navigation agent was further obtained, as shown below:

[0032]

[0033] in, This involves semantic segmentation of the ground projection from the RGB view; thus, the egocentric semantic map is complete. The reasoning and generation not only include a top-down egocentric semantic map of the current field of vision of the navigation agent, but also supplements the semantic map of unknown areas that cannot be observed outside its field of vision, thus making full preparations for subsequent steps.

[0034] Furthermore, in step S4, the specific process for generating several navigable waypoints leading to the target based on the waypoint generation network is as follows:

[0035] First, ResNet18 was used again to refine the egocentric semantic map. Encode to obtain an egocentric semantic map Feature representation

[0036] Then, feature representations of egocentric semantic maps are used. As the query Q, the instruction feature Λ serves as the key K and value V; waypoint attention representations are obtained through a scaled dot product attention mechanism. The calculation method is as follows:

[0037]

[0038] Among them W q W k W v They are all learnable square matrices of parameters with dimension d;

[0039] Finally, by representing waypoint attention The initial waypoint is input into a Res-Unet network to obtain all currently navigable waypoints leading to the target.

[0040] Furthermore, the specific process of predicting the actions to be performed by the navigation agent in step S5 is as follows: all the predicted waypoints and the previously learned environmental decoupling knowledge are sent into the planning and decision module. In this planning and decision module, the navigable waypoint closest to the location of the navigation agent is first selected as the sub-target. Then, combined with the environmental decoupling knowledge, the DD-PPO policy pre-trained in the PointNav task is used to complete the action prediction.

[0041] Furthermore, in step S6, the navigation agent uses an action set consisting of four actions: moving forward 0.25m, turning left 15°, turning right 15°, and stopping. If a stopping decision is made within 3m of the target location, and the navigation agent completes a navigation cycle within a fixed time budget of 500 steps, then the navigation is considered successful.

[0042] Furthermore, the navigation agent is trained within a limited set of visible data categories. If unseen data categories appear during testing, the navigation agent uses the semantic embeddings of all provided data categories to perform generalized navigation on the unseen data categories. The data categories include object categories and environment categories.

[0043] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the zero-sample continuous visual language navigation method based on visual semantics and map integration.

[0044] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the zero-sample continuous visual language navigation method based on visual semantics and map integration.

[0045] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0046] 1. Lower Data and Computational Requirements: Unlike most current Visual Language Navigation (VLN) research that relies on large amounts of data and supervised learning, the zero-shot continuous visual language navigation method provided in this invention does not depend on a large amount of training data. It can be trained on a limited number of visible data categories and effectively generalizes navigation tasks on unseen data categories. This significantly reduces data requirements and computational resource consumption, resulting in greater application flexibility and economic benefits.

[0047] 2. No large model required for navigation planning: Unlike existing zero-shot VLN methods, this invention does not rely on large models (such as large language models or visual language models). Instead, it extracts navigation information from limited environmental knowledge by combining techniques such as visual semantics, map ensemble, and cross-modal attention mechanisms. This not only avoids the high computational cost associated with large models but also enables the method to better address navigation challenges in unseen data categories without a large amount of training data.

[0048] 3. Excellent Zero-Shot Navigation Performance: By combining direction, object, and scene cues to learn relational knowledge in navigation, and utilizing a top-down egocentric semantic map to indicate direction, the method obtains object and scene representations by decoupling environmental knowledge. Since it does not learn representational knowledge for specific data categories, the navigation agent can effectively generalize navigation on unfamiliar data categories, achieving zero-shot navigation. Extensive experiments and analysis under the VLN-CE setting verify that the technical solution of this invention has excellent zero-shot navigation performance, surpassing the performance of methods based on large models, and demonstrating that the large-model-free architecture of this invention has high navigation efficiency. Compared with large-model methods, this method can achieve or surpass their performance with lower computational overhead, exhibiting excellent zero-shot navigation capabilities.

[0049] 4. Adapting to Dynamically Changing Real-World Environments: The zero-shot navigation method of this invention is closer to navigation tasks in real-world environments, as unseen object categories are frequently encountered. This invention can effectively navigate objects of unseen categories without relying on large-scale training data, promoting the wider application of VLN tasks in the real world.

[0050] 5. Low hardware requirements and easy integration: Because the core method of this invention has low hardware requirements, it can be easily integrated into existing smart devices, such as self-driving cars and intelligent service robots. Its modular structure allows the technology to be seamlessly embedded into various automation systems, enhancing the navigation and operation capabilities of embodied robots and expanding the application scenarios of this technology.

[0051] 6. Improved navigation efficiency: Compared with traditional methods, the method of this invention can achieve efficient navigation planning without relying on large-scale computing models, and can handle more types of environmental data, enabling the navigation agent to complete tasks more accurately and efficiently. Attached Figure Description

[0052] Figure 1 This is a flowchart illustrating the method of the present invention.

[0053] Figure 2 This is a schematic diagram of the overall framework structure of the method of the present invention. Detailed Implementation

[0054] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention.

[0055] The technical solution of this invention has been described in detail with reference to the accompanying drawings and specific embodiments, aiming to demonstrate its basic concept and operation process. The schematic diagrams provided in the drawings are for illustrative purposes only and do not represent actual product dimensions or limit the scope of the patent. Where there is no conflict, features in the embodiments can be combined with each other to explore more implementation possibilities. Those skilled in the art should understand that, in order to more clearly convey the concept of this invention, certain components may be omitted, enlarged, or reduced in the drawings, and well-known structures and their descriptions may be omitted.

[0056] Please refer to the attached diagram for details. Figure 1 and Figure 2 This embodiment relates to a zero-sample continuous visual language navigation method based on visual semantics and map ensemble. The detailed steps are as follows:

[0057] S1. Receive the natural language instructions for the current navigation task, encode them using a text encoder, and extract features to obtain instruction features. The instructions in this step are issued before each navigation session begins; they can be either detailed or rough. Rough descriptions are particularly challenging for the navigation agent, requiring better reasoning capabilities. This embodiment does not limit the level of detail in the instruction description. The text encoder in this embodiment uses the BERT model, which captures the semantic information of the instructions through its bidirectional encoding capabilities.

[0058] S2. Capture the RGB view of the current location, and use the environment knowledge decoupling module to decouple the environment knowledge in the RGB view to obtain environment decoupling knowledge. The RGB view in this step needs to be actively acquired, and it mainly depends on the current environment location; the RGB view acquired at different locations will differ. The specific process of environment knowledge decoupling is as follows:

[0059] 201) Using SAM

[48] , zero-sample segmentation is performed on the acquired RGB view with rich environmental information to obtain the corresponding object semantic class. Subsequent operations will be based on this semantic segmentation map.

[0060] 202) Decouple the scene information in the RGB view. Since the overall layout information in the scene is composed of the layout information of each object, the layout information of object category i can be represented as S. i =[C x C y A r ], where C x and C y Indicate the coordinates of its centroid, A rThis represents the area percentage of the object relative to the current view scene. It's important to consider that when multiple objects of the same category appear simultaneously in the current scene, the one with the largest area percentage is taken, as larger objects tend to be more visible and contribute more to the layout information. Ultimately, all S... i Stacked together, they form a scene decoupling matrix, which stores the key navigation information in the current RGB view scene;

[0061] 203) Decouple the object information in the RGB view. For object category i, it can be represented as O i = [V, K, S], where V represents the visibility of the object; if the object is observable in the current view, its value is 1, otherwise it is 0. Key objects that help the navigation agent understand the environment are extracted sequentially from the instructions; these objects reflect the navigation agent's navigation path. K is used to indicate whether the current object is a key object; if so, its value is 1, otherwise it is 0. S represents the weighted average of the cosine similarity between the semantic embedding of the current object and the semantic embedding of key objects. Key objects appearing in the already executed instruction portions have a weight of 0; as instruction execution progresses, the weights of subsequent key objects gradually increase, and the sum of all weights is 1. Finally, all O's are... i Stacked together, they form an object decoupling matrix; thus, it can be seen that the decoupling representation of an object is actually unrelated to the specific object category, and it does not contain specific representation data of a certain category (such as specific visual data representing a certain category of objects). Because it uses a relative representation, even if a new, unseen category appears, the navigation agent will not feel unfamiliar with it.

[0062] 204) The scene decoupling matrix and the object decoupling matrix are concatenated row by row to form a view observation decoupling matrix containing rich semantic information and independent of specific data categories. This matrix is ​​then input into a variational autoencoder, which uses its ability to capture the latent structure and the correlation between features of the input data. In this way, environmental decoupling knowledge is extracted from the latent space. This knowledge representation method reveals the spatial semantic relationship between environmental layout and objects more effectively.

[0063] S3. Capture the depth view of the current location, combine it with the instruction features from S1 and the RGB view from S2, and obtain an egocentric semantic map through a cross-modal attention network. The depth view in this step needs to be actively acquired; it depends primarily on the current environmental location. The depth view acquired at different locations differs, and it differs from the RGB view. Figure 1 One-to-one correspondence. This embodiment explains the specific process of obtaining an egocentric semantic map through a cross-modal attention network as follows:

[0064] 301) For the obtained depth view of the current location, project its ground onto a self-centered grid map φ t This grid map represents the current spatial occupancy, where t represents the current time step. For ground projection, inverse projection is first performed using camera intrinsics to convert depth pixels into point cloud data in 3D space. Then, the point cloud data is projected onto a 2D plane parallel to the ground. Furthermore, a truncated ResNet18 network is used to process this grid map φ. t Feature extraction is performed to generate a grid map feature representation. For the input instruction, the encoded instruction features Λ are obtained through the BERT model;

[0065] 302) During each navigation task, the instructions are given in advance and remain unchanged, but the egocentric map constantly changes based on the position of the navigation agent. Therefore, grid map features are used here. As the query Q, the instruction feature Λ serves as the key K and value V, and the attention representation is obtained through a scaled dot product attention mechanism, calculated as follows:

[0066]

[0067] Among them W q W k W v They are all learnable square matrices of parameters with dimension d. It is a grid map φ t Attention representation on;

[0068] 303) Natural language instructions often implicitly contain spatial semantic descriptions. Here, we further calculate the semantic similarity between text instruction features and map features, i.e., cosine similarity. By combining this metric with the raw attention score, the navigation agent can more accurately identify the map regions most relevant to the instruction semantics, thus encouraging the navigation agent to act towards the most relevant regions, as shown below:

[0069]

[0070] Among them, W R and W Λ It is a learnable parameter matrix. In the calculation... When multiplying by matrices, the numerator is multiplied by matrix multiplication, and the denominator is multiplied element-wise. ||·|| represents the L2 norm; due to cosine similarity... The value of is between -1 and 1, so it needs to be scaled to between 0 and 1 to get . To ensure and The values ​​of these values ​​are within the same range; finally, they are combined with the original attention score for a weighted calculation, where, This indicates the corrected grid map φ t The comprehensive attention representation on the graph, where λ1 and λ2 represent the weights used to balance the two attention scores, are both greater than 0 and sum to 1. For ease of description later, they are referred to as the "joint balancing factor".

[0071] 304) By using two Unet networks based on the ResNet18 architecture (i.e., Res-Unet), semantic information reasoning and completion were performed, and semantic information outside the observation area of ​​the navigation agent was further obtained, as shown below:

[0072]

[0073] in, This is a semantic segmentation of the ground projection of the RGB view. In order to better handle unseen classes, the SAM model

[48] is used here for zero-sample semantic segmentation; thus, the egocentric semantic map is completed. The reasoning and generation not only include a top-down egocentric semantic map of the current field of vision of the navigation agent, but also supplements the semantic map of unknown areas that cannot be observed outside its field of vision, thus making full preparations for subsequent steps.

[0074] S4. Combining the instruction features from S1 and the egocentric semantic map obtained in S3, a series of navigable waypoints leading to the target are generated using the waypoint generation network. Specifically, a truncated ResNet18 is first used to process the egocentric semantic map. Encode to obtain its map feature representation Then, based on equations (9) and (10), and taking them... Replace with Further, waypoint attention representations were obtained. Finally, by representing waypoint attention The initial waypoints are input into a waypoint prediction network based on the Res-Unet architecture to obtain all currently navigable waypoints leading to the target.

[0075] S5. Combining the environmental decoupling knowledge obtained in S2 and the series of navigable waypoints obtained in S4, predict the navigation action that the navigation agent should perform. Specifically, firstly, all the predicted waypoints and the previously learned environmental decoupling knowledge are sent to the planning and decision module. In this module, the navigable waypoint closest to the location of the navigation agent is selected as the sub-target. Then, the environmental decoupling knowledge is combined, and the improved DD-PPO strategy

[50] pre-trained in the PointNav task

[49] is used to complete the action prediction.

[0076] S6. Repeat the above steps until the navigation agent determines that the navigation task is complete or the maximum number of steps has been reached. It should be noted that the navigation agent uses an action set consisting of four actions: move forward 0.25m, turn left 15°, turn right 15°, and stop. If a stop decision is made within 3m of the target location, and the navigation agent completes a navigation operation within a fixed time budget of 500 steps, then the navigation is considered successful.

[0077] This embodiment uses a zero-shot approach to perform navigation tasks. That is, the navigation agent is trained in a limited set of visible data categories (including object categories and environment categories). However, during the test navigation process, if unseen data categories appear, the navigation agent needs to use the semantic embeddings of all provided data categories to complete generalized navigation on the unseen data categories.

[0078] Preferably, to evaluate the zero-shot navigation performance of the navigation agent, this embodiment carefully examines the data in the train split and validation unseen (val-unseen) split of the R2R-CE dataset, and enhances the data categories in the val-unseen split that the navigation agent did not see during training by removing data categories from the val-unseen split contained in the train split. Finally, this embodiment uses the zero-shot performance on the val-unseen split for comparison with other methods. The experimental process mainly focuses on the following five performance evaluation metrics:

[0079] 1) TL: Represents the length of the proxy navigation path (in meters);

[0080] 2) NE: Represents the average distance (in meters) between the agent's final location and the target;

[0081] 3) SR: Indicates the percentage of times the agent stops within 3 meters of the target (navigation is considered successful as long as the agent stays within 3 meters of the target).

[0082] 4) OSR: Represents the percentage of tasks in which the agent successfully reaches the target location when assisted by an "Oracle".

[0083] 5) SPL: Path-weighted success rate, which measures both the accuracy and efficiency of navigation.

[0084] For all results in the table below, bold text indicates the optimal value for the current indicator, "↓" indicates the lower the better, "↑" indicates the higher the better, "TL" and "NE" are in meters, and "SR", "SPL" and "OSR" are reported as percentage values.

[0085] In this implementation, a GPU server equipped with multiple GetForce RTX 3090 graphics cards was used for parallel computing to improve data processing efficiency. To optimize the performance of the navigation agent in visual and language navigation tasks, the Adam optimizer was selected and trained with a learning rate of 0.0002. Furthermore, the navigation agent has access to a egocentric RGB-D view with a 90-degree horizontal field of view.

[0086] This embodiment was compared with current high-performing zero-shot learning-based navigation agents, and the results are shown in Table 1. As can be seen from the table, this navigation agent achieved the best performance under the zero-shot setting. Compared to A... 2 Nav

[47] is a cutting-edge method for zero-shot navigation in VLN-CE and was the best performing method at the time. This navigation agent still significantly outperforms it with an absolute improvement of 5.2% (relative improvement of 23.0%) on SR and an absolute improvement of 4.1% (relative improvement of 36.9%) on SPL. For comparison methods such as CLIP-Nav

[42] and Seq CLIP-Nav

[42] , the performance of this navigation agent is also several times better than them. It is not difficult to find that other methods use large models (such as large language models or visual language models) for navigation planning, while this navigation agent mainly relies on its own experience, indicating that this navigation agent has an advantage in experience generalization. And it is precisely because it does not use large models for navigation planning that this navigation agent is better in navigation efficiency due to its lighter weight.

[0087] This embodiment was also compared with a very good supervised learning-based navigation agent, and the results are shown in Table 1. It can be found that the present navigation agent can outperform Seq2Seq

[21] on SR with an absolute improvement of 2.8% (relative improvement of 11.2%), and on SASRA

[25] SR, the present navigation agent also has an absolute improvement of 3.8% (relative improvement of 15.8%). The present navigation agent outperforms all supervised learning methods in Table 1 on NE, especially NaVid

[52] , a pioneering video-based navigation method. The present navigation agent can also beat its NE, which shows that the present navigation agent is closer to the target point at the end of the navigation and reduces the navigation error.

[0088] In addition to the comparisons above, ablation experiments were conducted on the algorithm proposed in this embodiment. The specific results are shown in Tables 2 and 3. Table 2 shows that different combinations of λ1 and λ2 significantly impact the model's performance. When either λ1 or λ2 is 0, meaning only the original attention score or only the cosine similarity score is used, the performance of the navigation agent is greatly reduced. When both λ1 and λ2 are 0.5, the navigation agent achieves optimal performance. This indicates that both the original attention score and the cosine similarity score make significant contributions to the navigation agent, and the comprehensive attention representation under this combination of values ​​yields the greatest benefit.

[0089] Table 3 further quantitatively describes the contribution of environmental decoupling knowledge to zero-shot navigation. The results show that without environmental decoupling knowledge for zero-shot navigation, the performance of the navigation agent is significantly reduced. This is mainly because the navigation agent only learns knowledge about specific data categories, but not the relationships between categories, resulting in the inability to generalize the learned experience to unseen data.

[0090] Preferably, embodiments of this application also provide a specific implementation of an electronic device capable of implementing all steps of the zero-sample continuous visual language navigation method based on visual semantics and map integration described in the above embodiments. The electronic device specifically includes the following:

[0091] Processor, memory, communications interface, and bus;

[0092] The processor, memory, and communication interface communicate with each other via a bus; the communication interface is used to realize information transmission between server-side devices, metering devices, and user-side devices.

[0093] The processor is used to call the computer program in memory. When the processor executes the computer program, it implements all the steps in the zero-sample continuous visual language navigation method based on visual semantics and map integration in the above embodiments.

[0094] Table 1. Performance comparison of this embodiment with other excellent navigation agents based on supervised learning and zero-shot learning on the R2R-CE val-unseen dataset. The third column indicates whether the navigation agent uses a large model (e.g., a large language model or a visual language model) for navigation planning.

[0095]

[0096] Table 2. Ablation Study: The Impact of Different Combinations of Joint Balance Factors on the Performance of Navigation Agents

[0097] Joint balancing factor OSR↑ SR↑ SPL↑ [lambda1 = 0.00, lambda2 = 1.00] 28.6 16.5 9.2 [lambda]1 = 0.25, [lambda]2 = 0.75 33.3 16.7 12.9 ​ 33.3 27.8 15.2 [lambda]1 = 0.75, [lambda]2 = 0.25 33.2 18.8 12.6 [lambda1 = 1.00, lambda2 = 0.00] 16.7 11.1 11.1

[0098] Table 3. Ablation Study: Impact of Environmental Decoupling Knowledge on the Performance of Navigation Agents

[0099] Model TL↓ NE↓ OSR↑ SR↑ SPL↑ The current navigation agent without using environment decoupling knowledge 6.49 4.38 16.7 5.6 5.6 The current navigation agent using environment decoupling knowledge 11.60 4.10 33.3 27.8 15.2

[0100] Preferably, embodiments of this application also provide a specific implementation of an electronic device capable of implementing all steps in the zero-sample continuous visual language navigation method based on visual semantics and map integration described in the above embodiments. The electronic device specifically includes the following:

[0101] Processor, memory, communications interface, and bus;

[0102] The processor, memory, and communication interface communicate with each other via a bus; the communication interface is used to realize information transmission between server-side devices, metering devices, and user-side devices.

[0103] The processor is used to call the computer program in memory. When the processor executes the computer program, it implements all the steps in the policy reinforcement learning method based on fuzzy rules and decision trees in the above embodiments.

[0104] Preferably, embodiments of this application also provide a computer-readable storage medium capable of implementing all steps of the zero-sample continuous visual language navigation method based on visual semantics and map integration in the above embodiments. The computer-readable storage medium stores a computer program that, when executed by a processor, implements all steps of the zero-sample continuous visual language navigation method based on visual semantics and map integration in the above embodiments.

[0105] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0106] While this application provides method operation steps as shown in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-inventive labor. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only execution order. In actual device or client product execution, the method can be executed sequentially as shown in the embodiments or drawings, or in parallel (e.g., in a parallel processor or multi-threaded processing environment).

[0107] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0108] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0109] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0110] It should be stated that the positional relationships and detailed descriptions shown in the accompanying drawings are intended to provide illustrative examples of the technical solutions to facilitate understanding of the core concepts and operating principles of the invention. These examples are not intended to limit the scope of patent protection, but rather serve as auxiliary means of explaining the technical solutions. Within the framework of this invention, those skilled in the art can, based on prior art knowledge and the guidance of this invention, make appropriate modifications, equivalent substitutions, or innovations to the technical solutions through logical analysis, reasoning, and experimentation to adapt to different application scenarios and needs. Such changes and adjustments, as long as they do not depart from the basic principles and spirit of the invention, should be considered within the scope of protection defined in the claims.

[0111] This detailed description of the embodiments illustrates multiple aspects of the technical solution, but it does not exclude other possible implementations. Those skilled in the art can make various specific modifications based on the teachings of this invention without departing from the spirit and scope of the claims, including but not limited to adjustments in hardware configuration, software implementation, system architecture, and operating procedures. These modifications, aimed at optimizing the performance, efficiency, or reliability of the technical solution, or expanding its application scope, all fall within the protection scope of this invention.

[0112] Furthermore, the terminology and expressions used in this invention are not intended to provide a restrictive definition of the technical solutions, but rather to offer clarity and consistency in the description. The use of terminology in the description is intended to facilitate understanding and description of the technical content and should not be considered as a limitation on the scope of patent protection. Therefore, any reasonable interpretation and application of the terminology, as long as it conforms to the purpose of the invention and the substance of the claims, should be included within the scope of protection of this invention.

[0113] This invention is not limited to the embodiments described above. The above description of specific embodiments is intended to illustrate and explain the technical solutions of this invention. The specific embodiments described above are merely illustrative and not restrictive. Without departing from the spirit and scope of the claims, those skilled in the art can make many specific modifications based on the teachings of this invention, and these modifications all fall within the scope of protection of this invention.

[0114] References:

[0115] [1]Gu J, Stefani E, Wu Q, et al. Vision-and-language navigation: A survey of tasks, methods, and future directions[J]. arXiv preprint arXiv:2203.12667,2022.

[0116] [2]Wu W,Chang T,Li X,et al.Vision-language navigation:a survey and taxonomy[J].Neural Computing and Applications,2024,36(7):3291-3316.

[0117] [3]Chen S,Guhur P L,Tapaswi M,et al.Learning from unlabeled 3denvironments for vision-and-language navigation[C] / / European Conference onComputer Vision.Cham:Springer Nature Switzerland,2022:638-655.

[0118] [4]Tan H,Yu L,Bansal M.Learning to navigate unseen environments:Backtranslation with environmental dropout[J].arXiv preprint arXiv:1904.04195,2019.

[0119] [5]Li J,Tan H,Bansal M.Envedit:Environment editing for vision-and-language navigation[C] / / Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition.2022:15407-15417.

[0120] [6]Li X,Wang Z,Yang J,et al.Kerm:Knowledge enhanced reasoning forvision-and-language navigation[C] / / Proceedings of the IEEE / CVF Conference onComputer Vision and Pattern Recognition.2023:2583-2592.

[0121] [7]Zhu F,Lee V,Chang X,et al.Vision language navigation withknowledge-driven environmental dreamer[J].2023.

[0122] [8]Zhu F,Zhu Y,Chang X,et al.Vision-language navigation with self-supervised auxiliary reasoning tasks[C] / / Proceedings of the IEEE / CVFconference on computer vision and pattern recognition.2020:10012-10022.

[0123] [9]Ma C Y,Wu Z,AlRegib G,et al.The regretful agent:Heuristic-aidednavigation through progress estimation[C] / / Proceedings of the IEEE / CVFconference on Computer Vision and Pattern Recognition.2019:6732-6740.

[0124]

[10] Ma C Y,Lu J,Wu Z,et al.Self-monitoring navigation agent viaauxiliary progress estimation[J].arXiv preprint arXiv:1901.03035,2019.

[0125]

[11] Wang X,Huang Q,Celikyilmaz A,et al.Reinforced cross-modalmatching and self-supervised imitation learning for vision-languagenavigation[C] / / Proceedings of the IEEE / CVF conference on computer vision andpattern recognition.2019:6629-6638.

[0126]

[12] Lu J,Batra D,Parikh D,et al.Vilbert:Pretraining task-agnosticvisiolinguistic representations for vision-and-language tasks[J].Advances inneural information processing systems,2019,32.

[0127]

[13] Guhur P L,Tapaswi M,Chen S,et al.Airbert:In-domain pretrainingfor vision-and-language navigation[C] / / Proceedings of the IEEE / CVFInternational Conference on Computer Vision.2021:1634-1643.

[0128]

[14] Hao W,Li C,Li X,et al.Towards learning a generic agent forvision-and-language navigation via pre-training[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition.2020:13137-13146.

[0129]

[15] Qiao Y,Qi Y,Hong Y,et al.Hop:History-and-order aware pre-trainingfor vision-and-language navigation[C] / / Proceedings of the IEEE / CVF Conferenceon Computer Vision and Pattern Recognition.2022:15418-15427.

[0130]

[16] Li X,Li C,Xia Q,et al.Robust navigation with language pretrainingand stochastic sampling[J].arXiv preprint arXiv:1909.02244,2019.

[0131]

[17] Magassouba A,Sugiura K,Kawai H.CrossMap transformer:A crossmodalmasked path transformer using double back-translation for vision-and-languagenavigation[J].IEEE Robotics and Automation Letters,2021,6(4):6258-6265.

[0132]

[18] Chen S,Guhur P L,Tapaswi M,et al.Think global,act local:Dual-scale graph transformer for vision-and-language navigation[C] / / Proceedings ofthe IEEE / CVF Conference on Computer Vision and Pattern Recognition.2022:16537-16547.

[0133]

[19] Zhou X,Liu W,Mu Y.Rethinking the spatial route prior in vision-and-language navigation[J].arXiv preprint arXiv:2110.05728,2021.

[0134]

[20] Wang Z,Li X,Yang J,et al.Gridmm:Grid memory map for vision-and-language navigation[C] / / Proceedings of the IEEE / CVF International Conferenceon Computer Vision.2023:15625-15636.

[0135]

[21] Krantz J,Wijmans E,Majumdar A,et al.Beyond the nav-graph:Vision-and-language navigation in continuous environments[C] / / Computer Vision–ECCV2020:16th European Conference,Glasgow,UK,August 23–28,2020,Proceedings,PartXXVIII 16.Springer International Publishing,2020:104-120.

[0136]

[22] Hong Y,Wang Z,Wu Q,et al.Bridging the gap between learning indiscrete and continuous environments for vision-and-language navigation[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition.2022:15439-15449.

[0137]

[23] Georgakis G,Schmeckpeper K,Wanchoo K,et al.Cross-modal maplearning for vision and language navigation[C] / / Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition.2022:15460-15470.

[0138]

[24] Wang T,Wu Z,Yao F,et al.Graph-Based Environment Representationfor Vision-and-Language Navigation in Continuous Environments[C] / / ICASSP2024-2024 IEEE International Conference on Acoustics,Speech and SignalProcessing(ICASSP).IEEE,2024:8331-8335.

[0139]

[25] Irshad M Z,Mithun N C,Seymour Z,et al.Semantically-aware spatio-temporal reasoning agent for vision-and-language navigation in continuousenvironments[C] / / 2022 26th International Conference on Pattern Recognition(ICPR).IEEE,2022:4065-4071.

[0140]

[26] Wang H,Liang W,Van Gool L,et al.Dreamwalker:Mental planning forcontinuous vision-language navigation[C] / / Proceedings of the IEEE / CVFInternational Conference on Computer Vision.2023:

[0141] 10873-10883.

[0142]

[27] Krantz J,Gokaslan A,Batra D,et al.Waypoint models forinstruction-guided navigation in continuous environments[C] / / Proceedings ofthe IEEE / CVF International Conference on Computer Vision.2021:15162-15171.

[0143]

[28] He Z,Wang L,Li S,et al.Mlanet:Multi-level attention network withsub-instruction for continuous vision-and-language navigation[J].arXivpreprint arXiv:2303.01396,2023.

[0144]

[29] Verma V K,Arora G,Mishra A,et al.Generalized zero-shot learningvia synthesized examples[C] / / Proceedings of the IEEE conference on computervision and pattern recognition.2018:4281-4289.

[0145]

[30] Naeem M F,Xian Y,Tombari F,et al.Learning graph embeddings forcompositional zero-shot learning[C] / / Proceedings of the IEEE / CVF Conferenceon Computer Vision and Pattern Recognition.2021:953-962.

[0146]

[31] Rahman S,Khan S,Porikli F.Zero-shot object detection:Learning tosimultaneously recognize and localize novel concepts[C] / / Asian Conference onComputer Vision.Cham:Springer International Publishing,2018:547-563.

[0147]

[32] Hayat N,Hayat M,Rahman S,et al.Synthesizing the unseen for zero-shot object detection[C] / / Proceedings of the Asian conference on computervision.2020.

[0148]

[33] Rahman S,Khan S,Barnes N.Polarity loss for zero-shot objectdetection[J].arXiv preprint arXiv:1811.08982,2018.

[0149]

[34] Bucher M,Vu T H,Cord M,et al.Zero-shot semantic segmentation[J].Advances in Neural Information Processing Systems,2019,32.

[0150]

[35] Kato N,Yamasaki T,Aizawa K.Zero-shot semantic segmentation viavariational mapping[C] / / Proceedings of the IEEE / CVF International Conferenceon Computer Vision Workshops.2019:0-0.

[0151]

[36] Majumdar A,Aggarwal G,Devnani B,et al.Zson:Zero-shot object-goalnavigation using multimodal goal embeddings[J].Advances in Neural InformationProcessing Systems,2022,35:32340-32352.

[0152]

[37] Li X,Zhang S,Lu Y,et al.Zero-Shot Object Goal Visual NavigationWith Class-Independent Relationship Network[J].arXiv preprint arXiv:2310.09883,2023.

[0153]

[38] Gadre S Y,Wortsman M,Ilharco G,et al.Cows on pasture:Baselinesand benchmarks for language-driven zero-shot object navigation[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition.2023:23171-23181.

[0154]

[39] Zhao Q,Zhang L,He B,et al.Zero-shot object goal visual navigation[C] / / 2023 IEEE International Conference on Robotics and Automation(ICRA).IEEE,2023:2025-2031.

[0155]

[40] Al-Halah Z,Ramakrishnan S K,Grauman K.Zero experience required:Plug&play modular transfer learning for semantic visual navigation[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition.2022:17031-17041.

[0156]

[41] Sun X,Lau L,Zhi H,et al.Prioritized Semantic Learning for Zero-shot Instance Navigation[J].arXiv preprint arXiv:2403.11650,2024.

[0157]

[42] Dorbala V S,Sigurdsson G,Piramuthu R,et al.Clip-nav:Using clipfor zero-shot vision-and-language navigation[J].arXiv preprint arXiv:2211.16649,2022.

[0158]

[43] Huang J,Zhang H,Zhao M,et al.IVLMap:Instance-Aware VisualLanguage Grounding for Consumer Robot Navigation[J].arXiv preprint arXiv:2403.19336,2024.

[0159]

[44] Chen J,Lin B,Xu R,et al.MapGPT:Map-Guided Prompting with AdaptivePath Planning for Vision-and-Language Navigation[C] / / Proceedings of the 62ndAnnual Meeting of the Association for Computational Linguistics(Volume 1:LongPapers).2024:9796-9810.

[0160]

[45] Zhou G,Hong Y,Wu Q.Navgpt:Explicit reasoning in vision-and-language navigation with large language models[C] / / Proceedings of the AAAIConference on Artificial Intelligence.2024,38(7):7641-7649.

[0161]

[46] Li D,Chen W,Lin X.TINA:Think,Interaction,and Action Framework forZero-Shot Vision Language Navigation[J].arXiv preprint arXiv:2403.08833,2024.

[0162]

[47] Chen P,Sun X,Zhi H,et al.$A^2$Nav:Action-Aware Zero-Shot RobotNavigation by Exploiting Vision-and-Language Ability of Foundation Models[J].arXiv preprint arXiv:2308.07997,2023.

[0163]

[48] Kirillov A,Mintun E,Ravi N,et al.Segment anything[C] / / Proceedingsof the IEEE / CVF International Conference on Computer Vision.2023:4015-4026.

[0164]

[49] Anderson P,Chang A,Chaplot D S,et al.On evaluation of embodiednavigation agents[J].arXiv preprint arXiv:1807.06757,2018.

[0165]

[50] Wijmans E,Kadian A,Morcos A,et al.Dd-ppo:Learning near-perfectpointgoal navigators from 2.5billion frames[J].arXiv preprint arXiv:1911.00357,2019.

[0166]

[51] Raychaudhuri S,Wani S,Patel S,et al.Language-aligned waypoint(law)supervision for vision-and-language navigation in continuousenvironments[J].arXiv preprint arXiv:2109.15207,2021.

[0167]

[52] Zhang J,Wang K,Xu R,et al.NaVid:Video-based VLM Plans the NextStep for Vision-and-Language Navigation[J].arXiv preprint arXiv:2402.15852,2024.

Claims

1. A zero-sample continuous visual-language navigation method based on visual semantics and map ensemble, characterized in that, Includes the following steps: S1. Receive the natural language instructions for the current navigation task, encode and extract features from them using a text encoder to obtain instruction features; S2. Capture the RGB view of the current location and decouple the environmental knowledge in the RGB view to obtain environmental decoupling knowledge; S3. Capture the Depth view of the current location, combine the instruction features in S1 and the RGB view in S2, and obtain an egocentric semantic map through a cross-modal attention network; S4. Combining the instruction features in S1 and the egocentric semantic map obtained in S3, generate several navigable waypoints leading to the target; S5. Combining the environmental decoupling knowledge obtained in S2 and the navigable waypoints obtained in S4, predict the navigation action that the navigation agent should perform; S6. Repeat the above steps until the navigation agent determines that the navigation task is completed or the maximum number of steps has been reached.

2. The zero-sample continuous visual-language navigation method based on visual semantics and map ensemble according to claim 1, characterized in that, In step S1, the BERT model is used as a text encoder to extract features from the input natural language instructions and capture semantic information through its bidirectional encoding capability.

3. The zero-sample continuous visual-language navigation method based on visual semantics and map ensemble according to claim 1, characterized in that, The specific workflow for decoupling environmental knowledge using the environmental knowledge decoupling module in step S2 is as follows: First, the SAM model is used to perform zero-shot segmentation on the acquired RGB view with environmental information to obtain the corresponding object semantic class; Then, the scene information in the RGB view is decoupled. For the layout information of object category i, it is represented as S. i =[C x C y A r ], where C x and C y Indicate the coordinates of its centroid, A r This represents the area percentage of the object relative to the current RGB view scene; ultimately, all S... i These are stacked together to form a scene decoupling matrix, which stores navigation information in the current RGB view scene; Next, the object information in the RGB view is decoupled, and for object category i, it is represented as O. i = [V, K, S], where V represents the visibility of the object; if the object is observable in the current view, its value is 1, otherwise it is 0. Several key objects that help the navigation agent understand the environment are extracted sequentially from the instructions; these objects reflect the navigation agent's navigation path. K is used to indicate whether the current object is a key object; if it is, its value is 1, otherwise it is 0. S represents the weighted average of the cosine similarity between the semantic embedding of the current object and the semantic embedding of the key objects. Key objects appearing in the already executed instruction portion have a weight of 0; as the instruction execution progresses, the weight of subsequent key objects gradually increases. Finally, all O's are... i Stacked together, they form an object decoupling matrix; Finally, the scene decoupling matrix and the object decoupling matrix are concatenated row by row to form a view-observation decoupling matrix that is independent of specific data categories. The view-observation decoupling matrix is ​​then input into a variational autoencoder to capture the latent structure and correlations between features of the input data and extract environmental decoupling knowledge.

4. The zero-sample continuous visual-language navigation method based on visual semantics and map ensemble according to claim 1, characterized in that, In step S3, the specific process of obtaining the egocentric semantic map through a cross-modal attention network is as follows: First, for the obtained depth view of the current location, project its ground onto a grid map. The grid map represents the current space occupancy, where t represents the current time step; for ground projection operations, inverse projection is first performed using camera intrinsic parameters to convert depth pixels into point cloud data in three-dimensional space, and then the point cloud data is projected onto a two-dimensional plane parallel to the ground. Then, a truncated ResNet18 network was used to process this grid map. Feature extraction is performed to generate a grid map feature representation. For the input instruction, the encoded instruction features Λ are obtained through the BERT model; Next, use grid map features As the query Q, the instruction feature Λ serves as the key K and value V, and the attention representation is obtained through a scaled dot product attention mechanism, calculated as follows: Among them W q W k W v They are all learnable square matrices of parameters with dimension d. It is a grid map Attention representation on; Further calculations were performed using command features Λ and grid map features. The degree of semantic similarity between them, i.e., cosine similarity By combining cosine similarity with the original attention score, the navigation agent can more accurately identify the map region most relevant to the semantics of the instruction feature Λ, thus encouraging the navigation agent to act towards the most relevant region, as shown below: Among them, W R and W Λ It is a learnable parameter matrix; in computation When calculating the numerator, matrix multiplication is used, and element-wise multiplication is used in the denominator. ||·|| represents the L2 norm; cosine similarity. The value ranges from -1 to 1. Scaled to between 0 and 1 To ensure and The range of values ​​is the same; ultimately, it will... The weighted calculation is performed by combining the original attention score with the original attention score. This indicates the corrected version on the grid map. The comprehensive attention representation on the graph, where λ1 and λ2 represent the weights used to balance the two attention scores, are both greater than 0, and sum to 1; Finally, semantic information was inferred and completed using two Unet networks based on the ResNet18 architecture, namely Res-Unet, and semantic information outside the observation area of ​​the navigation agent was further obtained, as shown below: in, This involves semantic segmentation of the ground projection from the RGB view; thus, the egocentric semantic map is complete. The reasoning and generation not only include a top-down egocentric semantic map of the current field of vision of the navigation agent, but also supplements the semantic map of unknown areas that cannot be observed outside its field of vision, thus making full preparations for subsequent steps.

5. The zero-sample continuous visual-language navigation method based on visual semantics and map ensemble according to claim 1, characterized in that, In step S4, the specific process for generating several navigable waypoints leading to the target based on the waypoint generation network is as follows: First, ResNet18 was used again to refine the egocentric semantic map. Encode to obtain an egocentric semantic map Feature representation Then, feature representations of egocentric semantic maps are used. As the query Q, the instruction feature Λ serves as the key K and value V; waypoint attention representations are obtained through a scaled dot product attention mechanism. The calculation method is as follows: Among them W q W k W v They are all learnable square matrices of parameters with dimension d; Finally, by representing waypoint attention The initial waypoint is input into a Res-Unet network to obtain all currently navigable waypoints leading to the target.

6. The zero-sample continuous visual-language navigation method based on visual semantics and map ensemble according to claim 1, characterized in that, The specific process of predicting the actions to be performed by the navigation agent in step S5 is as follows: all the predicted waypoints and the previously learned environmental decoupling knowledge are sent into the planning and decision module. In this planning and decision module, the navigable waypoint closest to the location of the navigation agent is selected as the sub-target. Then, combined with the environmental decoupling knowledge, the DD-PPO policy pre-trained in the PointNav task is used to complete the action prediction.

7. The zero-sample continuous visual-language navigation method based on visual semantics and map ensemble according to claim 1, characterized in that, In step S6, the navigation agent uses an action set consisting of four actions: moving forward 0.25m, turning left 15°, turning right 15°, and stopping. If a stop decision is made within 3m of the target location, and the navigation agent completes a navigation cycle within a fixed time budget of 500 steps, then the navigation is considered successful.

8. The zero-sample continuous visual-language navigation method based on visual semantics and map ensemble according to claim 1, characterized in that, The navigation agent is trained in a limited number of visible data categories. If an unseen data category appears during the test, the navigation agent completes generalized navigation on the unseen data category by using the semantic embeddings of all provided data categories. Data categories include object categories and environment categories.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the zero-sample continuous visual language navigation method based on visual semantics and map integration as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the zero-sample continuous visual language navigation method based on visual semantics and map integration as described in any one of claims 1 to 8.

Images