Zero-sample continuous visual language navigation method based on visual semantics and map integration

Abstract

The invention discloses a zero-sample continuous visual language navigation method based on visual semantics and map integration, which comprises the following steps: S1, receiving a natural language instruction of a current navigation task, and carrying out encoding and feature extraction on the natural language instruction through a text encoder to obtain an instruction feature; s2, capturing an RGB view during task execution, and decoupling environment knowledge in the RGB view to obtain environment decoupling knowledge; s3, capturing a Depth view during task execution, and obtaining a semantic map of a self-center through a cross-modal attention network in combination with an RGB view and instruction features; s4, generating a series of navigable waypoints leading to the target according to the waypoint generation network in combination with the semantic map and the instruction features; s5, predicting the current navigation action of the navigation agent by using the navigable waypoints and the environment decoupling knowledge; and S6, repeating the steps until the navigation task is completed or the maximum moving step number is reached.

Description

Technical Field

[0001] The present invention relates to the technical field of visual - language navigation cross - modality, and particularly relates to a zero - shot continuous visual - language navigation method based on visual semantics and map integration. Background Art

[0002] Visual - Language Navigation (VLN) [1 - 2] requires a navigation agent to be able to understand natural - language instructions and complete the navigation task from a starting point to a target point according to the instructions. It is mainly divided into two categories: one is discrete VLN executed in a discrete environment, which mainly relies on a predefined connected graph for navigation and can be regarded as an abstract version in a real - world situation; the other is continuous VLN (VLN - CE) for navigation in a continuous environment. Compared with discrete VLN, it can better reflect the complexity of navigation in a real - world environment and has positive significance for promoting the practical application of the model.

[0003] Discrete VLN is an early version of the development of VLN. The research work therein mainly focuses on data augmentation [3 - 7], auxiliary measures [8 - 11], Pretraining [12 - 16], map learning [17 - 20], etc. Continuous VLN [21 - 28] is a newly proposed research paradigm in recent years. Among them, Krantz et al.

[21] first proposed the VLN task in a continuous environment and initially solved it using the WPN model in

[27] ; Hong et al.

[22] promoted the transition of many models working on discrete VLN to continuous VLN by bridging the gap between discrete VLN and continuous VLN; ERG

[24] promoted the understanding of the environment by the navigation agent by representing the environment in the form of a graph; other methods such as map prediction

[23] and mental planning

[26] were proposed to further improve the navigation performance.

[0004] Zero-shot learning (ZSL) is one of the ways for current artificial intelligence models to transition to real-world applications. Since the data categories in the real world are diverse, it is impossible for models to exhaustively learn each data category during the training process. Early zero-shot learning was mainly applied to classification tasks [29-30]. With the rise and development of other computer vision tasks, zero-shot learning has gradually been applied to tasks such as object detection [31-33] and image segmentation [34-35]. In recent years, the dilemmas of some navigation tasks such as Object-Goal Navigation [36-39], Semantic Visual Navigation

[40] , Instance Navigation

[41] , and Vision-and-Language Navigation [42-47] have also been solved through zero-shot learning. Among them, the method of using CLIP [36, 38, 41, 42] to participate in the navigation process has achieved good results. Since large language models have powerful cognitive and reasoning abilities, using GPT to assist navigation [44-47] is a direction worthy of exploration, and many works have demonstrated the feasibility of this method and the excellent performance of the model after integrating GPT.

[0005] From the above research, it can be seen that a large number of VLN studies mainly focus on supervised learning. This method mainly relies on a large amount of training data, which will cause the model to be unable to generalize to new and unseen objects and environments. Moreover, real-world navigation contains various complex and diverse data categories, which cannot be fully included in the training data. To address this challenge, enabling the navigation agent to effectively generalize and navigate on unseen data categories after learning limited knowledge is a very effective solution, which is called "zero-shot navigation", and this VLN task is also "zero-shot VLN".

[0006] Recently, a small number of relatively new studies [42-47] have focused on zero-shot VLN tasks. However, it is undeniable that these studies mainly achieve navigation planning by relying on the rich internal knowledge and powerful reasoning ability of large models, and the navigation ability of the navigation agent itself has not been significantly improved. Moreover, due to the usually large size of large models, this leads to low efficiency of relying on large models for navigation and a high computational cost. Summary of the Invention

[0007] The object of the present invention is to overcome the deficiencies in the prior art and provide a zero-shot continuous vision-language navigation method based on visual semantics and map integration. This method provides a way to perform navigation planning without relying on large models (including large language models or vision-language models). This approach can learn navigation experience from limited visible data and flexibly utilize spatial semantic information to make up for the missing information in unseen data classes, ultimately achieving generalization navigation in unseen data classes. Compared with the previous methods, since large models are not familiar with the VLN task and lack the ability to directly learn from the navigation environment to acquire knowledge, the method of the present invention exactly makes up for this defect.

[0008] The object of the present invention is achieved by the following technical solutions:

[0009] A zero-shot continuous vision-language navigation method based on visual semantics and map integration, comprising the following steps:

[0010] S1. Receive the natural language instruction of the current navigation task, encode and extract features from it using a text encoder to obtain instruction features;

[0011] S2. Capture the RGB view of the current location and decouple the environmental knowledge in this RGB view to obtain environmental decoupled knowledge;

[0012] S3. Capture the Depth view of the current location, combine the instruction features in S1 and the RGB view in S2, and obtain an ego-centric semantic map through a cross-modal attention network;

[0013] S4. Combine the instruction features in S1 and the ego-centric semantic map obtained in S3 to generate several navigable waypoints leading to the target;

[0014] S5. Combine the environmental decoupled knowledge obtained in S2 and the navigable waypoints obtained in S4 to predict the navigation actions that the navigation agent should execute;

[0015] S6. Repeat the above steps until the navigation agent determines that the navigation task is completed or the maximum number of movement steps is reached.

[0016] Further, in step S1, the BERT model is used as the text encoder to extract features from the input natural language instruction, and semantic information is captured through its bidirectional encoding ability.

[0017] Further, the specific workflow of using the environmental knowledge decoupling module to decouple environmental knowledge in step S2 is as follows:

[0018] First, use the SAM model to perform zero-shot segmentation on the obtained RGB view with environmental information to obtain the corresponding object semantic classes;

[0019] Then the scene information in the RGB view is decoupled, and the layout information of object category i is expressed as S i =[C x ,C y ,A r ], where C x and C y represents its center of mass coordinates, A r Indicates the area ratio of the object relative to the current RGB view scene; eventually all J i Stacked together to form a scene decoupling matrix, which stores the navigation information in the current RGB view scene;

[0020] Then the object information in the RGB view is decoupled, and the object category i is represented as O i = [V, K, S], where V represents the visibility of the object. If the object can be observed in the current view, its value is 1, otherwise it is 0; a number of key objects that can help the navigation agent understand the environment are sequentially extracted from the instructions, which reflect the navigation path of the navigation agent; K is used to indicate whether the current object is a key object. If so, its value is 1, otherwise it is 0; S represents the weighted average of the cosine similarity between the semantic embedding of the current object and the semantic embedding of the key object, where the weight of the key object that appears in the executed instruction part is 0, and as the instruction execution progresses, the weight of the subsequent key objects gradually increases; finally, all O i Stacked together to form an object decoupling matrix;

[0021] Finally, the scene decoupling matrix and the object decoupling matrix are concatenated row by row to form a view-observation decoupling matrix that is independent of the specific data category; the view-observation decoupling matrix is input into the variational autoencoder to capture the potential structure of the input data and the correlation between the features, and extract the environment decoupling knowledge.

[0022] Furthermore, in step S3, the specific process of obtaining the egocentric semantic map through the cross-modal attention network is as follows:

[0023] First, for the obtained depth view of the current position, its ground is projected onto a grid map φ t , the grid map represents the occupancy of the current space, where t represents the current time step; for the ground projection operation, first perform inverse projection through the camera intrinsic parameters to convert the depth pixels into point cloud data in three-dimensional space, and then project the point cloud data onto a two-dimensional plane parallel to the ground;

[0024] Then use the truncated ResNet18 network to train this grid map φ t Perform feature extraction and generate grid map feature representation For the input instruction, the encoded instruction feature Λ is obtained through the BERT model;

[0025] Next, the grid map feature feature is used as the query Q, the instruction feature Λ as the key K and value V, and the attention representation is obtained through the scaled dot-product attention mechanism, and the calculation method is:

[0026]

[0027] where W q , W k , W v are all learnable square matrices with dimension d, is the attention representation on the grid map φ t ;

[0028] Furthermore, by calculating the semantic proximity degree, i.e., the cosine similarity between the instruction feature Λ and the grid map feature and combining the cosine similarity with the original attention score, the navigation agent can more accurately identify the map area that is semantically most relevant to the instruction feature Λ, promoting the navigation agent to act towards the most relevant area, as follows:

[0029]

[0030] where, W R and W Λ are learnable parameter matrices; when calculating , the numerator uses matrix multiplication, the denominator uses element-wise multiplication, and ‖·‖ represents the L2 norm; the cosine similarity takes values between -1 and 1, and is scaled to between 0 and 1 to obtain to ensure the same value range as ; finally, is combined with the original attention score for weighted calculation, where, represents the comprehensive attention representation on the grid map φ t after correction, and λ1 and λ2 represent the weights used to balance the two attention scores, both of which are greater than 0 and sum to 1;

[0031] Finally, by using two Unet networks based on the ResNet18 architecture, namely Res-Unet, semantic information is inferred and complemented, and further semantic information outside the observation area of the navigation agent is obtained, as follows:

[0032]

[0033] where, is the ground projection semantic segmentation of the RGB view; thus, the egocentric semantic map is completed. The inference and generation of it not only include the top-down egocentric semantic map in the current field of view of the navigation agent, but also supplement the semantic map of the unknown area that cannot be observed outside its field of view, making full preparations for the subsequent steps.

[0034] Furthermore, in step S4, the specific process of generating several navigable waypoints leading to the target according to the waypoint generation network is as follows:

[0035] First, use ResNet18 again to encode the egocentric semantic map to obtain the feature representation of the egocentric semantic map of

[0036] Then, use the feature representation of the egocentric semantic map as the query Q, the instruction feature Λ as the key K and the value V; obtain the waypoint attention representation through the scaled dot-product attention mechanism The calculation method is:

[0037]

[0038] where W q , W k , W v are all learnable square matrices of dimension d;

[0039] Finally, by inputting the waypoint attention representation and the initial waypoint into a Res-Unet network, all the currently navigable waypoints leading to the target are obtained.

[0040] Furthermore, in step S5, the specific process of predicting the actions of the navigation agent is as follows: Send all the predicted waypoints and the previously learned environment decoupling knowledge into the planning and decision-making module. In this planning and decision-making module, first select the navigable waypoint closest to the position of the navigation agent as the sub-goal, then combine the environment decoupling knowledge, and use the DD-PPO policy pre-trained on the PointNav task to complete the action prediction.

[0041] Furthermore, in step S6, the navigation agent uses an action set including four actions, namely moving forward 0.25m, turning left 15°, turning right 15°, and stopping; if a stop decision is made within 3m of the target position and the navigation agent completes a navigation within a fixed time budget of 500 steps, then this navigation is considered successful.

[0042] Furthermore, the navigation agent is trained in a limited set of visible data categories. If an unseen data category appears during the testing process, the navigation agent can complete generalized navigation on the unseen data category by leveraging the semantic embeddings of all provided data categories. The data categories include object categories and environmental categories.

[0043] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the steps of the zero-shot continuous vision-language navigation method based on vision semantics and map integration.

[0044] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the zero-shot continuous vision-language navigation method based on vision semantics and map integration.

[0045] Compared with the prior art, the beneficial effects brought by the technical solution of the present invention are as follows:

[0046] 1. Lower data and computing power requirements: Different from most current Vision-Language Navigation (VLN) research that relies on a large amount of data and supervised learning, the zero-shot continuous vision-language navigation method provided by the present invention does not rely on a large amount of training data. It can be trained with a limited set of visible data categories and effectively generalize to unseen data categories for navigation tasks. This significantly reduces the data requirements and consumption of computing resources, and has higher application flexibility and economic benefits.

[0047] 2. No need for large models for navigation planning: Different from existing zero-shot VLN methods, the method of the present invention does not rely on large models (such as large language models or vision-language models). Instead, by combining technologies such as vision semantics, map integration, and cross-modal attention mechanisms, it extracts navigation information from limited environmental knowledge. This not only avoids the high computational costs brought by large models but also enables the method to better handle navigation challenges for unseen data categories with less training data.

[0048] 3. Excellent zero-shot navigation performance: By combining direction, object, and scene cues to learn relational knowledge in navigation and using a top-down egocentric semantic map to indicate direction, object representations and scene representations are obtained by decoupling environmental knowledge. Since no representational knowledge of specific data categories is learned, the navigation agent can effectively generalize navigation on unseen data categories, achieving zero-shot navigation. Through detailed experiments and analysis under the VLN-CE setting, it is verified that the technical solution of the present invention has excellent zero-shot navigation performance, while surpassing the performance of those large model-based methods, and also indicating that the large model-free architecture of the technical solution of the present invention has high navigation efficiency; compared with large model methods, this method can achieve or exceed their performance with lower computational overhead, demonstrating excellent zero-shot navigation ability.

[0049] 4. Adapt to dynamic and ever-changing real-world environments: The zero-shot navigation method of the present invention is closer to navigation tasks in real-world environments because unseen object categories are often encountered in reality. The present invention can effectively navigate objects of unseen categories without relying on large-scale training data, promoting the wider application of the VLN task in the real world.

[0050] 5. Low hardware requirements and easy to integrate and apply: Since the core method of the present invention has low requirements for hardware, it can be easily integrated into existing intelligent devices, such as driverless cars, intelligent service robots, etc. Its modular structure enables this technology to be seamlessly embedded into various automation systems, enhancing the navigation and operation capabilities of embodied robots and expanding the application scenarios of this technology.

[0051] 6. Improve navigation efficiency: Compared with traditional methods, the method of the present invention can achieve efficient navigation planning without relying on large computational models and can process more types of environmental data, enabling the navigation agent to complete tasks more accurately and efficiently. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a schematic flowchart of the method of the present invention.

[0053] Figure 2 is a schematic diagram of the overall framework structure of the method of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0054] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0055] The technical solution of the present invention has been described in detail through the accompanying drawings and specific embodiments, aiming to show its basic concept and operation process. The schematic diagrams provided by the accompanying drawings are only for auxiliary understanding, and do not represent the actual product size or limit the scope of the patent. Without conflict, the features in the embodiments can be combined with each other to explore more implementation possibilities. Those skilled in the art should understand that in order to more clearly convey the concept of the present invention, some components may be omitted, enlarged or reduced in the accompanying drawings, and the well-known structures and their descriptions may be omitted.

[0056] Please refer in detail to Figure 1 and Figure 2 in the accompanying drawings. This embodiment relates to a zero-shot continuous vision-language navigation method based on visual semantics and map integration. The detailed steps are as follows:

[0057] S1. Receive the natural language instruction of the current navigation task, and use the text encoder to encode and extract features from it to obtain the instruction features. The instruction in this step is given before the start of each navigation. It can be either detailed or rough. Especially when it is rough, it is more difficult for the navigation agent to navigate, and the navigation agent needs to have better reasoning ability. In this embodiment, there is no limitation on the detailed degree of the instruction description; the text encoder used in this embodiment is the BERT model, which captures the semantic information of the instruction through its bidirectional encoding ability.

[0058] S2. Capture the RGB view of the current location, and use the environmental knowledge decoupling module to decouple the environmental knowledge in the RGB view to obtain the environmental decoupled knowledge. The RGB view in this step needs to be actively acquired, and it mainly depends on the current environmental location. The RGB views obtained at different locations are different. The specific process of environmental knowledge decoupling is as follows:

[0059] 201) Use SAM

[48] to perform zero-shot segmentation on the obtained RGB view with rich environmental information to obtain the corresponding object semantic classes, and subsequent operations will be based on this semantic segmentation map;

[0060] 202) Decouple the scene information in the RGB view. Since the overall layout information in the scene is composed of the layout information of each object, therefore, for the layout information of object category i, it can be expressed as S i = [C x , C y , A r , where C x and C y represent its centroid coordinates, and A rRepresents the proportion of the area of the object relative to the current view scene. It should be noted that when multiple objects of the same object category appear in the current scene, directly take the one with the largest area proportion, because objects with larger areas are often more visible and contribute more to the layout information. Finally, all S i are stacked to form a scene decoupling matrix, which stores the key navigation information in the current RGB view scene;

[0061] 203) Decouple the object information in the RGB view. For object category i, it can be represented as O i = [V, K, S], where V represents the visibility of the object. If the object is observable in the current view, its value is 1, otherwise it is 0; Sequentially extract some key objects from the instructions that can help the navigation agent understand the environment, which reflect the navigation path of the navigation agent; Use K to represent whether the current object is a key object. If so, its value is 1, otherwise it is 0; S represents the weighted average of the cosine similarity between the semantic embedding of the current object and the semantic embeddings of the key objects. Among them, the weights of the key objects that appear in the already executed instruction part are 0. As the instruction execution progresses, the weights of the subsequent key objects gradually increase, and the sum of all weights is 1; Finally, all O i are stacked to form an object decoupling matrix; It can be seen that the decoupled representation of the object is actually independent of the specific object category, and it does not contain specific representation data of a certain category (such as: specific visual data representing objects of a certain category). Because it uses relative representation, even if a new unseen category appears, the navigation agent will not feel unfamiliar;

[0062] 204) Concatenate the scene decoupling matrix and the object decoupling matrix row by row to form a view observation decoupling matrix that contains rich semantic information and is independent of the specific data category; Input this matrix into the variational autoencoder, and use its ability to capture the potential structure of the input data and the associations between features, and then extract environmental decoupling knowledge in the latent space. This way of knowledge representation more effectively reveals the spatial semantic connections between the environmental layout and objects.

[0063] S3. Capture the Depth view of the current location, combine the instruction features in S1 and the RGB view in S2, and obtain an egocentric semantic map through a cross-modal attention network. The Depth view in this step needs to be actively obtained, which mainly depends on the current environmental location. The Depth views obtained at different locations are different, and it corresponds one-to-one with the RGB view Figure 1 This embodiment makes the following explanations on the specific process of obtaining an egocentric semantic map through a cross-modal attention network:

[0064] 301) For the obtained Depth view of the current position, project its ground onto an egocentric grid map φ t , which represents the occupancy of the current space, where t represents the current time step; for the ground projection operation, first perform inverse projection through the camera intrinsics to convert the depth pixels into point cloud data in three-dimensional space, and then project the point cloud data onto a two-dimensional plane parallel to the ground; further, use a truncated ResNet18 network to extract features from this grid map φ t to generate a grid map feature representation For the input instruction, obtain the encoded instruction feature Λ through the BERT model;

[0065] 302) During each navigation task, the instruction is pre-given and remains unchanged, but the egocentric map is constantly changing according to the position of the navigation agent. Therefore, here use the grid map feature as the query Q, the instruction feature Λ as the key K and value V, and obtain the attention representation through the scaled dot-product attention mechanism, and the calculation method is:

[0066]

[0067] where W q , W k , W v are all learnable square matrices with dimension d, is the attention representation on the grid map φ t ;

[0068] 303) Natural language instructions often imply spatial semantic descriptions. Here, further calculate the semantic proximity degree, i.e., cosine similarity, between the text instruction feature and the map feature and combine this metric with the original attention score. The navigation agent can more precisely identify the map area most relevant to the instruction semantics and promote the navigation agent to move towards the most relevant area, as follows:

[0069]

[0070] where, W R and W Λ are learnable parameter matrices. When calculating , the numerator uses matrix multiplication, the denominator uses element-wise multiplication, and ‖·‖ represents the L2 norm; since the value of the cosine similarity is between -1 and 1, it is necessary to scale it to between 0 and 1 to obtain to ensure the same value range as ; finally, combine it with the original attention score for weighted calculation, where, represents the integrated attention representation on the grid map φ after correction t where λ1 and λ2 represent the weights used to balance the two attention scores. Both of them are greater than 0 and their sum is 1. For the convenience of subsequent description, they are called the "joint balance factor".

[0071] 304) Semantic information is inferred and complemented by using two Unet networks based on the ResNet18 architecture (i.e., Res-Unet), and further semantic information outside the observation area of the navigation agent is obtained as follows:

[0072]

[0073] Among them, is the ground projection semantic segmentation of the RGB view. To better handle unseen classes, the SAM model

[48] is used here for zero-shot semantic segmentation; thus, the inference and generation of the egocentric semantic map are completed. It not only includes the top-down egocentric semantic map in the current field of view of the navigation agent but also supplements the semantic map of the unknown area that cannot be observed outside its field of view, making full preparations for the subsequent steps.

[0074] S4. Combine the instruction features in S1 and the egocentric semantic map obtained in S3, and generate a series of navigable waypoints leading to the target according to the waypoint generation network. Specifically, first use the truncated ResNet18 to encode the egocentric semantic map to obtain its map feature representation Then, according to Equations (9) and (10), and replace the in them with to further obtain the waypoint attention representation Finally, by inputting the waypoint attention representation and the initial waypoint into a waypoint prediction network with the Res-Unet architecture, all the currently navigable waypoints leading to the target are obtained.

[0075] S5. Combine the environmental decoupling knowledge obtained in S2 and the series of navigable waypoints obtained in S4 to predict the navigation actions that the navigation agent should execute. Specifically, first send all the predicted waypoints and the previously learned environmental decoupling knowledge into the planning and decision-making module. In this module, first select the navigable waypoint closest to the position of the navigation agent as the sub-goal, then combine the environmental decoupling knowledge, and use the improved DD-PPO strategy

[50] pre-trained on the PointNav task

[49] to complete the action prediction.

[0076] S6. Repeat the above steps until the navigation agent determines that the navigation task is completed or the maximum number of movement steps is reached. It should be noted that the navigation agent uses an action set including four actions, namely moving forward by 0.25 m, turning left by 15°, turning right by 15°, and stopping; if a stop decision is made within 3 m of the target position and the navigation agent completes a navigation within a fixed time budget of 500 steps, then this navigation is considered successful.

[0077] This embodiment executes the navigation task in a zero-shot manner, that is, the navigation agent is trained in a limited visible data category (including object category and environment category), but in the test navigation process, unseen data categories appear. The navigation agent needs to complete the generalization navigation on the unseen data categories with the help of the semantic embeddings of all provided data categories.

[0078] Preferably, in order to evaluate the zero-shot navigation performance of the navigation agent, this embodiment carefully examines the data in the train split and the validation unseen (val-unseen) split in the R2R-CE dataset, and enhances the data categories unseen by the navigation agent during training in the val-unseen split by removing the data categories in the val-unseen split contained in the train split. Finally, this embodiment uses the zero-shot performance on the val-unseen split for comparison with other methods. The experimental process mainly focuses on the following 5 performance evaluation metrics:

[0079] 1) TL: Represents the trajectory length of the agent's navigation path (in meters);

[0080] 2) NE: Represents the average distance between the agent's final position and the target (in meters);

[0081] 3) SR: Represents the ratio of the agent stopping within 3 m of the target range (as long as the agent is within 3 m of the target range, the navigation is considered successful);

[0082] 4) OSR: Represents the ratio of the tasks in which the agent successfully reaches the target position with the assistance of "Oracle" to the total tasks;

[0083] 5) SPL: The path weighted success rate that measures both the accuracy and efficiency of navigation.

[0084] For all the results in the following table, the bold font represents the best value of the current metric, "↓" means the lower the better, "↑" means the higher the better, "TL" and "NE" are in meters, and "SR", "SPL", and "OSR" are reported as percentage values.

[0085] In an embodiment, a GPU server equipped with multiple GetForce RTX 3090 graphics cards is used for parallel computing to improve data processing efficiency. To optimize the performance of the navigation agent in visual and language navigation tasks, the Adam optimizer is selected and a learning rate of 0.0002 is set for training. In addition, the navigation agent can access ego-centered RGB-D view observations with a 90-degree horizontal field of view.

[0086] The performance of this embodiment is compared with that of the current very excellent zero-shot learning-based navigation agent, and the results are shown in Table 1. As can be seen from the table, this navigation agent achieves the best performance under the zero-shot setting. Compared with A 2 Nav

[47] , which is a cutting-edge method for zero-shot navigation in VLN-CE and was the best-performing at that time, this navigation agent still greatly exceeds it with an absolute improvement of 5.2% (a relative improvement of 23.0%) in SR and an absolute improvement of 4.1% (a relative improvement of 36.9%) in SPL. For comparison methods such as CLIP-Nav

[42] and Seq CLIP-Nav

[42] , the performance of this navigation agent is also several times that of theirs. And it is not difficult to find that other methods all use large models (such as large language models or vision-language models) for navigation planning, while this navigation agent mainly relies on its own experience, indicating that this navigation agent has an advantage in experience generalization. And precisely because it does not use a large model for navigation planning, this navigation agent has better navigation efficiency due to its lighter weight.

[0087] This embodiment is also compared with the very excellent supervised learning-based navigation agent in terms of performance, and the results are shown in Table 1. It can be found that this navigation agent can exceed Seq2Seq

[21] with an absolute improvement of 2.8% (a relative improvement of 11.2%) in SR, and on the SR of SASRA

[25] , this navigation agent also has an absolute improvement of 3.8% (relative to 15.8%). This navigation agent surpasses all the supervised learning methods in Table 1 in terms of NE. Especially for NaVid

[52] , a pioneering video-based navigation method, this navigation agent can also defeat its NE, indicating that this navigation agent is closer to the target point at the end of navigation, reducing the navigation error.

[0088] In addition to the above comparison, ablation experiments were also conducted on the algorithm proposed in this embodiment. For specific results, please refer to Tables 2 and 3. As can be seen from Table 2, different combinations of λ1 and λ2 have a significant impact on the performance of the model. When either λ1 or λ2 is 0, that is, either only the original attention score or only the cosine similarity score is taken, the performance of the navigation agent will be greatly reduced; when both λ1 and λ2 are set to 0.5, the navigation agent achieves the best performance, indicating that both the original attention score and the cosine similarity score make important contributions to the navigation agent, and the comprehensive attention representation under this combination of values can obtain the maximum benefit.

[0089] Table 3 further quantitatively describes the contribution of environment decoupling knowledge to zero-shot navigation. From the results, it can be found that when zero-shot navigation is performed without using environment decoupling knowledge, the performance of the navigation agent is greatly reduced, mainly because the navigation agent only learns knowledge of specific data categories and does not learn the knowledge of the relationships between categories, resulting in the inability to generalize the learned experience to unseen data.

[0090] Preferably, the embodiment of the present application also provides a specific implementation manner of an electronic device capable of implementing all steps in the above-mentioned zero-shot continuous visual language navigation method based on visual semantics and map integration. The electronic device specifically includes the following content:

[0091] A processor, a memory, a communication interface, and a bus;

[0092] Among them, the processor, the memory, and the communication interface complete mutual communication through the bus; the communication interface is used to implement information transmission between related devices such as server-side devices, metering devices, and client-side devices.

[0093] The processor is used to call the computer program in the memory, and when the processor executes the computer program, it implements all steps in the above-mentioned zero-shot continuous visual language navigation method based on visual semantics and map integration.

[0094] Table 1. Performance comparison between this embodiment and other excellent navigation agents based on supervised learning and zero-shot learning on the val-unseen dataset of R2R-CE. The third column indicates whether the navigation agent uses a large model (such as a large language model or a vision-language model) for navigation planning.

[0095]

[0096] Table 2. Ablation study: The influence of different combinations of joint balance factor values on the performance of the navigation agent

[0097] Combined balance factor OSR↑ SR↑ SPL↑ <![CDATA[λ1 = 0.00, λ2 = 1.00]]> 28.6 16.5 9.2 <![CDATA[λ1 = 0.25, λ2 = 0.75]]> 33.3 16.7 12.9 <![CDATA[λ2 = 0.50, λ2 = 0.50]]> 33.3 27.8 15.2 <![CDATA[λ1 = 0.75, λ2 = 0.25]]> 33.2 18.8 12.6 <![CDATA[λ1 = 1.00, λ2 = 0.00]]> 16.7 11.1 11.1

[0098] Table 3. Ablation study: Impact of environment decoupled knowledge on the performance of navigation agents

[0099] Model TL↓ NE↓ OSR↑ SR↑ SPL↑ This navigation agent without using environmental decoupling knowledge 6.49 4.38 16.7 5.6 5.6 This navigation agent using environmental decoupling knowledge 11.60 4.10 33.3 27.8 15.2

[0100] Preferably, an embodiment of the present application further provides a specific implementation manner of an electronic device capable of implementing all steps in the above - mentioned zero - shot continuous visual - language navigation method based on visual semantics and map integration. The electronic device specifically includes the following:

[0101] A processor, a memory, a communications interface, and a bus;

[0102] Among them, the processor, the memory, and the communications interface complete mutual communication through the bus; the communications interface is used to implement information transmission between related devices such as server - side devices, metering devices, and user - side devices. [[ID=**17]]

[0103] The processor is used to call the computer program in the memory. When the processor executes the computer program, it implements all steps in the above - mentioned policy reinforcement learning method based on fuzzy rules and decision trees.

[0104] Preferably, an embodiment of the present application further provides a computer - readable storage medium capable of implementing all steps in the above - mentioned zero - shot continuous visual - language navigation method based on visual semantics and map integration. A computer program is stored on the computer - readable storage medium, and when the computer program is executed by the processor, it implements all steps in the above - mentioned zero - shot continuous visual - language navigation method based on visual semantics and map integration.

[0105] The above describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired result. In certain implementations, multitasking and parallel processing are also possible or may be advantageous.

[0106] Although the present application provides method operation steps such as in the embodiments or flowcharts, more or fewer operation steps may be included based on routine or non-creative labor. The order of steps listed in the embodiments is only one way among the execution orders of numerous steps and does not represent the only execution order. When the actual device or client product is executed, it may be executed in the order of the method shown in the embodiments or the drawings or executed in parallel (such as in an environment of parallel processors or multi-threaded processing).

[0107] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) that contain computer-usable program code.

[0108] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured article including an instruction device, and the instruction device implements the functions specified in one process Figure 1 one process or multiple processes and / or boxes Figure 1 specified in one box or multiple boxes.

[0109] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one process Figure 1 one process or multiple processes and / or boxes Figure 1 specified in one box or multiple boxes.

[0110] It should be noted that the positional relationships and detailed descriptions shown in the drawings are intended to provide an exemplary illustration of the technical solution to facilitate understanding of the core concepts and operating principles of the invention. These examples are not intended to limit the protection scope of the patent, but are used as auxiliary means for explaining the technical solution. Under the framework of the present invention, those of ordinary skill in the art can, based on the existing technical knowledge and the guidance of the present invention, through logical analysis, reasoning, and experiments, make appropriate modifications, equivalent replacements, or innovations to the technical solution to adapt to different application scenarios and requirements. These changes and adjustments, as long as they do not deviate from the basic principles and spirit of the invention, should be regarded as within the protection scope defined by the claims.

[0111] The detailed description of this embodiment demonstrates multiple aspects of the technical solution, but they do not exclude other possible implementation forms. Without departing from the spirit of the invention and the scope protected by the claims, those of ordinary skill in the art can, based on the inspiration of the present invention, make various specific transformations, including but not limited to adjustments in aspects such as hardware configuration, software implementation, system architecture, and operation process. These transformations are aimed at optimizing the performance, efficiency, or reliability of the technical solution, or expanding its application scope, and all fall within the protection scope of the present invention.

[0112] In addition, the terms and expressions used in the present invention are not intended to restrictively define the technical solution, but rather to provide clarity and consistency in the description. In the description, the use of terms is intended to facilitate the understanding and description of the technical content, and should not be regarded as a limitation of the patent protection scope. Therefore, any reasonable interpretation and application of the terms, as long as they conform to the purpose of the invention and the essence of the claims, should be included in the protection scope of the present invention.

[0113] The present invention is not limited to the embodiments described above. The above description of the specific embodiments is intended to describe and illustrate the technical solution of the present invention. The above specific embodiments are merely illustrative and not restrictive. Without departing from the spirit of the present invention and the scope protected by the claims, those of ordinary skill in the art can, under the inspiration of the present invention, also make many forms of specific transformations, and all of these fall within the protection scope of the present invention.

[0114] References:

[0115] [1]Gu J,Stefani E,Wu Q,et al.Vision-and-language navigation:A survey of tasks,methods,and future directions[J].arXiv preprint arXiv:2203.12667,2022.

[0116] [2]Wu W,Chang T,Li X,et al.Vision-language navigation:a survey and taxonomy[J].Neural Computing and Applications,2024,36(7):3291-3316.

[0117] [3] Chen S, Guhur PL, Tapaswi M, et al. Learning from unlabeled 3 environments for vision-and-language navigation[C] / / European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 638-655.

[0118] [4] Tan H, Yu L, Bansal M. Learning to navigate unseen environments: Backtranslation with environmental dropout[J]. arXiv preprint arXiv:1904.04195, 2019.

[0119] [5] Li J, Tan H, Bansal M. Envedit: Environment editing for vision-and-language navigation[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022: 15407-15417.

[0120] [6] Li X, Wang Z, Yang J, et al. Kerm: Knowledge enhanced reasoning for vision-and-language navigation[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2023: 2583-2592.

[0121] [7] Zhu F, Lee V, Chang X, et al. Vision language navigation with knowledge-driven environmental dreamer[J]. 2023.

[0122] [8] Zhu F, Zhu Y, Chang X, et al. Vision-language navigation with self-supervised auxiliary reasoning tasks[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020:10012-10022.

[0123] [9] Ma C Y, Wu Z, AlRegib G, et al. The regretful agent: Heuristic-aided navigation through progress estimation[C] / / Proceedings of the IEEE / CVF conference on Computer Vision and Pattern Recognition. 2019:6732-6740.

[0124]

[10] Ma C Y, Lu J, Wu Z, et al. Self-monitoring navigation agent via auxiliary progress estimation[J]. arXiv preprint arXiv:1901.03035, 2019.

[0125]

[11] Wang X, Huang Q, Celikyilmaz A, et al. Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2019:6629-6638.

[0126]

[12] Lu J, Batra D, Parikh D, et al. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks[J]. Advances in neural information processing systems, 2019, 32.

[0127]

[13] Guhur P L, Tapaswi M, Chen S, et al. Airbert: In-domain pretraining for vision-and-language navigation[C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2021:1634-1643.

[0128]

[14] Hao W, Li C, Li X, et al. Towards learning a generic agent for vision-and-language navigation via pre-training[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020:13137-13146.

[0129] [(15)] Qiao Y, Qi Y, Hong Y, et al. Hop: History-and-order aware pre-training for vision-and-language navigation[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022:15418-15427.

[0130] Note: In the original text, the number in reference

[15] seems to be incorrect as it has an extra left parenthesis. It is corrected to [(15)] in the translation for better consistency. If this is not an error in the original, please let me know and I can adjust accordingly.

[16] Li X,Li C,Xia Q,et al.Robust navigation with language pretrainingand stochastic sampling[J].arXiv preprint arXiv:1909.02244,2019.

[0131]

[17] Magassouba A,Sugiura K,Kawai H.CrossMap transformer:A crossmodalmasked path transformer using double back-translation for vision-and-languagenavigation[J].IEEE Robotics and Automation Letters,2021,6(4):6258-6265.

[0132]

[18] Chen S,Guhur P L,Tapaswi M,et al.Think global,act local:Dual-scale graph transformer for vision-and-language navigation[C] / / Proceedings ofthe IEEE / CVF Conference on Computer Vision and Pattern Recognition.2022:16537-16547.

[0133]

[19] Zhou X,Liu W,Mu Y.Rethinking the spatial route prior in vision-and-language navigation[J].arXiv preprint arXiv:2110.05728,2021.

[0134]

[20] Wang Z,Li X,Yang J,et al.Gridmm:Grid memory map for vision-and-language navigation[C] / / Proceedings of the IEEE / CVF International Conferenceon Computer Vision.2023:15625-15636.

[0135]

[21] Krantz J, Wijmans E, Majumdar A, et al. Beyond the nav-graph: Vision-and-language navigation in continuous environments[C] / / Computer Vision–ECCV2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVIII 16. Springer International Publishing, 2020: 104-­120.

[0136]

[22] Hong Y, Wang Z, Wu Q, et al. Bridging the gap between learning in discrete and continuous environments for vision-and-language navigation[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022: 15439-­15449.

[0137]

[23] Georgakis G, Schmeckpeper K, Wanchoo K, et al. Cross-modal map learning for vision and language navigation[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022: 15460-­15470.

[0138]

[24] Wang T, Wu Z, Yao F, et al. Graph-Based Environment Representation for Vision-and-Language Navigation in Continuous Environments[C] / / ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP). IEEE, 2024: 8331 - 8335.

[0139]

[25] Irshad M Z, Mithun N C, Seymour Z, et al. Semantically-aware spatio-temporal reasoning agent for vision-and-language navigation in continuous environments[C] / / 2022 26th International Conference on Pattern Recognition(ICPR). IEEE, 2022: 4065 - 4071.

[0140]

[26] Wang H, Liang W, Van Gool L, et al. Dreamwalker: Mental planning for continuous vision-language navigation[C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2023: 10873 - 10883.

[0142]

[27] Krantz J, Gokaslan A, Batra D, et al. Waypoint models for instruction-guided navigation in continuous environments[C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2021: 15162 - 15171.

[0143]

[28] He Z,Wang L,Li S,et al.Mlanet:Multi-level attention network withsub-instruction for continuous vision-and-language navigation[J].arXivpreprint arXiv:2303.01396,2023.

[0144]

[29] Verma V K,Arora G,Mishra A,et al.Generalized zero-shot learningvia synthesized examples[C] / / Proceedings of the IEEE conference on computervision and pattern recognition.2018:4281-4289.

[0145]

[30] Naeem M F,Xian Y,Tombari F,et al.Learning graph embeddings forcompositional zero-shot learning[C] / / Proceedings of the IEEE / CVF Conferenceon Computer Vision and Pattern Recognition.2021:953-962.

[0146]

[31] Rahman S,Khan S,Porikli F.Zero-shot object detection:Learning tosimultaneously recognize and localize novel concepts[C] / / Asian Conference onComputer Vision.Cham:Springer International Publishing,2018:547-563.

[0147]

[32] Hayat N, Hayat M, Rahman S, et al. Synthesizing the unseen for zero-shot object detection[C] / / Proceedings of the Asian conference on computervision. 2020.

[0148]

[33] Rahman S, Khan S, Barnes N. Polarity loss for zero-shot objectdetection[J]. arXiv preprint arXiv:1811.08982, 2018.

[0149]

[34] Bucher M, Vu T H, Cord M, et al. Zero-shot semantic segmentation[J]. Advances in Neural Information Processing Systems, 2019, 32.

[0150]

[35] Kato N, Yamasaki T, Aizawa K. Zero-shot semantic segmentation viavariational mapping[C] / / Proceedings of the IEEE / CVF International Conferenceon Computer Vision Workshops. 2019:0-0.

[0151]

[36] Majumdar A, Aggarwal G, Devnani B, et al. Zson:Zero-shot object-goalnavigation using multimodal goal embeddings[J]. Advances in Neural InformationProcessing Systems, 2022, 35:32340-32352.

[0152]

[37] Li X, Zhang S, Lu Y, et al. Zero-Shot Object Goal Visual NavigationWith Class-Independent Relationship Network[J]. arXiv preprint arXiv:2310.09883, 2023.

[0153]

[38] Gadre S Y, Wortsman M, Ilharco G, et al. Cows on pasture: Baselinesand benchmarks for language-driven zero-shot object navigation[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition. 2023:23171-23181.

[0154]

[39] Zhao Q, Zhang L, He B, et al. Zero-shot object goal visual navigation[C] / / 2023 IEEE International Conference on Robotics and Automation(ICRA). IEEE, 2023:2025-2031.

[0155]

[40] Al-Halah Z, Ramakrishnan S K, Grauman K. Zero experience required: Plug&play modular transfer learning for semantic visual navigation[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition. 2022:17031-17041.

[0156]

[41] Sun X,Lau L,Zhi H,et al.Prioritized Semantic Learning for Zero-shot Instance Navigation[J].arXiv preprint arXiv:2403.11650,2024.

[0157]

[42] Dorbala V S,Sigurdsson G,Piramuthu R,et al.Clip-nav:Using clipfor zero-shot vision-and-language navigation[J].arXiv preprint arXiv:2211.16649,2022.

[0158]

[43] Huang J,Zhang H,Zhao M,et al.IVLMap:Instance-Aware VisualLanguage Grounding for Consumer Robot Navigation[J].arXiv preprint arXiv:2403.19336,2024.

[0159]

[44] Chen J,Lin B,Xu R,et al.MapGPT:Map-Guided Prompting with AdaptivePath Planning for Vision-and-Language Navigation[C] / / Proceedings of the 62ndAnnual Meeting of the Association for Computational Linguistics(Volume 1:LongPapers).2024:9796-9810.

[0160]

[45] Zhou G,Hong Y,Wu Q.Navgpt:Explicit reasoning in vision-and-language navigation with large language models[C] / / Proceedings of the AAAIConference on Artificial Intelligence.2024,38(7):7641-7649.

[0161]

[46] Li D, Chen W, Lin X. TINA: Think, Interaction, and Action Framework for Zero-Shot Vision Language Navigation[J]. arXiv preprint arXiv:2403.08833, 2024.

[0162]

[47] Chen P, Sun X, Zhi H, et al. $A^2$Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Models[J]. arXiv preprint arXiv:2308.07997, 2023.

[0163]

[48] Kirillov A, Mintun E, Ravi N, et al. Segment anything[C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2023:4015 - 4026.

[0164]

[49] Anderson P, Chang A, Chaplot D S, et al. On evaluation of embodied navigation agents[J]. arXiv preprint arXiv:1807.06757, 2018.

[0165]

[50] Wijmans E, Kadian A, Morcos A, et al. Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames[J]. arXiv preprint arXiv:1911.00357, 2019.

[0166]

[51] Raychaudhuri S,Wani S,Patel S,et al.Language-aligned waypoint(law)supervision for vision-and-language navigation in continuousenvironments[J].arXiv preprint arXiv:2109.15207,2021.

[0167]

[52] Zhang J,Wang K,Xu R,et al.NaVid:Video-based VLM Plans the NextStep for Vision-and-Language Navigation[J].arXiv preprint arXiv:2402.15852,2024.

Claims

1. A zero-shot continuous vision-language navigation method based on visual semantics and map integration, characterized in that It includes the following steps: S1. Receive the natural language instruction of the current navigation task, encode and extract features from it using a text encoder to obtain instruction features; S2. Capture the RGB view of the current location and decouple the environmental knowledge in this RGB view to obtain environmental decoupled knowledge; S3. Capture the Depth view of the current location, combine the instruction features in S1 and the RGB view in S2, and obtain an egocentric semantic map through a cross-modal attention network; S4. Combine the instruction features in S1 and the egocentric semantic map obtained in S3 to generate several navigable waypoints leading to the target; S5. Combine and use the environmental decoupled knowledge obtained in S2 and the navigable waypoints obtained in S4 to predict the navigation actions that the navigation agent should execute; S6. Repeat the above steps until the navigation agent determines that the navigation task is completed or the maximum number of movement steps is reached.

2. The zero-shot continuous vision-language navigation method based on visual semantics and map integration according to claim 1, wherein In step S1, the BERT model is used as the text encoder to extract features from the input natural language instruction, and semantic information is captured through its bidirectional encoding ability.

3. The zero-shot continuous vision-language navigation method based on visual semantics and map integration according to claim 1, wherein The specific workflow of using the environmental knowledge decoupling module to decouple environmental knowledge in step S2 is as follows: First, use the SAM model to perform zero-shot segmentation on the obtained RGB view with environmental information to obtain the corresponding object semantic classes; Then, decouple the scene information in the RGB view. For the layout information of object category i, denoted as S i =[C x , C y , A t , where C x and C y represent their centroid coordinates, and A r represents the area ratio of the object relative to the current RGB view scene; finally, stack all S i to form a scene decoupling matrix, which stores the navigation information in the current RGB view scene; Next, decouple the object information in the RGB view. For object category i, it is denoted as O i = [V, K, S], where V represents the visibility of the object. If the object can be observed in the current view, its value is 1; otherwise, it is 0. Sequentially extract several key objects from the instructions that can help the navigation agent understand the environment. They reflect the navigation path of the navigation agent. Use K to represent whether the current object is a key object. If so, its value is 1; otherwise, it is 0. S represents the weighted average of the cosine similarity between the semantic embedding of the current object and the semantic embeddings of the key objects. Among them, the weight of the key objects that appear in the already executed part of the instructions is 0, and as the execution of the instructions progresses, the weights of the subsequent key objects gradually increase. Finally, stack all the O i to form an object decoupling matrix; Finally, the scene decoupling matrix and the object decoupling matrix are concatenated row by row to form a view observation decoupling matrix independent of specific data categories; the view observation decoupling matrix is input into a variational autoencoder to capture the latent structure of the input data and the correlation between features, and environmental decoupled knowledge is extracted.

4. The zero-shot continuous vision-language navigation method based on visual semantics and map integration according to claim 1, wherein In step S3, the specific process of obtaining the egocentric semantic map through the cross-modal attention network is as follows: First, for the Depth view of the obtained current position, project its ground onto a grid map φ t , where the grid map represents the occupancy of the current space, and t represents the current time step; for the ground projection operation, first perform inverse projection through the camera intrinsics to convert the depth pixels into point cloud data in three-dimensional space, and then project the point cloud data onto a two-dimensional plane parallel to the ground; Subsequently, the truncated ResNet18 network is used to extract features from this grid map φ t to generate a feature representation of the grid map For the input instruction, the encoded instruction feature Λ is obtained through the BERT model; Next, the grid map feature is used as the query Q, the instruction feature Λ as the key K and value V, and the attention representation is obtained through the scaled dot-product attention mechanism, and the calculation method is as follows: Among them, W q , W k , W v are all learnable parameter square matrices with dimension d, is the attention representation on the grid map φ t ; Further, by calculating the semantic proximity, i.e., the cosine similarity, between the instruction feature Λ and the grid map feature and combining the cosine similarity with the original attention score, the navigation agent can more accurately identify the map area that is semantically most relevant to the instruction feature Λ, promoting the navigation agent to act towards the most relevant area, as follows: ​ Among them, W R and W Λ are learnable parameter matrices; when calculating , the numerator uses matrix multiplication, the denominator uses element-wise multiplication, and ‖·‖ represents the L2 norm; the value of the cosine similarity is between -1 and 1. Scale to between 0 and 1 to obtain to ensure that it has the same value range as ; finally, combine with the original attention score for weighted calculation, where represents the comprehensive attention representation on the grid map φ t . λ1 and λ2 represent the weights used to balance the two attention scores. They are both greater than 0 and their sum is 1; Finally, semantic information is inferred and completed through the use of two Unet networks based on the ResNet18 architecture, namely Res-Unet, and semantic information outside the observation area of the navigation agent is further obtained, as follows: Among them, is the ground projection semantic segmentation of the RGB view; thus, the inference and generation of the egocentric semantic map are completed. It not only includes the top-down egocentric semantic map in the current field of view of the navigation agent but also supplements the semantic map of the unknown areas that cannot be observed outside its field of view, making full preparations for the subsequent steps.

5. The zero-shot continuous vision-language navigation method based on visual semantics and map integration according to claim 1, wherein In step S4, the specific process of generating several navigable waypoints leading to the target according to the waypoint generation network is: First, use ResNet18 again to encode the egocentric semantic map to obtain the feature representation of the egocentric semantic map ​ Then, use the feature representation of the egocentric semantic map as the query Q, the instruction feature Λ as the key K and value V; obtain the waypoint attention representation through the scaled dot-product attention mechanism The calculation method is as follows: where W q , W k , W v are all learnable square parameter matrices of dimension d; Finally, by feeding the waypoint attention representation and the initial waypoint into a Res-Unet network, all currently navigable waypoints to the target are obtained.

6. The zero-shot continuous vision-language navigation method based on visual semantics and map integration according to claim 1, characterized in that The specific process of predicting the actions of the navigation agent in step S5 is: Send all the predicted waypoints and the previously learned environmental decoupled knowledge into the planning and decision-making module. In this planning and decision-making module, first select the navigable waypoint closest to the position of the navigation agent as the sub-goal, then combine the environmental decoupled knowledge, and use the DD-PPO policy pre-trained on the PointNav task to complete the action prediction.

7. The zero-shot continuous visual language navigation method based on visual semantics and map integration according to claim 1 is characterized in that: In step S6, the navigation agent uses an action set including four actions, namely moving forward 0.25m, turning left 15°, turning right 15°, and stopping; if a stop decision is made within 3m of the target position and the navigation agent completes a navigation within a fixed time budget of 500 steps, this navigation is considered successful.

8. The zero-shot continuous vision-language navigation method based on visual semantics and map integration according to claim 1, characterized in that The navigation agent is trained in a finite set of visible data categories. If an unseen data category appears during the test process, the navigation agent completes generalization navigation on the unseen data category by means of the semantic embeddings of all provided data categories; The data categories include object categories and environmental categories.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the zero-shot continuous vision-language navigation method based on visual semantics and map integration according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the zero-shot continuous vision-language navigation method based on visual semantics and map integration according to any one of claims 1 to 8.

Images