End-to-end visual language navigation method based on network occupation awareness

Abstract

The invention relates to the technical field of navigation, in particular to an end-to-end visual language navigation method and device based on occupancy network awareness and computer equipment, and the method comprises the following steps: receiving a language instruction, and obtaining a color image and a depth map collected by an RGB camera and a depth camera; performing feature extraction on the color image and the depth image through a ViT model to generate image features and depth features, and performing rotation and translation transformation on the depth features through a camera model to generate a blank 3D voxel space; mapping the image features to a blank 3D voxel space to generate 3D voxel features, and constructing the image features into a topological graph; inputting the 3D voxel features, the topological graph and the language information into a BERT model to obtain text features; and inputting the text features, the 3D voxel features and the topological graph into a cross-modal model, and predicting a navigation path. The method and the device are helpful for improving the accuracy of navigation decision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of navigation, in particular to an end-to-end vision-language navigation method based on occupancy network perception, a device and a computer equipment. BACKGROUND

[0002] Under the background of rapid development of global artificial intelligence (AI) and automation technology, embodied intelligence (Embodied AI) as a key application field is attracting more and more widespread attention. With the acceleration of this trend, the vision-and-language navigation (VLN) task has gradually become one of the core research directions in academia and industry.

[0003] As a complex task combining visual and language information, vision-and-language navigation (VLN) aims to guide an agent to navigate in a complex environment through natural language instructions, and has gradually become the focus of multi-modal understanding research. Although existing VLN methods have achieved good results on standard datasets, there are still some problems and challenges in the current vision-and-language navigation task: first, existing visual representation methods are insufficient in capturing three-dimensional spatial details and complex geometric structures; second, existing models have weak text understanding ability when facing ambiguous or unclear instructions; third, the high computational cost of existing methods in spatio-temporal information processing makes it difficult to deploy, limiting the feasibility of the model in real-time navigation tasks. In addition, the existing model also has insufficient generalization ability when dealing with unseen environments and scenes. These problems will lead to poor accuracy of the navigation path. SUMMARY

[0004] Therefore, it is necessary to provide an end-to-end vision-language navigation method based on occupancy network perception to improve the accuracy of the navigation path.

[0005] In a first aspect, the present application provides an end-to-end vision-language navigation method based on occupancy network perception, which comprises: receiving a language instruction and obtaining a color image and a depth map collected by an RGB camera and a depth camera, the language instruction carrying language information; extracting features of the color image and the depth map through a ViT model to generate image features and depth features, and performing rotation and translation transformation on the depth features through a camera model to generate a blank 3D voxel space; mapping the image features to the blank 3D voxel space to generate 3D voxel features, and constructing the image features into a topological graph; inputting the 3D voxel features, the topological graph and the language information into a BERT model to obtain text features; inputting the text features, the 3D voxel features and the topology graph into a cross-modal model to predict a navigation path.

[0006] In one of the embodiments, the mapping the image features onto the blank 3D voxel space, generating 3D voxel features, and constructing the image features into a topology graph comprises: mapping the image features onto the blank 3D voxel space, generating current-time 3D voxel features, and modeling the current-time 3D voxel features as a spatial dimension; obtaining image features of previous n time points, and generating a topology graph as a time dimension modeling according to the image features of the previous n time points.

[0007] In one of the embodiments, after the obtaining image features of previous n time points, and generating a topology graph as a time dimension modeling according to the image features of the previous n time points, the method further comprises: defining a plurality of different time scale history information windows for the topology graph through the multi-scale time self-attention mechanism, wherein each history information window contains history information within a corresponding time range; calculating self-attention of each time scale through the history information window; fusing the calculated self-attention results of each time scale to obtain comprehensive history information, so as to effectively capture and adjust the dependence degree on recent or long-term history information.

[0008] In one of the embodiments, the mapping the image features onto the blank 3D voxel space, generating current-time 3D voxel features, and modeling the current-time 3D voxel features as a spatial dimension comprises: mapping the image features onto the blank 3D voxel space, generating a plurality of voxel units through a three-dimensional occupancy network, wherein the three-dimensional occupancy network is used to divide the occupancy state of a three-dimensional space, and each voxel unit represents a corresponding region in the three-dimensional space and corresponds to different occupancy states respectively; generating current-time 3D voxel features according to the plurality of voxel units, and modeling the current-time 3D voxel features as a spatial dimension to obtain spatial information.

[0009] In one of the embodiments, after the generating current-time 3D voxel features according to the plurality of voxel units, and modeling the current-time 3D voxel features as a spatial dimension to obtain spatial information, the method further comprises: dividing the spatial information into a plurality of different resolution levels through a multi-scale spatial self-attention mechanism, and each level represents a different spatial scale; The spatial self-attention of each scale layer is calculated, and the calculated spatial self-attention results of the respective scale layers are fused to simultaneously capture local and global spatial information.

[0010] In one of the embodiments, the feature extraction of the color image and the depth map by the ViT model, the generation of image features and depth features, and the rotation and translation transformation of the depth features by the camera model to generate a blank 3D voxel space include: The feature extraction of the color image and the depth map by the ViT model to generate image features and depth features; The rotation and translation transformation of the depth features by the camera model to map the depth map into a pseudo point cloud; The pseudo point cloud is voxelized to generate a blank 3D voxel space.

[0011] In one of the embodiments, the input of the 3D voxel features, the topological graph, and the language information into the BERT model to obtain text features includes: According to the 3D voxel features and the topological graph, environmental perception information is obtained, and according to the environmental perception information, environmental features are obtained by a visual perception model; According to the language information, language features are extracted by a pre-trained language model; The interaction between the language features and the environmental features is calculated, and the environmental feature representation is combined with the language features to form a unified joint representation, which changes over time with the change of the environmental state; Each language feature and environmental feature in the joint representation is aligned, and the aligned language features and environmental features are input into the BERT model to obtain text features.

[0012] In one of the embodiments, the input of the text features, the 3D voxel features, and the topological graph into the cross-modal model to predict a navigation path includes: According to the 3D voxel features and the topological graph, visual features are obtained; The text features and the visual features are input into the cross-modal model for feature fusion through an end-to-end architecture; According to the fused features, the optimal navigation path is predicted by a global-local attention mechanism.

[0013] In a second aspect, the present application also provides an end-to-end visual language navigation device based on an occupancy network perception. The device includes: An instruction receiving module is configured to receive a language instruction and obtain a color image and a depth map collected by an RGB camera and a depth camera, the language instruction carrying language information; A voxel space generating module is configured to perform feature extraction on the color image and the depth map by using a ViT model to generate image features and depth features, and perform rotation and translation transformation on the depth features by using a camera model to generate a blank 3D voxel space; A voxel and topology map generating module is configured to map the image features onto the blank 3D voxel space to generate 3D voxel features, and construct the image features into a topology map; A text feature obtaining module is configured to input the 3D voxel features, the topology map and the language information into a BERT model to obtain text features; A navigation path predicting module is configured to input the text features, the 3D voxel features and the topology map into a cross-modal model to predict a navigation path.

[0014] In a third aspect, the present application further provides a computer device. The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program: An instruction receiving module is configured to receive a language instruction and obtain a color image and a depth map collected by an RGB camera and a depth camera, the language instruction carrying language information; A voxel space generating module is configured to perform feature extraction on the color image and the depth map by using a ViT model to generate image features and depth features, and perform rotation and translation transformation on the depth features by using a camera model to generate a blank 3D voxel space; A voxel and topology map generating module is configured to map the image features onto the blank 3D voxel space to generate 3D voxel features, and construct the image features into a topology map; A text feature obtaining module is configured to input the 3D voxel features, the topology map and the language information into a BERT model to obtain text features; A navigation path predicting module is configured to input the text features, the 3D voxel features and the topology map into a cross-modal model to predict a navigation path.

[0015] The end-to-end visual language navigation method, device and computer equipment based on the occupancy network perception map image features to a blank 3D voxel space, generate 3D voxel features, and construct the image features into a topological graph, so that the three-dimensional perception ability is fused in the visual representation method, and the agent can more accurately capture the geometric details and spatial relationships of the environment in a complex scene; the 3D voxel features, the topological graph and the language information are input into the BERT model to obtain text features, which can assist the BERT model in extracting key information in the language instruction through the environmental information, not only improving the robustness of text information extraction, but also significantly improving the overall performance in the navigation task; the text features, the 3D voxel features and the topological graph are input into the cross-modal model to predict the navigation path, which can enhance the understanding of the fine-grained relationship between the instruction and the environmental information of the agent by combining the text features, the 3D voxel features and the topological graph, and improve the accuracy of navigation decision. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 A flowchart of an embodiment of the end-to-end visual language navigation method based on the occupancy network perception; Figure 2 A schematic diagram of the overall task flow of the end-to-end visual language navigation method based on the occupancy network perception; Figure 3 A schematic diagram of the spatio-temporal constraint modeling process; Figure 4 A schematic diagram of the three-dimensional occupancy network processing process; Figure 5 A schematic diagram of the environment-guided text understanding process; Figure 6 A structural block diagram of an embodiment of the end-to-end visual language navigation device based on the occupancy network perception; Figure 7 An internal structure diagram of the computer equipment in an embodiment. DETAILED DESCRIPTION

[0017] The embodiment of the present application provides an end-to-end visual language navigation method, device and computer equipment based on occupancy network perception.

[0018] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms, and should not be interpreted as being limited to the embodiments described herein, but rather, these embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes, and are not intended to limit the scope of protection of the present application.

[0019] In the description of the embodiments disclosed in the present application, the term "comprising" and its similar terms shall be understood as open inclusion, i.e. "including but not limited to". The term "based on" shall be understood as "at least partially based on". The term "one embodiment" or "the embodiment" shall be understood as "at least one embodiment". The terms "first", "second", and the like can refer to different or same objects. Other explicit and implicit definitions can also be included below.

[0020] For ease of understanding, the specific process of the embodiments of the present application is described below. Please refer to Figure 1 An embodiment of the end-to-end visual language navigation method based on the occupancy network perception in the embodiments of the present application includes: S100, accepting language instructions, and acquiring color images and depth maps collected by RGB cameras and depth cameras.

[0021] Specifically, in the VLN task, the traditional two-dimensional perspective representation (such as bird's eye view, BEV) has limitations in capturing the spatial details and complex geometry of the environment, especially in navigation tasks that need to deal with multi-story building structures or involve vertical direction reasoning. The two-dimensional view is difficult to fully describe the height changes in the scene, the three-dimensional spatial layout of objects and their mutual relationship. Therefore, the present application adopts an efficient three-dimensional spatial representation method based on three-dimensional occupancy network (3D Occupancy Networks, OccNet) to enhance the spatial reasoning ability of the agent in a complex scene. The three-dimensional occupancy network can more comprehensively model the geometry of the environment by constructing a three-dimensional voxel representation of the environment, capturing the spatial relationship of walls, furniture and other objects. This not only allows detailed restoration of complex spatial layout, but also processes the space-time changes in the environment by dynamically updating the occupancy map, greatly improving the adaptability of the agent in dynamic scenes. In addition, the introduction of three-dimensional occupancy network also optimizes the computing efficiency. Through efficient voxel sampling and compression technology, the computing overhead is reduced while maintaining high-precision spatial reasoning.

[0022] The core components of the visual-language navigation (VLN) framework include an agent (Agent), an environment (Environment), and an "oracle" (usually a human) that provides guidance. The agent in the framework communicates with the oracle through natural language. The agent can request help according to the instructions, and the oracle will provide feedback and guidance according to the state of the environment and the needs of the agent. The agent accepts language instructions sent by the "oracle", wherein the language instructions carry language information, and at the same time, color images and depth maps of RGB cameras and depth cameras are collected, so as to realize semantic understanding and spatial geometry understanding of the environment.

[0023] S200, feature extraction is performed on the color image and the depth map through the ViT model to generate image features and depth features, and a blank 3D voxel space is generated by performing rotation and translation transformation on the depth features through a camera model.

[0024] Specifically, a ViT (Vision Transformer) model is used to extract features from the color image. The ViT model divides the image into multiple small patches, and then uses a self-attention mechanism to capture the relationships between different parts of the image, thereby generating image feature representations. For the depth map, a similar method can be used, such as using a pre-trained DenseNet network to downsample the depth map, and then encoding it into a feature sequence suitable for ViT. In addition, a hybrid architecture combining convolutional neural networks (CNN) and Transformers can also be used to further improve the feature representation ability of the depth map through multi-scale feature fusion. The depth features are rotated and translated through a camera model to generate a blank 3D voxel space. The camera model usually includes an intrinsic matrix and an extrinsic matrix, which describe the camera's perspective and position.

[0025] S300, map the image features to the blank 3D voxel space to generate 3D voxel features, and construct the image features into a topological graph.

[0026] Specifically, the image features are mapped to the blank 3D voxel space, and the three-dimensional space is divided into a regular voxel grid, with each voxel occupying a fixed spatial unit. A voxel is the smallest unit in three-dimensional space, similar to a pixel in two-dimensional space. It is a regular small cube used to represent discretized information in three-dimensional space, such as shape, color, density, etc.

[0027] The extracted image features are further converted into a topological structure, and the topological features are integrated into a topological graph. The topological graph is an effective graph structure representation method that can capture the global structure of the environment and help the agent effectively plan a global path in a long navigation task. Each node represents a specific position in the environment that the agent has passed through, and the edge represents the passability or navigation path between different positions.

[0028] Let the exploration path of the agent in the environment be represented as a graph , where represents the set of nodes visited by the agent, represents the edges connecting these nodes.

[0029] In the topological graph, the state of a node can be represented by its spatial position . The agent needs to plan a path from a given starting point to a target point An optimal path is found between the start and goal points. Path planning can be done through shortest path algorithms such as Dijkstra or A* algorithm, the goal is to find the shortest path that satisfies the following conditions: where, is the set of all feasible paths from the start point to the goal point, is the weight of the edge , usually representing the cost (such as distance or time) of the agent navigating between two nodes. Through the topological graph structure, the agent can remember the explored path nodes, thereby avoiding redundant exploration, significantly improving the efficiency of global path planning.

[0030] S400, input 3D voxel features, topological graph and language information into BERT model to obtain text features.

[0031] Specifically, in order to solve the problem of insufficient text understanding ability of current VLN technology, a text understanding method of mutual constraint between environment and text is proposed, which aims to enhance the parsing ability of text model to inaccurate or too long / short instructions. Traditional VLN methods mainly rely on text to understand the environment, but when the instruction information is not clear, the text model (BERT model) is difficult to accurately capture the key navigation information. Therefore, the strategy of environment guidance will be adopted, and the environment features will be used to assist the text model to extract the key information in the instruction. Specifically, through 3D voxel features and topological graph, the visual features in the environment are obtained. First, use language information to preliminarily analyze the environment, when encountering language information that is difficult to analyze, through the visual features in the environment such as path planning, key objects, etc., provide additional context clues, so as to help the text model more accurately extract useful information.

[0032] In this embodiment, the alignment accuracy of text and environment is significantly improved through the text-environment bidirectional constraint mechanism, especially when facing complex and ambiguous language descriptions. Through the reverse constraint of environment on text understanding, the model not only improves the robustness of language information extraction, but also significantly improves the overall performance in navigation tasks.

[0033] S500, input text features, 3D voxel features and topological graph into cross-modal model to predict navigation path.

[0034] Specifically, in VLN tasks, traditional navigation systems usually adopt a hierarchical architecture, which separates visual feature extraction, language understanding, and navigation decision-making into independent modules. Although these modules can be optimized individually, the modular design often leads to poor global information flow, insufficient cross-modal reasoning, and poor collaboration between modules. To solve this problem, an end-to-end framework-based navigation system is adopted, which integrates vision, language, and navigation strategy into a unified model. Through end-to-end training, the model can directly generate decisions from input multi-modal information, improving the overall performance and generalization ability of navigation.

[0035] In the end-to-end framework, the visual, language, and action decision modules are no longer separated but integrated into the same network. Let the visual input be an image sequence , the language input be a natural language instruction , and the agent needs to generate the current time action decision from these two inputs . In the end-to-end architecture, the entire cross-modal model can be represented as a continuous mapping function , whose goal is to minimize the length of the navigation path or the cumulative loss during navigation: The overall loss function of the model can be represented as: where represents the error between the agent's action decision and the true label action, and represent the loss terms of visual features and language features, respectively, and are the weights of the loss terms.

[0036] In the end-to-end framework, visual and language information need to be jointly modeled to ensure that the agent can make reasonable decisions based on language instructions in complex environments. Given the current visual data and language data T, the visual features are first extracted by a convolutional neural network (CNN) , and the text features are extracted by a language model such as Transformer or LSTM .

[0037] Next, the visual features and language features need to be fused through a multi-modal fusion mechanism. Common methods include attention-based or bilinear pooling-based methods. For example, when using a self-attention mechanism, the multi-modal fusion representation can be defined as: wherein, and is a weight matrix for projection, is a scaling factor. Through the self-attention mechanism, the model can dynamically adjust the weight of visual information and language instructions, ensuring that the current environment and task goals are considered simultaneously when generating actions.

[0038] In this embodiment, an end-to-end learning-based navigation system is adopted, which processes visual and language information in a unified framework. This framework enables multi-modal information to interact and fuse directly by sharing the embedding space of visual and language features, thereby improving the coherence and efficiency of the reasoning process. In addition, the end-to-end architecture simplifies the interface design between the feature extraction and task planning modules, avoiding information loss and delay problems in traditional methods. The cross-modal model based on the end-to-end framework not only achieves significant improvement in navigation accuracy, but also effectively reduces the reasoning time, with stronger real-time performance and robustness.

[0039] The overall process of the above-mentioned end-to-end visual language navigation method based on occupancy network perception is shown in Figure 2 As shown in the figure, first, the input depth map and RGB image are respectively processed by the ViT model to generate depth features and image features. Then, the depth features are rotated and translated by the camera model to generate a blank 3D voxel space. In this 3D voxel space, the image features are mapped to the blank voxels, thereby constructing 3D voxel features. At the same time, the image features are constructed into a topological graph to represent the temporal association of environmental information. After the text input, topological graph, and three-dimensional voxel features are aligned by the embedding layer, they are input into the text encoder module (BERT model) to obtain text features. Finally, the text features, topological graph, and three-dimensional voxel features are input into the fusion module (Cross-Modal Transformer) for cross-modal fusion to generate the final navigation path prediction.

[0040] In one embodiment, mapping the image features onto the blank 3D voxel space to generate 3D voxel features and constructing the image features into a topological graph include: mapping the image features onto the blank 3D voxel space to generate current time 3D voxel features, modeling the current time 3D voxel features as spatial dimensions; obtaining image features of the previous n time, and generating a topological graph as a modeling of time dimensions according to the image features of the previous n time.

[0041] Specifically, after the agent receives the language instruction, it acquires the color image captured by the camera in real time and extracts the image features, and then maps the image features to the blank 3D voxel space to generate the 3D voxel features at the current time, and models the 3D voxel features at the current time as the spatial dimension. At the same time, the image features of the previous n time points are acquired, and a topological graph is generated according to the image features of the previous n time points as the modeling of the time dimension. By constructing the topological graph, the change of the object over time can be better understood.

[0042] In one embodiment, after acquiring the image features of the previous n time points and generating a topological graph according to the image features of the previous n time points as the modeling of the time dimension, it further comprises: By a multi-scale time self-attention mechanism, a plurality of different time scale history information windows are defined for the topological graph, wherein each history information window contains historical information within a corresponding time range; by the history information window, self-attention of each time scale is calculated; and the calculated self-attention results of each time scale are fused to obtain comprehensive history information, so as to effectively capture and adjust the degree of dependence on recent or long-term history information.

[0043] Specifically, a multi-scale spatio-temporal attention mechanism based on Transformer is introduced, which can effectively combine historical observation information and local details of the current environment by capturing long-time dependency, so as to improve the decision-making ability of the agent in multi-step navigation. Specifically, the spatio-temporal self-attention mechanism integrates the time sequence features of the historical navigation trajectory and the local visual information of the current scene, thereby capturing the dynamic changes in the environment and the motion trajectory of the target object. Through multi-scale modeling, the dependency relationship in different time ranges can be processed, ensuring that the navigation decision in complex scenes is more accurate and robust.

[0044] The spatio-temporal constraint modeling is as shown in Figure 3 In the spatio-temporal constraint modeling, the self-attention mechanism (Self-Attention) can capture key features in multi-step time sequence through weighted processing of historical information. However, the traditional self-attention mechanism tends to average all historical information when facing long time sequences, which may result in the inability to effectively capture recent key decision points. Therefore, a multi-scale time and space self-attention mechanism is adopted to enhance the short-term and long-term reasoning ability of the model.

[0045] In order to better process historical information of different time spans, a multi-scale time self-attention mechanism is introduced. Let respectively represent the query, key and value of the agent at time step The standard self-attention calculation formula is: wherein is the dimension of the key vector. To introduce the multi-scale information of the time dimension, multiple history information windows of different time scales are defined , each window contains the history information within a time range. For each time window , its corresponding self-attention is calculated: Finally, the attention results from different time scales are weighted and fused to obtain the comprehensive history information representation: where is the weight parameter, which automatically adjusts the importance of different time scales through training. In this way, the model can flexibly adjust the degree of dependence on recent or long-term history information according to the requirements of the task, enhancing the decision-making ability in dynamic scenarios.

[0046] In this embodiment, the multi-scale time self-attention mechanism enables the model to adjust the attention to different history information according to the current task requirements, so that it is not disturbed by irrelevant information when processing long time sequences.

[0047] In one embodiment, the image features are mapped to a blank 3D voxel space to generate the current time 3D voxel features, and modeling the current time 3D voxel features as a spatial dimension includes: The image features are mapped to a blank 3D voxel space, and a plurality of voxel units are generated through a three-dimensional occupancy network, wherein the three-dimensional occupancy network is used to divide the occupancy state of a three-dimensional space, and each voxel unit represents a corresponding region in the three-dimensional space, corresponding to different occupancy states respectively; according to the plurality of voxel units, the current time 3D voxel features are generated, and the current time 3D voxel features are modeled as a spatial dimension to obtain spatial information.

[0048] Specifically, the three-dimensional occupancy network is a voxel-based spatial representation method, which models the environment as a grid structure composed of multiple three-dimensional small cubes (voxel units) by dividing the occupancy state of a three-dimensional space. Each voxel can represent a region in space, and its value is usually 0 or 1, corresponding to "empty" or "occupied by an object" state respectively. Given a three-dimensional spatial coordinate of a scene, the occupancy network represents whether the point is occupied by the following function: where is the occupancy function, and its output value is 1 when the voxel unit is occupied by an object, and 0 when it is empty.

[0049] In traditional voxel representation, excessively high voxel resolution will lead to a sharp increase in computational cost. Therefore, in order to improve the computational efficiency and adapt to the dynamically changing environment, an optimization method based on adaptive voxel division is adopted, which models the important areas with high resolution, while the non-critical areas are divided coarsely, thereby reducing the computational overhead while ensuring accuracy.

[0050] Assuming that the voxel grid dimension in the scene is The total number of voxel division is: Through adaptive division, the resolution of the voxel can be dynamically adjusted according to the object density and its geometric complexity in the scene, thereby effectively reducing unnecessary computational burden. The final optimized voxel quantity can be expressed as: wherein, represents the voxel dimension after adaptive division. In this way, the number of voxels in the important area increases, while the non-important area decreases, thereby reducing the computational complexity of the system as a whole.

[0051] In addition to the optimization of spatial dimension, OccNet can also handle the spatio-temporal changes in dynamic scenes, and introduces a time dimension occupancy state update mechanism, allowing the occupancy network to dynamically update the environment model during multi-step navigation. For each time step t, the occupancy function is updated according to the observed environmental changes of the agent: wherein, represents the observation result of the agent at time step t, is an update function that dynamically adjusts the state of the current voxel based on the new observation information. Through this spatio-temporal dynamic update mechanism, the agent can timely capture the movement, change or disappearance of objects in the environment, thereby adapting to more complex navigation tasks.

[0052] In this embodiment, through the introduction of three-dimensional occupancy network, the agent not only has a more comprehensive understanding of the geometric structure of the environment, but also dynamically perceives the spatio-temporal changes in the scene, thereby improving the spatial reasoning ability of the agent in complex, multi-layered environments. At the same time, through adaptive voxel division and dynamic spatio-temporal update mechanism, the system significantly reduces the computational load while maintaining high accuracy.

[0053] In one embodiment, according to a plurality of voxel units, a current moment 3D voxel feature is generated, and the current moment 3D voxel feature is taken as spatial dimension modeling to obtain spatial information, and further comprising: By the multi-scale spatial self-attention mechanism, the spatial information is divided into multiple levels of different resolutions, each level representing a different spatial scale; the spatial self-attention of each scale layer is calculated, and the calculated spatial self-attention results of each scale layer are fused to capture local and global spatial information at the same time.

[0054] Specifically, in the spatial dimension, the VLN task requires the agent to capture both local details and global structure simultaneously. To process spatial information at different scales, a multi-scale spatial self-attention mechanism is introduced. Specifically, the spatial information is divided into multiple levels of different resolutions, each level representing a different spatial scale. Let the spatial feature map be F, and the multi-scale spatial feature map be , where L represents the number of scale layers, and the resolution of each layer decreases layer by layer. For each scale layer , the spatial self-attention is calculated: Then, the self-attention results of each scale are fused to capture local and global spatial information at the same time: , where is the weighting parameter of the spatial scale. Through this multi-scale processing, the model can better understand the details in the local environment while maintaining the ability to plan the global navigation path.

[0055] In this embodiment, by the multi-scale spatial self-attention mechanism, local fine features and global spatial structure are captured at the same time, improving the navigation accuracy in complex environments.

[0056] In one embodiment, the ViT model is used to extract features from the color image and the depth map, generating image features and depth features, and the depth features are rotated and translated by the camera model to generate a blank 3D voxel space including: The ViT model is used to extract features from the color image and the depth map, generating image features and depth features; the depth features are rotated and translated by the camera model to map the depth map into a pseudo point cloud; and the pseudo point cloud is voxelized to generate a blank 3D voxel space.

[0057] Specifically, as shown in Figure 4 , the depth map is input into the image feature extraction module (ViT model) to extract features, thereby generating depth features. Then, the depth features are converted into a pseudo point cloud in space through the rotation matrix and translation vector of the camera, and further voxelized to generate a blank 3D voxel space.

[0058] In one embodiment, the 3D voxel features, topological graph, and language information are input into the BERT model to obtain text features including: Based on the 3D voxel features and topological maps, environmental perception information is obtained, and based on the environmental perception information, environmental features are obtained through the visual perception model; based on the language information, language features are extracted through the pre-trained language model; the interaction between language features and environmental features is calculated, and based on the interaction relationship, the environmental feature representation is combined with the language features to form a unified joint representation, which changes with the change of environmental status over time; each language feature and environmental feature in the joint representation is aligned, and the aligned language features and environmental features are input into the BERT model to obtain text features.

[0059] Specifically, in VLN tasks, intelligent agents need to navigate complex three-dimensional environments based on natural language instructions. Therefore, accurate text understanding is crucial to the successful navigation of the agent. However, traditional text understanding methods often fail to fully incorporate environmental context, resulting in inaccurate interpretation of instructions by the agent. To address this issue, environment-guided text understanding technology is employed. By combining environmental perception with language instructions, it enables dynamic interaction between text and the environment, improving the accuracy of text understanding.

[0060] In order for the agent to dynamically adjust its understanding of language according to different environmental contexts, it is necessary to achieve a joint representation of language and environment. Specifically, given a natural language instruction and environmental perception information , the goal is to build a joint multimodal representation , to capture the fine-grained relationship between language instructions and the environment.

[0061] In order for the agent to dynamically adjust its understanding of language according to different environmental contexts, it is necessary to achieve a joint representation of language and environment. Specifically, given a natural language instruction and environmental perception information , the goal is to build a joint multimodal representation , to capture the fine-grained relationship between language instructions and the environment.

[0062] The text understanding process guided by the environment is as follows Figure 5 As shown, first, the language instruction T can be extracted through a pre-trained language model (such as BERT or Transformer) to extract language features. Let the text feature be represented as ,in For words The eigenvector of .

[0063] Meanwhile, environmental information E can be extracted through visual perception models (such as convolutional neural networks, CNN) or sensors such as lidar, resulting in an environmental feature representation where is a feature vector representing the location in the environment.

[0064] Next, through multi-modal fusion technology, we combine language features L with environmental features V to form a unified joint representation A common fusion method is to calculate the interaction between language features and environmental features through weighted inner product: where is the dimension of the feature vector. Through this joint representation, the agent can dynamically interpret the meaning of the language instructions in combination with the environmental context. For example, when the instruction mentions "turn right and walk to the red door", the agent will combine the visual information in the environment to identify and understand the actual location of "turn right" and "red door".

[0065] In navigation tasks, instructions often contain complex spatial descriptions (such as "go up the stairs", "turn around the corner"), requiring the agent to accurately understand the relationship between language instructions and the spatial layout in the environment. To achieve alignment between language and spatial information, a semantic-spatial alignment module based on attention mechanism is introduced.

[0066] For each language feature , the alignment degree with the environmental feature is calculated through the attention mechanism to generate an attention distribution which is defined as: where the attention distribution represents the relevance of the language feature to a specific location in the environment. For example, the word "stairs" in the instruction should correspond to the stair location in the environmental feature. By weighting and summing the relevance of all words to environmental locations, a semantic-spatial alignment representation is formed: Here, is the language feature modulated by environmental information, which aligns the spatial semantics in language with the corresponding physical locations in the environment, enabling the agent to more accurately understand the spatial relationships in the instructions.

[0067] In addition, environmental information should not only include static geometry and object positions, but also dynamically consider changes in the environment. For example, during navigation, some objects in the environment may move or disappear, and the agent needs to dynamically adjust its reliance on the environment when understanding language. To this end, a dynamic environmental information guidance mechanism is adopted, which dynamically adjusts the weight of language understanding through real-time perception of environmental state.

[0068] Specifically, let the current environmental state be , the next time step of the environmental state be , and the goal is to automatically adjust the part of the instruction related to the environmental state by introducing a dynamic information guidance model. Assume that the language features and environmental features are represented by linear transformation: Then, the joint representation under dynamic guidance will depend on the change of the environmental state over time: Through this dynamic guidance mechanism, the agent can perceive changes in the environment and adjust its interpretation of language instructions in a timely manner. For example, when the agent needs to "pass through an open door", if the door suddenly closes, the model will be able to update its understanding of the language instruction through dynamic perception of this change, and thus make more reasonable navigation decisions.

[0069] To further improve the agent's understanding of language instructions and the environment, a joint optimization strategy based on reinforcement learning is also introduced. In this strategy, the agent's reward function not only depends on the success or failure of navigation, but also considers the accuracy of the agent's interpretation of language instructions and environmental information.

[0070] Let the state of the agent be , the action be , and the reward be , then the agent's goal is to maximize the long-term return under the given state through action : where is the discount factor. The agent's action at each step is guided by the joint representation , so the accuracy of language understanding and environmental perception will directly affect the quality of its decision-making. Through the reinforcement learning framework, the agent can gradually optimize its joint understanding of language and environment, thereby improving the success rate of the task.

[0071] In this embodiment, through the multimodal joint representation and attention mechanism, the agent can better understand the spatial description in the language instruction and align the language instruction with the environmental information, enabling the agent to dynamically associate the language instruction with the environmental context, improving the execution accuracy of the agent under ambiguous instructions or complex descriptions, making the navigation decision more flexible and meeting the actual needs; through the perception of the environmental state changes, the agent can dynamically adjust the language understanding, and update its navigation strategy in real time when the environment changes (such as the appearance or disappearance of dynamic obstacles), improving the response ability to real-time changing scenes; by introducing language-environment joint optimization based on reinforcement learning, the agent can improve its understanding and application of language instructions and environmental perception in continuous interaction.

[0072] In one embodiment, the text features, 3D voxel features and topological graph are input into the cross-modal model, and the predicted navigation path comprises: According to the 3D voxel features and the topological graph, the visual features are obtained; the text features and the visual features are input into the cross-modal model for feature fusion through an end-to-end architecture; and the optimal navigation path is predicted through a global-local attention mechanism according to the fused features.

[0073] Specifically, in the end-to-end architecture, the training of the navigation strategy not only depends on supervised learning (such as optimization through standard action labels), but also needs to combine a reinforcement learning (RL) framework for global optimization. Through the RL framework, the agent can learn how to choose the optimal path according to the language instruction in the process of continuous interaction with the environment.

[0074] Let the state represent the environmental perception and task instruction of the agent at time t, the action represent the current navigation decision of the agent, and the reward function represent the reward after executing the action The goal of the model is to maximize the cumulative reward : wherein is a discount factor for balancing short-term and long-term rewards. During the training process, the agent selects an action according to the current state, updates the policy to maximize the long-term cumulative reward. Through this reinforcement learning mechanism, the agent can optimize the navigation strategy in different environments and improve its generalization ability to unknown scenes.

[0075] Through this RL training framework, the agent can gradually learn how to efficiently plan paths, process dynamic environmental information, and flexibly respond to different navigation tasks in complex environments.

[0076] In the end-to-end navigation system, the agent needs to balance the requirements of global planning and local navigation. To this end, a global-local attention mechanism is introduced, which can help the agent make decisions at different scales.

[0077] The local attention mechanism is used to capture detailed information in the current environment, such as the agent needs to make fine-tuning operations in a complex environment (such as avoiding obstacles or entering a specific room). The representation of local attention is: where, is the weight matrix of the local feature, represents the attention weight of the local feature.

[0078] The global attention mechanism is used to help the agent plan long-term goals, such as finding the optimal path in a large-scale environment. The representation of global attention is: where, is the weight matrix of the global feature, represents the attention weight of the global feature.

[0079] In this embodiment, the information flow in the end-to-end framework is consistent and there is no segmentation, so the model can capture the relationship between global information and local details, improving the ability of global planning and local navigation. By introducing a reinforcement learning framework, the end-to-end architecture can dynamically adjust its navigation strategy according to environmental feedback, achieving efficient decision-making and path planning. By combining local and global attention mechanisms, the model can focus on both current navigation details and overall task goals, thereby improving the accuracy and efficiency of navigation decisions.

[0080] In one embodiment, as shown in Figure 6 An apparatus for end-to-end visual language navigation based on occupancy network perception is provided, comprising: an instruction receiving module 10, a voxel space generation module 20, a voxel and topology graph generation module 30, a text feature acquisition module 40, and a navigation path prediction module 50, wherein: The instruction receiving module 10 is used to accept language instructions and acquire color images and depth maps collected by RGB cameras and depth cameras, and the language instructions carry language information; The voxel space generation module 20 is used to extract features from the color images and depth maps through a ViT model, generate image features and depth features, and perform rotation and translation transformation on the depth features through a camera model to generate a blank 3D voxel space; The voxel and topology graph generation module 30 is used to map the image features to the blank 3D voxel space to generate 3D voxel features, and construct the image features into a topology graph; The text feature acquisition module 40 is configured to input the 3D voxel feature, the topological graph, and the language information into a BERT model to obtain a text feature. The navigation path prediction module 50 is configured to input the text feature, the 3D voxel feature, and the topological graph into a cross-modal model to predict a navigation path.

[0081] In an embodiment, the voxel and topological graph generation module 30 is further configured to map the image feature to a blank 3D voxel space to generate a current time 3D voxel feature, and model the current time 3D voxel feature as a spatial dimension; and acquire image features of previous n times, and generate a topological graph as a time dimension modeling according to the image features of the previous n times.

[0082] In an embodiment, the device for end-to-end visual language navigation based on occupancy network perception further comprises a time scale processing module configured to define a plurality of different time scale history information windows for the topological graph through a multi-scale time self-attention mechanism, wherein each history information window contains history information within a corresponding time range; calculate self-attention of each time scale through the history information window; and fuse the calculated self-attention results of each time scale to obtain comprehensive history information, so as to effectively capture and adjust the degree of dependence on recent or long-term history information.

[0083] In an embodiment, the voxel and topological graph generation module 30 is further configured to map the image feature to a blank 3D voxel space, and generate a plurality of voxel units through a three-dimensional occupancy network, wherein the three-dimensional occupancy network is used to divide the occupancy state of a three-dimensional space, and each voxel unit represents a corresponding region in the three-dimensional space and corresponds to different occupancy states; and generate a current time 3D voxel feature according to the plurality of voxel units, model the current time 3D voxel feature as a spatial dimension, and obtain spatial information.

[0084] In an embodiment, the device for end-to-end visual language navigation based on occupancy network perception further comprises a spatial scale processing module configured to divide the spatial information into a plurality of different resolution levels through a multi-scale spatial self-attention mechanism, and each level represents a different spatial scale; calculate spatial self-attention of each scale layer, and fuse the calculated spatial self-attention results of each scale layer to capture local and global spatial information at the same time.

[0085] In an embodiment, the voxel space generation module 20 is further configured to perform feature extraction on the color image and the depth map through a ViT model to generate image features and depth features; perform rotation and translation transformation on the depth features through a camera model to map the depth map into a pseudo point cloud; and generate a blank 3D voxel space through voxelization of the pseudo point cloud.

[0086] In one embodiment, the text feature acquisition module 40 is also used to obtain environmental perception information based on 3D voxel features and topological maps, and obtain environmental features based on the environmental perception information through a visual perception model; extract language features based on language information through a pre-trained language model; calculate the interaction between language features and environmental features, and combine the environmental feature representation with the language features based on the interaction relationship to form a unified joint representation, which changes with the change of environmental status over time; align each language feature and environmental feature in the joint representation, and input the aligned language features and environmental features into the BERT model to obtain text features.

[0087] In one embodiment, the navigation path prediction module 50 is also used to obtain visual features based on 3D voxel features and topological maps; through an end-to-end architecture, the text features and visual features are input into a cross-modal model for feature fusion; based on the fused features, the optimal navigation path is predicted through a global-local attention mechanism.

[0088] Each module in the above-mentioned end-to-end visual language navigation device based on occupancy network perception can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0089] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store infrared image data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, an end-to-end visual language navigation method based on network perception is implemented.

[0090] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0091] The above are all preferred embodiments of the present application, and do not limit the protection scope of the present application, so that: all equivalent changes made according to the structure, shape, principle of the present application should be covered in the protection scope of the present application.

Claims

1. An end-to-end visual language navigation method based on occupancy network perception, characterized in that: include: Accepting language instructions and obtaining color images and depth maps captured by the RGB camera and the depth camera, wherein the language instructions carry language information; Performing feature extraction on the color image and the depth map using the ViT model to generate image features and depth features, and performing rotation and translation transformation on the depth features using the camera model to generate a blank 3D voxel space; Mapping the image features onto the blank 3D voxel space to generate 3D voxel features, and constructing the image features into a topological map; Inputting the 3D voxel features, the topological map, and the language information into a BERT model to obtain text features; The text features, the 3D voxel features and the topological map are input into a cross-modal model to predict a navigation path.

2. The end-to-end visual language navigation method based on occupancy network perception according to claim 1 is characterized in that: Mapping the image features onto the blank 3D voxel space to generate 3D voxel features, and constructing the image features into a topological map includes: Mapping the image features onto the blank 3D voxel space to generate a current moment 3D voxel feature, and using the current moment 3D voxel feature as a spatial dimension for modeling; The image features of the previous n moments are obtained, and a topological map is generated based on the image features of the previous n moments as a model of the time dimension.

3. The end-to-end visual language navigation method based on occupancy network perception according to claim 2 is characterized in that: After acquiring the image features of the previous n moments and generating a topological map as a model of the time dimension based on the image features of the previous n moments, the method further includes: By means of the multi-scale temporal self-attention mechanism, a plurality of historical information windows of different time scales are defined for the topological graph, wherein each historical information window contains historical information within a corresponding time range; Calculate the self-attention at each time scale through the historical information window; The self-attention results of each time scale are fused to obtain comprehensive historical information to effectively capture and adjust the degree of dependence on recent or distant historical information.

4. The end-to-end visual language navigation method based on occupancy network perception according to claim 2 is characterized in that: Mapping the image features onto the blank 3D voxel space to generate a current moment 3D voxel feature, and using the current moment 3D voxel feature as a spatial dimension for modeling includes: Mapping the image features onto the blank 3D voxel space, and generating a plurality of voxel units through a three-dimensional occupancy network, wherein the three-dimensional occupancy network is used to divide the occupancy state of the three-dimensional space, and each voxel unit represents a corresponding area in the three-dimensional space, corresponding to a different occupancy state; A 3D voxel feature at the current moment is generated according to the multiple voxel units, and the 3D voxel feature at the current moment is used as a model of a spatial dimension to obtain spatial information.

5. The end-to-end visual language navigation method based on occupancy network perception according to claim 4 is characterized in that: After generating a 3D voxel feature at a current moment based on the multiple voxel units and using the 3D voxel feature at the current moment as a model of a spatial dimension to obtain spatial information, the method further includes: Through the multi-scale spatial self-attention mechanism, the spatial information is divided into multiple layers of different resolutions, each layer representing a different spatial scale; The spatial self-attention of each scale layer is calculated and the calculated spatial self-attention results of each scale layer are fused to capture both local and global spatial information.

6. The end-to-end visual language navigation method based on occupancy network perception according to claim 1 is characterized in that: The extracting features of the color image and the depth map by using the ViT model to generate image features and depth features, and performing rotation and translation transformation on the depth features by using a camera model to generate a blank 3D voxel space includes: Performing feature extraction on the color image and the depth map using a ViT model to generate image features and depth features; Performing rotation and translation transformation on the depth feature through a camera model to map the depth map into a pseudo point cloud; The pseudo point cloud is voxelized to generate a blank 3D voxel space.

7. The end-to-end visual language navigation method based on occupancy network perception according to claim 1 is characterized in that: Inputting the 3D voxel features, the topological map, and the language information into the BERT model to obtain text features includes: Obtaining environmental perception information based on the 3D voxel features and the topological map, and obtaining environmental features through a visual perception model based on the environmental perception information; Extracting language features using a pre-trained language model based on the language information; Calculating an interaction relationship between the language feature and the environment feature, and combining the environment feature representation with the language feature according to the interaction relationship to form a unified joint representation, wherein the joint representation changes with changes in the environment state over time; Each language feature and environmental feature in the joint representation is aligned, and the aligned language features and environmental features are input into the BERT model to obtain text features.

8. The end-to-end visual language navigation method based on occupancy network perception according to claim 1 is characterized in that: Inputting the text features, the 3D voxel features, and the topological map into a cross-modal model to predict a navigation path includes: Obtaining visual features according to the 3D voxel features and the topological map; Through an end-to-end architecture, the text features and the visual features are input into a cross-modal model for feature fusion; Based on the fused features, the optimal navigation path is predicted through the global-local attention mechanism.

9. An end-to-end visual language navigation device based on occupancy network perception, characterized in that: include: The command receiving module is used to receive language commands and obtain the color image and depth map captured by the RGB camera and the depth camera. The language commands carry language information. A voxel space generation module is used to extract features from the color image and the depth map using a ViT model to generate image features and depth features, and to perform rotation and translation transformation on the depth features using a camera model to generate a blank 3D voxel space; a voxel and topology map generation module, configured to map the image features onto the blank 3D voxel space, generate 3D voxel features, and construct the image features into a topology map; A text feature acquisition module, configured to input the 3D voxel features, the topological map, and the language information into a BERT model to obtain text features; The navigation path prediction module is used to input the text features, the 3D voxel features and the topology map into a cross-modal model to predict the navigation path.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images