Dynamic gate guiding multi-view observation fusion visual language navigation method and system

Abstract

The present application relates to the technical field of robot navigation and cross-modal information fusion, and particularly relates to a visual language navigation method and system based on dynamic gating and multi-view observation fusion, which comprises: collecting RGB images, depth images, pose data and task instructions at each time step t, extracting and fusing and compressing RGB features, depth features and direction features, and encoding instructions to generate text semantic features; gating and modulating the RGB and depth features after text expansion, calculating cosine similarity, removing redundancy, linearly fusing, and predicting candidate waypoints; obtaining three-dimensional coordinates through back projection, updating a topological graph and generating a path, selecting a sub-target to calculate geometric error, generating discrete low-level actions for execution, and entering the next time step. The present application improves the accuracy and generalization ability of visual language navigation in continuous environments.

Description

Technical Field

[0001] This invention relates to the field of robot navigation and cross-modal information fusion technology, and in particular to a visual language navigation method and system based on dynamic gating-guided multi-view observation fusion. Background Technology

[0002] Existing visual language navigation methods generally suffer from problems in the waypoint prediction stage in continuous environments, such as fixed multimodal feature fusion methods, inability to dynamically adapt to the different semantic and spatial information requirements of different commands, and neglect of environmental spatial accessibility constraints, resulting in generated waypoints with high semantic matching that are actually impassable. At the same time, single-view observation is difficult to fully model feasible areas in the environment, affecting the navigation accuracy and generalization ability of the model in complex and unknown scenarios. Summary of the Invention

[0003] This invention provides a visual language navigation method and system based on dynamic gating and multi-view observation fusion, which solves the problems in existing visual language navigation methods, such as fixed multimodal feature fusion methods, neglect of spatial reachability constraints in waypoint prediction, and difficulty in fully modeling feasible regions by single-view observation.

[0004] The objective of this invention can be achieved through the following technical solutions:

[0005] The first aspect of this invention is to provide a visual-language navigation method based on dynamic gating and multi-view observation fusion, comprising: At each time step t, RGB panoramic images, depth images, pose data, and pre-input task commands from the user are acquired; RGB feature vectors are extracted from the RGB panoramic images, and depth feature vectors are extracted from the depth images; the orientation feature vectors corresponding to the heading angles in the pose data are obtained, and the depth feature vectors and the orientation feature vectors are fused and smoothed to obtain compressed depth feature vectors; the task commands are encoded to generate text semantic features. By extending the semantic features of the text, an extended text feature vector is obtained; by applying gated modulation processing to the RGB feature vector using the extended text feature vector, a gated modulation RGB feature vector is obtained; by applying gated modulation processing to the compressed depth feature vector using the extended text feature vector, a gated modulation depth feature vector is obtained. Calculate the cosine similarity between the feature vector corresponding to each viewpoint in the gated modulation RGB feature vector and the extended text feature vector; based on the relationship between the cosine similarity and the preset similarity threshold, remove the gated modulation RGB feature vector to obtain the removed gated modulation RGB feature vector; Row linear fusion is performed on the gated modulation RGB feature vector and the gated modulation deep feature vector after the removal of features to obtain the RGB deep fusion feature vector; the candidate waypoint set is determined by the RGB deep fusion feature vector. Based on the depth image and pose data, the 3D coordinates of each candidate waypoint in the candidate waypoint set are mapped by 2D-to-3D backprojection to obtain the 3D coordinates of each candidate waypoint; the candidate waypoints are judged and classified by the 3D coordinates of each candidate waypoint, and the topology map is updated to obtain the updated topology map. A topology planning path consisting of nodes is generated from the updated topology graph; the next sub-target node is selected from the topology planning path, and the geometric error from the current position to the sub-target is calculated based on the current position information to obtain the Euclidean distance and heading error; the current position information includes three-dimensional coordinates and heading angle; based on the Euclidean distance and heading error, discrete low-level actions are generated using a segmented strategy heuristic decision logic; then the robot executes the discrete low-level actions to move to the new position, and then proceeds to the next time step.

[0006] Further, the step of obtaining the direction feature vector corresponding to the heading angle in the pose data, and fusing and smoothing the depth feature vector and the direction feature vector to obtain a compressed depth feature vector includes: The directional feature vector is represented as In the formula, Indicates the current heading angle. It is a cosine function. It is a sine function; The process of smoothing and blending includes: The depth feature vector and the direction feature vector corresponding to the heading angle are concatenated and fused along the channel dimension. Then, through adaptive average pooling, the spatial feature map of each channel is compressed into a single value, and finally the compressed depth feature vector is obtained.

[0007] Furthermore, the step of extending the semantic features of the text to obtain the extended text feature vector includes: Based on the text semantic features and a vector of length N consisting entirely of 1s, the text semantic features are expanded through a dimensionality broadcasting operation to obtain an expanded text feature vector; where N is the number of panoramic views.

[0008] Further, the step of performing gated modulation processing on the RGB feature vector by extending the text feature vector to obtain the gated modulated RGB feature vector includes: The RGB feature vector and the extended text feature vector are concatenated along the channel dimension to obtain the image-text joint feature vector; The image-text joint feature vector is processed through two layers of nonlinear mapping to generate the first gated weight vector; the specific process of obtaining the first gated weight vector is expressed by the formula:

[0009] In the formula, Represents the joint feature vector of image and text. This represents the first fully connected linear transformation. This is the activation function for the first layer; This represents the second fully connected linear transformation; This is the activation function for the second layer; This represents the first gating weight vector; Based on the first gating weight vector, the RGB feature vector is modulated element-wise through the Hadamard product to obtain the gated modulated RGB feature vector; the specific process of obtaining the gated modulated RGB feature vector is expressed by the formula:

[0010] In the formula, This represents the gated modulation RGB feature vector. Represents the RGB feature vector. This represents the Hadamard product.

[0011] Further, the step of performing gated modulation processing on the compressed deep feature vector by expanding the text feature vector to obtain the gated modulation deep feature vector includes: Global average pooling is performed on the compressed deep feature vector to obtain the channel-level deep global representation vector; where global average pooling is performed through an adaptive average pooling layer; The channel-level deep global representation vector and the extended text feature vector are concatenated along the channel dimension to obtain the deep text joint feature vector. The deep text joint feature vector is then processed through two layers of nonlinear mapping to generate a second gating weight vector. The specific process for obtaining the second gating weight vector is expressed by the following formula:

[0012] In the formula, Represents the joint feature vector of deep text. This represents the second gating weight vector. This represents the first fully connected linear transformation. This is the activation function for the first layer; This represents the second fully connected linear transformation; This is the activation function for the second layer; Based on the second gating weight vector, the channel-level depth global representation vector is modulated element-wise using the Hadamard product to obtain the gated modulation depth feature vector; the specific process of obtaining the gated modulation depth feature vector is expressed by the following formula:

[0013] In the formula, This represents the channel-level depth global representation vector. This represents the depth feature vector of gated modulation. This represents the Hadamard product.

[0014] Further, the step of removing gated modulation RGB feature vectors based on the relationship between the cosine similarity and a preset similarity threshold to obtain gated modulation RGB feature vectors includes: Feature vectors corresponding to all viewpoints with cosine similarity less than the similarity threshold are selected, and the feature vectors corresponding to the viewpoints are set to zero or masked, and do not participate in subsequent feature fusion and waypoint prediction. The similarity threshold is a preset value; An RGB panoramic image is composed of images from multiple perspectives; the RGB feature vector of an RGB panoramic image is also composed of feature vectors from multiple RGB images from multiple perspectives.

[0015] Further, the row linear fusion of the removed gated modulation RGB feature vector and the gated modulation deep feature vector is performed to obtain an RGB deep fusion feature vector; the candidate waypoint set is determined through the RGB deep fusion feature vector, including: The specific formula for obtaining the RGB deep fusion feature vector is as follows:

[0016] In the formula, This represents the gated modulation RGB feature vector after removal. This represents the depth feature vector of gated modulation. It is a weighting parameter that balances image information and spatial information. ； This represents the RGB deep fusion feature vector; The process of obtaining the candidate waypoint set includes: inputting the RGB deep fusion feature vector into a multilayer perceptron classifier, and the multilayer perceptron classifier mapping the fused features into a waypoint probability heatmap; The waypoint probability heatmap can be generated in the following form:

[0017] In the formula, It is a waypoint probability heatmap; It is a multilayer perceptron classifier; The waypoint probability heatmap is normalized to obtain the probability value of each candidate point. The specific formula for obtaining the probability value of each candidate point is as follows:

[0018] In the formula, For activation function, This represents the probability value of each candidate point; Non-maximum suppression technique is used to retain the waypoints with the highest probability values ​​as a set of candidate waypoints.

[0019] Further, the step of classifying and updating the topology map by using the three-dimensional coordinates of each candidate waypoint, and obtaining the updated topology map, includes: Calculate the Euclidean distance between candidate waypoints and each node in the topology graph; set a preset distance threshold, and classify the candidate waypoints by comparing the calculated Euclidean distance with the preset distance threshold, and update the topology graph to obtain the updated topology graph; The specific process of classifying candidate waypoints and updating the topology map is as follows: (1) If the Euclidean distance between each candidate waypoint and any visited node in the topology graph at the current time step t is less than a preset distance threshold, then the candidate waypoint is discarded and an edge is added between the current node and the visited node. (2) If the Euclidean distance between each candidate waypoint and any node to be navigated in the topology at the current time step t is less than a preset distance threshold, the original node to be navigated is updated using the position information and visual information of the candidate waypoint. The updated node position and visual representation are the average of the corresponding information of the candidate waypoint and the original node to be navigated. The position information is the three-dimensional coordinates of the point, and the visual information is the RGB feature vector. (3) If the Euclidean distance between each candidate waypoint and all nodes in the topology map is greater than the threshold, then the candidate waypoint is determined as a new node to be navigated.

[0020] Furthermore, the Euclidean distance and heading error are specifically expressed by the following formulas:

[0021]

[0022]

[0023] In the formula, The three-dimensional coordinates representing the current location information Represents the three-dimensional coordinates of the next sub-target node. Indicates relative displacement. express The second norm, i.e., the modulus; Indicates Euclidean distance; Represents the arctangent function in the four quadrants; The heading angle represents the current location information; This indicates mapping the angle to... ; For heading error, This represents the component of the relative displacement on the y-axis. This represents the component of the relative displacement along the x-axis; The decision logic for the discrete low-level actions can be specifically represented as follows:

[0024] In the formula, Indicates the stopping distance threshold. Indicates the rotation angle threshold. It is the absolute value symbol. This is a sign function; it outputs +1 when the input is positive, -1 when the input is negative, and 0 when the input is zero. Represents the rotation action function; For forward motion function; To stop the action indicates the end of the navigation task; Indicates other situations; These are discrete low-level actions.

[0025] A second aspect of the present invention is to provide a visual-language navigation system with dynamic gating guidance and multi-view observation fusion, comprising: The multimodal feature extraction module is used to acquire RGB panoramic images, depth images, pose data, and pre-input task instructions from the user at each time step t; extract RGB feature vectors from the RGB panoramic images and depth feature vectors from the depth images; obtain the direction feature vectors corresponding to the heading angles in the pose data; fuse and smooth the depth feature vectors and the direction feature vectors to obtain compressed depth feature vectors; and encode the task instructions to generate text semantic features. The dynamic gating fusion module is used to obtain an extended text feature vector by extending the semantic features of the text; to obtain a gated modulated RGB feature vector by gating the extended text feature vector to the RGB feature vector; and to obtain a gated modulated deep feature vector by gating the extended text feature vector to the deep feature vector. The visual text semantic alignment and elimination module is used to calculate the cosine similarity between the feature vector corresponding to each viewpoint in the gated modulation RGB feature vector and the extended text feature vector; and to eliminate the gated modulation RGB feature vector according to the relationship between the cosine similarity and the preset similarity threshold, so as to obtain the eliminated gated modulation RGB feature vector. The multi-view observation fusion and waypoint prediction module is used to perform row linear fusion on the gated modulation RGB feature vector and the gated modulation deep feature vector after rejection to obtain the RGB deep fusion feature vector; and to determine the candidate waypoint set through the RGB deep fusion feature vector. The waypoint-driven online topology update module is used to map the 3D coordinates of each candidate waypoint in the candidate waypoint set to 2D-3D backprojection based on depth image and pose data, thereby obtaining the 3D coordinates of each candidate waypoint; and to classify the candidate waypoints based on their 3D coordinates and update the topology map to obtain the updated topology map. The low-level motion control and navigation action execution module is used to generate a topology planning path composed of nodes from the updated topology graph; select the next sub-target node from the topology planning path, and calculate the geometric error from the current position to the sub-target based on the current position information to obtain the Euclidean distance and heading error; wherein the current position information includes three-dimensional coordinates and heading angle; based on the Euclidean distance and heading error, a segmented strategy heuristic decision logic is used to generate discrete low-level actions; then the robot executes the discrete low-level actions to move to the new position, and then proceeds to the next time step.

[0026] Compared with existing technologies, the beneficial effects of this invention are as follows: At each time step t, RGB panoramic images, depth images, pose data, and user-inputted task commands are acquired; RGB feature vectors are extracted from the RGB panoramic images, and depth feature vectors are extracted from the depth images; the direction feature vectors corresponding to the heading angles in the pose data are obtained, and the depth feature vectors and the direction feature vectors are fused and smoothed to obtain compressed depth feature vectors; the task commands are encoded to generate text semantic features; multimodal features are extracted and compressed to provide structured input for subsequent fusion; extended text feature vectors are obtained by expanding the text semantic features; and the RGB feature vectors are gated and modulated using the extended text feature vectors to obtain... Gated modulation RGB feature vectors are generated by: gating modulation of depth feature vectors with extended text feature vectors to obtain gated modulation depth feature vectors; adaptive weighted fusion of RGB and depth features through gating modulation to improve the matching degree between features and instruction semantics; calculating the cosine similarity between the feature vector corresponding to each viewpoint in the gated modulation RGB feature vector and the extended text feature vector; removing gated modulation RGB feature vectors based on the relationship between the cosine similarity and a preset similarity threshold to obtain gated modulation RGB feature vectors; removing redundant visual information unrelated to instructions to reduce invalid interference and improve cross-modal alignment accuracy; and adjusting the gated modulation RGB feature vectors and gated modulation depth feature vectors accordingly. The feature vectors are linearly fused to obtain RGB deep fused feature vectors; a candidate waypoint set is determined using these RGB deep fused feature vectors; multi-view observation information is fused to generate a candidate waypoint set, enhancing the joint modeling capability of spatial reachability and semantic consistency; based on depth images and pose data, 2D-to-3D backprojection is used to map the 3D coordinates of each candidate waypoint in the candidate waypoint set, obtaining the 3D coordinates of each candidate waypoint; the candidate waypoints are classified and the topology graph is updated using their 3D coordinates, obtaining the updated topology graph; waypoints are mapped to 3D coordinates and the topology graph is dynamically updated to build a stable global environment memory; a waypoint path is generated from the updated topology graph consisting of nodes... The topology planning path is constructed; the next sub-target node is selected from the topology planning path, and the geometric error from the current position to the sub-target is calculated in combination with the current position information to obtain the Euclidean distance and heading error; the current position information includes three-dimensional coordinates and heading angle; based on the Euclidean distance and heading error, discrete low-level actions are generated using a segmented strategy heuristic decision logic; then the robot executes the discrete low-level actions to move to the new position, and then proceeds to the next time step; the topology planning is converted into continuous actions that the robot can execute, realizing closed-loop navigation from decision to execution; this solves the problems of fixed multimodal feature fusion methods, waypoint prediction ignoring spatial reachability constraints, and difficulty in fully modeling feasible areas by single-view observation in existing visual language navigation methods. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 A flowchart illustrating the steps of the visual language navigation method for dynamic gating guidance and multi-view observation fusion provided by the present invention; Figure 2 This invention provides a schematic diagram of the module flow of a visual language navigation system that provides dynamic gating guidance and multi-view observation fusion. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] To address the problems existing in the background technology, a visual language navigation method and system with dynamic gating guidance and multi-view observation fusion has been designed, which has important practical significance.

[0032] like Figure 1 As shown, the first aspect of the present invention is to provide a visual-language navigation method for dynamic gating-guided multi-view observation fusion, comprising the following steps: Step S1: At each time step t, acquire RGB panoramic image, depth image, pose data, and user-inputted task instructions; extract RGB feature vectors from the RGB panoramic image and depth feature vectors from the depth image; obtain the direction feature vector corresponding to the heading angle in the pose data, and fuse and smooth the depth feature vector and the direction feature vector to obtain a compressed depth feature vector; encode the task instructions to generate text semantic features.

[0033] It should be noted that since the navigation instructions received by the robot are natural language text, while environmental perception comes from multimodal sensor data such as RGB images and depth images, the three are heterogeneous and inconsistent information sources. Therefore, in order to unify the text semantics and visual spatial information into a feature space that can be jointly modeled, and to retain the structured information of panoramic multi-view, multimodal feature extraction and encoding operations are performed first.

[0034] Specifically, at each time step t, the agent acquires an RGB panoramic image of the environment at the current moment through the onboard panoramic camera or a combination of multiple monocular cameras, and obtains a depth image. The depth image is an image that records the distance from each pixel in the RGB panoramic image of the scene to the camera plane. Each pixel value represents the distance from that point to the sensor (in meters or millimeters), reflecting the three-dimensional geometry and spatial position of the object.

[0035] At each time step t, the agent collects the robot's pose data using sensors such as odometry and IMU. This pose data includes three-dimensional position coordinates, heading angle, and elevation angle. Simultaneously, the robot acquires pre-input task commands from the user; these pre-input task commands typically remain unchanged throughout the navigation process.

[0036] Features are extracted from the RGB panoramic image using a pre-trained CLIP image encoder (ViT-B / 32) to obtain RGB feature vectors; features are extracted from the depth image using a depth encoder (ResNet50) to obtain depth feature vectors. The RGB panoramic image is composed of images from multiple viewpoints; the RGB feature vector of the RGB panoramic image is also composed of feature vectors from multiple viewpoints of RGB images. In this embodiment, the 360° panoramic view is divided into 12 equal parts, meaning the RGB panoramic image consists of RGB images from 12 viewpoints.

[0037] Obtain the direction feature vector corresponding to the heading angle in the pose data. In the formula, This indicates the heading angle at the current moment. It is a cosine function. It is a sine function.

[0038] The depth feature vector and the orientation feature vector are fused and smoothed to obtain a compressed depth feature vector. The fusion and smoothing process includes: concatenating and fusing the depth feature vector and the orientation feature vector corresponding to the heading angle in the channel dimension, and then performing adaptive average pooling to compress the spatial feature map of each channel into a single value, finally obtaining the compressed depth feature vector.

[0039] The BERT model is used to encode task instructions to obtain text semantic features.

[0040] Thus, we have obtained the RGB feature vector, the deep feature vector, the compressed deep feature vector, and the text semantic features.

[0041] Step S2: Extend the semantic features of the text to obtain an extended text feature vector; perform gated modulation processing on the RGB feature vector using the extended text feature vector to obtain a gated modulation RGB feature vector; perform gated modulation processing on the compressed depth feature vector using the extended text feature vector to obtain a gated modulation depth feature vector.

[0042] It should be noted that in visual language navigation tasks, RGB images mainly provide semantic information, while depth images reflect spatial structure. Different navigation commands have different degrees of dependence on the two types of information (e.g., target recognition commands rely more on RGB, while path planning commands rely more on depth). Therefore, in order to adaptively adjust the fusion weights of RGB and the fused and compressed depth features according to the command content and avoid the problems of poor feature adaptability and loss of key information caused by fixed weight fusion, a text semantic-driven dynamic gating fusion operation is performed.

[0043] Specifically, based on the text semantic features and a vector of length N consisting entirely of 1s, the text semantic features are expanded through a dimensionality broadcasting operation to obtain an expanded text feature vector L. Here, N is the number of panoramic views, which is 12 in this embodiment.

[0044] Gated RGB feature vectors are obtained by gating the RGB feature vectors with extended text feature vectors; gated depth feature vectors are obtained by gating the compressed depth feature vectors with extended text feature vectors.

[0045] Specifically, the RGB feature vector is obtained by gating and modulating the RGB feature vector through extended text feature vector, including: The RGB feature vector and the extended text feature vector are concatenated along the channel dimension to obtain the image-text joint feature vector. The image-text joint feature vector is then processed through two layers of nonlinear mapping to generate the first gating weight vector. The specific process for obtaining the first gating weight vector is expressed by the following formula:

[0046] In the formula, Represents the joint feature vector of image and text. This represents the first fully connected linear transformation, which maps the input dimension to an intermediate dimension; This is the activation function for the first layer; This indicates the second fully connected linear transformation, which maps the intermediate dimension to the output dimension (consistent with the 512 RGB feature channels). This is the activation function for the second layer; This represents the first gating weight vector (used for channel-by-channel modulation of the RGB feature vector).

[0047] Based on the first gating weight vector, the RGB feature vector is modulated element-wise through the Hadamard product to obtain the gated modulated RGB feature vector; the specific process of obtaining the gated modulated RGB feature vector is expressed by the formula:

[0048] In the formula, This represents the gated modulation RGB feature vector. Represents the RGB feature vector. This represents the Hadamard product. This represents the first gating weight vector.

[0049] Specifically, the compressed deep feature vector is subjected to gated modulation processing by expanding the text feature vector to obtain a gated modulation deep feature vector, including: Global average pooling is performed on the compressed deep feature vector to obtain the channel-level deep global representation vector; where global average pooling is performed by an adaptive average pooling layer (AdaptiveAvgPool2d).

[0050] The channel-level deep global representation vector and the extended text feature vector are concatenated along the channel dimension to obtain the deep text joint feature vector. The deep text joint feature vector is then processed through two layers of nonlinear mapping to generate a second gating weight vector. The specific process for obtaining the second gating weight vector is expressed by the following formula:

[0051] In the formula, Represents the joint feature vector of deep text. This represents the second gating weight vector. This represents the first fully connected linear transformation. This is the activation function for the first layer; This represents the second fully connected linear transformation; This is the activation function for the second layer.

[0052] Based on the second gating weight vector, the channel-level depth global representation vector is modulated element-wise using the Hadamard product to obtain the gated modulation depth feature vector; the specific process of obtaining the gated modulation depth feature vector is expressed by the following formula:

[0053] In the formula, This represents the channel-level depth global representation vector. Represents the depth feature vector of gated modulation; This represents the second gating weight vector. This represents the Hadamard product.

[0054] Thus, the gated modulation RGB feature vector and the gated modulation depth feature vector are obtained.

[0055] Step S3: Calculate the cosine similarity between the feature vector corresponding to each viewpoint in the gated modulation RGB feature vector and the extended text feature vector; remove the gated modulation RGB feature vector according to the relationship between the cosine similarity and the preset similarity threshold to obtain the removed gated modulation RGB feature vector.

[0056] It should be noted that because RGB images and natural language instructions are heterogeneous modal data, each residing in different feature spaces and lacking a unified semantic measurement standard, a visual-text semantic alignment operation is performed to enable the model to determine the degree of semantic matching between image content and text instructions and to eliminate redundant visual information unrelated to the instructions. This operation maps image features and text features to the same shared semantic space and measures their matching degree using cosine similarity.

[0057] It's important to further clarify that depth features reflect spatial geometry (such as distance and obstacles) and don't directly correspond to the semantic content of text commands. RGB features, on the other hand, carry semantic information like color, texture, and objects, resulting in a higher degree of matching with text commands. Therefore, only the similarity between RGB features and text is calculated for filtering. For example, text commands might be: "Leave the bedroom, turn right into the kitchen, and stop next to the counter," or "Go straight from the doorway, past the sofa, and stop in front of the dining table." Thus, only the similarity between the RGB image and the text command is analyzed to remove redundant information from the RGB image, reducing interference from invalid visual information during model decision-making and improving the accuracy of cross-modal alignment and navigation decisions.

[0058] Specifically, the cosine similarity between the feature vector corresponding to each viewpoint in the gated modulation RGB feature vector and the extended text feature vector is calculated; this is expressed by the formula:

[0059] In the formula, Represents the first eigenvector in the gated modulation RGB feature vector. Feature vectors corresponding to each viewpoint Represents the extended text feature vector. Represents the magnitude of a vector. Indicates the first Cosine similarity between the feature vectors corresponding to each viewpoint and the feature vectors of the extended text.

[0060] Among them, the degree of semantic matching between images and text is judged based on cosine similarity. The higher the similarity, the better the semantic matching. This similarity is used to remove redundant visual information that is not related to the instructions, thereby reducing the interference of invalid information on subsequent navigation decisions.

[0061] A preset similarity threshold is set, which is 0.5 in this embodiment. However, the similarity threshold is not specifically limited in this embodiment, and the implementer can determine it according to the specific situation.

[0062] Based on the relationship between the cosine similarity between the feature vector corresponding to each viewpoint and the extended text feature vector and a preset similarity threshold, the gated modulation RGB feature vectors are eliminated to obtain the eliminated gated modulation RGB feature vectors. The specific elimination process includes: selecting all feature vectors corresponding to all viewpoints with a cosine similarity less than the similarity threshold, setting the feature vectors corresponding to the viewpoints to zero or masking them, and not participating in subsequent feature fusion and waypoint prediction.

[0063] Thus, the gated modulation RGB feature vector after removal is obtained.

[0064] Step S4: Perform row linear fusion on the removed gated modulation RGB feature vector and gated modulation deep feature vector to obtain the RGB deep fusion feature vector; determine the candidate waypoint set through the RGB deep fusion feature vector.

[0065] It should be noted that, because observation information from a single perspective is difficult to comprehensively model feasible areas in the environment, and traditional waypoint prediction, which uses a discrete candidate set screening method, is prone to missing passable paths, a waypoint prediction operation based on multi-view observation fusion is performed in order to more completely model feasible areas in the environment in continuous space and improve the spatial accessibility and semantic matching degree of waypoint prediction. By fusing RGB image features and depth image features, a waypoint probability heatmap is generated, which expands waypoint prediction from a discrete candidate set to a continuous spatial probability distribution.

[0066] Specifically, the gated modulation RGB feature vector and the gated modulation deep feature vector after removal are linearly fused to obtain the RGB deep fused feature vector; the specific formula for obtaining the RGB deep fused feature vector is as follows:

[0067] In the formula, This represents the gated modulation RGB feature vector after removal. This represents the depth feature vector of gated modulation. These are weighting parameters that balance image information and spatial information; This represents the RGB deep fusion feature vector. In this embodiment... In this embodiment, for No specific restrictions are imposed; implementers can decide based on the specific circumstances.

[0068] The RGB deep fusion feature vector is input into a multilayer perceptron classifier, which maps the fused features to a waypoint probability heatmap. This waypoint probability heatmap covers multiple angles (120 angles in total, with 3-degree intervals) and multiple distances (12 distances in total, ranging from 0.25m to 3m, with 0.25m intervals), representing the waypoint probability at each viewpoint. The waypoint probability heatmap can be generated as follows:

[0069] In the formula, It is a waypoint probability heatmap; It is a multilayer perceptron classifier.

[0070] The waypoint probability heatmap is normalized to obtain the probability value of each candidate point. The specific formula for obtaining the probability value of each candidate point is as follows:

[0071] In the formula, For activation function, This represents the probability value of each candidate point.

[0072] In the waypoint probability heatmap, candidate points are formed by the intersection of multiple angles and multiple distances (i.e., the intersection of angle lines and distance lines).

[0073] The non-maximum suppression (NMS) technique is used to retain the waypoints with the highest probability values ​​as a set of candidate waypoints; in this embodiment, the five candidate waypoints with the highest probability values ​​are retained as a set of candidate waypoints.

[0074] It should be noted that because candidate locations with adjacent angles and distances will generate multiple waypoints with similar probability values ​​but overlapping spatial locations, directly selecting the Top-5 will lead to a lot of redundancy. Non-maximum suppression can eliminate spatially overlapping candidate points and retain the one with the highest probability in a local region, ensuring that the selected 5 candidate points are spatially dispersed and do not overlap.

[0075] At this point, a set of candidate waypoints has been obtained.

[0076] Step S5: Based on the depth image and pose data, perform 3D coordinate mapping on each candidate waypoint in the candidate waypoint set through 2D to 3D back projection to obtain the 3D coordinates of each candidate waypoint; classify the candidate waypoints based on the 3D coordinates of each candidate waypoint and update the topology map to obtain the updated topology map.

[0077] It should be noted that because robots cannot obtain a complete environmental map before the task begins in a continuous environment, and the waypoints predicted in step four need to be integrated with existing spatial memory to support long-term navigation planning, waypoint-driven online topology map update operations are performed in order to structurally embed the newly predicted waypoints into the existing environmental representation to form a dynamically updatable global topology map and provide stable spatial memory for subsequent navigation decisions.

[0078] Specifically, determine the topology graph at the current time step t; where the nodes in the topology graph are divided into visited nodes, current nodes, and nodes to be navigated.

[0079] Based on the depth image and pose data, the 3D coordinates of each candidate waypoint in the candidate waypoint set are mapped by 2D to 3D backprojection to obtain the 3D coordinates of each candidate waypoint; the 2D to 3D backprojection is a well-known technique and will not be described in detail here.

[0080] Calculate the Euclidean distance between the candidate waypoints and each node in the topology graph; preset a distance threshold (0.5 in this embodiment, but not specifically limited), and classify the candidate waypoints by comparing the calculated Euclidean distance with the preset distance threshold, and update the topology graph to obtain the updated topology graph; then use the updated topology graph as the topology graph for the next time step.

[0081] The specific process of classifying candidate waypoints and updating the topology map is as follows: (1) If the Euclidean distance between each candidate waypoint and any visited node in the topology graph at the current time step t is less than a preset distance threshold, then the candidate waypoint is discarded and an edge is added between the current node and the visited node. (2) If the Euclidean distance between each candidate waypoint and any node to be navigated in the topology at the current time step t is less than a preset distance threshold, the original node to be navigated is updated using the position information and visual information of the candidate waypoint. The updated node position and visual representation are the average of the corresponding information of the candidate waypoint and the original node to be navigated. The position information is the three-dimensional coordinates of the point, and the visual information is the RGB feature vector. (3) If the Euclidean distance between each candidate waypoint and all nodes in the topology map is greater than the threshold, then the candidate waypoint is determined as a new node to be navigated.

[0082] This completes the classification and topology update of candidate waypoints.

[0083] Step S6: Generate a topology planning path composed of nodes from the updated topology graph; select the next sub-target node from the topology planning path, and calculate the geometric error from the current position to the sub-target based on the current position information to obtain the Euclidean distance and heading error; wherein the current position information includes three-dimensional coordinates and heading angle; based on the Euclidean distance and heading error, generate discrete low-level actions using a segmented strategy heuristic decision logic; then the robot executes the discrete low-level actions to move to the new position, and then proceeds to the next time step.

[0084] It should be noted that, because the nodes in the topology map generated in step S5 are discrete spatial locations, while the robot actually needs to perform continuous low-level actions (such as moving forward, rotating, and stopping), and the topology planning results need to be converted into robot-executable instructions, in order to decode the discrete topology sub-target sequence into a series of parameterized low-level action sequences and drive the agent to gradually approach the navigation endpoint along the path in the topology map, low-level motion controller and navigation action execution operations are performed.

[0085] Specifically, a topology plan path consisting of nodes is generated from the updated topology graph: In the formula, Indicates the current node position. Indicates the target location node; This is the topology planning path. The next sub-target node in the topology planning path is predicted and determined using the ETPNav model. The ETPNav (Evolving Topological Planning for Vision-Language Navigation in Continuous Environments) model is a well-known technique and will not be described in detail here.

[0086] from Select the next sub-target node The system calculates the geometric error from the current position to the sub-target by combining the current position information with the current position information, and obtains the Euclidean distance and heading error; the current position information includes three-dimensional coordinates and heading angle.

[0087] Specifically, this can be expressed as a formula:

[0088]

[0089]

[0090] In the formula, The three-dimensional coordinates representing the current location information Represents the three-dimensional coordinates of the next sub-target node. Indicates relative displacement. express The second norm, i.e., the modulus; Indicates Euclidean distance; This represents the arctangent function in the four quadrants, used to calculate the angle with the positive x-axis based on the coordinate difference (the error between the x-axis and y-axis); The heading angle represents the current location information; This indicates mapping the angle to... ; For heading error, This represents the component of the relative displacement on the y-axis. This represents the component of the relative displacement along the x-axis.

[0091] Based on Euclidean distance and heading error, a segmented strategy-based heuristic decision logic generates discrete low-level actions. The control action space consists of a set of discrete low-level actions. It includes basic movements such as moving forward, rotating, and stopping. These movements can be combined to represent complex behaviors such as path following or termination determination. Among them, discrete low-level movements... .

[0092] The decision logic for discrete low-level actions can be specifically represented as follows:

[0093] In the formula, Indicates the stopping distance threshold. Indicates the rotation angle threshold. It is the absolute value symbol. This is a sign function; it outputs +1 when the input is positive, -1 when the input is negative, and 0 when the input is zero. It is used to determine the rotation direction (left or right). This represents a rotation function that causes the robot to rotate around the vertical axis by a specified angle. The forward motion function causes the robot to move a fixed step size (0.25m) along the current orientation. To stop the action indicates the end of the navigation task; This indicates other cases, i.e., when the stopping condition is not met. And does not satisfy the rotation condition () When ), perform the forward movement.

[0094] Specifically, when the absolute value of the heading error is greater than the rotation angle threshold, a rotation maneuver is performed first to eliminate the angle error; after the direction is aligned, a forward maneuver is then performed to reduce the distance from the walking distance. Approaching the sub-target; a stopping action is triggered when the Euclidean distance is less than the stopping distance threshold. This translates the sub-target sequence at the topology level into a sequence of control actions that can be executed step-by-step in simulation or real-world environments.

[0095] When an unreachable target is encountered, infeasible nodes are removed from the topology map to prevent navigation deadlock; upon detecting a deadlock, the agent is immediately controlled to rotate and continue moving forward to ensure that it does not stop.

[0096] The implementation process of the above solution with specific data is as follows: (1) Experimental dataset.

[0097] This chapter selects R2R-CE as the experimental dataset. This dataset is built based on the Habitat simulator. By transforming the discrete navigation paths in the original R2R dataset into a continuous environment, a dataset specifically adapted for continuous environment navigation tasks is formed.

[0098] (2) Evaluation indicators.

[0099] To quantitatively evaluate the performance of the method of this invention, the R2R-CE dataset was used for experimental verification. Commonly used core metrics in this field were selected for evaluation, namely trajectory length, navigation error, success rate, OSR, and path length-weighted success rate. Among them: Trajectory Length (TL) is defined as the actual distance an agent travels from its starting point to its target location. A smaller TL value indicates higher navigation efficiency. The formula for calculating TL is as follows:

[0100] In the formula, Indicates the first The total distance actually moved by the agent in this task. This indicates the total number of navigation tasks.

[0101] Navigation Error (NE) is defined as the distance between the agent's final docking position and the preset target position. The smaller the NE value, the better the navigation accuracy. The formula for calculating NE is as follows:

[0102] In the formula, Indicates the first The Euclidean distance between the agent's stopping position and the target position in this task. Indicates the target location. Indicates the first The stopping position of the agent in this task.

[0103] Oracle Success Rate (OSR) is defined as checking whether there exists a location on the entire navigation path of the agent whose distance from the target point is within a predefined threshold range. If such a location exists, the condition is met. In this embodiment, the threshold is set to 3m.

[0104] Success Rate (SR) is defined as the number of times an agent completes navigation. It is defined as either the agent being able to observe the target object upon reaching the navigation endpoint, or the distance between the stopping point and the target location being within a preset threshold range. A single navigation attempt is considered successful if either condition is met. The SR calculation formula is as follows:

[0105] In the formula, Indicates the first The value that can be achieved upon successful completion of the next task, when the first... If the Euclidean distance between the stopping position and the target position of the agent in the next task is less than a preset threshold, then the... The value can be 1 if the second task succeeds; otherwise, the value is 1. The success rate for this task is 0.

[0106] Success weighted by path length (SPL) is a comprehensive measure of an agent's navigation success rate and path length. It penalizes redundant routes, encouraging the agent to complete the navigation task using a shorter path. The SPL calculation formula is as follows:

[0107] In the formula, Indicates the first The longest path length in this task. This represents the function that takes the maximum value.

[0108] (3) Model performance comparison and analysis.

[0109] To verify the effectiveness of the dynamically gated guided multi-view observation fusion method proposed in this chapter, a comparative experiment was conducted on the R2R-CE dataset with existing typical VLN-CE methods. The results are shown in Tables 1 and 2. Overall, the method in this chapter achieves superior performance on most metrics, demonstrating its effectiveness.

[0110] Table 1. Performance comparison of different models on the VLN-CE visible validation set.

[0111] Table 2. Performance comparison of different models on the VLN-CE invisible validation set.

[0112] In the validation set, the navigation error was further reduced to 3.43m, and the navigation success rate (SR) and path length-weighted success rate (SPL) reached 68% and 61%, respectively. This demonstrates that the method presented in this chapter can reach the target location more accurately in a known environment, improving navigation accuracy. This indicates that in a known environment, the method not only improves navigation position accuracy but also optimizes navigation path efficiency while maintaining success rate, achieving better navigation task completion. Compared to cross-modal fusion enhancement methods, both are based on topological graphs for long-term planning. However, this chapter further optimizes spatial accessibility and multimodal adaptability at the waypoint prediction level, allowing the model to flexibly adjust the weights of the two types of features according to different navigation command requirements, avoiding the problems of poor feature adaptability and loss of key information caused by fixed-weight fusion. Furthermore, the waypoint prediction method based on multi-view observation fusion more comprehensively models feasible areas in the environment, improving the spatial accessibility and semantic matching degree of waypoint prediction.

[0113] In the unseen validation set, the navigation error NE decreased to 4.39m, while SR and SPL improved to 60% and 50%, respectively. This demonstrates that the method presented in this chapter can more effectively approach the target area in complex and unknown environments and increase the probability of successfully reaching the target. This shows the effectiveness of the method presented in this chapter. By introducing textual semantic information into the waypoint prediction process in advance and combining it with a dynamic gating mechanism to adaptively adjust RGB and depth features, the model can dynamically select more important semantic or spatial cues based on different navigation commands. In addition, the waypoint prediction method based on multi-view observation fusion replaces the original discrete candidate waypoint selection method, which can perform more complete probabilistic modeling of potential feasible areas in continuous space and reduce waypoint deviation problems caused by missing candidate sets. Finally, by combining visual textual semantic alignment and an online topology map update mechanism, the model can not only obtain more accurate local targets in unknown environments, but also maintain the consistency of global path planning by relying on topological memory, achieving better generalization ability and navigation stability.

[0114] This concludes the embodiment.

[0115] like Figure 2 As shown, a second aspect of the present invention is to provide a visual-language navigation system with dynamic gating guidance and multi-view observation fusion, comprising: The multimodal feature extraction module 101 is used to acquire RGB panoramic images, depth images, pose data, and pre-input task instructions from the user at each time step t; extract RGB feature vectors from the RGB panoramic images and depth feature vectors from the depth images; obtain the direction feature vectors corresponding to the heading angles in the pose data; fuse and smooth the depth feature vectors and the direction feature vectors to obtain compressed depth feature vectors; and encode the task instructions to generate text semantic features. The dynamic gating fusion module 102 is used to obtain an extended text feature vector by extending the semantic features of the text; to obtain a gated modulation RGB feature vector by gating the extended text feature vector to the RGB feature vector; and to obtain a gated modulation deep feature vector by gating the extended text feature vector to the deep feature vector. The visual text semantic alignment and elimination module 103 is used to calculate the cosine similarity between the feature vector corresponding to each viewpoint in the gated modulation RGB feature vector and the extended text feature vector; and to eliminate the gated modulation RGB feature vector according to the relationship between the cosine similarity and the preset similarity threshold to obtain the eliminated gated modulation RGB feature vector. The multi-view observation fusion and waypoint prediction module 104 is used to perform row linear fusion on the gated modulation RGB feature vector and the gated modulation deep feature vector after rejection to obtain the RGB deep fusion feature vector; and to determine the candidate waypoint set through the RGB deep fusion feature vector. The waypoint-driven online topology update module 105 is used to map the three-dimensional coordinates of each candidate waypoint in the candidate waypoint set according to the depth image and pose data through 2D to 3D back projection, so as to obtain the three-dimensional coordinates of each candidate waypoint; and to judge and classify the candidate waypoints through the three-dimensional coordinates of each candidate waypoint and update the topology map to obtain the updated topology map. The low-level motion control and navigation action execution module 106 is used to generate a topology planning path composed of nodes from the updated topology graph; select the next sub-target node from the topology planning path, and calculate the geometric error from the current position to the sub-target by combining the current position information to obtain the Euclidean distance and heading error; wherein the current position information includes three-dimensional coordinates and heading angle; based on the Euclidean distance and heading error, a segmented strategy heuristic decision logic is used to generate discrete low-level actions; then the robot executes the discrete low-level actions to move to a new position, and then proceeds to the next time step.

[0116] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, etc.) containing computer-usable program code.

[0117] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, systems, and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0118] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0119] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the present invention.

Claims

1. A visual-language navigation method based on dynamic gating and multi-view observation fusion, characterized in that, include: At each time step t, RGB panoramic images, depth images, pose data, and pre-input task instructions from the user are acquired. Extract RGB feature vectors from RGB panoramic images and depth feature vectors from depth images; obtain the direction feature vectors corresponding to the heading angles in pose data; fuse and smooth the depth feature vectors and the direction feature vectors to obtain compressed depth feature vectors; encode task instructions to generate text semantic features. By extending the semantic features of the text, an extended text feature vector is obtained; by applying gated modulation processing to the RGB feature vector using the extended text feature vector, a gated modulation RGB feature vector is obtained; by applying gated modulation processing to the compressed depth feature vector using the extended text feature vector, a gated modulation depth feature vector is obtained. Calculate the cosine similarity between the feature vector corresponding to each viewpoint in the gated modulation RGB feature vector and the extended text feature vector; based on the relationship between the cosine similarity and the preset similarity threshold, remove the gated modulation RGB feature vector to obtain the removed gated modulation RGB feature vector; Row linear fusion is performed on the gated modulation RGB feature vector and the gated modulation deep feature vector after the removal of features to obtain the RGB deep fusion feature vector; the candidate waypoint set is determined by the RGB deep fusion feature vector. Based on the depth image and pose data, the 3D coordinates of each candidate waypoint in the candidate waypoint set are mapped by 2D-to-3D backprojection to obtain the 3D coordinates of each candidate waypoint; the candidate waypoints are judged and classified by the 3D coordinates of each candidate waypoint, and the topology map is updated to obtain the updated topology map. Generate a topology plan path consisting of nodes from the updated topology graph; The next sub-target node is selected from the topology planning path, and the geometric error from the current position to the sub-target is calculated by combining the current position information to obtain the Euclidean distance and heading error; the current position information includes three-dimensional coordinates and heading angle; based on the Euclidean distance and heading error, discrete low-level actions are generated using a segmented strategy heuristic decision logic; then the robot executes the discrete low-level actions to move to the new position, and then proceeds to the next time step.

2. The visual-language navigation method based on dynamic gating and multi-view observation fusion according to claim 1, characterized in that, The step of acquiring the direction feature vector corresponding to the heading angle in the pose data, and fusing and smoothing the depth feature vector and the direction feature vector to obtain a compressed depth feature vector includes: The directional feature vector is represented as In the formula, Indicates the current heading angle. It is a cosine function. It is a sine function; The process of smoothing and blending includes: The depth feature vector and the direction feature vector corresponding to the heading angle are concatenated and fused along the channel dimension. Then, through adaptive average pooling, the spatial feature map of each channel is compressed into a single value, and finally the compressed depth feature vector is obtained.

3. The visual-language navigation method based on dynamic gating and multi-view observation fusion according to claim 1, characterized in that, The process of expanding the semantic features of the text to obtain an extended text feature vector includes: Based on the text semantic features and a vector of length N consisting entirely of 1s, the text semantic features are expanded through a dimensionality broadcasting operation to obtain an expanded text feature vector; where N is the number of panoramic views.

4. The visual-language navigation method based on dynamic gating and multi-view observation fusion according to claim 1, characterized in that, The step of performing gated modulation processing on the RGB feature vector by extending the text feature vector to obtain the gated modulated RGB feature vector includes: The RGB feature vector and the extended text feature vector are concatenated along the channel dimension to obtain the image-text joint feature vector; The image-text joint feature vector is processed through two layers of nonlinear mapping to generate the first gated weight vector; the specific process of obtaining the first gated weight vector is expressed by the formula: In the formula, Represents the joint feature vector of image and text. This represents the first fully connected linear transformation. This is the activation function for the first layer; This represents the second fully connected linear transformation; This is the activation function for the second layer; This represents the first gating weight vector; Based on the first gating weight vector, the RGB feature vector is modulated element-wise through the Hadamard product to obtain the gated modulated RGB feature vector; the specific process of obtaining the gated modulated RGB feature vector is expressed by the following formula: In the formula, This represents the gated modulation RGB feature vector. Represents the RGB feature vector. This represents the Hadamard product.

5. The visual-language navigation method based on dynamic gating and multi-view observation fusion according to claim 1, characterized in that, The step of performing gated modulation processing on the compressed deep feature vector by expanding the text feature vector to obtain the gated modulation deep feature vector includes: Global average pooling is performed on the compressed deep feature vector to obtain the channel-level deep global representation vector; where global average pooling is performed through an adaptive average pooling layer; The channel-level deep global representation vector and the extended text feature vector are concatenated along the channel dimension to obtain the deep text joint feature vector. The deep text joint feature vector is then processed through two layers of nonlinear mapping to generate a second gating weight vector. The specific process for obtaining the second gating weight vector is expressed by the following formula: In the formula, Represents the joint feature vector of deep text. This represents the second gating weight vector. This represents the first fully connected linear transformation. This is the activation function for the first layer; This represents the second fully connected linear transformation; This is the activation function for the second layer; Based on the second gating weight vector, the channel-level depth global representation vector is modulated element-wise using the Hadamard product to obtain the gated modulation depth feature vector; the specific process of obtaining the gated modulation depth feature vector is expressed by the following formula: In the formula, This represents the channel-level depth global representation vector. This represents the depth feature vector of gated modulation. This represents the Hadamard product.

6. The visual-language navigation method based on dynamic gating and multi-view observation fusion according to claim 1, characterized in that, The step of removing gated modulation RGB feature vectors based on the relationship between the cosine similarity and a preset similarity threshold, to obtain gated modulation RGB feature vectors after removal, includes: Feature vectors corresponding to all viewpoints with cosine similarity less than the similarity threshold are selected, and the feature vectors corresponding to the viewpoints are set to zero or masked, and do not participate in subsequent feature fusion and waypoint prediction. The similarity threshold is a preset value; An RGB panoramic image is composed of images from multiple perspectives; the RGB feature vector of an RGB panoramic image is also composed of feature vectors from multiple RGB images from multiple perspectives.

7. The visual-language navigation method based on dynamic gating and multi-view observation fusion according to claim 1, characterized in that, The process involves performing row linear fusion on the gated modulation RGB feature vector and the gated modulation deep feature vector after rejection to obtain an RGB deep fused feature vector; the candidate waypoint set is then determined using the RGB deep fused feature vector, including: The specific formula for obtaining the RGB deep fusion feature vector is as follows: In the formula, This represents the gated modulation RGB feature vector after removal. This represents the depth feature vector of gated modulation. These are weighting parameters that balance image information and spatial information; Represents the RGB deep fusion feature vector; The process of obtaining the candidate waypoint set includes: inputting the RGB deep fusion feature vector into a multilayer perceptron classifier, and the multilayer perceptron classifier mapping the fused features into a waypoint probability heatmap; The waypoint probability heatmap can be generated in the following form: In the formula, It is a waypoint probability heatmap; It is a multilayer perceptron classifier; The waypoint probability heatmap is normalized to obtain the probability value of each candidate point. The specific formula for obtaining the probability value of each candidate point is as follows: In the formula, For activation function, This represents the probability value of each candidate point; Non-maximum suppression technique is used to retain the waypoints with the highest probability values ​​as a set of candidate waypoints.

8. The visual-language navigation method based on dynamic gating and multi-view observation fusion according to claim 1, characterized in that, The process of classifying and updating the topology map by using the three-dimensional coordinates of each candidate waypoint, and obtaining the updated topology map, includes: Calculate the Euclidean distance between candidate waypoints and each node in the topology graph; set a preset distance threshold, and classify the candidate waypoints by comparing the calculated Euclidean distance with the preset distance threshold, and update the topology graph to obtain the updated topology graph; The specific process of classifying candidate waypoints and updating the topology map is as follows: (1) If the Euclidean distance between each candidate waypoint and any visited node in the topology graph at the current time step t is less than a preset distance threshold, then the candidate waypoint is discarded and an edge is added between the current node and the visited node. (2) If the Euclidean distance between each candidate waypoint and any node to be navigated in the topology at the current time step t is less than a preset distance threshold, the original node to be navigated is updated using the position information and visual information of the candidate waypoint. The updated node position and visual representation are the average of the corresponding information of the candidate waypoint and the original node to be navigated. The position information is the three-dimensional coordinates of the point, and the visual information is the RGB feature vector. (3) If the Euclidean distance between each candidate waypoint and all nodes in the topology map is greater than the threshold, then the candidate waypoint is determined as a new node to be navigated.

9. The visual-language navigation method based on dynamic gating and multi-view observation fusion according to claim 1, characterized in that, The Euclidean distance and heading error are specifically expressed by the following formulas: In the formula, The three-dimensional coordinates representing the current location information Represents the three-dimensional coordinates of the next sub-target node. Indicates relative displacement. express The second norm, i.e., the modulus; Indicates Euclidean distance; Represents the arctangent function in the four quadrants; The heading angle represents the current location information; This indicates mapping the angle to... ; For heading error, This represents the component of the relative displacement on the y-axis. This represents the component of the relative displacement along the x-axis; The decision logic for the discrete low-level actions can be specifically represented as follows: In the formula, Indicates the stopping distance threshold. Indicates the rotation angle threshold. It is the absolute value symbol. This is a sign function; it outputs +1 when the input is positive, -1 when the input is negative, and 0 when the input is zero. Represents the rotation action function; For forward motion function; To stop the action indicates the end of the navigation task; Indicates other situations; These are discrete low-level actions.

10. A visual language navigation system with dynamic gating-guided multi-view observation fusion, used to implement the visual language navigation method with dynamic gating-guided multi-view observation fusion as described in any one of claims 1-9, characterized in that, include: The multimodal feature extraction module is used to acquire RGB panoramic images, depth images, pose data, and pre-input task instructions from the user at each time step t. Extract RGB feature vectors from RGB panoramic images and depth feature vectors from depth images; obtain the direction feature vectors corresponding to the heading angles in pose data; fuse and smooth the depth feature vectors and the direction feature vectors to obtain compressed depth feature vectors; encode task instructions to generate text semantic features. The dynamic gating fusion module is used to obtain an extended text feature vector by extending the semantic features of the text; to obtain a gated modulated RGB feature vector by gating the extended text feature vector to the RGB feature vector; and to obtain a gated modulated depth feature vector by gating the compressed depth feature vector to the extended text feature vector. The visual text semantic alignment and elimination module is used to calculate the cosine similarity between the feature vector corresponding to each viewpoint in the gated modulation RGB feature vector and the extended text feature vector; and to eliminate the gated modulation RGB feature vector according to the relationship between the cosine similarity and the preset similarity threshold, so as to obtain the eliminated gated modulation RGB feature vector. The multi-view observation fusion and waypoint prediction module is used to perform row linear fusion on the gated modulation RGB feature vector and the gated modulation deep feature vector after rejection to obtain the RGB deep fusion feature vector; and to determine the candidate waypoint set through the RGB deep fusion feature vector. The waypoint-driven online topology update module is used to map the 3D coordinates of each candidate waypoint in the candidate waypoint set to 2D-3D backprojection based on depth image and pose data, thereby obtaining the 3D coordinates of each candidate waypoint; and to classify the candidate waypoints based on their 3D coordinates and update the topology map to obtain the updated topology map. The low-level motion control and navigation action execution module is used to generate a topology planning path consisting of nodes from the updated topology graph; The next sub-target node is selected from the topology planning path, and the geometric error from the current position to the sub-target is calculated by combining the current position information to obtain the Euclidean distance and heading error; the current position information includes three-dimensional coordinates and heading angle; based on the Euclidean distance and heading error, discrete low-level actions are generated using a segmented strategy heuristic decision logic; then the robot executes the discrete low-level actions to move to the new position, and then proceeds to the next time step.

Images