Real-time visual language navigation method based on frontier exploration and neural relationship inference

Abstract

The invention discloses a real-time visual language navigation method based on frontier exploration and neural relationship inference. The method comprises the following steps: constructing an initial occupation grid map and a confidence value map; based on the occupied grid map, identifying a boundary between the explored area and the unexplored area, and generating a candidate leading edge waypoint set; a vision-language model is used for detecting instance objects existing in the current visual field, and points closest to the robot are extracted to serve as a candidate instance route point set; calculating visual language comprehensive scores of all waypoints in the candidate frontier waypoint set and the candidate instance waypoint set, and selecting the waypoint with the highest score as a next navigation target; and based on the selected target waypoint, calling a local path planner to generate a collision-free path, driving the robot to move, and updating the occupied grid map and the confidence value map in a rolling manner. Task specific training is not needed, vision and language semantic information can be efficiently fused, and intelligent exploration and target navigation in an unknown environment are achieved.

Description

Technical Field

[0001] This invention belongs to the field of autonomous navigation for mobile robots, and in particular relates to a real-time visual language navigation method based on cutting-edge exploration and neural relationship inference. Background Technology

[0002] In the intersection of robotics and artificial intelligence, enabling robots to understand human natural language commands and navigate autonomously in unknown environments—a core challenge that researchers have been striving to overcome. This technology is a crucial foundation for future intelligent home assistants, disaster relief robots, and autonomous exploration systems. An ideal visual language navigation system not only needs to accurately perceive the geometry of its environment and avoid obstacles like traditional robots, but also needs to possess a deep semantic understanding of human commands. For example, upon receiving a command such as "find a person wearing gray clothes," the robot should be able to actively and strategically search for the target in complex spaces, rather than engaging in aimless random wandering.

[0003] Looking back at existing technologies, traditional navigation methods, represented by simultaneous localization and mapping (SMR) and frontier exploration strategies, construct geometric maps of the environment and guide exploration by identifying the boundaries of unknown areas. While these methods are efficient and have a clear architecture, their fundamental flaw lies in their inability to understand and integrate the rich semantic information provided by humans. They are like a guide who can only read a map but cannot understand instructions, unable to connect the linguistic concept of "a person in gray clothes" with visual perception, and therefore incapable of handling semantic-based search tasks.

[0004] To endow robots with semantic understanding capabilities, end-to-end deep learning models have emerged. These methods attempt to directly map raw sensor data into navigation actions using deep neural networks. While they have shown potential in specific simulation environments, their "black box" nature leads to serious interpretability problems, making it difficult for developers to diagnose and correct errors that are bound to occur in real-world scenarios. More importantly, their enormous computational demands make real-time operation on embedded robot platforms difficult, and their performance heavily relies on an environment that closely matches the training data, resulting in questionable generalization capabilities when faced with the diversity and variability of the real world.

[0005] In recent years, a modular approach combining the robustness of classical architectures with modern semantic understanding capabilities has gained popularity, especially with the rise of large-scale foundational models such as visual language models, enabling robots to understand the semantics of open-ended words like "chair" and "computer" with zero-shot processing. Some existing advanced systems have attempted to combine the semantic information output by visual language models (VLMs) with cutting-edge exploration frameworks, for example, by calculating the global similarity between the entire image and the instruction to score different exploration directions.

[0006] However, this approach has significant limitations. First, the image-level global features it relies on are too coarse to capture fine-grained relationships between multiple objects in a scene. When instructions involve tasks requiring understanding spatial relationships between objects, such as "the keys on the table" or "the backpack next to the sofa," accurate judgments cannot be made based solely on global features. Second, many graph neural networks used in navigation to model multi-agent or object interactions often learn implicit and difficult-to-interpret interactions, lacking the ability to explicitly model the inherent, sparse physical connections in the real world. Finally, the inherent computational bottleneck of large VLM models forces many systems to use expensive and power-intensive hardware, which greatly limits their practical application and widespread adoption on lightweight, low-power mobile robot platforms.

[0007] Therefore, the current technology community urgently needs a brand-new solution that can achieve real-time computing on resource-constrained embedded systems, while possessing a semantic understanding depth that surpasses existing methods—that is, not only can it see objects, but it can also understand the relationships between objects, thereby performing logical and interpretable spatial reasoning like humans, and ultimately achieving robust and efficient visual language navigation in diverse real-world environments. Summary of the Invention

[0008] To overcome the problems of waypoint confusion, redundant exploration, and insufficient zero-shot adaptability caused by the lack of high-level semantic relationships in existing visual language navigation technologies in unknown environments, this invention provides a real-time visual language navigation method based on frontier exploration and neural relationship inference. It can fuse pixel-level visual-language features and offline general latent graph relationships within a millisecond cycle, and achieve dynamic alignment of exploration direction and semantic commands, thereby significantly improving the zero-shot navigation success rate and path efficiency of mobile robots on resource-constrained platforms.

[0009] A real-time visual language navigation method based on cutting-edge exploration and neural relationship inference includes: (1) The robot rotates at its initial position and uses its depth camera and odometry data to construct an initial occupancy grid map and confidence value map; (2) Based on the occupied grid map, the boundary between the explored and unexplored areas is identified as the frontier area, and a set of candidate frontier waypoints is generated by breadth-first search algorithm. ; (3) Based on the images acquired by the depth camera, the pre-trained vision-language model is used to detect the instance objects existing in the current field of view, and the points closest to the robot are extracted as the candidate instance waypoint set. ; (4) Combine the semantic channel values ​​in the confidence value map to calculate the visual language comprehensive score for all waypoints in the candidate front waypoint set and the candidate instance waypoint set. And select the waypoint with the highest score as the next navigation target; (5) Based on the selected target waypoint, call the local path planner to generate a collision-free path, drive the robot to move, and continuously update the occupied grid map and confidence value map. (6) Repeat steps (2) to (5) until the robot reaches the target object or reaches the maximum number of exploration steps.

[0010] In step (1), the confidence value map is derived from a semantic channel. With confidence channel The composition and update formula are as follows: ; ; In this context, the superscript "new" indicates the updated value, the superscript "curr" indicates the current value, and the superscript "prev" indicates the previous value. The cosine similarity score is calculated by inputting the current RGB image and the target object text prompt into a pre-trained visual-language model. The calculation formula is: ; in, For grid cells The angle between the direction and the camera's optical axis, This is the horizontal field of view of the camera.

[0011] In step (4), the visual language comprehensive score The calculation formula is: in, To reach the waypoint with the robot The score is inversely proportional to the distance. For waypoints The score of the unknown region is positively correlated with the proportion of unknown grid cells in the neighborhood. The consistency score is calculated by the neural relationship inference model. and For weight hyperparameters.

[0012] Consistency score The calculation formula is: For waypoints , ; For waypoints , ; in, waypoints The corresponding semantic channel value, This represents text instructions inferred by a neural relationship inference model. With waypoint The posterior probability that there is a semantic matching relationship between objects.

[0013] The construction and training process of the neural relationship inference model is as follows: Graph construction steps: Given a text instruction and an RGB-D image, extract the text embedding. and visual embedding Mapped to the same d-dimensional space, forming a set of nodes. And construct a fully connected set of edges. ; Feature initialization steps: For each edge Define initial features ,in and For node features, Relative geometric deviations are accounted for in text-object edges; Message passing steps: Perform two rounds of message passing to update edge embeddings and node representations: ; ; ; Relationship inference steps: through fully connected layers Embed the final edge The mapping is to the relation type logits, and the posterior distribution is obtained by the softmax function. ; Model training steps: Using weak supervision signals based on task success or failure, minimize the loss function. To train the model, the loss function is: ; in, This is an indicator function for task success. For hyperparameters, As a prior distribution, Represents text instructions With objects The relationship between implicit variables is specified here, with label 1 designated as a text instruction. With objects Semantic matching.

[0014] With the robot to the waypoint The distance score is inversely proportional to the distance. The formula is as follows: ; in, agent For robots and waypoints The Euclidean distance between them; with waypoints The score of the unknown region is positively correlated with the proportion of unknown grid cells in the neighborhood. The formula is as follows: ; in, This indicates that the number of unknown lattice elements is counted within the neighborhood of waypoint g. This indicates the number of visited grid cells at waypoints. Statistics within the neighborhood.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. Traditional waypoint scoring mechanisms based on pixel-level visual-language similarity are prone to producing multiple candidate points with similar scores in topological bifurcation or cross-room scenarios, causing the robot to frequently switch target directions, resulting in path detours and energy waste. This invention introduces a consistency score based on neural relationship inference on the basis of the existing navigation framework based on frontier exploration. By enhancing neural relationships and coupling with frontier point exploration, a unified evaluation function is constructed, which assigns higher weights to semantically related directions, effectively suppressing the tendency of ineffective exploration and significantly improving path efficiency and command alignment.

[0016] 2. The method of the present invention has zero-sample capability and good generalization ability; the lightweight GNN relation tensor and decoder are encapsulated as independent modules, which can seamlessly replace the detector or planner, and the system has cross-platform migration and life cycle evolution capabilities. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a real-time visual language navigation method based on cutting-edge exploration and neural relationship inference, according to an embodiment of the present invention.

[0019] Figure 2 This is a schematic diagram of the system framework corresponding to the method of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.

[0022] The method of this invention is applicable to mobile robot platforms deployed on low-power edge computing terminals. This method utilizes offline neural relation inference to pre-miner general interaction patterns between objects, regions, and functions from large-scale trajectories, and compresses these patterns into lightweight relation tensors. In the online phase, instance nodes in the current field of view are generated in real time using an open vocabulary detector, and the relation tensor is quickly queried through a decoder to obtain knowledge-enhanced node embeddings. Furthermore, the above embeddings are coupled with the frontier point exploration score to form an adaptively adjustable fusion score, used to select the next-hop waypoint that best matches the language instructions and is most conducive to exploration in each frame. Finally, a real-time navigation path that satisfies both semantic alignment and computational resource constraints is generated for zero-shot visual-language navigation in continuous environments.

[0023] Specifically, such as Figure 1 As shown, a real-time visual language navigation method based on cutting-edge exploration and neural relationship inference, suitable for deployment on indoor mobile robot platforms with limited computing resources, includes the following steps: S1 Environment Initialization: The robot constructs an initial occupancy grid map and confidence value map by rotating in its initial position, using data from its depth camera and odometry.

[0024] The confidence value map consists of two channels: a semantic channel and a semantic channel. With confidence channel Updated on a rolling basis using a weighted average: ; ; in, Calculate the cosine similarity score between the current RGB image and the target object text prompt input using a pre-trained visual-language model. From this grid cell Angle between direction and optical axis and the horizontal field of view of the front camera pose It is calculated using the following formula: ; in, The angle between the direction of this grid cell and the camera's optical axis. This is the horizontal field of view of the camera.

[0025] S2 Forward Waypoint Generation: Based on the grid map occupied in step S1, the boundary between the explored and unexplored areas, namely the "forward area", is identified, and candidate forward waypoints are generated through a breadth-first search algorithm.

[0026] S3 Instance Waypoint Generation: Based on the depth image acquired in step S1, a pre-trained vision-language model is used to detect instance objects in the current field of view, extract the outline of the instance objects, and calculate the point closest to the robot as a candidate instance waypoint.

[0027] S4 Target Waypoint Selection: The set of leading waypoints generated in steps S2 and S3 With instance waypoint set Calculate the visual language comprehensive score separately. The waypoint with the highest score is selected as the navigation target for the next moment.

[0028] For waypoints The formula for calculating the visual language comprehensive score is as follows: ; a) Consistency score It is obtained by decoding the compressed relation tensor of the neural relation inference model; b) Distance score It is inversely proportional to the Euclidean distance from the robot to the target point; c) Scores for unknown areas , and the target point The proportion of unknown grid cells within the neighborhood is positively correlated; in, ; Specifically, the encoder-decoder architecture neural relationship inference model is trained using the following steps: (4-1) Given a text instruction and an RGB-D image, extract the text embedding and the visual embedding of each detection box using a vision-language pre-trained model, and linearly map them to the same... Dimensional space: ; in To detect object numbers, Embedding features for text The objects observed within the current field of view Feature embedding.

[0029] The set of nodes in the graph is thus formed. The edge set is fully connected. .

[0030] (4-2) Define initial features for each edge and assign contextual information to the edges: ; Here and Pick or Assign values ​​according to node type. The relative geometric deviation is included in the text-object edges, and zeros are filled in the remaining edges.

[0031] (4-3) Then, two rounds of message passing are performed: the first round generates edge embeddings on local binary pairs, and the second round aggregates the full graph context and updates the node representations: ; ; ; (4-4) Finally, each edge yields a relation type. hyperparameters Let type 1 be the "semantic matching" relation type. Obtain the posterior probability distribution using softmax. ; The training objective relies solely on weakly supervised signals: if the entire trajectory successfully finds the target, the probability of type 1 on all "text-object" edges is increased; otherwise, it is decreased. A simplified loss function is used. : ; in, This is an indicator function for task success. For hyperparameters, As a prior distribution, Represents text instructions With objects The implicit variables in the relationship are specified here as type 1, which represents a text instruction. With objects Semantic matching.

[0032] S5 Waypoint Navigation: Based on the target waypoint selected in step S4, a point-target navigation strategy is adopted. The local path planner is called to generate a collision-free path in real time, driving the robot to move towards the selected waypoint. At the same time, the occupation grid map and confidence value map are updated in a rolling manner to ensure the timeliness and accuracy of the environmental model.

[0033] S6 Loop and Termination: Repeat the above steps to continuously update the environmental representation and navigation target until the robot successfully reaches the vicinity of the target object and triggers the stop action, or reaches the preset maximum number of exploration steps, at which point the task ends.

[0034] To verify the effectiveness of this invention, in this embodiment, a wheeled service robot (using an NVIDIA Jetson Orin NX computing platform) is required to perform a visual-language navigation task in an unfamiliar home environment of approximately 300 square meters containing multiple rooms (living room, kitchen, and bedroom). The goal is to find the target object based on the user's instruction to "get the blue cup on the desk in the bedroom." This environment features a typical home layout, dynamic lighting changes, and partial occlusion. Traditional VLN methods tend to fail on such resource-constrained robots due to high computational load, inaccurate semantic understanding, and low exploration efficiency. The method of this invention is applied below to achieve efficient and robust real-time navigation.

[0035] like Figure 2 As shown, the system of the present invention adopts a closed-loop process of perception-reasoning-decision-execution, including the following specific implementation steps: Step S1: Environment Initialization and Real-time Sensing The robot is deployed at the living room entrance. Upon startup, it first rotates 360 degrees from its initial position (angular velocity 0.5 rad / s), with its Intel RealSense D455 depth camera (outputting 30Hz RGB-D images) and wheeled odometry working synchronously. The occupying grid map building module initializes the map at a 5cm resolution. The confidence value map building module is then initialized synchronously: semantic channel. Initially 0, confidence channel The initial value is 0.1. During rotation, the real-time acquired RGB image and the text command "blue cup" are input into the CLIP pre-trained model to calculate the cosine similarity between the image blocks and the text, which serves as the basis for the calculation. Simultaneously, based on the camera's intrinsic and extrinsic parameters and the azimuth angle of the grid cells... and horizontal field of view According to the formula Calculate the current observation confidence level. After initialization, an initial confidence value map of approximately 10 square meters is obtained.

[0036] Step S2: Multimodal waypoint generation The waypoint generation module uses breadth-first search (BFS) on the "explored-unknown" boundary, based on the updated occupied grid map, to search for waypoints within the "explored-unknown" boundary. It sets the minimum cluster size of waypoints to 10 grid cells and generates a set of candidate waypoints located 2-4 meters from the robot. There are a total of 5 points, mainly distributed at the corridor entrance and an unknown corner of the living room. The instance waypoint generation module works in parallel: using the current RGB image, it detects instances such as "chair" (confidence 0.7), "coffee table" (confidence 0.8), and "sofa" (confidence 0.9) using the Grounding DINO model. For the pixel region of each detection box, its 3D point cloud is calculated by combining it with the depth image, and the 3D point closest to the robot is extracted. After deduplication (merging points with a distance <0.3m), a candidate instance waypoint set is generated. There are 3 points in total, each pointing to the aforementioned object.

[0037] Step S3: Target selection based on neural relationship inference The target waypoint selection module receives all 8 candidate points (5 leading points and 3 instance points). For each point... Calculate its visual language comprehensive score .

[0038] Distance Score Calculate the robot's current position (known) to Euclidean distance For example, if the distance to the front edge of a corridor is 3.2 meters, then... .

[0039] Unknown area score :statistics The grid status within a 0.5m radius. A leading edge point pointing to an unknown area of ​​the living room has an unknown / visited grid ratio of approximately 8. .

[0040] Consistency score This is the core. The Neural Relationship Inference (NRI) model is invoked. The model receives the instruction "blue cup" and a list of objects detected in the current frame ("chair", "coffee table", "sofa"). After graph construction (nodes: 1 text node + 3 object nodes; edges: fully connected) and two rounds of message passing, the model infers the semantic matching probability between "blue cup" and "coffee table". The probability of a match for "sofa" is 0.02, which is 0.05. Therefore, the waypoint pointing to "coffee table" has a probability of matching 0.02. ( (This is the semantic channel value at that point, assumed to be 0.6). And for all the frontier points, Finally, weights are set. , Calculate each point Calculations showed that a forward point pointing towards the bedroom and containing abundant information scored the highest (0.72) and was selected as the next navigation target.

[0041] Step S4: Real-time waypoint navigation After receiving the target waypoint coordinates, the pathpoint navigation module invokes the FAR Planner. The planner, operating at 10Hz and based on the latest occupied grid map, generates a collision-free path from the current position to the target point. The robot then begins to move, with a maximum linear velocity set to 0.4. .

[0042] Step S5: Map Update During the movement, the map scrolling update module continues to operate. Each new frame of RGB-D data is used to update the occupying raster map. Simultaneously, new observation data... Confidence value map compared to the previous moment Weighted fusion is performed, and the update formula is strictly performed according to the present invention. This ensures that the environmental model remains up-to-date, especially in corridors with changing lighting, where the semantic confidence of "blue" can be continuously revised.

[0043] Step S6: Looping Exploration and Mission Termination After the robot reaches the first target leading edge, the loop and termination judgment module is triggered, repeating steps S2 to S4. This time, the robot's field of vision enters the bedroom, and the instance detection module successfully detects the "desk" (confidence 0.85) and the "cup" on it (confidence 0.65, and color feature matches "blue"). The NRI model calculates that the matching probability between the "blue cup" and the current "cup" instance is as high as 0.88. Therefore, a new waypoint pointing to this "cup" is generated. The score far exceeded that of other points, and it was selected as the final target. The robot navigated to the vicinity of this point, triggered a stop action, and the mission successfully ended.

[0044] Verified by the above embodiments, the system of this invention can run stably on Jetson Orin NX. In this navigation task in a 300-square-meter home environment, the robot traveled a total path of 15.3 meters from the starting point to finding the target object in 128 seconds, without any collisions. Compared with the baseline method (VLFM), this invention, by introducing an NRI model for semantic relationship reasoning, avoids accidental stops at irrelevant objects such as a "coffee table," shortening the exploration path by approximately 30% and increasing the task success rate to 43%. The system's average operating frequency reaches 10Hz, meeting the real-time requirements of indoor mobile robots. Therefore, the solution provided by this invention can effectively integrate active exploration and semantic reasoning under conditions of limited computing resources, achieving accurate and efficient visual-language navigation.

[0045] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A real-time visual language navigation method based on cutting-edge exploration and neural relationship inference, characterized in that, include: (1) The robot rotates at its initial position and uses its depth camera and odometry data to construct an initial occupancy grid map and confidence value map; (2) Based on the occupied grid map, the boundary between the explored and unexplored areas is identified as the frontier area, and a set of candidate frontier waypoints is generated by breadth-first search algorithm. ; (3) Based on the images acquired by the depth camera, the pre-trained vision-language model is used to detect the instance objects existing in the current field of view, and the points closest to the robot are extracted as the candidate instance waypoint set. ; (4) Combine the semantic channel values ​​in the confidence value map to calculate the visual language comprehensive score for all waypoints in the candidate front waypoint set and the candidate instance waypoint set. And select the waypoint with the highest score as the next navigation target; (5) Based on the selected target waypoint, call the local path planner to generate a collision-free path, drive the robot to move, and continuously update the occupied grid map and confidence value map. (6) Repeat steps (2) to (5) until the robot reaches the target object or reaches the maximum number of exploration steps.

2. The real-time visual language navigation method based on cutting-edge exploration and neural relationship inference according to claim 1, characterized in that, In step (1), the confidence value map is derived from a semantic channel. With confidence channel The composition and update formula are as follows: ； ； In this context, the superscript "new" indicates the updated value, the superscript "curr" indicates the current value, and the superscript "prev" indicates the previous value. The cosine similarity score is calculated by inputting the current RGB image and the target object text prompt into a pre-trained visual-language model. The calculation formula is: ； in, For grid cells The angle between the direction and the camera's optical axis, This is the horizontal field of view of the camera.

3. The real-time visual language navigation method based on cutting-edge exploration and neural relationship inference according to claim 1, characterized in that, In step (4), the visual language comprehensive score The calculation formula is: in, To reach the waypoint with the robot The distance score is inversely proportional to the distance. For waypoints The score of the unknown region is positively correlated with the proportion of unknown grid cells in the neighborhood. The consistency score is calculated by the neural relationship inference model. and For weight hyperparameters.

4. The real-time visual language navigation method based on cutting-edge exploration and neural relationship inference according to claim 3, characterized in that, Consistency score The calculation formula is: For waypoints , ; For waypoints , ; in, waypoints The corresponding semantic channel value, This represents text instructions inferred by a neural relationship inference model. With waypoint The posterior probability that there is a semantic matching relationship between objects.

5. The real-time visual language navigation method based on cutting-edge exploration and neural relationship inference according to claim 3 or 4, characterized in that, The construction and training process of the neural relationship inference model is as follows: Graph construction steps: Given a text instruction and an RGB-D image, extract the text embedding. and visual embedding Mapped to the same d-dimensional space, forming a set of nodes. And construct a fully connected set of edges. ; Feature initialization steps: For each edge Define initial features ,in and For node features, Relative geometric deviations are accounted for in text-object edges; Message passing steps: Perform two rounds of message passing to update edge embeddings and node representations: ； ； ； Relationship inference steps: through fully connected layers Embed the final edge The mapping is to the relation type logits, and the posterior distribution is obtained by the softmax function. ; Model training steps: Using weak supervision signals based on task success or failure, minimize the loss function. To train the model, the loss function is: ； in, This is an indicator function for task success. For hyperparameters, As a prior distribution, Represents text instructions With objects The implicit variables in the relationship are specified here as type 1, which represents a text instruction. With objects Semantic matching.

6. The real-time visual language navigation method based on cutting-edge exploration and neural relationship inference according to claim 3, characterized in that, With the robot to the waypoint The distance score is inversely proportional to the distance. The formula is as follows: ； in, agent For robots and waypoints The Euclidean distance between them; with waypoints The score of the unknown region is positively correlated with the proportion of unknown grid cells in the neighborhood. The formula is as follows: ； in, This indicates that the number of unknown lattice elements is counted within the neighborhood of waypoint g. This indicates the number of visited grid cells at waypoints. Statistics within the neighborhood.

Images