A visual language navigation method based on multi-level semantic driving

Abstract

This invention relates to a multi-level semantic-driven visual language navigation method, belonging to the field of computer vision language navigation. The invention constructs a perception-planning-control end-to-end collaborative structure. The method involves: First, in the front-end perception stage, combining target-oriented semantic masks and spatial geometric features, a semantic enhancement potential field is constructed through morphological contour projection and a dynamic fusion mechanism to extract local candidate waypoints. Second, in the high-level planning stage, a spatiotemporal topology map is dynamically maintained, and a graph-aware cross-modal attention mechanism is used for global path planning. Finally, in the low-level action control stage, a multi-level escape mechanism is introduced, achieving proactive resolution of microscopic physical deadlocks through physical fault tolerance, semantic redirection based on near-field area ratio, and negative feedback topology pruning. This invention effectively overcomes the defects of blind exploration and significantly improves the navigation success rate and system robustness in complex continuous environments.

Description

Technical Field

[0001] This invention belongs to the field of computer vision language navigation, specifically relating to a visual language navigation method based on multi-level semantic driving. Background Technology

[0002] With the rapid development of artificial intelligence and embodied intelligence technologies, visual-language navigation (VLN) has gradually become an important research direction in this field. This task requires an intelligent agent to autonomously perceive the visual scene and execute step-by-step navigation decisions based on input natural language instructions in a 3D simulated or real environment, ultimately reaching the target location.

[0003] Early research on visual language navigation primarily focused on discrete environments (relying on predefined navigation topologies, where agents only need to make discrete jumps between known nodes). This setup greatly simplifies the complexity of navigation in the real physical world, making it difficult to directly deploy the models in real-world robotic systems. In contrast, Visual Language Navigation in Continuous Environments (VLN-CE) requires agents to autonomously explore in a continuous 3D physical grid without prior topologies, relying on low-level action commands (such as fixed-distance forward movement and fixed-angle rotation). This is more in line with the practical application needs of embodied intelligence.

[0004] Currently, for visual language navigation in continuous environments, the industry generally adopts a modular pipeline approach of "high-level planning - low-level control." However, this mainstream paradigm still has significant shortcomings and technical bottlenecks in deployment in real-world complex physical environments:

[0005] Firstly, in terms of perception and planning, the lack of deep semantic guidance easily leads to blind exploration. Existing methods, in order to avoid model overfitting, often blindly discard or weaken fine-grained semantic information during feature extraction, relying solely on pure depth images to predict geometric reachability. This results in the agent lacking spatial guidance in perceiving key landmarks and target objects mentioned in natural language instructions. Not only does it fail to effectively filter out irrelevant visual noise in the environment, but it also easily wanders into meaningless empty areas in a vast and disordered continuous physical space, leading to extremely low pathfinding efficiency.

[0006] Secondly, at the control end, there is a lack of proactive deadlock escape mechanisms, resulting in poor robustness to rigid body physical constraints. Continuous environments introduce realistic rigid body dynamic constraints (such as chassis collision volume limitations). Existing low-level control and obstacle avoidance mechanisms largely rely on passive trial and error after hitting obstacles. When faced with strict physical constraints that limit sliding (such as narrow corridors or dense furniture obstacle zones), even small perceptual deviations can easily cause the agent to collide with physical boundaries. Due to the lack of proactive obstacle prediction and flexible cross-level escape feedback mechanisms, the agent is prone to getting stuck in microscopic local physical deadlocks, repeatedly oscillating in the deadlock region and exhausting its action budget, rendering high-level perfect path planning a mere theoretical exercise.

[0007] Therefore, how to break the dilemma of disordered exploration in continuous environments, effectively integrate deep semantic information into local waypoint generation and global planning, and endow intelligent agents with the ability to actively predict and self-correct when encountering complex physical deadlocks are the core pain points that urgently need to be solved in the field of visual language navigation in continuous environments. Summary of the Invention

[0008] The purpose of this invention is to fill the gaps and deficiencies in the prior art and provide a visual language navigation method based on multi-level semantic driving, which can overcome the problems of insufficient semantic guidance in continuous environments and easy physical deadlock in the prior art.

[0009] To achieve the above objectives, the technical solution of the present invention is: a visual language navigation method based on multi-level semantic driving, comprising the following steps:

[0010] Step S1: The agent acquires natural language navigation instructions. Panoramic visual observation of the current location ; for the natural language navigation instructions Perform semantic parsing and feature mapping to extract a list of entities containing landmarks and navigation targets. and generate a global instruction representation. Using the entity list Guide the pre-trained segmentation network to perform visual observations Perform targeted reasoning to extract the panoramic semantic mask of the target category. ;

[0011] Step S2: The agent constructs a geometric traffic heatmap based on deep visual features. Used to assess the spatial geometric reachability of the environment; for the panoramic semantic mask The external interaction boundary of the target object is generated using a morphological contour extraction strategy and projected onto the local three-dimensional coordinate system of the agent to construct a polar coordinate semantic potential field with semantic attraction properties. Dynamic semantic activation weights are calculated using an instruction-image decision mechanism. Geometric heatmap With polar coordinate semantic potential field Weighted fusion is performed to generate a semantic enhancement potential field with local semantic highlighting. And sample from them to generate local candidate waypoints ;

[0012] Step S3: The agent maintains a dynamic topology map in real time during navigation. The local candidate waypoints As incremental nodes, they are integrated into the map; a matrix containing map node features is then used via a cross-modal interaction mechanism. Global instruction representation Perform semantic alignment to predict the optimal long-term target node. And calculate the macro-planning path to that goal. ;

[0013] Step S4: The intelligent agent uses a low-level action controller to convert the macroscopic planned path... The process is transformed into a sequence of low-level motion commands for execution. The pose changes and collision status of the agent are monitored in real time. When a physical deadlock is detected, a multi-level escape mechanism is activated in sequence, including physical fault tolerance, semantic guidance and steering based on area comparison, and negative feedback topology pruning, to achieve path correction and logical escape. The above steps are repeated until the agent reaches the target position of the navigation command.

[0014] Furthermore, step S1 includes:

[0015] Step S11: For lengths of Natural language navigation instruction sequence This invention first performs multi-dimensional feature mapping on the text to explicitly preserve its temporal context and modal information. This is then applied to natural language navigation instruction sequences. Each word in Perform multi-dimensional feature overlay to calculate initial text features. The calculation formula is as follows:

[0016]

[0017] in, For word embedding functions, Encode the position of the word. Encode text types;

[0018] Step S12: Initialize the text feature sequence Deep contextual attention interactions are performed on the input multi-layer Transformer encoder, ultimately generating a global instruction representation with cross-modal perceptual guidance capabilities. :

[0019]

[0020] Step S13: To achieve efficient directional segmentation of the downstream visual perception module and reduce computational redundancy, this invention addresses the original instructions. A heuristic text parsing mapping function was designed. This function, based on part-of-speech tagging and syntactic analysis, accurately extracts a subset of entities containing navigation landmarks and the final target from a lengthy sequence of instructions. The extraction process is defined as follows:

[0021]

[0022] in, For the target entity list, The number of key references activated in the current instruction, and satisfying the following conditions: Through this step, the agent obtains explicit semantic priors to constrain the subsequent visual segmentation range, effectively avoiding the huge computational overhead caused by the visual network performing pixel-level classification of irrelevant background pixels.

[0023] Step S14: During the navigation time step The intelligent agent obtains local panoramic visual observations of its current pose through sensors. This observation is composed of 12 horizontally equidistant single-view RGB images and depth images stitched together, and can be formally represented as:

[0024]

[0025] in, and These represent the yaw angles. RGB and depth images captured from a specific viewpoint.

[0026] Step S15: Obtaining prior instructions Then, the model initiates an efficient and goal-driven two-dimensional semantic filtering mechanism. Specifically, the model utilizes the pre-trained lightweight semantic segmentation network SegFormer-B5 to process the RGB images from various viewpoints. Perform targeted local masking inference. The model no longer performs dense segmentation across all categories, but only targets the list of target entities. Specific landmarks and target categories included Activate and output the corresponding set of panoramic semantic masks. :

[0027]

[0028] This on-demand segmentation strategy accurately reduces and purifies the high-dimensional and noisy original visual space into a customized entity distribution matrix that is highly relevant to the current instruction, laying the data foundation for the construction of the semantic enhancement potential field in subsequent steps.

[0029] Furthermore, step S2 includes:

[0030] Step S21: The agent first establishes a purely physical geometric accessibility assessment, utilizing the extracted depth features. With viewpoint orientation features Through the cross-view interaction mechanism in the waypoint predictor, a local spatial two-dimensional probability distribution based on polar coordinates, namely a geometrical transit heatmap, is output. :

[0031]

[0032] This heatmap represents a physically unobstructed open space, serving as a secure geometric basis for the subsequent fusion process.

[0033] Step S22: To avoid the agent directly colliding with the center of the object, this invention designs a donut projection strategy based on morphological contours. This strategy is applied to the input target 2D segmentation results. Define the morphological expansion nucleus The original mask is dilated to generate an expanded mask covering the object and its edges. Then, contour subtraction is used to accurately extract the object's external interactive safety boundary. :

[0034]

[0035] This operation forcibly moves the semantically high responsive area from inside the object to the periphery of the object where it can be safely docked.

[0036] Step S23: After obtaining the two-dimensional external contour, the model is combined with the original depth image. Spatial mapping is performed using a pinhole camera model. The two-dimensional pixels on the contour mask are then mapped. By back-projecting the data onto the agent's local 3D physical coordinate system, its relative depth can be calculated. With lateral offset This transforms the two-dimensional contour into a three-dimensional point cloud distribution that includes local spatial geometric relationships.

[0037] Step S24: To filter out depth noise and prevent long-distance semantic interference, a dynamic weight decay mechanism is designed for the model. The Euclidean distance between the projected points is calculated. so that its potential weight The semantic attraction decreases negatively with distance; regions that are closer together have stronger semantic appeal.

[0038]

[0039] in, To ensure a minimum weighting coefficient, The effective depth range is defined. The weights of projection points within the same region are accumulated to form a preliminary three-dimensional contour potential field.

[0040] Step S25: Map the three-dimensional contour potential energy field to the polar coordinate system matrix. To eliminate discontinuous noise, a Gaussian smoothing kernel is used to perform convolutional blurring on the matrix, and its peak values ​​are normalized to output a smooth and continuous polar coordinate semantic heatmap. :

[0041]

[0042] Step S26: The model designs a dynamic fusion mechanism based on dynamic mask compensation. This is achieved by comparing the prior text with the set of semantic categories activated in the current field of view. Calculate dynamic semantic activation weights For process landmarks With the ultimate goal Each landmark threshold compensation coefficient is assigned separately Compensation coefficient with endpoint threshold This enables the on-demand allocation of semantic interventions.

[0043] Step S27: Use the Boolean mask generated by the geometrical heatmap Spatially constrain the semantic potential field and calculate the fused semantic enhancement potential field. :

[0044]

[0045] This mechanism ensures that the semantic potential associated with the instruction is strongly highlighted on the heatmap, while accurately falling within the agent's passable area.

[0046] Step S28: On the polar coordinate heatmap The nonmaximum suppression algorithm is applied to sample and extract the previous values. The grid coordinates with the highest confidence are obtained. Using preset angle and distance resolutions, the polar coordinate indices are inversely mapped back to the agent's local two-dimensional Cartesian coordinate system, ultimately generating a grid coordinate system containing... A set of local candidate waypoints with discrete relative poses , as incremental nodes for subsequent path planning.

[0047] Furthermore, step S3 includes:

[0048] Step S31: When exploring the unknown continuous global space, the agent dynamically maintains an undirected topological map through a self-organizing mechanism. The set of nodes in the topology graph It is strictly divided into three subsets: the current node The set of visited nodes and the set of unexplored boundary nodes First, the local candidate waypoints generated in step S2 are... Transform to global coordinate system to obtain It also defines a node localization function based on Euclidean distance to calculate the new waypoint and topology map. Minimum distance between existing nodes :

[0049]

[0050] Step S32: Based on the minimum distance Combined with preset merging threshold The system executes three branch update logics on the topology map:

[0051] (1) If And the matching node belongs to the set of visited nodes. If the waypoint points to an already explored area, discard the waypoint and add a connecting edge between the current node and the matching node, adding it to the edge set. middle;

[0052] (2) If And the matching node belongs to the existing exploration boundary. Then, a multi-step exponential moving average is accumulated between the position and visual features of the boundary node to enhance feature robustness:

[0053]

[0054] in, This represents the number of times the node has been matched.

[0055] (3) If If a completely new region is discovered, it is initialized as a new boundary node and added to the graph. .

[0056] Step S33: To enrich the spatiotemporal representation of the topological nodes, the model is used for each node. Basic visual features Superimposed relative pose encoding With navigation time step coding The comprehensive node representation is obtained. :

[0057]

[0058] Step S34: The feature matrix containing all graph nodes... Global instruction representation The input is fed to a cross-modal graph converter. To explicitly inject structural priors about the physical environment, this invention introduces a graph-aware self-attention mechanism, utilizing topologically connected edges. Calculate the shortest path across all sources and construct the spatial distance matrix. And inject it as a bias term into the attention calculation:

[0059]

[0060] in, All are learnable linear projection matrices. The feature dimension is used. This mechanism assigns higher receptive field weights to nodes that are physically closer, ultimately outputting a node representation matrix that achieves cross-modal depth alignment. .

[0061] Step S35: Based on the aligned node representations, the planning module evaluates the graph using a feedforward neural network. Confidence score of each node as the final navigation target To prevent the agent from getting stuck in a redundant backtracking loop within the explored area, the model uses an indicator function. A heuristic history masking strategy is introduced to force the scores of visited nodes and the current node to be reduced to a minimum:

[0062]

[0063] Under this mask constraint, the system explores the set of unexplored boundary nodes. Among the virtual nodes representing the stopping action, the node with the highest score is selected as the global long-term goal. Finally, the system in the current topology graph The Dijkstra shortest path search algorithm is executed to calculate the macroscopic planned path composed of discrete graph nodes. :

[0064] .

[0065] Furthermore, step S4 includes:

[0066] Step S41: The agent uses the base controller to plan the path of the high-level topology. This is translated into specific low-level micro-motion commands (such as rotation and forward movement) for execution. To handle physical collisions that are easily triggered in continuous environments, the system defines a deadlock state indication function. Real-time monitoring of pose changes:

[0067]

[0068] in, Forward command This represents the actual displacement change. This is the minimum displacement threshold. Once... Continuous collision counter Accumulate.

[0069] Step S42: Design a three-level progressive escape mechanism for deadlock states of different severity:

[0070] Level 1 (Physical Fault Tolerance): When At this time, the agent temporarily suspends its original navigation intention, forces itself to select an angle (such as a 15° left or right offset) from a predefined set of trial angles based on the current orientation, rotates, and attempts to move forward one step at a time again.

[0071] Level 2 (Semantic Direction): When Level 1 fails and When in the intermediate range, the system requests visual observations from the higher-level perception layer and uses a two-dimensional segmentation network to extract the original semantic mask of the passable area. To eliminate visual noise and false security zones, morphological erosion of the nucleus was utilized. The mask is shrunken inward; then the bottom region of the eroded mask is truncated as the near-field sensing frustum. Furthermore, the near-field mask is divided into left and right sides along the central axis, and the difference ratio of the passable area between the left and right sides is calculated. :

[0072]

[0073] like ( If the area ratio threshold is preset, the agent will issue a turning action to the side with the larger area.

[0074] Level 3 (Closed-Loop Topology Pruning): If the target still cannot escape after the first two levels of intervention, it indicates that the sub-target is physically inaccessible at the microscopic level. The controller sends a strong negative feedback signal to the higher-level module, and the high-level planning module immediately disconnects the global memory graph. Current node With failed sub-target nodes Topologically connected edges between them:

[0075]

[0076] At the same time, an absolute logical mask is applied to the predicted score of the failed node. This forces the high-level planner to rerun the pathfinding algorithm based on the updated graph structure and the remaining action budget to generate a completely new alternative topology path.

[0077] The present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0078] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described above.

[0079] Compared with existing technologies, this invention has the following advantages: This invention constructs a multi-level semantic-driven navigation architecture with full-link collaboration of perception, planning, and control. By introducing a morphological donut strategy and a dynamic fusion mechanism to construct a semantically enhanced potential energy field, it effectively overcomes the shortcomings of blind exploration in traditional methods and significantly improves the accuracy of local waypoint sampling and navigation efficiency. At the same time, in response to the collision deadlock problem that is easily caused by continuous physical environments, it innovatively designs a three-level progressive escape mechanism from shallow to deep, which transforms the microscopic deadlock signal at the bottom layer into negative feedback for the global planning at the high level. This gives the agent the ability to actively avoid deadlock and self-repair to escape difficulties, thereby effectively bridging the gap between macroscopic ideal planning and microscopic real execution, and greatly enhancing the navigation success rate and system robustness of the agent in complex continuous environments. Attached Figure Description

[0080] Figure 1 This is a schematic diagram of the overall process of the present invention.

[0081] Figure 2 This is a schematic diagram illustrating the construction process of the semantic potential field in an embodiment of the present invention.

[0082] Figure 3 This is a visualization diagram of the dynamic fusion mechanism in an embodiment of the present invention.

[0083] Figure 4 This is a logic block diagram of the multi-level escape mechanism of the underlying action controller in an embodiment of the present invention.

[0084] Figure 5 This is a visualization of the semantic enhancement potential field generation process implemented in this invention. Detailed Implementation

[0085] The technical solution of the present invention will be specifically described below with reference to the accompanying drawings. It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0086] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the invention; as used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise.

[0087] This invention provides a visual language navigation method based on multi-level semantic driving, comprising the following steps:

[0088] Step S1: The agent acquires natural language navigation instructions. Panoramic visual observation of the current location ; for the natural language navigation instructions Perform semantic parsing and feature mapping to extract a list of entities containing landmarks and navigation targets. and generate a global instruction representation. Using the entity list Guide the pre-trained segmentation network to perform visual observations Perform targeted reasoning to extract the panoramic semantic mask of the target category. ;

[0089] Step S2: The agent constructs a geometric traffic heatmap based on deep visual features. Used to assess the spatial geometric reachability of the environment; for the panoramic semantic mask The external interaction boundary of the target object is generated using a morphological contour extraction strategy and projected onto the local three-dimensional coordinate system of the agent to construct a polar coordinate semantic potential field with semantic attraction properties. Dynamic semantic activation weights are calculated using an instruction-image decision mechanism. Geometric heatmap With polar coordinate semantic potential field Weighted fusion is performed to generate a semantic enhancement potential field with local semantic highlighting. And sample from them to generate local candidate waypoints ;

[0090] Step S3: The agent maintains a dynamic topology map in real time during navigation. The local candidate waypoints As incremental nodes, they are integrated into the map; a matrix containing map node features is then used via a cross-modal interaction mechanism. Global instruction representation Perform semantic alignment to predict the optimal long-term target node. And calculate the macro-planning path to that goal. ;

[0091] Step S4: The intelligent agent uses a low-level action controller to convert the macroscopic planned path... The process is transformed into a sequence of low-level motion commands for execution. The pose changes and collision status of the agent are monitored in real time. When a physical deadlock is detected, a multi-level escape mechanism is activated in sequence, including physical fault tolerance, semantic guidance and steering based on area comparison, and negative feedback topology pruning, to achieve path correction and logical escape. The above steps are repeated until the agent reaches the target position of the navigation command.

[0092] The following is a detailed implementation process of the present invention.

[0093] like Figure 1 As shown, the present invention provides an intelligent agent visual language navigation method based on multi-level semantic driving, comprising the following steps:

[0094] Step S1: Obtain natural language navigation instructions and panoramic visual observations of the current location; perform semantic parsing and feature mapping on the navigation instructions, extract an entity list containing landmarks and navigation targets, and generate a global instruction representation; use the entity list to guide a pre-trained segmentation network to perform directional reasoning on the visual observations and extract panoramic semantic masks of target categories.

[0095] Step S2: The agent constructs a geometric accessibility heatmap based on deep visual features to assess the spatial geometric accessibility of the environment; for the semantic mask, the agent uses a morphological contour extraction strategy (i.e., the "donut" strategy) to extract the external interaction safety boundary of the target object and projects it into the agent's local three-dimensional coordinate system to construct a polar coordinate semantic potential field with semantic gravity characteristics; the agent calculates dynamic semantic activation weights through an instruction-image decision mechanism, and performs weighted fusion of the geometric accessibility heatmap and the polar coordinate semantic potential field to generate a semantic enhancement potential field with local semantic highlights, and samples from it to generate local candidate waypoints.

[0096] Step S3: During navigation, the agent maintains a dynamic undirected topology map in real time, integrates the local candidate waypoints as incremental nodes into the map, and uses a graph-aware cross-modal interaction mechanism to perform semantic depth alignment between the matrix containing map node features and the global command representation, predicts the optimal long-term target node, and calculates the macroscopic planned path to the target.

[0097] Step S4: The agent uses a low-level motion controller to convert the macroscopic planned path into a sequence of low-level motion instructions for execution; it monitors the pose changes and continuous collision status of the agent in real time, and when a physical deadlock is detected, it sequentially activates a multi-level escape mechanism including physical fault tolerance, semantic guidance and turning based on near-field passable area comparison, and closed-loop negative feedback topology pruning to achieve active path correction and logical escape; the above steps are repeated until the agent reaches the target position described in the navigation instructions.

[0098] Furthermore, in one embodiment of the present invention, as Figure 1 As shown, the agent makes decisions at each navigation time step. First, it receives global natural language commands and the current panoramic visual observation.

[0099] Step S11: For the navigation instruction sequence Initial text features are calculated by superimposing word embeddings, positional encoding, and text type encoding. :

[0100]

[0101] This is then input into a multi-layer Transformer encoder to interactively generate a global instruction representation:

[0102]

[0103] Step S12: Utilize heuristic text parsing mapping functions Accurately extract a list of entities containing landmarks and navigation targets from the instruction sequence. :

[0104]

[0105] Step S13: The agent acquires local panoramic visual observations from 12 perspectives. Using the pre-trained semantic segmentation network SegFormer-B5, only targeting... The specified categories are used to perform directional mask inference on RGB images, outputting a set of panoramic semantic masks that are highly correlated with the instructions. .

[0106] In one embodiment of the present invention, such as Figure 2 and Figure 3 As shown, the system constructs and dynamically merges heterogeneous potential energy fields to guide the generation of local waypoints.

[0107] Step S21: Utilize a pre-trained waypoint predictor to evaluate spatial geometrical accessibility based on depth features and output a geometrical accessibility heatmap. :

[0108]

[0109] Step S22: As Figure 2 As shown, the two-dimensional segmentation result of the target Utilizing morphologically expanded nuclei Subtract the original mask from the execution contour to extract the external interactive safety boundary of the target object. :

[0110]

[0111] This operation shifts semantic gravity from the interior of an object to its edges, preventing the agent from colliding with the object's geometric center.

[0112] Step S23: Combine the original depth image , two-dimensional contour pixels Back projection to three-dimensional physical coordinates Subsequently, the potential energy weights were calculated using a dynamic decay mechanism. :

[0113]

[0114] Step S24: Map the three-dimensional potential energy field to the polar coordinate system and perform Gaussian smoothing normalization to output the semantic potential energy field in the polar coordinate system. .like Figure 2 As shown, this process transforms semantic features into a task-oriented gravitational distribution.

[0115] Step S25: As Figure 3 As shown, the instruction-image decision mechanism is used to calculate the dynamic semantic activation weights. (Landmark weight) Weight of the endpoint ), and with Perform grid-level fusion to generate semantically enhanced potential energy fields :

[0116]

[0117] Step S26: In The nonmaximum suppression algorithm is applied to extract... A set of high-confidence grid coordinates is generated and mapped back to the Cartesian coordinate system, ultimately producing a set of local candidate waypoints. .

[0118] In one embodiment of the present invention, such as Figure 1 As shown, the generated waypoints are integrated into the continuously evolving topology map. In the middle. Step S3 also includes the following:

[0119] Step S31: The agent dynamically maintains the topology map. Calculate the minimum distance between the new waypoint and existing nodes. If it is less than the threshold Then perform node clustering and feature moving average accumulation; if it exceeds the threshold... Then it is initialized as a new boundary node and added to the graph.

[0120] Step S32: Utilize a graph-aware self-attention mechanism to perform deep alignment between the global graph structure and language instruction cues, and inject a spatial distance matrix. As a priori physical environment:

[0121]

[0122] Step S33: Evaluate map node scores and implement the history masking strategy. Predict the optimal long-term target node And calculate the macro-planning path. .

[0123] In one embodiment of the present invention, such as Figure 4 As shown, the controller is responsible for performing local transfers and handling physical deadlocks.

[0124] Step S41: The agent uses the base controller to plan the path of the high-level topology. This is translated into specific low-level micro-motion commands (such as rotation and forward movement) for execution. To handle physical collisions that are easily triggered in continuous environments, the system defines a deadlock state indication function. Real-time monitoring of pose changes:

[0125]

[0126] in, Forward command This represents the actual displacement change. This is the minimum displacement threshold. Once... Continuous collision counter Accumulate.

[0127] Step S42: Design a three-level progressive escape mechanism for deadlock states of different severity:

[0128] (1) Level 1 (Physical fault tolerance): When At this time, the agent temporarily suspends its original navigation intention, forces itself to select an angle (such as a 15° left or right offset) from a predefined set of trial angles based on the current orientation, rotates, and attempts to move forward one step at a time again.

[0129] (2) Second level (semantic redirection): when the first level fails and When in the intermediate range, the system requests visual observations from the higher-level perception layer and uses a two-dimensional segmentation network to extract the original semantic mask of the passable area. To eliminate visual noise and false security zones, morphological erosion of the nucleus was utilized. Shrink the mask inward: Subsequently, the bottom region of the eroded mask was extracted as the near-field perception frustum. Furthermore, the near-field mask is divided into left and right sides along the central axis, and the difference ratio of the passable area between the left and right sides is calculated. :

[0130]

[0131] like ( If the area ratio threshold is preset, the agent will issue a turning action to the side with the larger area.

[0132] (3) Third level (closed-loop topology pruning): If the target still cannot escape after the first two levels of intervention, it indicates that the sub-target is physically inaccessible. The controller sends a strong negative feedback signal to the higher level, and the high-level planning module immediately cuts off the global memory graph. Current node With failed sub-target nodes Topologically connected edges between them:

[0133]

[0134] At the same time, an absolute logical mask is applied to the predicted score of the failed node. This forces the high-level planner to rerun the pathfinding algorithm based on the updated graph structure and the remaining action budget to generate a completely new alternative topology path.

[0135] In addition to the above, the present invention also includes the following embodiments:

[0136] To verify the effectiveness of the aforementioned key modules in the overall system, this invention conducts ablation experiments on the RxR-CE dataset, a continuous environment visual-language navigation dataset with extremely stringent physical constraints. Semantic heatmaps, dynamic fusion mechanisms, and multi-level escape mechanisms are removed respectively, and the changes in navigation performance are observed. Under the strict constraints of RxR-CE, deadlock and displacement failure occur once the agent experiences a geometric collision with the environment. SDTW (Success Rate Penalized Dynamic Time Warping) is the most stringent comprehensive metric for measuring the robustness of low-level obstacle avoidance and the consistency of high-level planning. The experimental results are shown in Table 1, and this invention achieves the best performance.

[0137] Table 1 Experimental Results

[0138]

[0139] In addition, the following are combined with the appendix Figure 5 This paper provides a specific embodiment illustrating the visualization evolution process of the navigation heatmap in this invention. This embodiment visually demonstrates how the model reshapes the probability distribution of the action space under the guidance of specific entity commands. For example... Figure 5As shown in the blue geometric heatmap in the second column, traditional pure geometric prediction networks, lacking explicit task semantic guidance, rely solely on free-space features extracted from depth images for their attention distribution. Therefore, high-probability local waypoints are uniformly and divergently projected onto the open area directly ahead. While this strategy ensures physical collision-free operation, it completely ignores the spatial orientation required by the command, easily causing the agent to deviate from the navigation axis and enter meaningless empty areas. When the system introduces the multimodal semantic intervention described in this invention, such as... Figure 5 As shown in the red semantic heatmap in the third column, thanks to the donut strategy and polar coordinate mapping, the model accurately generates a highly concentrated semantic gravity peak at a position 315° to the left and in front (i.e., the actual spatial coordinates of the sink). This demonstrates that the semantic potential field can filter out environmental noise and accurately lock onto the task target. Figure 5 As shown in the purple semantic enhancement potential field on the far right, through the dynamic fusion mechanism, the geometric probability density, which was originally blindly spread out in front, is significantly pulled and shifted towards the direction of the target object by a strong semantic attraction. This dynamic superposition mechanism not only retains the basic physical obstacle avoidance capability, but also gives the sampling waypoints a strong mission orientation.

[0140] This embodiment powerfully demonstrates through visual comparison that a precise semantically enhanced potential field completely corrects the blind exploration tendency of a purely geometric model. It enables the agent's local waypoint sampling to approach the target object via the most efficient physical path, thus laying a solid foundation for high-success-rate continuous navigation at the microscopic level.

[0141] The present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0142] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described above.

[0143] The above are preferred embodiments of the present invention. Any changes made to the technical solution of the present invention that do not exceed the scope of the technical solution of the present invention shall fall within the protection scope of the present invention.

Claims

1. A visual language navigation method based on multi-level semantic driving, characterized in that, Includes the following steps: Step S1: The agent acquires natural language navigation instructions. Panoramic visual observation of the current location ; For the natural language navigation instructions Perform semantic parsing and feature mapping to extract a list of entities containing landmarks and navigation targets. and generate a global instruction representation. Using the entity list Guide the pre-trained segmentation network to perform visual observations Perform targeted reasoning to extract the panoramic semantic mask of the target category. ; Step S2: The agent constructs a geometric traffic heatmap based on deep visual features. It is used to assess the spatial geometric accessibility of an environment; For the aforementioned panoramic semantic mask The external interaction boundary of the target object is generated using a morphological contour extraction strategy and projected onto the local three-dimensional coordinate system of the agent to construct a polar coordinate semantic potential field with semantic attraction properties. ; Dynamic semantic activation weights are calculated using an instruction-image decision mechanism. Geometric heatmap With polar coordinate semantic potential field Weighted fusion is performed to generate a semantic enhancement potential field with local semantic highlighting. And sample from them to generate local candidate waypoints ; Step S3: The agent maintains a dynamic topology map in real time during navigation. The local candidate waypoints As incremental nodes, they are integrated into the map; a matrix containing map node features is then used via a cross-modal interaction mechanism. Global instruction representation Perform semantic alignment to predict the optimal long-term target node. And calculate the macro-planning path to that goal. ; Step S4: The intelligent agent uses a low-level action controller to convert the macroscopic planned path... The process is transformed into a sequence of low-level motion commands for execution. The pose changes and collision status of the agent are monitored in real time. When a physical deadlock is detected, a multi-level escape mechanism is activated in sequence, including physical fault tolerance, semantic guidance and steering based on area comparison, and negative feedback topology pruning, to achieve path correction and logical escape. The above steps are repeated until the agent reaches the target position of the navigation command.

2. The visual language navigation method based on multi-level semantic driving according to claim 1, characterized in that, In step S1, the specific process of feature mapping and entity extraction for natural language navigation instructions is as follows: Step S11: Process the natural language navigation instruction sequence Each word in Perform multi-dimensional feature overlay to calculate initial text features. The calculation formula is as follows: in, For word embedding functions, Encode the position of the word. Encode text types; Step S12: Initialize the text feature sequence Input a multi-layer Transformer encoder to generate a global instruction representation with deep contextual semantic interaction. : Step S13: Utilize heuristic text parsing mapping functions Based on part-of-speech tagging and syntactic analysis, from natural language navigation instruction sequences In the process of separating navigation landmarks from target entity subsets : in, The number of key references that are activated, and L is the length of the instruction sequence.

3. The visual language navigation method based on multi-level semantic driving according to claim 1, characterized in that, In step S2, a polar coordinate semantic potential energy field with semantic gravity properties is constructed. The specific process includes: Step S21: Two-dimensional segmentation result of the target entity Define morphological expansion nucleus Extract the external interactive security boundary contour mask of the object Its expression is: Step S22: Combine depth images Using camera intrinsics, a pinhole camera model is used to represent two-dimensional pixels. Back projection to the agent's local three-dimensional physical coordinates : in, The center point of the image. The equivalent focal length is derived from the field of view. Step S23: Calculate the Euclidean distance between the projection point and the agent. The potential energy weight of each effective projection point is calculated through a dynamic weight decay mechanism. : in, To ensure a minimum weighting coefficient, For the effective depth range, It is a smoothing constant; Step S24: Map the projection points to the polar coordinate system matrix. In the middle, and using a 3*3 Gaussian smoothing kernel Convolution and peak normalization are performed to obtain the final polar coordinate semantic potential field. : 。 4. The visual language navigation method based on multi-level semantic driving according to claim 1, characterized in that, In step S2, a semantic enhancement potential field with local semantic highlighting is generated. The specific process is as follows: Step S25: Calculate the dynamic semantic activation weights by comparing text priors with visual observations. The calculation process is as follows: in, The endpoint threshold compensation coefficient, This is the landmark threshold compensation coefficient, and ; This is the set of semantic categories that were successfully activated during visual observation. The final target entity in the navigation instructions. This is the set of process landmarks for the current navigation phase. Step S26: Utilize the geometrical heatmap The generated Boolean mask Spatially constrain the semantic potential field and perform grid-level weighted fusion in conjunction with dynamic semantic activation weights to ensure that the semantic potential field distribution strictly falls within the physically traversable region of the agent: in, These are the angle and the distance, respectively.

5. A visual language navigation method based on multi-level semantic driving according to claim 1, characterized in that, In step S3, the dynamic topology map is maintained in real time. The specific process includes: Step S31: Select local candidate waypoints Transform to global coordinate system to obtain And calculate its relationship with existing nodes in the topology graph. The minimum Euclidean distance: in, It is the set of boundary nodes at the current moment. It is the set of connected edges at the current moment; Step S32, if Less than the preset merging threshold If the corresponding node belongs to the set of visited nodes, then discard the waypoint and add a connecting edge between the current node and the corresponding node; Step S33, if If the corresponding node belongs to the set of boundary nodes, then the features of that node are accumulated using an exponential moving average to enhance feature robustness. in, This represents the number of times the node has been matched. The boundary node features before the update. The updated boundary node features; for Features; Step S34, if If so, it is initialized as a new boundary node and added to the graph.

6. The visual language navigation method based on multi-level semantic driving according to claim 1, characterized in that, In step S3, the matrix containing map node features is used to... Global instruction representation Perform semantic alignment to predict the optimal long-term target node. And calculate the macro-planning path to that goal. The specific process is as follows: Step S35: Utilize graph-aware cross-modal self-attention mechanism node feature matrix Perform feature aggregation and cross-modal alignment: in, To explicitly inject structural priors into the physical environment by utilizing the spatial distance matrix calculated using topologically connected edges; All are learnable linear projection matrices, where d is the feature dimension; Step S36: Evaluate node confidence scores using a feedforward neural network. A historical masking strategy is introduced, and the score is calculated after masking. : in, For indicator functions, The set of visited nodes. The current node is selected; the node with the highest score among the unexplored boundary nodes is chosen as the global long-term goal. And calculate the macro-planning path. .

7. The visual language navigation method based on multi-level semantic driving according to claim 1, characterized in that, In step S4, the multi-level escape mechanism includes a first-level escape mechanism and a second-level escape mechanism. The specific process of deadlock detection and the first-level and second-level escape mechanisms is as follows: Step S41: Monitor the pose changes of the intelligent agent in real time and define a deadlock state indication function. : in, For forward movement, This is a constant identifier for the forward motion. The minimum displacement threshold, Let be the actual spatial coordinates of the agent at time step t. The actual spatial coordinates of the agent at time step t−1; once Then the continuous collision counter Accumulation; Step S42, when Within the first threshold range, the agent is forced to select a trial angle to rotate based on the current orientation and attempt to move forward one step at a time again; Step S43, when Within the second threshold range, the visual mask directly in front is extracted and morphological erosion kernels are used to shrink inward, truncating the near-field perceptual cone; further, the difference ratio of the passable area on the left and right sides is calculated. : in, and These represent the total number of pixels that can pass through on the left and right sides, respectively; if If the value exceeds a preset threshold, the agent issues a turning action to the side with the larger area. A very small constant to prevent overflow when divided by zero.

8. A visual language navigation method based on multi-level semantic driving according to claim 7, characterized in that, The multi-stage escape mechanism also includes a third-stage escape mechanism, the specific process of which is as follows: Step S44: When the number of consecutive collisions exceeds the second threshold range or the second-level escape fails, disconnect the current node in the global memory graph. With problem sub-goal nodes Topologically connected edges between them: It is a dynamic topology map The set of connected edges at the current moment; Step S45: Apply an absolute logical mask to the network prediction confidence score of the failed sub-target nodes to prevent the planner from repeatedly getting stuck in deadlock. It also forces the advanced topology path planning module to rerun the pathfinding algorithm based on the updated graph structure, generating entirely new alternative topology paths.

9. An electronic device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 8.

Images