Embodied visual language navigation method based on multi-layer dynamic memory network

Abstract

The embodiment of the application provides a kind of embodied visual language navigation method based on multi-layer dynamic memory network, comprising: in response to embodied intelligent agent reaches the current position in target scene in current time step, obtains natural language instruction and environmental visual data;Build the target node associated with position coordinate, visual embedding feature, time step and access state;Based on the node updates abstract meta-graph and fixed size node memory queue, abstract meta-graph is expressed by adaptive clustering, and the latest access node is maintained in memory queue to realize constant level inference complexity;Local planner outputs node level navigation action probability distribution based on memory queue, and meta-planner outputs region level strategy preference distribution based on abstract meta-graph, and local sensor outputs immediate termination action probability distribution based on visual data;Fusion generates target action of next time step and executes, under given computing power budget and real-time constraint, improve navigation success rate.

Description

Technical Field

[0001] This application relates to the fields of embodied intelligence and visual language navigation technology, and in particular to an embodied visual language navigation method based on a multi-layer dynamic memory network. Background Technology

[0002] Embodied Artificial Intelligence refers to the ability of artificial intelligence systems to understand and execute complex tasks through continuous perception, action, and learning in physical or simulated environments. The core of embodied intelligence lies in the real-time interaction and experience accumulation between the AI ​​system and its environment.

[0003] Embodied Vision-and-Language Navigation, a core technology of embodied intelligence, requires embodied intelligent agents to perform spatial reasoning and path planning in unknown real or virtual 3D environments, relying solely on their own visual observations, based on natural language instructions, in order to achieve goal-oriented movement. An embodied intelligent agent refers to an autonomous intelligent entity possessing perception, decision-making, and movement capabilities. It can acquire environmental information through sensors, generate navigation strategies based on internal models, and achieve spatial displacement through action execution modules.

[0004] However, existing embodied visual language navigation methods struggle to simultaneously meet the demands of high-precision navigation and real-time response within limited computing resources in practical applications. Specifically, methods supporting long-distance backtracking and global path replanning typically require maintaining complete access trajectories and performing global correlation calculations, resulting in high inference latency and large memory consumption, making them unsuitable for embedded platforms or low-latency human-computer interaction scenarios. Methods that limit the scope of historical memory to control computational overhead, lacking a long-term representation of the overall topology of the environment, cannot effectively handle complex spatial layouts such as circular corridors and multi-branch intersections, and are prone to getting trapped in local optima or prematurely terminating at incorrect locations.

[0005] Therefore, there is an urgent need for a new embodied visual language navigation method that can dynamically construct and utilize lightweight but semantically complete environmental memory while ensuring bounded reasoning complexity, so as to achieve high-precision and high-robustness navigation behavior under given computing power budget and real-time constraints. Summary of the Invention

[0006] This application provides an embodied visual language navigation method based on a multi-layer dynamic memory network, which can achieve high-precision and high-robustness navigation behavior with low inference complexity under given computing power budget and real-time constraints.

[0007] In a first aspect, embodiments of this application provide an embodied visual language navigation method based on a multi-layer dynamic memory network, applied to an embodied intelligent agent, wherein the embodied intelligent agent is used to perform an embodied visual language navigation task in a target scene, and the method includes: In response to the embodied intelligent agent reaching its current position in the target scene at the current time step, it acquires the natural language instruction to be executed and the environmental visual data collected at the current position; Construct a target node corresponding to the current location. The target node is associated with the location coordinates of the current location, the visual embedding features corresponding to the environmental visual data, the current time step, and the node access status. Based on the target node, update the abstract metagraph and node memory queue corresponding to the target scene. The abstract metagraph is used to cluster the nodes visited by the embodied agent according to the region. The node memory queue is used to store a preset number of nodes recently visited by the embodied agent. Based on the natural language instructions and the memory node queue, a node-level navigation action probability distribution is output through a local planner; based on the natural language instructions and the abstract metagraph, a region-level strategic preference distribution is output through a metaplanner; based on the environmental visual data, an instant termination action probability distribution is output through a local perceptron. By integrating the node-level navigation action probability distribution, the region-level strategic preference distribution, and the instant termination action probability distribution, the target action to be executed by the embodied agent in the next time step is obtained, and the target action is executed in the next time step to complete the embodied visual language navigation task.

[0008] Secondly, embodiments of this application provide an embodied visual language navigation device based on a multi-layer dynamic memory network, applied to an embodied intelligent agent, the embodied intelligent agent being used to perform an embodied visual language navigation task in a target scene, the device comprising: The acquisition module is used to acquire the natural language instruction to be executed and the environmental visual data collected at the current location in the target scene in response to the embodied intelligent agent reaching the current position in the current time step. The construction module is used to construct the target node corresponding to the current position. The target node is associated with the position coordinates of the current position, the visual embedding features corresponding to the environmental visual data, the current time step, and the node access status. The processing module is used to update the abstract metagraph and node memory queue corresponding to the target scene based on the target node. The abstract metagraph is used to cluster the nodes visited by the embodied agent according to regions, and the node memory queue is used to store a preset number of nodes recently visited by the embodied agent. Based on the natural language instructions and the memory node queue, the module outputs a node-level navigation action probability distribution through a local planner. Based on the natural language instructions and the abstract metagraph, the module outputs a region-level strategic preference distribution through a meta planner. Based on the environmental visual data, the module outputs an instant termination action probability distribution through a local perceptron. The module fuses the node-level navigation action probability distribution, the region-level strategic preference distribution, and the instant termination action probability distribution to obtain the target action to be executed by the embodied agent in the next time step, and executes the target action in the next time step to complete the embodied visual language navigation task.

[0009] Thirdly, embodiments of this application provide an electronic device, including: a memory, a processor, and a communication interface; wherein, the memory stores a computer program, and when the computer program is executed by the processor, the processor can at least implement the embodied visual language navigation method based on a multi-layer dynamic memory network as described in the first aspect.

[0010] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor of an electronic device, enables the processor to at least implement the embodied visual language navigation method based on a multilayer dynamic memory network as described in the first aspect.

[0011] Fifthly, embodiments of this application provide a computer program product, including: a computer program or instructions that, when executed by a processor of an electronic device, enable the processor to at least implement the embodied visual language navigation method based on a multilayer dynamic memory network as described in the first aspect.

[0012] The solution provided in this application provides a complete closed loop of memory network construction, inference execution, action feedback, and memory network updating. By introducing a fixed-size working memory queue, the computational complexity of the local planner is limited to the constant level. An abstract metagraph is introduced by automatically adjusting clustering parameters according to different environments (e.g., narrow corridors vs. open halls) to generate reasonable region representations. This significantly reduces the number of nodes that the global planner needs to process, thus reducing the overall long-term navigation complexity from O(|V1|V2|V3|V4|V5|V6|V7|V8|V9|V1|V2 ... t | 2 ) reduced to O(|R tThis addresses the inherent high latency inherent in high-precision technologies. Furthermore, in this scheme, the abstract metagraph preserves the global topology of the environment. When the agent needs to return from a dead end or traverse a large area, the metaplanner can provide the correct strategic direction based on region connectivity, enhancing long-distance backtracking capabilities in complex environments and resolving the problem of local methods easily getting trapped in local optima. Based on this, the dual-planner fusion mechanism in this scheme ensures reliance on local vision during precise operations and on global memory during periods of confusion, significantly improving navigation success rates. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 A flowchart illustrating an embodied visual language navigation method based on a multi-layer dynamic memory network, provided as an embodiment of this application; Figure 2 A flowchart illustrating an abstract metagraph update method provided in this application embodiment; Figure 3 A schematic diagram of a dual-planner collaborative decision-making architecture provided in an embodiment of this application; Figure 4 A flowchart of a target action prediction method provided for the implementation of this application; Figure 5 A schematic diagram of the structure of an embodied visual language navigation device based on a multi-layer dynamic memory network provided in this application embodiment; Figure 6 To and Figure 5 The illustrated embodiment provides a schematic diagram of the electronic device corresponding to the embodied visual language navigation device based on a multi-layer dynamic memory network. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0016] It should be noted that, in the cases involving user information in the embodiments of this application, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse. In addition, the various models involved in this application (including but not limited to language models or large models) comply with relevant laws and standards.

[0017] Furthermore, the timing of the steps in the following method embodiments is merely an example and not a strict limitation.

[0018] In related technologies, embodied visual language navigation faces a contradiction between the need for high-precision navigation and the constraints of limited computing power. For example, in embodied visual language navigation schemes based on global topology graphs or global attention mechanisms, although global reasoning and path optimization can be performed on the entire map or over long time periods, exhibiting strong planning and backtracking capabilities, their computational complexity typically increases quadratically with the number of accessed nodes N, reaching O(N²). In long-term, long-distance embodied visual language navigation tasks, this can easily lead to excessive inference latency and high memory overhead, making it difficult to meet the requirements of real-time decision-making and embedded deployment.

[0019] For example, navigation schemes that rely solely on local history or streaming windows, while theoretically capable of operating at constant or linear computational complexity in computationally limited scenarios, lack explicit global topological memory and multi-level abstraction structures, making it impossible to backtrack and replan long-distance paths. Consequently, they suffer from insufficient adaptability when facing challenging environments such as complex intersections, long corridors, and loop structures, easily leading to local optima, redundant paths, or inaccurate termination.

[0020] For example, in hybrid schemes that directly combine local and global graph methods, the lack of a systematic hierarchical memory architecture and adaptive abstraction mechanism makes it impossible to simultaneously guarantee bounded inference cost and global topological inference capability. This results in limited scalability as navigation length and environment size increase, making it difficult to operate stably in scenarios such as long-term embodied intelligence tasks and complex environment migrations.

[0021] To address at least one technical problem in the prior art, this application provides an embodied visual language navigation scheme based on a multi-layer dynamic memory network. This scheme enables embodied intelligent agents to retain global topological reasoning and strategic planning capabilities when executing natural language navigation commands in complex 3D scenes. It also effectively reduces the reasoning complexity from the quadratic complexity of traditional global graph methods to a scalable logarithmic linear complexity. Under given computing power budget and real-time constraints, it still achieves high-precision and high-robustness navigation behavior.

[0022] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0023] Figure 1 This document provides a flowchart of an embodied visual language navigation method based on a multi-layer dynamic memory network, applicable to an embodied intelligent agent. The embodied intelligent agent performs an embodied visual language navigation task in a target scene. Figure 1 As shown, it may include the following steps: 101. In response to the embodied intelligent agent arriving at its current position in the target scene at the current time step, acquire the natural language instructions to be executed and the environmental visual data collected at the current position.

[0024] 102. Construct the target node corresponding to the current location. The target node is associated with the location coordinates of the current location, the visual embedding features corresponding to the environmental visual data, the current time step, and the node access status.

[0025] 103. Based on the target node, update the abstract metagraph and node memory queue corresponding to the target scene. The abstract metagraph is used to cluster the nodes visited by the embodied agent according to the region. The node memory queue is used to store the preset number of nodes recently visited by the embodied agent.

[0026] 104. Based on natural language instructions and memory node queues, output node-level navigation action probability distribution through local planner; based on natural language instructions and abstract metagraph, output region-level strategic preference distribution through meta planner; based on environmental visual data, output instant termination action probability distribution through local perceptron.

[0027] 105. By integrating the node-level navigation action probability distribution, the regional-level strategic preference distribution, and the instant termination action probability distribution, the target action to be executed by the embodied agent in the next time step is obtained, and the target action is executed in the next time step to complete the embodied visual language navigation task.

[0028] In this embodiment of the application, an embodied intelligent agent refers to an autonomous intelligent entity that possesses perception, decision-making, and movement capabilities. It can acquire environmental information through sensors, generate navigation strategies based on internal models, and achieve spatial displacement through an action execution module.

[0029] In practical applications, embodied intelligent agents can be categorized into different types based on the deployment environment and task requirements. For example, in indoor environments, embodied intelligent agents may include service robots, medical assistive robots, or automated guided vehicles for warehousing and logistics; in outdoor environments, embodied intelligent agents may include driverless vehicles, drones, or mobile robot inspection systems.

[0030] Taking service robots as an example, they are equipped with multimodal sensors such as cameras, lidar, and RGB-D cameras, which can acquire visual observation data of the indoor environment in real time. They can parse user commands (such as: go to the kitchen to get a cup) through natural language processing models, and perform path planning in combination with semantic maps. During navigation, they can perform spatial displacement actions through motion execution modules such as wheeled chassis to achieve target-oriented movement.

[0031] Similarly, medical assistive robots can move autonomously between wards under the voice commands of medical staff, completing tasks such as drug delivery and patient care; automated guided vehicles can autonomously navigate to designated storage locations in warehouses according to dispatch instructions to perform goods retrieval and placement operations; driverless vehicles can acquire road environment information through onboard cameras and radar, generate a global path based on natural language commands (such as: go to the nearest gas station), and continuously update local trajectories during driving to achieve highly robust navigation; drones can be used for tasks such as outdoor patrols and post-disaster search and rescue. They acquire three-dimensional environmental information through onboard vision systems, complete flight path planning in combination with voice commands, and perform hovering, ascent, and turning actions through flight control systems to adapt to complex terrain and dynamic obstacles.

[0032] It should be noted that the embodied visual language navigation method based on a multi-layer dynamic memory network provided in this application embodiment can be applied to various embodied intelligent agents, and is not limited to the examples mentioned above.

[0033] The embodied visual language navigation method based on a multi-layer dynamic memory network provided in this application includes a three-layer memory network: a global graph, an abstract metagraph, and a node memory queue. The global graph records the complete historical record of nodes visited by the embodied agent in a directed graph manner, i.e., all nodes visited by the embodied agent, supporting long-distance backtracking and global path analysis. The abstract metagraph is used to cluster the nodes visited by the embodied agent according to regions. The node memory queue stores a preset number of recently visited nodes by the embodied agent.

[0034] This application embodiment, through a three-layered memory network, achieves high-precision and highly robust navigation behavior with low inference complexity under given computing power budget and real-time constraints. The following will combine... Figure 1The illustrated embodiment further illustrates the application of a three-layer memory network in embodied visual language navigation.

[0035] 101. In response to the embodied intelligent agent arriving at its current position in the target scene at the current time step, acquire the natural language instructions to be executed and the environmental visual data collected at the current position.

[0036] It is understood that the current time step can be the starting time step in the embodied visual language navigation task, or it can be an intermediate time step in the embodied visual language navigation task. Correspondingly, at the current time step, the embodied agent may be directly placed at its current position in the target scene, or it may reach its current position in the target scene by performing an action. The embodiments of this application do not limit the specific way in which the embodied agent reaches its current position.

[0037] Natural language instructions refer to text instructions used to guide an embodied intelligent agent in embodied visual language navigation tasks in a target scene (such as a three-dimensional physical environment, a three-dimensional virtual environment, etc.). These include, but are not limited to: target instructions pointing to specific objects or areas, such as {go to the kitchen to get a cup}, {go to the sofa in the living room}, etc.; instructions containing relative directions and path descriptions, such as {turn left first, then walk straight to the bedroom door}, {walk along the corridor until you see the red door}, etc.; and complex instructions with conditional triggers, such as {if you see a blue vase, stop next to it}, {find the refrigerator and open it}, etc.

[0038] Optionally, natural language instructions may include spatial relation words (e.g., front, left, beside), action verbs (e.g., walk, stop, take), and the name of the target object (e.g., cup, door). The embodied agent can parse the natural language instructions into navigation intentions through the language understanding module.

[0039] The embodied intelligent agent is equipped with a positioning device and a visual information acquisition device. Based on this, when the embodied intelligent agent arrives at the current location, it can obtain the location coordinates (e.g., three-dimensional coordinates) of the current location through the positioning device and collect the environmental visual data that can be observed at the current location through the visual information acquisition device.

[0040] 102. Construct the target node corresponding to the current location. The target node is associated with the location coordinates of the current location, the visual embedding features corresponding to the environmental visual data, the current time step, and the node access status.

[0041] It's understandable that when the embodied agent reaches its current location, it's equivalent to visiting a new location, and the corresponding target node for that location is also a new node. Based on this, the target node corresponding to the current location can be denoted as... , It can be represented using the form of a quadruple: = ( , , t, 1). Among them, The three-dimensional position coordinates of the current position. The visual embedding features are extracted from the environmental visual data by the visual encoder. t represents the access timestamp corresponding to the current position, τ is the current time step t, and 1 indicates that the access status of the node corresponding to the current position is S, which is visited (if S is 0, it means that the access status of the node is not visited).

[0042] In an optional embodiment, during the process of the embodied intelligent agent performing the embodied intelligent navigation task, in response to the construction of a new target node, the global graph corresponding to the target scene is further updated according to the target node, that is, the target node is added to the global graph, and relationship edges between the target node and other nodes contained in the global graph are established.

[0043] In this embodiment, the global graph is represented as a directed graph. For time step t, the global graph can be represented as follows: = ( , ).in, The set of nodes includes the nodes visited by the embodied agent before time step t. Let be the set of edges, used to represent The edges connecting the nodes.

[0044] For any node ∈ ,node Includes quadruplets: = ( , , , ) in, ∈ R 3 Represents a node The three-dimensional position coordinates, ∈ R d This represents the visual embedding feature vector (d is the feature dimension). Indicates the access timestamp. ∈ {0,1} represents the node access status (1 indicates visited, 0 indicates unvisited).

[0045] For any relation edge ∈ It represents the relation edge Used to connect nodes and nodes Relationship edge Corresponding edge weights Used to describe nodes and nodes The distance between them, such as Euclidean distance, correspondingly, =‖ - ‖2.

[0046] In the specific implementation process, the relationship edge This is used to reflect the spatial proximity between nodes. Based on this, during the process of updating the target node to the global graph, when establishing the relationship edges between the target node and other nodes contained in the global graph, nodes whose distance to the target node is less than a threshold are considered. Establish relational edges. Optionally, a threshold... It can be set to, for example, 2.5 meters, to ensure the accuracy of the local topology in the global graph.

[0047] After constructing the target node corresponding to the current position, in addition to updating the global graph, the abstract metagraph and the node memory queue will also be updated. The following explanation is based on step 103.

[0048] 103. Based on the target node, update the abstract metagraph and node memory queue corresponding to the target scene. The abstract metagraph is used to cluster the nodes visited by the embodied agent according to the region. The node memory queue is used to store the preset number of nodes recently visited by the embodied agent.

[0049] In one optional embodiment, as an alternative way to update the node memory queue corresponding to the target scene, the target node can be added to the node memory queue corresponding to the target scene, and the earliest added node in the node memory queue can be removed in the order of node addition time from earliest to latest, so as to determine the number of nodes to retain in the node memory queue.

[0050] Optionally, in the embodiments of this application, the node memory queue It can be a fixed-capacity double-ended queue. Assume a node-memory queue. The maximum capacity is 30 nodes. During updates at time step t, the working memory queue... The 30 most recently visited nodes are retained, while the earliest visited nodes are automatically removed.

[0051] In the specific implementation process, working memory queues As a fixed-capacity deque, at time step t, it can be defined as: ={ } Where k is the number of nodes that the working memory queue can hold, for example, k=30.

[0052] When | | = k and reach a new node At that time, perform the update operation: = { }∪{ }, that is, remove the earliest node And add the latest node .

[0053] In this scheme, the bounded window design of the node memory queue ensures that the receptive field is bounded and the computational complexity is constant, i.e., O(k), during the subsequent computation of the embodied intelligent navigation task based on the node memory queue. This is independent of the navigation length and avoids the problem of quadratic complexity in the global graph method.

[0054] In another alternative embodiment, as an optional way to update the abstract metagraph corresponding to the target scene, such as Figure 2 As shown, it may include the following steps: 201. Determine the set of nodes to be clustered, consisting of the target node and other visited nodes that are not currently clustered.

[0055] 202. For each node in the set of nodes to be clustered, construct the hybrid feature vector corresponding to the node to be clustered based on the location coordinates and visual embedding features associated with the node.

[0056] 203. In response to the number of nodes in the set of nodes to be clustered being greater than a set threshold, the nodes in the set of nodes to be clustered are clustered according to the mixed feature vector to form at least one node cluster.

[0057] 204. Determine the region node corresponding to each node cluster. The region node is associated with the region center position of the region described by the corresponding node cluster, as well as the aggregated features of the visual embedding features described by the corresponding node cluster.

[0058] 205. Update the region nodes to the abstract metagraph, and establish region edges between the region nodes and other region nodes in the abstract metagraph based on the relationship edges between the nodes in the global graph.

[0059] To elaborate, the abstract metagraph in the embodiments of this application =( , ), consisting of a set of regional nodes and region edge set Composition. Each region node. ∈ This represents a cluster of nodes that are spatially and visually similar, obtained through clustering. ={ , , ..., v j | c _ j In the abstract metagraph, when storing region nodes, the corresponding information includes the region center position, the aggregated features of the visual embedding features, and the list of nodes contained in the node cluster corresponding to each region node.

[0060] Among them, for abstract metagraph Any region node ∈ Its corresponding node cluster The central location of the area described For node clusters The nodes contained in Corresponding position coordinates The average value, that is: =(1 / | |)Σ{ ∈ } .

[0061] For abstract metagraph Any region node ∈ Its corresponding node cluster Aggregated features of the described visual embedding features For node clusters The nodes contained in Corresponding visual embedding features The average value, that is: =(1 / | |)Σ{ ∈ } .

[0062] In the calculation of the aggregated features of the above-mentioned regional center location and visual embedding features, the averaging aggregation operation is equivalent to spatial low-pass filtering, which preserves regional semantic information while removing high-frequency visual noise.

[0063] In this embodiment, the region edges in the abstract metagraph are determined based on the relationships between nodes in the global graph. For example, for region nodes... and regional nodes If a node exists ∈ and nodes ∈ And nodes in the global graph and nodes There are relation edges between them ∈ Then establish a regional node. and regional nodes The boundary between the regions ,in, That is to say .

[0064] As described in the introduction to the abstract metagraph, in this embodiment, the abstract metagraph uses region nodes as its basic unit. Therefore, the abstract metagraph is not updated in real-time on a node-by-node basis. Instead, it is updated by clustering visited nodes with similar spatial and visual features into new region nodes, which are then added to the abstract metagraph as a whole.

[0065] In the specific implementation process, the timing of clustering visited nodes can be flexibly configured. Optionally, clustering can be triggered immediately after each new node is added, or batch clustering can be performed only when the number of nodes to be clustered reaches a set threshold. The clustering frequency can also be dynamically adjusted according to environmental complexity, thereby achieving a balance between computational efficiency and map accuracy. Based on this, the abstract metagraph update process in this solution can actually be designed as periodic, event-driven, or conditionally triggered, and the specific implementation method can be customized according to the actual application scenario requirements.

[0066] To facilitate understanding of the embodiments of this application, an example is given, which is that batch clustering is only performed when the number of nodes to be clustered reaches a set threshold.

[0067] Specifically, first, a set of nodes to be clustered is determined, consisting of the target node and other visited nodes that are not currently clustered. These other visited nodes can be nodes that have not been clustered because they haven't yet met the clustering trigger conditions, or isolated nodes that failed to form effective clusters with other nodes in previous clustering processes. Therefore, the set of nodes to be clustered contains two types of nodes: newly added nodes participating in clustering for the first time, and residual nodes that have not yet been included in any clusters due to insufficient proximity or low density accumulated in history. By including both types of nodes in the set of nodes to be clustered, continuous optimization and completion of the environmental structure can be achieved in subsequent clustering operations, avoiding the loss of local information or map fragmentation.

[0068] In this embodiment, when clustering nodes in the set of nodes to be clustered, the geometric proximity and semantic similarity between nodes are comprehensively considered to achieve an accurate abstraction of the environmental structure of the target scene. To this end, a hybrid feature vector that integrates spatial location information and visual content information is used as a reference for the clustering process. This ensures that the clustering results not only reflect the relative positional relationships of nodes in three-dimensional space but also demonstrate the consistency of their visual appearance, thereby improving the rationality and robustness of region division.

[0069] Specifically, for each node in the set of nodes to be clustered, a hybrid feature vector corresponding to the node to be clustered is constructed based on the location coordinates and visual embedding features associated with that node.

[0070] Optionally, the construction process of the hybrid feature vector includes: First, scaling the position coordinates associated with the nodes to be clustered according to a preset position scaling factor to obtain a position coordinate scaling result. Then, performing dimensionality reduction on the visual embedding features associated with the nodes to be clustered using principal component analysis projection matrix; scaling the dimensionality reduction result according to a preset visual feature scaling factor to obtain a visual feature scaling result. The position scaling factor and the visual feature scaling factor are used to balance the relative contributions of geometric proximity and semantic proximity in the clustering process. Finally, the position coordinate scaling result and the visual feature scaling result are concatenated to obtain the hybrid feature vector corresponding to the nodes to be clustered.

[0071] For example, suppose the node to be clustered is Its position coordinates are Visual embedding features are For position coordinates = [x i , y i , z i ] T After scaling, we get α· For visual embedding features ∈ R d First, project the PCA matrix P ∈ R. d '× d Dimensionality reduction: f' i = P·f i Then, after scaling, we obtain β·f' i Ultimately, α· and β· f' i The concatenation yields a mixed feature vector. Mixed features h i ∈R 3+d Specifically, the mixed feature vector It can be represented as: = [α· ; β·PCA( , d')] Where α is a preset position scaling factor (e.g., α = 1.0), β is a preset visual feature scaling factor (e.g., β = 0.5), and PCA(f i , d') represents the visual embedding f iPerform Principal Component Analysis (PCA) to reduce the dimension to d' dimension (e.g., d' = 64).

[0072] Optionally, cross-validation can be used to determine a location scaling factor α and a visual feature scaling factor β that balance the contributions of geometric distance and semantic distance.

[0073] In this scheme, the location coordinates and visual embedding features, after being scaled by the location scaling factor and the visual feature scaling factor, can balance the relative contributions of geometric distance and semantic distance in the clustering process. Specifically, since spatial coordinates and visual embedding features are usually on different scales (e.g., coordinate values ​​are on the meter scale, while visual features are normalized high-dimensional vectors), direct fusion may lead to one dimension dominating the clustering decision. By using preset location scaling factors and visual feature scaling factors to adjust the scale of the location coordinates and the dimensionality-reduced visual embedding features respectively, the two types of features can be unified under the same scale. This allows the clustering algorithm to make more reasonable region divisions based on a comprehensive consideration of both "spatial proximity" and "visual similarity," avoiding clustering bias caused by a single modality dominating.

[0074] After constructing the hybrid feature vector corresponding to each node in the set of nodes to be clustered, further, in response to the number of nodes in the set of nodes to be clustered being greater than a set threshold (e.g., 3), the nodes in the set of nodes to be clustered are clustered according to the hybrid feature vector corresponding to the nodes to be clustered, so as to form at least one node cluster.

[0075] Alternatively, the Density-Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm can be used for clustering.

[0076] When using the adaptive DBSCAN algorithm for clustering, the clustering process can be divided into three stages: 1) parameter initialization, 2) iterative adjustment, and 3) experience reuse. These will be explained in detail below.

[0077] 1) Parameter initialization phase: For each node in the set of nodes to be clustered, calculate the distance between the node to be clustered and its k nearest neighbors in the set of nodes to be clustered to form a set of k nearest neighbor distances. The set of k nearest neighbor distances is used to determine the distance threshold of the clustering algorithm. Based on the number of nodes in the set of nodes to be clustered and the preset granularity of the clustering region, determine the minimum number of samples for the clustering algorithm.

[0078] In the parameter initialization phase, the distance threshold obtained is also the initial distance threshold, and the minimum number of samples obtained is also the initial minimum number of samples.

[0079] For example, for the distance threshold ε, the k-nearest neighbor distance analysis method is used: for each node to be clustered, the distance between it and the k-th nearest neighbor node in the set of nodes to be clustered is calculated (where k can be 4 or 5, etc.); then, the k-nearest neighbor distances of all nodes to be clustered are collected to form the k-nearest neighbor distance set D_knn; then, the initial value of the distance threshold ε is determined by analyzing the distribution of D_knn, for example, the median or 75th percentile of D_knn is taken as the initial distance threshold ε0 to ensure that the local density features of most nodes to be clustered can be covered.

[0080] For the minimum number of samples MinPts, based on the total number of nodes to be clustered n and the desired region granularity (e.g., the desired number of regions is between n / 10 and n / 5), the empirical formula MinPts0 = max(4,⌊ ⌋) or MinPts0 = max(4, ⌊ ⌋) Make an initial estimate of the initial minimum sample size MinPts0 to ensure that meaningful regional clusters are formed without generating too many noisy nodes.

[0081] After estimating the initial distance threshold and the initial minimum sample size, the nodes in the set of nodes to be clustered are clustered based on the estimated initial distance threshold and the initial minimum sample size, according to the mixed feature vector, to form at least one node cluster. Specifically, when clustering the nodes in the set of nodes to be clustered, for any two nodes to be clustered... and Calculate its distance as d(h) i , h j )=‖h i - h j ||2; after that, d(h i , h j Clustering is performed by combining distance thresholds.

[0082] Clustering algorithms can classify nodes in a set of nodes to be clustered into three categories: 1. Core nodes (with at least MinPts nodes in their ε-neighborhood); 2. Boundary nodes (within the ε-neighborhood of core nodes but not themselves); 3. Noise nodes (neither core nodes nor boundary nodes). Core nodes and boundary nodes form regions, i.e., node clusters; noise nodes do not participate in region formation for the time being.

[0083] In this scheme, when using spatial indexes (such as kd-trees), the adaptive DBSCAN has a time complexity of O(n log n), where n is the number of nodes, which is significantly better than the O(n log n) time complexity of the global attention mechanism. 2 Complexity.

[0084] Understandably, the initial distance threshold and initial minimum sample size estimated based on the k-nearest neighbor distance distribution and the number of nodes are only initial reference values ​​for clustering. Their values ​​may deviate from the optimal range due to environmental complexity, uneven data density distribution, or differences in clustering objectives. Therefore, when the clustering results do not meet expectations, it is necessary to enter stage 2) – iterative adjustment. By dynamically correcting the values ​​of the distance threshold and minimum sample size used for clustering, the clustering effect is gradually optimized until a reasonable region division that conforms to topological connectivity and semantic consistency is obtained.

[0085] 2) Iterative adjustment phase: If the number of node clusters formed is less than the preset minimum number of clusters, the distance threshold is reduced and / or the minimum number of samples is increased, and clustering is re-executed; if the number of node clusters formed is greater than the preset maximum number of clusters, the distance threshold is increased and / or the minimum number of samples is decreased, and clustering is re-executed; the adjustment process of the distance threshold and the minimum number of samples and the clustering process are repeated until the number of node clusters is greater than or equal to the preset minimum number of clusters, less than or equal to the preset maximum number of clusters, or the preset number of adjustments is reached.

[0086] For example, if the number of generated clusters (i.e., regions) R is too small (e.g., less than the preset minimum number of clusters, R < R_min = 3), it indicates that the current region division is too coarse, resulting in excessive merging of nodes and potential omission of important structures. Correspondingly, the distance threshold can be reduced proportionally (e.g., ε_new = 0.8 × ε_old) and / or the minimum number of samples can be increased (e.g., MinPts_new = MinPts_old + 2).

[0087] Conversely, if the number of generated clusters R is too large (e.g., greater than the preset maximum number of clusters, R > R_max = n / 3), it indicates that the division is too fine, resulting in excessive segmentation of nodes, which may lead to map fragmentation. Correspondingly, the distance threshold can be increased proportionally (e.g., ε_new = 1.2 × ε_old) or the minimum number of samples can be decreased (e.g., MinPts_new = max(4, MinPts_old -1)).

[0088] An iterative adjustment strategy is adopted, and clustering is re-executed after each adjustment until the number of regions R falls within a reasonable range [R_min, R_max], or the maximum number of iterations is reached (e.g., 3-5 times).

[0089] 3) Experience reuse stage The distance threshold and minimum number of samples corresponding to the number of node clusters being greater than or equal to the preset minimum number of clusters and less than or equal to the preset maximum number of clusters are stored as association parameter pairs in the clustering experience database. Based on the environment type corresponding to the nodes in the node set to be clustered, the target distance threshold and target minimum number of samples matching the environment type are selected from the clustering experience database. Based on the target distance threshold and target minimum number of samples, the nodes in the node set to be clustered are clustered according to the mixed feature vector to form at least one node cluster.

[0090] In the experience reuse phase, after clustering is executed, if the current parameters produce good clustering results (e.g., a reasonable number of regions and high similarity of nodes within regions), the current parameter pair is added to the clustering experience database to achieve continuous learning and parameter optimization of the clustering algorithm. By maintaining the clustering experience database, historically optimal parameters can be reused in similar environments, accelerating the adaptive convergence of clustering. The clustering experience database can record historically optimal parameter pairs (ε*, MinPts*) under different environment types (e.g., corridors, rooms, open spaces) and node densities.

[0091] Based on the maintained clustering experience database, before executing a new clustering task, similar historical scenes can be searched in the experience database according to the visual features and spatial distribution features of the current environment. If a historical record with scene similarity exceeding a threshold (such as 0.7) is found, the historical optimal parameter corresponding to the historical record can be directly used as the initial value to accelerate the adaptive convergence process.

[0092] It is important to emphasize that these three stages do not have a mandatory dependency relationship; that is, they do not necessarily have to be executed sequentially or simultaneously. In other words, one or more stages can be flexibly selected to execute based on the needs of the actual application scenario. For example, only the parameter initialization stage can be executed, directly using preset parameters for clustering; or, initialization can be skipped and the iterative adjustment stage can be directly entered, dynamically optimizing parameters based on current data; or, based on existing historical clustering results, only the experience reuse stage can be executed to achieve rapid map updates; or, multiple stages can be combined to form a complete adaptive process. The clustering mechanism in this embodiment has high flexibility and configurability, and can be applied to navigation scenarios with different complexities and real-time requirements.

[0093] After clustering the nodes to be clustered using a clustering algorithm to obtain at least one node cluster, each node cluster is further represented, that is, the corresponding region node is represented. This region node is then added to the abstract metagraph, and region edges are established between the region node and other region nodes in the abstract metagraph. The process of representing region nodes and establishing region edges can be found in the aforementioned section on the abstract metagraph, and will not be repeated here.

[0094] In this scheme, an adaptive clustering algorithm is introduced to dynamically adjust the clustering parameters (distance threshold and minimum number of samples) online based on the spatial density and visual similarity of environmental nodes, thereby compressing the navigation trajectory into a regional topology (i.e., an abstract metagraph) in real time.

[0095] After updating the abstract metagraph and node memory queue corresponding to the target scene in step 103, the prediction of the action to be executed corresponding to the giant visual language navigation task is performed based on the updated abstract metagraph and node memory queue.

[0096] 104. Based on natural language instructions and memory node queues, output node-level navigation action probability distribution through local planner; based on natural language instructions and abstract metagraph, output region-level strategic preference distribution through meta planner; based on environmental visual data, output instant termination action probability distribution through local perceptron.

[0097] For ease of understanding, combined with Figure 3 The specific implementation process of step 104 is explained. Figure 3 This is a schematic diagram of a dual-planner collaborative decision-making architecture provided in an embodiment of this application.

[0098] like Figure 3 As shown, the dual-planner collaborative decision-making architecture includes three components: the local planner π local Meta-planner π meta and local perceptron π percept .

[0099] The local planner, based on natural language instructions and a memory node queue, makes node-level action decisions and outputs the probability distribution of node-level navigation actions.

[0100] As an alternative implementation, candidate nodes that can be accessed in the next time step in the target scene can be determined first based on the environmental visual data; then, a local context is constructed based on the nodes contained in the memory node queue and the candidate nodes; finally, the local context and natural language instructions are input into the local planner, so that the local planner can perform graph encoding and convolution processing on the nodes in the local context under the guidance of the natural language instructions to obtain the node-level navigation action probability distribution corresponding to each node.

[0101] In detail, the local planner needs to process the working memory queue, candidate nodes (ghost nodes), and natural language instructions. Candidate nodes can be understood as unvisited neighboring nodes corresponding to the current position, representing the current exploration frontier. Optionally, environmental visual data collected at the current position, such as 12 currently acquired panoramic images (12*30°), can be used to determine which nodes in the panoramic image are walkable using a trained multimodal model. Walkable (i.e., traversable) nodes are then identified as candidate nodes.

[0102] In this scheme, a structured and reasonable input representation is provided to the local planner by constructing the local context corresponding to the nodes in the memory node queue and the candidate nodes. Optionally, the local context can be obtained by performing cross-attention processing on the nodes in the memory node queue and the candidate nodes. Then, the local planner performs graph encoding on the nodes in the working memory queue based on the local context and the context described by the embedded natural language instructions, generating the log probabilities of candidate actions (including moving forward, turning, stopping, etc.). In other words, the local planner is constrained by the local context (i.e., the local topology) and the embedded natural language instructions during the graph encoding process.

[0103] Alternatively, the local planner can employ a graph neural network to process the working memory queue. Encode the code. For working memory queues... Nodes in Its initial node characteristics are denoted as h. 0 _i =[ , , , During graph convolution, nodes Feature updates can be represented as: h g+1 i = σ(W g · AGG({ : ∈ N( )})) Where N( ) is a node The neighbor set, AGG is the aggregation function (e.g., mean or attention aggregation), W g Let σ be the weight matrix of the g-th convolutional layer, and σ be the activation function.

[0104] After L layers of convolution, the local planner outputs a node-level navigation action probability distribution: P local (a t = softmax(MLP([hL _current; I t ])) Among them, h L _current represents the L-layer graph convolution output of the current node, I t Embedded natural language instructions; MLP stands for Multilayer Perceptron.

[0105] In this scheme, during node-level action decision-making, the local planner uses spatial filtering implemented through a working memory queue and candidate nodes to ensure that it only focuses on reachable geometry, such as the visible space within 7.5 meters, thus guaranteeing that the local planner can complete reasoning in constant time. Since the working memory queue has a fixed capacity, the computational complexity of the local planner is constant, i.e., O(k·L), independent of the navigation length, ensuring real-time performance.

[0106] The meta-planner performs regional strategic planning based on natural language instructions and abstract metagraphs, and outputs the regional strategic preference distribution.

[0107] As an alternative implementation, an abstract context can be constructed first based on the region nodes and the region edges between them in the abstract metagraph. Then, the abstract context and natural language instructions are input into the metaplanner, which uses the metaplanner to perform graph encoding on the region nodes in the abstract context under the guidance of the natural language instructions, thereby obtaining the region-level strategic preference distribution corresponding to each region.

[0108] In detail, the meta-planner needs to process abstract metagraphs and natural language instructions. In this scheme, an abstract context corresponding to the region nodes and the edges between them in the abstract metagraph is constructed, providing the meta-planner with a structured and reasonable input representation. Then, the meta-planner performs graph encoding on the region nodes in the abstract context based on the context described by the abstract context and the embedded natural language instructions, evaluates the strategic preferences of the node clusters corresponding to each region node, integrates region connectivity and language alignment information, and generates preference scores for the regions corresponding to each region node in the abstract metagraph to obtain the region-level strategic preference distribution.

[0109] Alternatively, the local planner may employ a graph neural network to abstract the metagraph. Encode it. For the abstract metagraph... regional nodes The corresponding regional preference score can be expressed as: score ) = w connect · S connect ( ) + w lang · S lang ( , I t ) Among them, S connect ( S represents the region connectivity score, determined based on the centrality of region nodes in the abstract metagraph; lang ( , I t () represents the language alignment score, based on region nodes. Corresponding aggregate features With Natural Language Commands I t The similarity between them is determined; w connect and w lang These are the weighting coefficients.

[0110] It is worth noting that the regional preference score corresponding to a regional node can be propagated to the node level through the mapping relationship between regional nodes and the nodes contained in the corresponding node cluster: P meta ( ) = score(r( )) Where, r( ) is a node The region to which it belongs, for example, if the region node Corresponding node clusters ={ , , ..., v j | c _ j |}, then node The corresponding regional preference score can be expressed as: P meta ( ) = score( ).

[0111] In this scheme, the metaplanner, while retaining global topological reasoning capabilities, operates on a compressed decision space (i.e., an abstract metagraph), achieving a computational complexity of O(|Ri|i). t |·L'), where L' is the metagraph encoding layer. Since the number of region nodes contained in the abstract metagraph is much smaller than the number of nodes contained in the global graph, therefore, |R t | << |V t Therefore, the computational complexity of the metaplanner in this scheme is significantly lower than that of the global graph method (O(|V)). t | 2 Complexity. Furthermore, in this scheme, regional preference scores can be propagated to node-level actions through the mapping function between regional node clusters and nodes, guiding local decision-making.

[0112] The local perceptron provides high-resolution visual processing, calculates stopping scores and auxiliary cues to improve the reliability of action termination / task termination. In simpler terms, it directly processes current panoramic visual features (i.e., environmental visual data), identifies target locations and key visual landmarks, generates stopping signals, and can improve termination accuracy in visually blurred areas.

[0113] As an optional implementation, environmental visual data can be input into a local perceptron to identify target locations and key visual markers within the data. Based on the identification results, a stopping score is generated. The target location is the spatial region in the environmental visual data corresponding to the target object described by the natural language instruction; the target object could be, for example, a cup, refrigerator, sofa, or door. Key visual markers are objects or structures in the target environment that indicate the execution of a stopping action, such as a red doorknob, a vase on a dining table, a kitchen countertop, or a wall area marked with a "STOP" sign. Finally, the stopping score is mapped using a non-linear activation function to serve as the probability distribution for immediate termination of the action.

[0114] Optionally, environmental visual data The probability distribution of the immediate termination action output by the input local perceptron can be expressed as: P percept = sigmoid(MLP percept ( )) Among them, MLP percept A multilayer perceptron specifically designed for local perceptrons.

[0115] 105. By integrating the node-level navigation action probability distribution, the regional-level strategic preference distribution, and the instant termination action probability distribution, the target action to be executed by the embodied agent in the next time step is obtained, and the target action is executed in the next time step to complete the embodied visual language navigation task.

[0116] In summary, the dual-planner collaborative decision-making architecture provided in this solution obtains the node-level navigation action probability distribution by having a local planner perform constant-complexity reasoning within a fixed-size window; the meta-planner performs region-level strategic reasoning on an abstract metagraph to obtain the region-level strategic preference distribution. Furthermore, through a node-region mapping function and a learnable gating mechanism, the meta-planner's region-level preferences are mapped back to the node-level action space, enabling global strategy to guide local actions. Combined with the immediate termination action probability distribution output by the local perceptron, the target action to be executed by the embodied agent in the next time step is predicted.

[0117] Figure 4 A flowchart of a target action prediction method provided for the implementation of this application is shown below. Figure 4As shown, it may include the following steps: 401. Generate dynamic gating signals based on environmental visual data and natural language instructions. The dynamic gating signals are used to adjust the relative weights between the regional strategic preference distribution and the node-level navigation action probability distribution.

[0118] The dynamic gating signal ranges from 0 to 1 and is generated by cross-attention processing of environmental visual data (such as 12 currently acquired panoramic observation images of 12*30°) and natural language instructions.

[0119] The gating signal is implemented through a gating network, reflecting the relative needs of precise local decision-making and global strategic reasoning in the current navigation phase. When navigation is in a complex spatial structure (such as an intersection or a long corridor), the gating signal tends to increase, emphasizing the regional-level guidance of the meta-planner; when navigation approaches the target or requires fine-grained operations, the gating signal tends to decrease, emphasizing the precise control of the local planner.

[0120] For example, the natural language instruction is: Turn left at the kitchen door, walk straight to the bedroom, and stop at the vase in front of the bedroom. Assuming the embodied agent arrives at the kitchen door at the current time step and has already turned left, if the meta-planner indicates that it does not recommend going to the group of explorable nodes in the kitchen, the local planner indicates that it recommends going straight, and the local perceptron does not provide valid information, then the gating network will output a gating signal similar to 0.2:0.7:0.1, allowing the embodied agent to walk straight.

[0121] 402. Based on the mapping relationship between regions and nodes, the regional-level strategic preference distribution is propagated to the corresponding node-level action space to obtain the node-level strategic preference distribution.

[0122] As mentioned earlier, the regional preference score corresponding to a regional node can be propagated to the node level through the mapping relationship between regional nodes and the nodes contained in the corresponding node cluster: P meta ( ) = score(r( )) Where, r( ) is a node The region to which it belongs, for example, if the region node Corresponding node clusters ={ , , ..., v j | c _ j |}, then node The corresponding regional preference score can be expressed as: P meta ( ) = score( ).

[0123] The node-level preference scores of each node in the node cluster corresponding to a regional node constitute the node-level strategic preference distribution.

[0124] Determine the number of action pairs corresponding to the node-level action probability distribution, the node-level strategic preference distribution, and the immediate termination action probability distribution.

[0125] 404. Inject exploration rewards and termination signals into the fused action logarithm according to the relative weights. The exploration rewards are used to encourage access to candidate nodes, and the termination signals are used to increase the probability of stopping the action.

[0126] 405. Normalize the logarithm of actions after the injection of auxiliary signals to obtain the final action probability distribution.

[0127] Optionally, at the current time step, the final action probability distribution can be represented as: P(a t | W t M t , I t , o t = softmax(log P) local + λ meta · log P meta + λ percept · log P percept ) Among them, P local = π local (W t , I t ), which represents the node-level navigation action probability distribution output by the local planner; P meta = π meta (M t , I t ), which is the regional-level strategic preference distribution output by the meta-planner; P percept = π percept (o t ), where λ is the probability distribution of the immediate termination action output by the local perceptron. meta and λ percept For fusion weights (e.g., λ) meta =0.3, λ percept =0.2). Optionally, λ percept It can be dynamically adjusted based on the current visual clarity. The softmax function is used to normalize the fused logarithm.

[0128] 406. Select the action with the highest probability as the target action to be executed by the embodied agent in the next time step.

[0129] In practice, the action with the highest probability is selected for execution. If the selected action is stopped and the termination condition is met, the navigation task is completed; otherwise, the selected action is executed, the multi-layer dynamic memory network is updated, and the next navigation step is initiated.

[0130] In summary, this application provides a complete closed loop encompassing memory network construction, inference execution, action feedback, and memory network updates. By introducing a fixed-size working memory queue, the computational complexity of the local planner is limited to constant level. Furthermore, by introducing an abstract metagraph through a clustering mechanism that automatically adjusts clustering parameters based on different environments (e.g., narrow corridors vs. open halls) to generate reasonable region representations, the scale of nodes processed by the global planner is significantly reduced, resulting in an overall long-term navigation complexity of O(|V1|V2|V3|V4|V5|V6|V7|V8|V9|V1|V2 ... t | 2 ) reduced to O(|R t This addresses the inherent high latency inherent in high-precision technologies. Furthermore, in this scheme, the abstract metagraph preserves the global topology of the environment. When the agent needs to return from a dead end or traverse a large area, the metaplanner can provide the correct strategic direction based on region connectivity, enhancing long-distance backtracking capabilities in complex environments and resolving the problem of local methods easily getting trapped in local optima. Based on this, the dual-planner fusion mechanism in this scheme ensures reliance on local vision during precise operations and on global memory during periods of confusion, significantly improving navigation success rates.

[0131] The following describes in detail one or more embodiments of the embodied visual language navigation device based on a multilayer dynamic memory network. Those skilled in the art will understand that these devices can be configured using commercially available hardware components through the steps taught in this solution.

[0132] Figure 5 This application provides a schematic diagram of the structure of an embodied visual language navigation device based on a multi-layer dynamic memory network, applied to an embodied intelligent agent. The embodied intelligent agent is used to perform embodied visual language navigation tasks in a target scene, such as... Figure 5 As shown, the device includes: an acquisition module 11, a construction module 12, and a processing module 13.

[0133] The acquisition module 11 is used to acquire the natural language instruction to be executed and the environmental visual data collected at the current location in the target scene in response to the embodied intelligent agent reaching the current position in the current time step. Construction module 12 is used to construct the target node corresponding to the current position. The target node is associated with the position coordinates of the current position, the visual embedding features corresponding to the environmental visual data, the current time step, and the node access status. Processing module 13 is configured to update the abstract metagraph and node memory queue corresponding to the target scene based on the target node. The abstract metagraph is used to cluster the nodes visited by the embodied agent according to regions, and the node memory queue is used to store a preset number of nodes recently visited by the embodied agent. Based on the natural language instructions and the memory node queue, a local planner outputs a node-level navigation action probability distribution. Based on the natural language instructions and the abstract metagraph, a meta planner outputs a region-level strategic preference distribution. Based on the environmental visual data, a local perceptron outputs an instant termination action probability distribution. The node-level navigation action probability distribution, the region-level strategic preference distribution, and the instant termination action probability distribution are fused to obtain the target action to be executed by the embodied agent in the next time step, and the target action is executed in the next time step to complete the embodied visual language navigation task.

[0134] Figure 5 The device shown can perform the steps described in the foregoing embodiments. For detailed execution process and technical effects, please refer to the description in the foregoing embodiments, which will not be repeated here.

[0135] In one possible design, the above Figure 5 The structure of the embodied visual language navigation device based on a multi-layer dynamic memory network shown can be implemented as an electronic device, such as... Figure 6 As shown, the electronic device may include: a memory 21, a processor 22, and a communication interface 23. The memory 21 stores a computer program, which, when executed by the processor 22, enables the processor 22 to at least implement the embodied visual language navigation method based on a multi-layer dynamic memory network as provided in the foregoing embodiments.

[0136] The aforementioned memory 21 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0137] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the above-described method embodiments. The computer-readable storage medium includes volatile or non-volatile components, or a combination thereof, and can be removable or non-removable. Examples of computer-readable storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), flash memory or other memory technologies, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium. Accordingly, this application also provides a computer program product, which includes a computer program or instructions that, when executed by a processor, cause the processor to implement the steps in the above method embodiments. It should be understood that each step or combination of steps in the above method flow can be implemented by the computer program or instructions. Furthermore, these computer programs or instructions can be applied to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable embodied visual language navigation device based on a multi-layer dynamic memory network, enabling the processor of the general-purpose computer, special-purpose computer, embedded processor, or other programmable embodied visual language navigation device based on a multi-layer dynamic memory network to function as an apparatus for implementing the corresponding functions in the above method embodiments.

[0138] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0139] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of a necessary general-purpose hardware platform, or by a combination of hardware and software. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a computer product. This application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0140] Finally, it should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0141] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. An embodied visual-language navigation method based on a multi-layer dynamic memory network, applied to an embodied intelligent agent, wherein the embodied intelligent agent performs an embodied visual-language navigation task in a target scene, characterized in that, include: In response to the embodied intelligent agent reaching its current position in the target scene at the current time step, it acquires the natural language instruction to be executed and the environmental visual data collected at the current position; Construct a target node corresponding to the current location. The target node is associated with the location coordinates of the current location, the visual embedding features corresponding to the environmental visual data, the current time step, and the node access status. Based on the target node, update the abstract metagraph and node memory queue corresponding to the target scene. The abstract metagraph is used to cluster the nodes visited by the embodied agent according to the region. The node memory queue is used to store a preset number of nodes recently visited by the embodied agent. Based on the natural language instructions and the memory node queue, a node-level navigation action probability distribution is output through a local planner; based on the natural language instructions and the abstract metagraph, a region-level strategic preference distribution is output through a metaplanner; based on the environmental visual data, an instant termination action probability distribution is output through a local perceptron. By integrating the node-level navigation action probability distribution, the region-level strategic preference distribution, and the instant termination action probability distribution, the target action to be executed by the embodied agent in the next time step is obtained, and the target action is executed in the next time step to complete the embodied visual language navigation task.

2. The method according to claim 1, characterized in that, The step of updating the abstract metagraph corresponding to the target scene based on the target node includes: Based on the target node, update the global graph corresponding to the target scene. The global graph is used to represent the nodes visited by the embodied agent and the relationship edges between the nodes. The edge weights of the relationship edges are used to describe the distance between the nodes. Determine the set of nodes to be clustered, consisting of the target node and other visited nodes that are not currently clustered; For each node in the set of nodes to be clustered, a hybrid feature vector corresponding to the node is constructed based on the location coordinates and visual embedding features associated with the node. In response to the number of nodes in the set of nodes to be clustered being greater than a set threshold, the nodes in the set of nodes to be clustered are clustered according to the mixed feature vector to form at least one node cluster; Determine the region node corresponding to each node cluster. The region node is associated with the region center position of the region described by the corresponding node cluster, as well as the aggregated features of the visual embedding features described by the corresponding node cluster. The region nodes are updated to the abstract metagraph, and region edges are established between the region nodes and other region nodes in the abstract metagraph based on the relationship edges between the nodes in the global graph.

3. The method according to claim 2, characterized in that, The step of constructing a hybrid feature vector corresponding to the node to be clustered based on the location coordinates and visual embedding features associated with the node to be clustered includes: The position coordinates associated with the nodes to be clustered are scaled according to a preset position scaling factor to obtain the position coordinate scaling result; The visual embedding features associated with the nodes to be clustered are reduced in dimensionality by using the projection matrix of principal component analysis. The dimensionality reduction result is scaled according to a preset visual feature scaling factor to obtain a visual feature scaling result. The position scaling factor and the visual feature scaling factor are used to balance the relative contributions of geometric proximity and semantic proximity in the clustering process. The scaling results of the location coordinates and the scaling results of the visual features are concatenated to obtain the hybrid feature vector corresponding to the node to be clustered.

4. The method according to claim 2, characterized in that, The step of clustering the nodes in the set of nodes to be clustered according to the mixed feature vector to form at least one node cluster includes: For each node in the set of nodes to be clustered, the distance between the node to be clustered and the k nearest neighbor nodes in the set of nodes to be clustered is calculated to form a set of k nearest neighbor distances. The set of k nearest neighbor distances is used to determine the distance threshold of the clustering algorithm. The minimum number of samples for the clustering algorithm is determined based on the number of nodes in the set of nodes to be clustered and the preset granularity of the clustering region. Based on the distance threshold and the minimum number of samples, the nodes in the set of nodes to be clustered are clustered according to the mixed feature vector to form at least one node cluster; If the number of node clusters formed is less than the preset minimum number of clusters, then the distance threshold is reduced and / or the minimum number of samples is increased, and clustering is re-executed; If the number of node clusters formed is greater than the preset maximum number of clusters, then the distance threshold is increased and / or the minimum number of samples is decreased, and clustering is re-executed; Repeat the adjustment process of the distance threshold and the minimum number of samples, and the clustering process, until the number of node clusters is greater than or equal to the preset minimum number of clusters, less than or equal to the preset maximum number of clusters, or reaches the preset number of adjustments.

5. The method according to claim 4, characterized in that, The method further includes: The distance threshold and minimum number of samples corresponding to the number of node clusters being greater than or equal to the preset minimum number of clusters and less than or equal to the preset maximum number of clusters are stored as association parameter pairs in the clustering experience database. Based on the environment type corresponding to the nodes in the set of nodes to be clustered, target distance thresholds and target minimum sample numbers that match the environment type are selected from the clustering experience database. Based on the target distance threshold and the target minimum number of samples, the nodes in the set of nodes to be clustered are clustered according to the mixed feature vector to form at least one node cluster.

6. The method according to claim 1, characterized in that, The step of updating the node memory queue corresponding to the target scene based on the target node includes: The target node is added to the node memory queue corresponding to the target scene, and the earliest added node in the node memory queue is removed in order of node addition time from earliest to latest, so as to determine the number of nodes to retain in the node memory queue.

7. The method according to claim 1, characterized in that, The step of outputting a node-level navigation action probability distribution through a local planner based on the natural language instructions and the memory node queue includes: Based on the environmental visual data, candidate nodes that can be accessed in the next time step in the target scene are determined; A local context is constructed based on the nodes contained in the memory node queue and the candidate nodes; The local context and the natural language instruction are input into the local planner, so that the local planner can perform graph encoding and convolution processing on the nodes in the local context under the guidance of the natural language instruction to obtain the node-level navigation action probability distribution corresponding to each node.

8. The method according to claim 1, characterized in that, The step of outputting a regional-level strategic preference distribution through a meta-planner based on the natural language instructions and the abstract metagraph includes: An abstract context is constructed based on the region nodes and the region edges between them in the abstract metagraph; The abstract context and the natural language instructions are input into the metaplanner, which performs graph encoding on the regional nodes in the abstract context under the guidance of the natural language instructions to obtain the regional-level strategic preference distribution corresponding to each region.

9. The method according to claim 1, characterized in that, The step of outputting an instantaneous termination action probability distribution through a local perceptron based on the environmental visual data includes: The environmental visual data is input into a local perceptron to identify the target location and key visual markers in the environmental visual data. A stopping score is generated based on the identification results of the target location and the key visual markers. The target location is the spatial region in the environmental visual data corresponding to the target object described by the natural language instruction. The key visual markers are objects or structures in the target environment that indicate the execution of the stopping action. The stop score is mapped using a nonlinear activation function and used as the probability distribution of the immediate termination action.

10. The method according to claim 1, characterized in that, The process of fusing the node-level navigation action probability distribution, the region-level strategic preference distribution, and the immediate termination action probability distribution to obtain the target action to be executed by the embodied agent in the next time step includes: Based on the environmental visual data and the natural language instructions, a dynamic gating signal is generated, which is used to adjust the relative weight between the regional strategic preference distribution and the node-level navigation action probability distribution; Based on the mapping relationship between regions and nodes, the regional-level strategic preference distribution is propagated to the corresponding node-level action space to obtain the node-level strategic preference distribution; Determine the number of action pairs corresponding to the node-level action probability distribution, the node-level strategic preference distribution, and the instant termination action probability distribution, respectively; Based on the relative weights, exploration rewards and termination signals are injected into the logarithm of the actions, wherein the exploration rewards are used to encourage visits to the candidate nodes, and the termination signals are used to increase the probability of stopping the action; The logarithm of actions after the injection of the auxiliary signal is normalized to obtain the final action probability distribution. The action with the highest probability is selected as the target action to be executed by the embodied agent in the next time step.

Images