Visual-semantic map construction method in cross-scene generalization

Abstract

The invention discloses a visual-semantic map construction method in cross-scene generalization, relates to the technical field of robot vision, and is characterized in that the method comprises three steps of adaptive visual feature extraction, semantic alignment and fusion and hierarchical path planning. The problem that hierarchical path planning based on language instructions is affected due to the fact that a semantic topological map is difficult to construct due to visual feature distribution deviation of an intelligent agent in a non-contact scene can be solved. According to the method, through an innovative multi-module cooperation mechanism, the accuracy of visual-semantic map construction of the intelligent agent under cross-scene and the efficiency and robustness of navigation based on language instructions are remarkably improved, and the method has wide application prospects in the fields of smart home, service robots, automatic driving and the like.

Description

Technical Field

[0001] This invention relates to the field of robot vision technology, and more specifically, to a method for constructing visual-semantic maps across scenarios. Background Technology

[0002] With the rapid development of artificial intelligence and robotics, visual-language navigation has shown enormous application potential in fields such as smart homes, service robots, and autonomous driving. Visual-semantic maps, as the core foundation of visual-language navigation, aim to combine scene information perceived by the robot with semantic information to construct a map that includes both spatial topological relationships and semantic understanding, thereby enabling intelligent navigation based on language commands. However, in practical applications, the cross-scene generalization problem has become a key bottleneck restricting the effective construction of visual-semantic maps and the improvement of visual-language navigation performance.

[0003] Traditional visual-semantic map construction methods are mostly based on training data from specific scenarios, constructing maps by extracting visual features and associating semantic information. In training scenarios, these methods perform well in map construction and navigation tasks. However, when the agent enters building structures or spatial layouts unfamiliar from the training phase, problems arise due to significant shifts in the distribution of visual features. The vast differences in lighting conditions, object placement, spatial scale, and architectural style across different scenarios cause the visual features extracted by traditional methods to lose representativeness and stability. For example, visual features used to identify doors in a training scenario may fail to accurately identify doors in new scenarios due to changes in material, color, and decoration.

[0004] Furthermore, traditional methods also have limitations in semantic association. They often rely on fixed semantic templates and predefined scene models, making it difficult to adapt to the diversity and variability of semantic concepts in new scenes. When encountering object categories or spatial semantic relationships unique to a new scene, they cannot accurately match visual information with semantic information, thus failing to build an effective semantic topology map. Since the semantic topology map is the foundation for hierarchical path planning based on language instructions, if the semantic topology map construction fails, the agent will be unable to understand the semantic information in the language instructions, unable to plan a reasonable path, leading to navigation task failure.

[0005] Some existing improvement methods attempt to alleviate the cross-scene generalization problem by increasing the diversity of training data, but this approach is not only costly but also difficult to cover all possible scenarios. Other studies employ transfer learning strategies, transferring knowledge learned in the source scene to the target scene, but due to the significant differences between the scenes, the transfer effect is not ideal. Summary of the Invention

[0006] The purpose of this invention is to provide a visual-semantic map construction method for cross-scene generalization, which can effectively address the problem of visual feature distribution offset in cross-scene generalization, achieve accurate semantic topology map construction, and improve the hierarchical path planning capability based on language instructions.

[0007] The above-mentioned technical objective of the present invention is achieved through the following technical solution: a visual-semantic map construction method in cross-scene generalization, comprising the following steps:

[0008] (1) Adaptive visual feature extraction: By performing meta-training in multiple different source scenes, the network learns the scene-independent general visual feature extraction capability. In the target scene, the network is quickly fine-tuned using a small amount of data to extract visual features adapted to the target scene.

[0009] (2) Semantic alignment and fusion: Construct a multimodal semantic space, map visual features, text semantic information and spatial topology information to a unified semantic space, use graph neural networks to model nodes and edges in the semantic space, calculate the similarity between visual features and text semantics in the semantic space to achieve semantic alignment, and combine spatial topology information to fuse semantic information into the visual map to establish a semantic topology map.

[0010] (3) Hierarchical path planning: Based on the constructed semantic topology map, a hierarchical reinforcement learning algorithm is used for path planning. The navigation task is decomposed into two levels: high-level semantic decision-making and low-level action execution. The high-level semantic decision-making selects appropriate semantic target nodes in the semantic topology map according to language instructions. The low-level action execution generates specific action instructions to guide the agent to move based on the relative positional relationship between the current position and the target node.

[0011] The present invention is further configured such that: the meta-training stage specifically comprises: collecting source scene data of different types and dividing them into corresponding meta-learning tasks, each task containing a support set and a query set; constructing a meta-learning feature extraction network based on a convolutional neural network; performing forward propagation to calculate feature representations on the support set of each task; performing classification prediction based on features and calculating the loss on the support set; updating network parameters through a gradient descent algorithm to obtain task-specific parameters; performing prediction on the query set using the updated parameters; and iteratively training the network with the goal of minimizing the expected loss of all task query sets.

[0012] The present invention is further configured such that: the target scene fine-tuning stage specifically involves: collecting a small amount of image data in the target scene to form a fine-tuning dataset, fixing the parameters of the bottom convolutional layers of the meta-training network, adjusting only the parameters of the high-level networks, and obtaining a feature extraction network adapted to the target scene by optimizing the fine-tuning loss function.

[0013] The present invention is further configured such that: the construction of the multimodal semantic space specifically involves: using a fully connected layer to map the visual feature vector output by the visual feature extraction network to the semantic space; using a pre-trained language model to encode the semantic information of the text and mapping it to the semantic space through a fully connected layer; constructing a topology graph based on the agent's motion odometer data and sensor data, and mapping the node and edge features of the topology graph to the semantic space through an encoding function.

[0014] The present invention is further configured such that: the semantic alignment specifically involves: calculating the similarity score between each visual feature semantic node and all text semantic nodes; using a graph neural network to perform information propagation and feature updates on nodes in the semantic space; selecting the text semantic node with the highest score and aligning it with the visual feature node by comparing the similarity scores; and temporarily not aligning if the highest score is lower than a set threshold.

[0015] The present invention is further configured such that the semantic fusion specifically involves attaching the aligned semantic information as an attribute to the corresponding visual map node in the spatial topology graph, updating the weights of the edges in the topology graph according to the relationship between the semantic nodes, and obtaining the semantic topology map through multiple iterations.

[0016] The present invention is further configured such that: the high-level semantic decision-making specifically involves: inputting natural language instructions into a natural language processing module, extracting a set of semantic keywords and semantic information of the target location, calculating the similarity between the keyword embedding vector and the semantic representation vector of the semantic node in the semantic topology map, selecting nodes with similarity higher than a threshold to form a target node set; defining the state space, action space and reward function of the high-level semantic decision-making, and using a reinforcement learning algorithm to train and obtain the optimal semantic path strategy from the current semantic node to the target semantic node.

[0017] The present invention is further configured such that: the execution of the underlying action specifically includes: defining the state space of the underlying action execution, including the current position coordinates of the agent, sensor data, and the next semantic node specified by the higher layer; defining the action space as a set of discrete actions that the agent can execute; defining the reward function of the underlying action execution; using a reinforcement learning algorithm to train and obtain the optimal action execution strategy that performs appropriate actions according to the current state; and during navigation, if the distance to an obstacle is detected to be less than a safety threshold, triggering the higher-level semantic decision module to replan the path.

[0018] In summary, the present invention has the following beneficial effects:

[0019] 1. Superior Cross-Scene Adaptability: The adaptive visual feature extraction module is based on a meta-learning mechanism. During the meta-training phase, the network learns from multiple different source scenes, enabling it to master scene-independent general visual feature extraction capabilities. When fine-tuning the target scene, only a small amount of data is needed to quickly adapt to the visual feature distribution of the new scene. This feature effectively overcomes the problem of visual feature distribution shift caused by changes in building structure and spatial layout. Compared with traditional methods that rely on a large amount of specific scene data for training, it significantly reduces data acquisition costs and training time, and significantly improves the accuracy and stability of visual feature extraction in unknown scenes, laying a solid foundation for subsequent map building and navigation.

[0020] 2. Precise Semantic Processing Capabilities: The multimodal semantic space constructed by the semantic alignment and fusion module unifies the mapping of visual, textual, and spatial topological information, breaking the limitations of isolated semantic information processing in traditional methods. By leveraging graph neural networks to model semantic nodes and edges, precise alignment of visual and textual semantics is achieved, fully considering the complex relationships between semantic concepts. During semantic fusion, the map's semantic attributes are dynamically updated based on the topological graph, enabling the semantic topological map to flexibly respond to the diversity and variability of semantic concepts in different scenarios. Compared to existing technologies, this represents a qualitative leap in the accuracy of semantic understanding and the richness of map semantic expression, providing strong support for agents to understand complex language instructions.

[0021] 3. Highly Efficient Path Planning Performance: The hierarchical path planning module processes navigation tasks in layers. High-level semantic decision-making, based on semantic topology maps and reinforcement learning, accurately parses semantic information in language commands and plans reasonable semantic paths, effectively avoiding the blindness of traditional methods in semantic understanding and path planning. Low-level action execution combines environmental perception and reinforcement learning, generating precise action commands based on the current position and sensor data, enabling stable movement of the agent in complex environments. When encountering obstacles or other unexpected situations, the information interaction mechanism between upper and lower layers can promptly trigger path replanning, ensuring the smooth completion of the navigation task. This layered architecture significantly reduces the complexity of path planning, improves its efficiency and accuracy, and enhances the robustness and flexibility of the agent in cross-scenario language command-based navigation.

[0022] 4. Wide Range of Applications and Expanded Value: This patented technology effectively solves the challenge of cross-scenario generalization, greatly improving the performance of visual language navigation. It can be widely applied in multiple fields such as smart homes, service robots, and autonomous driving. In smart home scenarios, service robots can quickly adapt to different home layouts and accurately execute user commands. In the field of autonomous driving, vehicles can better cope with complex and ever-changing road environments, safely reaching their destinations based on navigation instructions, thus promoting the improvement of the intelligence level of related industries and possessing significant economic and social value. Attached Figure Description

[0023] Figure 1 This is an information flow diagram of the new scene navigation generalization model in this embodiment of the invention;

[0024] Figure 2 This invention demonstrates the collaborative process of semantic alignment and hierarchical path planning.

[0025] Figure 3 This invention demonstrates a complete information processing closed loop from multi-source perception to navigation actions. Detailed Implementation

[0026] The following is in conjunction with the appendix Figure 1-3 The present invention will be described in further detail below.

[0027] Example: A visual-semantic map construction method in cross-scene generalization, including the following steps:

[0028] I. Adaptive Visual Feature Extraction

[0029] 1. Data preparation for the meta-training phase

[0030] Data Collection: Collect at least 100 source scene data of different types from public datasets (such as SUNCG, Matterport3D) and real-world scene acquisitions. Each source scene contains 200-500 images, covering indoor (living room, bedroom, office, etc., different decoration styles) and outdoor (street, square, etc., different environments) scenes. Simultaneously, annotate each image with semantic labels, such as common object categories like "door," "window," and "table."

[0031] Task partitioning: All source scene data is divided into 500-1000 meta-learning tasks. For each task, 20-30 images are randomly selected as the support set and 10-15 images as the query set. This ensures that the data in each task has a certain degree of diversity in terms of scene type, object category, etc.

[0032] 2. Network Architecture and Training

[0033] Network Construction: ResNet-50 is used as the backbone network, and three fully connected layers are added on top of it to construct a meta-learning feature extraction network. The network input is an RGB image with a resolution of 224×224, and the output is a feature vector with a dimension of 256.

[0034] Training process: Setting the initial learning rate The Adam optimizer is used. For each meta-learning task... In supporting sets Perform forward propagation to calculate the feature vector. Then, the sample category is predicted using the classification head, and the cross-entropy loss is calculated. Update network parameters using gradient descent algorithm. Obtain task-specific parameters Then in the query set Used Perform prediction and calculate query set loss. Repeat the above process to traverse all tasks, minimizing the expected loss of the query set for all tasks. With the goal of training the network iteratively for 20-30 epochs.

[0035] 3. Fine-tuning the target scenario

[0036] Data acquisition: When the intelligent agent enters the target scene, it uses its onboard camera to collect 50-100 images at different positions and angles to form a fine-tuning dataset.

[0037] Parameter tuning: Training the network with fixed parameters Parameters of the first 3 residual blocks (corresponding to the bottom convolutional layer) Only for the parameters of the subsequent fully connected layers Make adjustments. Set the fine-tuning learning rate. We employ stochastic gradient descent to fine-tune the cross-entropy loss on the dataset. To optimize the target, train for 5-10 epochs to obtain a feature extraction network adapted to the target scene.

[0038] II. Semantic Alignment and Fusion

[0039] 1. Construction of Multimodal Semantic Space

[0040] Visual feature mapping: This involves mapping the image feature vectors obtained through the adaptive visual feature extraction module. (Dimension 256), input to the fully connected layer (dimension) After passing through the ReLU activation function, the visual features are represented in the semantic space. (Dimension 512).

[0041] Text semantic mapping: The pre-trained BERT model is used to encode the input semantic text, resulting in a semantic vector with a dimension of 768. Then through the fully connected layer (dimension) The ReLU activation function is mapped to the semantic space to obtain... (Dimension 512).

[0042] Spatial topology information encoding: Based on the agent's motion odometry data and LiDAR point cloud data, construct a topology graph containing 10-20 nodes. For each node and edge, a multilayer perceptron (MLP) is used for encoding to obtain a spatial topological semantic representation with a dimension of 512. .

[0043] 2. Semantic alignment

[0044] Similarity calculation: For each visual feature semantic node Calculate its relationship with all text semantic nodes cosine similarity score A graph neural network with 3 graph convolutional layers is constructed, with each convolutional kernel size set to 3×3 and the number of channels to 64. Information propagation and feature updates are performed on nodes in the semantic space 3-5 times, using the following update formula: ,in This is the ReLU activation function.

[0045] Alignment decision: By comparing similarity scores, the text semantic node with the highest score is selected for alignment with the visual feature node. If the highest score is lower than the set threshold of 0.7, alignment is temporarily suspended, and subsequent data supplementation or recalculation is required.

[0046] 3. Semantic fusion

[0047] Attribute Attachment: In the spatial topology graph In this process, the aligned semantic information is attached as node attributes to the corresponding visual map node. For example, if a visual node corresponds to the semantics of "door", the relevant attributes of "door" (such as passability, location information, etc.) are added to that node.

[0048] Edge weight update: Based on the relationships between semantic nodes, use the formula Update the weights of edges in the topology graph. Specifically, set... , Cosine similarity is used as a metric. It is a Euclidean distance metric. To measure semantic relevance, weight coefficients are learned through training. After 5-10 iterations, a semantic topology map is obtained.

[0049] III. Hierarchical Path Planning

[0050] 1. High-level semantic decision-making

[0051] Instruction parsing: Input natural language instructions into the natural language processing module based on the Transformer architecture to extract a set of semantic keywords. The target location semantic information is obtained by calculating the cosine similarity between the keyword embedding vector and the semantic representation vector of the nodes in the semantic topology map, and selecting nodes with a similarity greater than 0.6 to form the target node set. .

[0052] Reinforcement learning training: Defining the state space Action space and reward function A Deep Q-Network (DQN) architecture is adopted, consisting of three fully connected layers with 256, 128, and 64 neurons per layer, respectively. A discount factor is set. The experience replay buffer size is 10,000. During training, 32 samples are randomly selected from the buffer each time for learning, and the training is iterated for 5,000-10,000 episodes to learn the optimal semantic path strategy. .

[0053] 2. Execution of underlying actions

[0054] State definition: State space Includes the agent's current position coordinates LiDAR scanning data (Processed as a 128-dimensional vector) and the next semantic node specified by the higher level. Action space It includes four discrete actions: "Forward", "Turn Left (30 degrees)", "Turn Right (30 degrees)" and "Stop".

[0055] Reinforcement learning training: The Deep Deterministic Policy Gradient (DDPG) algorithm is used to construct an Actor-Critic network architecture. The Actor network contains three fully connected layers (256, 128, and 4 neurons), and the Critic network contains three fully connected layers (256, 128, and 1 neuron). A discount factor is set. Soft update coefficient During training, the agent executes actions according to the policy, obtains rewards and new states, and stores the data in an experience replay buffer. 64 samples are sampled each time for learning, and the training is iterated for 8000-12000 steps to obtain the optimal action execution policy. During navigation, if the agent detects that the distance to an obstacle is less than the safety threshold of 0.5 meters, it triggers the high-level semantic decision-making module to replan the path.

[0056] Through the detailed implementation steps described above, the visual-semantic map construction method in cross-scenario generalization described in this patent can be realized, effectively improving the navigation ability of intelligent agents based on language commands in different scenarios.

[0057] Working principle: The method provided by this invention mainly includes the following key parts:

[0058] Adaptive Visual Feature Extraction Module: This module employs a meta-learning-based feature extraction network. Through meta-training in multiple different source scenes, it learns scene-independent general visual feature extraction capabilities. When faced with a new target scene, the network is rapidly fine-tuned using a small amount of target scene data, enabling it to adaptively extract visual features suitable for the target scene. Specifically, during the meta-learning process, the network learns how to quickly adjust network parameters under different scenes by optimizing the meta-loss function to adapt to the visual feature distribution of the new scene. For example, the network can learn stable extraction methods for basic visual features such as edges and textures under different lighting conditions and changes in object shape, avoiding feature extraction failure due to scene differences.

[0059] (I) Detailed Technical Solution of Adaptive Visual Feature Extraction Module

[0060] This module addresses the problem of cross-scene visual feature distribution shift by constructing a dynamic feature extraction framework based on meta-learning. Its core consists of two parts: general feature learning in the meta-training stage and rapid adaptation to the target scene. Through parameter optimization and structural design, a balance between scene independence and specificity is achieved.

[0061] 1. Meta-training stage: Building general feature extraction capabilities

[0062] Let the source scene set be Each source scene Include There are samples, and the sample is represented as... ,in For image data, The corresponding semantic tags are used. The source scene data is divided into... Individual learning task Each task Includes support set

[0063] and query set .

[0064] Constructing a meta-learning feature extraction network based on convolutional neural networks (CNN) Using ResNet-50 as the backbone network, among which This represents the network parameters. During meta-training, for each task... Firstly, in the support set The forward propagation is performed to compute the feature representation:

[0065] in For the sample Extracted feature vectors. Based on features. For classification prediction, the loss on the support set is calculated using the cross-entropy loss function. :

[0066]

[0067] in Predicting categories for the model The probability of [the outcome]. Network parameters are updated using the gradient descent algorithm. Obtain task-specific parameters :

[0068] in This is the learning rate. Next, in the query set... Use updated parameters Perform prediction and calculate query set loss. The goal of meta-learning is to minimize the expected loss of the query set for all tasks:

[0069] Through iterative optimization Make the network The network learns to quickly adapt to different scenarios and extract general features. During training, the MAML (Model-Agnostic Meta-Learning) algorithm is used to enhance the network's generalization performance through multiple intra-task gradient updates and inter-task parameter averaging.

[0070] 2. Target Scene Fine-Tuning Stage: Specific Feature Adaptation

[0071] When the intelligent agent enters the target scene At that time, collect Image samples The fine-tuning dataset is constructed. The network is trained using fixed meta-databases. parameters of the bottom convolutional layer (e.g., the parameters of the first 3 residual blocks of ResNet-50), only for high-level network parameters Adjustments will be made.

[0072] Define the fine-tuning loss function Similarly, cross-entropy loss is used:

[0073] in , For the fine-tuned model to predict samples category The probability of is minimized using the stochastic gradient descent (SGD) algorithm. :

[0074] in To fine-tune the learning rate. The network after fine-tuning. It can adaptively extract stable and representative visual features in the target scene, effectively addressing the feature distribution shift caused by changes in lighting and differences in object shape, and providing a reliable feature foundation for subsequent semantic alignment and map construction.

[0075] The innovation of this module lies in its use of a meta-learning framework to transform the differences in feature distributions across different source scenes into learnable parameter adjustment patterns. By combining a strategy of fixing low-level parameters with fine-tuning high-level parameters, it ensures the universality of feature extraction while achieving rapid adaptation to the target scene. Compared to traditional transfer learning methods, this approach does not require a large amount of target scene data; the feature extractor can be optimized with only a small number of samples, significantly improving the efficiency and accuracy of cross-scene visual feature extraction.

[0076] Semantic Alignment and Fusion Module: To address the semantic association problem, this module constructs a multimodal semantic space, mapping visual features, textual semantic information, and spatial topological information into a unified semantic space. Graph neural networks are used to model nodes and edges in the semantic space, where nodes represent semantic concepts and edges represent relationships between semantic concepts. Accurate semantic alignment is achieved by calculating the similarity between visual features and textual semantics in the semantic space. Simultaneously, combined with spatial topological information, semantic information is fused into the visual map to establish a semantic topological map. For example, when an agent perceives a new object, it searches for similar semantic concepts in the semantic space and, combined with the spatial topological relationships of the surrounding environment, determines the object's semantic label and integrates it into the map.

[0077] (II) Detailed Technical Solution for Semantic Alignment and Fusion Module

[0078] This module addresses the challenges of semantic concept diversity and visual-semantic association difficulties across different scenarios by constructing a semantic alignment and fusion framework based on multimodal semantic space and graph neural networks. This framework enables accurate matching and map fusion of visual and semantic information, providing a structured semantic topology map for hierarchical path planning.

[0079] 1. Constructing a multimodal semantic space

[0080] Define a high-dimensional semantic space ,in This represents the semantic space dimension. For visual features, a visual feature extraction network is used. (i.e., the network trained by the adaptive visual feature extraction module) extracts images. eigenvectors Then, using a fully connected layer Map it to the semantic space:

[0081] in It is a linear rectified function. This represents the visual features in the semantic space.

[0082] For semantic information in text, pre-trained language models (such as BERT) are used to process the semantic text. Encode to obtain semantic vectors Then through the fully connected layer Mapping to semantic space:

[0083] in This represents the semantics of text in the semantic space.

[0084] Spatial topology information is obtained through the agent's motion odometry data. A topology graph is constructed using (representing planar coordinates and orientation angles respectively) and sensor data (such as lidar point cloud data). ,in For a set of nodes, This is a set of edges. The node and edge features of the topological graph are encoded using an encoding function. Mapping to the semantic space yields a spatial topological semantic representation. :

[0085] 2. Semantic alignment

[0086] Construct a semantic alignment model based on a graph neural network (GNN). Let the set of nodes in the semantic space be denoted as . ,in For the first Semantic representation of visual features For the first The representation of textual semantics For the first A spatial topological semantic representation. Edge set. It represents the semantic relationships between nodes.

[0087] For each visual feature semantic node Calculate its relationship with all text semantic nodes Similarity score The cosine similarity formula is used:

[0088] Simultaneously, a graph neural network (such as a graph convolutional network GCN) is used to perform information propagation and feature updates for nodes. For nodes... Its update formula is:

[0089] in For nodes The set of neighboring nodes, The normalization coefficient is... The weight matrix is ​​a learnable matrix. For bias vectors, For example, ReLU is an activation function.

[0090] By iteratively updating node features and combining them with similarity scores, the text semantic node that best matches the visual semantic features is selected, achieving visual-semantic alignment. For example, if Then visual features With text semantics Alignment.

[0091] 3. Semantic fusion

[0092] After semantic alignment, semantic information is fused into the visual map. This is done using a spatial topology map. Based on this, for each visual feature corresponding to a node (In the visual map), the semantic information of its alignment The attributes are attached to the nodes. At the same time, the connection weights and semantic relationships between nodes in the map are updated based on the relationships between semantic nodes (represented by the edge information updated by the graph neural network).

[0093] Let the updated spatial topology be For the edge Its weight The updated formula is:

[0094] in For different semantic relationship measurement functions (such as distance measurement, semantic similarity measurement). These are the corresponding weight coefficients, obtained through training.

[0095] Through the above process, visual, textual, and spatial topological information are deeply integrated to construct a semantic topological map containing rich semantic information, providing an accurate semantic foundation for hierarchical path planning based on language instructions.

[0096] The innovation of this module lies in: organically integrating visual, textual, and spatial topological information by constructing a unified multimodal semantic space; using graph neural networks to achieve dynamic updating of semantic nodes and relationship modeling, solving the problem that traditional methods struggle to handle complex relationships of semantic concepts; and dynamically adjusting the semantic attributes of nodes and edges based on the topological graph structure during the semantic fusion stage, enabling the semantic topological map to flexibly adapt to semantic changes in different scenarios, resulting in a significant improvement in semantic alignment accuracy and map semantic expression capabilities compared to existing technologies.

[0097] Hierarchical Path Planning Module: Based on the constructed semantic topology map, this module employs a hierarchical reinforcement learning algorithm for path planning. The navigation task is decomposed into two levels: high-level semantic decision-making and low-level action execution. High-level semantic decision-making selects appropriate semantic target nodes from the semantic topology map based on language instructions; low-level action execution generates specific action instructions based on the relative position of the current location and the target node, guiding the agent's movement. This hierarchical structure allows for better utilization of information in the semantic topology map, improving the efficiency and accuracy of path planning and achieving hierarchical path planning based on language instructions.

[0098] (III) Detailed Technical Solution for Hierarchical Path Planning Module

[0099] Based on the constructed semantic topology map, this module designs a hierarchical reinforcement learning framework that decomposes the complex visual language navigation task into two collaborative layers: high-level semantic decision-making and low-level action execution. This effectively utilizes semantic information to improve the efficiency and accuracy of path planning and solves the problem of language command-based navigation in cross-scenario environments.

[0100] 1. High-level semantic decision-making

[0101] Define semantic topology map as ,in A set of semantic nodes, each node Each corresponds to a semantic concept (such as "living room", "door", "stairs", etc.). For the set of edges connecting semantic nodes, the edges Represents a node and The reachability relationship between them.

[0102] Natural language instructions The input is processed by the natural language processing module, which extracts a set of semantic keywords from the instructions using word embedding (such as Word2Vec) and semantic parsing techniques. And semantic information about the target location. In the semantic topology map, search for the set of semantic target nodes that match the semantic keywords. The matching process can be achieved by calculating the embedding vectors of semantic keywords. semantic representation vector of semantic node Cosine similarity is used to achieve this:

[0103] Select those with similarity higher than the threshold semantic nodes constitute .

[0104] Construct a reinforcement learning model for high-level semantic decision-making and define the state space. It is the combination of the semantic node where the agent is currently located and the target semantic node set, i.e. Action space Let the set of the next semantic nodes reachable from the current semantic node be defined; reward function. Designed as follows:

[0105]

[0106] in This is the current state. The action to be performed. A high-level decision-making model is trained using a Deep Q-Network (DQN) or a policy gradient algorithm (such as PPO) to maximize the cumulative reward. (in (using a discount factor) to learn the optimal semantic path from the current semantic node to the target semantic node. .

[0107] 2. Execution of underlying actions

[0108] The underlying action execution module, guided by the semantic path output from the high-level semantic decision, plans specific actions based on the agent's current position and environmental perception information. It defines the agent's state space. Including current position coordinates For planar coordinates, Orientation angle), sensor data (such as lidar scan data) and the next semantic node specified by the high-level semantic decision. Action space A set of discrete actions that an agent can perform, such as .

[0109] Construct a reinforcement learning model for underlying action execution, reward function Designed as follows:

[0110]

[0111] in This is the current state. The action to be performed is based on calculating the distance between the current position and the corresponding region of the target semantic node. And the reward value is determined by detecting whether there are obstacles in the sensor data.

[0112] The underlying action execution model is trained using the Deep Deterministic Policy Gradient (DDPG) algorithm or other continuous control reinforcement learning algorithms to learn the optimal action policy. This allows the agent to perform appropriate actions based on its current state, gradually moving towards the target semantic node. During execution, it continuously updates its state information and feeds it back to the high-level semantic decision-making module. If the current semantic path is found to be infeasible (e.g., encountering an insurmountable obstacle), the high-level semantic decision-making module is triggered to replan the path.

[0113] The innovation of this module lies in the following aspects: It separates semantic path planning from specific action execution through a hierarchical structure, reducing the complexity of path planning; in high-level semantic decision-making, it achieves semantic understanding and path planning of language instructions based on semantic topology maps and reinforcement learning, adapting to changes in semantic concepts across scenarios; and in low-level action execution, it combines environmental perception and reinforcement learning to achieve precise action control, improving the agent's navigation capabilities in complex environments. Compared to traditional path planning methods, this hierarchical path planning module exhibits greater robustness and flexibility in handling cross-scenario, language-instruction-based navigation tasks.

[0114] This specific embodiment is merely an explanation of the present invention and is not intended to limit the invention. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they are within the scope of the claims of the present invention.

Claims

1. A visual-semantic map construction method in cross-scene generalization, characterized by: Includes the following steps: (1) Adaptive visual feature extraction: By performing meta-training in multiple different source scenes, the network learns the scene-independent general visual feature extraction capability. In the target scene, the network is quickly fine-tuned using a small amount of data to extract visual features adapted to the target scene. (2) Semantic alignment and fusion: Construct a multimodal semantic space, map visual features, text semantic information and spatial topology information to a unified semantic space, use graph neural networks to model nodes and edges in the semantic space, calculate the similarity between visual features and text semantics in the semantic space to achieve semantic alignment, and combine spatial topology information to fuse semantic information into the visual map to establish a semantic topology map. (3) Hierarchical path planning: Based on the constructed semantic topology map, a hierarchical reinforcement learning algorithm is used for path planning. The navigation task is decomposed into two levels: high-level semantic decision-making and low-level action execution. The high-level semantic decision-making selects appropriate semantic target nodes in the semantic topology map according to language instructions. The low-level action execution generates specific action instructions to guide the agent to move based on the relative positional relationship between the current position and the target node.

2. The visual-semantic map construction method in cross-scene generalization according to claim 1, characterized in that: The meta-training phase specifically involves: collecting source scene data of different types and dividing them into corresponding meta-learning tasks, each task containing a support set and a query set; constructing a meta-learning feature extraction network based on a convolutional neural network; performing forward propagation to calculate feature representations on the support set of each task; performing classification prediction based on features and calculating the loss on the support set; updating network parameters using the gradient descent algorithm to obtain task-specific parameters; using the updated parameters to make predictions on the query set; and iteratively training the network with the goal of minimizing the expected loss of all task query sets.

3. The visual-semantic map construction method in cross-scene generalization according to claim 2, characterized in that: The target scene fine-tuning stage specifically involves: collecting a small amount of image data from the target scene to form a fine-tuning dataset; fixing the parameters of the bottom convolutional layers of the meta-training network; adjusting only the parameters of the high-level networks; and obtaining a feature extraction network adapted to the target scene by optimizing the fine-tuning loss function.

4. The visual-semantic map construction method in cross-scene generalization according to claim 1, characterized in that: The construction of the multimodal semantic space specifically involves: using a fully connected layer to map the visual feature vectors output by the visual feature extraction network to the semantic space; using a pre-trained language model to encode the semantic information of the text and mapping it to the semantic space through a fully connected layer; constructing a topology graph based on the agent's motion odometer data and sensor data, and mapping the node and edge features of the topology graph to the semantic space through an encoding function.

5. The visual-semantic map construction method in cross-scene generalization according to claim 4, characterized in that: The semantic alignment specifically involves: calculating the similarity score between each visual feature semantic node and all text semantic nodes; using a graph neural network to perform information propagation and feature updates on nodes in the semantic space; selecting the text semantic node with the highest similarity score and aligning it with the visual feature node; and temporarily not aligning if the highest score is lower than a set threshold.

6. The visual-semantic map construction method in cross-scene generalization according to claim 5, characterized in that: The semantic fusion specifically involves attaching the aligned semantic information as an attribute to the corresponding visual map node in the spatial topology graph, updating the weights of the edges in the topology graph based on the relationships between the semantic nodes, and obtaining the semantic topology map through multiple iterations.

7. The visual-semantic map construction method in cross-scene generalization according to claim 1, characterized in that: The high-level semantic decision-making process specifically involves: inputting natural language instructions into the natural language processing module, extracting a set of semantic keywords and semantic information of the target location, calculating the similarity between the keyword embedding vector and the semantic representation vector of the semantic node in the semantic topology map, and selecting nodes with similarity higher than a threshold to form a target node set. Define the state space, action space, and reward function for high-level semantic decision-making, and use reinforcement learning algorithms to train the optimal semantic path strategy from the current semantic node to the target semantic node.

8. The visual-semantic map construction method in cross-scene generalization according to claim 1, characterized in that: The underlying action execution specifically involves: defining the state space of the underlying action execution, including the agent's current position coordinates, sensor data, and the next semantic node specified by the higher layer; defining the action space as a set of discrete actions that the agent can execute; defining the reward function of the underlying action execution; using reinforcement learning algorithms to train and obtain the optimal action execution strategy that executes appropriate actions based on the current state; and during navigation, if the distance to an obstacle is detected to be less than a safety threshold, triggering the higher-level semantic decision module to replan the path.

Images