Robot vision language navigation method and device based on multi-modal knowledge graph

Abstract

The application provides a robot visual language navigation method and device based on a multi-modal knowledge graph, and relates to the technical field of visual language navigation. The method comprises the following steps: constructing a front map according to the mobile robot pose and the depth observation information to extract candidate front points; identifying semantic entities based on visual perception information, constructing nodes according to the semantic entities, semantic feature representations and spatial position information, and further constructing a multi-modal environment knowledge graph; generating target entity nodes and associated nodes through a text encoder for a natural language navigation instruction, and constructing a target semantic knowledge subgraph; defining the semantic value, distance weight and explorability of the front points, and selecting sub-targets for navigation from the candidate front points; introducing a proximal policy optimization algorithm to construct a path planning strategy based on reinforcement learning, and realizing the path planning of the mobile robot. The application can improve the navigation decision-making ability, exploration efficiency and task success rate of the mobile robot in an unknown environment.

Description

Technical Field

[0001] This invention relates to the field of visual language navigation technology, and in particular to a robot visual language navigation method and device based on multimodal knowledge graph. Background Technology

[0002] As the development of robotics technology becomes a crucial component driving intelligent manufacturing and the digital economy, the robotics industry boasts significant advantages, including immense development potential and broad application prospects. Among these, mobile robots, due to their unique mobility, are widely used across various industries and environments. Especially those mobile robots capable of performing tasks without human intervention play an irreplaceable role in exploring unknown environments, carrying out rescue missions, and working in hazardous conditions. In such applications, robots often need to autonomously complete target search and navigation tasks in unknown environments lacking prior maps, based on task instructions given by humans in natural language, recognizing visual observations, and performing these tasks. This type of problem is commonly referred to as the Vision-and-Language Navigation (VLN) problem.

[0003] Most existing visual-language navigation methods rely on imitation learning or reinforcement learning to jointly model visual information and language commands, training network parameters with large amounts of data to directly output navigation decisions in an end-to-end manner. With the development of large language models in recent years, many methods have emerged that use large-scale pre-trained models to reason and make decisions based on environmental semantics. These methods have improved robots' ability to understand open-vocabulary targets and complex language commands to some extent, but their decision-making process mainly relies on implicit feature representations, lacking explicit modeling of environmental and task knowledge, making logical reasoning and knowledge reuse difficult.

[0004] Furthermore, when performing goal-oriented navigation tasks in unknown environments, robots often need to balance exploration efficiency with decision-making accuracy. Existing methods mostly rely on frontier exploration strategies based on geometric information or heuristic scoring mechanisms based on model output to select candidate regions, making it difficult to fully utilize the high-level semantic relationships inherent in visual observations and the prior knowledge contained in language commands. When the environment structure is complex or the semantic associations of the target are hidden, problems such as redundant exploration paths, low navigation efficiency, and task failure can easily occur. Therefore, how to introduce a structured knowledge representation method that can uniformly represent multi-source information such as vision, language, and space in the visual-language navigation process, and how to use this knowledge to reason and guide mobile robots to autonomously explore and navigate, has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] To address the technical problems of existing visual language navigation methods, such as reliance on implicit feature representations in unknown environments and a lack of explicit modeling of environmental semantics and task knowledge, leading to insufficient reasoning ability, low navigation efficiency, and poor interpretability, this invention provides a robot visual language navigation method and device based on a multimodal knowledge graph. The technical solution is as follows: On the one hand, a robot visual language navigation method based on a multimodal knowledge graph is provided. This method is implemented by a mobile robot visual language navigation device and includes: S1. Obtain the pose of the mobile robot and the depth observation information collected by the depth camera. Construct a frontal map based on the pose and depth observation information. Extract candidate frontal points based on the frontal map to obtain a set of candidate frontal points.

[0006] S2. Obtain visual perception information observed by the mobile robot in the environment, identify semantic entities based on the visual perception information, construct nodes according to the semantic entities, semantic feature representations of the semantic entities and spatial location information, construct the association edges between nodes, and construct a multimodal environmental knowledge graph to describe the semantic state of the environment based on the nodes and association edges.

[0007] S3. Receive natural language navigation instructions input by the user, generate target entity nodes and associated nodes through the text encoder of the pre-trained visual language model, and construct a target semantic knowledge subgraph based on the target entity nodes and associated nodes.

[0008] S4. Obtain the semantic similarity between nodes based on the multimodal environment knowledge graph and the target semantic knowledge subgraph. Define the semantic value of the frontier points based on the semantic similarity. Define the distance weight and explorability of the frontier points. Construct a comprehensive evaluation function based on the semantic value, distance weight, and explorability. Select the sub-targets for navigation from the candidate frontier point set based on the comprehensive evaluation function.

[0009] S5. Introduce a proximal policy optimization algorithm to construct a path planning strategy based on reinforcement learning, and realize the path planning of the mobile robot according to the path planning strategy and sub-objectives.

[0010] Optionally, before S1, it also includes: Visual language navigation is modeled as a partially observable Markov decision process, as shown in equation (1): (1) In the formula, This represents the comprehensive evaluation function. Represents a set of environmental observation information. This indicates the historical background information of the intelligent agent. Represents the action space, Represents natural language navigation instructions. Indicates time Find the optimal candidate frontier point. Indicates time Next Observational information of candidate frontier points, Indicates the candidate front index, Indicates time, This indicates a single navigation action.

[0011] Optionally, in S1, candidate front point extraction based on the front map includes: By analyzing the boundaries between explored and unknown areas in the frontier map, boundary locations that meet the accessibility criteria are extracted as candidate frontier points.

[0012] Optionally, S2 includes: S21. Obtain visual perception information through the visual sensors mounted on the mobile robot itself.

[0013] S22. Use an open vocabulary target detection model to detect semantic entities from visual perception information, and obtain the candidate regions, category semantic information and confidence scores corresponding to each semantic entity.

[0014] S23. Use an image segmentation model to perform fine segmentation on the candidate region to obtain pixel-level masks of semantic entities.

[0015] S24. The pixel-level mask is used to encode features using a pre-trained visual language model to obtain a high-dimensional semantic feature vector of the semantic entity as a semantic feature representation.

[0016] S25. Obtain the spatial location information of semantic entities in the environment, and construct the semantic entities as nodes in the multimodal environment knowledge graph based on the semantic feature representation and spatial location information.

[0017] S26. Based on the spatial proximity or semantic similarity between semantic entities, construct associated edges between corresponding nodes; wherein, the spatial proximity is constructed based on the spatial location information of the semantic entities, and the semantic similarity is constructed based on the category semantic information and semantic feature representation of the semantic entities.

[0018] S27. Construct a multimodal environmental knowledge graph to describe the semantic state of the environment based on nodes and associated edges.

[0019] Optionally, S3 includes: S31. Receive the natural language navigation instruction input by the user, and determine whether the natural language navigation instruction is a word, a short phrase containing a clear target entity, or an instruction with a clear navigation constraint relationship. If it is a word or a short phrase containing a clear target entity, then execute step S32. If it is an instruction with a clear navigation constraint relationship, then execute step S33.

[0020] S32. Based on the natural language navigation instructions, candidate related entities are given through reasoning using a large language model; the natural language navigation instructions are input into the text encoder of a pre-trained visual language model to generate target entity nodes; the candidate related entities are input into the text encoder of a pre-trained visual language model to generate related nodes; and a target semantic knowledge subgraph is constructed based on the target entity nodes and related nodes.

[0021] S33. Perform target entity parsing on the target entities and their positional relationships in the natural language navigation instructions. Based on the target entity parsing results, infer candidate related entities through a large language model. Input the target entity parsing results into the text encoder of a pre-trained visual language model to generate target entity nodes. Input the candidate related entities into the text encoder of the pre-trained visual language model to generate related nodes. Construct a target semantic knowledge subgraph based on the target entity nodes and related nodes.

[0022] Optionally, S4 includes: S41. Calculate the semantic similarity between nodes based on the high-dimensional semantic feature vectors of semantic entities in the multimodal environment knowledge graph and the high-dimensional vectors of nodes obtained by the text encoder in the target semantic knowledge subgraph.

[0023] S42. Define the neighborhood semantic entity set of the front point based on the candidate front point set and semantic entities, and define the semantic value of the front point based on the neighborhood semantic entity set and semantic similarity.

[0024] S43. Define the distance weight of the leading edge point based on the geometric distance between the robot and the leading edge point.

[0025] S44. Define the explorability of a frontier point based on its potential to bring new environmental information to the robot after it has been explored.

[0026] S45. Construct a comprehensive evaluation function based on the semantic value, distance weight, and explorability of the frontier points, and select sub-targets for navigation from the candidate frontier point set based on the comprehensive evaluation function.

[0027] On the other hand, a robot visual language navigation device based on a multimodal knowledge graph is provided. This device is applied to a robot visual language navigation method based on a multimodal knowledge graph. The device includes: The candidate front point set construction module is used to acquire the pose of the mobile robot and the depth observation information collected by the depth camera. Based on the pose and depth observation information, a front map is constructed, and candidate front points are extracted based on the front map to obtain the candidate front point set.

[0028] The multimodal environment knowledge graph construction module is used to acquire visual perception information observed by the mobile robot in the environment, identify semantic entities based on the visual perception information, construct nodes according to the semantic entities, semantic feature representations of the semantic entities and spatial location information, construct the association edges between nodes, and construct a multimodal environment knowledge graph to describe the semantic state of the environment based on the nodes and association edges.

[0029] The target semantic knowledge subgraph construction module is used to receive natural language navigation instructions input by the user, generate target entity nodes and associated nodes through the text encoder of the pre-trained visual language model, and construct the target semantic knowledge subgraph based on the target entity nodes and associated nodes.

[0030] The sub-target selection module is used to obtain the semantic similarity between nodes based on the multimodal environment knowledge graph and the target semantic knowledge subgraph, define the semantic value of the frontier points based on the semantic similarity, define the distance weight and explorability of the frontier points, construct a comprehensive evaluation function based on the semantic value, distance weight and explorability, and select sub-targets for navigation from the candidate frontier point set based on the comprehensive evaluation function.

[0031] The output module is used to introduce a proximal policy optimization algorithm to construct a path planning policy based on reinforcement learning, and to realize the path planning of the mobile robot according to the path planning policy and sub-objectives.

[0032] On the other hand, a mobile robot visual language navigation device is provided, the mobile robot visual language navigation device comprising: a processor; a memory, the memory storing computer-readable instructions, which, when executed by the processor, implement any of the methods described above for robot visual language navigation based on multimodal knowledge graphs.

[0033] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, the at least one instruction being loaded and executed by a processor to implement any of the above-described robot vision-language navigation methods based on multimodal knowledge graphs.

[0034] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: In this invention, a multimodal knowledge graph integrating visual information, spatial location information, and natural language instructions is constructed, realizing a structured representation of environmental semantics and task objectives. This overcomes the problems of existing visual language navigation methods that rely on implicit feature representation and lack explicit knowledge modeling, and is conducive to improving the interpretability of navigation decisions.

[0035] This invention uses a multimodal knowledge graph to reason about the semantic relationships between environmental semantic entities and navigation targets, and uses the reasoning results to perform semantic evaluation of navigation frontier points. This enables mobile robots to conduct more goal-oriented autonomous exploration in unknown environments, thereby improving navigation efficiency and task success rate.

[0036] This invention dynamically constructs and updates a multimodal knowledge graph during navigation, enabling the robot to continuously integrate newly acquired environmental semantic information, reduce redundant exploration and invalid paths, and enhance the system's stability and adaptability in complex environments. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart of a robot vision-language navigation method based on a multimodal knowledge graph, provided by an embodiment of the present invention. Figure 2 This is an overall structural diagram of a robot vision-language navigation method based on multimodal knowledge graphs provided in an embodiment of the present invention; Figure 3 This is a flowchart of the frontier map construction process provided in the embodiments of the present invention; Figure 4 This is a flowchart of the multimodal environment knowledge graph construction process provided in this embodiment of the invention; Figure 5 This is a CLIP image encoding flowchart provided in an embodiment of the present invention; Figure 6 This is a flowchart of the target semantic knowledge subgraph construction process provided in the embodiments of the present invention; Figure 7 This is a CLIP text encoding flowchart provided in an embodiment of the present invention; Figure 8 This is a structural diagram of the self-attention mechanism provided in an embodiment of the present invention; Figure 9 This is a multi-head attention structure diagram provided in an embodiment of the present invention; Figure 10 This is a schematic diagram of the PPO algorithm provided in an embodiment of the present invention; Figure 11 This is a block diagram of a robot visual language navigation device based on a multimodal knowledge graph, provided in an embodiment of the present invention. Figure 12This is a structural schematic diagram of a mobile robot visual language navigation device provided in an embodiment of the present invention. Detailed Implementation

[0039] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0040] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0041] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0042] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0043] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0044] This invention provides a robot visual language navigation method based on a multimodal knowledge graph. This method can be implemented by a mobile robot visual language navigation device, which can be a terminal or a server. Figure 1 , Figure 2 The flowchart shown is for a robot vision-language navigation method based on a multimodal knowledge graph. The processing flow of this method may include the following steps: S1. Obtain the pose of the mobile robot and the depth observation information collected by the depth camera. Construct a frontal map based on the pose and depth observation information. Extract candidate frontal points based on the frontal map to obtain a set of candidate frontal points.

[0045] In one feasible implementation, the present invention first performs visual language navigation modeling and frontier point map construction.

[0046] Specifically, visual language navigation can be modeled as a partially observable Markov Decision Process (POMDP). In a Markov Decision Process, future observations are only related to the current state. It is relevant and independent of all past states. For a given state containing One word instruction Natural Language The agent needs to navigate the corresponding navigation topology to reach the designated target point, guided by natural language. At each moment... An intelligent agent can obtain observational information about an environment. Under observation at every moment, there will be There are 3 different candidate viewpoints (i.e., candidate frontier points), denoted as _____. , , For candidate viewpoint index, .in and Indicates the first Visual features and location information from various perspectives, in this invention This is defined as the decision space for the agent. Therefore, the agent only needs to navigate within this decision space. Candidate navigation viewpoints can be selected from the available information. For a navigation trajectory of an agent, all observation information... and historical actions performed , The action space represents the historical context of the agent. During the agent's navigation process, it predicts the next optimal navigable viewpoint based on natural language instructions, historical information, and current observations, and executes navigation actions to ultimately reach the target point. This can be represented as: (1) In the formula, This represents the comprehensive evaluation function. Represents a set of environmental observation information. This indicates the historical background information of the intelligent agent. Represents the action space. Represents natural language navigation instructions. Indicates time Find the optimal candidate frontier point. Indicates time Next Observational information of candidate frontier points, Indicates the candidate front index, Indicates time, This indicates a single navigation action.

[0047] Furthermore, a frontier-based exploration strategy is adopted for acquiring the navigation viewpoint. In the specific implementation, the mobile robot uses its own odometry information to obtain its current pose and combines this with depth observation information continuously acquired by the depth camera. The 3D point cloud at the current viewpoint is then filtered and projected onto a 2D plane, thereby constructing an online occupancy grid map of the environment. The occupancy grid map distinguishes between traversable areas, obstacle areas, and unexplored unknown areas in the environment. Based on the occupancy grid map, the boundaries between explored and unknown areas are analyzed, and boundary locations that meet the accessibility criteria are extracted as candidate frontier points. These candidate frontier points represent potential spatial locations that the robot can preferentially explore in the current navigation state and constitute a set of candidate frontier points. The frontier map construction process is as follows: Figure 3 As shown.

[0048] In this context, candidate front points serve only as candidate spatial locations for navigation decisions and do not directly determine the robot's final trajectory. Subsequent steps will combine semantic entity information from the multimodal knowledge graph with target semantic constraints to perform semantic association reasoning and target relevance evaluation on the candidate front points, thereby selecting the front point that best matches the natural language navigation target from among multiple candidate front points as the navigation sub-target.

[0049] S2. Obtain visual perception information observed by the mobile robot in the environment, identify semantic entities based on the visual perception information, construct nodes according to the semantic entities, semantic feature representations of the semantic entities and spatial location information, construct the association edges between nodes, and construct a multimodal environmental knowledge graph to describe the semantic state of the environment based on the nodes and association edges.

[0050] In one feasible implementation, after step S1, the present invention identifies semantic entities based on visual perception information and constructs a multimodal environment knowledge graph.

[0051] Specifically, during mobile robot navigation, to structurally store and continuously update semantic information in the environment, a multimodal environmental knowledge graph is constructed to represent the semantic entities in the environment observed by the robot and their relationships. The knowledge graph is represented in graph structure, where nodes describe semantic entities in the environment. Node attributes include the entity's spatial location information, category semantic information, and semantic feature representation; connections between nodes represent spatial or semantic relationships between different semantic entities. The construction process of the multimodal environmental knowledge graph is as follows: Figure 4 As shown.

[0052] Optionally, step S2 above may include the following steps S21-S27: S21. Obtain visual perception information through the visual sensors mounted on the mobile robot itself.

[0053] In one feasible implementation, the mobile robot acquires an RGB-D image from its current perspective using its onboard visual sensors as visual perception information.

[0054] S22. Use an open vocabulary target detection model to detect semantic entities from visual perception information, and obtain the candidate regions, category semantic information and confidence scores corresponding to each semantic entity.

[0055] In one feasible implementation, based on visual perception information, the Grounding Dino open vocabulary object detection model is used to detect semantic entities in the image, and to obtain the candidate regions, category information and confidence scores corresponding to each semantic entity.

[0056] S23. The candidate region is finely segmented using an image segmentation model to obtain pixel-level masks of semantic entities.

[0057] In one feasible implementation, the detected candidate regions are finely segmented using an image segmentation model (SegmentAnything Model, SAM) to obtain pixel-level masks of the corresponding semantic entities, thereby separating the semantic entities from the background information.

[0058] S24. The pixel-level mask is used to encode features using a pre-trained visual language model to obtain a high-dimensional semantic feature vector of the semantic entity as a semantic feature representation.

[0059] In one feasible implementation, after the detection and segmentation of semantic entities are completed, a pre-trained visual language model (Contrastive Language-Image Pre-training, CLIP) is used to encode the visual region of each semantic entity, mapping it into a high-dimensional semantic feature vector, and the high-dimensional semantic feature vector is used as the semantic attribute of the corresponding semantic entity node.

[0060] Specifically, the CLIP image coding structure is as follows: Figure 5 As shown. The image encoder section adopts a VisionTransform structure. The encoder receives the input image. and adjust it to a fixed size. Decompose the input image into There are 1 image patch, each image patch being 1. At the same time, ensure the width of the adjusted image. and height Can be Divisible by integer. Then linearly project the image patch as... 3D vectors, also known as patch embedding. This represents the hidden dimensions of all layers in the Transformer. Compared to traditional Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTMs), since the model does not know the location information of image patches in the data, it is necessary to use position embeddings to represent the position of image patches in the image. ViT uses learnable position embedding vectors, which are added to the corresponding image patches. After extracting image features, the encoder encodes them into high-dimensional feature vectors, thus obtaining the semantic feature vector of the current image.

[0061] S25. Obtain the spatial location information of semantic entities in the environment, and construct the semantic entities as nodes in the multimodal environment knowledge graph based on the semantic feature representation and spatial location information.

[0062] In one feasible implementation, semantic entities are constructed as nodes in a multimodal environmental knowledge graph by combining their spatial location information in the environment.

[0063] S26. Based on the spatial proximity or semantic similarity between semantic entities, construct associated edges between corresponding nodes; wherein, the spatial proximity is constructed based on the spatial location information of the semantic entities, and the semantic similarity is constructed based on the category semantic information and semantic feature representation of the semantic entities.

[0064] In one feasible implementation, based on the spatial proximity or semantic similarity between semantic entities, an association edge is constructed between corresponding nodes to represent the degree of association between different semantic entities.

[0065] Spatial proximity is calculated by combining the current detection results and depth information with the object's pose to obtain the 3D coordinates of the object's center. Spatial proximity is obtained by comparing the 3D coordinates of different objects. Based on the category information and semantic feature representation of semantic entities, the degree of association between different semantic entities in the semantic space is evaluated. When the degree of association meets the preset conditions, semantic association edges are constructed between the corresponding semantic entity nodes.

[0066] S27. Construct a multimodal environmental knowledge graph to describe the semantic state of the environment based on nodes and associated edges.

[0067] Furthermore, as the mobile robot continues to explore the environment, when new semantic entities are observed or the attributes of existing semantic entities are updated, the multimodal environmental knowledge graph is dynamically updated online, thereby forming an environmental semantic representation that is gradually improved as the exploration process progresses, providing support for subsequent semantic reasoning and navigation decisions based on natural language instructions.

[0068] S3. Receive natural language navigation instructions input by the user, generate target entity nodes and associated nodes through the text encoder of the pre-trained visual language model, and construct a target semantic knowledge subgraph based on the target entity nodes and associated nodes.

[0069] In one feasible implementation, after step S2, the present invention parses natural language navigation instructions and constructs a target semantic knowledge subgraph.

[0070] Specifically, based on the completion of the multimodal environmental knowledge graph construction, to enable the mobile robot to perform target-oriented reasoning and filtering of environmental semantic information based on user-input natural language navigation commands, this invention further utilizes the text encoding part of the CLIP model to perform semantic parsing of the natural language navigation commands and maps them into high-dimensional semantic vectors as attributes of the target subgraph nodes. Furthermore, prompt words are provided to the large language model, allowing it to provide a set of candidate semantic entities related to the target object, thus constructing the target semantic knowledge subgraph. The target semantic knowledge subgraph construction process is as follows: Figure 6 As shown.

[0071] Optionally, step S3 above may include the following steps S31-S33: S31. Receive the natural language navigation instruction input by the user, and determine whether the natural language navigation instruction is a word, a short phrase containing a clear target entity, or an instruction with a clear navigation constraint relationship. If it is a word or a short phrase containing a clear target entity, then execute step S32. If it is an instruction with a clear navigation constraint relationship, then execute step S33.

[0072] S32. Based on the natural language navigation instructions, candidate related entities are given through reasoning using a large language model; the natural language navigation instructions are input into the text encoder of a pre-trained visual language model to generate target entity nodes; the candidate related entities are input into the text encoder of a pre-trained visual language model to generate related nodes; and a target semantic knowledge subgraph is constructed based on the target entity nodes and related nodes.

[0073] S33. Perform target entity parsing on the target entities and their positional relationships in the natural language navigation instructions. Based on the target entity parsing results, infer candidate related entities through a large language model. Input the target entity parsing results into the text encoder of a pre-trained visual language model to generate target entity nodes. Input the candidate related entities into the text encoder of the pre-trained visual language model to generate related nodes. Construct a target semantic knowledge subgraph based on the target entity nodes and related nodes.

[0074] In one feasible implementation, natural language instructions are divided into two types. The first type consists of simple words or short phrases containing a clear target object, such as: pillow, TV, table, or "there is a {object}". This simple text information is directly encoded into target entity nodes through CLIP text encoding. There is no sequential or spatial relationship between these nodes and other nodes. Candidate associated objects are given through large-scale model inference, and after encoding, associated nodes are generated and connected to form a target semantic knowledge subgraph. The second type of instruction has explicit navigation constraints, such as "walk to the dining room and find a bottle of water on the table". For this type of instruction, the target entity and the implicit positional relationship in the instruction need to be parsed first. This can be done by using a pre-trained large-scale language model to parse it into a triple <head entity, relation, tail entity>. After encoding, target nodes and associated nodes are constructed, which together form a target semantic knowledge subgraph. The relationship between different entities is a spatial positional relationship or a semantic association relationship.

[0075] CLIP's text encoder employs a Decoder-Only Transformer architecture for semantic modeling of natural language descriptions. First, the input text is segmented using Byte Pair Encoding (BPE) and mapped to corresponding word vectors, while learnable positional embeddings are introduced to preserve sequence order information. Subsequently, the embedded text sequence is fed into a multi-layer Transformer encoder to model the contextual dependencies between words. Unlike traditional methods that use classification tags as sentence representations, CLIP selects the hidden state corresponding to the End of Text (EOT) tag at the end of the sequence as the semantic representation of the entire sentence. Finally, this semantic feature is linearly projected and normalized to obtain a high-dimensional vector of text features, which is then mapped to a multimodal semantic embedding space shared with the image encoder, thereby achieving semantic alignment between text and visual information. The specific structure is as follows: Figure 7 As shown.

[0076] Both the image encoder and text encoder components consist of multi-head attention and a feedforward network (FFN). The self-attention mechanism is structured as follows: Figure 8 As shown, through calculation query s and keys The similarity between them is measured by the dot product between them, and this similarity is used as a weight to calculate the weighted value. MatMul represents matrix multiplication, the Scale operation is used to scale the dot product result, and the SoftMax function is defined as follows: (2) In the formula, Indicates the first The output of each node, This indicates the number of output nodes.

[0077] From the above structure, we know the output of the self-attention mechanism: (3) In the formula, Representation matrix Dimensions.

[0078] To allow the model to focus on different aspects of information, the model is divided into multiple heads, forming multiple subspaces. Multi-head attention is formed by combining multiple self-attention mechanisms, such as... Figure 9 As shown. Where Linear represents a linear layer. The number of self-attention mechanisms is used; the Concat layer is used to implement tensor concatenation.

[0079] The formula for calculating the multi-head attention mechanism is as follows: (4) (5) (6) In the formula, , , , This represents the parameter matrix.

[0080] A feedforward neural network (FFN) consists of two fully connected layers. The first layer uses the ReLU activation function, while the second layer does not use an activation function. The corresponding formula is as follows: (7) In the formula, , Represents the parameter matrix, , Indicates bias.

[0081] S4. Obtain the semantic similarity between nodes based on the multimodal environment knowledge graph and the target semantic knowledge subgraph. Define the semantic value of the frontier points based on the semantic similarity. Define the distance weight and explorability of the frontier points. Construct a comprehensive evaluation function based on the semantic value, distance weight, and explorability. Select the sub-targets for navigation from the candidate frontier point set based on the comprehensive evaluation function.

[0082] In one feasible implementation, after step S3, the present invention performs semantic association reasoning and navigation sub-target selection based on multimodal knowledge graph.

[0083] Specifically, based on the construction of a multimodal environmental knowledge graph and a target semantic knowledge subgraph, semantic association reasoning is performed based on the multimodal environmental knowledge graph to select sub-targets for navigation from candidate frontier points in the frontier map. Specifically, the target semantic knowledge subgraph obtained from parsing natural language navigation instructions is used as a semantic reference to analyze the association relationships between each semantic entity node in the environmental knowledge graph and the target semantics. Three scoring strategies are employed to comprehensively evaluate the candidate frontier point set: semantic similarity, distance weight, and explorability. A comprehensive evaluation function is constructed to select the frontier point with the highest value as the navigation sub-target.

[0084] Optionally, step S4 above may include the following steps S41-S45: S41. Calculate the semantic similarity between nodes based on the high-dimensional semantic feature vectors of semantic entities in the multimodal environment knowledge graph and the high-dimensional vectors of nodes obtained by the text encoder in the target semantic knowledge subgraph.

[0085] In one feasible implementation, in the multimodal environment knowledge graph constructed in step S2, the attributes of each entity node in the graph include a high-dimensional feature vector mapped using an image encoder, denoted as: (8) For the target semantic knowledge subgraph constructed in step S3, the attributes of each node in the subgraph are obtained by the text encoder as a high-dimensional vector of text features, denoted as: (9) In the formula, Indicates the first Image regions corresponding to each semantic entity This represents natural language navigation instructions or target descriptions. This represents the semantic embedding dimension. Semantically related images and text have high vector similarity in the same embedding space. Vector similarity is used as semantic similarity to measure the relevance between semantic entities in the environment and natural language targets in the same semantic space. The higher the value, the more semantically consistent the entity is with the navigation target. For environmental semantic entity nodes... semantic nodes of the target subgraph Its semantic similarity is defined as: (10) S42. Define the neighborhood semantic entity set of the front point based on the candidate front point set and semantic entities, and define the semantic value of the front point based on the neighborhood semantic entity set and semantic similarity.

[0086] In one feasible implementation, the robot selects targets based on a leading edge point map during navigation and exploration. The leading edge points themselves lack semantic information; their semantic value is indirectly obtained through nearby semantic entities. An association model between leading edge points and semantic entities is used, where the set of leading edge points is defined as follows: (11) Semantic entity point set: (12) Define the frontier point The set of semantic entities in the neighborhood: (13) In the formula, Indicates the spatial location of semantic entities. Indicates the spatial location of the leading edge point. This represents the neighborhood radius. In the semantic scoring strategy for frontier points, a goal-driven maximum semantic relevance is adopted, meaning that a frontier point has high semantic value if at least one highly relevant entity exists nearby. This is defined as: (14) S43. Define the distance weight of the leading edge point based on the geometric distance between the robot and the leading edge point.

[0087] In one feasible implementation, considering the efficiency of actual robot navigation and exploration of the environment, a distance weight is introduced as a factor in the scoring strategy. The robot will comprehensively consider the geometric distance of different leading edge points from its own position. Leading edge points with shorter distances will receive a larger distance weight. This consideration is particularly important for actual robots, as shorter travel distances can significantly reduce energy consumption and prevent unnecessary roaming.

[0088] Define the distance weight as follows: (15) In the formula, This represents the path distance from the robot's current position to the candidate front edge point. This represents the distance attenuation coefficient.

[0089] S44. Define the explorability of a frontier point based on its potential to bring new environmental information to the robot after it has been explored.

[0090] In one feasible implementation, explorability measures the potential of a frontier point to provide the robot with new environmental information after exploration, reflecting the scale and uncertainty of the unknown space corresponding to that frontier point. To avoid the robot exploring the same area multiple times, it is encouraged to expand the exploration range into unknown areas, covering as many areas as possible and reducing uncertainty during the exploration process. The explorability score is defined as: (16) In the formula, This represents the set of unknown grid cells in the neighborhood of the leading edge point.

[0091] S45. Construct a comprehensive evaluation function based on the semantic value, distance weight, and explorability of the frontier points, and select sub-targets for navigation from the candidate frontier point set based on the comprehensive evaluation function.

[0092] In one feasible implementation, the comprehensive evaluation function of the frontier point can be obtained by combining semantic similarity, distance weight, and explorability: (17) In the formula, the sum of the weights satisfies Based on the comprehensive scoring results, the highest-scoring front point is selected from the candidate front points on the front map as the navigation sub-target at the current moment, which is used to guide the mobile robot to perform subsequent path planning and autonomous navigation control.

[0093] S5. Introduce a proximal policy optimization algorithm to construct a path planning strategy based on reinforcement learning, and realize the path planning of the mobile robot according to the path planning strategy and sub-objectives.

[0094] In one feasible implementation, after step S4, the present invention implements mobile robot path planning based on a near-end strategy optimization algorithm.

[0095] Specifically, after selecting navigation sub-target points from the frontier map based on the multimodal knowledge graph and comprehensive evaluation function, in order to enable the robot to reach the given sub-target points safely, smoothly and efficiently in complex and unknown environments, the Proximal Policy Optimization (PPO) algorithm is introduced to construct a path planning strategy based on reinforcement learning.

[0096] The PPO algorithm is a policy gradient-based algorithm that integrates a dual-network structure using an Actor-Critic architecture. Compared to reinforcement learning algorithms that use confidence region-based policy optimization for step size selection, the PPO algorithm has lower computational complexity, faster training speed, and greater feasibility. The basic block diagram of the algorithm is shown below. Figure 10 As shown. The PPO algorithm based on the Actor-Critic framework consists of two networks: the Actor network and the Critic network. The Actor network is responsible for generating the policy, and its network parameters are... The policy function is The Critic network is used to estimate the state value function. , indicating from state The expected reward that can be obtained by taking the action. To measure the advantage of the action relative to the current strategy, an advantage function is introduced: (18) The objective function of the Actor network is: (19) (20) In the formula, Represents the shearing function. Indicates the shearing parameter. express The expected function of the next sampling. This represents the policy network to be optimized. This represents the policy network currently used to collect data, which estimates the new policy through importance sampling. The closer the ratio of the two is to 1, the smaller the offset between the old and new policies, and the higher the similarity between the policies. During the update process, the PPO algorithm utilizes parameters... This limits the magnitude of policy updates. When the offset between the old and new policies is too large, a pruning term is used instead. This ensures that the deviation between the old and new policies is not too great, allowing the Actor network to update in a relatively smooth way and converge faster.

[0097] To fully characterize the information required for the robot's current navigation decision, the state It consists of the following parts: (twenty one) In the formula, Indicates the robot's current pose. This indicates the position of the sub-target point relative to the robot. This represents the robot's local environmental perception information. This state design enables the policy network to comprehensively consider geometric structure and environmental safety under sub-objective constraints. The action space is defined as the robot's executable motion control commands, which take the form of continuous actions. (twenty two) In the formula, Indicates linear velocity. It represents angular velocity.

[0098] For the design of the reward function, considering target proximity, path safety, and motion smoothness, the following reward function is designed, where... The weighting coefficients can be adjusted accordingly: (twenty three) Reward for reaching the target: To enable the local path strategy to converge to the local target point more quickly, during training, if the robot successfully reaches the local target point, it means that the current strategy is closer to the target strategy, and an additional positive reward value is given to accelerate the training speed of the algorithm. The reward function for reaching the target is designed as follows: (twenty four) Distance: During navigation, the distance to the local target point needs to be gradually reduced. A corresponding distance reward function is designed based on the change in distance to the local target point after executing the current strategy action. Let the robot's coordinates at the current moment be... Execute the current strategy to obtain the action. The rear coordinates are The coordinates of the local target point are The change in distance from the target point is: (25) Angle: Correct navigation direction is a prerequisite for the robot to reach the target point. To prevent reward sparsity when the robot adjusts its direction, a reward is given based on the change in the relative angle between the robot and the local target point before and after the robot performs the action. Let the relative angle between the robot and the local target point at the current moment be θ. After executing the current strategy, the resulting action becomes The corresponding reward is: (26) Smoothness penalty: Used to prevent large steering changes during robot movement, improving path smoothness. (27) The PPO algorithm training process is as follows: 1. Initialize the parameters of the policy network and value network; 2. In the simulation environment, the robot interacts with the environment based on the current strategy and collects trajectory data. ; 3. Calculate the reward using the collected trajectory. and dominance function ; 4. Update the policy network parameters by truncating the objective function to maximize PPO; 5. Update the value network parameters by minimizing the error of the value function; 6. Repeat the above process until the strategy converges.

[0099] This invention addresses the autonomous navigation tasks of mobile robots in unknown and complex environments based on natural language commands, and is particularly suitable for indoor scenarios such as residences, office buildings, shopping malls, and airports where prior maps are unavailable or the environment is dynamically changing. By introducing a multimodal knowledge graph, a unified structured representation and relational modeling of the visual information, spatial location information, and natural language commands perceived by the mobile robot are performed. Based on the graph reasoning results, the invention guides the exploration of the frontier and path planning, enabling efficient decision-making for goal-oriented navigation tasks in unknown environments.

[0100] This invention discloses a robot visual language navigation method based on a multimodal knowledge graph. The main contents and steps are as follows: First, acquire visual perception information observed by the mobile robot in the environment and receive natural language navigation commands input by the user. Second, identify semantic entities in the environment based on the visual perception information and, combined with the spatial location information of the semantic entities, construct a multimodal environmental knowledge graph to describe the semantic state of the environment. Semantic entities serve as graph nodes, and node attributes include category semantic information, spatial location information, and semantic feature representation. Third, perform semantic parsing on the navigation commands to extract the navigation target and its related semantic constraints, constructing a target semantic knowledge subgraph. Fourth, based on the environmental semantic entities, target semantic nodes, and various relationships between them in the multimodal knowledge graph, design a comprehensive evaluation function to comprehensively score candidate navigation frontier points. Fifth, select the navigation sub-target point with the highest value and design a proximal optimization algorithm to control the mobile robot to perform autonomous exploration and path planning until the navigation task is completed.

[0101] This invention constructs a multimodal knowledge graph by performing unified structured modeling of visual information, spatial location information, and natural language navigation instructions perceived by a mobile robot. Based on the semantic associations and reasoning results of the knowledge graph, the robot is guided to conduct goal-oriented autonomous exploration and navigation, thereby improving the mobile robot's navigation decision-making ability, exploration efficiency, and task success rate in unknown environments.

[0102] In this embodiment of the invention, a multimodal knowledge graph integrating visual information, spatial location information, and natural language instructions is constructed, realizing a structured representation of environmental semantics and task objectives. This overcomes the problems of existing visual language navigation methods relying on implicit feature representation and lacking explicit knowledge modeling, and is conducive to improving the interpretability of navigation decisions.

[0103] This invention uses a multimodal knowledge graph to reason about the semantic relationships between environmental semantic entities and navigation targets, and uses the reasoning results to perform semantic evaluation of navigation frontier points. This enables mobile robots to conduct more goal-oriented autonomous exploration in unknown environments, thereby improving navigation efficiency and task success rate.

[0104] This invention dynamically constructs and updates a multimodal knowledge graph during navigation, enabling the robot to continuously integrate newly acquired environmental semantic information, reduce redundant exploration and invalid paths, and enhance the system's stability and adaptability in complex environments.

[0105] Figure 11 This is a block diagram illustrating a robot visual language navigation device based on a multimodal knowledge graph, according to an exemplary embodiment. The device is used in a robot visual language navigation method based on a multimodal knowledge graph. (Refer to...) Figure 11 The device includes a candidate frontier point set construction module 310, a multimodal environment knowledge graph construction module 320, a target semantic knowledge subgraph construction module 330, a sub-target selection module 340, and an output module 350. Wherein: The candidate front point set construction module 310 is used to acquire the pose of the mobile robot and the depth observation information collected by the depth camera, construct a front map based on the pose and depth observation information, extract candidate front points based on the front map, and obtain a candidate front point set.

[0106] The multimodal environment knowledge graph construction module 320 is used to acquire visual perception information observed by the mobile robot in the environment, identify semantic entities based on the visual perception information, construct nodes according to the semantic entities, semantic feature representations of the semantic entities and spatial location information, construct the association edges between the nodes, and construct a multimodal environment knowledge graph to describe the semantic state of the environment based on the nodes and association edges.

[0107] The target semantic knowledge subgraph construction module 330 is used to receive natural language navigation instructions input by the user, generate target entity nodes and associated nodes through the text encoder of the pre-trained visual language model, and construct the target semantic knowledge subgraph based on the target entity nodes and associated nodes.

[0108] The sub-target selection module 340 is used to obtain the semantic similarity between nodes based on the multimodal environment knowledge graph and the target semantic knowledge subgraph, define the semantic value of the frontier point based on the semantic similarity, define the distance weight and explorability of the frontier point, construct a comprehensive evaluation function based on the semantic value, distance weight and explorability, and select the sub-target for navigation from the candidate frontier point set based on the comprehensive evaluation function.

[0109] Output module 350 is used to introduce a proximal policy optimization algorithm to construct a path planning strategy based on reinforcement learning, and to realize the path planning of the mobile robot according to the path planning strategy and sub-objectives.

[0110] In this embodiment of the invention, a multimodal knowledge graph integrating visual information, spatial location information, and natural language instructions is constructed, realizing a structured representation of environmental semantics and task objectives. This overcomes the problems of existing visual language navigation methods relying on implicit feature representation and lacking explicit knowledge modeling, and is conducive to improving the interpretability of navigation decisions.

[0111] This invention uses a multimodal knowledge graph to reason about the semantic relationships between environmental semantic entities and navigation targets, and uses the reasoning results to perform semantic evaluation of navigation frontier points. This enables mobile robots to conduct more goal-oriented autonomous exploration in unknown environments, thereby improving navigation efficiency and task success rate.

[0112] This invention dynamically constructs and updates a multimodal knowledge graph during navigation, enabling the robot to continuously integrate newly acquired environmental semantic information, reduce redundant exploration and invalid paths, and enhance the system's stability and adaptability in complex environments.

[0113] Figure 12 This is a structural schematic diagram of a mobile robot visual language navigation device provided in an embodiment of the present invention, as shown below. Figure 12 As shown, the mobile robot visual language navigation device may include the above-mentioned Figure 11 The illustrated robot visual language navigation device is based on a multimodal knowledge graph. Optionally, the mobile robot visual language navigation device 410 may include a first processor 2001.

[0114] Optionally, the mobile robot visual language navigation device 410 may also include a memory 2002 and a transceiver 2003.

[0115] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0116] The following is combined Figure 12 A detailed description of each component of the mobile robot visual language navigation device 410 is provided below: The first processor 2001 is the control center of the mobile robot visual language navigation device 410. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0117] Optionally, the first processor 2001 can perform various functions of the mobile robot visual language navigation device 410 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0118] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 12 CPU0 and CPU1 are shown in the diagram.

[0119] In a specific implementation, as one example, the mobile robot visual language navigation device 410 may also include multiple processors, for example... Figure 12 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0120] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0121] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be accessed through the interface circuit of the mobile robot visual language navigation device 410. Figure 12 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0122] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0123] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 12 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0124] Optionally, the transceiver 2003 can be integrated with the first processor 2001, or it can exist independently and be connected to the interface circuit of the mobile robot visual language navigation device 410. Figure 12 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0125] It should be noted that, Figure 12 The structure of the mobile robot visual language navigation device 410 shown does not constitute a limitation on the router. Actual knowledge structure recognition devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0126] Furthermore, the technical effects of the mobile robot visual language navigation device 410 can be referenced from the technical effects of the robot visual language navigation method based on multimodal knowledge graph described in the above method embodiments, and will not be repeated here.

[0127] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0128] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0129] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0130] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0131] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0132] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0133] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0134] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0135] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0136] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0137] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0138] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0139] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A robot visual language navigation method based on multimodal knowledge graph, characterized in that, The method includes: S1. Obtain the pose of the mobile robot and the depth observation information collected by the depth camera. Construct a frontal map based on the pose and depth observation information. Extract candidate frontal points based on the frontal map to obtain a set of candidate frontal points. S2. Obtain visual perception information observed by the mobile robot in the environment, identify semantic entities based on visual perception information, construct nodes according to semantic entities, semantic feature representations of semantic entities and spatial location information, construct the association edges between nodes, and construct a multimodal environmental knowledge graph to describe the semantic state of the environment based on nodes and association edges. S3. Receive natural language navigation instructions input by the user, generate target entity nodes and associated nodes through the text encoder of the pre-trained visual language model, and construct a target semantic knowledge subgraph based on the target entity nodes and associated nodes. S4. Obtain the semantic similarity between nodes based on the multimodal environment knowledge graph and the target semantic knowledge subgraph. Define the semantic value of the frontier point based on the semantic similarity. Define the distance weight and explorability of the frontier point. Construct a comprehensive evaluation function based on the semantic value, distance weight, and explorability. Select the sub-target for navigation from the candidate frontier point set based on the comprehensive evaluation function. S5. Introduce a proximal policy optimization algorithm to construct a path planning strategy based on reinforcement learning, and realize the path planning of the mobile robot according to the path planning strategy and sub-objectives.

2. The robot visual language navigation method based on multimodal knowledge graph according to claim 1, characterized in that, Before S1, it also includes: Visual language navigation is modeled as a partially observable Markov decision process, as shown in equation (1): （1） In the formula, This represents the comprehensive evaluation function. Represents a set of environmental observation information. This indicates the historical background information of the intelligent agent. Represents the action space, Represents natural language navigation instructions. Indicates time Find the optimal candidate frontier point. Indicates time Next Observational information of candidate frontier points, Indicates the candidate front index, Indicates time, This indicates a single navigation action.

3. The robot visual language navigation method based on multimodal knowledge graph according to claim 1, characterized in that, The step S1, extracting candidate frontier points based on the frontier map, includes: By analyzing the boundaries between explored and unknown areas in the frontier map, boundary locations that meet the accessibility criteria are extracted as candidate frontier points.

4. The robot visual language navigation method based on multimodal knowledge graph according to claim 1, characterized in that, The S2 includes: S21. Obtain visual perception information through the visual sensors mounted on the mobile robot itself; S22. Use an open vocabulary target detection model to detect semantic entities from visual perception information, and obtain the candidate regions, category semantic information and confidence scores corresponding to each semantic entity. S23. The candidate region is finely segmented using an image segmentation model to obtain pixel-level masks of semantic entities; S24. The pixel-level mask is used to encode features using a pre-trained visual language model to obtain a high-dimensional semantic feature vector of the semantic entity as a semantic feature representation. S25. Obtain the spatial location information of semantic entities in the environment, and construct the semantic entities as nodes in the multimodal environment knowledge graph based on the semantic feature representation and spatial location information; S26. Based on the spatial proximity or semantic similarity between semantic entities, construct association edges between corresponding nodes; wherein, the spatial proximity is constructed based on the spatial location information of the semantic entities, and the semantic similarity is constructed based on the category semantic information and semantic feature representation of the semantic entities; S27. Construct a multimodal environmental knowledge graph to describe the semantic state of the environment based on nodes and associated edges.

5. The robot visual language navigation method based on multimodal knowledge graph according to claim 1, characterized in that, The S3 includes: S31. Receive the natural language navigation instruction input by the user, and determine whether the natural language navigation instruction is a word, a short phrase containing a clear target entity, or an instruction with a clear navigation constraint relationship. If it is a word or a short phrase containing a clear target entity, then execute step S32. If it is an instruction with a clear navigation constraint relationship, then execute step S33. S32. Based on the natural language navigation instructions, candidate related entities are given through reasoning using a large language model; the natural language navigation instructions are input into the text encoder of a pre-trained visual language model to generate target entity nodes; the candidate related entities are input into the text encoder of a pre-trained visual language model to generate related nodes; and a target semantic knowledge subgraph is constructed based on the target entity nodes and related nodes. S33. Perform target entity parsing on the target entities and their positional relationships in the natural language navigation instructions. Based on the target entity parsing results, infer candidate related entities through a large language model. Input the target entity parsing results into the text encoder of a pre-trained visual language model to generate target entity nodes. Input the candidate related entities into the text encoder of the pre-trained visual language model to generate related nodes. Construct a target semantic knowledge subgraph based on the target entity nodes and related nodes.

6. The robot visual language navigation method based on multimodal knowledge graph according to claim 1, characterized in that, The S4 includes: S41. Calculate the semantic similarity between nodes based on the high-dimensional semantic feature vectors of semantic entities in the multimodal environment knowledge graph and the high-dimensional vectors of nodes obtained by the text encoder in the target semantic knowledge subgraph. S42. Define the neighborhood semantic entity set of the front point based on the candidate front point set and semantic entities, and define the semantic value of the front point based on the neighborhood semantic entity set and semantic similarity. S43. Define the distance weight of the leading edge point based on the geometric distance between the robot and the leading edge point; S44. Define the explorability of a frontier point based on its potential to bring new environmental information to the robot after it has been explored. S45. Construct a comprehensive evaluation function based on the semantic value, distance weight, and explorability of the frontier points, and select sub-targets for navigation from the candidate frontier point set based on the comprehensive evaluation function.

7. A robot visual language navigation device based on a multimodal knowledge graph, wherein the robot visual language navigation device based on a multimodal knowledge graph is used to implement the robot visual language navigation method based on a multimodal knowledge graph as described in any one of claims 1-6, characterized in that, The device includes: The candidate front point set construction module is used to obtain the pose of the mobile robot and the depth observation information collected by the depth camera, construct a front map based on the pose and depth observation information, extract candidate front points based on the front map, and obtain a candidate front point set. The multimodal environment knowledge graph construction module is used to acquire visual perception information observed by the mobile robot in the environment, identify semantic entities based on visual perception information, construct nodes according to semantic entities, semantic feature representations of semantic entities and spatial location information, construct the association edges between nodes, and construct a multimodal environment knowledge graph to describe the semantic state of the environment based on nodes and association edges. The target semantic knowledge subgraph construction module is used to receive natural language navigation instructions input by the user, generate target entity nodes and associated nodes through the text encoder of the pre-trained visual language model, and construct the target semantic knowledge subgraph based on the target entity nodes and associated nodes. The sub-target selection module is used to obtain the semantic similarity between nodes based on the multimodal environment knowledge graph and the target semantic knowledge subgraph, define the semantic value of the frontier point based on the semantic similarity, define the distance weight and explorability of the frontier point, construct a comprehensive evaluation function based on the semantic value, distance weight and explorability, and select the sub-target for navigation from the candidate frontier point set based on the comprehensive evaluation function. The output module is used to introduce a proximal policy optimization algorithm to construct a path planning policy based on reinforcement learning, and to realize the path planning of the mobile robot according to the path planning policy and sub-objectives.

8. The robot visual language navigation device based on multimodal knowledge graph according to claim 7, characterized in that, The process involves acquiring visual perception information observed by the mobile robot in the environment, identifying semantic entities based on this information, constructing nodes according to the semantic entities, their semantic feature representations, and spatial location information, building connections between nodes, and constructing a multimodal environmental knowledge graph to describe the semantic state of the environment based on the nodes and connections. S21. Obtain visual perception information through the visual sensors mounted on the mobile robot itself; S22. Use an open vocabulary target detection model to detect semantic entities from visual perception information, and obtain the candidate regions, category semantic information and confidence scores corresponding to each semantic entity. S23. The candidate region is finely segmented using an image segmentation model to obtain pixel-level masks of semantic entities; S24. The pixel-level mask is used to encode features using a pre-trained visual language model to obtain a high-dimensional semantic feature vector of the semantic entity as a semantic feature representation. S25. Obtain the spatial location information of semantic entities in the environment, and construct the semantic entities as nodes in the multimodal environment knowledge graph based on the semantic feature representation and spatial location information; S26. Based on the spatial proximity or semantic similarity between semantic entities, construct association edges between corresponding nodes; wherein, the spatial proximity is constructed based on the spatial location information of the semantic entities, and the semantic similarity is constructed based on the category semantic information and semantic feature representation of the semantic entities; S27. Construct a multimodal environmental knowledge graph to describe the semantic state of the environment based on nodes and associated edges.

9. A mobile robot visual language navigation device, characterized in that, The mobile robot visual language navigation device includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 6.

Images