A robotic natural language navigation method, system, and device

Abstract

The application discloses a robot natural language navigation method, system and device, mainly relates to the language navigation technical field, in order to solve the existing scheme only contains the geometry and obstacle information of environment, lack of understanding of the semantics, function, attribute of object in environment, cannot handle complex, combined, or involve abstract concept instruction Problem. Including: the comprehensive similarity score is arranged in descending order, and the first K nodes constitute a candidate set;Take the text description of the first K nodes, construct a prompt word;Using prompt word, natural language instruction, LLM infers the candidate nodes in candidate set, outputs the inference probability of each candidate node satisfying natural language instruction;Obtain the retrieval probability of each candidate node;Using retrieval probability and inference probability, calculate the final confidence of each candidate node;Determine the candidate node with the highest final confidence as the target node, output the pose point of the target node;Control robot operation to target node.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of language navigation, and in particular to a robot natural language navigation method, system and device. BACKGROUND

[0002] Traditional navigation relies on geometric maps: such as point cloud maps or grid maps generated by SLAM (simultaneous localization and mapping) technology, which only contain geometric and obstacle information of the environment, lacking understanding of the semantics, functions and attributes of objects in the environment. Robots cannot understand instructions such as "get a coke on the living room coffee table". Deficiencies of early semantic navigation: some existing semantic navigation technologies usually label discrete object labels (such as "chair" and "table") in the map. But this way has limited information and cannot handle complex, combined or abstract concept-related instructions (such as "find a quiet place to work" or "go to the place where the children played just now"). Gap in instruction understanding:

[0003] However, traditional methods are difficult to establish effective association between rich semantics of natural language instructions and internal environment representation of robots. Simple keyword matching methods have poor fault tolerance and low precision, and cannot understand synonyms, context and intent. Therefore, there is an urgent need for a robot natural language navigation method, system and device to solve the above problems. SUMMARY

[0004] The present application provides a robot natural language navigation method, system and device to solve the problem that existing solutions only contain geometric and obstacle information of the environment, lack understanding of the semantics, functions and attributes of objects in the environment, and cannot handle complex, combined or abstract concept-related instructions.

[0005] In a first aspect, the present application provides a robot natural language navigation method, the method comprising:

[0006] controlling the robot to move in a working environment, and collecting scene images at a plurality of discrete pose points generating a natural language text description corresponding to the scene image mapping the scene image and the text description to the same high-dimensional semantic space respectively, and extracting a deep image feature vector , a deep text feature vector ;

[0007] defining each tuple , , , , as a node , and calculating a comprehensive association weight between the nodes ​semantic embedding saliency weight of the node ;

[0008] a weighted undirected graph is constructed by using the node and the weight greater than the preset threshold ;

[0009] receiving a natural language instruction input by a user , encoding the natural language instruction into an instruction query vector ; calculating the initial semantic similarity between the instruction query vector and all nodes in the weighted undirected graph ;

[0010] modulating the initial semantic similarity , obtaining an initial score ;

[0011] performing iterative propagation of the initial score on the weighted undirected graph, smoothing the similarity score by using the graph structure relationship, and obtaining a comprehensive similarity score of each node

[0012] ranking the comprehensive similarity scores in descending order, and taking the top K nodes to form a candidate set

[0013] taking the text description of the top K nodes to construct a prompt word ; using the prompt word and the natural language instruction, the LLM performs reasoning on the candidate nodes in the candidate set, and outputs a reasoning probability of each candidate node satisfying the natural language instruction

[0014] obtaining a retrieval probability of each candidate node; using the retrieval probability and the reasoning probability, calculating a final confidence of each candidate node

[0015] determining the candidate node with the highest final confidence as a target node, and outputting a pose point of the target node; controlling the robot to operate to the target node

[0016] In an implementation manner of the present application, a natural language text description corresponding to a scene image is generated , and specifically includes:

[0017] inputting the scene image into an image description generation model to generate a corresponding natural language text description .

[0018] In an implementation manner of the present application, the comprehensive association weight between nodes , the semantic embedding saliency weight of the node , specifically includes:

[0019] by the formula:

[0020] Calculate the overall correlation weight ;

[0021] in, , Indicates hyperparameters, and + =1, Indicates the spatial scale factor. , Indicates the first i The and the first j 1 node express , The overall correlation weight between them; , They represent the first i The and the first j The depth image feature vector of each node, and They represent the first i The and the first j The deep text feature vector of each node;

[0022] Through the formula:

[0023] Calculate the semantic embedding saliency weights of nodes. ;

[0024] Where C represents the total number of preset categories, Indicates will The probability of belonging to category c is obtained after inputting into a pre-trained semantic classifier.

[0025] In one implementation of this application, calculation With all nodes in the weighted undirected graph middle Initial semantic similarity Specifically, it includes:

[0026] Through the formula:

[0027] Calculate the initial semantic similarity .

[0028] In one implementation of this application, utilizing ,right Modulation is performed to obtain an initial score, specifically including:

[0029] Through the formula:

[0030] , calculate the initial score .

[0031] In an implementation of the present application, the initial score is iteratively propagated on a weighted undirected graph, and the similarity scores are smoothed using the graph structure relationship to obtain a comprehensive similarity score of each node, specifically including:

[0032] Through the iteration formula:

[0033] , calculate the comprehensive similarity iterative score of the i th node ;

[0034] Where D is the degree matrix, W is the weight matrix, is the propagation coefficient, represents the initial score, is the initial semantic similarity;

[0035] Iterate until convergence or reach the preset stopping condition to obtain the comprehensive similarity score.

[0036] In an implementation of the present application, the prompt, the natural language instruction, and the LLM are used to infer the candidate nodes in the candidate set, and output the inference probability of each candidate node satisfying the natural language instruction, specifically including:

[0037] Construct a prompt ;

[0038] Where U represents the natural language instruction, ,..., represent the text description of the candidate nodes in the candidate set;

[0039] The prompt, the natural language instruction, and the LLM are used to infer the candidate nodes in the candidate set, and output the inference probability of each candidate node satisfying the natural language instruction ;

[0040] Where, represents the k th candidate node in the candidate set, k The value range of is [1, K].

[0041] In an implementation of the present application, the retrieval probability and the inference probability are used to calculate the final confidence of each candidate node, specifically including:

[0042] Through the formula:

[0043] , calculate the final confidence of each candidate node;

[0044] Where, represents the inference probability of the i-th candidate node in the candidate set, k represents the i-th candidate node in the candidate set, represents the fusion weight hyperparameter, represents the retrieval probability, represents the inference probability of the i-th candidate node in the candidate set, k represents the inference probability of the i-th candidate node in the candidate set.

[0045] In a second aspect, the application provides a robot natural language navigation system, the system comprising:

[0046] a directed graph generation module for controlling the robot to move in a working environment and collecting a plurality of discrete pose points scene images ; generating natural language text descriptions corresponding to the scene images ; mapping the scene images and the text descriptions to the same high-dimensional semantic space respectively, extracting deep image feature vectors , deep text feature vectors ; defining each tuple , , , , as a node , calculating the comprehensive correlation weight between the nodes , calculating the semantic embedding saliency weight of the nodes ; using the nodes and the weight greater than the preset threshold to form a weighted directed graph;

[0047] a target node determination module for receiving a natural language instruction input by a user , encoding the natural language instruction into an instruction query vector ; calculating the initial semantic similarity between all nodes in the weighted directed graph; using to modulate to obtain the initial score; performing iterative propagation of the initial score on the weighted directed graph, smoothing the similarity score using the graph structure relationship, and obtaining the comprehensive similarity score of each node; ranking the comprehensive similarity score in descending order, and taking the top K nodes to form a candidate set; taking the text descriptions of the top K nodes to construct a prompt word ; the LLM uses the prompt word and the natural language instruction to infer the candidate nodes in the candidate set, and outputs an inference probability of each candidate node satisfying the natural language instruction; a retrieval probability of each candidate node is obtained; the retrieval probability and the inference probability are used to calculate a final confidence of each candidate node; a candidate node with the highest final confidence is determined as a target node, and a pose point of the target node is output; and the robot is controlled to operate to the target node.

[0048] In a third aspect, the present application provides a robot natural language navigation device, the device comprising:

[0049] a processor;

[0050] and a memory having executable code stored thereon, when the executable code is executed, causing the processor to perform a robot natural language navigation method according to any one of the above.

[0051] From the above technical solutions, the present application has the following advantages:

[0052] The present application solves the pain point of semantic loss of traditional navigation systems by constructing a multi-modal mapping mechanism of high-dimensional semantic space. Specifically, the operation of mapping the scene image and the natural language description to the same semantic space (co-extraction of the depth image feature vector and the text feature vector) enables the robot to have a deep understanding of the functional attributes and spatial relationships of environmental objects for the first time. This fusion of visual and language representation not only supports the positioning of concrete objects such as "a coke on the coffee table", but also can analyze abstract concepts such as "a quiet work area" through the calculation of semantic embedding saliency weight of nodes. This is a breakthrough that traditional discrete label annotation technology cannot achieve. When processing combined instructions, the weighted undirected graph structure dynamically establishes cross-node semantic connections by integrating correlation weights, for example, automatically associating "a place where children play" with scattered nodes such as toys and children's furniture to form a complete semantic network. Compared with the static label system of early semantic navigation, it realizes the dynamic reconstruction and context reasoning of environmental semantics.

[0053] The iterative similarity propagation algorithm based on the graph structure improves the fault tolerance and analysis accuracy of natural language instructions. The double matching mechanism of the instruction query vector and the node (initial semantic similarity calculation + graph structure relationship smoothing) effectively overcomes the synonym ambiguity problem. A hybrid decision framework of LLM is introduced: the environment semantics (candidate node text description) and user instructions are jointly input into the large language model through prompt word construction, and the common sense reasoning ability of the large language model is used to supplement the limitations of the pure retrieval system. The weighted fusion strategy (final confidence calculation) of retrieval probability and reasoning probability enables the system to balance the objective characteristics of the environment and subjective intention understanding, which can not only filter nodes based on distance weight, but also exclude actual unreachable points blocked by furniture through LLM. The final output of the target node position point is essentially the optimal solution of geometric accuracy and semantic rationality. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the present application, the drawings required to be used in the description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0055] Figure 1 It is a robot natural language navigation method flowchart provided by an embodiment of the present application.

[0056] Figure 2 It is a robot natural language navigation system internal structure schematic diagram provided by an embodiment of the present application.

[0057] Figure 3 It is a robot natural language navigation device internal structure schematic diagram provided by an embodiment of the present application. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0059] Those skilled in the art should understand that the embodiments described below are only preferred embodiments of the present disclosure, and do not mean that the present disclosure can only be implemented by this preferred embodiment. The preferred embodiment is only used to explain the technical principles of the present disclosure, and is not used to limit the protection scope of the present disclosure. Based on the preferred embodiments provided by the present disclosure, all other embodiments obtained by those skilled in the art without creative labor are still within the scope of protection of the present disclosure.

[0060] It should also be noted that the terms "comprising", "containing" or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not include only those elements recited, but can also include other elements not expressly listed or inherent to such process, method, article or apparatus. Without more limitations, an element defined by the phrase "comprising a" does not exclude the existence of additional identical elements in the process, method, article or apparatus that includes the element.

[0061] The technical solutions of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0062] The embodiments provide a robot natural language navigation method, as shown in Figure 1 The method provided by the embodiments of the present application mainly includes the following steps:

[0063] Step 110, control the robot to move in the working environment and collect scene images at a plurality of discrete pose points; generate natural language text descriptions corresponding to the scene images; map the scene images and the text descriptions to the same high-dimensional semantic space respectively, and extract a deep image feature vector and a deep text feature vector.

[0064] It should be noted that the pose points , wherein, x, y, z, θ respectively represent the horizontal high value and the angle value.

[0065] In some embodiments, the natural language text description corresponding to the scene image is generated , specifically including:

[0066] The scene image is input to an image description generation model to generate a corresponding natural language text description .

[0067] It should be noted that the image description generation model can be a model of Transformer.

[0068] Understandably, by acquiring scene images of discrete pose points through robot movement and generating corresponding text descriptions, a dual-modal acquisition of environmental information is achieved. After mapping images and text to the same high-dimensional semantic space, the system gains two key capabilities: first, preserving visual details (such as the spatial position x / y / z coordinates of objects and their orientation angle θ) through depth image feature vectors; and second, establishing interpretable semantic expressions through depth text feature vectors. This dual-channel processing enables the environmental representation to contain both precise geometric information (guaranteed by pose point data) and a human-understandable semantic dimension (e.g., descriptions generated by the Transformer model).

[0069] Step 120: Define each tuple pose point, scene image, text description, depth image feature vector, and depth text feature vector as a node; calculate the comprehensive association weight between nodes and the semantic embedding saliency weight of nodes; and construct a weighted undirected graph using nodes and weights greater than a preset threshold.

[0070] It should be noted that the pose point Scene Images Text description The depth image feature vector and the depth text feature vector are as follows: , , , , .

[0071] Among them, the comprehensive correlation weight between computing nodes Calculate the semantic embedding saliency weights of nodes. Specifically, it includes:

[0072] Through the formula:

[0073] Calculate the overall correlation weight ;

[0074] in, , Indicates hyperparameters, and + =1, Indicates the spatial scale factor. , Indicates the first i The and the first j 1 node express , The overall correlation weight between them; , They represent the first i The and the firstj a depth image feature vector of the i-th node, and a depth text feature vector of the i-th node and the j-th node, respectively; i j denotes a spatial weight, denotes a semantic weight.

[0075] The semantic embedding saliency weight of the node is calculated by the formula:

[0076] to measure the explicitness of the semantics of the node.

[0077] where C denotes the total number of preset categories, denotes the probability of belonging to the category c after inputting the i-th node into the pre-trained semantic classifier.

[0078] It should be noted that the semantic classifier is a multi-layer perceptron (MLP) classifier pre-trained on a general image-text dataset (such as ImageNet or COCO). The value of ranges from 0 to 1, and the larger the value, the more significant the semantics. The weighted undirected graph is a semantic forest map, which is a dynamic knowledge base that integrates environmental geometry, multi-modal semantics, and their internal correlations.

[0079] Based on the above description, first, by defining the tuple as a node and calculating the comprehensive correlation weight, the quantitative fusion of multi-modal features (depth image and text features) is realized, which establishes a calculable correlation between environmental geometric information and semantic information in a unified framework, avoiding the limitations of manually designed correlation rules. Second, the introduction of semantic embedding saliency weight objectively quantifies the explicitness of node semantics using the probability output of the pre-trained classifier, providing an interpretable confidence index for subsequent semantic reasoning. This statistical probability-based measurement method is more reliable than subjective semantic labeling. Finally, the generated weighted undirected graph (semantic forest map) retains the spatial distribution relationship of the original data (adjusted by the spatial scale factor) and ensures the effectiveness of the correlation through threshold filtering.

[0080] Step 130, receiving a natural language instruction input by a user, encoding the natural language instruction into an instruction query vector; calculating the initial semantic similarity between the instruction query vector and the depth text feature vector of all nodes in the weighted undirected graph; using the semantic embedding saliency weight to modulate the initial semantic similarity to obtain an initial score; iteratively propagating the initial score on the weighted undirected graph to smooth the similarity score using the graph structure relationship, and obtaining the comprehensive similarity score of each node. ​​​​

[0081] Among them, calculation With all nodes in the weighted undirected graph middle Initial semantic similarity Specifically, it includes:

[0082] Through the formula:

[0083] Calculate the initial semantic similarity .

[0084] use ,right Modulation is performed to obtain an initial score, specifically including:

[0085] Through the formula:

[0086] Calculate the initial score .

[0087] The initial scores are iteratively propagated across a weighted undirected graph, and the similarity scores are smoothed using graph structure relationships to obtain a comprehensive similarity score for each node. Specifically, this includes:

[0088] Through iterative formula:

[0089] Calculate the first i The comprehensive similarity iterative score of each node ;

[0090] Where D is the degree matrix and W is the weight matrix. For the propagation coefficient, Indicates the initial score. Initial semantic similarity;

[0091] The comprehensive similarity score is obtained after iteration until convergence or when the preset stopping condition is reached.

[0092] It should be noted that the preset stopping condition can be that the L2 norm of the score change between two iterations is less than a threshold. : .

[0093] Based on the above description, the present step realizes the systematic calculation of semantic matching through the vectorization processing of natural language instructions and the topological propagation mechanism of weighted undirected graph. The specific effects are as follows: the encoding conversion of instruction query vectors enables the direct comparison of unstructured natural language input with the multi-modal features of nodes; the initial semantic similarity calculation establishes a basic association relationship by quantifying the matching degree of instruction vectors and node features; the modulation function highlights the matching priority of key nodes by introducing semantic saliency weights and making targeted adjustments to the initial similarity; the iterative propagation based on the graph structure utilizes the topological relationship defined by the degree matrix and the weight matrix to smoothly transfer the initial scores between associated nodes through mathematical iteration, and the final comprehensive similarity score not only retains the original matching information but also incorporates the contextual relevance implied by the graph structure, providing a matching basis with geometric consistency and semantic coherence for subsequent instruction analysis and task execution. The entire process realizes automatic calculation through preset mathematical rules and convergence conditions (L2 norm threshold), and does not rely on subjective parameter settings.

[0094] Step 140, rank the comprehensive similarity scores in descending order, take the top K nodes to form a candidate set; take the text descriptions of the top K nodes to construct prompt words; use the prompt words, natural language instructions, and LLM to infer the candidate nodes in the candidate set, and output the inference probability of each candidate node satisfying the natural language instruction.

[0095] In some embodiments, using the prompt words, natural language instructions, and LLM to infer the candidate nodes in the candidate set outputs the inference probability of each candidate node satisfying the natural language instruction, specifically including:

[0096] Constructing prompt words ;

[0097] Wherein, U represents the natural language instruction, ,..., represents the text description of the candidate node in the candidate set;

[0098] Using the prompt words, natural language instructions, and LLM to infer the candidate nodes in the candidate set outputs the inference probability of each candidate node satisfying the natural language instruction ;

[0099] Wherein, represents the k th candidate node in the candidate set, k the value range of i is [1, K].

[0100] Based on the above description, this step achieves refined response to natural language instructions by sorting the comprehensive similarity scores in descending order and selecting the top K nodes to form a candidate set. Its direct effects are reflected in the following: the generation of the candidate set is based on the ranking of the comprehensive similarity scores, ensuring a high degree of semantic relevance between the candidate nodes and the instructions; by constructing cue words containing both the natural language instructions and the textual descriptions of the candidate nodes, clear contextual information is provided to the LLM; the LLM uses these cue words to perform inference, outputting the probability that each candidate node satisfies the instruction. This process combines the structured features of the nodes with the semantic understanding capabilities of the LLM, ensuring that the final result retains the accuracy of graph structure analysis while incorporating the LLM's deep semantic analysis of natural language instructions.

[0101] Step 150: Obtain the retrieval probability of each candidate node; use the retrieval probability and inference probability to calculate the final confidence of each candidate node; determine the candidate node with the highest final confidence as the target node, and output the pose of the target node; control the robot to move to the target node.

[0102] Specifically, the final confidence score of each candidate node is calculated using retrieval probability and inference probability, including:

[0103] Through the formula:

[0104] Calculate the final confidence score for each candidate node;

[0105] in, Indicates the first in the candidate set k Candidate nodes, To integrate weight hyperparameters, Indicates the retrieval probability. Indicates the first in the candidate set k The inference probability of each candidate node.

[0106] The candidate node with the highest final confidence level is selected as the target node: .

[0107] It should be noted that when the maximum confidence level is... When the value falls below a preset threshold, it has the ability to reflect and replan, and can generate clarifying questions for user interaction or adjust parameters. and Re-execute the decision-making process.

[0108] Based on the above description, this step calculates the final confidence of the candidate node by fusing the retrieval probability and the reasoning probability, and determines the target node based on the confidence, realizing the reliability and adaptability of robot decision-making. The direct effect is reflected in: the weighted fusion of retrieval probability and reasoning probability (the weight is adjusted by the hyperparameter in the formula) integrates the accuracy of graph structure matching (retrieval probability) and the rationality of semantic reasoning (reasoning probability), so that the final confidence can quantitatively evaluate the comprehensive credibility of the candidate node to meet the instruction; the selection of the highest confidence node ensures the double advantages of the target node in matching accuracy and semantic rationality, and the output pose point provides accurate spatial coordinates for robot motion; when the maximum confidence is lower than the threshold, the system realizes reflection and re-planning by generating clarifying questions or adjusting parameters, avoiding incorrect decisions due to insufficient information and improving the fault tolerance of task execution.

[0109] Based on the foregoing description, the embodiment of the present application solves the pain point of semantic missing of traditional navigation systems by constructing a multi-modal mapping mechanism of high-dimensional semantic space. Specifically, the operation of mapping the scene image and the natural language description to the same semantic space (co-extraction of deep image feature vector and text feature vector) enables the robot to have a deep understanding of the functional attributes and spatial relationships of environmental objects for the first time. This fusion of visual and linguistic representation not only supports the positioning of concrete objects such as "a coke on the coffee table", but also can analyze abstract concepts such as "a quiet work area" through the calculation of semantic embedding saliency weight of nodes - a breakthrough that cannot be achieved by traditional discrete label annotation technology. When processing combined instructions, the weighted undirected graph structure dynamically establishes semantic connections across nodes by integrating correlation weights, for example, automatically associating "a place where children play" with scattered nodes such as toys and children's furniture, forming a complete semantic network. Compared with the static label system of early semantic navigation, it realizes the dynamic reconstruction and context reasoning of environmental semantics.

[0110] The iterative similarity propagation algorithm based on graph structure improves the fault tolerance and analysis accuracy of natural language instructions. The dual matching mechanism of instruction query vector and node in the technical scheme (initial semantic similarity calculation + graph structure relationship smoothing) effectively overcomes the synonym ambiguity problem. The hybrid decision framework of LLM is introduced: the environment semantics (text description of candidate nodes) and user instructions are jointly input into the large language model through prompt word construction, and the common sense reasoning ability of the large language model is used to supplement the limitations of the pure retrieval system. This weighted fusion strategy (final confidence calculation) of retrieval probability and reasoning probability enables the system to balance the understanding of environmental objective features and subjective intentions, which can not only filter nodes based on distance weight, but also exclude actual unreachable points hidden by furniture through LLM. The final output of the target node pose point is essentially the optimal solution of geometric accuracy and semantic rationality.

[0111] In addition, the present application Figure 2This application provides a robot natural language navigation system as an embodiment. Figure 2 As shown in the embodiments of this application, the system mainly includes:

[0112] Undirected graph generation module 210 is used to control the robot's movement in the working environment and to generate graphs at several discrete pose points. Acquire scene images Generate natural language text descriptions corresponding to the scene images. The scene image and text description are mapped to the same high-dimensional semantic space, and the depth image feature vector is extracted. Deep text feature vectors ; each tuple , , , , Define as a node Calculate the comprehensive correlation weight between nodes Calculate the semantic embedding saliency weights of nodes. ; Utilizing nodes Weights greater than a preset threshold This forms a weighted undirected graph;

[0113] Target node determination module 220 is used to receive natural language instructions input by the user. Encode natural language instructions into instruction query vectors ;calculate With all nodes in the weighted undirected graph middle Initial semantic similarity ;use ,right Modulation is performed to obtain an initial score; the initial score is iteratively propagated on a weighted undirected graph, and the similarity score is smoothed using the graph structure relationships to obtain the comprehensive similarity score of each node; the comprehensive similarity scores are sorted in descending order, and the top K nodes are selected to form a candidate set; the text descriptions of the top K nodes are used to construct prompt words. Using prompts and natural language instructions, LLM infers the candidate nodes in the candidate set and outputs the inference probability of each candidate node satisfying the natural language instructions; obtains the retrieval probability of each candidate node; calculates the final confidence of each candidate node using the retrieval probability and inference probability; determines the candidate node with the highest final confidence as the target node and outputs the pose of the target node; and controls the robot to move to the target node.

[0114] Based on the foregoing description, it can be known that the embodiment solves the pain point of semantic loss of the traditional navigation system by constructing a multi-modal mapping mechanism of a high-dimensional semantic space. Specifically, the operation of mapping the scene image and the natural language description to the same semantic space (co-extraction of the depth image feature vector and the text feature vector) enables the robot to have a deep understanding of the functional attributes and spatial relationships of the environmental objects for the first time. This fusion of visual and language representation not only supports the positioning of concrete objects such as "a coke on the coffee table", but also can analyze abstract concepts such as "a quiet work area" through the semantic embedding significance weight calculation of the nodes. This is a breakthrough that cannot be achieved by the traditional discrete label annotation technology. When processing combined instructions, the weighted undirected graph structure dynamically establishes cross-node semantic connections by comprehensively correlating the weights, for example, automatically associating "a place where children play" with scattered nodes such as toys, children's furniture, etc., to form a complete semantic network. Compared with the static label system of the early semantic navigation, the dynamic reconstruction and context reasoning of the environmental semantics are realized.

[0115] The iterative similarity propagation algorithm based on the graph structure improves the fault tolerance and analysis accuracy of the natural language instruction. The double matching mechanism of the instruction query vector and the node (initial semantic similarity calculation + graph structure relationship smoothing) in the technical solution effectively overcomes the synonym ambiguity problem. The hybrid decision framework of the LLM is introduced: the environment semantics (candidate node text description) and the user instruction are jointly input into the large language model through the prompt word construction, and the common sense reasoning ability of the large language model is used to supplement the limitations of the pure retrieval system. The weighted fusion strategy (final confidence calculation) of the retrieval probability and the reasoning probability enables the system to balance the understanding of the objective features of the environment and the subjective intention, and can not only filter the nodes based on the distance weight, but also exclude the actual unreachable points blocked by furniture through the LLM. The final output of the target node position point is essentially the optimal solution of geometric accuracy and semantic rationality.

[0116] The above is a method embodiment in the present application, based on the same inventive concept, the present application also provides a robot natural language navigation device. As shown in Figure 3 , the device comprises a processor and a memory having executable code stored thereon, when the executable code is executed, the processor executes a robot natural language navigation method as described in one of the above embodiments.

[0117] Specifically, the server controls the robot to move in the working environment, and collects scene images at a plurality of discrete position points in the working environment. The server generates natural language text descriptions corresponding to the scene images. The server maps the scene images and the text descriptions to the same high-dimensional semantic space respectively, extracts depth image feature vectors and depth text feature vectors . The server extracts a feature vector of each tuple , , , , Define as a node Calculate the comprehensive correlation weight between nodes Calculate the semantic embedding saliency weights of nodes. ; Utilizing nodes Weights greater than a preset threshold This forms a weighted undirected graph; it receives natural language commands input by the user. Encode natural language instructions into instruction query vectors ;calculate With all nodes in the weighted undirected graph middle Initial semantic similarity ;use ,right Modulation is performed to obtain an initial score; the initial score is iteratively propagated on a weighted undirected graph, and the similarity score is smoothed using the graph structure relationships to obtain the comprehensive similarity score of each node; the comprehensive similarity scores are sorted in descending order, and the top K nodes are selected to form a candidate set; the text descriptions of the top K nodes are used to construct prompt words. Using prompts and natural language instructions, LLM infers the candidate nodes in the candidate set and outputs the inference probability of each candidate node satisfying the natural language instructions; obtains the retrieval probability of each candidate node; calculates the final confidence of each candidate node using the retrieval probability and inference probability; determines the candidate node with the highest final confidence as the target node and outputs the pose of the target node; and controls the robot to move to the target node.

[0118] As described above, this embodiment addresses the semantic deficiencies of traditional navigation systems by constructing a multimodal mapping mechanism in a high-dimensional semantic space. Specifically, the operation of mapping scene images and natural language descriptions to the same semantic space (cooperative extraction of depth image feature vectors and text feature vectors) enables the robot to, for the first time, possess a deep understanding of the functional attributes and spatial relationships of objects in the environment. This representation method, which integrates vision and language, not only supports the localization of concrete objects such as "the cola on the coffee table," but also, through the semantic embedding saliency weight calculation of nodes, can resolve abstract concepts such as "a quiet work area"—a breakthrough that traditional discrete labeling techniques cannot achieve. When processing combinatorial instructions, the weighted undirected graph structure dynamically establishes cross-node semantic connections through comprehensive association weights. For example, it automatically associates "the place where children play" with scattered nodes such as toys and children's furniture, forming a complete semantic network. Compared to the static labeling system of early semantic navigation, this achieves dynamic reconstruction of environmental semantics and contextual reasoning.

[0119] The iterative similarity propagation algorithm based on graph structure improves the fault tolerance and parsing accuracy of natural language instructions. The dual matching mechanism of instruction query vector and node (initial semantic similarity calculation + graph structure relationship smoothing) effectively overcomes the synonym ambiguity problem. A hybrid decision framework is introduced: the environment semantics (candidate node text description) and user instructions are jointly input into the large language model through prompt word construction, and the common sense reasoning ability of the large language model is used to make up for the limitations of the pure retrieval system. The weighted fusion strategy of retrieval probability and reasoning probability (final confidence calculation) makes the system balance the objective characteristics of the environment and the understanding of subjective intentions, which can not only filter nodes based on distance weight, but also exclude actual unreachable points blocked by furniture through LLM. The final output of the target node pose point is essentially the optimal solution of geometric accuracy and semantic rationality.

[0120] In addition, the embodiment of the present application further provides a non-volatile computer storage medium, which has executable instructions stored thereon, and the executable instructions, when executed, realize the robot natural language navigation method as described above.

[0121] The above description of disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of natural language navigation for a robot, the method comprising: The method comprises: Controlling a robot to move in a work environment and to perform a task at a number of discrete pose points Capturing scene images Generating natural language text descriptions corresponding to the scene images Mapping the scene images and the text descriptions to the same high-dimensional semantic space respectively, and extracting deep image feature vectors , deep text feature vectors ; Each tuple is defined as a node , , , , The semantic embedding significance weight of the node is calculated , and the comprehensive association weight between nodes is calculated , The semantic embedding significance weight of the node is calculated ; using the node and the weight greater than the preset threshold, a weighted undirected graph is formed; Receive natural language commands from user input. Encode natural language instructions into instruction query vectors ;calculate With all nodes in the weighted undirected graph middle Initial semantic similarity ;use ,right Modulation is performed to obtain an initial score; the initial score is iteratively propagated on a weighted undirected graph, and the similarity score is smoothed using the graph structure relationship to obtain the comprehensive similarity score of each node; The comprehensive similarity scores are arranged in descending order, and the first K nodes constitute a candidate set; the text descriptions of the first K nodes are taken to construct prompt words The LLM uses the prompt words and the natural language instruction to infer the candidate nodes in the candidate set, and outputs an inference probability of each candidate node satisfying the natural language instruction; Obtaining the retrieval probability of each candidate node; calculating the final confidence of each candidate node by using the retrieval probability and the reasoning probability; determining the candidate node with the highest final confidence as the target node, and outputting the pose point of the target node; and controlling the robot to operate to the target node.

2. The robotic natural language navigation method of claim 1, wherein, Generating natural language text descriptions corresponding to scene images , and specifically comprises: inputting the scene image to an image description generation model to generate a corresponding natural language textual description .

3. The robotic natural language navigation method of claim 1, wherein, comprehensive association weights between computing nodes semantic embedding saliency weights of the computing nodes , and specifically comprises: Through the formula: , computing the integrated correlation weight ; wherein, , denotes a hyper-parameter, and + = 1, denotes a spatial scale factor, , denotes a depth image feature vector of the i th and j th node, denotes a depth text feature vector of the , between the , denotes a depth image feature vector of the i th and j th node, and denotes a depth text feature vector of the i th and j th node; Through the formula: semantic embedding saliency weight of the computing node ; wherein C denotes a total number of preset categories, denotes a probability that input into the pre-trained semantic classifier.

4. The robotic natural language navigation method of claim 1, wherein, Computing the initial semantic similarity of all nodes in the weighted undirected graph , specifically comprising: the initial semantic similarity of all nodes in the weighted undirected graph​ Through the formula: , compute initial semantic similarity .

5. The robotic natural language navigation method of claim 1, wherein, use ,right Modulation is performed to obtain an initial score, specifically including: Through the formula: , compute initial score .

6. The robotic natural language navigation method of claim 1, wherein, The initial score is iteratively propagated on the weighted undirected graph, and the similarity score is smoothed by using the graph structure relationship to obtain the comprehensive similarity score of each node, specifically including: Through the iterative formula: , compute an overall similarity iterative score for the i ;​ wherein D is a degree matrix, W is a weight matrix, is a propagation coefficient, denotes an initial score, is an initial semantic similarity; Iterate to the comprehensive similarity score after convergence or reach the preset stopping condition.

7. The robotic natural language navigation method of claim 1, wherein, Using the prompt word and the natural language instruction, the LLM reasons the candidate nodes in the candidate set, and outputs the reasoning probability of each candidate node satisfying the natural language instruction, specifically including: Constructing a prompt word ; wherein U denotes a natural language instruction, ,..., denotes a textual description of a candidate node in the candidate set; The LLM reasons on the candidate nodes in the candidate set using the prompt words and the natural language instruction, and outputs a reasoning probability of each candidate node satisfying the natural language instruction ; wherein, represents the i-th candidate node in the candidate set, k k has a value range of [1, K].​ 8. The robotic natural language navigation method of claim 1, wherein, Calculating the final confidence of each candidate node by using the retrieval probability and the reasoning probability, specifically including: The system comprises: computing a final confidence for each candidate node; in, Indicates the first in the candidate set k Candidate nodes, To integrate weight hyperparameters, Indicates the retrieval probability. Indicates the first in the candidate set k The inference probability of each candidate node.

9. A robotic natural language navigation system, characterized in that, The device comprises: An undirected graph generation module is configured to control the robot to move in a working environment and to generate a plurality of discrete pose points Collecting a scene image Generating a natural language text description corresponding to the scene image Mapping the scene image and the text description to the same high-dimensional semantic space respectively, and extracting a deep image feature vector and a deep text feature vector Defining each tuple , , , , as a node , calculating a comprehensive correlation weight between the nodes , calculating a semantic embedding saliency weight of the node ; and using the node and the weight greater than a preset threshold to form a weighted undirected graph​ The target node determination module is used to receive natural language commands input by the user. Encode natural language instructions into instruction query vectors ;calculate With all nodes in the weighted undirected graph middle Initial semantic similarity ;use ,right Modulation is performed to obtain an initial score; the initial score is iteratively propagated on a weighted undirected graph, and the similarity score is smoothed using the graph structure relationships to obtain the comprehensive similarity score of each node; the comprehensive similarity scores are sorted in descending order, and the top K nodes are selected to form a candidate set; the text descriptions of the top K nodes are used to construct prompt words. Using prompts and natural language instructions, LLM infers the candidate nodes in the candidate set and outputs the inference probability of each candidate node satisfying the natural language instructions; obtains the retrieval probability of each candidate node; calculates the final confidence of each candidate node using the retrieval probability and inference probability; determines the candidate node with the highest final confidence as the target node and outputs the pose of the target node; and controls the robot to move to the target node.

10. A robotic natural language navigation device, characterized by, A processor; And a memory having executable code stored thereon, when the executable code is executed, causes the processor to perform a robot natural language navigation method according to any one of claims 1-8. ​

Images