Indoor navigation semantic orientation method based on large language model

Abstract

The invention discloses an indoor navigation semantic orientation method based on a large-scale language model, and relates to the technical field of indoor navigation, and the method comprises the steps: collecting time-synchronized continuous video streams and spatial position coordinates corresponding to each frame of image through a positioning sensor and an image sensor of a robot, so as to construct a complete data set containing a time alignment relationship; performing key frame extraction based on a semantic change threshold value between adjacent frames on the complete data set, and jointly inputting each key frame image, the spatial position coordinates strictly aligned with the key frame image in time and a first structured output control parameter into a multi-modal large language model, the first structured output control parameter is used for constraining model output to be a predefined JSON structure containing object names and attribute information, so that a structured JSON response containing preliminary description information of the target object is obtained. According to the invention, the verification result is generated and converted into voice broadcast, and a complete perception-planning-verification technology closed loop is formed.

Description

Technical Field

[0001] This invention relates to the field of indoor navigation technology, and in particular to an indoor navigation semantic orientation method based on a large language model. Background Technology

[0002] With the rapid development of artificial intelligence technology, indoor mobile robots have gradually become key carriers in fields such as intelligent services, warehousing and logistics, and smart healthcare. One of their core capabilities, autonomous navigation technology, has also evolved from path planning based on geometric maps to intelligent navigation incorporating semantic understanding. In recent years, breakthroughs in Large Language Models (LLMs) and Vision-Language Models (VLMs) have provided a new paradigm for robots to understand human natural language commands and interact semantically with their environment. Representative works in existing technologies, such as VLMaps or LM-Nav systems, explore how to use these pre-trained models to associate natural language queries with visual and spatial information of the environment. These methods typically rely on the model's ability to recognize open vocabulary and a certain degree of spatial reasoning ability, enabling them to handle relatively simple semantic navigation tasks such as asking users to go to the rest area with the coffee machine. The basic idea is to use the model to parse user commands into executable action sequences or descriptions of target locations, achieving a preliminary mapping from symbolic commands to physical space to a certain extent, marking an important step towards higher-level cognitive intelligence in indoor navigation technology.

[0003] However, despite the promising potential of the pre-trained model-based methods mentioned above, they still face significant challenges in handling complex and dynamic indoor scenes. A core limitation is that existing solutions mostly focus on task planning or instruction parsing, lacking a structured semantic knowledge base capable of persistently storing and efficiently retrieving fine-grained object attributes. Specifically, when user queries contain multiple complex attribute constraints, existing technologies typically rely on the model's immediate parsing of a single visual input, making it difficult to accurately distinguish and confirm multiple similar objects. This is because, while the model's immediate inference is flexible, it does not systematically extract, structure, store, and index the stable attributes of objects in the environment. This leads to deviations in the mapping relationship between semantic intent and physical objects when faced with dynamic changes or visually highly similar distractors, thus limiting the accuracy and reliability of navigation. Furthermore, existing methods typically lack a closed-loop verification mechanism after navigation actions are executed, failing to provide users with natural language feedback based on real-time visual confirmation upon reaching the target, affecting the integrity and credibility of the interaction. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides an indoor navigation semantic orientation method based on a large language model, which solves the problem in the prior art that the user's natural language intent and specific objects in the environment cannot be accurately aligned due to the lack of fine-grained semantic description and open command parsing capabilities.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides an indoor navigation semantic orientation method based on a large language model, which includes acquiring a time-synchronized continuous video stream and the spatial position coordinates corresponding to each frame of the image through the robot's positioning sensor and image sensor, so as to construct a complete dataset containing time alignment relationships. Keyframe extraction based on the semantic change threshold between adjacent frames is performed on the complete dataset. Each keyframe image, the spatial location coordinates that are strictly aligned with the keyframe image in time, and the first structured output control parameter are input into a multimodal large language model. The first structured output control parameter is used to constrain the model output to a predefined JSON structure containing object name and attribute information, thereby obtaining a structured JSON response containing preliminary description information of the target object. Based on the structured JSON response corresponding to all keyframes, the spatial location coordinates, object name and attribute information are parsed and extracted. A dedicated semantic database is constructed with the spatial location coordinates as the index key. The user's natural language navigation query is received. The natural language navigation query and the second query parsing control parameter are input into the large language model. The second query parsing control parameter is used to guide the model to parse the natural language query into target object keywords and attribute constraints and encapsulate them into a standardized JSON format to generate a standardized JSON description of the semantic constraints of the target object. Based on the standardized JSON description, cosine similarity retrieval of word embedding vectors and matching based on attribute key-value pairs are performed in the dedicated semantic database. Context filtering is completed by comparing the spatial coordinates of candidate entries with the robot's current coordinates in real time, thereby determining the precise spatial coordinates of the target object. The autonomous robot is controlled to navigate to the precise spatial coordinates. Upon arrival, the robot takes a snapshot of the current environment using its onboard image sensor. The snapshot, the standardized JSON description, and the third-vision verification control parameters are then input into the multimodal large-scale language model. The third-vision verification control parameters guide the model to compare the snapshot content with the JSON description and generate a verification result. A natural language response text is then generated and broadcast via a speech synthesis module.

[0007] As a preferred embodiment of the indoor navigation semantic orientation method based on a large language model described in this invention, the method involves: acquiring a time-synchronized continuous video stream and the spatial coordinates of each frame of the image through the robot's positioning sensor and image sensor to construct a complete dataset containing time alignment relationships, including the following steps: An autonomous robot equipped with vision sensors moves through an indoor environment and captures continuous video streams; Autonomous robots acquire the spatial coordinates of each video frame in a continuous video stream through simultaneous localization and mapping (SLAM) technology, and combine them to form a complete dataset.

[0008] As a preferred embodiment of the indoor navigation semantic orientation method based on a large language model according to the present invention, the method includes: extracting keyframes from the complete dataset based on a threshold for semantic change between adjacent frames; inputting each keyframe image, the spatial coordinates strictly aligned with the keyframe image in time, and a first structured output control parameter into a multimodal large language model; the first structured output control parameter constrains the model output to a predefined JSON structure containing object names and attribute information, thereby obtaining a structured JSON response containing preliminary description information of the target object; and including the following steps: Keyframe extraction is performed on the complete dataset based on the evaluation of the degree of semantic change between adjacent image frames, resulting in a list of keyframe images containing timestamps; From the complete dataset, based on the timestamps in the list of keyframe images, extract the spatial coordinates that are strictly aligned with the keyframe images in time, forming a keyframe image-spatial coordinate pairing list; From the keyframe image-spatial coordinate pairing list, extract each pair of keyframe images and spatial coordinates that are strictly aligned with the keyframe images in time; Each pair of keyframe images extracted, the spatial coordinates that are strictly aligned with the keyframe images in time, and the first structured output control parameters are input into the multimodal large-scale language model. The multimodal large-scale language model infers and outputs a structured JSON response based on the input keyframe image, the spatial coordinates that are strictly aligned with the keyframe image in time, and the first structured output control parameters. The structured JSON response contains preliminary description information of the target object.

[0009] As a preferred embodiment of the indoor navigation semantic orientation method based on a large language model described in this invention, the method includes the following steps: based on the structured JSON response corresponding to all keyframes, parsing and extracting spatial location coordinates, object names, and attribute information, and constructing a dedicated semantic database with the spatial location coordinates as the index key. Parse the structured JSON response and extract the spatial coordinates of each keyframe image, the name of the identified object, and the attribute information of each object. The spatial coordinates corresponding to each keyframe image are stored in the image position record table of the dedicated semantic database, the identified object names are stored in the object name record table of the dedicated semantic database, and the attribute information corresponding to each object is stored in the object attribute record table of the dedicated semantic database. A dedicated semantic database is built after data writing operations based on all structured JSON responses.

[0010] As a preferred embodiment of the indoor navigation semantic orientation method based on a large language model according to the present invention, the method includes: receiving a user's natural language navigation query, inputting the natural language navigation query and a second query parsing control parameter into a large language model, wherein the second query parsing control parameter is used to guide the model to parse the natural language query into target object keywords and attribute constraints and encapsulate them into a standardized JSON format, generating a standardized JSON description of the semantic constraints of the target object, including the following steps: Receive the user's natural language navigation query. After the user's natural language navigation query is obtained, input the user's natural language navigation query and the second preset prompt words into the large language model. The large language model processes the user's natural language navigation query and the second preset prompt words to generate a standardized JSON description; Based on the target keywords in the standardized JSON description, the cosine similarity between the target keywords and the word embedding vectors of all object categories and attribute values ​​is obtained in a dedicated semantic database; Based on the attribute conditions in the standardized JSON description, for each object in the candidate object set, query all its attribute information in a dedicated semantic database, and retain objects whose attribute information is consistent with the attribute conditions to form a valid object set. The object with the highest cosine similarity to the word embedding vector is selected from the effective object set, and the spatial coordinates of the keyframe image associated with the object are read from the dedicated semantic database as the precise spatial coordinates of the target object.

[0011] As a preferred embodiment of the indoor navigation semantic orientation method based on a large language model according to the present invention, the method includes the following steps: based on the standardized JSON description, performing cosine similarity retrieval of word embedding vectors and matching based on attribute key-value pairs in the dedicated semantic database: Based on the standardized JSON description, the target keywords and attribute conditions in the standardized JSON description are extracted, and the target keywords are converted into word embedding vectors of the target keywords; In the dedicated semantic database, the cosine similarity between the word embedding vector of the target keyword and the word embedding vectors of all object category names and attribute values ​​in the dedicated semantic database is used to obtain a semantic similarity set; Based on the semantic similarity set, the top few objects with the highest semantic similarity are selected to form a candidate object set. For each object in the candidate object set, all attribute information of the object is queried in the dedicated semantic database. The complete attribute information of the queried object is compared with the attribute conditions item by item. Objects whose attribute information is consistent with the attribute conditions are retained to form a valid object set. The object with the highest semantic similarity is selected from the valid object set. The spatial coordinates of the keyframe image associated with the object with the highest semantic similarity are read from the dedicated semantic database and used as the precise spatial coordinates of the target object.

[0012] As a preferred embodiment of the indoor navigation semantic orientation method based on a large language model described in this invention, the method involves determining the precise spatial coordinates of the target object by comparing the spatial coordinates of candidate items with the robot's current coordinates in real time to perform context filtering. This includes the following steps: Obtain the spatial coordinates of candidate items obtained through semantic retrieval, acquire the robot's current coordinates in real time through the robot's positioning sensors, and calculate the Euclidean distance between the robot's current coordinates and the spatial coordinates of each candidate item. The Euclidean distance is compared with a predetermined distance threshold, and the spatial coordinates of candidate items whose Euclidean distance is less than or equal to the predetermined distance threshold are selected. From the spatial coordinates of the selected candidate items, the precise spatial coordinates of the target object are determined.

[0013] As a preferred embodiment of the indoor navigation semantic orientation method based on a large language model according to the present invention, the method includes: controlling an autonomous robot to navigate to the precise spatial coordinates, and upon arrival, taking a snapshot of the current environment using an image sensor on the robot; inputting the current environment snapshot, the standardized JSON description, and third-vision verification control parameters into the multimodal large language model; the third-vision verification control parameters guiding the model to compare the snapshot content with the JSON description and generate a verification result; generating a natural language response text; and performing voice broadcasting through a speech synthesis module, including the following steps: The system plans the movement path of the autonomous robot and sends control commands to drive the autonomous robot to move to the precise spatial coordinates. The autonomous robot uses its built-in positioning sensors to determine whether it has reached the precise spatial coordinates. After the autonomous robot arrives at the precise spatial coordinates, the image sensor on the autonomous robot is triggered to take a snapshot of the current environment, and a third preset prompt word is prepared for visual verification and language generation using a large multimodal language model. The current environment snapshot, the standardized JSON description, and the third preset prompt words are input into the multimodal large language model. The multimodal large language model processes the input current environment snapshot, standardized JSON description, and third preset prompt words, and outputs natural language response text. The natural language response text is fed into the speech synthesis module, which converts the natural language response text into an audio signal and drives the speaker to broadcast the message.

[0014] In a second aspect, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, it implements any step of the indoor navigation semantic orientation method based on a large language model as described in the first aspect of the present invention.

[0015] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the indoor navigation semantic orientation method based on a large language model as described in the first aspect of the present invention.

[0016] The beneficial effects of this invention are as follows: By acquiring time-synchronized continuous video streams and spatial coordinates, and using a first structured output control parameter to drive a multimodal large-scale language model to parse keyframes, a structured JSON response containing object attribute information is generated. This, in turn, constructs a dedicated semantic database indexed by spatial location. This persistent and structured storage of environmental semantic information provides a stable and efficient data foundation for subsequent processing of complex queries with multi-attribute constraints, overcoming the retrieval delay and result instability issues caused by existing technologies relying on real-time model inference. Furthermore, by converting user natural language queries into standardized JSON descriptions using a second query parsing control parameter, and performing cosine similarity retrieval and attribute key-value pair matching based on word embedding vectors in the dedicated semantic database, combined with context filtering using the robot's current coordinates, a precise mapping from fuzzy semantic intent to specific physical coordinates is achieved. Crucially, after the robot reaches the target location, it takes a snapshot of the current environment and, in conjunction with the third-vision verification control parameters, calls the multimodal large-scale language model again to perform visual-language comparison, generating verification results and converting them into voice broadcasts. This forms a complete closed loop of perception-planning-verification technology, which not only ensures the accuracy of navigation results but also provides intuitive and reliable interactive feedback. Thus, it achieves reliable and accurate navigation and interaction from open natural language commands to specific physical objects. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of an indoor navigation semantic orientation method based on a large language model. Detailed Implementation

[0019] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0020] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0021] Secondly, the term "one embodiment" or "example" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the invention. The appearance of an embodiment in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that mutually excludes other embodiments.

[0022] Reference Figure 1 This is one embodiment of the present invention, which provides an indoor navigation semantic orientation method based on a large language model, including the following steps: S1. Using the robot's positioning and image sensors, collect time-synchronized continuous video streams and the spatial coordinates of each frame to construct a complete dataset containing time alignment relationships.

[0023] S1.1 An autonomous robot equipped with a vision sensor moves in an indoor environment and captures continuous video streams.

[0024] Furthermore, autonomous robots equipped with visual sensors initiate and perform movement operations in the target indoor environment. The movement path covers all or most of the passageway of the area to be served. During the movement, the visual sensors continuously capture video images of the surrounding environment, forming a continuous video stream. The continuous video stream contains multiple frames of images arranged in chronological order. Each frame of the image records the local environmental visual information observed by the autonomous robot at a certain moment. Through continuous movement and image capture, comprehensive visual coverage of the indoor environment is achieved, and finally, a continuous video stream is obtained.

[0025] S1.2 The autonomous robot obtains the spatial coordinates of each video frame in the continuous video stream through synchronous localization and mapping technology, and combines them to form a complete dataset.

[0026] Furthermore, while capturing continuous video streams, the autonomous robot runs simultaneous localization and mapping (SLAM) technology. Based on visual sensor input, inertial measurement unit data, and wheel odometry information, it estimates the pose of the autonomous robot in the indoor environment in real time and assigns a corresponding spatial coordinate to each video frame in the continuous video stream. The spatial coordinate represents the position and orientation of the autonomous robot in the global coordinate system when the video frame is captured. Then, the continuous video stream and the spatial coordinate corresponding to each video frame are timestamped, bound and stored, and combined to form a complete dataset.

[0027] S2. Extract keyframes from the complete dataset based on the semantic change threshold between adjacent frames. Input each keyframe image, the spatial coordinates that are strictly aligned with the keyframe image in time, and the first structured output control parameter into the multimodal large language model. The first structured output control parameter is used to constrain the model output to a predefined JSON structure containing object name and attribute information, thereby obtaining a structured JSON response containing preliminary description information of the target object.

[0028] S2.1 Extract keyframes from the complete dataset based on the evaluation of the degree of semantic change between adjacent image frames, and obtain a list of keyframe images containing timestamps.

[0029] Furthermore, based on continuous video streams in the complete dataset, the degree of semantic change between adjacent image frames is calculated by comparing the differences between adjacent image frames in the feature space. A semantic change threshold (e.g., 0.5) is set to determine whether the image content has changed significantly. Image frames whose semantic change degree first exceeds the semantic change threshold are marked as keyframes, and the timestamp corresponding to the keyframe in the video stream is recorded. Finally, a list of keyframe images containing timestamps is generated in chronological order.

[0030] S2.2 From the complete dataset, based on the timestamps in the list of keyframe images, extract the spatial coordinates that are strictly aligned with the keyframe images in time, forming a keyframe image-spatial coordinate pairing list.

[0031] Furthermore, in the spatial coordinate sequence of the complete dataset, based on each timestamp recorded in the list of keyframe images containing timestamps, a precise timestamp matching query is performed to find spatial coordinate records that are completely consistent with each keyframe timestamp from the spatial coordinate sequence. Each successfully matched keyframe image is bound to a spatial coordinate, forming a structured keyframe image-spatial coordinate pairing list. Each element in this list contains a keyframe image and a spatial coordinate that is strictly aligned in time.

[0032] S2.3 From the keyframe image-spatial coordinate pairing list, extract each pair of keyframe images and spatial coordinates that are strictly aligned with the keyframe images in time.

[0033] Furthermore, the keyframe image-spatial coordinate pairing list is traversed and accessed sequentially. Each element in the keyframe image-spatial coordinate pairing list is read sequentially, and the keyframe image data contained in the read element and the spatial coordinate data that are strictly aligned with the keyframe image in time are extracted from the read element, providing a processing unit for subsequent steps.

[0034] S2.4. Input each extracted pair of keyframe images, the spatial coordinates that are strictly aligned with the keyframe images in time, and the first structured output control parameters into the multimodal large-scale language model. Furthermore, from the keyframe image-spatial coordinate pairing list elements, the extracted keyframe image data and spatial coordinate data that are strictly aligned with the keyframe image in time are retrieved. At the same time, the first structured output control parameter is obtained from the preset instruction storage unit. The keyframe image data, the spatial coordinate data that are strictly aligned with the keyframe image in time, and the first structured output control parameter are combined into an input sample, and this input sample is loaded into the data input buffer of the multimodal large language model.

[0035] S2.5 The multimodal large-scale language model infers and outputs a structured JSON response based on the input keyframe image, the spatial coordinates that are strictly aligned with the keyframe image in time, and the first structured output control parameters. The structured JSON response contains preliminary description information of the target object.

[0036] Furthermore, after receiving the input sample, the multimodal large-scale language model activates its internal vision and text joint reasoning module to perform visual feature understanding on the keyframe images in the input sample, perform spatial information parsing on the spatial location coordinates that are strictly aligned with the keyframe images in time, and logically integrate the above understanding and parsing results according to the output format and task requirements defined by the first structured output control parameter, and finally generate a structured JSON response containing preliminary description information of the target object. This structured JSON response is returned in text format from the output interface of the multimodal large-scale language model.

[0037] S3. Based on the structured JSON response corresponding to all keyframes, parse and extract the spatial location coordinates, object name and attribute information, and construct a dedicated semantic database with the spatial location coordinates as the index key.

[0038] S3.1 Parse the structured JSON response and extract the spatial coordinates of each keyframe image, the name of the identified object, and the attribute information of each object.

[0039] Furthermore, each structured JSON response is parsed, and the content is read layer by layer according to the predefined field paths. From this, the spatial coordinates of the keyframe image associated with the response, one or more object names identified by the multimodal large language model, and several attribute information generated for each object name are extracted. The attribute information exists in the form of key-value pairs, including semantic dimensions such as color, size, and purpose. This ensures that all extracted data items remain consistent with the original description in the structured JSON response, thus completing the structured decomposition of the structured semantic content and obtaining storable structured data elements.

[0040] S3.2 Store the spatial coordinates corresponding to each keyframe image into the image position record table of the dedicated semantic database, store the identified object names into the object name record table of the dedicated semantic database, and store the attribute information corresponding to each object into the object attribute record table of the dedicated semantic database.

[0041] Furthermore, the spatial coordinates of each keyframe image extracted from the structured JSON response are written into the image location record table of a dedicated semantic database. Each record contains spatial coordinates and a unique image identifier. The name of each identified object, along with its associated unique image identifier, is written into the object name record table of the dedicated semantic database. For all attribute information corresponding to each object name, each attribute key and attribute value is combined into an independent record, associated with the same unique image identifier and object identifier, and then written into the object attribute record table of the dedicated semantic database. The logical association between the three record tables is achieved through the unique image identifier, thus completing the persistent storage of structured semantic data.

[0042] S3.3 After writing data based on all structured JSON responses, build a dedicated semantic database.

[0043] Furthermore, after traversing and processing all structured JSON responses, it is confirmed that the spatial coordinates, object names, and attribute information in each response have been completely written into the corresponding record table of the dedicated semantic database, and the relationship between the record tables is correct. At this point, the dedicated semantic database contains the objects corresponding to all keyframes in the indoor environment, as well as their fine-grained semantic attributes and spatial location information, and the construction of the dedicated semantic database is complete.

[0044] S4. Receive the user's natural language navigation query, and input the natural language navigation query and the second query parsing control parameter into the large language model. The second query parsing control parameter is used to guide the model to parse the natural language query into target object keywords and attribute constraints and encapsulate them into a standardized JSON format to generate a standardized JSON description of the semantic constraints of the target object.

[0045] S4.1 Receive the user's natural language navigation query. After the user's natural language navigation query is obtained, input the user's natural language navigation query and the second preset prompt words into the large language model. Furthermore, the system receives natural language navigation queries from users via a voice recognition device or text input interface. These queries express the user's navigation intent in natural language, such as asking for the location of a specific object or requesting to go to a certain functional area. After obtaining the complete natural language navigation query, the system concatenates or structurally combines the query with a second preset prompt word to form a complete text input. This text input explicitly instructs the large language model to extract the target object and required attributes from the natural language navigation query and organize the output according to a specified format. The text input is then submitted to the large language model, completing the initial encapsulation and transmission of the user's intent.

[0046] S4.2 The large-scale language model processes the user's natural language navigation query and the second preset prompt words to generate a standardized JSON description.

[0047] Furthermore, after receiving text input containing a natural language navigation query and a second preset prompt word, the large language model performs semantic parsing on the natural language navigation query according to the instructions in the second preset prompt word, identifies the core target keywords and possible attribute constraints, eliminates colloquial expressions, redundant information or ambiguous references, organizes the parsing results into a machine-readable format containing only necessary navigation elements according to a predefined data structure, and finally outputs a standardized JSON description. The standardized JSON description accurately reflects the user's real navigation needs and does not contain any additional explanatory content.

[0048] S4.3 Based on the target keywords in the standardized JSON description, obtain the cosine similarity between the target keywords and the word embedding vectors of all object categories and attribute values ​​in a dedicated semantic database.

[0049] Furthermore, target keywords are extracted from the standardized JSON description and input into the word embedding model to generate corresponding semantic vectors. Then, all object category names and attribute values ​​are traversed in the object name record table and object attribute record table of the dedicated semantic database. They are converted into semantic vectors by the same word embedding model. The cosine similarity between the semantic vector of the target keyword and the semantic vector of each object category name and attribute value is calculated to obtain the set of cosine similarity between the target keyword and the word embedding vectors of all object categories and attribute values. The top few items are selected to form a candidate object set after being sorted from high to low according to the cosine similarity of the word embedding vectors.

[0050] S4.4 Based on the attribute conditions in the standardized JSON description, query all attribute information of each object in the candidate object set in a dedicated semantic database, and retain objects whose attribute information matches the attribute conditions to form a valid object set.

[0051] Furthermore, one or more attribute conditions are parsed from the standardized JSON description. For each object in the candidate object set, all attribute information associated with the object is queried in the object attribute record table of the dedicated semantic database based on its unique image identifier. The queried attribute information is compared item by item with the attribute conditions in the standardized JSON description. Only when all attribute conditions are met is the object retained in the filtering results. Finally, all objects that meet the attribute conditions form the valid object set.

[0052] S4.5 Select the object with the highest cosine similarity to the word embedding vector from the effective object set, and read the spatial coordinates of the keyframe image associated with the object from the dedicated semantic database as the precise spatial coordinates of the target object.

[0053] Furthermore, the object with the highest cosine similarity to the word embedding vector is found in the effective object set. The cosine similarity of the word embedding vector comes from the highest matching score associated with the object. After the object is identified, its spatial location coordinates corresponding to the keyframe image associated with it are retrieved from the image location record table of the dedicated semantic database through its unique image identifier, and the spatial location coordinates are used as the precise spatial location coordinates of the target object.

[0054] S5. Based on the standardized JSON description, perform cosine similarity retrieval of word embedding vectors and matching based on attribute key-value pairs in the dedicated semantic database.

[0055] S5.1 Based on the standardized JSON description, extract the target keywords and attribute conditions from the standardized JSON description, and convert the target keywords into word embedding vectors of the target keywords.

[0056] Furthermore, the standardized JSON description is parsed to extract the explicitly specified target keywords and one or more attribute conditions; then, the extracted target keywords are input into a pre-trained word embedding model to generate a fixed-dimensional, numerical word embedding vector of the target keywords.

[0057] S5.2 In the dedicated semantic database, the cosine similarity between the word embedding vector of the target keyword and the word embedding vectors of all object category names and attribute values ​​in the dedicated semantic database is used to obtain a semantic similarity set.

[0058] Furthermore, all object category names and attribute values ​​stored in the dedicated semantic database are pre-converted into corresponding word embedding vectors using the same pre-trained word embedding model and stored. In this step, the cosine similarity between the word embedding vector of the target keyword and the word embedding vector of each object category name and attribute value in the dedicated semantic database is calculated, and all calculation results are summarized to form a semantic similarity set.

[0059] S5.3. Based on the semantic similarity set, select the top few objects with the highest semantic similarity to form a candidate object set. For each object in the candidate object set, query all attribute information of the object in the dedicated semantic database.

[0060] Furthermore, all entries in the semantic similarity set are sorted from high to low according to their similarity scores, and the top-ranked objects are selected to form a candidate object set. Then, each object in the candidate object set is traversed, and based on its unique identifier in the dedicated semantic database, all attribute information associated with the object is queried and obtained from the object attribute record table in the dedicated semantic database.

[0061] S5.4. Compare all the attribute information of the queried object with the attribute conditions item by item, retain the objects whose attribute information is consistent with the attribute conditions, form a valid object set, and select the object with the highest semantic similarity from the valid object set.

[0062] Furthermore, all attribute information of each object is compared item by item with the attribute conditions extracted from the standardized JSON description to check whether they are fully satisfied; all objects that meet the attribute conditions are retained to form a set of valid objects; finally, within the set of valid objects, the object with the highest semantic similarity is selected based on its value in the set of semantic similarity.

[0063] S5.5 Read the spatial coordinates of the keyframe image associated with the object with the highest semantic similarity from the dedicated semantic database, and use them as the precise spatial coordinates of the target object.

[0064] Furthermore, based on the object with the highest semantic similarity, the unique identifier of the keyframe image corresponding to the object is located through its association in the dedicated semantic database; then, in the image location record table of the dedicated semantic database, the unique identifier of the keyframe image is used to query and read its corresponding spatial location coordinates, and these coordinates are determined as the precise spatial location coordinates of the target object.

[0065] S6. By comparing the spatial coordinates of the candidate items with the robot's current coordinates in real time to complete context filtering, the precise spatial coordinates of the target object are determined.

[0066] S6.1 Obtain the spatial coordinates of the candidate items obtained through semantic retrieval, acquire the robot's current coordinates in real time through the robot's positioning sensor, and calculate the Euclidean distance between the robot's current coordinates and the spatial coordinates of each candidate item.

[0067] Furthermore, the spatial coordinates of candidate entries obtained through cosine similarity retrieval based on word embedding vectors and matching based on attribute key-value pairs are acquired. Meanwhile, the robot's current coordinates are acquired in real time through the robot's positioning sensors. Then, the spatial coordinates of each candidate entry are traversed, and the Euclidean distance between the robot's current coordinates and the spatial coordinates of each candidate entry is calculated, thus obtaining a list containing all corresponding Euclidean distance values.

[0068] S6.2. Compare the Euclidean distance with a predetermined distance threshold, filter out the spatial coordinates of candidate items whose Euclidean distance is less than or equal to the predetermined distance threshold, and determine the precise spatial coordinates of the target object from the spatial coordinates of the selected candidate items.

[0069] Furthermore, each Euclidean distance is compared with a predetermined distance threshold, such as 0.5 meters, to filter out the spatial coordinates of candidate items whose Euclidean distance is less than or equal to the predetermined distance threshold, forming a neighboring candidate set. Then, according to preset rules, such as directly selecting or combining semantic similarity scores, the spatial coordinates of a candidate item are finally determined from the neighboring candidate set as the precise spatial coordinates of the target object.

[0070] S7. Control the autonomous robot to navigate to the precise spatial coordinates, and after arrival, take a snapshot of the current environment using the image sensor on the robot. Input the current environment snapshot, the standardized JSON description, and the third visual verification control parameters into the multimodal large language model. The third visual verification control parameters are used to guide the model to compare the snapshot content with the JSON description and generate verification results. Generate natural language response text and broadcast it through the speech synthesis module.

[0071] S7.1 Plan the movement path of the autonomous robot and send control commands to the autonomous robot to drive it to move to the precise spatial coordinates. The autonomous robot uses its built-in positioning sensors to determine whether it has reached the precise spatial coordinates.

[0072] Furthermore, the system plans the movement path of the autonomous robot from its current position to its precise spatial coordinates. Based on the movement path, it generates a continuous sequence of control commands containing speed and direction instructions and sends this sequence to the robot's motion controller. The autonomous robot moves according to the sequence of control commands. During the movement, the robot's current coordinates are collected in real time by its positioning sensors. The current coordinates are compared with the precise spatial coordinates. When the distance between the current coordinates and the precise spatial coordinates is less than a predetermined threshold (e.g., 0.1 meters), it is determined that the autonomous robot has reached the precise spatial coordinates.

[0073] S7.2 After the autonomous robot arrives at the precise spatial coordinates, the image sensor on the autonomous robot is triggered to take a snapshot of the current environment, and a third preset prompt word is prepared for visual verification and language generation using a forced multimodal large language model.

[0074] Furthermore, after the conditions for the autonomous robot to reach the precise spatial coordinates are met, a shooting command is sent to the image sensor on the autonomous robot. The image sensor executes the shooting command to capture a snapshot of the current environment. At the same time, a third preset prompt word is read from the preset text storage unit. The third preset prompt word is used to force the multimodal large language model to perform visual verification and language generation tasks.

[0075] S7.3. Input the current environment snapshot, the standardized JSON description, and the third preset prompt words into the multimodal large-scale language model. The multimodal large-scale language model processes the input current environment snapshot, standardized JSON description, and third preset prompt words, and outputs natural language response text. Furthermore, the current environment snapshot is obtained from the output buffer of the image sensor, the standardized JSON description is obtained from the output parsing unit of the language model, and the third preset prompt word is obtained from the text storage unit. The current environment snapshot, the standardized JSON description, and the third preset prompt word are used as a set of multimodal input data and fed into the multimodal large-scale language model. The multimodal large-scale language model performs joint inference on the input current environment snapshot, standardized JSON description, and third preset prompt word to generate a natural language response text containing a state description of the matching between the visual content of the current environment snapshot and the standardized JSON description.

[0076] S7.4. The natural language response text is sent to the speech synthesis module, which converts the natural language response text into an audio signal and drives the speaker to broadcast the message. Furthermore, the natural language response text is obtained from the text output interface of the multimodal large language model. The natural language response text is sent as string data to the text input interface of the speech synthesis module. The speech synthesis module performs text-to-speech conversion processing on the received natural language response text string to generate an audio signal corresponding to the content of the natural language response text. The audio signal is transmitted to the speaker driver circuit, which drives the speaker to convert the audio signal into an audible sound signal for speech playback.

[0077] This invention also includes, for example, the first structured output control parameter can be designed as: Please analyze this image and associated coordinates. List all visible objects and output the name, color, size, status, and relative position to coordinates of each object in JSON format. The second preset prompt can be: Parse the user query [original user query text] into JSON format, strictly including the two keys target_object and attributes. The third preset prompt can be: Please verify whether the target object in the standardized JSON description exists in the current environment. If it exists, describe its status; if it does not exist, explain the possible reasons. Please reply in colloquial Chinese. Secondly, it is recommended to add a technical effect analysis paragraph at the end of the invention content or specific implementation, quantifying the advantages by citing simulation experiments or theoretical comparisons. For example, it can be stated: According to simulation tests, with the support of the method described in this invention, the success rate of the robot in handling multi-attribute constraint queries such as finding an empty workstation near a window is improved by about 35% compared to the traditional method that relies on real-time VLM inference; through real-time coordinate filtering, in a dynamic environment with multiple similar targets, the positioning accuracy (success rate) can be improved by about 20%. Finally, the technical field and invention content sections should further emphasize hardware dependence and system integration, clarifying that this method is based on an autonomous robot equipped with a visual sensor, a high-precision positioning sensor (such as LiDAR, IMU), a SLAM algorithm computing unit, and a speech synthesis module, thereby highlighting the technical solution's attributes rather than a pure algorithm.

[0078] This embodiment also provides a computer device applicable to the indoor navigation semantic orientation method based on a large language model, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the indoor navigation semantic orientation method based on a large language model as proposed in the above embodiment.

[0079] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0080] This embodiment also provides a storage medium on which a computer program is stored. When executed by a processor, the program implements the indoor navigation semantic orientation method based on a large language model as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0081] In summary, by acquiring time-synchronized continuous video streams and spatial coordinates, and using the first structured output control parameters to drive a multimodal large-scale language model to parse keyframes, a structured JSON response containing object attribute information is generated. This, in turn, constructs a dedicated semantic database indexed by spatial location. This approach persistently and structurally stores environmental semantic information, providing a stable and efficient data foundation for subsequent processing of complex queries with multi-attribute constraints. It overcomes the retrieval latency and result instability issues caused by existing technologies relying on real-time model inference. Furthermore, by converting user natural language queries into standardized JSON descriptions using the second query parsing control parameters, and performing cosine similarity retrieval and attribute key-value pair matching based on word embedding vectors in the dedicated semantic database, combined with context filtering using the robot's current coordinates, a precise mapping from fuzzy semantic intent to specific physical coordinates is achieved. Crucially, after the robot reaches the target location, it takes a snapshot of the current environment and, in conjunction with the third-vision verification control parameters, calls the multimodal large-scale language model again to perform visual-language comparison, generating verification results and converting them into voice broadcasts. This forms a complete closed loop of perception-planning-verification technology, which not only ensures the accuracy of navigation results but also provides intuitive and reliable interactive feedback. Thus, it achieves reliable and accurate navigation and interaction from open natural language commands to specific physical objects.

[0082] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for semantic orientation of indoor navigation based on large language models, characterized in that: This includes using the robot's positioning and image sensors to collect time-synchronized continuous video streams and the spatial coordinates of each frame to construct a complete dataset containing time alignment relationships. Keyframe extraction based on the semantic change threshold between adjacent frames is performed on the complete dataset. Each keyframe image, the spatial location coordinates that are strictly aligned with the keyframe image in time, and the first structured output control parameter are input into a multimodal large language model. The first structured output control parameter is used to constrain the model output to a predefined JSON structure containing object name and attribute information, thereby obtaining a structured JSON response containing preliminary description information of the target object. Based on the structured JSON response corresponding to all keyframes, the spatial location coordinates, object name and attribute information are parsed and extracted. A dedicated semantic database is constructed with the spatial location coordinates as the index key. The user's natural language navigation query is received. The natural language navigation query and the second query parsing control parameter are input into the large language model. The second query parsing control parameter is used to guide the model to parse the natural language query into target object keywords and attribute constraints and encapsulate them into a standardized JSON format to generate a standardized JSON description of the semantic constraints of the target object. Based on the standardized JSON description, cosine similarity retrieval of word embedding vectors and matching based on attribute key-value pairs are performed in the dedicated semantic database. Context filtering is completed by comparing the spatial coordinates of candidate entries with the robot's current coordinates in real time, thereby determining the precise spatial coordinates of the target object. The autonomous robot is controlled to navigate to the precise spatial coordinates. Upon arrival, the robot takes a snapshot of the current environment using its onboard image sensor. The snapshot, the standardized JSON description, and the third-vision verification control parameters are then input into the multimodal large-scale language model. The third-vision verification control parameters guide the model to compare the snapshot content with the JSON description and generate a verification result. A natural language response text is then generated and broadcast via a speech synthesis module.

2. The indoor navigation semantic orientation method based on a large language model as described in claim 1, characterized in that: By using the robot's positioning and image sensors, a time-synchronized continuous video stream and the spatial coordinates of each frame are acquired to construct a complete dataset containing time-aligned relationships. This includes the following steps: An autonomous robot equipped with vision sensors moves through an indoor environment and captures continuous video streams; Autonomous robots acquire the spatial coordinates of each video frame in a continuous video stream through simultaneous localization and mapping (SLAM) technology, and combine them to form a complete dataset.

3. The indoor navigation semantic orientation method based on a large language model as described in claim 2, characterized in that: Keyframe extraction based on a semantic change threshold between adjacent frames is performed on the complete dataset. Each keyframe image, its spatial coordinates that are strictly aligned with the keyframe image in time, and a first structured output control parameter are input into a multimodal large-scale language model. The first structured output control parameter is used to constrain the model output to a predefined JSON structure containing object names and attribute information, thereby obtaining a structured JSON response containing preliminary description information of the target object. The process includes the following steps: Keyframes were extracted from the complete dataset based on the evaluation of the degree of semantic change between adjacent image frames, resulting in a list of keyframe images containing timestamps. From the complete dataset, based on the timestamps in the list of keyframe images, extract the spatial coordinates that are strictly aligned with the keyframe images in time, and form a keyframe image-spatial coordinate pairing list. From the keyframe image-spatial coordinate pairing list, extract each pair of keyframe images and spatial coordinates that are strictly aligned with the keyframe images in time; Each pair of keyframe images extracted, the spatial coordinates that are strictly aligned with the keyframe images in time, and the first structured output control parameters are input into the multimodal large-scale language model. The multimodal large-scale language model infers and outputs a structured JSON response based on the input keyframe image, the spatial coordinates that are strictly aligned with the keyframe image in time, and the first structured output control parameters. The structured JSON response contains preliminary description information of the target object.

4. The indoor navigation semantic orientation method based on a large language model as described in claim 3, characterized in that: Based on the structured JSON response corresponding to all keyframes, the spatial location coordinates, object names, and attribute information are parsed and extracted to construct a dedicated semantic database indexed by the spatial location coordinates. This includes the following steps: Parse the structured JSON response and extract the spatial coordinates of each keyframe image, the name of the identified object, and the attribute information of each object. The spatial coordinates corresponding to each keyframe image are stored in the image position record table of the dedicated semantic database, the identified object names are stored in the object name record table of the dedicated semantic database, and the attribute information corresponding to each object is stored in the object attribute record table of the dedicated semantic database. A dedicated semantic database is built after data writing operations based on all structured JSON responses.

5. The indoor navigation semantic orientation method based on a large language model as described in claim 4, characterized in that: The system receives a user's natural language navigation query and inputs the natural language navigation query and a second query parsing control parameter into a large language model. The second query parsing control parameter guides the model to parse the natural language query into target object keywords and attribute constraints, encapsulate them into a standardized JSON format, and generate a standardized JSON description of the semantic constraints of the target object. This includes the following steps: Receive the user's natural language navigation query. After the user's natural language navigation query is obtained, input the user's natural language navigation query and the second preset prompt words into the large language model. The large language model processes the user's natural language navigation query and the second preset prompt words to generate a standardized JSON description; Based on the target keywords in the standardized JSON description, the cosine similarity between the target keywords and the word embedding vectors of all object categories and attribute values ​​is obtained in a dedicated semantic database; Based on the attribute conditions in the standardized JSON description, for each object in the candidate object set, query all its attribute information in a dedicated semantic database, and retain objects whose attribute information is consistent with the attribute conditions to form a valid object set. The object with the highest cosine similarity to the word embedding vector is selected from the effective object set, and the spatial coordinates of the keyframe image associated with the object are read from the dedicated semantic database as the precise spatial coordinates of the target object.

6. The indoor navigation semantic orientation method based on a large language model as described in claim 5, characterized in that: Based on the standardized JSON description, the following steps are performed in the dedicated semantic database: cosine similarity retrieval of word embedding vectors and matching based on attribute key-value pairs. Based on the standardized JSON description, the target keywords and attribute conditions in the standardized JSON description are extracted, and the target keywords are converted into word embedding vectors of the target keywords; In the dedicated semantic database, the cosine similarity between the word embedding vector of the target keyword and the word embedding vectors of all object category names and attribute values ​​in the dedicated semantic database is used to obtain a semantic similarity set; Based on the semantic similarity set, the top few objects with the highest semantic similarity are selected to form a candidate object set. For each object in the candidate object set, all attribute information of the object is queried in the dedicated semantic database. The complete attribute information of the queried object is compared with the attribute conditions item by item. Objects whose attribute information is consistent with the attribute conditions are retained to form a valid object set. The object with the highest semantic similarity is selected from the valid object set. The spatial coordinates of the keyframe image associated with the object with the highest semantic similarity are read from the dedicated semantic database and used as the precise spatial coordinates of the target object.

7. The indoor navigation semantic orientation method based on a large language model as described in claim 6, characterized in that: Context filtering is performed by comparing the spatial coordinates of candidate items with the robot's current coordinates in real time to determine the precise spatial coordinates of the target object. This includes the following steps: Obtain the spatial coordinates of candidate items obtained through semantic retrieval, acquire the robot's current coordinates in real time through the robot's positioning sensors, and calculate the Euclidean distance between the robot's current coordinates and the spatial coordinates of each candidate item. The Euclidean distance is compared with a predetermined distance threshold, and the spatial coordinates of candidate items whose Euclidean distance is less than or equal to the predetermined distance threshold are selected. From the spatial coordinates of the selected candidate items, the precise spatial coordinates of the target object are determined.

8. The indoor navigation semantic orientation method based on a large language model as described in claim 7, characterized in that, The autonomous robot is controlled to navigate to the precise spatial coordinates. Upon arrival, the robot's onboard image sensor captures a snapshot of the current environment. This snapshot, the standardized JSON description, and third-vision verification control parameters are input into the multimodal large-scale language model. The third-vision verification control parameters guide the model to compare the snapshot content with the JSON description and generate a verification result. A natural language response text is then generated and broadcast via a speech synthesis module. The process includes the following steps: The system plans the movement path of the autonomous robot and sends control commands to drive the autonomous robot to move to the precise spatial coordinates. The autonomous robot uses its built-in positioning sensors to determine whether it has reached the precise spatial coordinates. After the autonomous robot arrives at the precise spatial coordinates, the image sensor on the autonomous robot is triggered to take a snapshot of the current environment, and a third preset prompt word is prepared for visual verification and language generation using a large multimodal language model. The current environment snapshot, the standardized JSON description, and the third preset prompt words are input into the multimodal large language model. The multimodal large language model processes the input current environment snapshot, standardized JSON description, and third preset prompt words, and outputs natural language response text. The natural language response text is fed into the speech synthesis module, which converts the natural language response text into an audio signal and drives the speaker to broadcast the message.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the indoor navigation semantic orientation method based on a large language model as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the indoor navigation semantic orientation method based on a large language model as described in any one of claims 1 to 8.

Images