Intelligent agent, indoor navigation method and equipment thereof, medium and product

Abstract

The invention discloses an intelligent agent, an indoor navigation method and equipment thereof, a medium and a product, and relates to the field of navigation control. The method comprises the following steps: constructing an indoor scene multi-modal knowledge base by adopting a visual language model and utilizing a retrieval enhancement generation strategy; acquiring a user instruction and an indoor environment image; position information is retrieved from an indoor scene multi-mode knowledge base based on the user instruction and the indoor environment image; and matching a navigation strategy algorithm based on the position information, updating and verifying the indoor scene multi-mode knowledge base, and matching the information accuracy. According to the invention, multi-mode instructions including natural languages and images can be understood, and accurate navigation can be carried out.

Description

Technical Field

[0001] The present application relates to the field of navigation control, and in particular to an intelligent agent and its indoor navigation method, equipment, medium and product. Background Art

[0002] With the rapid development of large-scale visual language models (VLMs), the potential of VLMs in various practical applications has attracted more and more attention. Currently, it is still difficult to build a navigation strategy that can understand multimodal instructions including natural language and images and provide feedback for applying VLMs to navigation in unknown environments, especially in indoor zero-shot navigation. Summary of the invention

[0003] The purpose of this application is to provide an intelligent agent and its indoor navigation method, device, medium and product, which can understand multimodal instructions including natural language and images and perform precise navigation.

[0004] To achieve the above objectives, this application provides the following solutions: In a first aspect, the present application provides an indoor navigation method, comprising: A visual language model is adopted and a retrieval-enhanced generation strategy is used to construct a multimodal knowledge base of indoor scenes; the multimodal knowledge base of indoor scenes includes multiple sets of position information; each set of position information is triple data; the triple data is used to represent the scene type, the number of indoor objects, and the positional relationship and hierarchical relationship between indoor objects; Obtain user instructions and indoor environment images; Retrieving location information from the indoor scene multimodal knowledge base based on the user instruction and the indoor environment image; Based on the location information, the navigation strategy algorithm is matched and the indoor scene multimodal knowledge base is updated and verified to ensure the accuracy of the matching information.

[0005] Optionally, the process of building a visual language model includes: The BLIP-2 strategy is adopted to process the data in the SCENEVERSE dataset, and in the indoor data alignment and generation stage, Q-Former is used to align the query embedding with the text embedding of the visual language model to obtain the visual language model; the SCENEVERSE dataset includes diverse indoor scenes and semantic annotations corresponding to the indoor scenes.

[0006] Optionally, a hybrid retrieval strategy is adopted to retrieve location information from the indoor scene multimodal knowledge base based on the user instruction and the indoor environment image; the hybrid retrieval strategy includes global vector retrieval and graph retrieval.

[0007] Optionally, the process of global vector retrieval includes: The inner product of a document vector or an image vector is used to represent the correlation; the document vector is generated by a document encoder based on a user instruction; and the image vector is generated by an image encoder based on the indoor environment image.

[0008] Optionally, the graph retrieval process includes: generating a text subgraph based on the user instruction; A subgraph structure related to the text subgraph is retrieved from the indoor scene multimodal knowledge base.

[0009] In a second aspect, the present application provides an intelligent agent, comprising: a processing unit, a positioning unit, a speech recognition unit, a visual unit, and an instruction execution carrier; The processing unit, the positioning unit, the voice recognition unit and the visual unit are all mounted on the instruction execution carrier; the positioning unit, the voice recognition unit and the visual unit are all connected to the processing unit; the positioning unit is used to obtain positioning information; the voice recognition unit is used to obtain user instructions; the visual unit is used to obtain indoor environment images; The processing unit is used to implement the indoor navigation method provided above based on the positioning information, the user instruction and the indoor environment image, and execute the user instruction.

[0010] Optionally, the instruction execution carrier is a robot.

[0011] In a third aspect, the present application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any one of the indoor navigation methods described above.

[0012] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the indoor navigation methods described above.

[0013] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of any one of the indoor navigation methods described above.

[0014] According to the specific embodiments provided in this application, this application has the following technical effects: The present application provides an intelligent agent and its indoor navigation method, device, medium and product, which adopts a visual language model and uses a retrieval enhancement generation strategy to build a multimodal knowledge base of indoor scenes, and can understand multimodal instructions including natural language and images, so as to update the multimodal knowledge base of indoor scenes in real time. Based on user instructions and indoor environment images, location information is retrieved from the indoor scene multimodal knowledge base to generate a navigation strategy, which can achieve accurate navigation in the indoor environment. In addition, since the visual language model updates the indoor scene multimodal knowledge base in real time, combined with the indoor environment image obtained by the intelligent agent, the intelligent agent can complete user instructions between different floors. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 A flowchart of an indoor navigation method provided by an embodiment of the present application; Figure 2 An implementation architecture diagram of an indoor navigation method provided in one embodiment of the present application; Figure 3 A hybrid search strategy architecture diagram provided for an embodiment of the present application; Figure 4 A schematic diagram of an application flow of an indoor navigation method performed by an intelligent agent provided in an embodiment of the present application; Figure 5 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0018] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0019] like Figure 1 As shown, the present application provides an indoor navigation method, including the following steps 100 to 103. Among them: Step 100: Use the visual language model and the retrieval enhancement generation strategy to build a multimodal knowledge base for indoor scenes. The multimodal knowledge base for indoor scenes (referred to as the multimodal knowledge base or knowledge base) includes multiple sets of location information. Each set of location information is a triple data. The triple data is used to represent the scene type, the number of indoor objects, and the location relationship and hierarchical relationship between indoor objects.

[0020] Step 101: Obtain user instructions and indoor environment images.

[0021] Step 102: Retrieve location information from an indoor scene multimodal knowledge base based on user instructions and the indoor environment image.

[0022] Step 103: Match the navigation strategy algorithm based on the location information and update and verify the indoor scene multimodal knowledge base to ensure the accuracy of the matching information.

[0023] By implementing the above steps 100 to 103 , it is possible to understand multimodal instructions including natural language and images and perform precise navigation.

[0024] In another exemplary embodiment of the present application, in order to obtain a real-time updated contextual visual language model (VLM) to understand and process complex visual and language information. A large-scale 3D visual-language dataset, such as the SCENEVERSE dataset, can be used, the BLIP-2 strategy is used to process the data in the SCENEVERSE dataset, and in the indoor data alignment and generation stage, Q-Former is used to align the query embedding with the text embedding of the visual language model to obtain a visual language model. Among them, the SCENEVERSE dataset contains diverse indoor scenes and rich semantic annotations, which provides sufficient training data for the visual language model. Q-Former is a lightweight transformer model that uses a set of learnable query vectors to extract visual features from a frozen image encoder Q to provide the most useful visual features for the VLM to output the required text.

[0025] The BLIP-2 strategy (Bootstrapping Language-Image Pre-training) freezes the image encoder and uses the query transformer (Q-Former) to capture visual features while freezing the visual language model to facilitate the transition from vision to language generation. The BLIP-2 strategy can adopt image-text comparison (ITC) learning and image-text generation (ITG) strategies to ensure that the resulting visual language model can accurately generate high-quality text descriptions.

[0026] In the alignment and generation stages, the purpose of using Q-Former as a soft visual cue is to simplify the task of aligning visual and textual information in the visual language model.

[0027] In another exemplary embodiment of the present application, the ARKitScenes dataset, the HM3D (Habitat-Matterport 3D) dataset and the MultiScan dataset may also be used for implementation. ARKitScenes is a dataset designed using the ARKit framework that contains diverse indoor environments with high-quality 3D reconstructions and semantic annotations.

[0028] HM3D is a large-scale dataset for embodied AI (Artificial Intelligence) research, with thousands of high-resolution 3D scans of real-world indoor spaces, accompanied by detailed object and semantic information.

[0029] MultiScan combines 3D scan data from multiple sources to create comprehensive and diverse 3D scenes, including real and synthetic data, providing a wide range of environments.

[0030] In another exemplary embodiment of the present application, in step 100, in the process of constructing the indoor scene multimodal knowledge base using the Retrieval-Augmented Generation (RAG) strategy, since the RAG strategy combines contextual understanding and common sense reasoning, it ensures that more complex instructions can be processed by continuously updating and expanding the knowledge base. The constructed indoor scene multimodal knowledge base can integrate the associated information of different scenes and objects through the hierarchical affiliation of the layers to form a comprehensive knowledge system.

[0031] In another exemplary embodiment of the present application, based on the knowledge base result constructed in step 100, the obtained navigation strategy can be used to perform real-time navigation, so that the instruction execution body (such as a robot) can perform zero-sample navigation tasks in an unknown environment, understand and feedback complex multi-modal instructions. Among them, the process of implementing the indoor navigation method is as follows: Figure 2 shown. Figure 2 In the example, Json{imgID:caption(x,y)} is a representation of updating metadata using the Json tool.

[0032] The navigation strategy combines real-time contextual environment understanding and common sense reasoning to dynamically adjust the path and goal to ensure navigation accuracy and efficiency.

[0033] With the support of the knowledge base, the intelligent agent can use the known scene and object information to make inferences and decisions, thereby improving the success rate of navigation. The overall implementation process can be found in Figure 3 Content shown.

[0034] In another exemplary embodiment of the present application, in order to efficiently retrieve knowledge related to the query, especially focusing on the entity information in the query, a dual similarity measurement method can be provided, which consists of two parts: full set vector retrieval and graph retrieval. Based on this, in this embodiment, the present application adopts a hybrid retrieval strategy to implement step 102. Figure 3 As shown in Figure 2, the implementation process of the hybrid retrieval strategy can be described as: (1) Global vector retrieval: mainly measured by the inner product of the global vector.

[0035] In the global vector retrieval process, a multimodal (image and text) retrieval method is mixed to enhance the retrieval effect. Specifically, the relevance between the query and the document or image is obtained by fusing the vector representations of different modalities and calculating their inner product. This process not only considers the information of a single modality, but also integrates multimodal information, thereby improving the comprehensiveness and accuracy of the retrieval. The formula for determining the relevance is expressed as: .

[0036] In the formula, Represents a relevance scoring function that calculates the relevance between a query vector and a document or image vector. represents the fusion vector of query and image, represents the fusion vector of document and text, Indicates a query (Query), is the vector representation of the image. is the vector representation of the document, It is the vector representation of text.

[0037] (2) Graph retrieval: A text graph is a graph consisting of nodes and edges with text attributes, which can be formally represented as , where V and E represent the node set and edge set of the text graph respectively. and They represent the natural language attributes and image attributes of the corresponding nodes in the graph, Represents the natural language attributes of the edge.

[0038] The task to be achieved is to retrieve the subgraph from the graph. The retrieved subgraph is represented as: .

[0039] in and denote the power sets of V and E respectively, Represent the node set and edge set of the subgraph respectively.

[0040] For a multimodal knowledge base of indoor scenes, we can rely on image attributes Distinguish different scenes.

[0041] Graph retrieval enhancement generation means that given a specific question q on a graph G, there must be a relevant subgraph structure in the graph G that can be used to help generate more reliable and explanatory answers. Retrieve the top k most relevant subgraph structures.

[0042] The 3D feature vector of the corresponding noun and the vector of the subgraph entity are used to calculate the similarity and select the correct matching space. Then, the entity features identified by the noun extractor and the subgraph entity are used to calculate the discrete similarity. The sum of these two similarities is used to measure the entity relevance, which is: .

[0043] .

[0044] In the formula, Represented by maximizing the image vector and noun entity vector The inner product of , selects the best matching index, that is, finds the most relevant noun or entity. Noun entity vector The embedded feature vector representing the noun or entity corresponding to the image vector I is usually obtained through some vectorization method (such as word embedding or entity embedding). Represents the relevance score function, calculating the query vector With image embedding vector I k and the embedding vector of the entity E j The similarity between them is combined to get the final matching score. Indicates k The image embedding vectors are selected to be relevant to the query, usually obtained through multimodal retrieval methods. E j Indicates j The embedding vector of an entity, which can be an object represented by text or other modalities. Represents the query vector The embedding vector of the target entity The similarity score between them is usually used to measure the semantic similarity between the query and the target entity. The embedding vector representing the target entity is usually associated with a specific query and represents the target object to be retrieved. Represents the image embedding vector I k Embedding vector with the underlying entity The similarity score between them measures the similarity between the image and the underlying entity in a specific feature space. The embedding vector representing the underlying entity is used to measure the information of other reference entities.

[0045] In order to generate the search results, the retrieved information is filtered using re-ranking, and the two search results are re-ranked by sorting the similarity of the two searches.

[0046] Since hybrid retrieval mixes graph retrieval and text retrieval, the text information contained in it may also be repeated, which affects the generator performance. To avoid redundancy, when re-ranking, if the candidate information is too similar to the already retrieved documents (for example, the relevance score > 0.9), the retrieval result will be skipped.

[0047] Based on the above description, in this embodiment, the process of answer generation is: For a specific question q and image vector I, retrieve the text and subgraph topology from the knowledge base, convert the subgraph topology into text, and input it into the visual language model, as follows: .

[0048] In the formula, represents the text retrieved from the knowledge base, represents the subgraph retrieved from the knowledge base, Represents a distance metric based on d Before selection K Related entities. Indicates that in the query vector Under the condition of This is the retrieval step, where the system retrieves the most relevant text based on the query q. and subgraph . represents the query vector used for retrieval, Indicates that in the query vector Under the condition of The probability distribution of . Represents the number of time steps in a sequence model. Indicates that the condition Next, target The probability distribution of . Indicates the historical time step and conditions Next, the current time step The probability distribution of . Represents the query vector used for retrieval, usually a representation related to the question or query. Represents an attribute or feature associated with an image.

[0049] For the retrieved X t and X g , the input layer can be integrated, that is, the retrieved text in the question and image X t and subgraph X g Under the joint distribution of Forecast current time step words.

[0050] In summary, the indoor navigation method provided in this application applies the BLIP-2 strategy in the SCENEVERSE dataset to build a powerful retrieval-enhanced generation model (i.e., visual language model) for indoor scene understanding. The two-stage pre-training and advanced learning strategy of BLIP-2 significantly enhance the alignment and generation capabilities of the model, and performs well in 3D visual-language tasks, achieving state-of-the-art performance, which can provide a new solution for realizing zero-sample navigation of intelligent agents in unknown environments.

[0051] In another exemplary embodiment of the present application, an intelligent agent is provided, including: a processing unit, a positioning unit, a speech recognition unit, a visual unit and an instruction execution carrier.

[0052] The processing unit, positioning unit, speech recognition unit and vision unit are all mounted on the instruction execution carrier. The positioning unit, speech recognition unit and vision unit are all connected to the processing unit. The positioning unit is used to obtain positioning information. The speech recognition unit is used to obtain user instructions. The vision unit is used to obtain indoor environment images.

[0053] The processing unit is used to implement the indoor navigation method provided above and execute the user instruction based on the positioning information, the user instruction and the indoor environment image.

[0054] Based on the above intelligent agent structure, in the process of executing user instructions, the 2T open source indoor multimodal scene dataset can be used to build a RAG knowledge vector library (i.e., indoor scene multimodal knowledge base) for indoor scene multimodal image text retrieval. The trained weights are embedded in the instruction execution carrier (such as a mobile robot). This enables the retrieval of indoor scenes on mobile devices. And through real-time VLM questions, valid caption descriptions and real-time positioning information in indoor scenes are obtained. The image and text are aligned in real time, and the alignment results are saved in the indoor scene multimodal knowledge base, forming an autonomously updated real-time multimodal map connected to the instruction execution carrier, and completing the object index navigation strategy.

[0055] In another exemplary embodiment of the present application, the Yushu robot is used as an intelligent agent, and the constructed indoor scene multimodal knowledge base and multimodal large model (i.e., visual language model) are installed in its processing unit. Unknown indoor scene objects and location information are collected in real time, and stored and matched in the indoor scene multimodal knowledge base. In specific implementation, for example, if the user asks "find the computer", at this time, the intelligent agent can index the computer location in the indoor scene multimodal knowledge base, and navigate to the specified location to complete the user's instruction. Among them, the user instruction implementation process can be as follows: Figure 4 shown.

[0056] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store video tag processing data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an indoor navigation method is implemented.

[0057] Those skilled in the art will understand that Figure 5The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0058] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0059] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0060] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0061] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0062] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0063] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0064] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. An indoor navigation method, characterized in that: The indoor navigation method comprises: A visual language model is adopted and a retrieval-enhanced generation strategy is used to construct a multimodal knowledge base of indoor scenes; the multimodal knowledge base of indoor scenes includes multiple sets of position information; each set of position information is triple data; the triple data is used to represent the scene type, the number of indoor objects, and the positional relationship and hierarchical relationship between indoor objects; Obtain user instructions and indoor environment images; Retrieving location information from the indoor scene multimodal knowledge base based on the user instruction and the indoor environment image; Based on the location information, the navigation strategy algorithm is matched and the indoor scene multimodal knowledge base is updated and verified to ensure the accuracy of the matching information.

2. The indoor navigation method according to claim 1, characterized in that: The process of building a visual language model includes: The BLIP-2 strategy is adopted to process the data in the SCENEVERSE dataset, and in the indoor data alignment and generation stage, Q-Former is used to align the query embedding with the text embedding of the visual language model to obtain the visual language model; the SCENEVERSE dataset includes diverse indoor scenes and semantic annotations corresponding to the indoor scenes.

3. The indoor navigation method according to claim 1, characterized in that: A hybrid retrieval strategy is adopted to retrieve location information from the indoor scene multimodal knowledge base based on the user instruction and the indoor environment image; the hybrid retrieval strategy includes global vector retrieval and graph retrieval.

4. The indoor navigation method according to claim 3, characterized in that: The process of global vector retrieval includes: The inner product of a document vector or an image vector is used to represent the correlation; the document vector is generated by a document encoder based on a user instruction; and the image vector is generated by an image encoder based on the indoor environment image.

5. The indoor navigation method according to claim 3, characterized in that: The graph retrieval process includes: generating a text subgraph based on the user instruction; A subgraph structure related to the text subgraph is retrieved from the indoor scene multimodal knowledge base.

6. An intelligent agent, characterized in that: The intelligent agent includes: a processing unit, a positioning unit, a speech recognition unit, a visual unit and an instruction execution carrier; The processing unit, the positioning unit, the voice recognition unit and the visual unit are all mounted on the instruction execution carrier; the positioning unit, the voice recognition unit and the visual unit are all connected to the processing unit; the positioning unit is used to obtain positioning information; the voice recognition unit is used to obtain user instructions; the visual unit is used to obtain indoor environment images; The processing unit is used to implement the indoor navigation method according to any one of claims 1 to 5 based on the positioning information, the user instruction and the indoor environment image, and execute the user instruction.

7. The intelligent agent according to claim 6, characterized in that: The instruction execution carrier is a robot.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the indoor navigation method according to any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the indoor navigation method according to any one of claims 1 to 5 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the indoor navigation method according to any one of claims 1 to 5 is implemented.

Images