Embedded intelligent visual language large model knowledge base construction and application method, equipment, medium and product

Abstract

The invention discloses an embedded intelligent visual language large model knowledge base construction and application method and device, a medium and a product, and relates to the field of data intelligent processing. The method comprises the steps that an image encoder and a text encoder are adopted to encode image data and text data into vectors respectively, and image vectors and text vectors are obtained; fusing the image vector and the text vector, and generating a knowledge graph based on a fusion result to obtain an embedded intelligent visual language large model knowledge base; obtaining a user problem, and analyzing the user problem to obtain an analysis result; retrieving in a multi-modal knowledge base based on the analysis result to obtain a plurality of objects and a graph retrieval result; and combining the selected objects with the corresponding graph retrieval results to obtain triple information, converting the triple information into position text descriptions of the objects, inputting the position text descriptions into the multi-modal large language model, and outputting answers corresponding to the user questions. According to the method, the answer generation accuracy can be improved, and illusion phenomena are reduced.

Description

Technical Field

[0001] The present application relates to the field of intelligent data processing, and in particular to a method, device, medium and product for constructing and applying a large model knowledge base of an embedded embodied intelligent visual language. Background Art

[0002] Information Retrieval (IR) is a fundamental task in computer science and is crucial for obtaining relevant information that meets specific user needs from huge datasets. With the rise of generative AI, the importance of IR has become increasingly prominent. Generative AI models rely on IR systems to achieve accurate citations and reduce the risk of inaccurate information in generated content. Despite the indispensable role of IR in today's technological environment, existing literature, especially in the field of multimodal IR, remains narrow in scope and mainly focuses on homogeneous retrieval scenarios with predefined formats, usually limited to a single domain. For example, the MSCOCO dataset retrieves Flickr images by text descriptions, while the Electronic datainterchange system (EDIS) retrieves news headlines by news headlines. This homogeneous setting cannot meet the diverse information needs of users, which often span different domains and modalities.

[0003] Multimodal retrieval systems must evolve to accommodate these diverse needs. For example, some users may search for web images via text queries, while others may use a photo of a dress combined with text inputs such as "similar style" or "red" to find matching fashion products. Current multimodal retrieval systems fall short in addressing these diverse user needs and are typically limited to task-specific fine-tuning of pre-trained models such as CLIP (Contrastive Language-Image Pre-training). Therefore, there is an urgent need to conceive and develop more flexible general neural retrievers that can span different domains, modalities, and retrieval tasks to better meet the diverse needs of users.

[0004] In computer science, another important application is Retrieval-augmented Generation (RAG). Unlike generative models that create new content, RAG enhances the generation process by integrating retrieval to locate relevant existing objects in a large number of resources. This approach is particularly important in 3D vision-language (3D-VL) learning, especially for ground scene understanding in embodied intelligence scene tasks. However, the inherent complexity of 3D scenes exacerbates this gap. 3D scenes contain a variety of object instances with different properties, which are arranged in various ways and have complex relationships with each other. These unique 3D scene characteristics not only make it more difficult to accurately describe objects and their relationships, but also significantly increase the number of language descriptions required to fully describe the scene.

[0005] Visual-language learning has made significant progress, driven by Transformer-based pre-trained models and large-scale image-language datasets. These advances highlight the importance of data expansion, with data showing that a simplified contrastive pre-training pipeline can improve alignment and extend open vocabulary understanding. Despite these advances, applying these models to scene understanding remains a major challenge due to the lack of paired 3D scene language data. Current 3D scene localization models typically rely on task-specific knowledge and high-level optimization strategies, which limits their generality. To address this issue, researchers have begun to adapt pre-trained 2D visual-language models for 3D tasks, but these models still fall short in capturing the spatial relationships inherent in 3D data. This highlights the need for multi-level alignment between language and 3D scenes, especially for 3D-specific information.

[0006] In known research, the main components of hybrid retrieval, namely the Large Language Model (LLM) and the Vision-and-Language Pretrained Model (VLPM), both have hallucination problems. In particular, LLM tends to generate unexpected text hallucinations, while VLPM may generate non-existent objects in images, which is called object hallucination. Known MLLMs are prone to hallucinations of objects that appear frequently or co-occur in visual instruction datasets.

[0007] In traditional RAG scenarios, queries and paragraphs are usually encoded as a single vector. Most dense searchers fail to explicitly discover and utilize the key entities of the problem. This often leads to overly general knowledge and lacks specific focus. Based on this, there is an urgent need in the current technology to identify the key entities of the problem and specifically retrieve knowledge related to the problem and focused on these entities. General descriptions will only bring irrelevant noise to the model, thus affecting the generated results. It can be seen that this field urgently needs a good retrieval strategy to avoid possible hallucinations.

[0008] In summary, the object categories and attributes faced in the 3D visual language alignment process are diverse and complex. 3D visual language image-text matching is scarce, and the retrieval complexity and data duplication of 3D scenes themselves lack a unified method for aligning 3D multimodal knowledge bases. Summary of the invention

[0009] The purpose of this application is to provide a method, device, medium and product for constructing and applying an embedded embodied intelligent visual language large model knowledge base, which can improve the accuracy of generated answers and reduce the occurrence of hallucinations.

[0010] To achieve the above objectives, this application provides the following solutions: In a first aspect, the present application provides a method for constructing and applying an embedded embodied intelligent visual language large model knowledge base, including: Constructing a multimodal 3D knowledge retrieval framework; the multimodal 3D knowledge retrieval framework includes an image encoder and a text encoder; An image encoder and a text encoder are used to encode the image data and the text data into vectors respectively, thereby obtaining an image vector and a text vector; The image vector and the text vector are fused, and a knowledge graph is generated based on the fusion result, and the knowledge graph is stored to obtain a multimodal knowledge base; the multimodal knowledge base is used as an embedded embodied intelligent visual language large model knowledge base; Obtaining user questions, and parsing the user questions to obtain parsing results; Retrieving a plurality of objects and graph retrieval results in the multimodal knowledge base based on the parsing result; Merging the selected multiple objects with the corresponding graph search results to obtain triple information, and converting the triple information into a text description of the location of the object to obtain a search result; the triple information is used to describe the location relationship between the objects; The retrieval results are input into a multimodal large language model, and an answer corresponding to the user question is output.

[0011] Optionally, retrieving a plurality of objects and graph retrieval results in the multimodal knowledge base based on the parsing result specifically includes: Based on the analysis result, the corresponding object and graph retrieval result are retrieved in the multimodal knowledge base; A fine-grained contrastive learning method is used to determine the similarity between the parsed result and the corresponding object using the tag level between 3D vision, 2D vision and text tags; Arrange the objects in descending order according to their similarities to obtain an object sequence; The first k objects in the object sequence are retained, and the retained first k objects are the multiple objects obtained by retrieval.

[0012] Optionally, the similarity between the parsed result and the corresponding object includes one or more of 3D image to text similarity, text to text similarity, text to 3D image similarity, 2D image to text similarity, and text to 2D image similarity.

[0013] Optionally, a similarity matrix is ​​used to determine the similarity between the parsing result and the corresponding object; the similarity matrix is ​​expressed as: ; In the formula, Representing objects and objects The similarity between represents the encoding vector of the image, The encoding vector representing the text, and Embedding vectors representing different scenes or objects, Represents the embedding vector The transpose of , D represents the scene entity, represents the text feature vector, represents the number of samples, The number of tokens representing the text; They all represent embedding vectors.

[0014] Optionally, parsing the user question to obtain a parsing result includes: Parse the user question using a constructed parser to obtain a parse tree; Noun phrases are extracted from leaf nodes of the parse tree to obtain the parse result.

[0015] Optionally, in the process of retrieving multiple objects and graph retrieval results in the multimodal knowledge base based on the parsing result, a retrieval strategy adopted is a hybrid retrieval.

[0016] Optionally, the hybrid search includes global vector search and graph search; The global vector retrieval is measured by the inner product of the global vector; The graph retrieval is achieved by retrieving the subgraphs of the knowledge graph.

[0017] In a second aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method for constructing and applying a large model knowledge base of an embedded embodied intelligent visual language as described in any one of the above.

[0018] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for constructing and applying a large model knowledge base of an embedded embodied intelligent visual language as described in any one of the above.

[0019] In a fourth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the method for constructing and applying a large model knowledge base of an embedded embodied intelligent visual language as described in any one of the above.

[0020] According to the specific embodiments provided in this application, this application has the following technical effects: The present application provides a method, device, equipment, medium and product for constructing and applying an embedded embodied intelligent visual language large model knowledge base. By constructing a multimodal 3D knowledge retrieval framework, the relationship nodes within and between classes are constructed to meet the challenges of fine-grained mixed retrieval. The image encoder and text encoder are used to encode the relevant data into vectors, and then these vectors are fused to construct a multimodal knowledge base to derive the relationship between objects. After the obtained triples are converted into a text description of the object's position, they are input into a multimodal large language model, and the answers corresponding to the user's questions are output, which can not only significantly improve the accuracy of the generated answers, but also reduce the occurrence of hallucinations. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 A flowchart of a method for constructing and applying a large model knowledge base of an embedded embodied intelligent visual language provided in one embodiment of the present application; Figure 2 A schematic diagram of an example implementation process of a method for constructing and applying an embedded embodied intelligent visual language large model knowledge base provided in an embodiment of the present application; Figure 3 A schematic diagram of a 3D scene and text fine-grained interaction process provided by another embodiment of the present application; Figure 4 A schematic diagram of a similarity calculation process provided by another embodiment of the present application; Figure 5 A schematic diagram of triple information provided by another embodiment of the present application; Figure 6A schematic diagram of a document entity extraction process provided by another embodiment of the present application; Figure 7 A schematic diagram of a hybrid search process provided by another embodiment of the present application; Figure 8 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0024] Creating aligned 3D-language datasets is inherently challenging. Existing works such as ShapeNet leverage online resources to acquire 3D assets, leading to a boom in high-quality 3D object datasets. Recent developments include the emergence of a large-scale dataset, Objaverse, which facilitates 3D-language alignment by integrating object-level captions, demonstrating enhanced object understanding capabilities in classification, generation, and captioning tasks.

[0025] Developing 3D datasets for scene understanding is more complex due to the high requirements for scene acquisition and annotation. Existing datasets, such as RGB-D and scanned indoor scene datasets, were initially used to benchmark tasks such as 3D object detection and segmentation. These semantically annotated scenes are now applied to fine-grained tasks such as object reference, subtitle generation, visual-language navigation, and reasoning. Recent work has explored the use of 3D scene graphs (3DSGs) to represent scenes with a hierarchical structure, which is particularly beneficial for planning and subtitle generation tasks as they are compatible with large language models.

[0026] Based on the above description, in order to provide a multimodal 3D scene RAG retrieval framework, this application needs to solve the fine-grained matching problem of 3D images in the scene, expand the pre-trained extensive 3D-QV (Quality Assurance) and 2D-QA indoor scene knowledge base, and distinguish the intra-class similarity of the same objects during the retrieval process. This adaptability requires not only improving the retrieval process, but also optimizing the affiliation between 3D knowledge base vectors, while reducing costs by supporting context and eliminating certain generation steps.

[0027] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0028] The embedded embodied intelligent visual language large model knowledge base construction and application method provided in the embodiment of the present application can be implanted in the processing unit of any terminal for embedded intelligent application. Figure 2 As shown, the provided embedded embodied intelligent visual language large model knowledge base construction and application method includes the following steps 100 to 106. Among them: Step 100: construct a multimodal 3D knowledge retrieval framework. The multimodal 3D knowledge retrieval framework includes an image encoder and a text encoder.

[0029] Step 101: Use an image encoder and a text encoder to encode image data and text data into vectors respectively, thereby obtaining an image vector and a text vector.

[0030] Step 102: Fuse the image vector and the text vector, generate a knowledge graph based on the fusion result, store the knowledge graph, and obtain a multimodal knowledge base. The multimodal knowledge base is used as the knowledge base of the embedded embodied intelligent visual language large model.

[0031] Step 103: Obtain the user's question, and parse the user's question to obtain a parsing result.

[0032] Step 104: Based on the parsing result, multiple objects and graph retrieval results are retrieved in the multimodal knowledge base.

[0033] Step 105: Merge the selected multiple objects with the corresponding graph search results to obtain triple information, and convert the triple information into a text description of the location of the object to obtain a search result. The triple information is used to describe the location relationship between objects.

[0034] Step 106: Input the search results into the multimodal large language model and output the answer corresponding to the user's question.

[0035] Implementing the above steps 100 to 106 can not only significantly improve the accuracy of generating answers, but also reduce the occurrence of hallucinations.

[0036] In another exemplary embodiment of the present application, in order to improve the ability to understand the similarity of objects in 3D scenes and optimize the retrieval efficiency of related objects and their relationships. By constructing a multimodal 3D knowledge retrieval framework (UR3D), the construction of intra-class and inter-class relationship nodes is realized to meet the challenge of fine-grained hybrid retrieval. Based on this, the present application first uses the image encoder and text encoder in the aligned UR3D to encode the data into vectors, and then fuses these vectors to build a multimodal knowledge base. In this process, it is necessary to pay special attention to the objects in SenseVerse to be annotated through the visual language model to obtain annotation information, so as to deduce the relationship between objects. By merging the vector of the object with the entity word, the vertex of the graph is given features. In the retrieval process, a hybrid retrieval method is adopted. First, the multimodal knowledge base is searched, and the k objects with the highest cosine similarity are selected and merged with the first k graph retrieval results. For the retrieval results of the graph, the obtained triples are converted into text descriptions of the location of the object. For example, (sofa, next to, chair) can be converted into "the sofa is next to the chair". Finally, the retrieval results are input into the multimodal large model. The results show that the method provided by this application not only significantly improves the accuracy of generating answers, but also reduces the occurrence of hallucinations. Taking the user question "Is the sofa next to the TV?" as an example, the overall implementation process of the method provided by this application is as follows: Figure 2 As shown, the final answer corresponding to the user's question is "No, the sofa and the TV are opposite to each other in the room."

[0037] In another exemplary embodiment of the present application, the implementation process of step 104 can be described as follows: (1) Based on the parsing results, the corresponding objects and graph retrieval results are retrieved in the multimodal knowledge base.

[0038] (2) Using a fine-grained contrastive learning method, using the tag level between 3D vision, 2D vision and text tags, the similarity between the parsed result and the corresponding object is determined. The similarity between the parsed result and the corresponding object includes one or more of the similarity between 3D image and text, the similarity between text and text, the similarity between text and 3D image, the similarity between 2D image and text, and the similarity between text and 2D image.

[0039] (3) Arrange the objects in descending order according to their similarity to obtain an object sequence.

[0040] (4) Retain the first k objects in the object sequence, where the first k objects retained are the multiple objects retrieved.

[0041] In the actual application process, the present application realizes similarity determination by fine-grained interaction between 3D scene and text, wherein the entire fine-grained interaction process between 3D scene and text is as follows: Figure 3 As shown in Figure 2. On top of the 3D image, text encoder, and query encoder, the representations of textual and visual tags are linearly projected into a multimodal joint space. A novel fine-grained contrastive learning method is proposed that exploits the token-wise maximum similarity between 3D and 2D visual and textual tags.

[0042] Under the general formulation of cross-modal contrastive learning, we wish to learn an encoder for 2D image data I , learn the encoder of text data T , a queryer that learns to query data , learn the encoder of 3D image data D . So that for a given 3D image belong D , 2D image belong and text belong , query belong Under the cosine distance metric, if their information is related, the encoding representation and Should be close, otherwise should be far away. In each training batch, multiple 2D image-text-query-3D image pairs are sampled, denoted as ( , , , ). For each image-text pair, the text is its positive example, and other texts will be used as negative examples in the batch. ,image is its positive example, and other images will be used as negative examples in the batch. For a text-image contrast loss, it can be expressed as: (1) in, represents the calculation of the inner product, Represents the text-image contrast loss. In cross-modal contrastive learning, The set of related negative examples. Represents the index value selected from this batch. z represents the normalization constant, which is used to ensure that the entire similarity calculation is probabilistic. Represents a set of constants.

[0043] Similarly, , Expand to ( , , , ) can be expressed as: (2) In the formula, Representation Object Related negative examples, images and text The expansion of and express. Represents the expanded image , Represents the expanded text .

[0044] The above interaction methods ignore fine-grained alignment. To this end, this application proposes a fine-grained multimodal interaction method to model 3D images, query texts, and documents at the tag level.

[0045] PointBERT (Yu et al. 2022) was the first to introduce the standard Transformer to point cloud tasks. The encoder for 3D indoor scenes uses the design and pre-training model of EPCL (Emulated Pre-training for Contrastive Learning). Similar to ViT, the point cloud P∈R A×3 is divided into a series of point cloud segments P p ∈R M×(3K) , where A and M represent the number of points and the number of segments, respectively, and K represents the number of points in each segment. These segments are fed into a tokenizer to extract the token embeddings of the points. Subsequently, the token embeddings, position embeddings, and class tokens of these points are fed into a standard Transformer for feature extraction. Given a 3D scene point cloud S, it is segmented into a set of M segments S = {s1, s2, s3, s4…}. For each s ∈S, there are corresponding 3D position codes xyz. Similarly, for text or query, there is a word set D={d1,d2,d3,d4…}. Among them, the design and pre-training method of EPCL are mainly used for the encoder part of the point cloud task. Its role is to apply contrastive learning to 3D point cloud tasks by simulating pre-training, thereby improving the generalization ability and robustness of the model.

[0046] Based on the above description, if Figure 4As shown in Figure 1, the token-wise maximum similarity between text and image is calculated. The similarity matrix is ​​obtained by calculating the average token-wise maximum similarity of non-filled tags of a single image (or text).

[0047] Specifically, remember i and t j are the number of tags of the i-th image and the j-th text respectively, and the corresponding encoding features are and r is the number of samples, d L is the feature dimension. And d i ,t j <d L . Alignment mapper for 3DEncoder.

[0048] (3) (4) Then, the average token-level maximum similarity of all non-filled tokens (masks) in the image (or text) is used as d i (or t j ) to all text tags t(d). Take the average of all text tag-level similarities and scale the L2 norm to 1, so that the similarity matrix from 3D image (text) to text (image) is obtained: , (5) In the formula, Indicates the number of markers The encoded vector after image alignment mapping, Indicates the number of markers The text alignment mapped vector, Indicates transpose.

[0049] The same is true for other modalities. Let image be i and query be q. We get the encoding vectors I and Q as: (6) (7) In this way, the similarity matrix between any modalities can be calculated: (8) In the formula, Representing objects a and objects b The similarity between represents the encoding vector of the image, The encoding vector representing the text, and Embedding vectors representing different scenes or objects. Represents the embedding vector By transposing the vector, the inner product operation can be performed when calculating the similarity, thereby obtaining the relationship between different objects or scenes. D Represents scene entities, which refer to actual physical objects or environments and are used to describe the physical background involved. Represents a text feature vector, which is extracted from the text through natural language processing technology (such as language model) and reflects the main semantics and structure of the text. represents the number of samples, The number of tokens representing the text.

[0050] After obtaining the similarity matrix s, apply the above formula (2) and finally get the following loss function: (9) in, They serve as positive examples to each other.

[0051] In another exemplary embodiment of the present application, the implementation process of parsing the user question to obtain the parsing result in step 103 can be described as follows: (1) Use the constructed parser to parse the user question and obtain the parse tree.

[0052] (2) Extract noun phrases from the leaf nodes of the parse tree and obtain the parse results.

[0053] In practical applications, the present application can be implemented through entity extraction. In the senseverse dataset, for orientation retrieval, a graph structure is used to model relational queries. Compared with the orientation description at the sentence level, there is a lack of a unified reference system and it is impossible to obtain comprehensive information. The topological structure can better model the relationship between objects. Based on this, a triple is used to describe the relationship between (A, Edge, B) objects. And it is used as a method of meta-information of the object feature vector. Based on phrase-based dense paragraph retrieval, entities are collected from multiple aspects of questions and images, and an independent vocabulary model can be used. The vocabulary model is trained to imitate the performance of a sparse retriever that performs better in phrase matching. The DensePhrase model is also proposed in the known technology. The DensePhrase model extracts every possible phrase feature in the paragraph and uses only the most relevant phrases to measure the similarity between the query and the paragraph. In contrast, a multimodal entity is constructed, including 2D pictures, 3D images, text entities, etc., to construct a rich unified intermediate representation.

[0054] In short, the method provided in this application obtains the multimodal features of entities, which not only expands the scope of retrieval, but also enhances the ability to capture relevant knowledge.

[0055] Based on the above description, the implementation process of the method for obtaining multimodal features of an entity provided by the present application may be: First, the noun phrases in the question usually reveal key entities. The concepts present in the text may be implied but not explicitly stated in the text itself (such as implicit relationships). Therefore, the question can be parsed by constructing a parser and extracting noun phrases at the leaf nodes of the parse tree. In the senseverse dataset, 3D subgraphs are provided to construct the positional relationship between indoor objects. Instances of graph nodes and edges are identified and extracted from images. In this way, knowledge triples (subject, predicate, object) in the traditional sense can be constructed for reasoning. For example, triple information such as Figure 5 shown. Figure 5 In the example, 'scene_type' means scene type, 'Bedroom' means bedroom, and 'object_count' means number of objects. 'nightstand2' means 2 nightstands. 'relation' means relation. {nightstand', 'on', 'floor', 'backback', 'near', bed, ...} means the nightstand is on the floor near the back of the bed.

[0056] The relations in the dataset are used to construct triple information descriptions of different objects. Then, the 3D image encoder is used to extract 3D image features as attributes of nouns.

[0057] Based on the above description, if Figure 6 As shown, the entity extraction of the document is performed. This extraction method is mainly through fine-grained retrieval. Among them, after calculating the maximum token-wise similarity between the image features and the keyword features, the feature vector with the largest similarity is selected as the keyword feature.

[0058] In a multimodal scenario, in order to extract the feature vector T for keyword t, it is necessary to calculate its similarity with the scene entity feature d and select the scene entity with the greatest similarity as the keyword's feature vector. The specific steps are as follows: First, the keyword t is encoded by the text encoder. i Encode and generate its feature vector T i Assuming there are n keywords and the dimension of the feature vector of each keyword is R, the dimension of T is n×R. Use the 3D scene encoder to encode the entity d in the scene and generate its feature vector , the corresponding feature vector of the generated entity d ,have: (10) (11) In the formula, represents a text encoder, Represents a 3D image encoder.

[0059] In order to find the scene entity that best matches each keyword, it is necessary to calculate the feature vector t of each keyword. i With all scene entity feature vectors This can be achieved by calculating the dot product. Specifically, for each keyword And each scene entity , similarity The calculation formula is: (12) By calculating each keyword With all scene entities The similarity of the scene entity with the largest similarity is selected as the final feature vector of the keyword. In short, the probability distribution is calculated , based on the similarity measure s, in The probability of ,have: (13) The feature vector with the highest probability is selected as the 3D feature vector corresponding to the entity as its final feature representation. The entity name and the corresponding feature are merged to obtain the final entity. .

[0060] (14) In the formula, Indicates feature merging.

[0061] The created index can be modeled as an undirected graph, where entity nodes are connected by relationship edges, and the approximate locations are reflected by distance information.

[0062] In another exemplary embodiment of the present application, in the process of implementing step 104, the retrieval strategy adopted is a hybrid retrieval. The hybrid retrieval includes a global vector retrieval and a graph retrieval. The global vector retrieval is measured by the inner product of the global vector. The graph retrieval is implemented by retrieving a subgraph of the knowledge graph.

[0063] In practical applications, the hybrid retrieval strategy aims to efficiently retrieve knowledge related to the query, especially focusing on the entity information in the query. To this end, this application proposes a dual similarity measurement method, which consists of two parts: full set vector retrieval and graph retrieval.

[0064] Among them, global vector retrieval is mainly measured by the inner product of the global vector. In this process, a multimodal (image and text) retrieval method is mixed to enhance the retrieval effect. Specifically, the relevance between the query and the document or image is obtained by fusing the vector representations of different modalities and calculating their inner product. This process not only considers the information of a single modality, but also integrates multimodal information, thereby improving the comprehensiveness and accuracy of the retrieval. There are: (15) In the formula, Indicates the relevance between the query and the document or image.

[0065] In graph retrieval, the text graph is a graph consisting of nodes and edges with text attributes, which can be formally represented as , where V and E represent the node set and edge set respectively, and They represent the natural language attributes and image attributes of the corresponding nodes in the graph, Represents the natural language attributes of the edge.

[0066] Based on the above description, the task of this embodiment is to Retrieve subgraphs from .in and Represent the power sets of V and E respectively. For an indoor scene knowledge base, the image attributes Distinguish different scenes.

[0067] Graph retrieval enhancement generation means that given a specific question q on a graph G, there must be a relevant subgraph structure in G that can be used to help generate more reliable and explanatory answers. Retrieve the top k most relevant subgraph structures.

[0068] The 3D feature vector of the corresponding noun and the vector of the subgraph entity are used to calculate the similarity, and the correct matching space is selected. Then, the entity features identified by the noun extractor and the subgraph entity are used to calculate the discrete similarity, and the sum of these two similarities is used to measure the entity relevance. There are: (16) (17) In the formula, It is expressed by maximizing the image vector I and the noun entity vector T i The inner product of is used to select the best matching index, that is, to find the most relevant noun or entity. The embedded feature vector representing the noun or entity corresponding to the image vector I is usually obtained through some vectorization method (such as word embedding or entity embedding). Represents the relevance score function, calculating the query vector With image embedding vector I k and the embedding vector of the entity E j The similarity between them is combined to get the final matching score. Indicates k The image embedding vectors are selected to be relevant to the query, usually obtained through multimodal retrieval methods. E j Indicates j The embedding vector of an entity, which can be an object represented by text or other modalities. Represents the query vector The embedding vector of the target entity The similarity score between them is usually used to measure the semantic similarity between the query and the target entity. The embedding vector representing the target entity is usually associated with a specific query and represents the target object to be retrieved. Represents the image embedding vector I k Embedding vector with the underlying entity The similarity score between them measures the similarity between the image and the underlying entity in a specific feature space. The embedding vector representing the underlying entity is used to measure the information of other reference entities.

[0069] In order to generate the search results, the retrieved information is filtered using re-ranking, and the two search results are re-ranked by sorting the similarity of the two searches.

[0070] Since hybrid retrieval mixes graph retrieval and text retrieval, the text information contained in it may also be repeated, which affects the generator performance. To avoid redundancy, when re-ranking, if the candidate information is too similar to the already retrieved documents (for example, the relevance score > 0.9), the retrieval result will be skipped.

[0071] Multimodal retrieval is becoming increasingly important in tasks involving text-image matching, especially in navigation and embodied robotics. In these scenarios, an effective retrieval system can significantly improve the performance of navigation tasks, providing robots with the information they need to accurately navigate complex environments. Therefore, advancing the development of multimodal retrieval systems is crucial to fully realize the potential of IR and RAG techniques in a variety of applications, from web search to complex robotic navigation systems.

[0072] The main contributions are: 1) Multimodal 3D vision-language alignment: We propose a fine-grained alignment method for the retrieval-augmented generation (RAG) strategy, which enhances the metrics of zero-shot 3D vision-language alignment. This method exploits the complexity of 3D environments and addresses the challenges posed by diverse object configurations and complex relationships in scenes.

[0073] 2) Graph structure and text hybrid retrieval: The retrieval method is expanded, and the topological structure of objects is established based on the characteristics of the indoor data set. Graph retrieval can avoid redundant retrieval information. Experimental examples show that the method provided in this application effectively reduces the probability of hallucinations in the model.

[0074] 3) Multimodal 3D knowledge retrieval framework (UR3D) and retrieval evaluation index (M3DKR): A comprehensive multimodal 3D knowledge retrieval and alignment framework UR3D is proposed. This framework constructs a joint encoder of 3D image-2D image-text-query, which solves the challenge of fine-grained hybrid retrieval. It enhances the understanding of object similarity in 3D scenes and optimizes the retrieval of related objects and their relationships. This framework also supports dense description and visual positioning in indoor 3D environments, and provides a powerful solution for complex multimodal retrieval tasks, a new retrieval evaluation index M3DKR to evaluate multimodal models. Extensive ablation experiments point out the direction of future enhancements to multimodal retrieval systems.

[0075] Furthermore, based on the search strategy given above, the generation process of the answer to this application is as follows: For a specific question q and image vector I, retrieve the text and subgraph topology from the knowledge base, convert the subgraph topology into text, and input it into the visual language model, as follows: (18) In the formula, Represents text, represents a subgraph, Represents a subset of a specific type in the topology. In the given problem In the case of text The probability distribution of . Represents a given problem, which is the representation after embedding (encoding) the problem q through some encoder, usually used for model input. In the given problem In the case of The probability distribution of . Represents the time step. Indicates that the condition Next, target The probability distribution of . Indicates the historical time step and conditions Next, the current time step The probability distribution of . Represents the query vector used for retrieval, usually a representation related to the question or query. Represents an attribute or feature associated with an image.

[0076] For the retrieved X t and X g , the input layer can be integrated, that is, the retrieved text in the question and image X t and subgraph X g Under the joint distribution of Predict the current moment words.

[0077] In another exemplary embodiment of the present application, the M3DKR framework is evaluated by experimental verification. In this embodiment, an indoor multi-task multimodal knowledge retrieval (M2KR) benchmark suite is proposed. Nine senseverse datasets are converted into a multimodal retrieval format.

[0078] The M3DKR benchmark is used in the task to evaluate the UR3D framework. The M3DKR benchmark contains three types of tasks: 1).3D image + question - (DQ) > 3D image + text (DT).

[0079] This system evaluates the ability to find the corresponding scene image and question based on the current 3D image and question.

[0080] 2).3D image (D) > text (T).

[0081] Evaluate the ability to find the corresponding text based on the current image.

[0082] 3). Text (T) -> 3D image (D).

[0083] Assess the ability to find corresponding images based on text.

[0084] 4).2D image + question (IQ) -> 3D image + text (DT) retrieval.

[0085] By aligning 2D images and 3D images, a unified architecture can be used for 3D scene understanding, and 3D images can be converted into information of other modalities (such as text) without the need for extremely costly LLM alignment fine-tuning.

[0086] Furthermore, ablation experiments were performed.

[0087] 1) Dataset.

[0088] Use the senseverse dataset, which is a million-scale 3D-VL dataset for scene-based understanding. The SceneVerse data contains 68K 3D scenes and 2.5 million scene-language pairs, which are obtained through a combination of manual annotation and automatic generation methods. The SceneVerse dataset uses the power of LLM to generate comprehensive and high-quality language descriptions, covering object-level and scene-level descriptions. In addition, the SceneVerse dataset also includes the most extensive human-annotated object references to date, providing new training resources and benchmarks for this field. The SceneVerse dataset includes multiple datasets such as ARKitScenes, MultiScan, and 3RScan. The ScanNet, MultiScan, ARKitScenes, and 3RScan datasets were selected as experimental datasets. Among them, ScanNet contains 2D images, and the other three only have 3D images. LLama3-7b was used to generate query documents for the sentence description of each dataset.

[0089] In the experiment, a two-stage training method was adopted. The first stage: 3D image encoder pre-training. In the first stage, we focus on training the image 3D encoder PECL for classification tasks. The model is trained to correctly classify this data, aiming to improve the model's representation ability in 3D space. The second stage: multi-modal alignment training. In the second stage, we focus on multi-modal alignment training and align multiple modal features. This two-stage training strategy aims to first establish a strong 3D feature representation, and then use these representations for cross-modal alignment and optimization, thereby improving the performance of the entire system.

[0090] The 3D scene graph in the SceneVerse dataset is defined as a collection of tuples consisting of nodes and edges. Nodes represent unique 3D object instances, parameterized by their center points and bounding box sizes, while edges represent the spatial relationships between these nodes. These spatial relationships are divided into vertical proximity (support, interior, embedded, above, below), horizontal proximity (in front, beside, behind, left, right), and multi-object relationships (aligned, between). To construct the scene graph, the instantiated nodes are first annotated with instances in the point cloud and the object categories are assigned with corresponding semantic labels. Then, the spatial relationships of each object node are calculated, and an automatic verification process is performed to correct erroneous relationships. The detailed scene graph construction and relationship determination process are shown in the figure. Figure 7 shown.

[0091] Pre-trained 3D scene encoder EPCL based on CLIP's ViT-L / 14. PreFLMR pre-trained question encoder, image encoder and document encoder. The experiment uses 8 cards 3090, uses Adam optimizer, and sets the learning rate to 0.001. The training is performed for 30 rounds. The training set and test set use the default settings of the senseverse dataset. In the multimodal alignment retrieval ablation experiment shown in Table 1, the retrieval includes four modalities: query (Q), text (T), 3D image (D), and 2D image (I). Among them, only the ScanNet dataset has a 2D modality.

[0092] Table 1 Multimodal alignment retrieval ablation experiment results

[0093] In Table 1, Q+D->T+D represents the task from question and 3D image to text and 3D image. Q+I->T+D represents the task from question and 2D image to text and 3D image.

[0094] Datasets lists the different datasets used in the experiments.

[0095] Global and Fine-grained represent different levels of evaluation criteria: Rank 1 represents the performance in the 1st ranked retrieval result.

[0096] mAP (mean average precision) indicates the overall performance of the model in the retrieval task.

[0097] 1. Fine-grained contrastive learning vs single-grained contrastive learning.

[0098] Table 1 shows the advantages of fine-grained contrastive learning over single-grained contrastive learning. Experiments on four different datasets have achieved significant improvements.

[0099] For the 3D modal information in the query, data enhancement, rotation, and translation are used to distinguish the original data. The experiment has a certain degree of randomness, and the average value is taken from the three experiments.

[0100] It can be seen that the average mAP using fine-grained retrieval is 4.92 points higher than that without fine-grained retrieval. This shows that fine-grained visual-language alignment is crucial to achieve good performance.

[0101] 2. UR3D is an excellent zero-shot retriever.

[0102] By training and testing on different datasets, the method provided in this application demonstrates its strong cross-domain generalization ability, especially in fine-grained retrieval tasks.

[0103] For example, when the application trained the model on the ARKitScenes dataset and tested it on the MultiScan dataset, it achieved global retrieval results of 84.78% Rank 1, 98.91% Rank 5, and 90.40% mAP. In fine-grained retrieval, it achieved high scores of 97.82% Rank 1, 98.91% Rank 5, and 98.19% mAP, which is significantly better than the known methods. Similarly, when the model was trained on the MultiScan dataset and tested on the ARKitScenes dataset, the results were even better, with 95.65% and 100% Rank 1 and 5 in fine-grained retrieval, respectively, and an mAP of 97.82%, further verifying the robustness of the method.

[0104] Even in more challenging scenarios, such as when trained on the ScanNet dataset and tested on the 3RScan dataset, the method provided in this application still shows obvious advantages. Although the Rank 1 of global retrieval is only 65.78%, the mAP of fine-grained retrieval reaches 85.67%, surpassing the known global retrieval methods, showing the adaptability and superiority of the method in dealing with new tasks and datasets.

[0105] These data show that the fine-grained retrieval method provided by this application performs extremely well in zero-shot retrieval, and can surpass the global retrieval method on both known and unknown datasets. The zero-shot retrieval ablation experiment results are shown in Table 2.

[0106] Table 2 Zero-shot retrieval ablation experiment results

[0107] 3. Entity retrieval improves content relevance.

[0108] For object location questions, this application sets up a series of template questions as queries. For example, the question "Where would [obj] be located?, is the [obj1]on the [obj2]?" uses the positional relationship in the senseverse to construct a subgraph and fills in the template with the nouns in it. In hybrid retrieval, the results retrieved in the graph are converted into text. For example, [obj1]is next to [obj2] is used as text as the input of the multimodal language model.

[0109] As shown in Table 3, the results of adding graph search and not adding graph search are compared. Graph search greatly improves the accuracy and richness of the search content. Among them, the top 1 retrieved from the generated result comparison experiment Graph Text content and Textcontent are respectively. W / O graph means no graph search results, W graph means search results with graph, and Direct means direct input without search.

[0110] Table 3 Comparison of search results with and without graph search

[0111] 4. Hybrid retrieval reduces multimodal model hallucinations.

[0112] In the answer generation process, this application uses the QwenVL-7b model and generates it on a single NVIDIA A100 GPU. The generated results with graph retrieval have three positions: 'located next to', 'positioned nextto', 'in front of', while the results without graph retrieval have only two position words. Without retrieval, two examples of object hallucination are shown, where hallucinated objects such as "black trash" are generated by MLLM. It can be found that even if relevant instructions are used to require relative object descriptions, the model still gives coordinates and irrelevant objects.

[0113] This shows that the method provided by this application effectively solves the problem of hallucinations that may occur during the generation process of multimodal language models. By introducing image retrieval, the model can not only better understand and integrate visual information, but also significantly improve the accuracy of generated answers, thereby reducing the hallucinations produced by the model's insufficient processing of visual information and question understanding. This improvement demonstrates the effectiveness and potential of the method provided by this application in improving the generation capabilities of multimodal language models.

[0114] In summary, this application proposes a multimodal 3D scene retrieval-enhanced generation (UR3D) framework, and experimentally verifies the effectiveness of the UR3D framework in complex 3D visual language alignment and zero-shot retrieval tasks. Through fine-grained interaction and hybrid retrieval enhancement strategies, the accuracy and efficiency of zero-shot 3D scene retrieval are significantly improved. Experimental results on the senseverse dataset show that the method provided by this application improves the retrieval accuracy by an average of 12.3% and mAP by 13.4% in the visual language multimodal retrieval task. In addition, the implementation shows that the UR3D framework demonstrates excellent generalization capabilities.

[0115] This application also proposes a multimodal evaluation framework for indoor scenes (M3DKR), which includes most common retrieval and evaluation tasks. This framework is used to verify the effectiveness of multimodal alignment results.

[0116] In summary, the experiments in this application demonstrate the superiority of fine-grained contrastive learning over single-grained methods, especially in improving accuracy and relevance on multimodal retrieval tasks. The UR3D model has been shown to be an excellent zero-shot retriever that can effectively generalize to unseen tasks with outstanding performance. Moreover, the integration of entity-based retrieval significantly improves content relevance, ensuring that the retrieved information is closely aligned with the user query. Finally, the implementation of a hybrid retrieval approach helps reduce hallucinations in the multimodal model, contributing to more reliable and accurate retrieval results.

[0117] Furthermore, in response to the problems faced in the 3D visual language alignment process, such as diverse object category attributes, complex relationships, scarce 3D visual language image-text matching, 3D scene retrieval complexity, data repetitiveness, and lack of a unified alignment method for 3D multimodal knowledge bases, this application designs the UR3D pre-trained retrieval framework, a transferable 3D scene zero-sample RAG retrieval question-answering framework, which solves the problems of zero-sample 3D visual language alignment and 2D visual language alignment, image-text retrieval matching, and the ability to generate fine-grained descriptions through fine-grained interaction, 2D and 3D feature extraction, and hybrid retrieval enhancement strategies. To this end, this application also proposes a multimodal evaluation framework M3DKR for pre-training and fine-tuning on the senseverse dataset. M3DKR includes a variety of visual and language retrieval tasks. Through a series of cross-domain experiments with benchmark tasks, it is proved that the accuracy of the fine-grained retrieval provided by this application is improved by an average of 12.3%, and the mAP is improved by 4.92%. Experiments show that the method of this application has better retrieval effect on various different data sets in 3D scenes, and the retrieval index of the model is higher in zero-sample retrieval in unknown environments, providing a multimodal retrieval generalization solution for complex and changeable 3D embodied intelligent environments.

[0118] In addition, based on this application, we can further focus on exploring the application of indoor RAG (Retrieval-Augmented Generation) in embodied intelligence to improve the ability of artificial intelligence systems in navigating and understanding complex environments, especially in the context of indoor navigation tasks. RAG plays a vital role in promoting intelligent navigation and has the potential to significantly improve the performance and accuracy of AI-driven navigation systems in complex indoor environments.

[0119] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 8As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store a visual language large model knowledge base. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for constructing and applying an embedded embodied intelligent visual language large model knowledge base is implemented.

[0120] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0121] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0122] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0123] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0124] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0125] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0126] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0127] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for constructing and applying a large model knowledge base of embedded embodied intelligent visual language, characterized in that: The method for constructing and applying the embedded embodied intelligent visual language large model knowledge base includes: Constructing a multimodal 3D knowledge retrieval framework; the multimodal 3D knowledge retrieval framework includes an image encoder and a text encoder; An image encoder and a text encoder are used to encode the image data and the text data into vectors respectively, thereby obtaining an image vector and a text vector; The image vector and the text vector are fused, and a knowledge graph is generated based on the fusion result, and the knowledge graph is stored to obtain a multimodal knowledge base; the multimodal knowledge base is used as an embedded embodied intelligent visual language large model knowledge base; Obtaining user questions, and parsing the user questions to obtain parsing results; Retrieving a plurality of objects and graph retrieval results in the multimodal knowledge base based on the parsing result; Merging the selected multiple objects with the corresponding graph search results to obtain triple information, and converting the triple information into a text description of the location of the object to obtain a search result; the triple information is used to describe the location relationship between the objects; The retrieval results are input into a multimodal large language model, and an answer corresponding to the user question is output.

2. The method for constructing and applying a large model knowledge base of embedded embodied intelligent visual language according to claim 1 is characterized in that: Based on the parsing result, a plurality of objects and graph retrieval results are retrieved in the multimodal knowledge base, specifically including: Based on the analysis result, the corresponding object and graph retrieval result are retrieved in the multimodal knowledge base; A fine-grained contrastive learning method is used to determine the similarity between the parsed result and the corresponding object using the tag level between 3D vision, 2D vision and text tags; Arrange the objects in descending order according to their similarities to obtain an object sequence; The first k objects in the object sequence are retained, and the retained first k objects are the multiple objects obtained by retrieval.

3. The method for constructing and applying a large model knowledge base of embedded embodied intelligent visual language according to claim 2, characterized in that: The similarity between the parsed result and the corresponding object includes one or more of 3D image to text similarity, text to text similarity, text to 3D image similarity, 2D image to text similarity, and text to 2D image similarity.

4. The method for constructing and applying a large model knowledge base of embedded embodied intelligent visual language according to claim 2, characterized in that: A similarity matrix is ​​used to determine the similarity between the parsing result and the corresponding object; the similarity matrix is ​​expressed as: ； In the formula, Representing objects and objects The similarity between represents the encoding vector of the image, The encoding vector representing the text, and Embedding vectors representing different scenes or objects, Represents the embedding vector The transpose of , D represents the scene entity, represents the text feature vector, represents the number of samples, The number of tokens representing the text; They all represent embedding vectors.

5. The method for constructing and applying a large model knowledge base of embedded embodied intelligent visual language according to claim 1, characterized in that: Parsing the user question to obtain a parsing result includes: Parse the user question using a constructed parser to obtain a parse tree; Noun phrases are extracted from leaf nodes of the parse tree to obtain the parse result.

6. The method for constructing and applying a large model knowledge base of embedded embodied intelligent visual language according to claim 1, characterized in that: In the process of retrieving multiple objects and graph retrieval results in the multimodal knowledge base based on the parsing result, the retrieval strategy adopted is hybrid retrieval.

7. The method for constructing and applying a large model knowledge base of embedded embodied intelligent visual language according to claim 6, characterized in that: The hybrid search includes global vector search and graph search; The global vector retrieval is measured by the inner product of the global vector; The graph retrieval is achieved by retrieving the subgraphs of the knowledge graph.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for constructing and applying a large model knowledge base of an embedded embodied intelligent visual language as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for constructing and applying a large model knowledge base of an embedded embodied intelligent visual language described in any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for constructing and applying a large model knowledge base of an embedded embodied intelligent visual language described in any one of claims 1 to 7 is implemented.

Images