A method for combining complex problem decomposition of a large model and multi-modal knowledge retrieval

Abstract

The application provides a complex problem decomposition and multi-modal knowledge retrieval method combined with a large model, and relates to the technical field of artificial intelligence and information retrieval. The method is aimed at a multi-modal knowledge base, provides a knowledge retrieval method with image understanding capability by extracting semantic information contained in image modal knowledge data, constructs the semantic and structural relationship between numerous sub-problems possibly contained in a complex problem by decomposing the complex problem, and is used for complex problem decomposition and multi-modal knowledge retrieval to generate accurate and comprehensive answers, so as to help the large model to more accurately and comprehensively answer the complex problem. In addition, the retrieval ability of the retrieval tool to the multi-modal knowledge base is expanded by integrating the related background knowledge of the complex problem.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and information retrieval technology, and particularly relates to a complex problem decomposition and multi-modal knowledge retrieval method combined with a large model. BACKGROUND

[0002] With the widespread dissemination of Internet multimedia information and the rapid development of artificial intelligence technology, machine reading comprehension and question answering systems have become a research hotspot. The core of this task is to understand the user's question, combine the user-provided or system-built knowledge base, retrieve the relevant information required by the question, and generate an accurate answer. This research direction not only can significantly reduce the user's search and reading time, but also can improve the flexibility of the question and answer interaction process, and has important application value in human-computer dialogue, intelligent tutoring, and medical diagnosis scenarios.

[0003] In recent years, large language models (referred to as large models) represented by ChatGPT, GPT-4, and LLaMA systems have shown excellent induction and summarization capabilities and diversified instruction following capabilities due to their large model parameter quantity and pre-training corpus size. In order to further improve the accuracy of the large model in answering questions (i.e., reduce the large model hallucination problem), according to the user's question, the knowledge base is retrieved to provide context for the large model, and this method of combining knowledge retrieval enhancement capabilities has become the mainstream method for current machine reading comprehension and question answering tasks. However, this method has certain limitations: first, the ability to handle complex problems is insufficient. The user's question often contains multiple sub-questions, and even further reasoning of the question is required to understand the user's intention. In this scenario, the large model may have difficulty accurately understanding and analyzing the question, and the retrieval tool cannot directly match the content required to solve the complex problem from the knowledge base. Second, in many application scenarios (such as education and medicine), the user's question may involve information combining images, tables, or other non-textual data, and the key information may need to be combined with images, tables, or other non-textual data to answer the question. However, most current methods still focus on text modality retrieval and are difficult to efficiently integrate and process multi-modal information.

[0004] In summary, although the current machine reading comprehension and question answering system has made great progress under the promotion of the large model, it still faces challenges such as the lack of ability to decompose complex problems and retrieve and integrate multi-modal knowledge. Therefore, in order to solve the problem of what knowledge to retrieve and how to retrieve knowledge, developing a method that takes the large model as the core and has the ability to handle complex problems and retrieve multi-modal knowledge is the key to improving the accuracy and comprehensiveness of the question and answer system.

[0005] With the increasing general ability and instruction compliance of large models, existing related question and answer systems have shifted to knowledge retrieval research centered on large models, that is, different retrieval algorithms are used to retrieve corresponding knowledge for related problems, so as to rely on the summarization and reasoning ability of large models to answer questions.

[0006] In short, because people will ask flexible and diverse questions, the background knowledge required to answer corresponding questions is not the same, so traditional machine reading comprehension and question and answer systems have been difficult to meet people's needs. This is because as the complexity and flexibility of the question increase, it is difficult for understanding-based models to correctly answer users' questions, which is also the reason why traditional tasks answer questions in the form of multiple-choice questions.

[0007] And as a generative model, large models not only accumulate a large amount of basic knowledge in the pre-training stage, but also to a certain extent, follow various instructions given by users, and most importantly, large models can generate corresponding answers according to different question styles and requirements of users, so they are more practical.

[0008] The Chinese patent "CN202410213818 A large language model question and answer method and device based on knowledge retrieval enhancement" retrieves the knowledge that the question may involve from the knowledge base, and then combines the historical questions and answers in the multi-round dialogue to enhance the question and answer ability of the large model.

[0009] The Chinese patent "CN202410343664 A large language model reasoning method and system based on multi-level knowledge retrieval enhancement" optimizes the accuracy and relevance of the large model in generating answers through layer-by-layer retrieval of the operation knowledge base and industry knowledge base. The specific process is as follows: first, vectorize the question statement, compare it with the operation knowledge base, and if successful, output the corresponding answer; if not, turn to the industry knowledge base, obtain high-relevance text fragments through segmented vectorization matching, and generate answers combined with the large model. The core methods include BGE algorithm vectorization, cosine similarity calculation, multi-path recall and precision sorting. The purpose is to alleviate the "hallucination" of large models and achieve efficient and accurate industry knowledge question and answer

[0010] The current related technologies either focus on how to better retrieve a knowledge base or focus on how to sort the retrieved knowledge, but ignore the question of "what knowledge should be retrieved". That is, the existing technologies have an assumed premise that a given question (also known as a query) itself is a text that can be directly used to retrieve a knowledge base, and the text itself is not complex and can be directly understood by the retrieval tool in terms of its intention and semantics. However, in actual applications, the question asked by the user can be complex, and the sub-questions and reasoning paths that can be contained in the complex question need to be carefully analyzed and understood and decomposed before they can be directly retrieved by the retrieval tool or retrieved by the existing technology optimized retrieval method.

[0011] In addition, the knowledge base used by the existing technology is basically text-based, but in many cases, the knowledge base is given in a multi-modal form (especially in the form of documents), which contains a lot of image content such as pdf, word documents, etc., and part of the content is not directly implied in the text but needs to be understood by referring to the accompanying images. Therefore, the existing technology also lacks a simple and efficient method for retrieving a multi-modal knowledge base. SUMMARY

[0012] The technical problem to be solved by the present application is to solve the above-mentioned deficiencies of the prior art, and to provide a complex question decomposition and multi-modal knowledge retrieval method combined with a large model for complex question decomposition and multi-modal knowledge retrieval to generate accurate and comprehensive answers.

[0013] To solve the above technical problems, the technical solution adopted by the present application is:

[0014] The present application provides a complex question decomposition and multi-modal knowledge retrieval method combined with a large model, which specifically includes the following steps:

[0015] Step 1: Select a large model to be used as a base model;

[0016] Step 2: According to the specific business scenario requirements, construct a named entity recognition dataset for training the selected base model to obtain a trained base model f LLM ;

[0017] Construct a named entity recognition dataset D NER ={(S p ;{e p1 ,e p2 ,…,e pd})|p=1,2,…,D} for training the base model, where S p is the pth text in the dataset with a sample size of D, {e p1 ,e p2 ,…,e pd} for named entity recognition dataset D NER d entities contained in the pth text in the middle;

[0018] Step 3: Establish a multi-modal knowledge base, preprocess it, and convert the multi-modal knowledge base into a pure text knowledge base, which specifically includes: extracting semantic information contained in image modal knowledge data in the multi-modal knowledge base, converting the image modal knowledge data into text modal knowledge data, and then converting the text modal knowledge data into text modal knowledge vectors;

[0019] The multi-modal knowledge base refers to a knowledge base containing text-image modal knowledge data, and the data in the knowledge base does not have to be multi-modal data. The modality of the data depends on the actual business and use scenario. If the given knowledge base contains image modal knowledge data, the image modal data needs to be preprocessed;

[0020] The multi-modal knowledge base includes system-built knowledge bases and user-provided knowledge bases. The system-built knowledge base refers to existing professional knowledge bases, including but not limited to knowledge graphs, graph databases, and vector query databases. The user-provided knowledge base refers to the multi-modal context provided by the user to the large model to help answer questions, i.e., the general text or document provided by the user;

[0021] When preprocessing the multi-modal data contained in the multi-modal knowledge base, the system-built knowledge base only needs to be processed offline once. The data contained in the system-built knowledge base is preprocessed and can be directly retrieved for use. For the knowledge base provided by the user, real-time processing is required. If the user does not provide a knowledge base when asking a question, skip the real-time processing of the user-provided knowledge base and only preprocess the system-built knowledge base. Conversely, if the user provides a knowledge base when asking a question, skip the preprocessing of the system-built knowledge base and only perform real-time processing on the user-provided knowledge base;

[0022] Step 3.1: Use an optical character recognition tool f OCR Extract the optical character information of each image contained in all image modal knowledge data in the multi-modal knowledge base, and use a large model f with multi-modal processing capability MLLM Recognize the semantic information contained in the optical character information of each image;

[0023] Step 3.1.1: Use an optical character recognition tool f OCR Extract the optical character information of each image contained in all image modal knowledge data in the multi-modal knowledge base;

[0024] Use an optical character recognition tool f OCROptical character information is extracted from image modal knowledge data. For image I located at position P in the multimodal knowledge base, the optical character information of this image is represented as OCR = f OCR (I) If the image does not contain optical character information, then the OCR is empty;

[0025] Step 3.1.2: Based on the optical character recognition information, position information, and context information of the image, use a large model with multimodal processing capabilities to identify the semantic information contained in the optical character information of each image;

[0026] For the position P of image I, let C be the text content above it at a distance of δ1 rows. 1 (δ1), the following content of line δ2 is C 2 (δ2), the final identified optical character information of the image contains semantic information F as follows:

[0027] F = f MLLM (I;C) 1 (δ1); OCR; C 2 (δ2)) (1)

[0028] Step 3.2: Replace the images corresponding to each location in the multimodal knowledge base with the optical character information of the images, which contains semantic information, and convert the multimodal knowledge base into a plain text knowledge base;

[0029] Step 4: Receive a complex question given by the user, decompose the complex question based on the entity recognition model, and use the dependency parsing model to obtain the entity information in the complex question;

[0030] The complex problem is one consisting of two or more entities, or one composed of two or more sub-problems.

[0031] Step 4.1: Based on entity recognition model f NER Obtain the set of latent entities in a complex problem As shown in the formula below:

[0032]

[0033] Where Q is a complex problem, {e1,e2,…,e E Let E be the potential entities identified from the complex problem Q;

[0034] Step 4.2: Using syntactic analysis model f DSP Obtain the set of key entities in a complex problem And update the potential entity set to obtain the updated potential entity set.

[0035] Key entities are defined as all types of nouns obtained after dependency syntactic analysis;

[0036] Key entity set And the updated potential entity set is shown in the following formula:

[0037]

[0038] Where {s1, s2, …, sn} is the N key entities in the complex question obtained by the syntactic analysis model f N , {c1, c2, …, cm} is the m key potential entities in the updated potential entity set; DSP m

[0039] Step 4.3: According to the dependency syntactic structure and the key entity set Through the direct, indirect or clause form of association between nouns, the association entity set is formed Where (s i , s j ) is the i-th association entity and the j-th association entity, (s i , s j ) z is the z-th association entity pair in the association entity set, and n is the number of association entity pairs contained in the complex question Q;

[0040] Step 5: According to the entity information in the complex question, the complex question is converted into multiple sub-questions, and the direct knowledge and associated knowledge of each sub-question is retrieved from the pure text knowledge base;

[0041] Step 5.1: For the key potential entities in the updated potential entity set , a direct sub-question set Q sub is generated based on the direct question template T sub ;

[0042] Based on the direct question template T sub , the direct sub-question set Q sub is generated as shown in the following formula:

[0043]

[0044] Where q y is a direct sub-question in the direct sub-question set Q sub , and c y is a key potential entity; for any key potential entity c y , its corresponding direct sub-question q y is generated by filling the direct question template, until the updated potential entity set​​ all the key potential entities in the text are filled in;

[0045] Step 5.2: Retrieving the set of direct sub-questions from the pure text knowledge base based on the association question template T rel to generate the set of association sub-questions Q rel ;

[0046] based on the association question template T rel , to generate the set of association sub-questions Q rel as shown in the following formula:

[0047]

[0048] wherein q i,j is an association sub-question in the set of association sub-questions Q rel , (q i,j ) w is the wth association sub-question in the set of association sub-questions Q rel ;

[0049] Step 5.3: Retrieving the set of direct sub-questions Q sub from the pure text knowledge base y to obtain direct knowledge K y about each direct sub-question q sub ;

[0050] For the direct sub-question q y , use the retrieval tool f retrieval matched with the pure text knowledge base to retrieve its relevant knowledge from the pure text knowledge base, if no direct knowledge is retrieved, then the basic model is used to answer the question alone, as reference knowledge and generate direct knowledge K sub ; if the retrieved direct knowledge is one or more, then sort all the retrieved direct knowledge according to the relevance score of each piece of direct knowledge by the basic model, and splice each piece of direct knowledge from high to low according to the relevance score, to generate direct knowledge K sub :

[0051] K sub = f retrieval (Q sub ) = {k u |u = 1, 2, …, U} (6)

[0052] wherein k u is the uth direct knowledge of the direct sub-question q y , U is the number of direct knowledge, f retrieval is the retrieval tool used to retrieve knowledge from the pure text knowledge base and return the retrieved knowledge;

[0053] Step 5.4: Retrieving the associated sub-problem set Q in the plain text knowledge base rel each associated sub-problem q i,j , obtaining the associated knowledge K i,j about the associated sub-problem q rel ;

[0054] For the associated sub-problem q i,j , its relevant knowledge is retrieved from the plain text knowledge base. If no associated knowledge is retrieved, "there may be no relationship" and the answer obtained by directly answering the base model are spliced to form reference knowledge and generate the associated knowledge K rel ; if one or more associated knowledge is retrieved, all the retrieved associated knowledge is sorted according to the relevance score of the base model to each piece of associated knowledge, and each piece of associated knowledge is spliced from high to low according to the relevance score to generate the associated knowledge K rel :

[0055] K rel = f retrieval (Q rel ) = {(k i,j ) v |v = 1, 2, …, V} (7)

[0056] where (k i,j ) v is the vth associated knowledge of the associated sub-problem q i,j , and V is the number of associated knowledge;

[0057] Step 6: Based on the knowledge graph embedding model f KGE , the complex problem, the sub-problem set, the direct knowledge related to the sub-problem and the associated knowledge are constructed to build the knowledge graph G of the complex problem;

[0058] Step 6.1: Define the set of binary logical relationships R between all key entities in the complex problem;

[0059] The set of binary logical relationships R between the key entities in the complex problem includes symmetric relationship, anti-symmetric relationship and no relationship;

[0060] For any two entities e l and e h in the set of potential entities , the symmetric relationship is as follows:

[0061]

[0062] where r is the relationship between the two entities, and the relationship between e l and e hGiven the existence of a relation r, can e be directly derived? h and e l If there is also a relationship r between the two entities, then there is a symmetrical relationship between them.

[0063] The antisymmetric relation is shown in the following equation:

[0064]

[0065] In contrast to symmetry, when entity e is known l and e h Given that a relation r exists, can e be derived? h and e l If there is no relation r between the two entities, then there is an antisymmetric relation between them.

[0066] No relation, meaning there is no semantic relationship between the two entity pairs;

[0067] Step 6.2: Use the knowledge graph embedding model f KGE encoder f emb The direct knowledge corresponding to the direct sub-problems obtained from the decomposition of complex problems and the related knowledge corresponding to the related sub-problems obtained from the decomposition of complex problems are all converted into a knowledge graph embedding model f. KGE The required embedding vector;

[0068] Using knowledge graph embedding model f KGE encoder f emb This yields the embedding vector E of the complex problem Q. Q =f emb (Q);

[0069] Each direct subproblem q y It combines several direct knowledge points to integrate the entity and its background knowledge, using a knowledge graph embedding model f. KGE encoder f emb This yields the embedding vector E of direct knowledge. y =f emb (q y ;k u );

[0070] Each related subproblem q i,j By piecing together several related pieces of knowledge, the background knowledge of related entity pairs and the possible relationships between them is integrated, and a knowledge graph embedding model f is used. KGE encoder f emb The embedded vector E of the associated knowledge is obtained. i,j =f emb (q i,j ;(ki,j ) v );

[0071] Step 6.3: Input the embedding vector of the complex problem Q, the embedding vector of direct knowledge, and the embedding vector of related knowledge into the knowledge graph embedding model f. KGE We obtain the knowledge graph G corresponding to the decomposition of the complex problem;

[0072] Using knowledge graph embedding model f KGE The resulting set of potential entities any entity e in l and e h The logical relationship between them is:

[0073] r l.h =f KGE (E Q E l E h E l,h ),r l,h ∈R (10)

[0074] Further, for the potential entity set The logical relationship that exists in it is r lh any entity e l and e h The knowledge graph G is:

[0075] G={{(e l ,k l ),r l.h ,(e h ,k h )} g |g=1,2,…,G} (11)

[0076] Step 7: Integrate the elements contained in the knowledge graph G into a new question Q. * And by the trained base model f LLM Generate a new question Q * Complete and generate answer A;

[0077] Step 7.1: Determine the entity-direct knowledge pairs (e) contained in each element of the knowledge graph G. l ,k l If the length of the direct knowledge exceeds the maximum context length specified by the basic model, summarize and abbreviate the direct knowledge that exceeds the maximum context length specified by the basic model so that the total length of the direct knowledge does not exceed the maximum context length Θ of the basic model.

[0078] Define the maximum context length Θ that the base model can handle, and define each entity-direct knowledge pair (e l ,kl ) the maximum length l If the length of a certain entity-direct knowledge pair (e l ,k l ) exceeds its specified maximum length l , the direct knowledge k LLM of the base model f l is summarized and abbreviated until the length limit is met, so the maximum context length

[0079]

[0080] Step 7.2: Fill the original complex question Q and each element contained in the knowledge graph G into the new question template T new to generate a new question Q * , and the trained base model f LLM generates a reply A according to the new question Q * :

[0081]

[0082] Define a new question template T new , fill each element contained in the knowledge graph G into the new question template T new to form a new question Q * , and the trained base model f LLM generates a reply A, which is the answer to the new question Q * and also the answer to the original question Q.

[0083] The beneficial effects produced by the above technical solutions are that the complex question decomposition and multi-modal knowledge retrieval method combined with a large model provided by the present application provides a knowledge retrieval method with image understanding capability by extracting semantic information contained in image modal knowledge data for multi-modal knowledge base, and through the decomposition of complex questions, the semantic and structural relationships between numerous sub-questions that may be contained in complex questions are constructed, thereby helping the large model to more accurately and comprehensively answer complex questions. In addition, by integrating the relevant background knowledge of complex questions, the retrieval ability of the retrieval tool for the multi-modal knowledge base is expanded. BRIEF DESCRIPTION OF DRAWINGS

[0084] Figure 1 A flowchart of a complex question decomposition and multi-modal knowledge retrieval method combined with a large model provided by an embodiment of the present application. DETAILED DESCRIPTION

[0085] The specific embodiments of the present application are described in further detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present application but are not intended to limit the scope of the present application.

[0086] The present embodiment is a complex problem decomposition and multi-modal knowledge retrieval method combined with a large model, as shown in Figure 1 The method comprises the following steps:

[0087] Step 1: Select a large model to be used as a base model;

[0088] The present embodiment does not limit the selected large model (LLM), and a large model with multi-modal processing capability can be selected, including but not limited to GPT-4o, Gemini, LLaVA, and LLaMA, etc. In some embodiments, if a large model without multi-modal processing capability is selected, an additional model with image content description capability is needed, such as a BLIP model with image content understanding capability. In the present embodiment, a large model with multi-modal processing capability is selected as the base model f LLM .

[0089] Step 2: According to the specific business scenario requirements, a named entity recognition dataset is constructed for training the selected base model, to obtain a trained base model f LLM .

[0090] In the present embodiment, the business scenario includes but is not limited to real application scenarios in specific fields such as the education field, the medical field, the legal field, etc.

[0091] Named entity recognition (NER) refers to the technology of identifying all potential named entities (hereinafter referred to as entities) in a given text, including names, place names, organization names, and professional terms in specific business scenarios, etc.

[0092] A named entity recognition dataset D NER is constructed for training the base model, where S p is the pth text in the dataset with a sample number of D, which is usually a natural sentence, and {e p1 , e p2 , …, e pD} is the named entity recognition dataset D i . p1 p2 pd NER ​​​d entities contained in the pth text; for example, the text "A has B" is constructed as two entities "A" and "B" to form the named entity recognition dataset D NER , one sample in the dataset D

[0093] In this embodiment, it is not limited that the named entity recognition dataset must be constructed. Whether the named entity recognition dataset needs to be constructed can be determined according to whether there is an optimization requirement for one or more business scenarios. If there is an optimization requirement for one or more business scenarios, the named entity recognition dataset is constructed for optimizing the base model f LLM , otherwise, the named entity recognition dataset does not need to be constructed. The named entity recognition dataset required by the business scenario is constructed through prompting engineering and other technologies, and the training of the base model f LLM is performed through fine-tuning.

[0094] Step 3: Establishing a multi-modal knowledge base, preprocessing it, and converting the multi-modal knowledge base into a pure text knowledge base, specifically including: extracting semantic information contained in image modal knowledge data in the multi-modal knowledge base, converting the image modal knowledge data into text modal knowledge data, and then converting the text modal knowledge data into text modal knowledge vectors.

[0095] In this embodiment, the multi-modal knowledge base refers to a knowledge base containing text-image modal knowledge data, and it is not limited that the data in the knowledge base must be multi-modal data. The modality of the data depends on the actual business and use scenario. If the given knowledge base only contains pure text modal knowledge data, it does not affect the subsequent steps of this embodiment. If the given knowledge base contains image modal knowledge data, the image modal data needs to be preprocessed.

[0096] The multi-modal knowledge base includes a system built-in knowledge base and a user-provided knowledge base. The system built-in knowledge base refers to existing professional knowledge bases, including but not limited to knowledge graphs, graph databases, and vector query databases, etc. The user-provided knowledge base refers to the multi-modal context provided by the user to the large model to help answer the question, i.e., the general text or document provided by the user when using the large model.

[0097] When preprocessing the multi-modal data contained in the multi-modal knowledge base, the system built-in knowledge base only needs to be processed offline once. The preprocessing operation is performed on the data contained in the system built-in knowledge base, which can be directly retrieved and used without repeating the preprocessing operation. For the knowledge base provided by the user, real-time processing is needed to meet the user's specific needs. If the user does not provide a knowledge base when asking a question, the real-time processing of the user-provided knowledge base is skipped, and only the system built-in knowledge base is preprocessed. Conversely, if the user provides a knowledge base when asking a question, the preprocessing of the system built-in knowledge base is skipped, and only the real-time processing of the user-provided knowledge base is performed.

[0098] Step 3.1: using an optical character recognition tool f OCR Extracting the optical character information of each image contained in all image modal knowledge data in the multi-modal knowledge base, and using a large model f with multi-modal processing capability MLLM to identify the semantic information contained in the optical character information of each image;

[0099] Since existing knowledge retrieval tools are often based on text retrieval, they lack the ability to retrieve image data, or directly provide image content to large models through visual encoders, which greatly affects the context processing capability of large models. In order to enhance the retrieval capability of image modal knowledge data and reduce the data volume of image modal knowledge, the embodiment uses a modal conversion method to convert image modal information into text modal using an optical character recognition tool f OCR and image content description technology;

[0100] Step 3.1.1: using an optical character recognition tool f OCR Extracting the optical character information of each image contained in all image modal knowledge data in the multi-modal knowledge base;

[0101] For tables, bills and other formatted text or numerical knowledge information provided in image modal, traditional visual encoders and image content description tools are difficult to accurately identify the text or numerical information therein, therefore, the embodiment uses an optical character recognition tool f OCR to extract the optical character information in the image modal knowledge data. For an image I located at position P in the knowledge base, the representation of the optical character information of the image is OCR = f OCR (I), if the image does not contain optical character information, OCR is empty;

[0102] Step 3.1.2: using a large model with multi-modal processing capability to identify the semantic information contained in the optical character information of each image according to the optical character recognition information, position information and context information of the image;

[0103] In order to identify the semantic information in the image, such as the general content of the image, a large model with multi-modal processing capability is used, and the optical character recognition information, position information and context information of the image are provided to identify the semantic information in the image modal knowledge data; for the position P where the image I is located, the context content at a distance δ1 rows is defined as C 1 (δ1), the context content at a distance δ2 rows is defined as C 2 (δ2), and finally the semantic information F contained in the optical character information of the image is identified as: x

[0104] F = f LLM (I; C​1 (δ1); OCR; C 2 (δ2)) (1)

[0105] Step 3.2: replacing the corresponding image at each position in the multi-modal knowledge base with the semantic information of the optical character information of the image, and converting the multi-modal knowledge base into a pure text knowledge base;

[0106] Step 4: receiving a given complex question from a user, decomposing the complex question based on an entity recognition model, and obtaining entity information in the complex question by using a dependency syntax analysis model;

[0107] A complex question is defined as a question composed of two or more entities, or a question composed of two or more sub-questions. A complex question cannot be directly answered, but needs to be answered according to the reasoning path formed by the entities involved in the complex question and the relationship between the multiple entities. If the reasoning path formed is not clear, or the basic model used does not have the relevant background knowledge of the entity, the question may not be answered correctly or comprehensively. For example, for a question such as "Who is A", which is composed of a single entity, it can be regarded as a simple question; and for a question such as "A obtains B on the date", which contains multiple entities, not only does it need to know who A is, but also needs to know what B is, and the meaning of the date entity in this sentence, so this kind of question containing multiple entities is a complex question.

[0108] In this embodiment, the specific used named entity recognition model f NER (hereinafter referred to as an entity recognition model) can be selected from any existing model, such as a BERT-based entity recognition model; in addition, according to the named entity recognition data set D NER , it can be selected according to the specific business scenario whether the selected entity recognition model needs to be further trained, and in this embodiment, the training method used for the entity recognition model is not limited.

[0109] The dependency syntax analysis model f DSP (hereinafter referred to as a syntax analysis model) decomposes the complex question text input by the user into a syntax tree with a dependency structure including a noun subject, a direct object, an indirect object, an adjective clause, and a copula.

[0110] Step 4.1: obtaining a set of potential entities in the complex question based on the entity recognition model f NER As shown in the following formula:

[0111]

[0112] Wherein, Q is a complex question, {e1, e2, …, e E ​} are the E potential entities identified from the complex question Q;

[0113] Step 4.2: obtaining N key entities in the complex question Q through the syntactic analysis model f DSP obtaining a key entity set in the complex question and updating the potential entity set to obtain an updated potential entity set

[0114] The key entity is defined as all types of nouns obtained after dependency syntactic analysis, which act as the subject, direct object, indirect object and various types of clauses in the sentence, playing a key role in syntax. Since these nouns are also part of the entity, in order to avoid the difference between potential entities and key entities, after obtaining the potential entity set, the potential entity set is updated as shown in the following formula:

[0115]

[0116] wherein {s1, s2, …, s N} are the N key entities in the complex question Q obtained through the syntactic analysis model f DSP {c1, c2, …, c m} are m key potential entities in the updated potential entity set;

[0117] Step 4.3: obtaining the association entity set according to the dependency syntactic structure and the key entity set The association entity set is formed by the direct, indirect or clause form association between the nouns wherein (s i ,s j ) is the i th association entity and the j th association entity, (s i ,s j ) z is the z th association entity pair in the association entity set, and n is the number of association entity pairs contained in the complex question Q;

[0118] The association entity refers to the entity pair with the subject, object and clause form association in the dependency syntactic tree. Specifically, the subject noun and its direct object are associated, the direct object and its indirect object are associated, and any noun and its subsequent clause are associated. If a noun does not have an association with any other noun, it is considered an exception and is associated with any other noun;

[0119] Step 5: converting the complex question into multiple sub-questions according to the entity information in the complex question, and retrieving the direct knowledge and associated knowledge of each sub-question from the pure text knowledge base;

[0120] In this embodiment, the retrieval tool and the retrieval manner used are not limited, and a retrieval tool f matching the pure text knowledge base is used retrieval For example,

[0121] Step 5.1: Based on the direct question template T sub , a direct sub-question set Q sub is generated for the key potential entity in the updated potential entity set ;

[0122] The direct question template refers to a simple question that can be directly retrieved and answered after each entity is filled in according to the template; in this embodiment, “what / who is c y ?” is used as the direct question template to generate a direct sub-question corresponding to the key potential entity, for example, for the entity “A”, a direct sub-question like “what / who is A?” is generated;

[0123] Based on the direct question template T sub , the direct sub-question set Q sub is generated as follows:

[0124]

[0125] wherein q y is a direct sub-question in the direct sub-question set Q sub , and c y is a key potential entity; for any key potential entity c y , a direct sub-question q y corresponding to the key potential entity is generated by filling in the direct question template until all the key potential entities in the updated potential entity set are filled in;

[0126] Step 5.2: Based on the association question template T rel , an association sub-question set Q rel is generated for the association entity set ;

[0127] The association question template refers to a question that inquires about the relationship between two entities for each element included in the association entity set; in this embodiment, “what is the relationship between s i and s j ?” is used as the association question template, and for the association entity pair (A, B), an association sub-question like “what is the relationship between A and B?” is generated;

[0128] Based on the association question template T rel , the association sub-question set Q rel is generated as follows:

[0129]

[0130] where q i,j is the associated sub-problem in the associated sub-problem set Q rel , (q i,j ) w is the wth associated sub-problem in the associated sub-problem set Q rel ;

[0131] Step 5.3: retrieve each direct sub-problem q sub in the direct sub-problem set Q a from the pure text knowledge base to obtain direct knowledge K y about each direct sub-problem q sub ;

[0132] For the direct sub-problem q y , use the retrieval tool f retrieval matched with the pure text knowledge base to retrieve its relevant knowledge from the pure text knowledge base, and if no direct knowledge is retrieved, directly answer the question by the base model as reference knowledge and generate direct knowledge K sub ; if one or more direct knowledge is retrieved, sort all the retrieved direct knowledge according to the relevance score of each piece of direct knowledge by the base model, and concatenate each piece of direct knowledge in descending order of relevance score to generate direct knowledge K sub :

[0133] K sub = f retrieval (Q sub ) = {k u |u = 1,2,…,U} (6)

[0134] where k u is the uth direct knowledge of the direct sub-problem q y , U is the number of direct knowledge, and f retrieval is the retrieval tool used to retrieve knowledge from the pure text knowledge base and return the retrieved knowledge;

[0135] Step 5.4: retrieve each associated sub-problem q rel in the associated sub-problem set Q i,j from the pure text knowledge base to obtain associated knowledge K i,j about the associated sub-problem q rel ;

[0136] For the associated sub-problem q i,j , retrieve its relevant knowledge from the pure text knowledge base, and if no associated knowledge is retrieved, concatenate “there may be no relationship” and the answer directly answered by the base model as reference knowledge and generate associated knowledge Krel If one or more related knowledge items are retrieved, all retrieved related knowledge items are sorted according to the relevance score of each related knowledge item based on the basic model. The related knowledge items are then concatenated in descending order of relevance score to generate related knowledge K. rel :

[0137] K rel =f retrieval (Q rel )={(k i,j ) v |v=1,2,…,V} (7)

[0138] Among them, (k i,j ) v For the related subproblem q i,j The vth related knowledge, where V is the number of related knowledge;

[0139] Step 6: Based on the knowledge graph embedding model f KGE Construct a knowledge graph G for the complex problem, the set of subproblems, the direct knowledge related to the subproblems, and the related knowledge;

[0140] Knowledge Graph Embedding (KGE) models refer to a class of models that learn vector representations between entities and relations to capture semantic relationships between entities, thereby aiding in knowledge graph modeling and reasoning. These include, but are not limited to, TransE, RotateE, and ConvE. This embodiment does not limit the knowledge graph embedding model used; by establishing a knowledge graph with semantic relationships, it helps large models better perform reasoning on complex problems.

[0141] Step 6.1: Define the set R of binary logical relations that exist between all key entities in the complex problem;

[0142] The set of relation types mainly refers to the semantic relationship types that may exist between entities in a complex problem posed by a user, and even the relationships between those relationships. It is generally represented using binary logical relations in discrete mathematics, including symmetric relations, antisymmetric relations, inverse relations, transitive relations, and combinatorial relations, etc. By establishing a knowledge graph with entity semantics as nodes and logical relations as elements, and leveraging the theory of discrete mathematics, it is possible to help large models establish reasoning paths for complex problems, thereby providing more accurate and comprehensive answers. In this embodiment, the set R of binary logical relations between key entities in a complex problem includes three types: symmetric relations, antisymmetric relations, and no relations.

[0143] For potential entity set Any two entities e l and eh The symmetric relation is shown in the following formula:

[0144]

[0145] Wherein, r is the relationship between two entities, and e l and e h have the relationship r, it can be directly deduced that e h and e l also have the relationship r, then the two entities have a symmetric relationship, such as classmates, friends, lovers and the like;

[0146] The anti-symmetric relation is shown in the following formula:

[0147]

[0148] In contrast to the symmetric relation, under the premise that e l and e h have the relationship r, it can be deduced that e h and e l do not have the relationship r, then the two entities have an anti-symmetric relationship, such as parent-child relationship;

[0149] No relationship, that is, there is no semantic relationship between the two entity pairs;

[0150] Step 6.2: using the encoder f KGE of the knowledge graph embedding model f emb , the complex problem, the direct knowledge corresponding to the direct sub-problem obtained by decomposing the complex problem and the associated knowledge corresponding to the associated sub-problem obtained by decomposing the complex problem are all converted into the embedding vector required by the knowledge graph embedding model f KGE ;

[0151] Using the encoder f KGE of the knowledge graph embedding model f emb , the embedding vector E Q of the complex problem Q is obtained f emb (Q);

[0152] Splicing each direct sub-problem q y and several direct knowledge, thereby fusing the entity and the background knowledge of the entity together, using the encoder f KGE of the knowledge graph embedding model f emb , the embedding vector E y of the direct knowledge is obtained f emb (q y ; k u );

[0153] Splicing each associated sub-problem qi,j and their possible relationships, and fuse the background knowledge of the related entity pairs and their possible relationships together, using the knowledge graph embedding model f KGE , to obtain the embedding vector E emb of the related knowledge. i,j = f emb (q i,j ; (k i,j ) v );

[0154] Step 6.3: input the embedding vector of the complex problem Q, the embedding vector of the direct knowledge, and the embedding vector of the related knowledge into the knowledge graph embedding model f KGE , to obtain the knowledge graph G corresponding to the decomposition of the complex problem.

[0155] The logical relationship between any entity e l and e h in the latent entity set obtained using the knowledge graph embedding model f KGE is:

[0156] r l.h = f KGE (E Q ,E l ,E h ,E l,h ), r l,h ∈R (10)

[0157] Further, the knowledge graph G of any entity e l,h and e l with the logical relationship r h existing in the latent entity set is:

[0158] G = { {(e l , k l ), r l.h , (e h , k h )} g | g = 1, 2, …, G} (11)

[0159] Step 7: integrate each element contained in the knowledge graph G into a new problem Q * , and generate the new problem Q LLM by the trained base model f * to complete and generate the answer A.

[0160] Step 7.1: judge the entity-direct knowledge pair (e l , k lwhether the length of each entity-direct knowledge pair (e

[0161] The maximum context length Θ that the base model can handle is defined, and the maximum length θ l of each entity-direct knowledge pair (e l ,k l If the length of a certain entity-direct knowledge pair (e l ,k l ) exceeds its specified maximum length θ l , the base model f LLM performs summarization and abbreviation on its direct knowledge k l until the length limit is met, so the maximum context length Θ that the base model can handle is:

[0162]

[0163] The maximum context length that the base model can handle generally depends on the base model itself. In this embodiment, for simple calculation, the maximum length of each entity-knowledge pair is set to be equal, and the maximum context length that the base model can handle is equally divided into the number of elements in the knowledge graph.

[0164] Step 7.2: Fill the original complex question Q and each element contained in the knowledge graph G into the new question template T new to generate a new question Q * , and the trained base model f LLM generates a reply A according to the new question Q * :

[0165]

[0166] In this embodiment, the new question template T new is defined as: “Please answer the question based on the given background knowledge: Q; relevant background knowledge: (e l ,k l ), (e h ,k h ), the logical relationship between e l and e h is r l,h ”; after filling each element contained in the knowledge graph G into the new question template T new , a new question Q * is formed, and the reply A generated by the trained base model f LLM is the new question Q *The answer to the question is also the answer to the original question Q.

[0167] In this way, not only is the relevant background knowledge integrated and the retrieval ability of the retrieval tool for the multi-modal knowledge base expanded, but also the semantic and structural relationships between the numerous sub-questions that may be contained in the complex question are constructed through the decomposition of the complex question, thereby helping the large model to more accurately and comprehensively answer the complex question.

[0168] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, but not to limit it; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope defined by the claims of the present application.

Claims

1. A method combining complex problem decomposition of large models and multi-modal knowledge retrieval, characterized in that: The method comprises the following steps: Step 1, selecting a large model to be used as a base model; Step 2, constructing a named entity recognition dataset according to the specific business scenario requirements to train the selected base model to obtain a trained base model; Step 3: Establishing a multi-modal knowledge base, preprocessing it, and converting the multi-modal knowledge base into a pure text knowledge base; Step 4: Receiving a complex question given by a user, decomposing the complex question based on an entity recognition model, and obtaining entity information in the complex question using a dependency syntax analysis model; Step 5: According to the entity information in the complex question, converting the complex question into multiple sub-questions, and retrieving direct knowledge and associated knowledge of each sub-question from the pure text knowledge base; Step 6: Based on a knowledge graph embedding model, a knowledge graph of the complex question is constructed based on the complex question, the set of sub-questions, the direct knowledge related to the sub-questions, and the associated knowledge; Step 7: Integrating each element in the knowledge graph into a new question, and generating an answer by the trained base model; The step 5 comprises the following steps: Step 5.1: Generating direct sub-questions for the updated set of key potential entities in the updated set of potential entities Step 5.2: Selecting the best direct sub-question Step 5.3: Generating direct sub-questions for the updated set of key potential entities in the updated set of potential entities Step 5.4: Selecting the best direct sub-question Based on a direct problem template , generating a set of direct sub-problems As shown in the following equation: (4) wherein, is a direct sub-problem set in the direct sub-problem, is a key latent entity; for any key latent entity , a direct sub-problem corresponding to the key latent entity is generated after being filled by a direct question template , until all the key latent entities in the updated latent entity set are filled up. Step 5.2: Generating a set of associated entities based on the associated question template based on the associated question template based on the associated question template Generating a set of related sub-questions based on a related question template , generating a set of related sub-questions as follows: (5) wherein is the set of associated sub-problems is the associated sub-problem in the set of associated sub-problems is the set of associated sub-problems is the w-th associated sub-problem in the set of associated sub-problems Step 5.3: Retrieving the direct child question collection in the plain text knowledge base each direct child question , obtaining direct knowledge about each direct child question ; and ​ For direct sub-problems , use the retrieval tool matched with the pure text knowledge base Retrieve relevant knowledge from the pure text knowledge base, and if no direct knowledge is retrieved, answer the question alone by the base model as reference knowledge and generate direct knowledge If there is more than one piece of direct knowledge, sort all the retrieved direct knowledge according to the relevance score of each piece of direct knowledge by the base model, and splice each piece of direct knowledge from high to low according to the relevance score to generate direct knowledge : (6) wherein, is the u-th direct knowledge of the direct problem U is the number of direct knowledge, is the use of a search tool to search in a pure text knowledge base and return the searched knowledge; Step 5.4: Retrieving the associated sub-problem collection in the plain text knowledge base The associated sub-problems in each of the associated knowledge about the associated sub-problems ; For the sub-problem of association Retrieving relevant knowledge from the pure text knowledge base, if no relevant knowledge is retrieved, then "no relationship exists" and the answer obtained by directly answering the basic model are spliced together as reference knowledge and the relevant knowledge is generated If there is more than one piece of relevant knowledge retrieved, all the relevant knowledge retrieved is sorted according to the relevance score of the basic model for each piece of relevant knowledge, and each piece of relevant knowledge is spliced from high to low according to the relevance score to generate relevant knowledge : (7) wherein, is the vth associated sub-problem of the vth associated knowledge, V being the number of associated knowledge.

2. The method of claim 1, wherein: The named entity recognition dataset of step 2 is: Building a Named Entity Recognition Dataset Used to train the base model, where For a sample size of The first in the dataset A text, For named entity recognition datasets The Middle The text contains d entities.

3. The method of claim 2, wherein: The step 3 comprises: Extracting semantic information contained in image modal knowledge data in the multi-modal knowledge base, converting the image modal knowledge data into text modal knowledge data, and then converting the text modal knowledge data into text modal knowledge vectors; The multi-modal knowledge base refers to a knowledge base containing text-image modal knowledge data, and the data in the knowledge base is not limited to multi-modal data. The modality of the data depends on the actual business and use scenario. If the given knowledge base contains image modal knowledge data, the image modal data needs to be preprocessed; The multi-modal knowledge base includes a system built-in knowledge base and a user provided knowledge base. The system built-in knowledge base refers to existing professional knowledge bases, including but not limited to knowledge graphs, graph databases, and vector query databases. The user provided knowledge base refers to the multi-modal context provided by the user to help answer the question, i.e., the general text or document provided by the user when using the large model; When preprocessing the multi-modal data contained in the multi-modal knowledge base, the system built-in knowledge base only needs to be processed offline once. The data contained in the system built-in knowledge base is preprocessed and can be directly retrieved and used. For the user given knowledge base, real-time processing is required. If the user does not provide a knowledge base when asking a question, the real-time processing of the user given knowledge base is skipped, and only the preprocessing of the system built-in knowledge base is performed. Conversely, if the user provides a knowledge base when asking a question, the preprocessing of the system built-in knowledge base is skipped, and only the real-time processing of the user provided knowledge base is performed.

4. The method of claim 3, wherein: The step 3 comprises the following steps: Step 3.1: using an optical character recognition tool extracting optical character information of each image contained in all image modality knowledge data in the multi-modal knowledge base, and using a large model with multi-modal processing capability recognizing semantic information contained in the optical character information of each image; Step 3.1.1 : Using an optical character recognition tool extracting optical character information for each image contained in the image modality knowledge data of the multi-modal knowledge base; Using optical character recognition tools extracting optical character information in image modality knowledge data for an image located at a location in a multi-modal knowledge base a representation of the optical character information for the image is if the image does not contain optical character information, then null;​ Step 3.1.2: Using a large model with multimodal processing capabilities to process the image based on the optical character recognition information, location information, and context information Identifying semantic information contained in the optical character information of each image; For the image where the location , the content above its row is defined as , the content below its row is defined as , and the final recognition of the image's optical character information contains semantic information is: (1) Step 3.2: Replacing the images corresponding to each position in the multi-modal knowledge base with the semantic information contained in the optical character information of the images to convert the multi-modal knowledge base into a pure text knowledge base.

5. The method of claim 4, wherein: The complex question of step 4 is a question composed of two or more entities, or a question composed of two or more sub-questions; The step 4 comprises the following steps: Step 4.1: Entity recognition model based Obtaining a set of potential entities in a complex question As shown in the following equation: (2) wherein, is a complex problem, is a complex problem E potential entities identified from the complex problem; Step 4.2: passing the sentence to a syntactic analysis model obtaining a set of key entities in the complex question and updating the set of potential entities to obtain an updated set of potential entities ; Key entities are defined as all types of nouns after dependency syntactic analysis; Key entity set and the updated set of latent entities is given by the following equation: (3) wherein, are N key entities in the complex problem obtained by the syntactic analysis model are m key latent entities in the updated set of latent entities, and are m key latent entities in the updated set of latent entities, and Step 4.3: Forming the set of associated entities based on the dependency syntax structure and the set of key entities by the direct, indirect or clause form of association between the nouns, to form a set of associated entities wherein is an associated entity pair consisting of the ith associated entity and the jth associated entity, is the zth associated entity pair in the set of associated entities, and n is the number of associated entity pairs contained in the complex question Q.

6. The method of claim 1, wherein: The specific method of the step 6 is: Step 6.1 : Define the set of binary logical relationships that exist between all key entities in the complex problem ; A set of binary logical relations existing between key entities in the complex problem The set of binary logical relations includes symmetric relation, anti-symmetric relation and no relation. For any two entities in the set of potential entities and and The symmetric relation is given by the following equation: (8) wherein, is a relationship between two entities, given that and there is a relationship between it can be directly inferred that and there is also a relationship between then there is a symmetric relationship between the two entities; The anti-symmetry relation is shown in the following formula: (9) In contrast to the symmetric relation, it can be deduced that and have an anti-symmetric relation if it is known that there is a relation between and then there is no relation between the two entities.​ No relation, that is, there is no semantic relation between two entity pairs; Step 6.2: using a knowledge graph embedding model of the encoder , the direct knowledge corresponding to the direct sub-problems obtained by decomposing the complex problem and the associated knowledge corresponding to the associated sub-problems obtained by decomposing the complex problem are all converted into a knowledge graph embedding model embedding vectors required Using knowledge graph embedding models of an encoder to obtain embedding vectors for complex problems ; Each direct sub-problem and several pieces of direct knowledge thereof are spliced together, so as to fuse the entity and the background knowledge of the entity together, and obtain an embedding vector of the direct knowledge by using an encoder of a knowledge graph embedding model . ;​ Each related sub-problem By piecing together several related pieces of knowledge, the background knowledge of related entity pairs and the relationships between them is integrated, and a knowledge graph embedding model is used. encoder To obtain the embedding vector of related knowledge ; Step 6.3: Transform complex problems The embedding vectors of direct knowledge and the embedding vectors of related knowledge are input into the knowledge graph embedding model. To obtain the knowledge graph corresponding to the decomposition of complex problems. ; Using knowledge graph embedding models The resulting set of latent entities Any entity in the set of entities And The logical relationship between (10) Further, a set of potential entities is obtained The logical relationships existing in the set of potential entities are The knowledge graph G of any entity and is: (11)。 7. The method of claim 6, wherein: The specific method of the step 7 is: Step 7.1: Determine the entity-direct knowledge pairs contained in each element of the knowledge graph G. If the length of the direct knowledge exceeds the maximum context length specified by the base model, then the direct knowledge exceeding the maximum context length specified by the base model is summarized and abbreviated to ensure that the total length of the direct knowledge does not exceed the maximum context length of the base model. ; Define the maximum context length that the base model can handle. Define each entity-direct knowledge pair Maximum length If a certain entity-direct knowledge pair The length exceeds its specified maximum length Then, based on the basic model Direct knowledge of it Summarize and abbreviate until the length limit is met; therefore, the maximum context length that the base model can handle. for: (12) Step 7.2: Transform the original complex problem and knowledge graph The elements contained therein are sequentially filled into the new question template. New problems are generated in the middle And from the trained base model According to the new problem Generate reply : (13) Define a new question template Knowledge graph Fill the new question template with the elements contained therein in sequence. Then, new problems arose. From the trained base model generated response It is a new problem The answer is also the original question. The answer.

Images