A multimodal knowledge question-answering retrieval method and system for specific professional fields

Abstract

The present invention discloses a multimodal knowledge question and answer retrieval method and system for a specific professional field, belonging to the field of data processing technology. The present application calculates the cosine similarity between feature vectors in each mode through large model preprocessing to form a similarity matrix, and adopts the same vectorization processing method as the knowledge base text to realize the direct vector calculation of the question and the professional field knowledge base text. According to the input question vector, through the vectorization index retrieval in the professional field knowledge base, several text vectors closest to the question vector are selected from the professional field knowledge base to obtain several knowledge related to the question and the source of each piece of knowledge. The present application effectively improves the accuracy of knowledge question and answer replies in the target field, ensures the integrity of knowledge question and answer replies in the target field, and realizes basis support, supports localized deployment, solves information security issues, and realizes the intelligent application scenario of knowledge question and answer retrieval.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a multimodal knowledge question and answer retrieval method and system for a specific professional field. Background Art

[0002] The knowledge question answering system is an important application in the field of natural language processing. It enables computers to understand the meaning of questions raised by humans and return accurate and complete answers. It involves multiple research fields such as natural language processing, deep learning, knowledge graphs, and multimodality.

[0003] The big language model is an artificial intelligence technology based on deep learning. It has a wide range of applications in the field of natural language processing and can understand and generate natural language text. Its core is to use large-scale data sets to train the model. By learning the language rules and patterns in a large corpus, it can understand the grammatical structure, vocabulary and semantics of natural language, and then generate high-quality, natural and coherent text. At present, knowledge question answering based on the big language model has been widely used. Common big language models include BERT, GPT-2, GPT-3, GPT-4, LLaMa, etc. The big language model can answer most general questions and has achieved good performance. However, in a specific professional field, the accuracy, completeness and timeliness of the answers given by the general big language model are limited, and there are problems such as lack of basis and hallucination. For example, it may not accurately understand the meaning of professional vocabulary, thereby generating output that does not meet expectations, and giving irrelevant, specious or meaningless answers. At the same time, with continuous development, data shows a trend of modal diversification. It is not just a single text modality, but has expanded to multiple modalities such as images, videos, and audio. Therefore, when users ask questions, compared with traditional single-modal retrieval, cross-modality can obtain information from different types of data and effectively integrate them, which helps to improve the accuracy and effectiveness of information. Summary of the invention

[0004] The present application provides a multimodal knowledge question and answer retrieval method and system for specific professional fields to solve the problems of single modal information source of large language models in specific professional fields, and inaccurate, incomplete and unfounded knowledge question and answer responses. It is suitable for specific professional knowledge fields, supports localized deployment, solves information security issues, and realizes the intelligent application of knowledge question and answer retrieval.

[0005] In a first aspect, the present application provides a multimodal knowledge question-answering retrieval method for a specific professional field, the method comprising the following steps:

[0006] Steps for constructing a multimodal professional domain knowledge base: constructing a multimodal professional domain knowledge base in the form of a vector database, which is used to store basic information knowledge of different data modalities for the target professional field;

[0007] Input text preprocessing step: preprocess the question text input by the user to obtain the vector form of the question text, that is, the question vector; at the same time, find several prompt contents consisting of "question + professional knowledge" that are closest to the current question vector in the multimodal professional domain knowledge base, and create the source index of each professional knowledge corresponding to the prompt content;

[0008] Prompt input processing step: using prompt contents consisting of several "questions + professional knowledge" as the input message of the multimodal large language model, obtaining a preliminary original answer text based on the output of the large language model and transmitting it to the constraint large model output module;

[0009] Constrained output step: Taking the preliminary original answer text as input, and based on the knowledge graph for the target professional field and the set output constraints, the final output answer text is generated through the large language model and transmitted to the retrieval basis generation module;

[0010] Generate retrieval basis step: Generate the answer source and basis of the answer text based on the source index of each professional knowledge, and visually output the answer text and its answer source and basis to the user.

[0011] The second aspect of the present application provides a multimodal knowledge question-answering retrieval system for a specific professional field, which includes: a multimodal professional field knowledge base, an input text preprocessing module, a prompt input large model module, a constraint large model output module, and a retrieval basis generation module;

[0012] Among them, the multimodal professional domain knowledge base uses a vector database to store basic information knowledge of different data modalities for the target professional field; that is, the basic information knowledge of each data modality is stored in the multimodal professional domain knowledge base in the form of a vector;

[0013] The input text preprocessing module is used to preprocess the question text input by the user to obtain the vector form of the question text, that is, the question vector; at the same time, find out the prompt contents composed of several "question + professional knowledge" closest to the current question vector in the multimodal professional field knowledge base, and create the source index of each professional knowledge corresponding to the prompt content; and transmit the prompt contents composed of several "question + professional knowledge" to the prompt input large model module, and transmit the source index of each professional knowledge corresponding to the prompt content to the generation retrieval basis module;

[0014] The prompt input big model module uses the prompt content consisting of several "questions + professional knowledge" as the input message of the multimodal big model, obtains the preliminary original answer text based on the output of the big language model and transmits it to the constraint big model output module;

[0015] The constrained large model output module takes the preliminary original answer text as input, and based on the knowledge graph oriented to the target professional field and the set output constraints, generates the final output answer text through the large language model and transmits it to the retrieval basis generation module;

[0016] The retrieval basis generation module generates the answer source and basis of the answer text based on the source index of each professional knowledge, and visually outputs the answer text and its answer source and basis to the user.

[0017] Furthermore, basic information knowledge can be policies and regulations, document systems, speeches, case reports, etc. for the target professional field.

[0018] Furthermore, the data modalities of basic information knowledge include but are not limited to: text, voice, image, and video.

[0019] Furthermore, the multimodal professional domain knowledge base also includes a similarity matrix for each data modality, and the similarity matrix is ​​constructed as follows: content segmentation is performed on the basic information under the current data modality, and a content segmentation mode can be selected based on the actual application scenario to form unit content that can be used for user retrieval questions, and then the segmented single content (unit content) is converted into a feature vector (such as through a multimodal large language model to perform feature vector (i.e., text vector) conversion) to obtain multiple numerical vectors of the basic information under the current data modality and store them in the multimodal professional domain knowledge base; at the same time, in each modality, the cosine similarity between each feature vector is calculated to form a similarity matrix for the current data modality. The dimensions of the similarity matrices of each modality are the same, and insufficient data modalities can be filled based on specified values.

[0020] Furthermore, the input text preprocessing module finds several prompt contents consisting of "question + professional knowledge" that are closest to the current question vector in the multimodal professional domain knowledge base, and the specific prompt contents are:

[0021] Through vectorized index retrieval, the first k closest feature vectors are selected in the multimodal professional domain knowledge base, where k is greater than 1;

[0022] Based on the correlation between each selected feature vector and the question vector, they are sorted to obtain k prompt contents consisting of "question + professional knowledge" and the source index of each professional knowledge corresponding to each enhanced content.

[0023] Furthermore, the multimodal professional domain knowledge base is retrieved through vectorized indexing, and the top k closest feature vectors are selected in the multimodal professional domain knowledge base, specifically including:

[0024] The total similarity matrix is ​​obtained based on the similarity matrix of all data modalities ,in, represents the similarity matrix of the mth data modality, and ;

[0025] Select the problem vector The corresponding second-order interest feature vector:

[0026]

[0027] in, Represents the problem vector The set of closest multi-model feature vectors, Records for each The multimodal feature vector of The similarity score of represents the first k objects filtered out, Represents the total similarity matrix The Row, first-order interaction matrix is the set of all interactions between question vectors and multimodal feature vectors, Representing the problem vector set The number of problem vectors included, Represents a set of feature vectors The number of multimodal eigenvectors included, Used to represent the problem vector With multimodal feature vector The interaction status, This means that an interaction has taken place. This means that no interaction has taken place;

[0028] Create a second-order interaction matrix :

[0029]

[0030] The second-order interaction matrix As input, the question vector is obtained through the graph convolutional network With multimodal feature vector First collaboration information ;

[0031] According to the formula Calculating the Problem Vector With each multimodal feature vector The degree of association ,in, Represents the similarity matrix of the mth data modality The OK; Representation and Problem Vector First-order interest vectors with strong correlation;

[0032] In each Select the representative problem vector The interest vector is: ;

[0033] Constructing interest-related graphs:

[0034]

[0035] in, Represents the interest-related graph of the mth data modality The i-th row and j-th column of Representation and Problem Vector The second-order interest vector with strong correlation, Represents the multimodal feature vector index;

[0036] Similarity matrix of each data modality To perform sparseness:

[0037]

[0038] in, Represents the sparse similarity matrix of m data modalities The i-th row and j-th column of Represents the similarity matrix The OK;

[0039] According to interest related graph With the sparse similarity matrix Construct the interest-related features-feature map of the m-th data modality:

[0040]

[0041] Interest-related features-feature maps for each data modality Perform feature extraction to obtain the problem vector for each data modality With multimodal feature vector The eigenvector of ;

[0042] Fusion of feature vectors from all modalities Get the question vector With multimodal feature vector Second collaborative information ;

[0043] Based on the first collaborative information Information about collaboration with the second Get the feature embedding matrix , based on the feature embedding matrix The corresponding row of the problem representation is , feature embedding matrix The corresponding columns of , and then according to Calculating the Problem Vector and the multimodal feature vector The interaction score between

[0044] For each problem vector , select the first k multimodal feature vectors with the highest scores as the first k closest eigenvectors.

[0045] Furthermore, the interest-related features of each data modality are analyzed separately. Perform feature extraction to obtain the problem vector for each data modality With multimodal feature vector The eigenvector of Specifically include:

[0046] Use graph neural network to analyze the interest-related features of each data modality - feature map Perform feature extraction to obtain multimodal feature vectors Feature information ;

[0047] Based on the interaction matrix Integration to obtain feature vector .

[0048] Furthermore, the feature vector of each data modality is obtained based on the attention mechanism The attention weight , and the sum of the attention weights of all data modalities is 1;

[0049] According to the formula Get the second collaboration information .

[0050] Furthermore, the large model output module is constrained to directly output the answer text "This question cannot be answered" for input without generated prompt content.

[0051] Furthermore, it also includes a knowledge graph updating module, which is used to update and adjust the knowledge graph for the target professional field based on the regular evaluation results of the answer text generated by the constraint large model.

[0052] The technical solution provided by this application brings at least the following beneficial effects:

[0053] This application uses a large language model for preprocessing. For content feature vectors of different modes, in each mode, the cosine similarity between feature vectors is calculated to form a similarity matrix. Through the processing or dialogue of the large model, the accuracy of language expression is enhanced, and the actual problem that is easier for the machine to understand and identify is output. Then, the same vectorization processing method as the knowledge base text is used to convert the actual problem into a semantic vector for the next step of directly calculating the vector of the problem with the professional field knowledge base text. According to the input question vector, through the vectorization index retrieval in the professional field knowledge base, several text vectors closest to the question vector are selected from the professional field knowledge base (which can be set according to the number of professional knowledge base texts or the needs of professional field experts), and sorted according to the relevance of the text vectors, so as to obtain several knowledge related to the problem and the source of each piece of knowledge. This application effectively improves the accuracy of knowledge question and answer responses in the target field, ensures the integrity of knowledge question and answer responses in the target field, and realizes basis support, supports localized deployment, solves information security issues, and innovates the intelligent application scenarios of knowledge question and answer retrieval, providing efficient and intelligent solutions for automatic knowledge question and answer responses in specific professional fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0055] Figure 1 A schematic diagram of the structure of a multimodal knowledge question-answering retrieval system for a specific professional field provided in an embodiment of the present application;

[0056] Figure 2 A schematic diagram of the overall processing process of a multimodal knowledge question-answering retrieval system for a specific professional field provided in an embodiment of the present application;

[0057] Figure 3 A schematic diagram of the construction process of a multimodal professional domain knowledge base according to an embodiment of the present application;

[0058] Figure 4 A schematic diagram of the processing flow of the input text preprocessing module of an embodiment of the present application;

[0059] Figure 5 A schematic diagram of the processing flow of the prompt input large model module of an embodiment of the present application;

[0060] Figure 6 A schematic diagram of the processing flow of the constraint large model output module of an embodiment of the present application;

[0061] Figure 7 A schematic diagram of the processing flow of the retrieval basis generating module according to an embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions of the embodiments of the present application will be described in detail and completely in conjunction with the drawings in the embodiments of the present application. Obviously, the embodiments described with reference to the drawings are exemplary and are intended to be used to explain the present application, and cannot be understood as limiting the present application.

[0063] The present application provides a multimodal knowledge question and answer retrieval method for specific professional fields, which solves the problems of single modal information source of large language models in specific professional fields, and inaccurate, incomplete and unfounded knowledge question and answer responses. It is suitable for specific professional knowledge fields, supports localized deployment, and solves information security issues.

[0064] In one embodiment, a multimodal knowledge question and answer retrieval method for a specific professional field provided by the present application includes the following steps:

[0065] Step 1: Construction of a multimodal professional domain knowledge base: A multimodal professional domain knowledge base is constructed in the form of a vector database, and the multimodal professional domain knowledge base is used to store basic information knowledge of different data modalities for the target professional field;

[0066] Step 2, input text preprocessing: preprocess the question text input by the user to obtain the vector form of the question text, that is, the question vector; at the same time, find several prompt contents consisting of "question + professional knowledge" that are closest to the current question vector in the multimodal professional domain knowledge base, and create the source index of each professional knowledge corresponding to the prompt content;

[0067] Step 3, prompt input processing: the prompt content consisting of several "questions + professional knowledge" is used as the input message of the multimodal large language model, and the preliminary original answer text is obtained based on the output of the large language model and transmitted to the constraint large model output module;

[0068] Step 4, Constrained Output: Taking the preliminary original answer text as input, and based on the knowledge graph for the target professional field and the set output constraints, the final output answer text is generated through the large language model and transmitted to the retrieval basis generation module;

[0069] Step 5, generating the retrieval basis step; generating the answer source and basis of the answer text based on the source index of each professional knowledge, and visually outputting the answer text and its answer source and basis to the user.

[0070] In one example, taking the application in the field of discipline inspection and supervision as an example, this application also provides a multimodal knowledge question-answering retrieval system for a specific professional field, which mainly includes five modules, such as Figure 1 As shown, each module is specifically:

[0071] (1) Multimodal professional domain knowledge base. The constructed multimodal professional domain knowledge base uses a vector database to store basic information knowledge of different data modalities for the target professional field. It can dynamically add, delete and modify the text, voice, image and video of policies, regulations, documents, speeches, case reports, etc. in the field of discipline inspection and supervision, so as to form a multimodal knowledge base in the field of discipline inspection and supervision. The professional domain knowledge base is built in the form of a vector database, and the multimodal data is divided into blocks and vectorized. It is combined with a general large language model as an external knowledge base to filter user input, thereby forming prompt words for input into the large language model.

[0072] (2) Input text preprocessing module. When a user asks a question to the system, a question text is first formed. This text can be parsed by a large model to obtain a more accurate description of the question. The question text is then converted into a vector. Through data vectorization, the similarity between the question vector and the content in the professional domain knowledge base is calculated. A query search is performed in the professional domain knowledge base to filter the question text and find the prompt content consisting of several "question + professional knowledge" closest to the question vector. At the same time, an index of the source of relevant professional knowledge is created. That is, the input text preprocessing module is used to preprocess the question text input by the user to obtain the vector form of the question text, namely the question vector, and the vector format of the question vector is consistent with the vector format of the basic information knowledge stored in the multimodal professional domain knowledge base; then, the prompt content consisting of several "question + professional knowledge" closest to the question vector is found in the multimodal professional domain knowledge base.

[0073] (3) Prompt input big model module. The selected prompt contents of several "questions + professional knowledge" are provided as input messages to the multimodal big model. The specific knowledge is placed in the request, and the big model analyzes and processes the knowledge contained in the message, integrates information from different sources and in different forms, and performs joint reasoning and decision-making on this basis, giving full play to the big model's ability to summarize, extract context, and remove duplicates, and obtain a preliminary original answer text.

[0074] (4) Constrained large model output module. Based on the original answer, combined with the knowledge graph of a specific professional field, the specified constraints are introduced to moderately control the output of the large model, form a structured answer, enhance the professional accuracy of the answer, improve the interpretability of the reasoning, and form the final output answer text.

[0075] (5) Generate retrieval basis module. While outputting the generated answer, the module gives the knowledge base matching results or related knowledge based on the multimodal professional knowledge base, clarifies the source and basis of the answer, and provides users with the opportunity to verify the accuracy of the answer and for reference. In other words, the module visually outputs the answer text and its source and basis to the user.

[0076] The framework diagram of a multimodal knowledge question answering retrieval system for a specific professional field provided in an embodiment of the present application is as follows: Figure 2 As shown in the figure, it mainly includes four parts. After the user asks a question, in the first part, information is filtered first, and then pre-processed by the LLM model (large language model), which can enhance the understanding and analysis of the problem, correct the error deviation of the problem, extract the key information of the problem, and transform the user's problem into a practical problem. In the second part, the actual problem is input into the professional knowledge base, and through knowledge retrieval (such as RAG (Retrieval-Augmented Generation) retrieval), the query and matching of professional knowledge is solved to form several combinations of "question + relevant professional knowledge", and the source and basis of professional knowledge are marked. In the third part, several combinations of "question + relevant professional knowledge" are input into LLM, and the knowledge graph of the professional field is introduced as a specified constraint. After LLM processing, the answer is generated. In the fourth part, the answer and the source basis are output together.

[0077] In one embodiment, the construction process of the multimodal professional domain knowledge base of the embodiment of the present application is as follows: Figure 3 As shown in the figure, it mainly presents a way of constructing a professional knowledge base. Taking the professional knowledge base of discipline inspection and supervision as an example, the text, audio, and picture content of policies, regulations, system documents, web pages, etc. related to discipline inspection and supervision are collected to form a local knowledge base. By reading the local knowledge base files, they are converted into feature vectors. That is, the information content is first segmented according to the characteristics of the professional field, and the segmented content is converted into numerical vectors to form a vector database of professional field knowledge, which provides a basis for the next step of vector indexing and similarity. For example, for text knowledge, it is necessary to segment, tag, fragment, and block it, and then perform word embedding processing to obtain the corresponding text vector and store it in the multimodal professional field knowledge base. At the same time, for content feature vectors of different modalities, in each modality, the cosine similarity between feature vectors is calculated to form a similarity matrix. Taking visual feature vectors and text feature vectors as examples, and Represent the similarity matrices of visual feature vectors and text feature vectors respectively.

[0078] In one embodiment, the processing flow of the input text preprocessing module of the embodiment of the present application is as follows: Figure 4 As shown, the question text raised by the user is first processed or communicated through a large model to enhance the accuracy of language expression and output actual questions that are easier for the machine to understand and recognize. Then, the actual question is converted into a semantic vector using the same vectorization processing method as the knowledge base text, which is used for the next step of vector calculation to directly compare the question with the professional knowledge base text.

[0079] In one embodiment, the processing flow of the prompt input large model module of the embodiment of the present application is as follows: Figure 5 As shown, the question data is first converted into a question vector, and then based on the current question vector, a number of text vectors closest to the question vector are selected from the professional knowledge base (which can be set according to the number of professional knowledge base texts or the needs of professional experts) through vectorized index retrieval in the professional knowledge base, and sorted according to the relevance of the text vectors, thereby obtaining a number of knowledge related to the question and the source of each piece of knowledge. The vectorized index retrieval of the prompt input large model module specifically includes:

[0080] By constructing a similarity matrix and , get the similarity = · , the second-order interest feature vector corresponding to the question vector is selected by the following formula:

[0081]

[0082] in, Represents the problem vector The set of closest multi-model feature vectors, Records for each The multimodal feature vector of The similarity score of represents the first k objects filtered out, Represents the similarity matrix The Row, first-order interaction matrix is the set of all interactions between question vectors and multimodal feature vectors, Representing the problem vector set The number of problem vectors included, Represents a set of feature vectors The number of multimodal eigenvectors included, Used to represent the problem vector With multimodal feature vector The interaction status, This means that an interaction has taken place. This means that no interaction has taken place. Characterizing multimodal feature vectors is the problem vector The second-order interaction matrix can be created by the following formula: :

[0083]

[0084] With the first-order interaction matrix Have the same shape. A lightweight graph convolutional network can be used to calculate the collaborative information on the question vector-feature vector interaction graph to obtain the question vector With the eigenvector Collaboration information .

[0085] In order to strengthen the relationship between the question vector and the multimodal feature vector, the interest or relevance of the question vector to the multimodal feature vector is calculated by the following formula:

[0086]

[0087]

[0088] in, Representation and Problem Vector The first-order interest vector with a strong correlation (can be considered as a vector with high similarity). and Select the first k vectors to represent the interest of the question vector:

[0089]

[0090] Construct the interest-related graph using the representative vectors obtained above:

[0091]

[0092]

[0093] in, Representation and Problem Vector A second-order interest vector with a strong correlation.

[0094] In order to maintain the hidden relationship in multimodal features, and Use knn (K-Nearest Neighbors, K nearest neighbors) to sparse:

[0095]

[0096]

[0097] in, Represents the similarity matrix The OK.

[0098] Then the interest-related features of vision and text - feature map are obtained by the following formula:

[0099]

[0100]

[0101] exist and Implementing graph neural network to obtain collaborative information and , integrating them with the interaction matrix to enhance the interest of the question vector to the multimodal feature vector:

[0102]

[0103]

[0104] The interaction matrix , and finally the multimodal representations of all modalities are fused through the attention mechanism:

[0105]

[0106] in, represents the attention weight, represents a linear attention layer, is the output of the linear attention layer.

[0107] From the feature embedding matrix Get problem representation and characterization , that is, based on the feature embedding matrix The corresponding row of the problem representation is , based on the feature embedding matrix The corresponding row of , and then by Calculate the interaction score between the question vector and the feature vector. For each question vector, select the top k feature vectors with the highest scores and their sources as the final hints.

[0108] In one embodiment, the processing flow of the constraint large model output module of the embodiment of the present application is as follows: Figure 6 As shown in the figure, by combining the generated prompt content (prompt 1, prompt 2, ..., prompt n) and the question, the large language model (LLM) is input, and the large model performs reasoning and answering and results integration based on the constraints and processing of the output answer by the knowledge graph in the professional field according to the known information, and generates an answer with professional knowledge, that is, the answer text. At the same time, for the input without generated prompt content, the large model directly processes the question that cannot be answered, instead of making up the answer. In addition, in order to further improve the accuracy of the answer text, the generated answer text set can also be evaluated regularly through expert evaluation, and the knowledge graph can be updated and adjusted accordingly based on the evaluation results. First, if it is found that there is a lack of entity or relationship, the corresponding node can be supplemented and the necessary information (such as basic nodes and relationships) can be supplemented. Secondly, for erroneous or outdated data, the conflicting items should be corrected or merged in time to ensure the accuracy of the graph. Finally, if the evaluation points out that there are defects in the logic or reasoning rules, the reasoning strategy can be modified accordingly or a new relationship type can be defined.

[0109] In one embodiment, the processing flow of the generation retrieval basis module of the embodiment of the present application is as follows: Figure 7 As shown, based on the vector retrieval of the professional field, the text in the professional knowledge base corresponding to the answer is provided as a supporting basis, so as to fully answer the question raised by the user.

[0110] This application uses a large language model for preprocessing. For content feature vectors of different modes, in each mode, the cosine similarity between feature vectors is calculated to form a similarity matrix. Through the processing or dialogue of the large model, the accuracy of language expression is enhanced, and the actual problem that is easier for the machine to understand and identify is output. Then, the same vectorization processing method as the knowledge base text is used to convert the actual problem into a semantic vector for the next step of directly calculating the vector of the problem with the professional field knowledge base text. According to the input question vector, through the vectorization index retrieval in the professional field knowledge base, several text vectors closest to the question vector are selected from the professional field knowledge base (which can be set according to the number of professional knowledge base texts or the needs of professional field experts), and sorted according to the relevance of the text vectors, so as to obtain several knowledge related to the problem and the source of each piece of knowledge. This application effectively improves the accuracy of knowledge question and answer responses in the target field, ensures the integrity of knowledge question and answer responses in the target field, and realizes basis support, supports localized deployment, solves information security issues, and innovates the intelligent application scenarios of knowledge question and answer retrieval, providing efficient and intelligent solutions for automatic knowledge question and answer responses in specific professional fields.

[0111] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0113] The above are only some embodiments of the present application. For those skilled in the art, several modifications and improvements can be made without departing from the creative concept of the present application, and these all belong to the protection scope of the present application.

Claims

1. A multimodal knowledge question-answering retrieval method for a specific professional field, characterized in that: The following steps are involved: Steps for constructing a multimodal professional domain knowledge base: constructing a multimodal professional domain knowledge base in the form of a vector database, which is used to store basic information knowledge of different data modalities for the target professional field; Input text preprocessing step: preprocess the question text input by the user to obtain the vector form of the question text, that is, the question vector; at the same time, find several prompt contents consisting of "question + professional knowledge" that are closest to the current question vector in the multimodal professional domain knowledge base, and create the source index of each professional knowledge corresponding to the prompt content; Prompt input processing steps: using prompt contents consisting of several "questions + professional knowledge" as the input message of the multimodal large language model, obtaining the preliminary original answer text based on the output of the large language model and transmitting it to the constraint large model output module; Constrained output step: Taking the preliminary original answer text as input, and based on the knowledge graph for the target professional field and the set output constraints, the final output answer text is generated through the large language model and transmitted to the retrieval basis generation module; Generate retrieval basis step: Generate the answer source and basis of the answer text based on the source index of each professional knowledge, and visually output the answer text and its answer source and basis to the user; In the input text preprocessing step, the prompt contents consisting of several "question + professional knowledge" closest to the current question vector are found in the multimodal professional domain knowledge base, specifically: Through vectorized index retrieval, the first k closest feature vectors are selected in the multimodal professional domain knowledge base, where k is greater than 1; Based on the correlation between each selected feature vector and the question vector, the selected feature vector is sorted to obtain k pieces of prompt content consisting of "question + professional knowledge" and the source index of each piece of professional knowledge corresponding to each piece of improved content; Among them, selecting the first k closest feature vectors in the multimodal professional domain knowledge base specifically includes: Based on the similarity matrix of all data modalities, the total similarity matrix S is obtained a =S 1 ·S 2 ·…·S M , where S m represents the similarity matrix of the mth data mode, and m=1,2…,M; Select the second-order interest feature vector corresponding to the problem vector u: Among them, SOI_I(u) represents the set of multimodal feature vectors closest to the problem vector u, SOI_W(u,i) is used to record the similarity score of each multimodal feature vector i belonging to SOI_I(u), top_k() represents the first k objects screened out, Represents the total similarity matrix S a The i-th row in the first-order interaction matrix R∈{0,1} |U|×|I| is the set of all interactions between problem vectors and multimodal feature vectors, |U| represents the number of problem vectors included in the problem vector set U, |I| represents the number of multimodal feature vectors included in the feature vector set I, and r u,i ∈{0,1} is used to represent the interaction state between the problem vector u and the multimodal feature vector i, r u,i =1 means interaction has occurred, r u,i =0 means no interaction; Create the second-order interaction matrix R a : The second-order interaction matrix R a As input, the first collaborative information E of the question vector u and the multimodal feature vector i is obtained through the graph convolutional network. ui ; According to the formula Calculate the correlation degree S between the problem vector u and each multimodal feature vector i m (u), where Represents the similarity matrix S of the mth data modality m The i-th row in the u,i =1,r u,i ∈R} represents the first-order interest vector that is strongly associated with the question vector u; In each S m (u) selects the interest vector representing the question vector u: I m (u) = top_k(S m (u)); Constructing interest-related graphs: in, Represents the interest-related graph of the mth data modality The i-th row and j-th column of represents the second-order interest vector that is strongly associated with the question vector u, j′ represents the multimodal feature vector index; The similarity matrix S of each data modality m To perform sparseness: in, Represents the sparse similarity matrix of m data modalities The i-th row and j-th column of Represents the similarity matrix S m The i-th row in ; According to interest related graph With the sparse similarity matrix Construct the interest-related features-feature map of the m-th data modality: For each data modality, the relevant features of interest - feature map G m Perform feature extraction to obtain the feature vector of the problem vector u and the multimodal feature vector i under each data mode Fusion of feature vectors from all modalities Get the second collaborative information of the problem vector u and the multimodal feature vector i Based on the first collaboration information E ui Information about collaboration with the second Get the feature embedding matrix E, and get the problem representation based on the corresponding rows of the feature embedding matrix E The corresponding columns of the feature embedding matrix E get the feature representation According to Calculate the interaction score between the question vector u and the multimodal feature vector i; For each question vector u, the first k multimodal feature vectors i with the highest scores are selected as the first k closest feature vectors.

2. A multimodal knowledge question-answering retrieval system for a specific professional field, characterized in that: include: Multimodal professional domain knowledge base, input text preprocessing module, prompt input large model module, constraint large model output module and retrieval basis generation module; in, The multimodal professional domain knowledge base uses a vector database to store basic information knowledge of different data modalities for the target professional domain; The input text preprocessing module is used to preprocess the question text input by the user to obtain the vector form of the question text, that is, the question vector; at the same time, find out the prompt contents composed of several "question + professional knowledge" closest to the current question vector in the multimodal professional field knowledge base, and create the source index of each professional knowledge corresponding to the prompt content; and transmit the prompt contents composed of several "question + professional knowledge" to the prompt input large model module, and transmit the source index of each professional knowledge corresponding to the prompt content to the generation retrieval basis module; The prompt input large model module uses the prompt content consisting of several "questions + professional knowledge" as the input message of the multimodal large language model, obtains the preliminary original answer text based on the output of the large language model and transmits it to the constraint large model output module; The constrained large model output module takes the preliminary original answer text as input, and based on the knowledge graph oriented to the target professional field and the set output constraints, generates the final output answer text through the large language model and transmits it to the retrieval basis generation module; Generate a retrieval basis module, which generates the answer source and basis of the answer text based on the source index of each professional knowledge, and visually outputs the answer text and its answer source and basis to the user; Among them, the input text preprocessing module finds several prompt contents consisting of "question + professional knowledge" that are closest to the current question vector in the multimodal professional field knowledge base, and the specific contents are: Through vectorized index retrieval, the first k closest feature vectors are selected in the multimodal professional domain knowledge base, where k is greater than 1; Based on the correlation between each selected feature vector and the question vector, they are sorted to obtain k prompt contents consisting of "question + professional knowledge" and the source index of each professional knowledge corresponding to each enhanced content. Among them, the multimodal professional domain knowledge base is retrieved through vectorized index, and the first k closest feature vectors are selected in the multimodal professional domain knowledge base, specifically including: Based on the similarity matrix of all data modalities, the total similarity matrix S is obtained a =S 1 ·S 2 ·…·S M , where S m represents the similarity matrix of the mth data mode, and m=1,2…,M; Select the second-order interest feature vector corresponding to the problem vector u: Among them, SOI_I(u) represents the set of multimodal feature vectors closest to the problem vector u, SOI_W(u,i) is used to record the similarity score of each multimodal feature vector i belonging to SOI_I(u), top_k() represents the first k objects screened out, Represents the total similarity matrix S a The i-th row in the first-order interaction matrix R∈{0,1} |U|×|I| is the set of all interactions between problem vectors and multimodal feature vectors, |U| represents the number of problem vectors included in the problem vector set U, |I| represents the number of multimodal feature vectors included in the feature vector set I, and r u,i ∈{0,1} is used to represent the interaction state between the problem vector u and the multimodal feature vector i, r u,i =1 means interaction has occurred, r u,i =0 means no interaction; Create the second-order interaction matrix R a : The second-order interaction matrix R a As input, the first collaborative information E of the question vector u and the multimodal feature vector i is obtained through the graph convolutional network. ui ; According to the formula Calculate the correlation degree S between the problem vector u and each multimodal feature vector i m (u), where Represents the similarity matrix S of the mth data modality m The i-th row in the u,i =1,r u,i ∈R} represents the first-order interest vector that is strongly associated with the question vector u; In each S m (u) selects the interest vector representing the question vector u: I m (u) = top_k(S m (u)); Constructing interest-related graphs: in, Represents the interest-related graph of the mth data modality The i-th row and j-th column of represents the second-order interest vector that is strongly associated with the question vector u, j′ represents the multimodal feature vector index; The similarity matrix S of each data modality m To perform sparseness: in, Represents the sparse similarity matrix of m data modalities The i-th row and j-th column of Represents the similarity matrix S m The i-th row in ; According to interest related graph With the sparse similarity matrix Construct the interest-related features-feature map of the m-th data modality: For each data modality, the relevant features of interest - feature map G m Perform feature extraction to obtain the feature vector of the problem vector u and the multimodal feature vector i under each data mode Fusion of feature vectors from all modalities Get the second collaborative information of the problem vector u and the multimodal feature vector i Based on the first collaboration information E ui Information about collaboration with the second Get the feature embedding matrix E, and get the problem representation based on the corresponding rows of the feature embedding matrix E The corresponding columns of the feature embedding matrix E get the feature representation According to Calculate the interaction score between the question vector u and the multimodal feature vector i; For each question vector u, the first k multimodal feature vectors i with the highest scores are selected as the first k closest feature vectors.

3. A multimodal knowledge question-answering retrieval system for a specific professional field as claimed in claim 2, characterized in that: Basic information knowledge includes policies, regulations, documents, systems, speeches and / or case reports for the target professional field.

4. A multimodal knowledge question-answering retrieval system for a specific professional field as claimed in claim 2, characterized in that: The multimodal professional domain knowledge base also includes a similarity matrix for each data modality. The similarity matrix is ​​constructed by performing content segmentation on the basic information under the current data modality, and then converting the segmented single content into a feature vector to obtain multiple numerical vectors of the basic information under the current data modality and storing them in the multimodal professional domain knowledge base; In each mode, the cosine similarity between each feature vector is calculated to form a similarity matrix for the current data mode.

5. A multimodal knowledge question-answering retrieval system for a specific professional field as claimed in claim 2, characterized in that: For each data modality, the relevant features of interest - feature map G m Perform feature extraction to obtain the feature vector of the problem vector u and the multimodal feature vector i under each data mode Specifically include: The graph neural network is used to analyze the interest-related features of each data modality - feature map G m Perform feature extraction to obtain the feature information E of the multimodal feature vector i im ; And based on the interaction matrix G ui =R+R a Integration to obtain feature vector 6. A multimodal knowledge question-answering retrieval system for a specific professional field as claimed in claim 2, characterized in that: Obtain feature vectors for each data modality based on attention mechanism The attention weight α m , and the sum of the attention weights of all data modalities is 1; According to the formula Get the second collaboration information 7. A multimodal knowledge question-answering retrieval system for a specific professional field as claimed in claim 2, characterized in that: The constraint large model output module directly outputs the answer text "Unable to answer this question" for input without generated prompt content.

8. A multimodal knowledge question-answering retrieval system for a specific professional field as claimed in claim 2, characterized in that: It also includes a knowledge graph updating module, which is used to update and adjust the knowledge graph for the target professional field based on the regular evaluation results of the answer text generated by the constraint large model.

Images