Question answering system construction method and system based on large language model

Abstract

The invention provides a question and answer system construction method and system based on a large language model, and the method comprises the steps: obtaining multi-modal data, constructing a question and answer knowledge base and a knowledge graph, and carrying out the dynamic updating of the question and answer knowledge base; obtaining a query text, and respectively carrying out vectorization processing on the query text and the multi-modal data to generate a corresponding query semantic vector and a multi-modal vector; an entity in the query text is extracted by using the recognition model, a triple associated with the entity is extracted from the knowledge graph, the query text and the triple are spliced and vectorized, and a query semantic enhancement vector is generated; according to the method, through a dynamic knowledge base incremental updating mechanism, a context-aware hybrid retrieval strategy, a cross-modal semantic enhancement technology and a user feedback-driven continuous optimization method, real-time processing requirements of various modal data such as texts, images and voices can be effectively met, and accurate semantic understanding and answer generation of complex queries are achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of question and answer systems, and particularly relates to a large language model-based question and answer system construction method and system. BACKGROUND

[0002] As a core technology of human-computer interaction, question and answer systems have been widely used in intelligent customer service, medical consultation, financial analysis and other fields in recent years. A question and answer system is a high-level form of an information retrieval system, which can accurately and concisely answer a user's question in natural language. A question and answer system analyzes a user's question through natural language understanding technology, then generates an answer using information in a knowledge base or database, and returns the answer to the user in the form of natural language through natural language generation technology.

[0003] A knowledge base question and answer construction method and device based on a large model are disclosed in CN118035405A. The method includes: constructing a knowledge database; constructing a knowledge vector library using the knowledge database; obtaining a user question, matching the semantic vector corresponding to the user question with the content of the knowledge vector library; merging the text paragraphs of the matching corresponding knowledge vector library and the user question to generate inference information; and using a pre-configured large language model to generate a corresponding answer to the inference information.

[0004] Existing question and answer systems have problems such as insufficient real-time performance and weak cross-modal semantic understanding capability when processing multi-modal data, making it difficult to accurately analyze the deep intention of complex queries and resulting in low answer accuracy. In addition, dynamic knowledge updating is lagging behind, and the retrieval strategy is single, further limiting the effective integration of timely knowledge and multi-source information, which cannot meet the user's demand for accurate answers in high-dynamic scenarios and reduces the answer accuracy. SUMMARY

[0005] Therefore, the present application provides a large language model-based question and answer system construction method and system, which can effectively adapt to the real-time processing needs of text, image, voice and other multi-modal data, accurately understand the semantics of complex queries and generate answers, and improve the answer accuracy.

[0006] The technical solution of the present application is as follows: In a first aspect, the present application provides a large language model-based question and answer system construction method, including the following steps: S1, obtaining multi-modal data, constructing a question and answer knowledge base and a knowledge graph, and dynamically updating the question and answer knowledge base; S2, obtaining a query text, and performing vectorization processing on the query text and the multi-modal data respectively to generate corresponding query semantic vectors and multi-modal vectors; S3, using the recognition model to extract entities in the query text, and extracting triples associated with the entities from the knowledge graph, splicing the query text and the triples and vectorizing to generate a query semantic enhancement vector; S4, combining the keyword matching algorithm and the similarity retrieval algorithm to retrieve the question and answer knowledge base, respectively calculating the similarity and matching degree of the query semantic enhancement vector and the multi-modal vector, dynamically weighting the similarity and matching degree, and obtaining the text segment with the highest score; S5, splicing the query text and the text segment with the highest score, using a pre-trained language model to encode the spliced text, outputting the semantic vector of each word, and respectively obtaining a query text semantic vector matrix and a document semantic vector matrix; S6, according to the query text semantic vector matrix and the document semantic vector matrix, obtaining the attention weight of each word of the query text and the text segment through the attention mechanism, and fusing the query text semantic vector matrix and the document semantic vector matrix to obtain a fused semantic vector; S7, obtaining words in the text segment that are greater than a preset word frequency and attention weight threshold, and based on the fused semantic vector, using a GPT-2 model to generate natural language, adding the extracted words to the natural language, using a beam search algorithm to generate a coherent sentence, and outputting a natural language answer text.

[0007] On the basis of the above technical scheme, preferably, the step S1 includes the following sub-steps of obtaining multi-modal data, constructing a question and answer knowledge base and a knowledge graph containing entity associated triples, and dynamically updating the question and answer knowledge base: S11, obtaining multi-modal data from documents, databases, API interfaces and web pages, and performing data cleaning on the multi-modal data to obtain standard sample data, constructing a question and answer knowledge base according to the standard sample data, and constructing a knowledge graph containing entity associated triples according to entity-relation-property; S12, dynamically updating the question and answer knowledge base, judging whether the standard sample data has version update time information, if there is, then according to the update time, obtaining the corresponding update data and the to-be-updated data, positioning the added content interval between the update data and the to-be-updated data through binary search, updating the version difference segment, and obtaining the updated standard sample data; If not, calculate the hash value of the data block of the update data and the to-be-updated data, filter the difference block segment of the update data and the to-be-updated data, compare the semantic similarity of the difference block segment, obtain the similarity difference value, and preset a similarity update threshold, when the similarity difference value is greater than the similarity update threshold, update the difference block segment of the to-be-updated data.

[0008] Based on the above technical solutions, preferably, in step S2, the query text is obtained, and the query text and the multi-modal data are respectively subjected to vectorization processing to generate corresponding query semantic vectors and multi-modal vectors, including the following steps: S21, obtaining query text and text-image-table multi-modal data; S22, performing vectorization processing on the text data and the query text by using a pre-trained BERT language model to generate text semantic vectors from the text data and query semantic vectors from the query text; S23, performing vectorization processing on the image data by using a pre-trained ResNet visual model to extract image features to generate image feature vectors; S24, identifying table data boundaries and cells, determining header rows and data rows, establishing a row-column index structure, embedding and combining table header and cell contents to obtain table semantic vectors.

[0009] Based on the above technical solutions, preferably, in step S3, the identified entities in the query text are extracted using a recognition model, and triplets associated with the entities are extracted from a knowledge graph, the query text and the triplets are spliced and subjected to vectorization processing to generate query semantic enhancement vectors, including the following steps: S31, identifying entities in the query text by using a BLINK model, linking the entities to a knowledge graph, and extracting triplets associated with the entities from the knowledge graph; S32, splicing the extracted triplets, and inputting the spliced text into a pre-trained BERT language model to generate triplet vectors; S33, fusing the triplet vectors and the query semantic vectors by using an MLP multi-layer perceptron combined with a first gating coefficient to output query semantic enhancement vectors; S34, calculating the cosine similarity of the query semantic vectors and the triplet vectors to obtain a relevance score result, and dynamically adjusting the first gating coefficient based on the relevance score result.

[0010] Based on the above technical solutions, preferably, in step S4, the query semantic enhancement vectors and the multi-modal vectors are respectively calculated for similarity and matching degree by using a keyword matching algorithm and a similarity retrieval algorithm, the similarity and the matching degree are dynamically weighted, the text segment with the highest score is obtained, including the following sub-steps: S41, performing dependency syntax analysis on the query text by using a StanfordCoreNLP tool to generate a syntax tree and calculate the maximum syntax depth; S42, obtaining the ratio between the number of entity words in the query text and the total number of entities in the knowledge graph to obtain an entity density; S43, initialize the weight proportion of the BM25 keyword matching algorithm and the similarity retrieval algorithm, calculate the dynamic weights of the BM25 keyword matching algorithm and the similarity retrieval algorithm according to the maximum syntactic depth, entity density and query complexity of the current query text, and the query complexity is calculated by the syntactic tree depth and the entity density; S44, retrieve the question and answer knowledge base based on the BM25 keyword matching algorithm, calculate the matching degree score of the query text and the text segment, retrieve the question and answer knowledge base according to the similarity retrieval algorithm, calculate the cosine similarity of the semantic enhancement vector and the multi-modal vector, calculate the sum score of the matching degree score and the cosine similarity according to the dynamic weights of the BM25 keyword matching algorithm and the similarity retrieval algorithm, and sort according to the final sum score to obtain the text segment with the highest score as the retrieval result.

[0011] On the basis of the above technical scheme, preferably, in step S5, the query text and the text segment with the highest score retrieved are spliced, a pre-trained language model is used to encode the spliced text, the semantic vector of each word is output, and the query text semantic vector matrix and the document semantic vector matrix are obtained respectively, including the following substeps: S51, splice the user query text and the text segment with the highest score retrieved according to the following format, and use the pre-trained RoBERTa-large model to encode the spliced text to output the semantic vector of each word; S52, determine whether the spliced text exceeds the maximum input length of the RoBERTa-large model, if yes, keep the head and tail words according to half of the maximum input length, and truncate the middle part; S53, construct the query text semantic vector matrix according to the semantic vector of the corresponding word of the query text, and construct the document semantic vector matrix according to the semantic vector of the corresponding word of the text segment; S54, if the number of retrieved text segments is more than one, perform mean pooling on the semantic vector matrix of each text segment to obtain the comprehensive document semantic vector.

[0012] On the basis of the above technical scheme, preferably, in step S6, the attention weight of each word of the query text and the text segment is obtained through the attention mechanism according to the query text semantic vector matrix and the document semantic vector matrix, and the query text semantic vector matrix and the document semantic vector matrix are fused to obtain the fusion semantic vector, including the following substeps: S61, calculate the attention weight of each word of the query text and the text segment through the attention mechanism according to the query text semantic vector matrix and the document semantic vector matrix; S62, fuse the query text semantic vector matrix and the document semantic vector matrix, and dynamically adjust the contribution proportion of the query text and the text segment through a second gating coefficient, to calculate and generate a fused semantic vector.

[0013] On the basis of the above technical solutions, preferably, in step S7, the words in the text segment greater than the preset word frequency and attention weight threshold are obtained, and based on the fused semantic vector, a GPT-2 model is used to generate natural language, the extracted words are added to the natural language, a beam search algorithm is used to generate a coherent sentence, and a natural language answer text is output. The preset word frequency threshold and the attention weight threshold are used to filter and obtain the words in the text segment greater than the preset word frequency and the attention weight threshold, and the filtered words are added to the generated word list of the GPT-2; According to the current context and the fused semantic vector, the generation probability of each word is calculated through the GPT-2 model; According to the attention weight and the word frequency, the replication probability of each word is calculated; According to the generation probability and the replication probability of each word and the weighted calculation, the candidate probability of each word is obtained; Based on the candidate probability of each word and using the beam search algorithm, a natural language answer is output.

[0014] In a second aspect, the present application also provides a large language model-based question answering system construction system, which is implemented by using the large language model-based question answering system construction method, and comprises: A knowledge base construction module is configured to acquire multi-modal data, construct a question answering knowledge base and a knowledge graph comprising entity association triples, and dynamically update the question answering knowledge base; A data processing module is configured to acquire a query text, and perform vectorization processing on the query text and the multi-modal data respectively to generate corresponding query semantic vectors and multi-modal vectors; A semantic increasing module is configured to use an identification model to extract entities in the query text, extract triples associated with the entities from the knowledge graph, concatenate the query text and the triples, and perform vectorization processing to generate query semantic enhancement vectors; A retrieval matching module is configured to combine a keyword matching algorithm and a similarity retrieval algorithm to retrieve the question answering knowledge base, calculate the similarity and the matching degree of the query semantic enhancement vectors and the multi-modal vectors respectively, dynamically weight the similarity and the matching degree, and acquire a text segment with the highest score; An encoding processing module is configured to concatenate the query text and the text segment with the highest score, use a pre-trained language model to encode the concatenated text, output semantic vectors of each word, and respectively obtain a query text semantic vector matrix and a document semantic vector matrix; The attention module is used to obtain the attention weight of each vocabulary of the query text and the text segment according to the query text semantic vector matrix and the document semantic vector matrix through an attention mechanism, and the query text semantic vector matrix and the document semantic vector matrix are fused to obtain a fused semantic vector. The generation module is used to obtain the vocabulary in the text segment that is greater than a preset word frequency and attention weight threshold, generate natural language based on the fused semantic vector by using a GPT-2 model, add the extracted vocabulary to the natural language, generate a coherent sentence by using a beam search algorithm, and output a natural language answer text.

[0015] The large language model-based question and answer system construction method and system have the following beneficial effects compared with the prior art: (1) Through the dynamic knowledge base incremental updating mechanism, the mixed retrieval strategy of context perception, the cross-modal semantic enhancement technology and the user feedback driven continuous optimization method, the real-time processing demand of text, image, voice and other multi-modal data can be effectively adapted, and the precise semantic understanding and answer generation of complex queries can be realized; (2) A high-time-efficiency question and answer knowledge base and knowledge graph are constructed through a multi-source data fusion and dynamic updating mechanism, and an abnormal fuse, a distributed transaction lock and a multi-mirror source retry mechanism are introduced to ensure the updating stability, ensure the knowledge timeliness, and improve the response ability of the question and answer system to dynamic knowledge and the answer reliability; (3) The query complexity is quantified by syntax analysis, a query complexity model is constructed in combination with entity density and text length, a Sigmoid function with learnable parameters is introduced to dynamically adjust the weights of BM25 and semantic similarity, so that the system can adaptively allocate algorithm weights according to the query characteristics, avoid the matching deviation caused by traditional static weights, and improve the recall rate and accuracy of retrieval. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0017] Figure 1 The flowchart of the large language model-based question and answer system construction method of the present application; Figure 2 The dynamic context window adjustment schematic diagram of the large language model-based question and answer system construction method of the present application; Figure 3 The multi-round dialogue performance comparison line graph of the large language model-based question and answer system construction method of the present application. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present application will be clearly and completely described in combination with the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0019] As shown in Figures 1-3 A method for constructing a large language model-based question answering system according to the present application comprises the following steps: S1, acquiring multi-modal data, constructing a question answering knowledge base and a knowledge graph, and dynamically updating the question answering knowledge base.

[0020] In the step S1 in the present embodiment, the following sub-steps are included: S11, acquiring multi-modal data from documents, databases, API interfaces and web pages, and performing data cleaning on the multi-modal data to obtain standard sample data, constructing a question answering knowledge base according to the standard sample data, and constructing a knowledge graph containing entity association triplets according to entities-relationships-attributes; It should be noted that knowledge data is collected in real time from channels such as documents, databases, API interfaces and web pages, redundant data is cleaned by using regular expression matching and semantic disambiguation technology, and repeated, expired or invalid data is removed through data cleaning to ensure the accuracy and timeliness of the knowledge base content; the cleaned data is stored in a structured question answering knowledge base for subsequent retrieval and updating; and NLP technology is used to extract entities, relationships and attributes to construct triplets and establish a knowledge graph.

[0021] S12, dynamically updating the question answering knowledge base, judging whether the standard sample data has version update time information, if yes, acquiring corresponding update data and to-be-updated data according to the update time, positioning the added content interval between the update data and the to-be-updated data through binary search, updating the version difference segment, and obtaining the updated standard sample data; If not, calculate the hash values of the update data and the to-be-updated data by data block, screen the difference block segment of the update data and the to-be-updated data, perform semantic similarity calculation and comparison on the difference block segment, obtain a similarity difference value, preset a similarity update threshold, and when the similarity difference value is greater than the similarity update threshold, update the difference block segment of the to-be-updated data.

[0022] It should be noted that if the data source provides version timestamp, the incremental data is directly obtained according to the time interval, the difference segment is quickly located through binary search, the version difference segment is updated, and the updated standard sample data is obtained; if there is no timestamp, the data is divided into fixed size blocks, each block is 512 bytes, the SHA-256 hash value of each block is calculated, the hash fingerprint is generated, the hash blocks of the update data and the data to be updated are compared, the difference blocks are screened, the cosine similarity of the difference blocks is calculated by using the pre-trained Sentence-BERT model to avoid hash collision misjudgment, only the difference segment is replaced in the update strategy, the calculation resource consumption is reduced, and an abnormal fuse mechanism is implemented in the update process. When the node processing fails, it is automatically switched to the standby data mirror source for retry, and the distributed transaction lock is used to ensure data consistency. The cleaned data is stored according to the ontology model, a three-level traceability system of version-branch-commit is established, the retrieval priority is dynamically adjusted according to the data credibility, and if the knowledge base after updating is abnormal, it can be rolled back to the last stable version.

[0023] S2, obtaining the query text, and performing vectorization processing on the query text and the multi-modal data respectively to generate corresponding query semantic vectors and multi-modal vectors.

[0024] In this embodiment, step S2 includes the following steps: S21, obtaining the query text and the text-image-table multi-modal data; S22, performing vectorization processing on the text data and the query text by using a pre-trained BERT language model, generating text semantic vectors from the text data, and generating query semantic vectors from the query text; the expression is:

[0025] In the formula, V text is the query semantic vector, the dimension is the hidden layer dimension of BERT, BERT(q) [CLS] is the output vector of the CLS mark of the BERT model for the input text, and a sliding window is used for division when the input text is too long, wherein the window size is 128 and the overlap rate is 30%; S23, performing vectorization processing on the image data by using a pre-trained ResNet visual model, extracting image features to generate image feature vectors; the expression is:

[0026] In the formula, v image is the image feature vector; the global feature adopts the GAP output of the last layer of ResNet-50 with a dimension of 2048, and the Top-50 regional features are extracted by FasterR-CNN, and the dimension is reduced to 2048 by MaxPool.

[0027] S24, identify the table data boundary and cell, determine the header row and data row; establish row and column index structure, respectively embed and combine the header and cell content, and obtain the table semantic vector; the expression is:

[0028] In the formula, v table is the table semantic vector, is the vector splicing, MLP(·) is the multi-layer perception machine, Embed( H i ) is the embedding vector of the header H i , and Embed( C ij ) is the embedding vector of the cell C ij . It should be noted that the numerical and text field types are distinguished, and the embedding matrix is initialized respectively. Meanwhile, the numerical cells are embedded after Z-Score standardization.

[0029] S3, the entity in the query text is extracted using the recognition model, and the triple associated with the entity is extracted from the knowledge graph. The query text and the triple are spliced and vectorized to generate the query semantic enhancement vector.

[0030] In the embodiment, step S3 includes the following steps: S31, the entity in the query text is recognized and obtained using the BLINK model, the entity is linked to the knowledge graph, and the triple associated with the entity is extracted from the knowledge graph; the expression is:

[0031] In the formula, Expand(q) is the result set after knowledge expansion of the query text q , Entities ( q ) is the entity set recognized from the query text q , h is the head entity in the knowledge graph, r is the relationship in the knowledge graph, t is the tail entity in the knowledge graph, R c is the candidate relationship set; S32, the extracted triple is spliced, and the spliced text is input into the pre-trained BERT language model to generate a triple vector; S33, the triple vector and the query semantic vector are fused by the MLP multi-layer perception machine combined with the first gating coefficient, and the query semantic enhancement vector is output; the expression is:

[0032] wherein Eenh(q) is a query semantic enhancement vector, λ is a first gating coefficient, E(q) is a query semantic vector, and MLP(·) is a multi-layer perceptron.

[0033] S34, calculating a cosine similarity between the query semantic vector and the triple vector to obtain a relevance score result, expressed as:

[0034] wherein Score(q ， hrt ) is the relevance score result. dynamically adjusting the first gating coefficient based on the relevance score result, expressed as: .

[0035] It should be noted that the precise entity recognition and knowledge graph association based on the BLINK model effectively supplements the implicit semantics of the query; the structured triples are converted into semantic vectors by BERT vectorization, retaining the context association features; and the dynamic fusion mechanism of the MLP and the first gating coefficient can adaptively adjust the weight of the original query and the expanded knowledge according to the query complexity, avoiding information redundancy or loss; finally, the feedback mechanism based on the cosine similarity optimizes the gating parameters in real time, so that the system can generate an enhanced semantic representation that combines the original intent and domain knowledge, improving the recall rate and accuracy of the retrieval.

[0036] S4, retrieving the question and answer knowledge base by combining the keyword matching algorithm and the similarity retrieval algorithm, calculating the similarity and matching degree of the query semantic enhancement vector and the multi-modal vector respectively, and dynamically weighting the similarity and matching degree to obtain the text segment with the highest score.

[0037] In this embodiment, step S4 includes the following sub-steps: S41, performing dependency syntax analysis on the query text using the StanfordCoreNLP tool to generate a syntax tree and calculate the maximum syntax depth thereof; S42, obtaining the ratio between the number of entity words in the query text and the total number of entities in the knowledge graph to obtain an entity density; S43, initializing the weight ratio of the BM25 keyword matching algorithm and the similarity retrieval algorithm, calculating the dynamic weight of the BM25 keyword matching algorithm and the similarity retrieval algorithm according to the maximum syntax depth, the entity density and the query complexity of the current query text, expressed as:

[0038] where ω is a weight coefficient, σ is a Sigmoid function, α and β are learnable parameters, and Complexity(q) is query complexity. The query complexity is calculated by the syntax tree depth and entity density, and the expression is:

[0039] where Depth(q) is the syntax tree depth, EntityCount(q) is the entity density, and Length(q) is the text length of the query text. q S44, based on the BM25 keyword matching algorithm, retrieves the question and answer knowledge base, calculates the matching degree score of the query text and the text segment, retrieves the question and answer knowledge base according to the similarity retrieval algorithm, calculates the cosine similarity of the semantic enhancement vector and the multi-modal vector, calculates the sum score of the matching degree score and the cosine similarity according to the dynamic weight of the BM25 keyword matching algorithm and the similarity retrieval algorithm, sorts according to the final sum score, and obtains the text segment with the highest score as the retrieval result, and the expression is: .

[0040] where Sim(q,d) is the sum score of the matching degree score and the cosine similarity, BM25(q,d) is the matching degree score calculated based on the BM25 algorithm, and cos(E(q),E(d)) is the cosine similarity score calculated by the similarity retrieval algorithm.

[0041] It should be noted that the syntax analysis based on StanfordCoreNLP quantifies the query structure complexity, combines the entity density and the text length to construct the query complexity model, and introduces the Sigmoid function of the learnable parameter to dynamically adjust the weight of BM25 and semantic similarity, so that the system can adaptively allocate the algorithm weight according to the query characteristics, avoiding the matching deviation caused by the traditional static weight; improve the recall rate and accuracy of retrieval.

[0042] In addition, step S43 also includes randomly extracting 1000 queries from historical queries, determining the optimal weight combination of each query through manual annotation, randomly generating 50 groups of weight parameters, calculating the average absolute error of each group of parameters on the validation set to evaluate the model performance, and dynamically adjusting the learnable parameters α and β optimization, until the error change is less than 0.0001 or reaches 200 iterations, and the expression is:

[0043] where N is the total number of validation set samples, is the predicted weight of the i-th i sample, is the predicted weight of the i-th i ​The true weights of the samples are determined by a grid search on the validation set.

[0044] Furthermore, during the retrieval process, the dimension of the original vector is... v ∈ R D Divide into multiple sub-vector matrices uniformly For each sub-vector matrix, k-means clustering is performed on the sample data to generate multiple cluster center vectors. In the formula, c i,k For the first i The codebook for each subspace contains k There are 1 cluster center vector; during compressed storage, the index of the nearest neighbor cluster center vector in the corresponding codebook is recorded for each sub-vector. During inspection, the approximate distance is calculated by looking up a table, and the expression is:

[0045] In the formula, q i For the first i A query semantic vector, c i,idx(vi) for v i The corresponding cluster center vector; In this embodiment, product quantization technology is used to decompose a high-dimensional vector into multiple low-dimensional subspaces, independently cluster them to generate codebooks, and record the most recent codeword index, thereby achieving efficient vector compression and fast retrieval.

[0046] S5 concatenates the query text and the highest-scoring text segment, encodes the concatenated text using a pre-trained language model, and outputs the semantic vector of each word, thus obtaining the semantic vector matrix of the query text and the semantic vector matrix of the document.

[0047] Step S5 in this embodiment includes the following sub-steps: S51, the user query text and the highest-scoring text fragment are concatenated in the following format, and the pre-trained RoBERTa-large model is used to encode the concatenated text, outputting the semantic vector of each word; S52, determine whether the concatenated text words exceed the maximum input length of the RoBERTa-large model. If they do, retain the first and last words according to half of the maximum input length, and truncate the middle part. S53, construct a semantic vector matrix for the query text based on the semantic vectors of the words corresponding to the query text, and construct a semantic vector matrix for the document based on the semantic vectors of the words corresponding to the text fragments; S54, if the number of retrieved text segments is more than one, performing mean pooling on the semantic vector matrix of each text segment to obtain a comprehensive document semantic vector.

[0048] It should be noted that the RoBERTa-large pre-training model is used, the maximum input length is 512 words, and the super-long text truncation strategy is to reserve 256 words at the beginning and end and then perform joint encoding.

[0049] S6, obtaining the attention weight of each word of the query text and the text segment according to the query text semantic vector matrix and the document semantic vector matrix through the attention mechanism, and fusing the query text semantic vector matrix and the document semantic vector matrix to obtain a fused semantic vector.

[0050] The step S6 in the embodiment includes the following sub-steps: S61, calculating the attention weight of each word of the query text and the text segment according to the query text semantic vector matrix and the document semantic vector matrix through the attention mechanism, and the expression is:

[0051] In the formula, Q is the encoding of the query text, K / V is the encoding of the text segment; S62, fusing the query text semantic vector matrix and the document semantic vector matrix, and dynamically adjusting the contribution ratio of the query text and the text segment through a second gating coefficient to calculate and generate a fused semantic vector, and the expression is:

[0052] In the formula, H final is the fused semantic vector, g is a learnable second gating coefficient, and dynamically controls the contribution ratio of the query text and the text segment, H query is the query text semantic vector matrix, H doc is the document semantic vector matrix.

[0053] It should be noted that through the attention mechanism and the gating fusion strategy, the deep semantic interaction and dynamic weight distribution of the query text and the document text segment are realized, and the precision, model adaptability and calculation efficiency of the semantic matching are improved.

[0054] S7, obtaining the words in the text segment that are greater than the preset word frequency and attention weight threshold, and based on the fused semantic vector, using the GPT-2 model to generate natural language, adding the extracted words to the natural language, using the beam search algorithm to generate a coherent sentence, and outputting to obtain a natural language answer text.

[0055] Step S7 in this embodiment includes the following sub-steps: A preset word frequency threshold and an attention weight threshold are set, words in the obtained text segment greater than the preset word frequency and attention weight threshold are screened, and the screened words are added to the generated vocabulary table of GPT-2; According to the current context and the fused semantic vector, the generation probability of each word is calculated through the GPT-2 model; According to the attention weight and the word frequency, the replication probability of each word is calculated; According to the generation probability and the replication probability of each word and the weighted calculation, the candidate probability of each word is obtained; the expression is:

[0056] In the formula, P final (ω) is the overall generation probability of the model for the word ω, λ1 is the replication probability weight coefficient, P copy (ω) is the replication probability, P generate (ω) is the generation probability.

[0057] Based on the candidate probability of each word and using the beam search algorithm, the natural language answer is output.

[0058] It should be noted that by combining the word screening based on word frequency and attention weight, the dynamic fusion of generation and replication probability, and the optimization of the beam search algorithm, the accuracy, coherence and information richness of the natural language answer are improved.

[0059] The embodiment also includes supporting multi-round context-aware dialogue, and adaptively optimizing the retrieval and generation strategy based on historical interaction.

[0060] The initial window length W base is set as the basic reserved content; According to the dialogue complexity and the historical relevance, an expansion coefficient is calculated to dynamically adjust the final window size, and the expression is:

[0061] Wherein, W max is the maximum allowed window length of the model, Δ is the expansion step, a is the expansion coefficient.

[0062] Specifically, the initial setting is to reserve the last 3 rounds of dialogue history, with a maximum length of 512 tokens per round, and the memory capacity is realized by a ring buffer, covering typical short dialogue scenarios.

[0063] The expansion trigger condition is set: entity density > 40%, calculated according to the ratio of the number of current dialogue entities to the total number of words; topic jump frequency > 2 times, judged according to the similarity of adjacent rounds of topics < 0.5.

[0064] The system dynamically determines the number of historical dialogue turns that need to be retained by analyzing the complexity and topic consistency of the dialogue, and the expansion rate coefficient is used to control the sensitivity of window expansion, and experience shows that 0.6 can balance the response speed and information integrity; the cumulative entity number counts the number of domain entities mentioned in the current dialogue, and the more entities, the more complex the topic; the topic similarity variance calculates the topic similarity between adjacent dialogue turns, and the larger the variance, the stronger the topic jump.

[0065] At the same time, the system adjusts the context window length based on the calculation result of the expansion coefficient according to the following strategy: the basic window retains the last 3 rounds of dialogue by default, covering the context demand of short dialogue; each expansion increases 2 rounds of historical dialogue to avoid frequent small adjustments; prevent memory overload, and retain up to 8 rounds of dialogue.

[0066] The accuracy and response efficiency of the scheme in the multi-turn dialogue scene are significantly better than those of the traditional scheme, and the performance comparison line chart is shown in Figure 3 The accuracy and coherence of the system in the multi-turn dialogue scene are significantly improved, and the experiment shows that the first round answer F1 value reaches 89.7%, the multi-turn dialogue accuracy is improved by 37.2%, and the response delay is reduced to within 500ms through quantitative compression and retrieval optimization. In the intelligent government affairs, industrial diagnosis, multi-language customer service and other scenes, it shows high efficient knowledge iteration ability and strong robustness, and the error rate is reduced by 62% compared with the traditional scheme, which has broad commercial application prospect.

[0067] In a second aspect, the present application also provides a large language model question and answer system construction system, which is realized by using a large language model question and answer system construction method, comprising: A knowledge base construction module is used to acquire multi-modal data, construct a question and answer knowledge base and a knowledge graph containing entity association triples, and dynamically update the question and answer knowledge base; A data processing module is used to acquire query text and perform vectorization processing on the query text and multi-modal data respectively to generate corresponding query semantic vectors and multi-modal vectors; A semantic increasing module is used to extract entities in the query text using a recognition model, and extract triples associated with the entities from the knowledge graph, splice the query text and the triples and perform vectorization processing to generate query semantic enhancement vectors; A retrieval matching module is used to retrieve the question and answer knowledge base in combination with a keyword matching algorithm and a similarity retrieval algorithm, calculate the similarity and matching degree of the query semantic enhancement vectors and the multi-modal vectors respectively, dynamically weight the similarity and matching degree, and acquire the text segment with the highest score; The encoding processing module is configured to splice the query text and the text segment with the highest retrieval score, encode the spliced text using a pre-trained language model, output semantic vectors of each vocabulary, and obtain a query text semantic vector matrix and a document semantic vector matrix respectively; The attention fusion module is configured to obtain attention weights of each vocabulary of the query text and the text segment according to the query text semantic vector matrix and the document semantic vector matrix through an attention mechanism, and fuse the query text semantic vector matrix and the document semantic vector matrix to obtain a fused semantic vector. The generation module is configured to obtain a vocabulary in the text segment that is greater than a preset word frequency and attention weight threshold, generate natural language based on the fused semantic vector using a GPT-2 model, add the extracted vocabulary to the natural language, generate a coherent sentence using a beam search algorithm, and output a natural language answer text.

[0068] Those skilled in the art can appreciate that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0069] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working process of the system and modules described above can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0070] In the embodiments provided by the present application, it should be understood that the disclosed system and method can be implemented in other ways. For example, the device embodiments described above are only schematic. The division of units is only a logical function division. There can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different units can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or in other forms.

[0071] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. they can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0072] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0073] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk, and various program code storage media.

[0074] In addition, it should be noted that in the system and method of the present application, it is obvious that each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions of the present application. Moreover, the steps of performing the above series of processes can naturally be executed in time sequence according to the order of description, but do not necessarily have to be executed in time sequence. Some steps can be executed in parallel or independently of each other. It can be understood by those skilled in the art that all or any steps or components of the method and device of the present application can be implemented in hardware, firmware, software or a combination thereof in any computing device (including a processor, a storage medium, etc.) or a network of computing devices, which can be implemented by those skilled in the art using their basic programming skills after reading the description of the present application.

[0075] Therefore, the purpose of the present application can also be achieved by running a program or a set of programs on any computing system. The computing system can be a commonly known general system. Therefore, the purpose of the present application can also be achieved by only providing a program product containing program code for implementing the method or device. That is, such a program product also constitutes the present application, and the storage medium storing such a program product also constitutes the present application. Obviously, the storage medium can be any commonly known storage medium or any storage medium developed in the future. It should be noted that in the device and method of the present application, it is obvious that each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions of the present application. Moreover, the steps of performing the above series of processes can naturally be executed in time sequence according to the order of description, but do not necessarily have to be executed in time sequence. Some steps can be executed in parallel or independently of each other.

[0076] The above merely provides the preferred embodiment of the present application, and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for constructing a large language model-based question answering system, the method comprising: Comprise the following steps: S1, acquire multi-modal data, build a question and answer knowledge base and a knowledge graph, and dynamically update the question and answer knowledge base; S2, acquire query text, and vectorize the query text and the multi-modal data respectively to generate corresponding query semantic vectors and multi-modal vectors; S3, use a recognition model to extract entities in the query text, and extract triples associated with the entities from the knowledge graph, concatenate the query text and the triples, and vectorize to generate query semantic enhancement vectors; S4, combine keyword matching algorithm and similarity retrieval algorithm to retrieve the question and answer knowledge base, calculate the similarity and matching degree of the query semantic enhancement vector and the multi-modal vector respectively, dynamically weight the similarity and matching degree, and obtain the text segment with the highest score; S5, concatenate the query text and the text segment with the highest score, encode the concatenated text using a pre-trained language model, output the semantic vector of each word, and obtain the query text semantic vector matrix and the document semantic vector matrix respectively; S6, according to the query text semantic vector matrix and the document semantic vector matrix, through the attention mechanism, the attention weight of each word of the query text and the text segment is obtained, and the query text semantic vector matrix and the document semantic vector matrix are fused to obtain a fused semantic vector; S7, acquire the words in the text segment that are greater than the preset word frequency and attention weight threshold, and based on the fused semantic vector, use the GPT-2 model to generate natural language, add the extracted words to the natural language, use the beam search algorithm to generate coherent sentences, and output the natural language answer text.

2. The method of claim 1, wherein the method further comprises: In step S1, the multi-modal data is acquired, the question and answer knowledge base and the knowledge graph containing entity associated triples are built, and the question and answer knowledge base is dynamically updated, comprising the following sub-steps: S11, acquire multi-modal data from documents, databases, API interfaces and web pages, and clean the multi-modal data to obtain standard sample data, build a question and answer knowledge base according to the standard sample data, and build a knowledge graph containing entity associated triples according to entity-relation-property; S12, dynamically update the question and answer knowledge base, judge whether the standard sample data has version update time information, if there is, acquire the corresponding update data and the data to be updated according to the update time, locate the added content interval between the update data and the data to be updated by binary search, update the version difference segment, and obtain the updated standard sample data; If not, calculate the hash value of the data block of the update data and the data to be updated, filter the difference block segment of the update data and the data to be updated, compare the semantic similarity of the difference block segment, obtain the similarity difference value, and preset the similarity update threshold value, when the similarity difference value is greater than the similarity update threshold value, update the difference block segment of the data to be updated.

3. The method of claim 2, wherein the method further comprises: In step S2, the query text is acquired, and the query text and the multi-modal data are vectorized respectively to generate corresponding query semantic vectors and multi-modal vectors, comprising the following steps: S21, acquire query text and text-image-table multi-modal data; S22, vectorizing the text data and the query text by using a pre-trained BERT language model to generate a text semantic vector from the text data and a query semantic vector from the query text; S23, vectorizing the image data by using a pre-trained ResNet visual model to extract image features and generate an image feature vector; S24, identifying the table data boundary and cells, determining the header row and data rows, establishing a row-column index structure, embedding and combining the header and cell contents to obtain a table semantic vector.

4. The method of claim 3, wherein the method further comprises: The step S3 uses a recognition model to extract entities in the query text, extracts triples associated with the entities from the knowledge graph, concatenates the query text and the triples, and vectorizes to generate a query semantic enhanced vector, including the following steps: S31, using a BLINK model to identify and obtain entities in the query text, linking the entities to the knowledge graph, and extracting triples associated with the entities from the knowledge graph; S32, concatenating the extracted triples, and inputting the concatenated text into a pre-trained BERT language model to generate a triple vector; S33, combining the triple vector and the query semantic vector by an MLP multi-layer perceptron with a first gating coefficient to output a query semantic enhanced vector; S34, calculating the cosine similarity of the query semantic vector and the triple vector to obtain a relevance score result, and dynamically adjusting the first gating coefficient based on the relevance score result.

5. The method of claim 4, wherein the method further comprises: The step S4 combines a keyword matching algorithm and a similarity retrieval algorithm to retrieve the question and answer knowledge base, respectively calculates the similarity and matching degree of the query semantic enhanced vector and the multi-modal vector, dynamically weights the similarity and matching degree, and obtains the text segment with the highest score, including the following sub-steps: S41, performing dependency syntax analysis on the query text by using a StanfordCoreNLP tool to generate a syntax tree and calculate the maximum syntax depth; S42, obtaining the ratio between the number of entity words in the query text and the total number of entities in the knowledge graph to obtain an entity density; S43, initializing the weight ratio of the BM25 keyword matching algorithm and the similarity retrieval algorithm, calculating the dynamic weights of the BM25 keyword matching algorithm and the similarity retrieval algorithm according to the maximum syntax depth, the entity density and the query complexity of the current query text, and the query complexity is calculated by the syntax tree depth and the entity density; S44, retrieving the question and answer knowledge base based on the BM25 keyword matching algorithm, calculating the matching degree score of the query text and the text segment, retrieving the question and answer knowledge base based on the similarity retrieval algorithm, calculating the cosine similarity of the semantic enhanced vector and the multi-modal vector; according to the dynamic weights of the BM25 keyword matching algorithm and the similarity retrieval algorithm, calculating the sum score of the matching degree score and the cosine similarity, and sorting according to the final sum score to obtain the text segment with the highest score as the retrieval result.

6. The method of claim 5, wherein the method further comprises: The query text and the highest-scored text segment are spliced in step S5, the spliced text is encoded using a pre-trained language model, and the semantic vector of each word is output, obtaining a query text semantic vector matrix and a document semantic vector matrix, including the following sub-steps: S51, the user query text and the highest-scored text segment are spliced in the following format, and the spliced text is encoded using a pre-trained RoBERTa-large model to output the semantic vector of each word; S52, determine whether the number of spliced text words exceeds the maximum input length of the RoBERTa-large model, if it exceeds, keep the first and last words according to half of the maximum input length, and truncate the middle part; S53, construct a query text semantic vector matrix according to the semantic vector of the corresponding word of the query text, and construct a document semantic vector matrix according to the semantic vector of the corresponding word of the text segment; S54, if the number of retrieved text segments is more than one, the semantic vector matrix of each text segment is mean-pooled to obtain a comprehensive document semantic vector.

7. The method of claim 6, wherein the method further comprises: determining a question type of the question based on the question type information; and determining a corresponding answer type of the answer based on the answer type information. In step S6, the attention mechanism is used to obtain the attention weight of each word of the query text and the text segment according to the query text semantic vector matrix and the document semantic vector matrix, and the query text semantic vector matrix and the document semantic vector matrix are fused to obtain a fusion semantic vector, including the following sub-steps: S61, the attention mechanism is used to calculate the attention weight of each word of the query text and the text segment according to the query text semantic vector matrix and the document semantic vector matrix; S62, the query text semantic vector matrix and the document semantic vector matrix are fused, and the contribution ratio of the query text and the text segment is dynamically adjusted by a second gating coefficient to calculate and generate a fusion semantic vector.

8. The method of claim 7, wherein the method further comprises: determining a question type of the question based on the question type information; and determining a corresponding answer based on the question type. In step S7, the words in the text segment that are greater than the preset word frequency and attention weight threshold are obtained, and based on the fusion semantic vector, a GPT-2 model is used to generate natural language, the extracted words are added to the natural language, a beam search algorithm is used to generate a coherent sentence, and a natural language answer text is output, including the following sub-steps: The preset word frequency threshold and the attention weight threshold are used to filter and obtain the words in the text segment that are greater than the preset word frequency and the attention weight threshold, and the filtered words are added to the GPT-2 generated word list; According to the current context and the fusion semantic vector, the generation probability of each word is calculated through the GPT-2 model; According to the attention weight and the word frequency, the replication probability of each word is calculated; According to the generation probability and the replication probability of each word and the weighted calculation, the candidate probability of each word is obtained; Based on the candidate probability of each word and using the beam search algorithm, a natural language answer is output.

9. A large language model-based question answering system construction system implemented by using the large language model-based question answering system construction method according to any one of claims 1 to 8. It includes: A knowledge base construction module for acquiring multi-modal data, constructing a question and answer knowledge base and a knowledge graph containing entity association triples, and dynamically updating the question and answer knowledge base; A data processing module for acquiring a query text and performing vectorization processing on the query text and multi-modal data to generate corresponding query semantic vectors and multi-modal vectors; The semantic increasing module is configured to extract an entity in the query text using a recognition model, extract a triple associated with the entity from a knowledge graph, splice the query text and the triple, and perform vectorization processing to generate a query semantic enhancement vector. The retrieval matching module is configured to retrieve the question and answer knowledge base by combining a keyword matching algorithm and a similarity retrieval algorithm, calculate similarity and matching degrees of the query semantic enhancement vector and the multi-modal vector, dynamically weight the similarity and the matching degrees, and obtain a text segment with the highest score. The encoding processing module is configured to splice the query text and the text segment with the highest score, encode the spliced text using a pre-trained language model, output semantic vectors of each word, and obtain a query text semantic vector matrix and a document semantic vector matrix. The attention fusion module is configured to obtain attention weights of each word in the query text and the text segment based on the query text semantic vector matrix and the document semantic vector matrix through an attention mechanism, and fuse the query text semantic vector matrix and the document semantic vector matrix to obtain a fused semantic vector. The generation module is configured to obtain words in the text segment that are greater than preset word frequency and attention weight thresholds, generate natural language based on the fused semantic vector using a GPT-2 model, add the extracted words to the natural language, generate a coherent sentence using a beam search algorithm, and output a natural language answer text.

10. A computer-readable storage medium, characterized in that, The storage medium has a large language model-based question and answer system construction method program stored thereon, and the large language model-based question and answer system construction method program is executed to implement the large language model-based question and answer system construction method of any one of claims 1 to 8.

Images