Output method, device and equipment for retrieval enhancement generation type questions and answers and storage medium

By integrating query questions and historical dialogue content into a large language model, enhanced query statements are generated and queried in the knowledge base. Answers are then filtered based on evaluation scores, solving the problem of low-quality answers in existing technologies and achieving more efficient and accurate answer generation.

CN119046433APending Publication Date: 2024-11-29CHINA PING AN PROPERTY INSURANCE CO LTD

Patent Information

Application Number
CN202411187876.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-27
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

In existing technologies, retrieval-enhanced generative question-answering systems based on large language models suffer from a lack of filtering procedures for low-quality answers, resulting in poor answer quality and low accuracy.

Method used

By receiving user-input query questions and historical dialogue content, the system integrates and generates an enhanced query statement using a pre-trained first language model. Based on this statement, the system searches the knowledge base, generates candidate answers using a second language model, and selects the target answer based on evaluation scores.

Benefits of technology

It improved the efficiency and accuracy of answer acquisition, enhanced the quality of answers, and reduced hallucinations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119046433A_ABST
    Figure CN119046433A_ABST
Patent Text Reader

Abstract

The invention belongs to the field of artificial intelligence, and relates to an output method for retrieval enhanced generative questions and answers, which comprises the following steps: integrating query questions and historical dialogue contents through a first big language model to obtain enhanced query statements, generating knowledge base query statements based on the enhanced query statements, and outputting the knowledge base query statements. Querying in a preset knowledge base according to the knowledge base query statement to obtain a knowledge base query result, inputting the knowledge base query result and the enhanced query statement into a second large language model, outputting candidate answers, and evaluating the candidate answers according to the knowledge base query result and the enhanced query statement. And screening out a target answer from the candidate answers according to the evaluation score. The invention further provides an output device and equipment for retrieval enhancement generation type questions and answers and a storage medium. In addition, the invention also relates to a block chain technology, and the query problem input by the user can be stored in a block chain. The answer generation efficiency and accuracy can be improved, and the answer quality is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence, financial technology and digital medical technology, and in particular to an output method, device, equipment and storage medium for retrieval-enhanced generative question and answer. Background Art

[0002] Large Language Model (LLM) is a natural language processing model based on transformer architecture. It has a large number of parameters and has been trained with a large amount of text. Therefore, LLM can be close to human language cognition and generation process. Compared with traditional NLP models, LLM can better understand and generate natural text, and has certain logic and reasoning capabilities, showing excellent capabilities in various question-answering scenarios. The cost of designing and training LLM from scratch is very high. With the open source of a number of excellent pre-trained models, such as LLaMa2, ChatGLM, Bloom, Ziya, etc., fine-tuning based on open source models has also become a research hotspot. In order to adapt to various specific scenarios, vertical field data is usually used to fine-tune the base LLM to form an industry large model. The fine-tuned model injected with high-quality industry data can have better performance in specific fields, but LLM is still prone to hallucinations, opacity, and untraceability during reasoning. A common practice is to integrate knowledge in the knowledge base to realize vector RAG, which is the abbreviation of Retrieval Augmented Generation. However, the connection between the answers generated by the RAG scheme and the user questions and the knowledge retrieved from the knowledge base is not intuitive, and the hallucination problem cannot be completely avoided. Moreover, the existing RAG scheme directly outputs the answers generated after retrieval enhancement, lacks a filtering procedure for low-quality answers, resulting in poor quality and low accuracy of the generated answers. Summary of the invention

[0003] The purpose of the embodiments of the present application is to propose an output method, device, computer equipment and storage medium for retrieval-enhanced generative question and answer, so as to solve the technical problem in the related art that there is a lack of filtering procedures for low-quality answers, resulting in poor quality and low accuracy of the generated answers.

[0004] In order to solve the above technical problems, the embodiment of the present application provides a method for outputting retrieval-enhanced generative question and answer, which adopts the following technical solution:

[0005] Receive a query question input by a user, and obtain historical conversation content corresponding to the query question;

[0006] Inputting the query question and the historical conversation content into the trained first language model for integration to obtain an enhanced query statement;

[0007] Generate a knowledge base query statement based on the enhanced query statement, and query a preset knowledge base according to the knowledge base query statement to obtain a knowledge base query result;

[0008] Inputting the knowledge base query result and the enhanced query statement into the trained second language model, and outputting candidate answers corresponding to the query question;

[0009] Evaluate the candidate answers according to the knowledge base query result and the enhanced query statement to obtain an evaluation score;

[0010] The candidate answers are screened according to the evaluation scores, and a target answer is output.

[0011] In order to solve the above technical problems, the embodiment of the present application also provides an output device for retrieval-enhanced generative question-answering, which adopts the following technical solution:

[0012] An acquisition module, used to receive a query question input by a user and acquire historical conversation content corresponding to the query question;

[0013] A refinement module, configured to input the query question and the historical conversation content into the trained first language model for integration to obtain an enhanced query statement;

[0014] A query module, used to generate a knowledge base query statement based on the enhanced query statement, and query a preset knowledge base according to the knowledge base query statement to obtain a knowledge base query result;

[0015] An answer generation module, used for inputting the knowledge base query result and the enhanced query sentence into the trained second language model, and outputting candidate answers corresponding to the query question;

[0016] An evaluation module, used to evaluate the candidate answers according to the knowledge base query result and the enhanced query statement to obtain an evaluation score;

[0017] The screening module is used to screen the candidate answers according to the evaluation scores and output a target answer.

[0018] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0019] The computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the output method of the retrieval-enhanced generative question answering as described above are implemented.

[0020] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0021] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the output method of the retrieval-enhanced generative question and answer are implemented as described above.

[0022] Compared with the prior art, this application has the following beneficial effects:

[0023] The present application obtains historical conversation content corresponding to a query question, integrates the query question and the historical conversation content through a trained first language model, obtains an enhanced query statement, generates a knowledge base query statement based on the enhanced query statement, and queries a preset knowledge base according to the knowledge base query statement to obtain a knowledge base query result, then inputs the knowledge base query result and the enhanced query statement into a trained second language model, outputs candidate answers, and then evaluates the candidate answers based on the knowledge base query result and the enhanced query statement to obtain an evaluation score, and finally screens out a target answer from the candidate answers based on the evaluation score; the present application obtains candidate answers based on the query question and the preset knowledge base through the first language model and the second language model, and then evaluates the candidate answers, which can improve the efficiency and accuracy of answer acquisition, thereby improving the quality of the answer. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0026] Figure 2 is a flowchart of an embodiment of the output method of retrieval-enhanced generative question answering according to the present application;

[0027] Figure 3 yes Figure 2 A flowchart of a specific implementation of step S202;

[0028] Figure 4 yes Figure 2 A flowchart of a specific implementation of step S204;

[0029] Figure 5 is a structural schematic diagram of an embodiment of an output device for retrieval-enhanced generative question answering according to the present application;

[0030] Figure 6 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0032] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0033] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0034] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0035] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0036] This application provides a retrieval-enhanced generative question-answering output method that can be applied to Figure 1In the system architecture 100 shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0037] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0038] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0039] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0040] It should be noted that the output method of the retrieval-enhanced generative question and answer provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the output device of the retrieval-enhanced generative question and answer is generally set in the server / terminal device.

[0041] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0042] Continue to refer Figure 2 , showing a flowchart of an embodiment of the output method of retrieval-enhanced generative question answering according to the present application, comprising the following steps:

[0043] Step S201, receiving a query question input by a user, and obtaining historical conversation content corresponding to the query question.

[0044] Receive a query question input by a user and pre-process the query question, wherein the pre-processing includes removing special characters, punctuation marks, extra spaces, and spelling checking and correction.

[0045] Determine the topic of the preprocessed query question and extract keywords based on the topic. Extracting keywords requires combining entity recognition and context understanding to ensure that the keywords are correctly identified and extracted. Then, query the context information of historical conversations based on the keywords and question topics to obtain the historical conversation content corresponding to the query question, ensuring that the explanation and answer to the query question are consistent with the historical conversation, thereby providing a coherent conversation experience.

[0046] For example, taking the query question as a medical question, the subject of the medical question is determined, and keywords are extracted from the medical question based on the subject, including entities such as drugs, symptoms, diseases, time, events, etc. The subject and intent of the question are determined based on these entities.

[0047] In this embodiment, the electronic device (eg, Figure 1 The server / terminal device shown in the figure can receive the query question input by the user through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0048] It should be emphasized that in order to further ensure the privacy and security of the query questions, the above query questions can also be stored in a node of a blockchain.

[0049] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0050] Step S202: Input the query question and the historical conversation content into the trained first language model for integration to obtain an enhanced query statement.

[0051] The first language model is trained to better understand and answer industry-related questions, which is achieved by introducing text data from the corresponding industry field for model training, where the industries include medical care, finance, etc. In this embodiment, the query question and historical conversation content are refined through the first language model to obtain a more refined question expression, that is, an enhanced query statement.

[0052] In some optional implementations, the first large language model includes a first embedding layer, a multi-level feature extraction layer, a head entity recognition layer, a first attention layer, and a relation and tail entity joint recognition layer connected in sequence, see Figure 3 As shown, the above step of inputting the query question and the historical conversation content into the trained first language model for integration to obtain the enhanced query statement includes:

[0053] Step S301: Input the query question and historical conversation content into the first embedding layer for vector conversion to obtain a sentence embedding vector.

[0054] In some optional implementations, before inputting the query questions and historical conversation contents into the first embedding layer, the query questions and historical conversation contents are cleaned to remove irrelevant characters in the sample data, such as special punctuation marks, blank characters, line breaks, spaces, etc.

[0055] After data cleaning, query questions and historical conversation content are divided into semantic units. A [start], [CLS], or [C] symbol is added before each semantic unit, and a [SEP] or [S] symbol is added at the end of the semantic unit. The semantic unit symbol is a token.

[0056] Sequence tags, such as [CLS] and [SEP], are added to query questions and historical conversation content to obtain sentence embedding vectors. The sentence embedding vectors include semantic unit (Token) vectors, text (Segment) vectors, and position (Position) vectors, which represent word information, sentence information, and position information, respectively.

[0057] Step S302, extracting features from the sentence embedding vector through a multi-level feature extraction layer to obtain multi-level text features.

[0058] The multi-level feature extraction layer includes a bottom layer network unit, an intermediate layer network unit and a high layer network unit. In this embodiment, the step of extracting features from the sentence embedding vector through the multi-level feature extraction layer to obtain multi-level text features includes:

[0059] Surface features of sentence embedding vectors are extracted through the bottom network units, and the extracted surface features are input into the middle network units and the high-level network units respectively;

[0060] The surface features are extracted through the middle layer network units to obtain syntactic features, and the syntactic features are input into the high-level network units;

[0061] The surface features and syntactic features are extracted through high-level network units to obtain semantic features;

[0062] Surface features, syntactic features and semantic features are concatenated to obtain multi-level text features.

[0063] The multi-level feature extraction layer includes a stacked 12-layer network structure. Each layer of the network structure uses the decoder (encoder) part of the transformer. Each layer of the encoder includes a multi-head attention mechanism (Muti-Head-Attention) and a position-wise feed-forward neural network (Position-Wise-Feed-Forward-Networks). Among them, the 12-layer network structure includes 1 to 4 layers of bottom-layer network units, 5 to 8 layers of middle-layer network units and 9 to 12 layers of high-layer network units. The bottom-layer network units are used to learn the surface features at the phrase level, the middle-layer network units are used to learn the syntactic features at the syntactic level, and the high-layer network units are used to learn the semantic features at the semantic level.

[0064] Among them, the 4th layer network, the 8th layer network and the 12th layer network are connected, that is, the input of the 8th layer network is the output of the 7th layer network and the output of the 4th layer network, and the input of the 12th layer network is the output of the 11th layer network structure, the 8th layer network and the 4th layer network. The purpose is to enhance the features and make the refined problem more accurate.

[0065] The surface feature X1, syntactic feature X2 and semantic feature X3 are concatenated to obtain the multi-level text feature X m , the splicing formula used is as follows:

[0066] X m =X1⊕X2⊕X3;

[0067] Among them, ⊕ is a vector concatenation operation.

[0068] In this embodiment, by extracting and concatenating features at different levels, semantic information can be enriched, feature complementarity can be improved, and context understanding can be facilitated, so that the model can be better generalized to unseen data, thereby improving the accuracy of the refined problem.

[0069] Step S303: input the multi-level text features into the head entity recognition layer for recognition, and output all head entities.

[0070] In this embodiment, the input sequence h is encoded by the head entity recognition layer. nDecode and then get the entity's start position (starting position) and end position (ending position) to identify all possible head entities. The head entity recognition layer uses the following calculation formula:

[0071]

[0072] in, and Respectively represent the input sequence h n The probability that the i-th multi-level text feature is recognized as the start position and the end position of the head entity. If the probability exceeds the preset threshold, the label value of the multi-level text feature is recorded as 1, otherwise it is 0; σ is the sigmoid activation function, ω start With ω end is the trainable weight, b start With b end represents the bias value; and x i =h n [i], represents the input sequence h n The multi-level text features of the i-th token in .

[0073] The closest matching principle is: get all and The nearest neighbor The start and The end of 1 is regarded as a complete head entity. This matching method ensures the integrity of the entity and improves the accuracy of head entity recognition.

[0074] Step S304, performing self-attention fusion calculation on all head entities and multi-level text features through the first attention layer to obtain a head entity tuple relationship matrix.

[0075] By assigning different weights to different head entities in the first attention layer, a vector that better represents the entity can be obtained. In this embodiment, the calculation formula of the self-attention function is as follows:

[0076]

[0077] Among them, d k Represents the dimension of the multi-level text features of the input; Q, K, V represent the query matrix, key matrix and value matrix respectively, and the three are connected through the parameter matrix W Q , W K , W V It is obtained by multiplying the vector of each entity, and then using the self-attention function to calculate the attention size of different positions, and finally linearly transformed by the softmax function.

[0078] Step S305 , input the head entity tuple relationship matrix into the relationship and tail entity joint recognition layer for recognition, and obtain all specific relationships and tail entities of each head entity.

[0079] After identifying the head entity, the joint identification of the relationship and the tail entity is performed. The structure of each layer of the joint identification layer of the relationship and the tail entity is the same as that of the head entity identification layer. The decoding of the obj entity (tail entity) is similar to the decoding of the sub entity (head entity). The difference between the joint identification layer of the relationship and the tail entity and the head entity identification layer is that the multi-level text features containing contextual semantic information of the encoding layer are added to the input, and the head entity feature information extracted by the head entity identification layer is also considered. The relationship modeling is used as a function of mapping the head entity to the tail entity.

[0080] The head entity tuple relation matrix containing contextual semantic information and head entity feature information is input into the relation and tail entity joint recognition layer for recognition, and all specific relations and tail entities of each head entity are obtained.

[0081] In this embodiment, the relationship and tail entity joint identification layer is calculated using the following formula:

[0082]

[0083] in, and Respectively represent the input sequence h n The probability of the i-th multi-level text feature being recognized as the start position and the end position of the tail entity. If the probability exceeds the preset threshold, the label value of the multi-level text feature is recorded as 1, otherwise it is 0; r represents a specific relationship; Represents the average vector of all multi-level text features between the start and end positions of the k-th head entity, that is, the head entity tuple relationship matrix.

[0084] It should be noted that when start_o and end_o are both judged as 0 after binary division, it means that there is no corresponding obj entity under this specific relationship, that is, when sub is the head entity, there is no triple of this relationship.

[0085] Step S306, obtaining entity relationship triples based on the head entity, the specific relationship and the tail entity, and generating enhanced query statements according to the entity relationship triples.

[0086] In this embodiment, the generated entity relationship triples are expressed as (head entity, specific relationship, tail entity). According to the entity relationship triples, the subject in the entity relationship triples is determined to be the head entity, the predicate is the specific relationship, and the object is the tail entity. Then, according to the word order and grammatical rules of natural language, sentences are usually constructed in the order of "subject + predicate + object". The constructed sentences should take into account contextual information to obtain more reasonable and accurate enhanced query statements.

[0087] In this embodiment, question refinement is performed through the first language model, which can improve the accuracy of question understanding and increase the efficiency and accuracy of question refinement.

[0088] Step S203, generating a knowledge base query statement based on the enhanced query statement, and querying a preset knowledge base according to the knowledge base query statement to obtain a knowledge base query result.

[0089] Among them, the knowledge base refers to a data warehouse or knowledge graph built in the field of artificial intelligence that covers knowledge in a specific industry field. This concept involves integrating and organizing a large amount of structured and unstructured information about a specific industry, including but not limited to technical standards, industry trends, market data, professional terms, and the relationship between related entities. By utilizing technologies such as natural language processing, machine learning, and knowledge graphs, industry knowledge bases help to achieve intelligent search, understanding, and analysis of large-scale data, thereby providing powerful tools and resources for decision support, innovative research and development, and information discovery in specific industry fields. For example, in the medical field, the knowledge base can include drug knowledge bases, disease knowledge bases, clinical trial knowledge bases, medical literature knowledge bases, medical imaging knowledge bases, epidemiological knowledge bases, medical dictionaries, and terminology knowledge bases; in the insurance field, the knowledge base can include insurance professional terminology knowledge bases, insurance product information knowledge bases, insurance case knowledge bases, and legal and policy knowledge bases. It should be understood that the knowledge base of each industry can be constructed according to actual needs.

[0090] The process of generating a knowledge base query statement based on an enhanced query statement is to convert a natural language query into a format that is more suitable for a specific knowledge base structure and query language. In some optional implementations of this embodiment, the step of generating a knowledge base query statement based on an enhanced query statement includes:

[0091] Perform word segmentation on the enhanced query statement to obtain the word segmentation result;

[0092] According to the word segmentation results, the frequency of each word in the preset knowledge base is counted, and the weight of each word is determined according to the frequency;

[0093] Extracting keywords in the enhanced query sentence based on the weight of each word, and determining the extended words of the keywords;

[0094] A knowledge base query statement corresponding to the enhanced query statement is generated according to the keywords and the extended words.

[0095] The enhanced query statement is analyzed. Specifically, the enhanced query statement can be subjected to semantic analysis, which includes word segmentation, part-of-speech analysis, named entity recognition, stop word removal, etc. The enhanced query statement is subjected to word segmentation processing based on semantics using a word segmentation controller to obtain a word segmentation result. The number of times each word appears in the preset knowledge base, i.e., the word frequency, is counted, and the weight of each word is determined according to the word frequency: word weight = word frequency of the word / sum of word frequencies of all words.

[0096] Keywords are an important part of questions and can help understand the user's question intention. Words with word weights greater than or equal to the preset weights are determined as keywords for enhanced query statements.

[0097] In this embodiment, the acquisition of extended words can be obtained through an associated word list, and words with high relevance to the keywords in the enhanced query statement are obtained through the associated word list, and the keywords are replaced with the words; they can also be obtained through a knowledge graph, and the knowledge graph can be pre-constructed, and the extended words are automatically generated according to the hyponyms or related synonyms of the keywords using the pre-constructed knowledge graph. It should be noted that the knowledge graph can continuously collect data for training based on the user's query so that the scale of the extended words becomes larger and more accurate. In this embodiment, the generation of extended words through keywords can expand the search scope and thereby improve the recall rate of user questions.

[0098] Obtain a query interface of a preset knowledge base, construct a query request according to the knowledge base query statement and the query interface, send the query request through the query interface, and receive a knowledge base query result returned by the preset knowledge base in response to the query request.

[0099] In this embodiment, searching a preset knowledge base through a knowledge base query statement can improve retrieval efficiency and retrieval accuracy.

[0100] Step S204: input the knowledge base query result and the enhanced query statement into the trained second language model, and output the candidate answers corresponding to the query question.

[0101] In this embodiment, the second largest language model includes a knowledge layer, a second embedding layer, a selection mechanism layer, a visible layer, an encoding layer, a CRF layer and an output layer. The second largest language model is already trained, and the question-answer pair data set is used to train the second largest language model.

[0102] In some optional implementations, see Figure 4 As shown, the above step of inputting the knowledge base query result and the enhanced query statement into the trained second language model and outputting the candidate answer corresponding to the query question includes:

[0103] Step S401, injecting the knowledge of the knowledge base query result into the enhanced query statement through the knowledge layer to form a sentence tree, and inputting the sentence tree into the second embedding layer and the visible layer respectively.

[0104] The knowledge layer is responsible for injecting the knowledge in the knowledge base query results into the enhanced query statement to form a sentence tree. For example, for each input sentence s = {w1, w2, w3, ..., w n}, first obtain the named entity from the enhanced query sentence, then query the corresponding relationship and value (tail entity) of the obtained named entity in the knowledge graph to form a triple of <head entity, relationship, value>, and then return this triple to the corresponding position of the sentence to form a tree structure called Sentence Tree. The sentence tree completes the background information of the sentence and solves the problem that the word vector deviates from the core semantics due to the lack of knowledge background in a single sentence.

[0105] In this embodiment, forming a sentence tree is divided into two steps: knowledge query (K-Query) and knowledge injection (K-Inject).

[0106] Furthermore, the knowledge query function of the knowledge layer is called to identify all entities corresponding to the enhanced query statement, and the triples corresponding to each entity are queried in the knowledge graph; the knowledge injection function of the knowledge layer is called to embed the triples into the corresponding positions in the enhanced query statement to obtain the sentence tree.

[0107] K-Query is responsible for querying the relationship and value corresponding to each entity in each enhanced query statement from the knowledge graph, that is, the triple. The specific query process is as follows:

[0108] E = K_Query(s, K);

[0109] Among them, the function K_Query represents the enhanced query statement s to query the knowledge graph K and obtain the triple set E = {(w i , r i0 , w i0 ),……,(w i , r ik , w ik )].

[0110] K-Inject is responsible for embedding the triple set E into the corresponding position in the enhanced query sentence s to form a sentence tree. Each triple constitutes a branch. The matrix sentence tree output by the knowledge layer is: t = {w1, w2, w3, ..., w i (r i0 , w i0 ), ..., w i (rik , w ik ),……,w n}.

[0111] Step S402, position embedding is performed on the sentence tree through the second embedding layer to obtain a sentence position encoding vector.

[0112] The sentence tree cannot be directly used as a sequence input. The sentence tree is converted into a text sequence through the second embedding layer. The second embedding layer includes the token embedding layer, the soft-position embedding layer, and the segment embedding layer. The token embedding layer maps each token in the sentence into a vector representation with a dimension of H. There is a special token [CLS] at the beginning of each sentence, which is mainly for sentence classification. The soft-position embedding layer uses soft positions to encode embedded values ​​and relationships, which is different from the position encoding of entities. The segment embedding layer is used to distinguish between two sentence fragments.

[0113] Specifically, the sentence tree is input into the second embedding layer to perform segment embedding operation, soft position embedding operation and word embedding operation respectively to obtain the corresponding sentence encoding vector, position encoding vector and word encoding vector; the sentence encoding vector, position encoding vector and word encoding vector are summed to obtain the sentence position encoding vector.

[0114] Through the token embedding layer, each word in the sentence tree is embedded to obtain the sentence encoding vector; through the soft-position embedding layer, each word in the sentence tree is soft-position embedded to obtain the position encoding vector, which retains the main position information of the sentence; through the segment embedding layer, each word in the sentence tree is embedded to obtain the word encoding vector, and the sentence encoding vector, position encoding vector and word encoding vector are summed to obtain the sentence position encoding vector, where the sentence position encoding vector retains the structural information of the tree structure, which can strengthen the feature representation and better obtain the semantic features of the text.

[0115] Step S403, performing weighted calculation on the sentence position encoding vector by selecting a mechanism layer to obtain a text weighted feature vector.

[0116] The selection mechanism layer includes an average pooling sublayer and a multi-layer perceptron sublayer. The sentence position encoding vector is pooled by the average pooling sublayer to obtain the pooled sentence position encoding; the sentence position encoding vector and the pooled sentence position encoding are input into the multi-layer perceptron layer to construct a weighted representation matrix; based on the pooled sentence position encoding and the weighted representation matrix, the text weighted feature vector is calculated.

[0117] Step S404, constructing a text visible matrix of the sentence tree through the visible layer.

[0118] Since the triples in the sentence tree may affect the meaning of the enhanced query sentence, in order to prevent the influence of knowledge noise on the sentence, a text visible matrix M is constructed to limit each word to only see the context and knowledge related to itself.

[0119] Assume that the two tokens are w i and w j , M ij = 0, indicating that in the text visible matrix w i to w j It is visible, M ij = -∞, indicating that in the text visible matrix w i to w j is invisible.

[0120] Step S405: input the text weighted feature vector and the text visibility matrix into the encoding layer for attention calculation, and output the text enhancement feature.

[0121] In this embodiment, the encoding layer includes a multi-head attention sublayer and a feedforward neural network sublayer. The text weighted feature vector and the text visible matrix are input into the multi-head attention sublayer for multi-head attention calculation to obtain multi-head attention features; the text weighted feature vector and the multi-head attention features are enhanced by the feedforward neural network sublayer to obtain text enhancement features.

[0122] Among them, the multi-head attention sublayer can capture contextual information when calculating the attention vector; the feedforward network sublayer is used to connect and enhance the features output by the multi-head attention sublayer to prevent insufficient fitting.

[0123] In this embodiment, the use of the encoding layer can limit the visible area of ​​the self-attention mechanism, thereby capturing the deep bidirectional structure in the text sentence. The multi-head attention sublayer is composed of multiple mask-self-attention layers stacked together, and adds a text visibility matrix M on the basis of mask-self-attention to perform attention calculation on the text weighted feature vector.

[0124] Assuming the number of mask-self-attention layers is L and the number of heads is A, determine the query vector parameter matrix W for each layer of mask-self-attention q , key vector parameter matrix W k Sum value vector parameter matrix W v , the self-attention calculation formula of each layer of mask-self-attention is as follows:

[0125]

[0126] Among them, Q i =H i-1 W q ; K i =H i-1 W k ; V i =H i-1 W v ;Q i represents the query vector of the i-th layer, K i represents the key vector of the i-th layer, V i Represents the value vector of the i-th layer; H i represents the mask-self-attention output of the i-th layer; d k Represents the dimension of the input text weighted feature vector.

[0127] The multi-head attention calculation formula is as follows:

[0128] MultiHead=Concat(head1,head2,…,head A )W0;

[0129] Among them, Concat represents the matrix concatenation function; W0 represents the parameter matrix when compressing each self-attention.

[0130] In this embodiment, attention calculation is performed through the encoding layer, which can capture deep features in text sentences, enrich the representation capability of features, and improve the generalization ability of the model.

[0131] Step S406: input the text enhancement features into the CRF layer for classification, and output a text classification sequence.

[0132] Among them, the CRF layer can effectively utilize sentence-level label information to further explore the relationship between different entities and set constraints to ensure the effectiveness of the final prediction. The constraints can be automatically learned by the CRF layer when training data.

[0133] The parameter of the CRF layer is a (k+2)-power matrix A. Since we need to add a start state and a stop state to the beginning and end of the sentence, we need to add 2. A ij represents the transition probability from the i-th annotation position to the j-th annotation position, so that when annotating a certain position, the previously annotated information can be fully utilized. Assume that the enhanced query statement x that needs to identify the entity is represented by (x1, x2…, x n ), a label sequence y with a length equal to the length of the sentence is (y1,y2,…,y n ), then the score corresponding to the sentence corresponding to the recognized entity is determined as follows:

[0134]

[0135] Among them, A is the transfer score matrix, A yi ,y i+ 1 means from label y i Move to label y i+1 The fraction of y0 and y n are the start and end tags of the enhanced query statement respectively; so the latitude of A is (k+2)*(k+2) (k is the number of tags); P is the text enhancement feature, the latitude is n*k (k is the number of tags), P i,yi Indicates that the label corresponding to the i-th word in the enhanced query is y i The probability value of .

[0136] Use the softmax activation function to normalize the scores and get the probability of label y. The calculation formula is as follows:

[0137]

[0138] Among them, y' represents the possible annotation sequence corresponding to the enhanced query statement x. That is to say, each annotation sequence corresponding to the enhanced query statement has a score and a probability. The purpose is to make the true annotation sequence corresponding to the enhanced query statement have the highest probability. The true annotation sequence with the highest probability is the text classification sequence.

[0139] Step S407, reorganize the text classification sequence through the output layer to obtain candidate answers.

[0140] The text classification sequence output by the CRF layer is reorganized according to the sequence classification label of the text classification sequence. For example, if the category of the text classification sequence is the beginning, it is placed at the beginning; if the category is the end, it is placed at the end.

[0141] In this embodiment, candidate answers are generated according to the knowledge base query results and enhanced query statements through the second largest language model, which can utilize external knowledge, have a stronger ability to understand the context, improve the accuracy of answer generation, and reduce hallucinations.

[0142] It should be noted that the first language model and the second language model may use the same model.

[0143] Step S205, evaluating the candidate answers according to the knowledge base query result and the enhanced query statement to obtain an evaluation score.

[0144] In order to improve the output quality of retrieval-enhanced generative question-answering answers, this embodiment evaluates candidate answers based on three evaluation indicators: confidence, completeness, and ethics, so as to filter out low-quality answers.

[0145] Specifically, the credibility of the candidate answers is evaluated based on the knowledge base query results to obtain a credibility score; the completeness of the candidate answers is evaluated based on the enhanced query statement to obtain a completeness score; the compliance of the candidate answers is evaluated according to the preset compliance evaluation rules to obtain a compliance score; the credibility score, completeness score and compliance score are weighted to obtain an evaluation score.

[0146] Among them, the score range of credibility is [0,1], and the standard is whether the answer is consistent with the facts in the text queried in the knowledge base, which is used to evaluate whether the answer of the large model is consistent with the retrieved content; the score range of completeness is [0,1], and the standard is whether the answer fully answers the user's query question, which is used to evaluate whether the answer of the large model fully responds to the user's question; the score range of compliance is [0,1], and the standard is whether the answer complies with laws, regulations and ethics, which is used to evaluate whether the answer of the large model complies with laws, regulations or ethics.

[0147] During the evaluation process, the importance of different dimensions is considered, and weight rules are designed to give a score interval of [0,1], and finally a comprehensive calculation is performed. For example, a weighted calculation formula is as follows:

[0148] final score =0.5*confidence score +0.3*ethic score +0.2*completeness score

[0149] In some optional implementations, a filter score threshold is preset. score If final score ≥pass score, output the answer to the user, if final score <pass score , output the preset content to the user, and output the question and answer evaluation results to the operation and maintenance personnel for subsequent analysis and correction.

[0150] In some optional implementations, candidate answers may be evaluated by constructing an evaluation model or by constructing an evaluation template.

[0151] For example, the evaluation template is as follows:

[0152] {"confidence":{"score":<The score range is [0,1], keep two decimal places>,"evidence":<Write the reason for the score, string format>},"completeness":{"score":<The score range is [0,1], keep two decimal places>,"evidence":<Write the reason for the score, string format>},"ethic":{"score":<The score range is [0,1], keep two decimal places>,"evidence":<Write the reason for the score, string format>}}.

[0153] In this embodiment, a unified evaluation system is used to evaluate the quality of answers. By weighting the evaluation dimensions, a final quality score is given, which is more in line with business requirements and can help the question-and-answer system quickly decide whether the answer can be output to the customer. When evaluating the answers, credibility analysis, completeness analysis, and compliance analysis are performed respectively, which can help operators trace the source of problems more quickly, supplement knowledge, and thus improve the quality and accuracy of the answers.

[0154] Step S206, screening candidate answers according to the evaluation scores and outputting the target answer.

[0155] Sort the candidate answers from high to low according to the evaluation scores, and confirm whether the predicted score of the first candidate answer is greater than or equal to the filtering score threshold. If so, output the candidate answer as the target answer. Otherwise, feed back the evaluation results to the operation and maintenance personnel for improvement.

[0156] This application uses the first language model and the second language model to obtain candidate answers based on the query question in combination with a preset knowledge base, and then evaluates the candidate answers, which can improve the efficiency and accuracy of answer acquisition and thus improve the quality of the answers.

[0157] The output method of retrieval-enhanced generative question answering provided in the present application can be applied to the medical field or the financial technology field. For the medical field, the output method of retrieval-enhanced generative question answering of the present application can be used to answer medical questions; for the financial technology field, the output method of retrieval-enhanced generative question answering of the present application can be used to answer financial knowledge questions, insurance underwriting questions, etc.

[0158] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0159] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0160] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0161] Further references Figure 5 , as a response to the above Figure 2The present application provides an embodiment of a device for outputting retrieval-enhanced generative question-answering. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0162] like Figure 5 As shown, the output device 500 of the retrieval enhanced generative question and answer described in this embodiment includes: an acquisition module 501, a refinement module 502, a query module 503, an answer generation module 504, an evaluation module 505 and a screening module 506. Among them:

[0163] The acquisition module 501 is used to receive a query question input by a user and obtain the historical conversation content corresponding to the query question;

[0164] The refining module 502 is used to input the query question and the historical conversation content into the trained first language model for integration to obtain an enhanced query statement;

[0165] The query module 503 is used to generate a knowledge base query statement based on the enhanced query statement, and perform a query in a preset knowledge base according to the knowledge base query statement to obtain a knowledge base query result;

[0166] The answer generation module 504 is used to input the knowledge base query result and the enhanced query sentence into the trained second language model, and output the candidate answer corresponding to the query question;

[0167] The evaluation module 505 is used to evaluate the candidate answer according to the knowledge base query result and the enhanced query statement to obtain an evaluation score;

[0168] The screening module 506 is used to screen the candidate answers according to the evaluation scores and output a target answer.

[0169] It should be emphasized that in order to further ensure the privacy and security of the query questions, the above query questions can also be stored in a node of a blockchain.

[0170] The output device 500 based on the above-mentioned retrieval-enhanced generative question and answer obtains candidate answers based on the query question through the first language model and the second language model in combination with the preset knowledge base, and then evaluates the candidate answers, which can improve the efficiency and accuracy of answer acquisition and thus improve the quality of the answers.

[0171] In some optional implementations, the first large language model includes a first embedding layer, a multi-level feature extraction layer, a head entity recognition layer, a first attention layer, and a relationship and tail entity joint recognition layer; the refining module 502 includes a vector conversion submodule, a feature extraction submodule, a head entity recognition submodule, a self-attention calculation submodule, a joint recognition submodule, and a refining submodule, wherein:

[0172] The vector conversion submodule is used to input the query question and the historical conversation content into the first embedding layer for vector conversion to obtain a sentence embedding vector;

[0173] The feature extraction submodule is used to extract features from the sentence embedding vector through the multi-level feature extraction layer to obtain multi-level text features;

[0174] The head entity recognition submodule is used to input the multi-level text features into the head entity recognition layer for recognition, and output all head entities;

[0175] The self-attention calculation submodule is used to perform self-attention fusion calculation on all the head entities and the multi-level text features through the first attention layer to obtain a head entity tuple relationship matrix;

[0176] The joint identification submodule is used to input the head entity tuple relationship matrix into the relationship and tail entity joint identification layer for identification, and obtain all specific relationships and tail entities of each head entity;

[0177] The refining submodule is used to obtain entity relationship triples based on the head entity, the specific relationship and the tail entity, and generate enhanced query statements according to the entity relationship triples.

[0178] Refining questions through the first language model can improve the accuracy of question understanding and increase the efficiency and accuracy of question refinement.

[0179] In some optional implementations of this embodiment, the multi-level feature extraction layer includes a bottom layer network unit, an intermediate layer network unit and a high layer network unit, and the feature extraction submodule is used to:

[0180] Extracting surface features of the sentence embedding vector through the bottom network unit, and inputting the extracted surface features into the middle network unit and the high-level network unit respectively;

[0181] Extracting the surface features through the middle layer network unit to obtain the syntactic features, and inputting the syntactic features into the high layer network unit;

[0182] Extracting the surface features and the syntactic features through the high-level network unit to obtain the semantic features;

[0183] The surface features, the syntactic features and the semantic features are concatenated to obtain multi-level text features.

[0184] By extracting and concatenating features at different levels, we can enrich semantic information, improve feature complementarity, and help understand the context, so that the model can be better generalized to unseen data, thereby improving the accuracy of refined problems.

[0185] In some optional implementations, the query module 503 includes a word segmentation submodule, a weight determination submodule, an expansion submodule, and a generation submodule, wherein:

[0186] The word segmentation submodule is used to segment the enhanced query statement to obtain a word segmentation result;

[0187] The weight determination submodule is used to count the word frequency of each word in the preset knowledge base according to the word segmentation result, and determine the weight of each word according to the word frequency;

[0188] The expansion submodule is used to extract the keywords in the enhanced query sentence based on the weight of each of the words, and determine the expansion words of the keywords;

[0189] The generating submodule is used to generate a knowledge base query statement corresponding to the enhanced query statement according to the keyword and the extended word.

[0190] Querying the preset knowledge base through knowledge base query statements can improve retrieval efficiency and retrieval accuracy.

[0191] In some optional implementations, the second language model includes a knowledge layer, a second embedding layer, a selection mechanism layer, a visible layer, an encoding layer, a CRF layer, and an output layer, and the answer generation module 504 includes a knowledge injection submodule, a position embedding submodule, a weighting submodule, a matrix construction submodule, an encoding submodule, a classification submodule, and a reorganization submodule, wherein:

[0192] The knowledge injection submodule is used to inject the knowledge of the knowledge base query result into the enhanced query statement through the knowledge layer to form a sentence tree, and input the sentence tree into the second embedding layer and the visible layer respectively;

[0193] The position embedding submodule is used to perform position embedding on the sentence tree through the second embedding layer to obtain a sentence position encoding vector;

[0194] The weighting submodule is used to perform weighted calculation on the sentence position encoding vector through the selection mechanism layer to obtain a text weighted feature vector;

[0195] The matrix construction submodule is used to construct a text visible matrix of the sentence tree through the visible layer;

[0196] The encoding submodule is used to input the text weighted feature vector and the text visibility matrix into the encoding layer for attention calculation, and output text enhancement features;

[0197] The classification submodule is used to input the text enhancement features into the CRF layer for classification, and output a text classification sequence;

[0198] The reorganization submodule is used to reorganize the text classification sequence through the output layer to obtain the candidate answer.

[0199] The second largest language model generates candidate answers based on the knowledge base query results and enhanced query statements, which can utilize external knowledge, have a stronger ability to understand the context, improve the accuracy of answer generation, and reduce hallucinations.

[0200] In some optional implementations of this embodiment, the encoding layer includes a multi-head attention sublayer and a feedforward neural network sublayer; the encoding submodule is further used for:

[0201] Inputting the text weighted feature vector and the text visible matrix into the multi-head attention sublayer to perform multi-head attention calculation to obtain a multi-head attention feature;

[0202] The text weighted feature vector and the multi-head attention feature are enhanced and calculated through the feedforward neural network sublayer to obtain text enhancement features.

[0203] By performing attention calculations at the encoding layer, we can capture deep features in text sentences, enrich the representation capabilities of features, and improve the generalization ability of the model.

[0204] In some optional implementations, the evaluation module 505 includes a credibility evaluation submodule, a completeness evaluation submodule, a compliance evaluation submodule, and a comprehensive submodule, wherein:

[0205] The credibility evaluation submodule is used to evaluate the credibility of the candidate answers based on the knowledge base query results to obtain a credibility score;

[0206] The completeness evaluation submodule is used to evaluate the completeness of the candidate answer based on the enhanced query statement to obtain a completeness score;

[0207] The compliance evaluation submodule is used to evaluate the compliance of the candidate answers according to the preset compliance evaluation rules to obtain a compliance score;

[0208] The comprehensive submodule is used to perform weighted calculation on the credibility score, the integrity score and the compliance score to obtain an evaluation score.

[0209] Evaluating answers can help operations staff trace problems more quickly and supplement knowledge, thereby improving the quality and accuracy of answers.

[0210] To solve the above technical problems, the present application also provides a computer device. Figure 6 , Figure 6 This is a basic structural block diagram of the computer device in this embodiment.

[0211] The computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 6 with components 61-63, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0212] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0213] The memory 61 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 61 can be an internal storage unit of the computer device 6, such as a hard disk or memory of the computer device 6. In other embodiments, the memory 61 can also be an external storage device of the computer device 6, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 6. Of course, the memory 61 can also include both the internal storage unit of the computer device 6 and its external storage device. In this embodiment, the memory 61 is generally used to store the operating system and various application software installed on the computer device 6, such as computer-readable instructions for the output method of the search enhanced generative question answering, etc. In addition, the memory 61 can also be used to temporarily store various types of data that have been output or are to be output.

[0214] The processor 62 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 62 is generally used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to run the computer-readable instructions stored in the memory 61 or process data, such as running the computer-readable instructions of the output method of the retrieval enhanced generative question answering.

[0215] The network interface 63 may include a wireless network interface or a wired network interface. The network interface 63 is generally used to establish a communication connection between the computer device 6 and other electronic devices.

[0216] This embodiment implements the steps of the output method of retrieving enhanced generative question and answer as in the above embodiment when the processor executes computer-readable instructions stored in the memory. The candidate answers are obtained according to the query question by using the first largest language model and the second largest language model in combination with the preset knowledge base, and the candidate answers are then evaluated, which can improve the efficiency and accuracy of answer acquisition and thus improve the quality of the answers.

[0217] The present application also provides another implementation mode, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor so that the at least one processor performs the steps of the output method of the retrieval enhanced generative question and answer as described above, and obtains candidate answers according to the query question through the first large language model and the second large language model in combination with a preset knowledge base, and then evaluates the candidate answers, which can improve the efficiency and accuracy of answer acquisition, thereby improving the quality of the answer.

[0218] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0219] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A method for outputting retrieval-enhanced generative question answering, characterized in that: The steps include: Receive a query question input by a user, and obtain historical conversation content corresponding to the query question; Inputting the query question and the historical conversation content into the trained first language model for integration to obtain an enhanced query statement; Generate a knowledge base query statement based on the enhanced query statement, and query a preset knowledge base according to the knowledge base query statement to obtain a knowledge base query result; Inputting the knowledge base query result and the enhanced query statement into the trained second language model, and outputting candidate answers corresponding to the query question; Evaluate the candidate answers according to the knowledge base query result and the enhanced query statement to obtain an evaluation score; The candidate answers are screened according to the evaluation scores, and a target answer is output.

2. The method for outputting retrieval-enhanced generative question-answering according to claim 1, characterized in that: The first language model includes a first embedding layer, a multi-level feature extraction layer, a head entity recognition layer, a first attention layer, and a relation and tail entity joint recognition layer; The step of inputting the query question and the historical conversation content into the trained first language model for integration to obtain an enhanced query statement comprises: Inputting the query question and the historical conversation content into the first embedding layer for vector conversion to obtain a sentence embedding vector; Performing feature extraction on the sentence embedding vector through the multi-level feature extraction layer to obtain multi-level text features; Input the multi-level text features into the head entity recognition layer for recognition, and output all head entities; Performing self-attention fusion calculation on all the head entities and the multi-level text features through the first attention layer, obtaining a head entity tuple relationship matrix; Input the head entity tuple relationship matrix into the relationship and tail entity joint recognition layer for recognition, and obtain all specific relationships and tail entities of each head entity; An entity relationship triple is obtained based on the head entity, the specific relationship and the tail entity, and an enhanced query statement is generated according to the entity relationship triple.

3. The method for outputting retrieval-enhanced generative question-answering according to claim 2, characterized in that: The multi-level feature extraction layer includes a bottom layer network unit, a middle layer network unit and a high layer network unit. The step of extracting features from the sentence embedding vector through the multi-level feature extraction layer to obtain multi-level text features includes: Extracting surface features of the sentence embedding vector through the bottom network unit, and inputting the extracted surface features into the middle network unit and the high-level network unit respectively; Extracting features from the surface features through the middle layer network unit to obtain syntactic features, and inputting the syntactic features into the high-level network unit; Extracting features from the surface features and the syntactic features through the high-level network unit to obtain semantic features; The surface features, the syntactic features and the semantic features are concatenated to obtain multi-level text features.

4. The output method of retrieval-enhanced generative question answering according to claim 1, characterized in that: The step of generating a knowledge base query statement based on the enhanced query statement comprises: Performing word segmentation on the enhanced query statement to obtain a word segmentation result; Counting the frequency of each word in the preset knowledge base according to the word segmentation result, and determining the weight of each word according to the frequency; Extracting keywords in the enhanced query sentence based on the weight of each of the words, and determining extended words of the keywords; A knowledge base query statement corresponding to the enhanced query statement is generated according to the keyword and the extended word.

5. The method for outputting retrieval-enhanced generative question-answering according to claim 1, characterized in that: The second language model includes a knowledge layer, a second embedding layer, a selection mechanism layer, a visible layer, an encoding layer, a CRF layer and an output layer. The step of inputting the knowledge base query result and the enhanced query statement into the trained second language model and outputting the candidate answer corresponding to the query question includes: Injecting the knowledge of the knowledge base query result into the enhanced query statement through the knowledge layer to form a sentence tree, and inputting the sentence tree into the second embedding layer and the visible layer respectively; Performing position embedding on the sentence tree through the second embedding layer to obtain a sentence position encoding vector; Performing weighted calculation on the sentence position encoding vector through the selection mechanism layer to obtain a text weighted feature vector; Constructing a text visible matrix of the sentence tree through the visible layer; Inputting the text weighted feature vector and the text visibility matrix into the encoding layer for attention calculation, and outputting text enhancement features; Input the text enhancement features into the CRF layer for classification, and output a text classification sequence; The text classification sequence is reorganized through the output layer to obtain the candidate answer.

6. The method for outputting retrieval-enhanced generative question-answering according to claim 5, characterized in that: The encoding layer includes a multi-head attention sublayer and a feedforward neural network sublayer; the step of inputting the text weighted feature vector and the text visibility matrix into the encoding layer for attention calculation and outputting the text enhancement feature includes: Inputting the text weighted feature vector and the text visible matrix into the multi-head attention sublayer to perform multi-head attention calculation to obtain a multi-head attention feature; The text weighted feature vector and the multi-head attention feature are enhanced and calculated through the feedforward neural network sublayer to obtain text enhancement features.

7. The method for outputting retrieval-enhanced generative question-answering according to claim 1, characterized in that: The step of evaluating the candidate answers according to the knowledge base query result and the enhanced query statement to obtain an evaluation score comprises: Evaluating the credibility of the candidate answers based on the knowledge base query results to obtain a credibility score; Evaluate the completeness of the candidate answer based on the enhanced query statement to obtain a completeness score; Evaluate the compliance of the candidate answers according to preset compliance evaluation rules to obtain a compliance score; The credibility score, the completeness score and the compliance score are weighted to obtain an evaluation score.

8. A retrieval-enhanced generative question-answering output device, characterized in that: include: An acquisition module, used to receive a query question input by a user and acquire historical conversation content corresponding to the query question; A refinement module, configured to input the query question and the historical conversation content into the trained first language model for integration to obtain an enhanced query statement; A query module, used to generate a knowledge base query statement based on the enhanced query statement, and query a preset knowledge base according to the knowledge base query statement to obtain a knowledge base query result; An answer generation module, used for inputting the knowledge base query result and the enhanced query sentence into the trained second language model, and outputting candidate answers corresponding to the query question; An evaluation module, used to evaluate the candidate answers according to the knowledge base query result and the enhanced query statement to obtain an evaluation score; The screening module is used to screen the candidate answers according to the evaluation scores and output a target answer.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the output method of the retrieval-enhanced generative question answering as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the method for outputting retrieval-enhanced generative question-answering according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Method and device for updating question-answer knowledge base

    CN109033262A

  • Knowledge graph question and answer method based on improved Embedded KGQA model, electronic equipment and storage medium

    CN114936293A

  • Pretraining language model-based common sense question and answer method and system

    CN116842157A

  • Knowledge retrieval enhancement-based large language model question and answer method and device

    CN118113836A

  • Questions and answers generation

    US20110125734A1

Cited By

  • Domain speech recognition method and system based on RAG

    CN119296516A

  • Knowledge graph-based large model answer effect evaluation method and system

    CN119537561A

  • Enterprise risk identification method and device based on retrieval enhancement framework

    CN119558667A

  • Multi-hop agricultural question-answering system based on knowledge graph reasoning and large model and text fragment retrieval and answer generation method thereof

    CN119782467A

  • Multi-hop agricultural question answering system based on knowledge graph reasoning and large model and text segment retrieval and answer generation method thereof

    CN119782467B