Answer generation method and device based on retrieval enhancement generation model and related equipment

Abstract

The invention provides an answer generation method and device based on a retrieval enhancement generation model and related equipment, and relates to the technical field of computers.The method comprises the steps that a target question inquired by a user is obtained; inputting the target question into the retrieval enhancement generation model, and outputting at least one retrieval result associated with the target question; reversely generating at least one question corresponding to the retrieval result based on the at least one retrieval result associated with the target question; and determining a target answer corresponding to the target question according to the similarity between the question corresponding to each retrieval result and the target question. The accuracy of the question and answer result can be improved.

Description

Background Art

[0002] The retrieval-generation model combines the advantages of information retrieval and generative models. It retrieves relevant information from external knowledge bases and then generates more accurate and context-relevant answers. This method significantly improves the accuracy, reliability and knowledge coverage of the generative model when dealing with open-domain problems, making it of great significance in intelligent question and answering, dialogue systems and knowledge-intensive tasks.

[0003] Traditional self-continuous optimization of retrieval generation models mainly improves performance through rule optimization, supervised learning and feedback loops. Specific practices include keyword matching, query expansion, model fine-tuning based on labeled data and recycling of user feedback, while combining regular error analysis and domain fine-tuning to gradually optimize model performance. These methods rely on manual intervention and high-quality labeled data.

[0004] In the process of achieving self-sustaining optimization of the retrieval generation model, the difficulty and core point lies in how to realize the automated evaluation of the entire retrieval generation model chain process. Realizing the automated evaluation of the entire retrieval generation model chain process is a complex task, involving each link from retrieval to generation, which requires accurate evaluation and continuous optimization feedback. At present, some methods and technologies have been gradually developed for the automated evaluation of the entire process of the retrieval generation model chain, but they are still in the stage of continuous improvement in research and application.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The present disclosure provides a method, apparatus and related equipment for generating an answer based on a retrieval enhanced generation model, which at least to a certain extent improve the accuracy of answers in a question-answering system in the related art.

[0007] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0008] According to one aspect of the present disclosure, a method for generating an answer based on a retrieval enhancement generation model is provided, comprising: obtaining a target question of a user query; inputting the target question into the retrieval enhancement generation model, and outputting at least one retrieval result associated with the target question; reversely generating at least one question corresponding to the retrieval result based on at least one retrieval result associated with the target question; and determining a target answer corresponding to the target question based on the similarity between each question corresponding to the retrieval result and the target question.

[0009] In some exemplary embodiments of the present disclosure, based on the aforementioned scheme, at least one question corresponding to the retrieval result is reversely generated based on at least one retrieval result associated with the target question, including: inputting at least one retrieval result associated with the target question into a pre-trained reverse answer generation model, and outputting the question corresponding to the retrieval result.

[0010] In some exemplary embodiments of the present disclosure, based on the aforementioned scheme, the target answer corresponding to the target question is determined according to the similarity between the question corresponding to each of the retrieval results and the target question, including: using a cosine similarity calculation algorithm to calculate the similarity between the question corresponding to each of the retrieval results and the target question; based on a preset sorting rule, sorting the similarity between the question corresponding to each of the retrieval results and the target question to obtain a similarity sequence; and determining that the retrieval result corresponding to the maximum value in the similarity sequence is the target answer corresponding to the target question.

[0011] In some exemplary embodiments of the present disclosure, based on the aforementioned scheme, after determining the target answer corresponding to the target question according to the similarity between the question corresponding to each of the retrieval results and the target question, the method also includes: obtaining user feedback information on the target answer; performing weighted calculation on the feedback information of the target answer to obtain a user feedback evaluation index; and optimizing the retrieval enhancement generation model based on the user feedback evaluation index.

[0012] In some exemplary embodiments of the present disclosure, based on the aforementioned scheme, before inputting the target question into the retrieval enhancement generation model and outputting at least one retrieval result associated with the target question, the method also includes: rewriting the target question to obtain a rewritten target question.

[0013] In some exemplary embodiments of the present disclosure, based on the aforementioned scheme, before determining the target answer corresponding to the target question according to the similarity between the question corresponding to each of the retrieval results and the target question, the method also includes: performing vectorization processing on the question corresponding to each retrieval result and the target question to obtain a question vector and a target question vector corresponding to each retrieval result; and calculating the similarity between the question corresponding to each retrieval result and the target question based on the question vector and the target question vector corresponding to each retrieval result.

[0014] According to another aspect of the present disclosure, there is also provided an answer generation device based on a retrieval enhancement generation model, including: a question acquisition module, used to obtain a target question of a user query; a retrieval result output module, used to input the target question into the retrieval enhancement generation model, and output at least one retrieval result associated with the target question; a question generation module, used to reversely generate at least one question corresponding to the retrieval result based on at least one retrieval result associated with the target question; and an answer determination module, used to determine a target answer corresponding to the target question based on the similarity between the question corresponding to each retrieval result and the target question.

[0015] According to another aspect of the present disclosure, an electronic device is also provided, which includes: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned answer generation methods based on the retrieval enhancement generation model by executing the executable instructions.

[0016] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the answer generation method based on the retrieval enhancement generation model described above is implemented.

[0017] According to another aspect of the present disclosure, a computer program product is also provided, including a computer program, which, when executed by a processor, implements any of the above-mentioned answer generation methods based on the retrieval enhancement generation model.

[0018] The answer generation method, apparatus and related equipment based on the retrieval enhancement generation model provided in the embodiments of the present disclosure retrieves the question queried by the user to obtain at least one retrieval result, reversely generates the corresponding question for the retrieval result, calculates the similarity between the question corresponding to the retrieval result and the target question, and finally determines the target answer corresponding to the target question. Compared with the problem of low accuracy of question and answer results in the related art, the embodiments of the present disclosure reversely generates the question corresponding to the retrieval result associated with the target question, calculates the similarity between the question corresponding to the retrieval result and the target question, and determines the target answer corresponding to the target question, thereby improving the accuracy of the answer output by the retrieval enhancement generation model.

[0019] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0021] Figure 1 A schematic diagram showing a system structure of a method for generating an answer based on a RAG model in an embodiment of the present disclosure;

[0022] Figure 2 A flow chart of a method for generating an answer based on a RAG model in an embodiment of the present disclosure is shown;

[0023] Figure 3 A schematic diagram of a method for generating an answer based on a RAG model in an embodiment of the present disclosure is shown;

[0024] Figure 4 A schematic diagram showing determination of evaluation indicators for an answer generation method based on a RAG model in an embodiment of the present disclosure is shown;

[0025] Figure 5 A schematic diagram of a RAG full-process self-assessment system in an embodiment of the present disclosure is shown;

[0026] Figure 6 A schematic diagram of an answer generation device based on a RAG model in an embodiment of the present disclosure is shown;

[0027] Figure 7 A structural block diagram of an electronic device using a method for generating an answer based on a RAG model in an embodiment of the present disclosure is shown; DETAILED DESCRIPTION

[0028] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0029] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0030] For ease of understanding, before introducing the embodiments of the present disclosure, several terms involved in the embodiments of the present disclosure are first explained as follows:

[0031] Retrieval-Augmented Generation (RAG): is a model architecture that combines information retrieval and text generation. It enhances the output of the generative model by retrieving relevant information (such as documents or data fragments), thereby improving the accuracy and relevance of the answer. RAG is often used in question-answering systems and dialogue generation, leveraging information from external knowledge bases to support the generation of richer and more accurate content.

[0032] Large Language Model (LLM): refers to a language model trained using deep learning techniques, which is usually very large in size, containing hundreds of millions to trillions of parameters. These models are trained with large amounts of text data and are able to understand and generate natural language. LLMs perform well in a variety of natural language processing tasks, including text generation, translation, question-answering, dialogue systems, etc. Common examples of LLMs include OpenAI's GPT series, Google's BERT and T5, etc.

[0033] Elasticsearch: is an open source distributed search engine based on Lucene that supports real-time data search and analysis. It is known for its powerful full-text search capabilities and efficient data indexing, and is often used for log analysis, data storage, and large-scale query processing. Elasticsearch adopts a distributed architecture, is easy to expand, and provides a simple integration method through RESTful API. It is an ideal choice for building search and analysis solutions.

[0034] Facebook AI Similarity Search (FAISS) is an open source library developed by the Facebook AI Research team for efficient similarity search and dense vector retrieval. It can quickly process large-scale vector data and find the most similar items between these vectors. It is an important tool for high-dimensional data search.

[0035] Query: A request sent by a user to a database or search engine to retrieve, update or operate data. It describes the information the user wants to obtain or the operation to be performed through structured statements or keywords. In databases, SQL statements are often used for queries, while in search engines, keywords or complex search conditions are used to find relevant content. Query is a core tool for data management and information retrieval, and is widely used in data analysis, information search, and data operations in various applications.

[0036] Term Frequency-Inverse Document Frequency (TF-IDF) is a statistical method for evaluating the importance of words in a text. It measures the importance of a word in a document by calculating term frequency (TF) and inverse document frequency (IDF): term frequency indicates the frequency of a word in a document, and inverse document frequency measures the rarity of a word in the entire corpus. TF-IDF reduces the weight of common words and increases the weight of rare words, thereby more accurately representing text features.

[0037] Latent Semantic Analysis (LSA) is a technology used in natural language processing. It discovers the latent semantic relationship between words by mathematically decomposing text (such as singular value decomposition, SVD). LSA maps documents and words into a low-dimensional semantic space, making semantically similar words and documents closer in the space, thereby effectively handling synonyms and polysemy problems. It is often used in information retrieval and text analysis.

[0038] Latent Dirichlet Allocation (LDA) is a topic model used to automatically discover hidden topics from large amounts of text data. It represents documents as a mixture of multiple topics and each topic as a probability distribution of a set of words. In this way, LDA can reveal the topic structure between documents and the semantic associations between words, and is widely used in text classification, information retrieval, and natural language processing.

[0039] The Bidirectional Encoder Representations from Transformers (BERT) model is a pre-trained language model developed by Google. It understands the context of text through bidirectional learning. Unlike traditional models, BERT considers the information before and after the words at the same time, so as to more accurately capture the semantic relationship between words. BERT performs well in multiple natural language processing tasks (such as text classification and question-answering systems) and is an important breakthrough in the field of NLP.

[0040] The specific implementation of the embodiment of the present disclosure is described in detail below with reference to the accompanying drawings.

[0041] Figure 1 FIG. 1 shows an exemplary application system architecture diagram to which the answer generation method based on the retrieval enhancement generation model in the embodiment of the present disclosure can be applied. Figure 1 As shown, the system architecture may include a terminal device 101 , a network 102 and a server 103 .

[0042] The network 102 is a medium for providing a communication link between the terminal device 101 and the server 103, and can be a wired network or a wireless network.

[0043] Optionally, the wireless network or wired network described above uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or any combination of a virtual private network). In some embodiments, the data exchanged through the network is represented by technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPSec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above data communication technologies.

[0044] The terminal device 101 can be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, smart speakers, smart watches, wearable devices, augmented reality devices, virtual reality devices, etc.

[0045] Optionally, the client of the application installed in different terminal devices 101 is the same, or the client of the same type of application based on different operating systems. Based on the different terminal platforms, the specific form of the client of the application can also be different, for example, the application client can be a mobile client, a PC client, etc.

[0046] The server 103 may be a server that provides various services, such as a background management server that provides support for the device operated by the user using the terminal device 101. The background management server may analyze and process the received request and other data, and feed back the processing results to the terminal device.

[0047] Optionally, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0048] Those skilled in the art will know that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration, and any number of terminal devices, networks and servers may be provided according to actual needs, and the embodiments of the present disclosure do not limit this.

[0049] Under the above system architecture, an embodiment of the present disclosure provides an answer generation method based on a retrieval enhancement generation model, which can be executed by any electronic device with computing and processing capabilities.

[0050] In some embodiments, the answer generation method based on the retrieval enhanced generation model provided in the embodiments of the present disclosure can be executed by a terminal device of the above-mentioned system architecture; in other embodiments, the answer generation method based on the retrieval enhanced generation model provided in the embodiments of the present disclosure can be executed by a server in the above-mentioned system architecture; in other embodiments, the answer generation method based on the retrieval enhanced generation model provided in the embodiments of the present disclosure can be implemented by the terminal device and the server in the above-mentioned system architecture through interaction.

[0051] Figure 2 A flow chart of a method for generating an answer based on a retrieval enhancement generation model in an embodiment of the present disclosure is shown as follows: Figure 2 As shown, the answer generation method based on the retrieval enhancement generation model provided in the embodiment of the present disclosure includes the following steps:

[0052] S202, obtaining the target question queried by the user.

[0053] It should be noted that the target question in the embodiments of the present disclosure can be any question that a user needs to query using natural language. In some embodiments, it can be a question asked by a user on various search platforms. For example, the target question can be a question asked by a user, such as "Who is the winner of the Nobel Prize in Physics in 2023?"

[0054] S204: Input the target question into the retrieval enhancement generation model, and output at least one retrieval result associated with the target question.

[0055] It should be noted that the retrieval-augmented generation (RAG) model in the embodiment of the present disclosure combines a language model and information retrieval technology. Specifically, when the retrieval-augmented generation model needs to generate text or answer questions, it will first retrieve relevant information from a large document collection, and then use the retrieved information to guide the generation of text retrieval results. For example, when a user asks "Who is the winner of the 2023 Nobel Prize in Physics", the retrieval result output by the retrieval-augmented generation model may be "The winners of the 2023 Nobel Prize in Physics are A, B, and C".

[0056] S206: Reversely generate at least one question corresponding to the retrieval result based on at least one retrieval result associated with the target question.

[0057] It should be noted that the method used in the embodiments of the present disclosure to reversely generate questions corresponding to the search results may be to use a pre-trained language model (such as the GPT series, T5, etc.) to generate possible questions based on the selected search results, or to adjust the parameters of the retrieval enhancement generation model to obtain multiple versions of questions with different expressions but the same semantics, etc., which is not limited in the embodiments of the present disclosure.

[0058] S208, determining a target answer corresponding to the target question according to the similarity between the question corresponding to each search result and the target question.

[0059] It should be noted that in the embodiment of the present disclosure, the similarity between the question corresponding to each search result and the target question is obtained by the cosine value of the angle between the question corresponding to each search result and the target question, and the range is [-1, 1]. The closer the value is to 1, the more similar they are. The more similar the question corresponding to the search result is to the target question, the more accurate the corresponding search result is.

[0060] In some embodiments, the answer generation method based on the retrieval enhancement generation model in the embodiments of the present disclosure first obtains the target question of the user query; secondly, inputs the target question into the retrieval enhancement generation model, and outputs at least one retrieval result associated with the target question; then, based on at least one retrieval result associated with the target question, reversely generates at least one question corresponding to the retrieval result; finally, according to the similarity between the question corresponding to each retrieval result and the target question, determines the target answer corresponding to the target question. Compared with the problem of low accuracy of question and answer results in related technologies, the embodiments of the present disclosure improve the accuracy of the answer output by the retrieval enhancement generation model by reversely generating questions corresponding to the retrieval results associated with the target question, and calculating the similarity between the question corresponding to the retrieval result and the target question, and determining the target answer corresponding to the target question.

[0061] In some embodiments, Figure 3 As shown, the answer generation method based on the retrieval enhancement generation model in the embodiment of the present disclosure specifically includes the following steps:

[0062] S302, obtaining a large amount of high-quality raw data.

[0063] In some embodiments, building the entire RAG model requires a large amount of high-quality raw data. The data types may include structured data, unstructured text and other data. After data collection, data cleaning and preprocessing are required to remove noise and duplicates, and format the data to ensure high data quality and consistency. The processed data is then segmented according to the retrieval construction rules to obtain key-value pairs.

[0064] S304: construct an index for the key of the key-value pair, convert it into vector data and store it in a vector database or other search engine.

[0065] In some embodiments, the disclosed embodiments construct an index for the key of the key-value pair, convert it into vector data and store it in a vector database or other retrieval engine, such as Elasticsearch, FAISS, etc. By building an efficient index structure, it is possible to quickly query and retrieve massive data. At this stage, the RAG model will retrieve relevant information from the knowledge base based on the user's input or query, and provide contextual support for the output of the RAG model.

[0066] S306, the generation part of the RAG model architecture. Based on the relevant information of the target question retrieved by the retrieval part of the RAG model, the generation part (usually a large language model, such as the GPT series) will use this information to generate answers. The generation part combines the user input and the retrieved data to generate answers that fit the context through natural language processing technology.

[0067] In some embodiments, this step in the disclosed embodiments is the core link in the entire process, ensuring that the generated answer is both grammatically correct and logically coherent.

[0068] S308, model output and user interaction. After the generation part of the RAG model is processed, the output results are presented to the user. The model output includes not only the generated text, but also the related information sources associated with it to improve the transparency and interpretability of the output content. The user interacts with the system through the interface, receives the results and provides feedback.

[0069] S310, user feedback collection and processing. During the interaction with the RAG model system, users may provide feedback or opinions on the output results. By collecting user feedback, the accuracy, relevance and user satisfaction of the model output can be evaluated. These feedbacks will become an important reference for subsequent system optimization, used to measure the performance of the current model and find possible improvements.

[0070] S312, Log management and data analysis. Throughout the process, the RAG model system will record every operation and system behavior through log management tools, including retrieval requests, response time of model generation, user feedback and other data. These log data will be stored in the log management system, and the log data will be mined and analyzed through data analysis tools to obtain insights into system performance, user behavior and model effects.

[0071] S314, RAG full-process self-evaluation system. In order to achieve automated process optimization, the RAG model system will also introduce a RAG self-evaluation mechanism, which will quantitatively analyze the model generation quality and retrieval accuracy through pre-set evaluation indicators to ensure that each link meets the set standards.

[0072] S316, model operation management and data accumulation. The model operation management system will comprehensively analyze user feedback, log data, model performance evaluation results, etc. to form a closed-loop operation data system. By accumulating this data, it continuously adjusts and optimizes each link of the large model, including retrieval algorithms, generation mechanisms, and the overall system architecture.

[0073] S318, continuous optimization and model fine-tuning. After accumulating a certain amount of data, enter the continuous optimization and model fine-tuning stage. By reintroducing operational data and user feedback data into the model training process, the model is trained again to improve the model's generation ability and accuracy. With the continuous addition of new data and model fine-tuning, the model will gradually improve its performance in different scenarios and better adapt to user needs.

[0074] In some embodiments, the embodiments of the present disclosure reversely generate at least one question corresponding to the retrieval result based on at least one retrieval result associated with the target question, including: inputting at least one retrieval result associated with the target question into a pre-trained reverse answer generation model, and outputting the question corresponding to the retrieval result.

[0075] In some embodiments, the embodiments of the present disclosure determine the target answer corresponding to the target question based on the similarity between the question corresponding to each retrieval result and the target question, including: using a cosine similarity calculation algorithm to calculate the similarity between the question corresponding to each retrieval result and the target question; based on a preset sorting rule, sorting the similarity between the question corresponding to each retrieval result and the target question to obtain a similarity sequence; determining the retrieval result corresponding to the maximum value in the similarity sequence as the target answer corresponding to the target question.

[0076] In some embodiments, after determining the target answer corresponding to the target question based on the similarity between the question corresponding to each retrieval result and the target question, the answer generation method based on the retrieval enhancement generation model in the embodiments of the present disclosure also includes: obtaining user feedback information on the target answer; performing weighted calculation on the feedback information of the target answer to obtain a user feedback evaluation index; and optimizing the retrieval enhancement generation model based on the user feedback evaluation index.

[0077] In some embodiments, after the RAG model outputs the results, the embodiments of the present disclosure will present the output results to the user. The RAG model output not only includes the generated target answer, but may also include relevant information sources associated with it to enhance the transparency and interpretability of the output content. The user can interact with the output results of the RAG model through the interface, receive results and provide feedback.

[0078] In some embodiments, during the interaction with the RAG model, users may provide feedback or opinions on the output results. By collecting user feedback, the accuracy, relevance and user satisfaction of the model output can be evaluated. These feedbacks will become an important reference for subsequent model optimization, used to measure the performance of the current model and find possible improvement points.

[0079] In some embodiments, before the target question is input into the retrieval enhancement generation model and at least one retrieval result associated with the target question is output, the answer generation method based on the retrieval enhancement generation model in the embodiment of the present disclosure also includes: rewriting the target question to obtain a rewritten target question.

[0080] In some embodiments, before determining the target answer corresponding to the target question based on the similarity between the question corresponding to each retrieval result and the target question, the answer generation method based on the retrieval enhancement generation model in the embodiments of the present disclosure also includes: vectorizing the question corresponding to each retrieval result and the target question to obtain a question vector and a target question vector corresponding to each retrieval result; and calculating the similarity between the question corresponding to each retrieval result and the target question based on the question vector and the target question vector corresponding to each retrieval result.

[0081] In some embodiments, the RAG model in the embodiments of the present disclosure will record each operation and system behavior through a log management tool, including retrieval requests, response time for generating models, user feedback and other data. These log data will be stored in the log management system, and the log data will be mined and analyzed through data analysis tools to obtain insights into model performance, user behavior, and model effects.

[0082] In some embodiments, the disclosed embodiments can also perform full-process self-evaluation of the RAG model. Specifically, in order to achieve automated process optimization, the system will also introduce a RAG self-evaluation mechanism. Through pre-set evaluation indicators, the generation quality and retrieval accuracy of the model are quantitatively analyzed to ensure that each link meets the set standards.

[0083] In some embodiments, Figure 4 As shown, the automatic evaluation indicators of the retrieval enhancement generation model in the embodiment of the present disclosure cover all links of the whole process of RAG, including question rewriting, retrieval part, and generation part. These indicators are obtained through regression of user feedback or comparison of online links. All indicators can be obtained without manual participation in annotation. Since each link of the whole process of RAG is closely linked, these evaluation indicators are also closely related. These indicators include: evaluation indicators of question rewriting, evaluation indicators of retrieval part, and evaluation indicators of generation part. Among them, the evaluation indicators of question rewriting include:

[0084] 1. Accuracy: The semantic similarity between the rewritten result and the original query. The methods for calculating the semantic similarity of short texts mainly include three methods based on vocabulary, statistics, and deep learning. The vocabulary method uses dictionaries and word vector models (such as TF-IDF) to compare the overlap of words; the statistical method uses topic models (such as LSA and LDA) to analyze the topic structure of text; the deep learning method uses pre-trained language models (such as BERT and GPT) to convert text into high-dimensional vectors, and evaluates semantic similarity by calculating cosine similarity or Euclidean distance. In this scenario, the last method is generally used.

[0085] 2. Completeness: Extract entities and concepts from the rewritten query and compare them with the entities and concepts of the original query to ensure that key information is not lost. Common methods include rule matching, named entity recognition (NER), and deep learning-based models. Rule matching relies on predefined dictionaries and patterns to recognize entities. NER uses machine learning or deep learning models (such as BERT-NER) to extract entity information. Deep learning-based methods can understand more complex concepts and semantics through context.

[0086] 3. Query effect: measured by the scores of the original query and the rewritten query after refined ranking. This indicator needs to be calculated in combination with the results of the retrieval part evaluation indicators, and is obtained by subtracting the support rate calculation results of the original query retrieval results and the support rate calculation results of the rewritten query retrieval results.

[0087] In some embodiments, Figure 4 As shown, the evaluation indicators of the retrieval part in the embodiment of the present disclosure include: Support rate: The support rate is used to measure how many of the results returned by the retrieval part from the knowledge base have a direct correlation or support relationship with the final generated answer (or the answer expected by the user). The calculation of the support rate needs to be reversed through the user feedback evaluation indicator. Support rate = fidelity of the generation module × user feedback evaluation indicator × 100%.

[0088] In some embodiments, Figure 4 As shown, the evaluation indicators of the generation part in the embodiment of the present disclosure include: fidelity and the correlation between the question and the generated answer, where fidelity (based on retrieval): fidelity evaluates whether the generated response is accurate, and is obtained by calculating the correlation between the reply generated by the large model and the context obtained by retrieval based on the retrieved document. It ensures that the response does not contain hallucinations or wrong information. The embodiment of the present disclosure uses cosine similarity to calculate the correlation between the large model reply and the retrieved content. The calculation formula is as follows:

[0089]

[0090] Among them, FID refers to the retrieval-based large model reply fidelity index; A is the vector representation of the reply generated by the large model; B is the vector representation of the retrieved context; n is the dimension of the vector, that is, the number of elements contained in each vector; i is the index variable; A i is the i-th element of vector A; B i is the i-th element of vector B.

[0091] The models commonly used in the embodiments of the present disclosure to convert text into vectors include Word2Vec, GloVe, BERT, and Sentence Transformers.

[0092] In some embodiments, Figure 3 As shown, the correlation between the question and the generated answer in the embodiment of the present disclosure refers to generating multiple questions by reversely generating multiple questions through the answer, calculating the similarity between the multiple questions and the original question, and the correlation between the question and the generated answer can be calculated by cosine similarity, and the specific method is as follows:

[0093]

[0094] Among them, QG is the correlation between the question and the generated answer; Q is the vector representation of the generated answer, and the result range is between [-1, 1]. The closer the value is to 1, the higher the correlation is; Q i is the i-th element of vector Q.

[0095] In some embodiments, the user's feedback information on the target answer in the embodiments of the present disclosure includes explicit feedback (such as likes and dislikes) and implicit feedback (such as evaluations inferred through text sentiment analysis), wherein the likes in the explicit feedback are clicks when the user expresses satisfaction or approval of the generated content, which is generally regarded as positive feedback, indicating that the content generated by the model meets the user's expectations; the dislikes are clicks when the user expresses dissatisfaction or disappointment with the generated content, which is negative feedback, indicating that the content generated by the model does not meet the user's needs or contains errors.

[0096] In some embodiments, the embodiments of the present disclosure can intuitively measure user satisfaction with model-generated content by counting the number of likes and dislikes. This explicit feedback can quickly reflect the user's attitude towards the generated results and is an important indicator for evaluating content quality.

[0097] In some embodiments, in order to obtain richer user feedback information, the disclosed embodiments may perform text sentiment analysis on user comments, suggestions or questions. Sentiment analysis can automatically identify the sentiment tendency in user comments and determine whether the feedback content is a positive (positive, satisfied) or negative (negative, dissatisfied) evaluation, wherein positive sentiment refers to words and sentences contained in user comments that express affirmation, praise or gratitude for the generated content; negative sentiment refers to user comments that contain criticism, dissatisfaction or complaints about the generated content. The disclosed embodiments combine explicit feedback (thumbs-up and thumbs-down) with implicit feedback (sentiment analysis) to more comprehensively understand the user's true attitude towards the generated content. Implicit feedback makes up for the shortcomings of explicit feedback, can capture more emotional details of the user, and help optimize the model more accurately.

[0098] In some embodiments, the disclosed embodiments may also combine explicit feedback (likes and dislikes) and implicit feedback (sentiment analysis scores) through final user feedback evaluation in a weighted manner to form a comprehensive score. The following is an example formula that can perform weighted calculation of the overall evaluation according to the importance of different feedback.

[0099] In order to combine explicit feedback (likes and dislikes) and implicit feedback (sentiment analysis) to evaluate the content generated by the large model, a weighted formula can be used to combine the two for quantitative evaluation. The following is an example formula that can weight the overall evaluation according to the importance of different feedback:

[0100]

[0101] Among them, UF refers to the comprehensive score of user feedback; U up Indicates the number of likes (positive explicit feedback); U down Indicates the number of taps (negative explicit feedback); S pos Represents the evaluation score identified as positive in sentiment analysis (implicit feedback); S neg Represents the evaluation score identified as negative in sentiment analysis (implicit feedback); Used to calculate the proportional difference of explicit feedback. The result range is between [-1, 1]. Positive values ​​indicate more positive feedback, and negative values ​​indicate more negative feedback. Used to calculate the proportional difference of implicit feedback. The result range is also between [-1, 1]. A positive value indicates that the sentiment analysis is biased towards positive, and a negative value indicates that it is biased towards negative. 1 and ω 2 is the weight of explicit feedback and implicit feedback, satisfying ω 1 +ω 2 =1, the relative importance of the two can be adjusted according to business needs.

[0102] Through this weighted formula, we can comprehensively consider the user's explicit behavior (likes and dislikes) and implicit emotional expression to obtain a more comprehensive and accurate user feedback evaluation, providing data support for the continuous optimization of the large model.

[0103] The calculation formula of the evaluation index of the retrieval part is obtained by inverting the evaluation index of the generation part as follows:

[0104]

[0105] Among them, SR refers to the support rate.

[0106] The calculation formula of the query effect index is as follows:

[0107] QE=SR post -SR pre (5)

[0108] Among them, QE represents the query effect index, SR post Represents the support rate after query rewriting, SR pre Indicates the support rate before query rewriting.

[0109] The above evaluation indicators cover all links of the entire RAG chain and realize automatic evaluation without manual labeling. These evaluation indicators are closely linked through the upper and lower relationships to form an interlocking system. Through user feedback, the quantitative evaluation results of other links can be reversed to obtain data support for the optimization of the entire RAG process. This automated evaluation method not only improves the evaluation efficiency, but also enhances the reliability and accuracy of the evaluation results.

[0110] It should be noted that if Figure 5 As shown, the entire process in the disclosed embodiment is a cyclic iterative process. Through continuous accumulation of operational data and user feedback, the model is regularly retrained and optimized to achieve self-sustaining optimization of the model. In each iteration, the accuracy of the retrieval and generation links is continuously improved, and the user experience is optimized based on user feedback to better meet user needs. Through such a closed-loop process from data to model output to user feedback, combined with the multi-dimensional support of modules such as log management analysis, RAG self-assessment, and model monitoring, the large model operation management system can efficiently accumulate operational data and gradually achieve continuous optimization and evolution of the large model.

[0111] Based on the same inventive concept, the embodiment of the present disclosure also provides an answer generation device based on a retrieval enhancement generation model, as described in the following embodiment. Since the principle of solving the problem in the embodiment of the device is similar to that in the above method embodiment, the implementation of the embodiment of the device can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0112] Figure 6 A schematic diagram of an answer generation device based on a retrieval enhancement generation model in an embodiment of the present disclosure is shown. Figure 6 As shown, the device comprises:

[0113] The question acquisition module 601 is used to acquire the target question queried by the user;

[0114] A retrieval result output module 602 is used to input the target question into the retrieval enhancement generation model and output at least one retrieval result associated with the target question;

[0115] A question generation module 603, configured to reversely generate at least one question corresponding to the search result based on at least one search result associated with the target question;

[0116] The answer determination module 604 is used to determine the target answer corresponding to the target question according to the similarity between the question corresponding to each search result and the target question.

[0117] In some embodiments, the answer generation device based on the retrieval enhancement generation model in the embodiments of the present disclosure obtains the target question of the user query through the question acquisition module; inputs the target question into the retrieval enhancement generation model through the retrieval result output module, and outputs at least one retrieval result associated with the target question; reversely generates at least one question corresponding to the retrieval result based on at least one retrieval result associated with the target question through the question generation module; and determines the target answer corresponding to the target question according to the similarity between the question corresponding to each retrieval result and the target question through the answer determination module. Compared with the problem of low accuracy of question and answer results in the related art, the embodiments of the present disclosure improve the accuracy of the answer output by the retrieval enhancement generation model by reversely generating questions corresponding to the retrieval results associated with the target question, and calculating the similarity between the question corresponding to the retrieval result and the target question, and determining the target answer corresponding to the target question.

[0118] In some embodiments, the question generation module in the embodiments of the present disclosure is also used to input at least one retrieval result associated with the target question into a pre-trained inverse answer generation model, and output the question corresponding to the retrieval result.

[0119] In some embodiments, the answer determination module in the embodiments of the present disclosure is also used to calculate the similarity between the question corresponding to each retrieval result and the target question using a cosine similarity calculation algorithm; based on a preset sorting rule, the similarity between the question corresponding to each retrieval result and the target question is sorted to obtain a similarity sequence; and the retrieval result corresponding to the maximum value in the similarity sequence is determined as the target answer corresponding to the target question.

[0120] In some embodiments, the answer generation device based on the retrieval enhancement generation model in the embodiments of the present disclosure also includes: a feedback information acquisition module, which is used to obtain user feedback information on the target answer after determining the target answer corresponding to the target question based on the similarity between the question corresponding to each retrieval result and the target question; a feedback evaluation index determination module, which is used to perform weighted calculation on the feedback information of the target answer to obtain the user feedback evaluation index; and a model optimization module, which is used to optimize the retrieval enhancement generation model based on the user feedback evaluation index.

[0121] In some embodiments, the answer generation device based on the retrieval enhancement generation model in the embodiments of the present disclosure also includes: a question rewriting module, which is used to rewrite the target question before inputting the target question into the retrieval enhancement generation model and outputting at least one retrieval result associated with the target question to obtain a rewritten target question.

[0122] In some embodiments, the answer generation device based on the retrieval enhancement generation model in the embodiments of the present disclosure also includes: a vectorization processing module, which is used to perform vectorization processing on the question corresponding to each retrieval result and the target question before determining the target answer corresponding to the target question based on the similarity between the question corresponding to each retrieval result and the target question, so as to obtain the question vector and the target question vector corresponding to each retrieval result; a similarity calculation module is used to calculate the similarity between the question corresponding to each retrieval result and the target question based on the question vector and the target question vector corresponding to each retrieval result.

[0123] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above method embodiments. It should be noted that the above modules as part of the device can be executed in a computer system such as a set of computer executable instructions.

[0124] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0125] Refer to the following Figure 7 An electronic device 700 according to this embodiment of the present disclosure is described. Figure 7 The electronic device 700 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0126] like Figure 7As shown, the electronic device 700 is in the form of a general computing device. The components of the electronic device 700 may include but are not limited to: at least one processing unit 710, at least one storage unit 720, and a bus 730 connecting different system components (including the storage unit 720 and the processing unit 710).

[0127] The storage unit stores program codes, which can be executed by the processing unit 710, so that the processing unit 710 performs the steps of various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 710 can perform the following steps of the above method embodiment:

[0128] Obtain a target question of a user query; input the target question into a retrieval enhancement generation model, and output at least one retrieval result associated with the target question; reversely generate at least one question corresponding to the retrieval result based on at least one retrieval result associated with the target question; and determine a target answer corresponding to the target question based on the similarity between the question corresponding to each retrieval result and the target question.

[0129] The storage unit 720 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 7201 and / or a cache storage unit 7202 , and may further include a read-only storage unit (ROM) 7203 .

[0130] The storage unit 720 may also include a program / utility 7204 having a set (at least one) of program modules 7205, such program modules 7205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0131] Bus 730 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0132] The electronic device 700 may also communicate with one or more external devices 740 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 700, and / or communicate with any device that enables the electronic device 700 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 750. Furthermore, the electronic device 700 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 760. As shown, the network adapter 760 communicates with other modules of the electronic device 700 via a bus 730. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0133] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0134] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart may be implemented as a computer program product, which includes: a computer program, which implements the method in the above embodiment when executed by a processor.

[0135] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. A program product capable of implementing the above method of the present disclosure is stored thereon. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above “Exemplary Method” section of this specification.

[0136] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0137] In the present disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0138] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0139] In a specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0140] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0141] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0142] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0143] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A method for generating answers based on a retrieval-enhanced generation model, characterized in that: include: Get the target question of the user's query; Inputting the target question into the retrieval enhancement generation model, and outputting at least one retrieval result associated with the target question; Reversely generate at least one question corresponding to the search result based on at least one search result associated with the target question; According to the similarity between the question corresponding to each of the retrieval results and the target question, a target answer corresponding to the target question is determined.

2. The answer generation method based on the retrieval enhanced generation model according to claim 1, characterized in that: Reversely generating at least one question corresponding to the search result based on at least one search result associated with the target question, comprising: At least one retrieval result associated with the target question is input into a pre-trained inverse answer generation model, and the question corresponding to the retrieval result is output.

3. The answer generation method based on the retrieval enhanced generation model according to claim 1, characterized in that: Determining a target answer corresponding to the target question according to the similarity between the question corresponding to each of the search results and the target question includes: Using a cosine similarity calculation algorithm, calculate the similarity between the question corresponding to each of the search results and the target question; Based on a preset sorting rule, sorting the similarities between the question corresponding to each of the search results and the target question to obtain a similarity sequence; Determine that the search result corresponding to the maximum value in the similarity sequence is the target answer corresponding to the target question.

4. The answer generation method based on the retrieval enhanced generation model according to claim 1, characterized in that: After determining a target answer corresponding to the target question according to the similarity between the question corresponding to each of the search results and the target question, the method further includes: Obtaining user feedback information on the target answer; Performing weighted calculation on the feedback information of the target answer to obtain a user feedback evaluation index; The retrieval enhancement generation model is optimized based on the user feedback evaluation index.

5. The answer generation method based on the retrieval enhanced generation model according to claim 1, characterized in that: Before inputting the target question into the retrieval enhancement generation model and outputting at least one retrieval result associated with the target question, the method further includes: The target problem is rewritten to obtain a rewritten target problem.

6. The answer generation method based on the retrieval enhanced generation model according to claim 1, characterized in that: Before determining a target answer corresponding to the target question according to the similarity between the question corresponding to each of the search results and the target question, the method further includes: Performing vectorization processing on the question corresponding to each search result and the target question to obtain a question vector and a target question vector corresponding to each search result; The similarity between the question corresponding to each retrieval result and the target question is calculated based on the question vector corresponding to each retrieval result and the target question vector.

7. An answer generation device based on a retrieval enhanced generation model, characterized in that: include: The question acquisition module is used to obtain the target question queried by the user; A retrieval result output module, used to input the target question into the retrieval enhancement generation model and output at least one retrieval result associated with the target question; A question generation module, configured to reversely generate at least one question corresponding to the search result based on at least one search result associated with the target question; The answer determination module is used to determine the target answer corresponding to the target question according to the similarity between the question corresponding to each of the retrieval results and the target question.

8. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the answer generation method based on the retrieval enhancement generation model as described in any one of claims 1 to 6 by executing the executable instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the answer generation method based on the retrieval enhancement generation model described in any one of claims 1 to 6 is implemented.

10. A computer program product comprising: A computer program or instruction, characterized in that when the computer program or instruction is executed by a processor, it implements the answer generation method based on the retrieval enhanced generation model as described in any one of claims 1 to 6.

Images