A method, apparatus, device, medium and product for generating response text

Abstract

The present invention discloses a method, apparatus, device, medium and product for generating answer texts. The method includes: obtaining a user question and inputting the user question into a question discriminator to obtain a retrieval type corresponding to the user question, where the question discriminator is obtained by iteratively training a text classification model through a target sample set, and the target sample set includes: question samples and text segments corresponding to the question samples; based on the retrieval type, retrieving in a target database according to the user question and sorting the retrieved target text segments to obtain a set of text segments corresponding to the user question; determining a model input text according to the set of text segments, the user question and an answer prompt word template, and inputting the model input text into a target model to obtain an answer text corresponding to the user question. Through the technical solution of the present invention, it is possible to accurately determine, by means of the question discriminator, the retrieval type suitable for the question raised by the user, improve the retrieval accuracy, and further improve the question-and-answer accuracy.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of large language models, and in particular, to a method, device, equipment, medium and product for generating answer texts. Background Art

[0002] Currently, technologies, applications and ecosystems related to large models have continuously become hot topics of concern. With the continuous improvement of the capabilities of general large language models and the continuous improvement and optimization of large model development frameworks, vertical applications, especially in the field of knowledge Q&A, have become more and more extensive. However, due to the high training time and resource costs of large models, the knowledge Q&A scenarios in vertical fields face the challenge that large models cannot quickly adapt to new knowledge and achieve timely updates of knowledge. To solve the above problems, RAG (Retrieval-Augmented Generation) has been proposed, which uses retrieved non-parametric knowledge to improve the performance of knowledge-intensive tasks, providing a good direction for solving problems such as "hallucinations" (answering off-topic), "timeliness" of knowledge, and private domain data of large models.

[0003] However, in knowledge Q&A using RAG technology, a core issue is how to improve the recall rate, thereby improving the Q&A accuracy rate, that is, how to find a limited number of text segments most relevant to the user's question from a large number of knowledge documents. Generally speaking, the recall rate can be improved by increasing the number of recall results. However, in actual Q&A applications, the number of recall results cannot be too large. One reason is that there are size limitations for the input window of large models (usually the number of Tokens is 4K, 16K, 32K, 128K), and appropriate text segmentation strategies and recall numbers need to be selected according to the size limitations of the input window of the selected model. The other reason is that even if the model context window is large enough, too many recall segments may introduce less relevant content, resulting in a decrease in the quality of the answer. Therefore, the optimal recall result is to cover the most relevant text segment content in as few recall segments as possible.

[0004] In summary, to improve the Q&A accuracy rate, there is an urgent need for a Q&A method that can combine the advantages of two retrieval methods. Summary of the Invention

[0005] The embodiments of the present invention provide a method, device, equipment, medium and product for generating answer texts to achieve improving the retrieval accuracy rate and thus the Q&A accuracy rate.

[0006] According to one aspect of the present invention, there is provided a method for generating answer texts, including:

[0007] Obtain the user's question, input the user's question into a question discriminator, and obtain the retrieval type corresponding to the user's question. The question discriminator is obtained by iteratively training a text classification model with a target sample set. The target sample set includes: question samples and text segments corresponding to the question samples;

[0008] Based on the retrieval type, retrieve in the target database according to the user's question, and sort the retrieved target text segments to obtain a set of text segments corresponding to the user's question;

[0009] Determine the model input text according to the set of text segments, the user's question, and the answer prompt word template, and input the model input text into the target model to obtain the answer text corresponding to the user's question.

[0010] According to another aspect of the present invention, there is provided an answer text generation device, which includes:

[0011] An acquisition and input module, configured to acquire the user's question, input the user's question into a question discriminator, and obtain the retrieval type corresponding to the user's question. The question discriminator is obtained by iteratively training a text classification model with a target sample set. The target sample set includes: question samples and text segments corresponding to the question samples;

[0012] A retrieval and sorting module, configured to retrieve in the target database according to the user's question based on the retrieval type, and sort the retrieved target text segments to obtain a set of text segments corresponding to the user's question;

[0013] A determination and input module, configured to determine the model input text according to the set of text segments, the user's question, and the answer prompt word template, and input the model input text into the target model to obtain the answer text corresponding to the user's question.

[0014] According to another aspect of the present invention, there is provided an electronic device, which includes:

[0015] At least one processor; and

[0016] A memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program executable by the at least one processor. The computer program is executed by the at least one processor so that the at least one processor can execute the answer text generation method according to any embodiment of the present invention.

[0018] According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for causing a processor to implement the answer text generation method according to any embodiment of the present invention when executed.

[0019] According to another aspect of the present invention, an embodiment of the present invention further provides a computer program product, which includes a computer program that implements the answer text generation method according to any embodiment of the present invention when executed by a processor.

[0020] In an embodiment of the present invention, a problem discriminator is obtained by iteratively training a text classification model with a problem sample set collected in advance and text segments corresponding to the problem sample set. During application, a user problem is obtained, and then the user problem is input into the problem discriminator. The problem discriminator is used to confirm the appropriate retrieval type for the user problem. Then, based on the retrieval type, a search is performed in a target database according to the user problem, and the retrieved target text segments are sorted to obtain a set of text segments corresponding to the user problem. Finally, based on the set of text segments, the user problem, and a preset answer prompt word template, the model input text is determined, and the model input text is input into a target model to obtain an answer text corresponding to the user problem. Through the technical solution of the present invention, the appropriate retrieval type for the user problem can be accurately determined by the problem discriminator, improving the retrieval accuracy and thus the Q&A accuracy.

[0021] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0023] Figure 1 is a flowchart of an answer text generation method in an embodiment of the present invention;

[0024] Figure 2 is a flowchart of a financial system Q&A method based on a large language model in an embodiment of the present invention;

[0025] Figure 3 is a flowchart of a text retrieval method in an embodiment of the present invention;

[0026] Figure 4 It is a schematic structural diagram of an answer text generation device in an embodiment of the present invention;

[0027] Figure 5 It is a schematic structural diagram of an electronic device for implementing the answer text generation method of the embodiment of the present invention. Detailed implementation manners

[0028] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described here can be implemented in an order different from those illustrated or described here. In addition, the terms "comprising" and "having" and their derivatives are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0030] It can be understood that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, usage scopes, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to users and the authorization of the users should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0031] Embodiment 1

[0032] To improve the accuracy of question answering, a joint retrieval scheme based on a question-answer database and a text and vector library is proposed in the existing methods. However, its limitations are as follows: First, the generation of question-answer pairs requires a large amount of computing and human resources to ensure the quality of question-answer pairs. Second, to cover enough information, the number of question-answer pairs also needs to be relatively high, which will greatly increase the size of the database and thus reduce the retrieval efficiency. In addition, another method is to first use vector retrieval to obtain a list of similar text fragments in the semantic dimension, then use keyword retrieval to screen from the obtained list of text fragments, and finally use a mixed score of vectors and keywords to screen out the final text fragments. This scheme uses vector retrieval for preliminary screening, so the upper limit of vector retrieval will become the upper limit of the entire retrieval scheme.

[0033] To improve the overall retrieval accuracy, an embodiment of the present invention proposes a new combined vector retrieval and keyword retrieval scheme. This scheme can combine the advantages of the two retrieval methods and does not require introducing a new database to affect the retrieval efficiency.

[0034] Figure 1 It is a flowchart of a method for generating answer text in an embodiment of the present invention. This embodiment is applicable to the situation of generating answer text for financial systems based on large language models. This method can be executed by the answer text generation device in the embodiment of the present invention, and the device can be implemented in software and / or hardware, such as Figure 1 As shown, the method specifically includes the following steps:

[0035] S101. Obtain the user's question, and input the user's question into the question discriminator to obtain the retrieval type corresponding to the user's question.

[0036] The method for generating answer text in the embodiment of the present invention can be applied to the financial field. That is, in this embodiment, the user's question can be, for example, a question about the financial system input by the user. Exemplarily, the user can input a question about the financial system on a financial APP on a smartphone, tablet, or computer.

[0037] In this embodiment, the question discriminator can be a discriminator model for determining which retrieval type is suitable for the question input by the user. Exemplarily, the question discriminator can be, for example, a trained text classification model.

[0038] Among them, the question discriminator is obtained by iteratively training the text classification model with a target sample set, and the target sample set includes: question samples and text segments corresponding to the question samples.

[0039] It should be noted that the question samples can be, for example, some questions about the financial system collected in advance, and the text segments corresponding to the question samples can be, for example, the text segments of the financial system referred to when obtaining the standard answers corresponding to each question sample determined in advance.

[0040] Among them, the text classification model can be, for example, a model for text classification. Exemplarily, the text classification model can be, for example, an untrained model, and after training, the question discriminator is obtained.

[0041] In this embodiment, the retrieval types can include keyword retrieval, vector retrieval, and hybrid retrieval, where hybrid retrieval can be understood as performing keyword retrieval and vector retrieval simultaneously.

[0042] Specifically, problem samples and corresponding text fragments are collected in advance as the target sample set, and a problem discriminator is obtained by iteratively training a text classification model with the target sample set. Then, during the application process, after obtaining the user's problem, the user's problem is input into the problem discriminator, and the appropriate retrieval type for the user's problem is confirmed by the problem discriminator.

[0043] S102. Based on the retrieval type, retrieve in the target database according to the user's problem, and sort the retrieved target text fragments to obtain a set of text fragments corresponding to the user's problem.

[0044] In this embodiment, the target database can be a database established based on local financial system text fragments. Exemplarily, the text content in the financial system text fragments can be stored in the target database, or the vectorized content corresponding to the text content in the financial system text fragments can be stored.

[0045] It should be noted that the target text fragment can be the financial system text fragment retrieved from the target database and most relevant to the user's problem. Further, the set of text fragments can be a set composed of K sorted target text fragments, where the value of K can be set by the user according to actual needs or empirical values, and this embodiment does not limit this.

[0046] Specifically, according to the retrieval type output by the problem discriminator, perform corresponding retrieval in the target database to obtain the K text fragments most relevant to the user's input problem, and re-sort them from high to low according to the similarity score to obtain a set of text fragments corresponding to the user's problem. Among them, the similarity score can be, for example, to calculate the similarity degree between the user's problem and each target text fragment, and this embodiment does not limit this.

[0047] S103. Determine the model input text according to the set of text fragments, the user's problem, and the answer prompt word template, and input the model input text into the target model to obtain the answer text corresponding to the user's problem.

[0048] In this embodiment, the target model can be a large language model, which can be abbreviated as a large model hereinafter. Among them, the answer prompt word template can be a pre-set large model answer prompt word template, which can assemble the user's problem, the set of target text fragments, and the user's instruction to form the large model input.

[0049] Among them, the answer text can be an answer to the user's problem generated according to the relevant financial system text content.

[0050] Specifically, after retrieving the text fragment most relevant to the user's question, an answer prompt word template is assembled using the user's question and the retrieval results of the target database to form the model input text. Then, the model input text can be input into a large language model for answering.

[0051] Exemplarily, if the retrieval type is keyword retrieval, the most relevant text fragment retrieved by the keyword is directly assembled into the answer prompt word template to obtain the answer text; if the retrieval type is vector retrieval, the most relevant text fragment retrieved by the vector is directly assembled into the answer prompt word template to obtain the answer text; if the retrieval type is hybrid retrieval, for example, K / 2 most relevant text fragments are obtained based on keyword retrieval and K / 2 most relevant text fragments are obtained based on vector retrieval, then after combining the K / 2 most relevant text fragments retrieved by keyword retrieval and the K / 2 most relevant text fragments retrieved by vector retrieval, the answer prompt word template is assembled to obtain the answer text. It should be noted that during the combination process, if the most relevant text fragments retrieved by the two retrieval methods are the same, one of the same parts in the retrieval results can be deleted and only the remaining content needs to be displayed. This embodiment does not limit this.

[0052] In the embodiment of the present invention, a problem discriminator is obtained by iteratively training a text classification model using the pre-collected problem samples and the text fragments corresponding to the problem samples as the target sample set. During the application process, the user's question is obtained, and then the user's question is input into the problem discriminator. The problem discriminator is used to confirm the appropriate retrieval type for the user's question. Then, based on the retrieval type, the user's question is retrieved in the target database, and the retrieved target text fragments are sorted to obtain the text fragment set corresponding to the user's question. Finally, based on the text fragment set, the user's question, and the preset answer prompt word template, the model input text is determined, and the model input text is input into the target model to obtain the answer text corresponding to the user's question. Through the technical solution of the present invention, the appropriate retrieval type for the question raised by the user can be accurately judged by the problem discriminator, the retrieval accuracy can be improved, and thus the question and answer accuracy can be improved.

[0053] Optionally, iteratively training the text classification model through the target sample set includes:

[0054] The problem samples in the target sample set are respectively subjected to keyword retrieval and vector retrieval to obtain the first predicted text fragment corresponding to the keyword retrieval and the second predicted text fragment corresponding to the vector retrieval.

[0055] It can be known that keyword retrieval is an information retrieval technology that searches for relevant documents, web pages, or other types of data through one or more keywords. Vector retrieval is an information retrieval technology based on the vector space model. It converts text (the main object in this embodiment is text), images, audio, or other types of data into numerical vectors and then conducts retrieval in the vector space. The core idea of this method is to map similar items to close positions in the vector space.

[0056] It should be noted that the first predicted text segment can be the most relevant text segment corresponding to the question sample obtained through keyword retrieval, and the second predicted text segment can be the most relevant text segment corresponding to the question sample obtained through vector retrieval.

[0057] Specifically, a financial system Q&A dataset is collected in advance as the target sample set, where the data can include questions raised by users and the institutional document segments referred to for obtaining answers. For the question samples in the target sample set, on the one hand, a keyword retrieval and re-ranking process is carried out to obtain the top k most relevant text segments of the keyword retrieval process; on the other hand, a vector retrieval and re-ranking process is carried out to obtain the top k most relevant text segments of the vector retrieval process.

[0058] Determine the retrieval type label corresponding to the question sample based on the text segment corresponding to the question sample, the first predicted text segment, and the second predicted text segment.

[0059] Among them, the retrieval type label can be a label used to characterize which retrieval type is suitable for the question sample for retrieval. Exemplarily, if a certain question sample uses keyword retrieval, the keyword retrieval output corresponding to this question sample can be set to 1, and the vector retrieval output can be set to 0; if a certain question sample uses vector retrieval, the vector retrieval output corresponding to this question sample can be set to 1, and the keyword retrieval output can be set to 0. Then the final retrieval type label can include four possibilities: [1,0], [0,1], [1,1], [0,0].

[0060] Specifically, for the question sample, keyword retrieval and vector retrieval are respectively carried out to obtain the optimal retrieval answer corresponding to keyword retrieval and the optimal retrieval answer corresponding to vector retrieval, and then the output results are evaluated, and the final retrieval type label is determined according to the evaluation results.

[0061] Establish a text classification model.

[0062] In this embodiment, the text classification model can be, for example, BERT (Bidirectional Encoder Representations from Transformers, a method of pre-training language representations. BERT is based on the Transformer architecture and can capture deep bidirectional text relationships by pre-training with a large amount of text data), TextCNN (Text Convolutional Neural Networks, a deep learning model that applies convolutional neural networks (CNNs) to text data. It uses convolutional layers to extract local features in the text and combines these features for text classification or other natural language processing tasks), or Transformer. This embodiment does not limit this.

[0063] Input the problem samples in the target sample set into the text classification model to obtain the predicted retrieval type corresponding to the problem samples.

[0064] Among them, the predicted retrieval type can be the retrieval type suitable for the predicted problem samples obtained by the text classification model based on retrieval judgment of the problem samples. Exemplarily, the predicted retrieval type can be, for example, keyword retrieval or vector retrieval.

[0065] Specifically, input the problem samples in the target sample set into the text classification model, and output the predicted retrieval type corresponding to the problem samples.

[0066] Train the parameters of the text classification model according to the objective function formed by the predicted retrieval type corresponding to the problem samples and the retrieval type label corresponding to the problem samples.

[0067] In the actual operation process, the specific training process of the text classification model can be described as follows: First, collect and preprocess a large amount of labeled text data; then, select a suitable loss function (such as the cross-entropy loss function) to measure the difference between the model prediction result and the true label; next, continuously adjust the model parameters based on the pre-trained basic model through an iterative optimization algorithm (such as the gradient descent method) to minimize the loss function value; during the training process, model verification is also required to ensure its generalization ability; finally, after multiple iterations, a model that can accurately predict text categories is obtained.

[0068] Return to execute the operation of inputting the problem samples in the target sample set into the text classification model to obtain the predicted retrieval type corresponding to the problem samples until a problem discriminator is obtained.

[0069] In this embodiment, the training process of the text classification model is a cyclic iterative process. The termination conditions of the cycle can be, for example, that the number of iterations reaches a preset number and / or the retrieval type output results of preset adjacent numbers are the same, that is, the iteration converges.

[0070] Among them, the number of iterations can be the number of loop iterations from the input of the problem sample to the final retrieval type output result. Among them, the preset number can be the number of loop iterations set in advance according to the actual situation or empirical value. The specific size of this value is not limited in this embodiment. It should be noted that the preset adjacent number can be the number of adjacent loop iterations set in advance according to the actual situation or empirical value. The specific value of this is not specifically limited in this embodiment. Exemplarily, the preset adjacent number is 3, that is, the retrieval type output results are the same for 3 consecutive times, and finally a trained problem discriminator is obtained.

[0071] Optionally, determining the retrieval type label corresponding to the problem sample according to the text segment corresponding to the problem sample, the first predicted text segment, and the second predicted text segment includes:

[0072] Determine the mean reciprocal rank corresponding to keyword retrieval according to the first predicted text segment and the text segment corresponding to the problem sample, and determine the category output label corresponding to keyword retrieval according to the mean reciprocal rank corresponding to keyword retrieval.

[0073] It can be known that MRR (Mean Reciprocal Rank) is a commonly used evaluation metric in the fields of information retrieval and machine learning, especially in tasks such as evaluating the ranking quality, such as search engine result ranking, recommendation systems, etc. MRR measures the average quality of query results, especially when considering the position of the most relevant results. Specifically, MRR calculates the average of the reciprocal ranks of each query result. Specifically, for each query, find the most relevant result (usually determined by manual annotation or predefined labels), then calculate the reciprocal of the position of this result in the sorted list, and finally take the average of the reciprocal ranks of all queries.

[0074] In the text classification model of this embodiment, for the problem samples in the target sample set, one is to carry out the keyword retrieval and re-ranking process to obtain the topk most relevant text segments of the keyword retrieval process, and the other is to carry out the vector retrieval and re-ranking process to obtain the topk most relevant text segments of the vector retrieval process, and then evaluate the output results. Exemplarily, for a certain problem sample, if the MRR of its keyword retrieval is 1, then the output of the "keyword retrieval" category of this problem sample can be set to 1.

[0075] Determine the mean reciprocal rank corresponding to vector retrieval according to the second predicted text segment and the text segment corresponding to the problem sample, and determine the category output label corresponding to vector retrieval according to the mean reciprocal rank corresponding to vector retrieval.

[0076] Exemplarily, for a certain problem sample, if the MRR of vector retrieval is 1, the output of the "vector retrieval" category for this problem sample can be set to 1.

[0077] Determine the retrieval type label corresponding to the problem sample according to the category output label corresponding to keyword retrieval and the category output label corresponding to vector retrieval.

[0078] In the actual operation process, the final category output label corresponding to the text classification model in this embodiment can include four possibilities: [1, 0], [0, 1], [1, 1], and [0, 0].

[0079] Optionally, input the user's question into the question discriminator to obtain the retrieval type corresponding to the user's question, including:

[0080] Input the user's question into the question discriminator to obtain the first function output value corresponding to keyword retrieval and the second function output value corresponding to vector retrieval.

[0081] In this embodiment, the activation function selected is sigmoid. Among them, the first function output value can be the sigmoid output value corresponding to keyword retrieval, and the second function output value can be the sigmoid output value corresponding to vector retrieval.

[0082] In the actual operation process, keyword retrieval and vector retrieval may both give the optimal retrieval answer at the same time, or may both fail to give the optimal answer at the same time. Therefore, the question discriminator in the embodiment of the present invention is designed as a multi-label classification model, whose output is two categories: "vector retrieval" and "keyword retrieval", and its activation function is selected as sigmoid. Then, a threshold is set for the output of sigmoid, and the retrieval type output result of the text classification model is finally determined by judging the size of the threshold.

[0083] Determine the retrieval type corresponding to the user's question according to the first function output value and the second function output value.

[0084] Specifically, the retrieval type corresponding to the final user's question can be determined by setting a threshold for the output of sigmoid and comparing the sigmoid values corresponding to the two retrieval types with the threshold.

[0085] Optionally, determining the retrieval type corresponding to the user's question according to the first function output value and the second function output value includes:

[0086] If the first function output value is greater than the preset threshold and the second function output value is less than the preset threshold, it is determined that the retrieval type corresponding to the user's question is keyword retrieval.

[0087] Among them, the preset threshold can be a sigmoid threshold set in advance according to the actual situation or empirical value, and the specific size of this value is not limited in this embodiment.

[0088] Specifically, if the output of the "keyword retrieval" category is greater than the threshold, and the output of the "vector retrieval" is less than the threshold, then the problem discriminator outputs "keyword retrieval".

[0089] If the output value of the first function is less than the preset threshold, and the output value of the second function is greater than the preset threshold, then it is determined that the retrieval type corresponding to the user's question is vector retrieval.

[0090] Specifically, if the output of the "vector retrieval" category is greater than the threshold, and the output of the "keyword retrieval" is less than the threshold, then the problem discriminator outputs "vector retrieval".

[0091] Otherwise, it is determined that the retrieval type corresponding to the user's question is hybrid retrieval.

[0092] Specifically, if the outputs of the "vector retrieval" category and the "keyword retrieval" are both greater than or less than the threshold, then the problem discriminator outputs "hybrid retrieval".

[0093] Optionally, before retrieving according to the user's question in the target database, it further includes:

[0094] Obtain an initial document set.

[0095] In this embodiment, the initial document set can be a local financial system document.

[0096] Chunk each document in the initial document set to obtain a text chunk set.

[0097] Among them, the text chunk set can be a set composed of text chunks obtained by chunking the text according to the characteristics of the financial system document.

[0098] Specifically, in this embodiment, the local financial system document can be first parsed and loaded into text, and then the text is chunked according to the characteristics of the system document. Exemplarily, typical chunking bases can be, for example, according to Article X, and relatively independent clauses are divided into separate text chunks.

[0099] Construct a text database according to the text chunk set.

[0100] Among them, the text database can be a database storing several text chunks.

[0101] Specifically, the text chunks in the text chunk set are stored in the database to form a text database.

[0102] Vectorize each text chunk in the text chunk set to obtain a vector database.

[0103] The vector database may be a database storing vectors of vectorized text blocks.

[0104] Specifically, the text blocks in the text block set are vectorized and stored in a database to form a vector database.

[0105] Construct a target database based on the text database and the vector database.

[0106] Specifically, the target database is composed of a text database and a vector database.

[0107] The technical solution of the embodiment of the present invention introduces keyword retrieval and vector retrieval in the retrieval stage and gives full play to the respective advantages of the two retrieval methods, and also introduces a question discriminator to automatically identify the retrieval method that is more suitable for the user's question, thereby further improving the retrieval accuracy. Then, in the answer generation stage, the retrieval enhancement generation technology is used to provide private domain financial system information to the large language model, thereby improving the retrieval accuracy and further improving the question and answer accuracy.

[0108] Embodiment 2

[0109] As an exemplary description of the above invention embodiment, Figure 2 It is a flowchart of a financial system question-answering method based on a large language model in an embodiment of the present invention.

[0110] The embodiment of the present invention proposes a financial system question-answering method based on a large language model, which uses retrieval enhancement generation technology to provide private domain financial system information to the large model. In order to improve the recall rate, since financial system question-answering may involve simple query questions such as special nouns and special phrases, it may also involve semantic query questions of complex questions. In order to improve the retrieval accuracy, the embodiment of the present invention proposes a solution based on keyword retrieval, vector retrieval, hybrid retrieval and re-ranking.

[0111] like Figure 2 As shown in the figure, the implementation process of the financial system question answering method based on the large language model can be described as follows:

[0112] In the first stage, the knowledge base is constructed (the knowledge base is the target database in the above embodiment): first, the local financial system document is loaded and parsed into text, and then the text is divided into blocks according to the characteristics of the system document (a typical block division basis is such as dividing relatively independent clauses into separate text blocks according to Article X). After that, the text blocks are directly stored in the knowledge base to form a text database, and the text is vectorized and stored in the knowledge base to form a vector knowledge base. The target database is composed of the text database and the vector knowledge base.

[0113] Second stage, knowledge base retrieval: The user inputs a question. (Vectorize the input question) Perform text retrieval based on the user's question to obtain K relevant text blocks (Text block #1, Text block #2... Text block #K). Then, re-rank the obtained K text blocks according to the user's input question, and re-rank the K text blocks from high to low according to the similarity score (e.g., Text block #5, Text block #2... Text block #1). Subsequently, select the top topk text blocks that are most relevant to the user's question.

[0114] Figure 3 It is a flowchart of a text retrieval method in an embodiment of the present invention. To improve the accuracy of knowledge base retrieval, the embodiment of the present invention proposes a text retrieval scheme as Figure 3 shown below, and its process can be described as follows:

[0115] After receiving the question input by the user, first input the question into the question discriminator. The question discriminator uses the question discrimination model to confirm whether the question is suitable for keyword retrieval, vector retrieval, or hybrid retrieval? If the question discriminator determines it is keyword retrieval, then start the keyword retrieval process and output the K text fragments (Text block #1, Text block #2... Text block #K) that are most relevant to the user's question; if the question discriminator determines it is vector retrieval, then vectorize the user's question and use vector retrieval to output the K text fragments that are most similar to the question vector; if the question discriminator determines it is hybrid retrieval, then start keyword retrieval and vector retrieval respectively, and each output K / 2 most relevant text fragments, and combine the two search results to obtain K retrieval text fragments (if the K / 2 most relevant text fragments output by keyword retrieval and vector retrieval respectively contain the same text fragment, they can be displayed separately, or the same part in one of the retrieval results can be deleted and only the remaining content can be shown. This embodiment does not limit this, and the user can set it according to actual needs).

[0116] In the above process, the steps for obtaining the question discrimination model are as follows:

[0117] a) Original data preparation: The user prepares a financial system Q&A dataset (i.e., the target sample set in the above embodiment), which includes questions and the institutional document fragments referred to for obtaining answers.

[0118] b) Model Selection: For text classification, fine-tuning based on BERT can be selected. Just input [CLS] + sentence (user question) + [SEP] into BERT, then extract the vector at the [CLS] position and connect it to a fully-connected layer for classification. In the actual operation process, keyword retrieval and vector retrieval may both give the optimal retrieval answer, or neither may give the optimal answer. Therefore, the question discriminant model in this embodiment is designed as a multi-label classification model, with its output being two categories: "vector retrieval" and "keyword retrieval", and its activation function is selected as sigmoid.

[0119] c) Data Annotation Acquisition: For the user questions in the financial system Q&A dataset, first, carry out the keyword retrieval and re-ranking process to obtain the top-k most relevant text fragments of the keyword retrieval process; second, carry out the vector retrieval and re-ranking process to obtain the top-k most relevant text fragments of the vector retrieval process, and then evaluate the output results. For a certain question, if the MRR of keyword retrieval is 1, the output of the "keyword retrieval" category is 1; if the MRR of vector retrieval is 1, the output of the "vector retrieval" category is 1. The final labels include four possibilities: [1,0], [0,1], [1,1], and [0,0].

[0120] d) Use the annotated data to train the model in b) to obtain a question multi-label classification model.

[0121] e) Model Usage: Set a threshold for the output of sigmoid. If the output of the "keyword retrieval" category is greater than the threshold and the output of "vector retrieval" is less than the threshold, the question discriminator outputs "keyword retrieval"; if the output of "vector retrieval" is greater than the threshold and the output of "keyword retrieval" is less than the threshold, the question discriminator outputs "vector retrieval"; if the outputs of both the "vector retrieval" category and the "keyword retrieval" category are greater than or less than the threshold at the same time, the question discriminator outputs "hybrid retrieval".

[0122] In the third stage, large model generation: After obtaining the most relevant text retrieval fragments, use the user question and the knowledge base retrieval results to assemble the prompt template (i.e., the answer prompt template in the above embodiment) to form a prompt (i.e., the answer text in the above embodiment) and input it to the large model for answering.

[0123] The technical solution of the embodiment of the present invention combines keyword retrieval and vector retrieval, which can give full play to the respective advantages of the two retrieval methods, thus solving the situation in the financial system Q&A scenario that includes both simple query problems such as special nouns and special phrases, and semantic query problems of complex questions. The technical solution of the embodiment of the present invention uses the question discriminant model to automatically identify the more suitable retrieval method for user questions, thereby improving the retrieval accuracy and further improving the Q&A accuracy.

[0124] Embodiment III

[0125] Figure 4 It is a schematic structural diagram of an answer text generation device in an embodiment of the present invention. This embodiment is applicable to the situation of generating answer texts for financial systems based on large language models. This device can be implemented in software and / or hardware, and can be integrated into any device that provides the function of generating answer texts, such as Figure 4 As shown, the answer text generation device specifically includes: an acquisition and input module 201, a retrieval and sorting module 202, and a determination and input module 203.

[0126] Among them, the acquisition and input module 201 is used to acquire a user question, input the user question into a question discriminator, and obtain the retrieval type corresponding to the user question. The question discriminator is obtained by iteratively training a text classification model with a target sample set. The target sample set includes: question samples and text segments corresponding to the question samples;

[0127] The retrieval and sorting module 202 is used to retrieve in a target database according to the retrieval type based on the user question, and sort the retrieved target text segments to obtain a set of text segments corresponding to the user question;

[0128] The determination and input module 203 is used to determine a model input text according to the set of text segments, the user question, and an answer prompt word template, and input the model input text into a target model to obtain an answer text corresponding to the user question.

[0129] Optionally, the acquisition and input module 201 includes:

[0130] A retrieval unit, used to perform keyword retrieval and vector retrieval on the question samples in the target sample set respectively, to obtain a first predicted text segment corresponding to the keyword retrieval and a second predicted text segment corresponding to the vector retrieval;

[0131] A first determination unit, used to determine a retrieval type label corresponding to the question sample according to the text segment corresponding to the question sample, the first predicted text segment, and the second predicted text segment;

[0132] An establishment unit, used to establish a text classification model;

[0133] A first input unit, used to input the question samples in the target sample set into the text classification model to obtain a predicted retrieval type corresponding to the question sample;

[0134] A training unit for training the parameters of the text classification model according to an objective function formed by a predicted retrieval type corresponding to the problem sample and a retrieval type label corresponding to the problem sample;

[0135] An execution unit for returning to execute an operation of inputting a problem sample in the target sample set into the text classification model to obtain a predicted retrieval type corresponding to the problem sample until a problem discriminator is obtained.

[0136] Optionally, the first determination unit is specifically configured to:

[0137] Determine an average reciprocal rank corresponding to keyword retrieval according to the first predicted text segment and the text segment corresponding to the problem sample, and determine a category output label corresponding to the keyword retrieval according to the average reciprocal rank corresponding to the keyword retrieval;

[0138] Determine an average reciprocal rank corresponding to vector retrieval according to the second predicted text segment and the text segment corresponding to the problem sample, and determine a category output label corresponding to the vector retrieval according to the average reciprocal rank corresponding to the vector retrieval;

[0139] Determine a retrieval type label corresponding to the problem sample according to the category output label corresponding to the keyword retrieval and the category output label corresponding to the vector retrieval.

[0140] Optionally, the obtaining and input module 201 includes:

[0141] A second input unit for inputting the user problem into the problem discriminator to obtain a first function output value corresponding to the keyword retrieval and a second function output value corresponding to the vector retrieval;

[0142] A second determination unit for determining a retrieval type corresponding to the user problem according to the first function output value and the second function output value.

[0143] Optionally, the second determination unit is specifically configured to:

[0144] If the first function output value is greater than a preset threshold and the second function output value is less than the preset threshold, determine that the retrieval type corresponding to the user problem is keyword retrieval;

[0145] If the first function output value is less than the preset threshold and the second function output value is greater than the preset threshold, determine that the retrieval type corresponding to the user problem is vector retrieval;

[0146] Otherwise, determine that the retrieval type corresponding to the user problem is hybrid retrieval.

[0147] Optionally, the device further includes:

[0148] An acquisition module for acquiring an initial document set;

[0149] A chunking module for chunking each document in the initial document set to obtain a text chunk set;

[0150] A component module for constructing a text database according to the text chunk set;

[0151] A vectorization module for vectorizing each text chunk in the text chunk set to obtain a vector database;

[0152] A construction module for constructing a target database according to the text database and the vector database.

[0153] The above product can execute the answer text generation method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0154] Embodiment 4

[0155] Figure 5 FIG. shows a schematic structural diagram of an electronic device 30 that can be used to implement an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, a personal digital processor, a cellular phone, a smart phone, a wearable device (such as a helmet, glasses, a watch, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0156] As Figure 5 shown, the electronic device 30 includes at least one processor 31, and a memory communicatively connected to at least one processor 31, such as a read-only memory (ROM) 32, a random access memory (RAM) 33, etc. Among them, the memory stores a computer program executable by at least one processor. The processor 31 can execute various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 32 or the computer program loaded from the storage unit 38 into the random access memory (RAM) 33. In the RAM 33, various programs and data required for the operation of the electronic device 30 can also be stored. The processor 31, the ROM 32, and the RAM 33 are connected to each other through a bus 34. The input / output (I / O) interface 35 is also connected to the bus 34.

[0157] Multiple components in the electronic device 30 are connected to the I / O interface 35, including: an input unit 36, such as a keyboard, a mouse, etc.; an output unit 37, such as various types of displays, speakers, etc.; a storage unit 38, such as a disk, an optical disc, etc.; and a communication unit 39, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 39 allows the electronic device 30 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0158] The processor 31 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 31 include but are not limited to a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 31 executes the various methods and processes described above, such as the answer text generation method:

[0159] Obtain a user question, input the user question into a question discriminator to obtain the retrieval type corresponding to the user question, where the question discriminator is obtained by iteratively training a text classification model with a target sample set, and the target sample set includes: question samples and text segments corresponding to the question samples;

[0160] Based on the retrieval type, retrieve in the target database according to the user question, and sort the retrieved target text segments to obtain a set of text segments corresponding to the user question;

[0161] Determine model input text according to the set of text segments, the user question, and an answer prompt word template, and input the model input text into a target model to obtain an answer text corresponding to the user question.

[0162] In some embodiments, the answer text generation method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as the storage unit 38. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 30 via the ROM 32 and / or the communication unit 39. When the computer program is loaded into the RAM 33 and executed by the processor 31, one or more steps of the answer text generation method described above can be executed. Alternatively, in other embodiments, the processor 31 can be configured to execute the answer text generation method in any other suitable manner (e.g., by means of firmware).

[0163] The various embodiments of the systems and techniques described above in this specification can be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGA), application specific integrated circuits (ASIC), application specific standard products (ASSP), systems on a chip (SOC), complex programmable logic devices (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that receives data and instructions from, and transmits data and instructions to, a storage system, at least one input device, and at least one output device.

[0164] The computer programs for implementing the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions / operations specified in the flowchart and / or block diagram to be implemented. The computer programs can be executed entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

[0165] In the context of the present invention, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0166] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the electronic device. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0167] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), blockchain network, and the Internet.

[0168] The computing system can include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The client-server relationship is created by computer programs running on respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system and solves the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services.

[0169] In one embodiment, the embodiment of the present invention further includes a computer program product, which includes a computer program that, when executed by a processor, implements the answer text generation method of any embodiment of the present invention.

[0170] In the process of implementing a computer program product, computer program code for performing the operations of the present invention may be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network - including a local area network (LAN) or a wide area network (WAN) - or, alternatively, may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0171] It should be understood that the various forms of the process shown above can be used, steps can be reordered, added or deleted. For example, the steps recited in the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved, and no limitation is made herein.

[0172] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for generating an answer text, characterized in that: include: Obtain a user question and input the user question into a question discriminator to obtain a search type corresponding to the user question, wherein the question discriminator is obtained by iteratively training a text classification model with a target sample set, wherein the target sample set includes: a question sample and a text segment corresponding to the question sample; the search type includes: keyword search, vector search, and hybrid search; Based on the search type, searching in the target database according to the user question, and sorting the retrieved target text fragments to obtain a set of text fragments corresponding to the user question; Determine a model input text according to the text segment set, the user question and the answer prompt word template, and input the model input text into the target model to obtain an answer text corresponding to the user question; The step of inputting the user question into the question discriminator to obtain the search type corresponding to the user question includes: Input the user question into a question discriminator to obtain a first function output value corresponding to keyword search and a second function output value corresponding to vector search; The retrieval type corresponding to the user question is determined according to the first function output value and the second function output value.

2. The method according to claim 1, characterized in that Iterative training of the text classification model through the target sample set includes: Perform keyword search and vector search on the question samples in the target sample set respectively, to obtain a first predicted text segment corresponding to the keyword search and a second predicted text segment corresponding to the vector search; Determine a retrieval type label corresponding to the question sample according to the text segment corresponding to the question sample, the first predicted text segment, and the second predicted text segment; Build a text classification model; Inputting the question sample in the target sample set into the text classification model to obtain the predicted retrieval type corresponding to the question sample; Training the parameters of the text classification model according to an objective function formed by the predicted retrieval type corresponding to the question sample and the retrieval type label corresponding to the question sample; Return to executing the operation of inputting the question sample in the target sample set into the text classification model to obtain the predicted retrieval type corresponding to the question sample, until a question discriminator is obtained.

3. The method according to claim 2, characterized in that Determining a retrieval type label corresponding to the question sample according to the text segment corresponding to the question sample, the first predicted text segment, and the second predicted text segment includes: Determine an average reciprocal ranking corresponding to the keyword search according to the first predicted text segment and the text segment corresponding to the question sample, and determine a category output label corresponding to the keyword search according to the average reciprocal ranking corresponding to the keyword search; Determining an average reciprocal ranking corresponding to the vector retrieval according to the second predicted text segment and the text segment corresponding to the question sample, and determining a category output label corresponding to the vector retrieval according to the average reciprocal ranking corresponding to the vector retrieval; The retrieval type label corresponding to the question sample is determined according to the category output label corresponding to the keyword retrieval and the category output label corresponding to the vector retrieval.

4. The method according to claim 1, characterized in that: Determining the search type corresponding to the user question according to the first function output value and the second function output value includes: If the output value of the first function is greater than a preset threshold, and the output value of the second function is less than the preset threshold, determining that the search type corresponding to the user question is a keyword search; If the output value of the first function is less than the preset threshold, and the output value of the second function is greater than the preset threshold, determining that the search type corresponding to the user question is vector search; Otherwise, it is determined that the search type corresponding to the user question is a mixed search.

5. The method according to claim 1, characterized in that Before searching in the target database according to the user question, the method further includes: Get the initial document collection; Dividing each document in the initial document set into blocks to obtain a text block set; Building a text database according to the text block set; Vectorizing each text block in the text block set to obtain a vector database; A target database is constructed according to the text database and the vector database.

6. A device for generating a reply text, characterized in that: include: The acquisition and input module is used to acquire user questions and input the user questions into a question discriminator to obtain a search type corresponding to the user questions. The question discriminator is obtained by iteratively training a text classification model with a target sample set. The target sample set includes: a question sample and a text segment corresponding to the question sample. The search types include: keyword search, vector search, and hybrid search. A retrieval and sorting module, used to search in a target database according to the user question based on the retrieval type, and sort the retrieved target text segments to obtain a set of text segments corresponding to the user question; A determination and input module, used to determine a model input text according to the text segment set, the user question and the answer prompt word template, and input the model input text into a target model to obtain an answer text corresponding to the user question; Wherein, the acquisition and input module includes: A second input unit, used to input the user question into a question discriminator, and obtain a first function output value corresponding to the keyword search and a second function output value corresponding to the vector search; A second determining unit is used to determine a retrieval type corresponding to the user question according to the first function output value and the second function output value.

7. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the answer text generation method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the answer text generation method according to any one of claims 1 to 5 when executed.

9. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the answer text generation method according to any one of claims 1 to 5.

Images