Answer text generation method and device, equipment, medium and product

Abstract

The invention discloses an answer text generation method and device, equipment, a medium and a product. The method comprises the steps that user questions are obtained and input into a question discriminator, retrieval types corresponding to the user questions are obtained, the question discriminator is obtained by iteratively training a text classification model through a target sample set, and the target sample set comprises question samples and text fragments corresponding to the question samples; based on the retrieval type, performing retrieval in a target database according to the user question, and sorting retrieved target text fragments to obtain a text fragment set corresponding to the user question; and determining a model input text according to the text fragment set, the user question and the answer prompt word template, and inputting the model input text into the target model to obtain an answer text corresponding to the user question. According to the technical scheme, the retrieval type suitable for the question proposed by the user can be accurately judged through the question discriminator, the retrieval accuracy is improved, and then the question and answer accuracy is improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the technical field of large language models, and in particular to a method, device, equipment, medium and product for generating answer texts. Background Art

[0002] At present, the technologies, applications and ecosystems related to large models continue to become a hot topic. With the continuous improvement of the capabilities of general large language models and the continuous improvement and optimization of large model development frameworks, applications in vertical fields, especially in the field of knowledge question and answer, are becoming more and more extensive. However, due to the high training time and resource costs of large models, knowledge question and answer scenarios in vertical fields face the challenge that large models cannot quickly adapt to new knowledge and realize timely updating of knowledge. To solve the above problems, RAG (Retrieval-Augmented Generation) was proposed, which uses the retrieval of non-parametric knowledge to improve the performance of knowledge-intensive tasks, and provides a good direction for solving problems such as the "hallucination" of large models (answering the wrong question), the "timeliness" of knowledge, and private domain data.

[0003] However, in knowledge question answering using RAG technology, a core issue is how to improve the recall rate and thus improve the accuracy of question answering, that is, how to find a limited number of text fragments that are most relevant to the user's question from a large number of knowledge documents. Generally speaking, the recall rate can be improved by increasing the number of recall results, but in actual question answering applications, the number of recall results cannot be too many. The reason is that the input window of a large model has a size limit (generally 4K, 16K, 32K, 128K tokens), and it is necessary to select a suitable text segmentation strategy and number of recall items based on the size limit of the selected model input window; second, even if the model context window is large enough, too many recall segments may introduce less relevant content, resulting in reduced answer quality. Therefore, the optimal recall result is to cover the most relevant text segmentation content in as few recall segments as possible.

[0004] In summary, in order to improve the accuracy of question answering, there is an urgent need for a question answering method that can combine the advantages of the two retrieval methods. Summary of the invention

[0005] The embodiments of the present invention provide a method, device, equipment, medium and product for generating an answer text, so as to improve the accuracy of retrieval and further improve the accuracy of question and answer.

[0006] According to one aspect of the present invention, there is provided a method for generating an answer text, comprising:

[0007] Obtaining a user question and inputting the user question into a question discriminator to obtain a retrieval type corresponding to the user question, wherein the question discriminator is obtained by iteratively training a text classification model with a target sample set, wherein the target sample set includes: a question sample and a text segment corresponding to the question sample;

[0008] Based on the search type, searching in the target database according to the user question, and sorting the retrieved target text fragments to obtain a set of text fragments corresponding to the user question;

[0009] The model input text is determined according to the text segment set, the user question and the answer prompt word template, and the model input text is input into the target model to obtain the answer text corresponding to the user question.

[0010] According to another aspect of the present invention, there is provided a device for generating an answer text, the device comprising:

[0011] An acquisition and input module is used to acquire user questions and input the user questions into a question discriminator to obtain a search type corresponding to the user questions. The question discriminator is obtained by iteratively training a text classification model with a target sample set, wherein the target sample set includes: a question sample and a text segment corresponding to the question sample;

[0012] A retrieval and sorting module, used to search in a target database according to the user question based on the retrieval type, and sort the retrieved target text segments to obtain a set of text segments corresponding to the user question;

[0013] The determination and input module is used to determine the model input text according to the text segment set, the user question and the answer prompt word template, and input the model input text into the target model to obtain the answer text corresponding to the user question.

[0014] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the answer text generation method described in any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the answer text generation method described in any embodiment of the present invention when executed.

[0019] According to another aspect of the present invention, an embodiment of the present invention further provides a computer program product, the computer program product includes a computer program, and when the computer program is executed by a processor, it implements the answer text generation method described in any embodiment of the present invention.

[0020] The embodiment of the present invention uses the previously collected question samples and the text fragments corresponding to the question samples as the target sample set, and iteratively trains the text classification model through the target sample set to obtain a question discriminator. During the application process, the user question is obtained, and then the user question is input into the question discriminator. The question discriminator confirms the retrieval type suitable for the user question. Then, based on the retrieval type, the user question is searched in the target database, and the retrieved target text fragments are sorted to obtain a text fragment set corresponding to the user question. Finally, the model input text is determined according to the text fragment set, the user question and the preset answer prompt word template, and the model input text is input into the target model to obtain the answer text corresponding to the user question. Through the technical solution of the present invention, the question discriminator can accurately determine the retrieval type suitable for the question raised by the user, improve the retrieval accuracy, and then improve the question-answering accuracy.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 is a flow chart of a method for generating an answer text in an embodiment of the present invention;

[0024] Figure 2 It is a flowchart of a financial system question-answering method based on a large language model in an embodiment of the present invention;

[0025] Figure 3 is a flow chart of a text retrieval method in an embodiment of the present invention;

[0026] Figure 4 is a structural schematic diagram of a device for generating an answer text in an embodiment of the present invention;

[0027] Figure 5 It is a structural schematic diagram of an electronic device for implementing the answer text generation method of an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and their meanings are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0031] Embodiment 1

[0032] In order to improve the accuracy of question-answering, existing methods propose joint retrieval schemes based on question-answering databases and text and vector libraries. However, their limitations are that the generation of question-answering pairs requires a lot of computing and human resources to ensure the quality of the question-answering pairs, and that the number of question-answering pairs needs to be high in order to cover enough information, which will greatly increase the size of the database and reduce the efficiency of retrieval. In addition, another method is to first use vector retrieval to obtain a list of similar text fragments in the semantic dimension, then use keyword retrieval to filter the obtained text fragment list, and finally apply the mixed score of vectors and keywords to filter out the final text fragments. This scheme uses vector retrieval for preliminary screening, so the upper limit of vector retrieval will become the upper limit of the entire retrieval scheme.

[0033] In order to improve the overall retrieval accuracy, the embodiment of the present invention proposes a new solution of joint vector retrieval and keyword retrieval, which can combine the advantages of the two retrieval methods without introducing a new database to affect the retrieval efficiency.

[0034] Figure 1 is a flow chart of a method for generating an answer text in an embodiment of the present invention. This embodiment is applicable to the generation of an answer text of a financial system based on a large language model. The method can be executed by an answer text generation device in an embodiment of the present invention. The device can be implemented in software and / or hardware, such as Figure 1 As shown, the method specifically comprises the following steps:

[0035] S101, obtaining a user question, and inputting the user question into a question discriminator to obtain a search type corresponding to the user question.

[0036] The answer text generation method of the embodiment of the present invention can be applied to the financial field, that is, in this embodiment, the user question can be, for example, a question about the financial system input by the user. For example, the user can input questions about the financial system on a financial APP on a smartphone, tablet or computer.

[0037] In this embodiment, the question discriminator may be a discriminator model for discriminating which search type is suitable for searching the question input by the user. Exemplarily, the question discriminator may be, for example, a trained text classification model.

[0038] The question discriminator is obtained by iteratively training a text classification model through a target sample set, and the target sample set includes: question samples and text fragments corresponding to the question samples.

[0039] It should be noted that the question samples may be, for example, some questions on the financial system collected in advance, and the text fragments corresponding to the question samples may be, for example, the financial system text fragments that are referenced when obtaining the standard answers corresponding to each question sample that are determined in advance.

[0040] The text classification model may be, for example, a model for text classification. Exemplarily, the text classification model may be, for example, an untrained model, and a problem discriminator is obtained after training.

[0041] In this embodiment, the search type may include keyword search, vector search, and hybrid search, wherein the hybrid search may be understood as performing keyword search and vector search simultaneously.

[0042] Specifically, question samples and text fragments corresponding to the question samples are collected in advance as the target sample set, and the text classification model is iteratively trained through the target sample set to obtain the question discriminator. Later, in the application process, after obtaining the user question, the user question is input into the question discriminator, and the question discriminator confirms the retrieval type that is suitable for the user question.

[0043] S102: Based on the search type, search the target database according to the user question, and sort the retrieved target text segments to obtain a set of text segments corresponding to the user question.

[0044] In this embodiment, the target database may be a database established based on the local financial system text fragment. Exemplarily, the target database may store the text content in the financial system text fragment, and may also store the vectorized content corresponding to the text content in the financial system text fragment.

[0045] It should be noted that the target text segment can be a financial system text segment that is most relevant to the user's question and is retrieved from the target database. Further, the text segment set can be a set composed of K target text segments after sorting, wherein the value of K can be set by the user according to actual needs or experience, and this embodiment does not limit this.

[0046] Specifically, according to the search type output by the question discriminator, a corresponding search is performed in the target database to obtain K text segments most relevant to the question input by the user, and the segments are reordered from high to low according to the similarity scores to obtain a set of text segments corresponding to the user question. The similarity score may be, for example, the degree of similarity between the user question and each target text segment, which is not limited in this embodiment.

[0047] S103: Determine the model input text according to the text segment set, the user question and the answer prompt word template, and input the model input text into the target model to obtain the answer text corresponding to the user question.

[0048] In this embodiment, the target model may be a large language model, which may be referred to as a large model in the following. The answer prompt word template may be a pre-set large model answer prompt word template, which may be assembled with user questions, target text segment sets, and user instructions to form a large model input.

[0049] The answer text may be an answer to a user's question generated based on the relevant financial system text content.

[0050] Specifically, after retrieving the text fragment most relevant to the user's question, the user's question and the target database retrieval results are used to assemble the answer prompt word template to form a model input text, and then the model input text can be input into the large language model for answering.

[0051] Exemplarily, if the search type is keyword search, the most relevant text fragment retrieved by the keyword is directly assembled with the answer prompt word template to obtain the answer text; if the search type is vector search, the most relevant text fragment retrieved by the vector is directly assembled with the answer prompt word template to obtain the answer text; if the search type is mixed search, for example, K / 2 most relevant text fragments are obtained based on keyword search and K / 2 most relevant text fragments are obtained based on vector search, then the K / 2 most relevant text fragments retrieved by the keyword and the K / 2 most relevant text fragments obtained by the vector search are combined, and then assembled with the answer prompt word template to obtain the answer text. It should be noted that in the process of combination, if the most relevant text fragments retrieved by the two search methods are the same, the same part in one of the search results can be deleted, and only the remaining content can be displayed. This embodiment does not limit this.

[0052] The embodiment of the present invention uses the previously collected question samples and the text fragments corresponding to the question samples as the target sample set, and iteratively trains the text classification model through the target sample set to obtain a question discriminator. During the application process, the user question is obtained, and then the user question is input into the question discriminator. The question discriminator confirms the retrieval type suitable for the user question. Then, based on the retrieval type, the user question is searched in the target database, and the retrieved target text fragments are sorted to obtain a text fragment set corresponding to the user question. Finally, the model input text is determined according to the text fragment set, the user question and the preset answer prompt word template, and the model input text is input into the target model to obtain the answer text corresponding to the user question. Through the technical solution of the present invention, the question discriminator can accurately determine the retrieval type suitable for the question raised by the user, improve the retrieval accuracy, and then improve the question-answering accuracy.

[0053] Optionally, iteratively training a text classification model using a target sample set includes:

[0054] The question samples in the target sample set are respectively subjected to keyword search and vector search to obtain a first predicted text segment corresponding to the keyword search and a second predicted text segment corresponding to the vector search.

[0055] It is known that keyword search is an information retrieval technology that uses one or more keywords to find relevant documents, web pages or other types of data. Vector search is an information retrieval technology based on the vector space model. It converts text (the main object in this embodiment is text), image, audio or other types of data into numerical vectors, and then searches in the vector space. The core idea of ​​this method is to map similar items to similar positions in the vector space.

[0056] It should be noted that the first predicted text segment may be the most relevant text segment corresponding to the question sample obtained based on keyword retrieval, and the second predicted text segment may be the most relevant text segment corresponding to the question sample obtained based on vector retrieval.

[0057] Specifically, a financial system question-answering dataset is collected in advance as the target sample set, where the data may include questions raised by users and system document fragments referenced for obtaining answers. For the question samples in the target sample set, the first step is to carry out a keyword search and re-ranking process to obtain the topk most relevant text fragments of the keyword search process, and the second step is to carry out a vector search and re-ranking process to obtain the topk most relevant text fragments of the vector search process.

[0058] A retrieval type label corresponding to the question sample is determined according to the text segment corresponding to the question sample, the first predicted text segment, and the second predicted text segment.

[0059] The search type label may be a label used to indicate which search type is suitable for the question sample. For example, if a question sample uses keyword search, the keyword search output corresponding to the question sample may be 1, and the vector search output may be 0; if a question sample uses vector search, the vector search output corresponding to the question sample may be 1, and the keyword search output may be 0. The final search type label may include four possibilities: [1,0], [0,1], [1,1], and [0,0].

[0060] Specifically, for the question samples, keyword search and vector search are performed respectively to obtain the optimal search answer corresponding to the keyword search and the optimal search answer corresponding to the vector search respectively, and then the output results are evaluated, and the final search type label is determined according to the evaluation results.

[0061] Build a text classification model.

[0062] In this embodiment, the text classification model can be, for example, BERT (Bidirectional Encoder Representations from Transformers, a method for pre-training language representation. BERT is based on the Transformer architecture and can capture deep bidirectional text relationships by pre-training using a large amount of text data), TextCNN (Text Convolutional Neural Networks, a deep learning model that applies convolutional neural networks (CNN) to text data. It uses convolutional layers to extract local features in text and combines these features for text classification or other natural language processing tasks) or Transformer, which is not limited in this embodiment.

[0063] The question samples in the target sample set are input into the text classification model to obtain the predicted retrieval type corresponding to the question samples.

[0064] The predicted search type can be a search type suitable for the predicted question sample, obtained after the text classification model performs search judgment based on the question sample. Exemplarily, the predicted search type can be, for example, a keyword search or a vector search.

[0065] Specifically, the question samples in the target sample set are input into the text classification model, and the predicted retrieval type corresponding to the question samples is output.

[0066] The parameters of the text classification model are trained according to the objective function formed by the predicted retrieval type corresponding to the question sample and the retrieval type label corresponding to the question sample.

[0067] In actual operation, the specific training process of the text classification model can be described as follows: first, collect and preprocess a large amount of labeled text data; then, select a suitable loss function (such as the cross entropy loss function) to measure the difference between the model prediction result and the true label; then, based on the pre-trained basic model, continuously adjust the model parameters through an iterative optimization algorithm (such as the gradient descent method) to minimize the loss function value; during the training process, model verification is also required to ensure its generalization ability; finally, after multiple iterations, a model that can accurately predict text categories is obtained.

[0068] Return to execute the operation of inputting the question samples in the target sample set into the text classification model to obtain the predicted retrieval type corresponding to the question samples until the question discriminator is obtained.

[0069] In this embodiment, the training process of the text classification model is a cyclic iterative process, and the termination condition of the cycle may be, for example, that the number of iterations reaches a preset number and / or the output results of the retrieval type of preset adjacent times are the same, that is, the iterations converge.

[0070] Among them, the number of iterations can be the number of loop iterations from the input of the question sample to the final retrieval type output result. Among them, the preset number of times can be the number of loop iterations set in advance according to actual conditions or experience values, and the specific size of this value is not limited in this embodiment. It should be explained that the preset number of adjacent times can be the number of adjacent loop iterations set in advance according to actual conditions or experience values, and this embodiment does not specifically limit this value. Exemplarily, the preset number of adjacent times is 3, that is, the retrieval type output results are the same for 3 consecutive times, and finally a trained question discriminator is obtained.

[0071] Optionally, determining a retrieval type label corresponding to the question sample according to the text segment corresponding to the question sample, the first predicted text segment, and the second predicted text segment includes:

[0072] An average reciprocal ranking corresponding to the keyword search is determined according to the first predicted text segment and the text segment corresponding to the question sample, and a category output label corresponding to the keyword search is determined according to the average reciprocal ranking corresponding to the keyword search.

[0073] It is known that MRR (Mean Reciprocal Rank) is a commonly used evaluation metric in the field of information retrieval and machine learning, especially in tasks that evaluate ranking quality, such as search engine result ranking, recommendation systems, etc. MRR measures the average quality of query results, especially when considering the position of the most relevant results. Specifically, MRR calculates the average of the reciprocal rankings of each query result. Specifically, for each query, find the most relevant result (usually determined by manual annotation or pre-defined labels), then calculate the reciprocal of the position of that result in the ranked list, and finally average the reciprocal rankings of all queries.

[0074] In the text classification model of this embodiment, for the question samples in the target sample set, first, a keyword search and re-ranking process is carried out to obtain the topk most relevant text fragments of the keyword search process, and second, a vector search and re-ranking process is carried out to obtain the topk most relevant text fragments of the vector search process, and then the output results are evaluated. For example, if the MRR of the keyword search for a certain question sample is 1, the output of the "keyword search" category of the question sample can be 1.

[0075] An average reciprocal ranking corresponding to the vector retrieval is determined according to the second predicted text segment and the text segment corresponding to the question sample, and a category output label corresponding to the vector retrieval is determined according to the average reciprocal ranking corresponding to the vector retrieval.

[0076] For example, for a certain question sample, if the MRR of vector retrieval is 1, the output of the “vector retrieval” category of the question sample can be set to 1.

[0077] The retrieval type label corresponding to the question sample is determined according to the category output label corresponding to the keyword retrieval and the category output label corresponding to the vector retrieval.

[0078] In actual operation, the final category output label corresponding to the text classification model in this embodiment may include four possibilities: [1,0], [0,1], [1,1], and [0,0].

[0079] Optionally, the user question is input into the question discriminator to obtain the retrieval type corresponding to the user question, including:

[0080] The user question is input into the question discriminator to obtain the first function output value corresponding to the keyword search and the second function output value corresponding to the vector search.

[0081] In this embodiment, the selected activation function is sigmoid, wherein the first function output value may be the sigmoid output value corresponding to the keyword search, and the second function output value may be the sigmoid output value corresponding to the vector search.

[0082] In actual operation, keyword search and vector search may give the best search answer at the same time, or may not give the best answer at the same time. Therefore, the question discriminator in the embodiment of the present invention is designed as a multi-label classification model, whose output is two categories of "vector search" and "keyword search", and its activation function is selected as sigmoid. Then, a threshold is set for the output of sigmoid, and the retrieval type output result of the text classification model is finally determined by judging the threshold value.

[0083] The search type corresponding to the user question is determined according to the output value of the first function and the output value of the second function.

[0084] Specifically, the retrieval type corresponding to the final user question can be determined by setting a threshold for the output of sigmoid and comparing the sigmoid values ​​corresponding to the two retrieval types and the threshold.

[0085] Optionally, determining the search type corresponding to the user question according to the output value of the first function and the output value of the second function includes:

[0086] If the output value of the first function is greater than the preset threshold, and the output value of the second function is less than the preset threshold, it is determined that the search type corresponding to the user question is a keyword search.

[0087] The preset threshold may be a sigmoid threshold set in advance based on actual conditions or empirical values, and this embodiment does not limit the specific size of this value.

[0088] Specifically, if the output of the "keyword search" category is greater than the threshold, and the output of the "vector search" category is less than the threshold, the problem discriminator output is "keyword search".

[0089] If the output value of the first function is less than the preset threshold, and the output value of the second function is greater than the preset threshold, it is determined that the search type corresponding to the user question is vector search.

[0090] Specifically, if the output of the "vector search" category is greater than a threshold, and the output of the "keyword search" category is less than a threshold, the problem discriminator outputs "vector search".

[0091] Otherwise, it is determined that the search type corresponding to the user question is a mixed search.

[0092] Specifically, if the output of the "vector search" category and the output of the "keyword search" category are both greater than or less than the threshold, the problem discriminator outputs "mixed search".

[0093] Optionally, before searching the target database according to the user's question, it also includes:

[0094] Get the initial collection of documents.

[0095] In this embodiment, the initial document set may be local financial system documents.

[0096] Each document in the initial document set is divided into blocks to obtain a text block set.

[0097] The text block set may be a set of text blocks obtained by dividing the text into blocks according to the characteristics of financial system documents.

[0098] Specifically, in this embodiment, the local financial system document can be parsed and loaded into text first, and then the text can be divided into blocks according to the characteristics of the system document. For example, a typical block division basis can be, for example, according to Article X, to divide relatively independent clauses into separate text blocks.

[0099] A text database is constructed based on a collection of text blocks.

[0100] The text database may be a database storing a number of text blocks.

[0101] Specifically, the text blocks in the text block set are stored in a database to form a text database.

[0102] Each text block in the text block set is vectorized to obtain a vector database.

[0103] The vector database may be a database storing vectors of vectorized text blocks.

[0104] Specifically, the text blocks in the text block set are vectorized and stored in a database to form a vector database.

[0105] Construct a target database based on the text database and the vector database.

[0106] Specifically, the target database is composed of a text database and a vector database.

[0107] The technical solution of the embodiment of the present invention introduces keyword retrieval and vector retrieval in the retrieval stage and gives full play to the respective advantages of the two retrieval methods, and also introduces a question discriminator to automatically identify the retrieval method that is more suitable for the user's question, thereby further improving the retrieval accuracy. Then, in the answer generation stage, the retrieval enhancement generation technology is used to provide private domain financial system information to the large language model, thereby improving the retrieval accuracy and further improving the question and answer accuracy.

[0108] Embodiment 2

[0109] As an exemplary description of the above invention embodiment, Figure 2 It is a flowchart of a financial system question-answering method based on a large language model in an embodiment of the present invention.

[0110] The embodiment of the present invention proposes a financial system question-answering method based on a large language model, which uses retrieval enhancement generation technology to provide private domain financial system information to the large model. In order to improve the recall rate, since financial system question-answering may involve simple query questions such as special nouns and special phrases, it may also involve semantic query questions of complex questions. In order to improve the retrieval accuracy, the embodiment of the present invention proposes a solution based on keyword retrieval, vector retrieval, hybrid retrieval and re-ranking.

[0111] like Figure 2 As shown in the figure, the implementation process of the financial system question answering method based on the large language model can be described as follows:

[0112] In the first stage, the knowledge base is constructed (the knowledge base is the target database in the above embodiment): first, the local financial system document is loaded and parsed into text, and then the text is divided into blocks according to the characteristics of the system document (a typical block division basis is such as dividing relatively independent clauses into separate text blocks according to Article X). After that, the text blocks are directly stored in the knowledge base to form a text database, and the text is vectorized and stored in the knowledge base to form a vector knowledge base. The target database is composed of the text database and the vector knowledge base.

[0113] The second stage is knowledge base retrieval: the user inputs a question, (the input question is vectorized) text retrieval is performed based on the user question, K relevant text blocks are obtained (text block #1, text block #2...text block #K), and then the K text blocks are reordered according to the user input question, and the K text blocks are reordered from high to low according to the similarity score (for example, text block #5, text block #2...text block #1), and then the top k text blocks that are most relevant to the user question are selected.

[0114] Figure 3 is a flowchart of a text retrieval method in an embodiment of the present invention. In order to improve the accuracy of knowledge base retrieval, the embodiment of the present invention proposes the following Figure 3 The text retrieval scheme shown in the figure can be described as follows:

[0115] After receiving the question input by the user, the question is first input into the question discriminator. The question discriminator uses the question discriminant model to confirm whether the question is suitable for keyword search, vector search or hybrid search. If the question discriminator determines that it is a keyword search, the keyword search process is carried out, and the K text segments most relevant to the user's question (text block #1, text block #2...text block #K) are output; if the question discriminator determines that it is a vector search, the user's question is vectorized and the vector search is used to output the K text segments most similar to the question vector; if the question discriminator determines that it is a hybrid search, the keyword search and vector search are carried out respectively, and each outputs K / 2 most relevant text segments, and the two search results are combined to obtain K search text segments (if the K / 2 most relevant text segments output by the keyword search and the vector search contain the same text segment, they can be displayed separately, or the same part in one of the search results can be deleted and only the rest of the content can be displayed. This embodiment does not limit this, and the user can set it according to actual needs).

[0116] In the above process, the steps for obtaining the problem discrimination model are as follows:

[0117] a) Raw data preparation: The user prepares a financial system question-answering dataset (i.e., the target sample set in the above embodiment), which includes questions and system document fragments referenced by the answers;

[0118] b) Model selection: You can select BERT-based fine-tuning to implement text classification. You only need to pass the input [CLS] + sentence (user question) + [SEP] into BERT, then take out the vector at the [CLS] position and connect it to the fully connected layer for classification. In actual operation, keyword search and vector search may give the optimal search answer at the same time, or may not give the optimal answer at the same time. Therefore, the question discrimination model in this embodiment is designed as a multi-label classification model, and its output is two categories of "vector search" and "keyword search", and its activation function is selected as sigmoid.

[0119] c) Data annotation acquisition: For user questions in the financial system question-answering dataset, we first conduct a keyword search and re-ranking process to obtain the topk most relevant text fragments of the keyword search process; second, we conduct a vector search and re-ranking process to obtain the topk most relevant text fragments of the vector search process, and then evaluate the output results. For example, if the MRR of the keyword search for a certain question is 1, then the output of the "keyword search" category is 1; if the MRR of the vector search is 1, then the output of the "vector search" category is 1. The final label contains four possibilities: [1,0], [0,1], [1,1], and [0,0].

[0120] d) Use the labeled data to train the model in b) to obtain a multi-label classification model for the problem.

[0121] e) Model usage: Set a threshold for the output of sigmoid. If the output of the "keyword retrieval" category is greater than the threshold, while the output of the "vector retrieval" category is less than the threshold, the problem discriminator output is "keyword retrieval"; if the output of the "vector retrieval" category is greater than the threshold, while the output of the "keyword retrieval" category is less than the threshold, the problem discriminator output is "vector retrieval". If the output of the "vector retrieval" category and the output of the "keyword retrieval" category are both greater than or less than the threshold, the problem discriminator output is "mixed retrieval".

[0122] The third stage is the generation of a large model: after obtaining the most relevant text retrieval fragment, the prompt word template (i.e., the answer prompt word template in the above embodiment) is assembled using the user question and the knowledge base retrieval results to form a prompt word (i.e., the answer text in the above embodiment) which is input into the large model for answering.

[0123] The technical solution of the embodiment of the present invention combines keyword search and vector search to give full play to the advantages of the two search methods, so as to solve the situation in the financial system question-answering scenario that includes both simple query problems such as special nouns and special phrases, and semantic query problems involving complex problems. The technical solution of the embodiment of the present invention uses the question discrimination model to automatically identify the more suitable search method for the user's question, thereby improving the search accuracy and further improving the question-answering accuracy.

[0124] Embodiment 3

[0125] Figure 4 is a schematic diagram of the structure of a reply text generation device in an embodiment of the present invention. This embodiment is applicable to the generation of financial system reply text based on a large language model. The device can be implemented in software and / or hardware. The device can be integrated in any device that provides a reply text generation function, such as Figure 4 As shown, the answer text generation device specifically includes: an acquisition and input module 201, a retrieval and sorting module 202 and a determination and input module 203.

[0126] The acquisition and input module 201 is used to acquire user questions and input the user questions into a question discriminator to obtain a search type corresponding to the user questions. The question discriminator is obtained by iteratively training a text classification model with a target sample set, and the target sample set includes: a question sample and a text segment corresponding to the question sample;

[0127] A retrieval and sorting module 202 is used to search the target database according to the user question based on the retrieval type, and sort the retrieved target text segments to obtain a set of text segments corresponding to the user question;

[0128] The determination and input module 203 is used to determine the model input text according to the text segment set, the user question and the answer prompt word template, and input the model input text into the target model to obtain the answer text corresponding to the user question.

[0129] Optionally, the acquisition and input module 201 includes:

[0130] A retrieval unit, configured to perform keyword retrieval and vector retrieval on the question samples in the target sample set, respectively, to obtain a first predicted text segment corresponding to the keyword retrieval and a second predicted text segment corresponding to the vector retrieval;

[0131] A first determining unit, configured to determine a retrieval type label corresponding to the question sample according to the text segment corresponding to the question sample, the first predicted text segment, and the second predicted text segment;

[0132] Establishing a unit for establishing a text classification model;

[0133] A first input unit, used to input the question sample in the target sample set into the text classification model to obtain a predicted retrieval type corresponding to the question sample;

[0134] A training unit, used for training the parameters of the text classification model according to an objective function formed by the predicted retrieval type corresponding to the question sample and the retrieval type label corresponding to the question sample;

[0135] The execution unit is used to return to execute the operation of inputting the question sample in the target sample set into the text classification model to obtain the predicted retrieval type corresponding to the question sample until a question discriminator is obtained.

[0136] Optionally, the first determining unit is specifically configured to:

[0137] Determine an average reciprocal ranking corresponding to the keyword search according to the first predicted text segment and the text segment corresponding to the question sample, and determine a category output label corresponding to the keyword search according to the average reciprocal ranking corresponding to the keyword search;

[0138] Determining an average reciprocal ranking corresponding to the vector retrieval according to the second predicted text segment and the text segment corresponding to the question sample, and determining a category output label corresponding to the vector retrieval according to the average reciprocal ranking corresponding to the vector retrieval;

[0139] The retrieval type label corresponding to the question sample is determined according to the category output label corresponding to the keyword retrieval and the category output label corresponding to the vector retrieval.

[0140] Optionally, the acquisition and input module 201 includes:

[0141] A second input unit, used to input the user question into a question discriminator, and obtain a first function output value corresponding to the keyword search and a second function output value corresponding to the vector search;

[0142] A second determining unit is used to determine a retrieval type corresponding to the user question according to the first function output value and the second function output value.

[0143] Optionally, the second determining unit is specifically configured to:

[0144] If the output value of the first function is greater than a preset threshold, and the output value of the second function is less than the preset threshold, determining that the search type corresponding to the user question is a keyword search;

[0145] If the output value of the first function is less than the preset threshold, and the output value of the second function is greater than the preset threshold, determining that the search type corresponding to the user question is vector search;

[0146] Otherwise, it is determined that the search type corresponding to the user question is a mixed search.

[0147] Optionally, the device further comprises:

[0148] An acquisition module, used to acquire an initial document set;

[0149] A block division module, used for dividing each document in the initial document set into blocks to obtain a text block set;

[0150] A component module, used for forming a text database according to the text block set;

[0151] A vectorization module, used for vectorizing each text block in the text block set to obtain a vector database;

[0152] A construction module is used to construct a target database according to the text database and the vector database.

[0153] The above-mentioned product can execute the answer text generation method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0154] Embodiment 4

[0155] Figure 5 A schematic diagram of the structure of an electronic device 30 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0156] like Figure 5 As shown, the electronic device 30 includes at least one processor 31, and a memory connected to the at least one processor 31 in communication, such as a read-only memory (ROM) 32, a random access memory (RAM) 33, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 31 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 32 or the computer program loaded from the storage unit 38 to the random access memory (RAM) 33. In the RAM 33, various programs and data required for the operation of the electronic device 30 can also be stored. The processor 31, the ROM 32, and the RAM 33 are connected to each other through a bus 34. An input / output (I / O) interface 35 is also connected to the bus 34.

[0157] A number of components in the electronic device 30 are connected to the I / O interface 35, including: an input unit 36, such as a keyboard, a mouse, etc.; an output unit 37, such as various types of displays, speakers, etc.; a storage unit 38, such as a disk, an optical disk, etc.; and a communication unit 39, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 39 allows the electronic device 30 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0158] The processor 31 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 31 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 31 performs the various methods and processes described above, such as the answer text generation method:

[0159] Obtaining a user question and inputting the user question into a question discriminator to obtain a retrieval type corresponding to the user question, wherein the question discriminator is obtained by iteratively training a text classification model with a target sample set, wherein the target sample set includes: a question sample and a text segment corresponding to the question sample;

[0160] Based on the search type, searching in the target database according to the user question, and sorting the retrieved target text fragments to obtain a set of text fragments corresponding to the user question;

[0161] The model input text is determined according to the text segment set, the user question and the answer prompt word template, and the model input text is input into the target model to obtain the answer text corresponding to the user question.

[0162] In some embodiments, the answer text generation method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 38. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 30 via the ROM 32 and / or the communication unit 39. When the computer program is loaded into the RAM 33 and executed by the processor 31, one or more steps of the answer text generation method described above can be performed. Alternatively, in other embodiments, the processor 31 can be configured to execute the answer text generation method in any other appropriate manner (for example, by means of firmware).

[0163] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0164] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0165] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0166] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0167] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0168] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0169] In one embodiment, the embodiment of the present invention further includes a computer program product, the computer program product includes a computer program, and the computer program implements the answer text generation method of any embodiment of the present invention when executed by a processor.

[0170] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0171] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0172] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating an answer text, characterized in that: include: Obtaining a user question and inputting the user question into a question discriminator to obtain a retrieval type corresponding to the user question, wherein the question discriminator is obtained by iteratively training a text classification model with a target sample set, wherein the target sample set includes: a question sample and a text segment corresponding to the question sample; Based on the search type, searching in the target database according to the user question, and sorting the retrieved target text fragments to obtain a set of text fragments corresponding to the user question; The model input text is determined according to the text segment set, the user question and the answer prompt word template, and the model input text is input into the target model to obtain the answer text corresponding to the user question.

2. The method according to claim 1, characterized in that Iterative training of the text classification model through the target sample set includes: Perform keyword search and vector search on the question samples in the target sample set respectively, to obtain a first predicted text segment corresponding to the keyword search and a second predicted text segment corresponding to the vector search; Determine a retrieval type label corresponding to the question sample according to the text segment corresponding to the question sample, the first predicted text segment, and the second predicted text segment; Build a text classification model; Inputting the question sample in the target sample set into the text classification model to obtain the predicted retrieval type corresponding to the question sample; Training the parameters of the text classification model according to an objective function formed by the predicted retrieval type corresponding to the question sample and the retrieval type label corresponding to the question sample; Return to executing the operation of inputting the question sample in the target sample set into the text classification model to obtain the predicted retrieval type corresponding to the question sample, until a question discriminator is obtained.

3. The method according to claim 2, characterized in that Determining a retrieval type label corresponding to the question sample according to the text segment corresponding to the question sample, the first predicted text segment, and the second predicted text segment includes: Determine an average reciprocal ranking corresponding to the keyword search according to the first predicted text segment and the text segment corresponding to the question sample, and determine a category output label corresponding to the keyword search according to the average reciprocal ranking corresponding to the keyword search; Determining an average reciprocal ranking corresponding to the vector retrieval according to the second predicted text segment and the text segment corresponding to the question sample, and determining a category output label corresponding to the vector retrieval according to the average reciprocal ranking corresponding to the vector retrieval; The retrieval type label corresponding to the question sample is determined according to the category output label corresponding to the keyword retrieval and the category output label corresponding to the vector retrieval.

4. The method according to claim 1, characterized in that: Inputting the user question into a question discriminator to obtain a search type corresponding to the user question includes: Input the user question into a question discriminator to obtain a first function output value corresponding to keyword search and a second function output value corresponding to vector search; The retrieval type corresponding to the user question is determined according to the first function output value and the second function output value.

5. The method according to claim 4, characterized in that Determining the search type corresponding to the user question according to the first function output value and the second function output value includes: If the output value of the first function is greater than a preset threshold, and the output value of the second function is less than the preset threshold, determining that the search type corresponding to the user question is a keyword search; If the output value of the first function is less than the preset threshold, and the output value of the second function is greater than the preset threshold, determining that the search type corresponding to the user question is vector search; Otherwise, it is determined that the search type corresponding to the user question is a mixed search.

6. The method according to claim 1, characterized in that Before searching in the target database according to the user question, the method further includes: Get the initial document collection; Dividing each document in the initial document set into blocks to obtain a text block set; Building a text database according to the text block set; Vectorizing each text block in the text block set to obtain a vector database; A target database is constructed according to the text database and the vector database.

7. A device for generating a reply text, characterized in that: include: An acquisition and input module is used to acquire user questions and input the user questions into a question discriminator to obtain a search type corresponding to the user questions. The question discriminator is obtained by iteratively training a text classification model with a target sample set, wherein the target sample set includes: a question sample and a text segment corresponding to the question sample; A retrieval and sorting module, used to search in a target database according to the user question based on the retrieval type, and sort the retrieved target text segments to obtain a set of text segments corresponding to the user question; The determination and input module is used to determine the model input text according to the text segment set, the user question and the answer prompt word template, and input the model input text into the target model to obtain the answer text corresponding to the user question.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the answer text generation method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the answer text generation method according to any one of claims 1 to 6 when executed.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the answer text generation method according to any one of claims 1 to 6.

Images