Question answering method and question answering system

Abstract

The embodiment of the invention provides a question answering method and a question answering system. In the question answering method, the question answering system obtains a target question to be answered, retrieves from an internal data source and an external data source of the question answering system based on the target question to obtain a plurality of reference texts, and performs at least two rounds of screening on the plurality of reference texts based on reference values of the plurality of reference texts for answering the target question, and determining one or more target texts from the plurality of reference texts, and guiding the first large model to answer the target question based on the one or more target texts to obtain an answer result of the target question. Wherein the reference value of the reference text rejected in each round of screening is lower than the reference value of the reference text not rejected, and the evaluation modes of the reference values in different rounds of screening are different.

Description

TECHNICAL FIELD

[0001] The present specification relates to the field of artificial intelligence, and in particular, to a question answering method and a question answering system. BACKGROUND

[0002] In the field of natural language processing (NLP), large models have shown good performance in multiple tasks. However, since large models are usually trained based on data up to a certain time period, and may lack knowledge in some professional fields, the generated content may have the problem of "hallucination", that is, the output of the large model looks reasonable but is actually inaccurate, fabricated, misleading or irrelevant to the question. Therefore, how to alleviate the "hallucination" problem of large models and improve the quality of their output content has become a problem that needs to be solved.

[0003] The content of the background art section is only information known to the inventors and does not mean that the above information has been in the public domain before the filing date of the present disclosure, nor does it mean that it can be prior art of the present disclosure. SUMMARY

[0004] The present specification provides a question answering method and a question answering system, which can guide a large model by referring to reference texts with high reference value, thereby avoiding the "hallucination" problem and improving the accuracy of the large model in answering questions.

[0005] In a first aspect, the present specification provides a question answering method applied to a question answering system, comprising: obtaining a target question to be answered; retrieving a plurality of reference texts based on the target question from a plurality of data sources, the plurality of data sources comprising an internal data source and an external data source of the question answering system; performing at least two rounds of screening on the plurality of reference texts based on reference values of the plurality of reference texts for answering the target question, to determine N target texts from the plurality of reference texts, wherein the reference value of a reference text eliminated in each round of screening is lower than the reference value of a reference text not eliminated, the evaluation method of the reference value is different in different rounds of screening, and N is an integer greater than or equal to 1; and then guiding a first large model to answer the target question based on the N target texts, to obtain an answer result of the target question.

[0006] In some embodiments, the determining the N target texts from the plurality of reference texts based on the reference values of the plurality of reference texts for answering the target question comprises: determining a first reference value of each of the plurality of reference texts for answering the target question, and removing the reference texts with the first reference value lower than a preset threshold from the plurality of reference texts to obtain M candidate texts, where M is an integer greater than or equal to N; and determining a second reference value of each of the M candidate texts for answering the target question, and determining the N candidate texts with the top N second reference values as the N target texts, where the evaluation dimension of the second reference value is more than that of the first reference value.

[0007] In some embodiments, the evaluation dimension of the first reference value comprises a content correlation dimension, and the evaluation dimension of the second reference value comprises at least two of the following: a content correlation dimension, a content timeliness dimension, a content authority dimension, and a content completeness dimension.

[0008] In some embodiments, the first reference value of each reference text is obtained by: inputting the reference text and the target question into a relevance evaluation model to obtain a relevance between the reference text and the target question, where the relevance evaluation model is trained to have the ability to evaluate the relevance between two texts from a content correlation dimension; and taking the relevance as the first reference value of the reference text.

[0009] In some embodiments, the relevance evaluation model is trained based on the following: obtaining a plurality of original samples, each original sample comprising an original question and an original reference text; expanding each original sample into at least one enhanced sample using at least one sample expansion method, thereby obtaining a plurality of enhanced samples, each enhanced sample comprising a question and a reference text; determining the corresponding label information of each of the plurality of original samples and the plurality of enhanced samples, where the label information of each sample represents the content correlation between the question and the reference text in the sample; and fine-tuning a first base large model using the plurality of original samples and the corresponding label information thereof, and the plurality of enhanced samples and the corresponding label information thereof, to obtain the relevance evaluation model.

[0010] In some embodiments, the augmenting each original sample to obtain at least one augmented sample by using at least one sample augmentation manner comprises at least one of the following: performing content expansion on original reference text in the original sample to obtain augmented reference text, and generating one augmented sample based on the augmented reference text and original question in the original sample; performing content extraction on original reference text in the original sample to obtain augmented reference text, and generating one augmented sample based on the augmented reference text and original question in the original sample; performing content mining on original reference text in the original sample to obtain augmented reference text, and generating one augmented sample based on the augmented reference text and original question in the original sample; or performing question decomposition on original question in the original sample to obtain a plurality of augmented questions, and generating a plurality of augmented samples based on the plurality of augmented questions and original reference text in the original sample.

[0011] In some embodiments, the performing content expansion on original reference text in the original sample to obtain augmented reference text comprises: generating first guide instructions based on original reference text and original question in the original sample, the first guide instructions being used to guide a second large model to add expansion content related to the original question in the original reference text; and inputting the first guide instructions into the second large model to obtain the augmented reference text output by the second large model.

[0012] In some embodiments, the performing content extraction on original reference text in the original sample to obtain augmented reference text comprises: generating second guide instructions based on original reference text and original question in the original sample, the second guide instructions being used to guide a second large model to extract part of content related to the original question in the original reference text; and inputting the second guide instructions into the second large model to obtain the augmented reference text output by the second large model.

[0013] In some embodiments, the performing content mining on original reference text in the original sample to obtain augmented reference text comprises: generating third guide instructions based on original reference text and original question in the original sample, the third guide instructions being used to guide a second large model to mine and summarize content directly related to the original question in the original reference text; and inputting the third guide instructions into the second large model to obtain the augmented reference text output by the second large model.

[0014] In some embodiments, the question decomposition on the original question in the original sample obtains a plurality of enhanced questions, including: generating a fourth guide instruction based on the original question in the original sample, the fourth guide instruction being used to guide a second large model to decompose the original question into a plurality of questions; and inputting the fourth guide instruction into the second large model to obtain the plurality of enhanced questions output by the second large model.

[0015] In some embodiments, the at least one sample expansion manner is configured to expand the original sample with the goal of improving the content correlation between the original question in the original sample and the original reference text.

[0016] In some embodiments, the second reference value corresponding to each candidate text is obtained by inputting the candidate text and the target question into a reference degree evaluation model to obtain the second reference value of the candidate text for answering the target question, where the reference degree evaluation model is trained to have the ability to evaluate the reference value of the current reference text for answering the current question from the at least two dimensions.

[0017] In some embodiments, the reference degree evaluation model is obtained by the following method: obtaining a plurality of original samples, each original sample including an original question and an original reference text; determining the corresponding label information of each of the plurality of original samples, where the label information corresponding to each original sample represents the second reference value of the original reference text in the original sample for answering the original question; and using the plurality of original samples and their corresponding label information, fine-tuning a second base large model to obtain the reference degree evaluation model.

[0018] In some embodiments, the retrieving a plurality of reference texts based on the target question from a plurality of data sources includes: modifying the target question to obtain a modified question, where the modification method includes at least one of the following: modifying incorrect information in the target question, or supplementing information in the target question; and retrieving the plurality of reference texts based on the modified question from the plurality of data sources.

[0019] In some embodiments, the target question belongs to a target domain, and a part of the plurality of data sources are general data sources, and another part of the plurality of data sources are special data sources of the target domain.

[0020] In some embodiments, the guiding the first large model to answer the target question based on the N target texts includes: guiding the first large model to answer the target question based on the N target texts and their reference values.

[0021] In a second aspect, the specification provides a question and answer method applied to a question and answer system, including: obtaining a target question to be answered; retrieving a plurality of reference texts from a plurality of data sources based on the target question, the plurality of data sources including an internal data source and an external data source of the question and answer system; performing a round of screening on the plurality of reference texts based on reference values of the plurality of reference texts for answering the target question to determine N target texts from the plurality of reference texts, wherein the reference value of a reference text eliminated in the screening process is lower than the reference value of a reference text not eliminated in the screening process, the evaluation dimensions of the reference value in the screening process include at least two dimensions: a content relevance dimension, a content timeliness dimension, a content authority dimension, and a content completeness dimension, and N is an integer greater than or equal to 1; and guiding a first large model to answer the target question based on the N target texts to obtain an answer result of the target question.

[0022] In a third aspect, the specification also provides a question and answer system, including: at least one storage medium storing at least one instruction set; and at least one processor in communication connection with the at least one storage medium, wherein the at least one processor reads the at least one instruction set when running, and implements the question and answer method according to the indication of the at least one instruction set.

[0023] In a fourth aspect, the specification provides a computer-readable non-transitory storage medium, wherein the computer-readable non-transitory storage medium stores at least one instruction set, and the at least one instruction set is executed by at least one processor to implement the question and answer method according to the first aspect.

[0024] Other functions of the question and answer method, the question and answer system and the storage medium provided by the specification will be partially listed in the following description. The creative aspects of the question and answer method, the question and answer system and the storage medium provided by the specification can be fully explained by practicing or using the methods, systems and combinations described in the following detailed examples. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the specification, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the specification, and those skilled in the art can also obtain other drawings according to these drawings without any creative labor.

[0026] Figure 1 A schematic diagram of an application scenario of a question and answer provided according to an embodiment of the specification is shown;

[0027] Figure 2 A hardware structure diagram of a question and answer system provided according to an embodiment of the specification is shown;

[0028] Figure 3 A flow chart of a question and answer method is shown according to an embodiment of the present specification;

[0029] Figure 4 A specific schematic diagram of a retrieval, screening and answering process is shown according to an embodiment of the present specification;

[0030] Figure 5 A schematic diagram of a training method of a relevance evaluation model is shown according to an embodiment of the present specification;

[0031] Figure 6 A generation schematic diagram of an enhanced sample is shown according to an embodiment of the present specification;

[0032] Figure 7 Screening performance of different models for reference text is shown according to an embodiment of the present specification; and

[0033] Figure 8 A schematic diagram of a training method of a relevance evaluation model is shown according to an embodiment of the present specification. DETAILED DESCRIPTION

[0034] The following description provides specific applications and requirements of the present specification, so as to enable those skilled in the art to manufacture and use the contents of the present specification. Various local modifications of the disclosed embodiments are obvious to those skilled in the art, and the general principles defined herein can be applied to other embodiments and applications without departing from the spirit and scope of the present specification. Therefore, the present specification is not limited to the shown embodiments, but to the widest scope consistent with the claims.

[0035] The terms used herein are only for the purpose of describing specific example embodiments, and are not limiting. For example, unless the context clearly indicates otherwise, as used herein, the singular forms "a", "an" and "the" can also include the plural forms. When used in the present specification, the terms "include", "contain" and / or "have" mean that the associated integers, steps, operations, elements and / or components exist, but do not exclude the presence of one or more other features, integers, steps, operations, elements, components and / or groups or additional features, integers, steps, operations, elements, components and / or groups can be added in the system / method.

[0036] These and other features of this specification, as well as the operation and function of the associated elements of the structure, and the economical assembly and manufacture of the components, can be significantly improved with consideration of the following description. Reference is made to the accompanying drawings, all of which form a part of this specification. However, it should be expressly understood that the drawings are for illustration and description purposes only and are not intended to limit the scope of this specification. It should also be understood that the drawings are not drawn to scale.

[0037] The flowcharts used in this specification illustrate operations implemented by systems according to some embodiments of the present specification. It should be clearly understood that the operations of the flowcharts may not be implemented in sequence. Rather, the operations may be implemented in reverse order or simultaneously. Furthermore, one or more additional operations may be added to the flowcharts. One or more operations may be removed from the flowcharts.

[0038] The following combination Figure 1 This section introduces the application scenarios of the questions and answers provided in this manual.

[0039] Figure 1 FIG2 shows a schematic diagram of an application scenario of question and answer provided according to an embodiment of this specification. Figure 1 As shown, the application scenario 001 includes a question object 100 and a question-answering system 200 .

[0040] In some embodiments, the question-answering system 200 may be an independent system, and the question object 100 is a user who can directly ask questions to the question-answering system 200 .

[0041] Exemplarily, the question-answering system 200 is an independent application (APP). Figure 1 As shown, question-answering system 200 includes client 10 and server 20. Client 10 can be deployed on a terminal device. The terminal device can display the interactive interface of client 10 to the user, where the user can enter a target question. Client 10 can send the target question to server 20. Server 20 can determine the answer to the target question using a large model and then send the answer to client 10. After receiving the answer, client 10 can display the answer to the user through the interactive interface.

[0042] The terminal device may include mobile devices such as mobile phones, smart watches, tablet computers, laptops, built-in devices in motor vehicles or the like, or any combination thereof, and this specification does not limit this.

[0043] In some embodiments, the question-answering system 200 may be built into an external system (a system external to the question-answering system 200 ) to provide question-answering services thereto. In this case, the question object 100 is the external system.

[0044] For example, the external system is a standalone APP, which can directly face users. The question and answer system 200 is an internal system embedded in the standalone APP. As shown in Figure 1 The question and answer system 200 includes a server 20. The external system can input a target question to the server 20 through an internal interface. The server 20 answers the target question through a large model and displays the answer result to the external system through the internal interface.

[0045] A large model refers to a deep learning model with a large number of parameters and a complex architecture, which can process massive data and complete various complex tasks such as natural language processing, computer vision, speech recognition, and multi-modal learning. The large model can have the following characteristics:

[0046] 1. Large-scale parameters: A large model can contain hundreds of millions, tens of billions, or even thousands of billions of parameters, so it can capture and learn extremely subtle and complex features in data, thus exhibiting strong performance in handling complex tasks.

[0047] 2. Complex architecture: A large model uses a deep neural network architecture, such as the Transformer architecture, which is based on the self-attention mechanism (Self-Attention). It can dynamically measure the importance of each part of the input data, support parallel processing, and effectively capture long-distance dependencies.

[0048] 3. Large-scale training set: A large model requires a large data set during training, containing hundreds of millions of text, images, or other types of data.

[0049] 4. High computing demand: A large model requires a large amount of computing resources during training. Specialized hardware such as GPU clusters or TPU (Tensor Processing Unit) can be used to accelerate the training process.

[0050] The large model in this specification can be a language-based large model, i.e., a type of large model used in the field of natural language processing (NLP) for processing text data and understanding natural language. For example, a large language model (LLM), which is not limited in this specification.

[0051] The large model can be deployed inside the question and answer system 200, and the question and answer system 200 can locally call the large model when answering the target question. The large model can also be deployed outside the question and answer system 200, and the question and answer system 200 can remotely call the large model, which is not limited in this specification.

[0052] In some embodiments, the question-answering system 200 can employ a retrieval augmented generation (RAG) technique before using the large model to answer the target question. That is, the question-answering system 200 retrieves reference text related to the target question from a data source, so as to guide the large model to answer the target question based on the retrieved reference text, so as to avoid the "hallucination" problem of the large model.

[0053] For example, the question-answering system 200 retrieves at least one reference text from a data source. The question-answering system 200 can evaluate the at least one reference text as a whole by using an evaluation tool to obtain a degree of relevance between the at least one reference text and the target question. When the evaluation result is that the at least one reference text is relevant to the target question, the question-answering system 200 can guide the large model to answer the target question based on the at least one reference text. When the evaluation result is that the at least one reference text is not relevant to the target question, the question-answering system 200 needs to retrieve other reference text from other data sources again, so as to evaluate the other reference text as a whole again.

[0054] The traditional evaluation tool uses a T5-Large model, which has a small number of parameters and is trained in a simple manner with less training data, resulting in insufficient performance in understanding text semantics. Therefore, the evaluation dimension is single and the evaluation is rough when evaluating as a whole, so the evaluation accuracy is not high. In addition, the process of retrieving different data sources each time and evaluating as a whole after each retrieval is time-consuming. For a single retrieval, if multiple reference texts are retrieved and the degrees of relevance of each reference text to the target question are uneven, and there is a lack of reference text highly relevant to the target question, the overall evaluation may determine that the multiple reference texts are not relevant to the target question. After similar multiple retrievals, if each overall evaluation determines that it is not relevant, the question-answering system will miss a small amount of reference text that is somewhat relevant to the target question, resulting in the inability to answer the target question and affecting the user's question-answering experience.

[0055] In some embodiments, the question-answering system 200 can expand the range of data sources for the first retrieval to cover more comprehensive data source types, so as to obtain a large amount of reference text. However, this approach may introduce noise text (i.e., text not relevant to the target question), which may interfere with the large model and result in low-quality answers output by the large model.

[0056] Therefore, the embodiments of the present specification provide a question-answering method, a question-answering system, and a storage medium thereof, which can obtain reference text with high reference value by screening, avoid the "hallucination" problem of the large model, and improve the accuracy of the large model in answering questions.

[0057] In some embodiments, the question-answering system 200 can store data and instructions for implementing the question-answering method P300 and can execute or be used to execute the data and instructions. In some embodiments, the question-answering system 200 can include a hardware device with data information processing functions and necessary programs required to drive the hardware device to work.

[0058] The present specification does not limit whether the question-answering system 200 includes all or part of the server 20 and the client 10. The question-answering method P300 can be executed by the client 10 or the server 20 alone, or can be executed by the client 10 and the server 20 in cooperation, and the present specification does not limit this.

[0059] Hereinafter, the question-answering system 200 is taken as the executor of the question-answering method P300, and the operations of the server 20 and the client 10 are not described separately. The technical details of the question-answering method P300 are described hereinafter, and will not be described here.

[0060] In addition, the question-answering system 200 can be applied to different subfields, including but not limited to the e-commerce field, the payment field, the medical field, etc. Hereinafter, the medical field is mainly taken as an example for introduction, but the present specification is not limited thereto.

[0061] Figure 2 A hardware structure diagram of the question-answering system 200 provided according to an embodiment of the present specification is shown.

[0062] As shown in Figure 2 , the question-answering system 200 can include at least one storage medium 230 and at least one processor 220. In some embodiments, the question-answering system 200 can further include a communication port 250 and an internal communication bus 210. In addition, the question-answering system 200 can further include an I / O component 260.

[0063] The internal communication bus 210 can connect different system components. For example, the internal communication bus 210 can connect the storage medium 230, the processor 220, the communication port 250, and the I / O component 260.

[0064] The I / O component 260 supports the input / output between the question-answering system 200 and other components.

[0065] The communication port 250 is used for data communication between the question-answering system 200 and the outside world. For example, the communication port 250 can be used for data communication between the question-answering system 200 and the network. The communication port 250 can be a wired communication port or a wireless communication port.

[0066] In some embodiments, the network can be any type of wired or wireless network, or a combination thereof. For example, the network can include a cable network, a wired network, a fiber optic network, a telecommunication network, an intranet, the Internet, a Local Area Network (LAN), a Wide Area Network (WAN), a Wireless Local Area Network (WLAN), a Metropolitan Area Network (MAN), a Public Switched Telephone Network (PSTN), a Bluetooth network™, a short-range wireless network (ZigBee™), a Near Field Communication (NFC) network, or the like.

[0067] In some embodiments, the network can include one or more network access points. For example, the network can include wired or wireless network access points, such as a base station or an Internet exchange point. Through the access point, one or more components of each device corresponding to the question-answering system 200 can be connected to the network to exchange data or information.

[0068] The storage medium 230 can include a data storage device. The data storage device can be a non-transitory storage medium or a transitory storage medium. For example, the data storage device can include one or more of a disk 232, a read-only memory (ROM) 234, or a random access memory (RAM) 236. The storage medium 230 further includes at least one instruction set stored in the data storage device. The instruction set can include computer program code, which can include programs, routines, objects, components, data structures, procedures, modules, and the like, that perform the question-answering methods provided in the present specification.

[0069] The processor 220 can be communicatively connected to the storage medium 230. The processor 220 is configured to execute the at least one instruction set. When the question-answering system 200 is running, the processor 220 reads the at least one instruction set and performs the question-answering methods provided in the present specification according to the instructions of the at least one instruction set.

[0070] The processor 220 can be in the form of one or more processors. In some embodiments, the processor 220 can include one or more hardware processors, such as a microcontroller, a microprocessor, a reduced instruction set computer (RISC), an application-specific integrated circuit (ASIC), an application-specific instruction set processor (ASIP), a central processing unit (CPU), a graphics processing unit (GPU), a physics processing unit (PPU), a microcontroller unit, a digital signal processor (DSP), a field programmable gate array (FPGA), an advanced RISC machine (ARM), a programmable logic device (PLD), any circuit or processor capable of executing one or more functions, or the like, or any combination thereof.

[0071] For the sake of illustration only, one processor 220 is shown in the question-answering system 200. However, it should be noted that the question-answering system 200 can also include multiple processors in the present specification. Therefore, the operations and / or method steps disclosed in the present specification can be performed by one processor as described in the present specification, or jointly performed by multiple processors. For example, if the processor 220 of the question-answering system 200 performs step A and step B in the present specification, it should be understood that step A and step B can also be performed jointly or separately by two different processors 220 (e.g., a first processor performs step A, a second processor performs step B, or the first and second processors jointly perform steps A and B). Figure 2

[0072] Figure 3 A flowchart of a question-answering method P300 provided according to an embodiment of the present specification is shown. The question-answering system 200 can perform the question-answering method P300.

[0073] As shown in the question-answering method P300, the question-answering system 200 can perform the following steps. Figure 3

[0074] S310: Obtain a target question to be answered.

[0075] The target question can be a question input by a questioning object to the question-answering system 200, for example, a question input by a user in an input box of an interactive interface. The target question can also be a question input by a user in an original question, and then the original question is transformed and automatically input to the question-answering system 200 by other systems (such as a browser, etc.) when the question-answering system 200 is called by the other systems, or the target question can also be an original question automatically input by other systems to the question-answering system 200.

[0076] The target question can be a question in a general field or a question in a subdivided field, which is not limited in the present specification. Hereinafter, the subdivided field will be mainly taken as an example for introduction. For example, taking the medical field (or referred to as the health management field) as an example, the target question is "What is the medical insurance payment base?".​​

[0077] S320: retrieving a plurality of reference texts based on the target question from a plurality of data sources, the plurality of data sources comprising internal data sources of the question answering system and external data sources.

[0078] The question answering system 200 can directly retrieve from the plurality of data sources if the target question is correct and sufficient information. However, if the target question has defects, such as typos or lack of important information, the question answering system 200 can correct the target question and then retrieve from the plurality of data sources. For example, in the medical field, different regions may have different policies, so if the target question lacks regional information, the question answering system 200 can hardly give an accurate answer.

[0079] In some embodiments, the question answering system 200 corrects the target question to obtain a corrected question, and then retrieves the plurality of reference texts based on the corrected question from the plurality of data sources. The correction method includes at least one of the following: modifying incorrect information in the target question or supplementing information in the target question.

[0080] For example, the target question is "What is the medical insurance payment count in region A?", and there is a typo "count". At this time, the question answering system 200 needs to correct the target question, and the corrected question is "What is the medical insurance payment base in region A?".

[0081] For example, the target question is "What is the medical insurance payment base?", and it lacks regional information. The question answering system 200 needs to supplement the target question. For example, the question answering system 200 can obtain the user's positioning information based on the terminal device used by the user, and then supplement the target question based on the positioning information. The corrected question is "What is the medical insurance payment base in region A?". In this way, it can avoid the problem that the question answering system 200 retrieves a large number of unsuitable reference texts due to incorrect or insufficient information in the target question, which increases the time-consuming problem of subsequent screening, and also avoids the problem that suitable reference texts are not retrieved, which affects the accuracy of subsequent answers.

[0082] Hereinafter, the corrected question and the original target question without correction are collectively referred to as the target question for convenience of description.

[0083] The internal data sources of the question answering system 200 can include one or more data sources, and the external data sources can also include one or more data sources, which are not limited in the present specification.

[0084] For example, one of the internal data sources can cover Frequently Asked Questions (FAQ) and their corresponding answers. For example, the data source includes a plurality of texts, each text containing a question and the answer corresponding to the question, or each text containing one or more knowledge pieces / chunks. The specific form of the content in each data source is not limited in the present specification.

[0085] The external data sources can be data sources provided by some platforms outside the question and answer system 200, including but not limited to data sources provided by official platforms and data sources provided by non-official platforms. The official platforms include official agencies related to the medical field, etc.

[0086] The more data sources there are, the more comprehensive the reference texts retrieved by the question and answer system 200 are, and the more all the content related to the target question can be covered, so that the question and answer system 200 can retrieve all the reference texts related to the target question from all the data sources at one time, avoiding the time-consuming problem of multiple retrievals and evaluations and screenings.

[0087] In some embodiments, the target question belongs to a target field, and some of the plurality of data sources are general data sources, and the other part are special data sources of the target field. Thus, the question and answer system 200 can retrieve both relatively professional reference texts in the target field and general reference texts related to the target question, providing rich and comprehensive reference basis for subsequent answers.

[0088] The general data source refers to a data source covering multiple fields and not limited to a specific field, which can provide general knowledge in multiple fields. The special data source refers to a data source providing knowledge in a specific field. Taking the medical field as an example, the special data source can provide professional knowledge in the medical field. When the special data source includes a plurality of data sources, different data sources can correspond to different professional directions. For example, some data sources provide professional knowledge on medical insurance payment base and payment ratio, and some data sources provide professional knowledge on medical insurance reimbursement ratio and reimbursement conditions. In addition, the data source can be in the form of a knowledge graph or other forms, which are not limited in the present specification.

[0089] The general data sources in the plurality of data sources can all be external data sources, or all be internal data sources, or part be external data sources and part be internal data sources, which are not limited in the present specification. The special data sources in the plurality of data sources can all be external data sources, or all be internal data sources, or part be external data sources and part be internal data sources, which are not limited in the present specification.

[0090] For example, Figure 4As shown, the general data sources in the plurality of data sources include data source T1, data source T2, …, and the special data sources in the plurality of data sources include data source Z1, data source Z2, … The plurality of reference texts retrieved by the question and answer system 200 from the plurality of data sources and matched with the target question include: reference text #1 and reference text #2 retrieved from data source T1, reference text #3 and reference text #4 retrieved from data source T2, …, reference text #7 retrieved from data source Z1, reference text #8 and reference text #9 retrieved from data source Z2, and so on.

[0091] It should be noted that the number of reference texts retrieved by the question and answer system 200 from each data source can be the same or different, and the present specification does not limit this.

[0092] S330: Based on the reference value of the plurality of reference texts for answering the target question, at least two rounds of screening are performed on the plurality of reference texts to determine N target texts from the plurality of reference texts, wherein the reference value of the reference texts eliminated in each round of screening is lower than the reference value of the reference texts not eliminated, the evaluation method of the reference value is different in different rounds of screening, and N is an integer greater than or equal to 1.

[0093] For example, the evaluation dimension set includes at least two evaluation dimensions in the content relevance dimension, the content timeliness dimension, the content authority dimension, and the content completeness dimension. The question and answer system 200 can combine the above evaluation dimensions in advance to form the evaluation method of the reference value in each round of screening.

[0094] In the combination process, the evaluation dimensions and the number of evaluation dimensions involved in the evaluation method of the reference value in each round of screening can be different. Alternatively, as the number of screening rounds increases, the number of evaluation dimensions involved in the evaluation method can also increase. The question and answer system 200 can flexibly set different evaluation dimension combinations for each round of screening under the premise that at least two rounds of screening cover the above plurality of evaluation dimensions.

[0095] Hereinafter, two rounds of screening are mainly described, and the execution method of more than two rounds of screening is similar and will not be described in detail. It is assumed that the evaluation method of the reference value in the first round of screening corresponds to evaluation dimension subset 1, the evaluation method of the reference value in the second round of screening corresponds to evaluation dimension subset 2, and evaluation dimension subset 1 and evaluation dimension subset 2 are both subsets of the evaluation dimension set.

[0096] Exemplarily, the number of evaluation dimensions in the evaluation dimension subset 2 is more than the number of evaluation dimensions in the evaluation dimension subset 1, and the evaluation dimensions in the evaluation dimension subset 2 include at least part of the evaluation dimensions in the evaluation dimension subset 1. Or the number of evaluation dimensions in the evaluation dimension subset 2 is equal to the number of evaluation dimensions in the evaluation dimension subset 1, and the evaluation dimensions in the evaluation dimension subset 2 and the evaluation dimensions in the evaluation dimension subset 1 are partially different or totally different. Or the evaluation dimension subset 1 and the evaluation dimension subset 2 can also have other combinations, which will not be listed one by one.

[0097] In some embodiments, the question and answer system 200 respectively determines a first reference value of the plurality of reference texts for answering the target question, and removes the reference texts with a first reference value lower than a first preset threshold from the plurality of reference texts to obtain M candidate texts, M being an integer greater than or equal to N.

[0098] The question and answer system 200 performs a first round of screening on the plurality of reference texts through the first reference value. In the first round of screening, the question and answer system 200 determines a first reference value of each reference text in the plurality of reference texts for answering the target question, thereby obtaining the respective first reference values of the plurality of reference texts.

[0099] Exemplarily, the evaluation dimension of the first reference value includes a content correlation dimension. That is, in the first round of screening, the question and answer system 200 mainly considers the content correlation between the reference text and the target question, and first removes the reference texts that are not related to the target question in content.

[0100] The question and answer system 200 can determine the first reference value of each reference text in various ways. For example, taking reference text #1 as an example, the question and answer system 200 can divide the reference text #1 into a token vector, divide the target question into a token vector, and determine the matching degree of the token vector corresponding to the reference text #1 and the token vector corresponding to the target question, and determine the first reference value of the reference text #1 according to the matching degree. For another example, the question and answer system 200 can generate appropriate guidance instructions based on the reference text #1 and the target question, guide the large model to output the content correlation between the reference text #1 and the target question, and process the correlation degree output by the large model as the first reference value. For another example, the question and answer system 200 can also determine the first reference value of each reference text based on a pre-trained correlation evaluation model.

[0101] This specification mainly introduces the determination of the first reference value of each reference text through the correlation evaluation model. The following takes reference text #1 in the plurality of reference texts as an example for introduction, and the other reference texts are similar and will not be described in detail.

[0102] In some embodiments, the question and answer system 200 inputs the reference text #1 into the relevance evaluation model to obtain the relevance between the reference text #1 and the target question, and takes the relevance as the first reference value corresponding to the reference text #1. The relevance evaluation model is trained to have the ability to evaluate the degree of relevance between two texts from the content relevance dimension.

[0103] In this embodiment, the question and answer system 200 only needs to pre-set the calling mode of the relevance evaluation model, does not need to master the evaluation principle of the relevance, has low use threshold and high efficiency. Moreover, when the relevance evaluation model is trained, it can learn rich features from a large amount of training data, and the determined first reference value is relatively accurate and reliable.

[0104] After the question and answer system 200 obtains the first reference value of each reference text through the relevance evaluation model, it can eliminate the reference texts in the plurality of reference texts whose first reference value is lower than the first preset threshold to obtain the M candidate texts. Thus, the question and answer system 200 can first coarsely screen all the retrieved reference texts from the basic content relevance dimension in the first round of screening, greatly reduces the number of reference texts that need to be finely screened subsequently, reduces the time consumption of subsequent screening, eliminates the reference texts with insufficient content relevance, can avoid misleading the large model subsequently, and thus improves the accuracy of the answer of the large model.

[0105] Continuing to take Figure 4 as an example, the question and answer system 200 eliminates the reference texts #2, #3 and #9 with insufficient first reference value through the first preset threshold, and thus obtains the candidate texts #1, #4, #7 and #8.

[0106] After the question and answer system 200 obtains the M candidate texts through the first round of screening, it can determine the second reference value of the M candidate texts for answering the target question, and determines the N candidate texts with the top N second reference value in the M candidate texts as the N target texts. The evaluation dimension of the second reference value is more than that of the first reference value, so that the question and answer system 200 can finely screen the M candidate texts based on the second reference value, obtain the target texts with relatively high reference value from each dimension, and further improve the accuracy of the answer of the large model subsequently.

[0107] Exemplarily, the evaluation dimensions of the second reference value include at least two of the following: a content relevance dimension, a content timeliness dimension, a content authority dimension, or a content completeness dimension. For example, the question and answer system 200 considers the content timeliness dimension on the basis of the first round of screening, and can eliminate reference texts with expired content to avoid the answer result of the large model being inconsistent with the current situation and misleading the user. For another example, the question and answer system 200 considers the content authority dimension on the basis of the first round of screening, and can eliminate reference texts retrieved from data sources that are not authoritative enough, so that the result of the answer of the large model is more reliable. For another example, the question and answer system 200 considers the content completeness dimension on the basis of the first round of screening, and can eliminate reference texts that are not complete enough, so as to avoid the answer result of the large model being incomplete and affecting the user experience. The more evaluation dimensions of the second reference value, the higher the comprehensive reference value of the target text obtained by the question and answer system 200.

[0108] Hereinafter, the evaluation dimensions of the second reference value are mainly taken as examples of including the content relevance dimension, the content timeliness dimension, the content authority dimension, and the content completeness dimension. The implementation manners of the evaluation dimensions of the second reference value including other combinations are similar, and will not be described in detail.

[0109] The question and answer system 200 can determine the second reference value of each candidate text in multiple ways. For example, the question and answer system 200 sets multiple judgment conditions for each evaluation dimension of the second reference value, the number of satisfied judgment conditions is different, the scores obtained in the evaluation dimension are different, and the weights corresponding to different evaluation dimensions can be different. Taking the candidate text #1 as an example, the question and answer system 200 can determine the second reference value of the candidate text #1 according to the number of judgment conditions satisfied by the candidate text #1 in each evaluation dimension, the importance of the satisfied judgment conditions, and the weight of each evaluation dimension. For another example, the question and answer system 200 can generate appropriate guide instructions based on the M candidate texts and the target question, and guide the large model to comprehensively consider the content relevance dimension, the content timeliness dimension, the content authority dimension, and the content completeness dimension to sort the second reference value of the M candidate texts. For another example, the question and answer system 200 can also determine the second reference value of each candidate text based on a pre-trained reference degree evaluation model.

[0110] This specification mainly introduces the determination of the second reference value of each candidate text by the reference degree evaluation model. Hereinafter, the candidate text #1 in the M candidate texts is mainly taken as an example for introduction, and the other candidate texts are similar, and will not be described in detail.

[0111] In some embodiments, question-answering system 200 inputs candidate text #1 and the target question into a reference value evaluation model to obtain a second reference value of candidate text #1 for answering the target question. The reference value evaluation model is trained to evaluate the reference value of the current reference text for answering the current question based on the dimensions of content relevance, content timeliness, content authority, and content completeness.

[0112] In this embodiment, question-answering system 200 only needs to pre-set the invocation method of the reference evaluation model, without having to understand the evaluation principles of the second reference value. This approach offers low barriers to entry and high efficiency. Furthermore, during training, the reference evaluation model can learn rich features from a large amount of training data, making the determined second reference value relatively accurate and reliable.

[0113] Continue with Figure 4 For example, question-answering system 200 similarly obtains the second reference value of each candidate text and, using a second preset threshold, eliminates reference text #1, reference text #8, and so on, which have insufficient second reference values, thereby obtaining target texts such as reference text #4 and reference text #7. Alternatively, question-answering system 200 may preset N, so that after obtaining the second reference value of each candidate text, the top N candidate texts ranked by second reference value are selected as the N target texts.

[0114] S340: guiding the first large model to answer the target question based on the N target texts, and obtaining an answer result of the target question.

[0115] Continue with Figure 4 For example, the question answering system 200 inputs the target texts such as reference text #4 and reference text #7 and the target question into the first large model, and can also input the following description information into the first large model ( Figure 4 (not shown): "Answer the target question using the text listed this time as a reference." Thus, the question-answering system 200 can obtain the answer to the target question output by the large model.

[0116] In some embodiments, the question-answering system 200 guides the first large model to answer the target question based on the N target texts and their reference values, and obtains the answer result of the target question, so that the first large model can give priority to referring to the texts with higher reference value to answer, further improve the quality of the answer, and avoid the answer result being too long.

[0117] Exemplarily, the question and answer system 200 inputs each target text, the second reference value corresponding to each target text, and the target question into the first large model, and inputs the following description information into the first large model: "answer the target question with reference to the listed texts and their reference values, and focus on the texts with high reference values". Thus, the question and answer system 200 can obtain the answer result of the target question output by the large model. The answer result is, for example, "the upper and lower limits of the staff medical insurance payment base in region A are generally determined according to 60% to 300% of the monthly average wage of the previous year. In xx year, the maximum base in region A is …, and the minimum base is …".

[0118] Exemplarily, the question and answer system 200 inputs each target text, the reference value ranking of each target text, and the target question into the first large model, and inputs the above similar description information into the first large model to obtain the above answer result output by the large model.

[0119] In addition, it should be noted that in S330, when the question and answer system 200 filters the plurality of reference texts based on the reference value of the plurality of reference texts for answering the target question, the question and answer system 200 can also perform only one round of filtering to obtain the N target texts. The number of filtering times is not limited in the present specification. However, when only one round of filtering is performed, the reference value of each reference text needs to be considered from the above-mentioned plurality of evaluation dimensions. For example, the question and answer system 200 considers at least two dimensions: content relevance dimension, content timeliness dimension, content authority dimension, and content completeness dimension, to obtain target texts with high reference value from multiple aspects. When determining the reference value of each reference text, the question and answer system 200 can directly call the above-mentioned reference degree evaluation model, and the specific process of filtering can be referred to the above, which will not be described here.

[0120] In summary, the question and answer method P300 and the question and answer system 200 provided in the present specification can perform extensive retrieval from multiple data sources, and obtain all reference texts related to the target question at one time, thereby avoiding the time-consuming problem of multiple retrievals and multiple evaluations. In addition, all the retrieved reference texts can be filtered for one or more rounds. When multiple rounds of filtering are performed, different evaluation methods are considered each time. When one round of filtering is performed, multiple evaluation dimensions can be considered. Finally, target texts with high reference value from multiple aspects can be filtered, so that the first large model can be guided to answer with reference to the target texts, and the accuracy of the answer result can be improved.

[0121] It can be understood that before the question-answering system 200 executes the question-answering method P300, the question-answering system 200 needs to deploy the pre-trained relevance evaluation model and reference evaluation model, so as to call the relevance evaluation model and reference evaluation model in the process of executing the question-answering method P300. The relevance evaluation model and the reference evaluation model can be pre-trained by the question-answering system 200, or can be pre-trained by other systems, and the present specification does not limit this. Hereinafter, the pre-training of the question-answering system 200 is mainly taken as an example for introduction.

[0122] Firstly, the training process of the relevance evaluation model is introduced below. Figure 5 A flowchart of a training method P400 provided according to an embodiment of the present specification is shown. The training method P400 is executed by the question-answering system 200. As shown in the figure, Figure 5 The training method P400 includes the following steps.

[0123] S410: Obtain a plurality of original samples, each of which includes an original question and an original reference text.

[0124] The question-answering system 200 can collect user questions, and can obtain at least one reference text related to the user question from a plurality of data sources through the user question, and process the user question and each reference text into an original sample.

[0125] For example, for the original question #1, the question-answering system 200 can obtain the original reference text #1 and the original reference text #2 from a plurality of data sources, and thus process them to obtain the original sample #1 and the original sample #2. The original sample #1 includes the original question #1 and the original reference text #1. The original sample #2 includes the original question #1 and the original reference text #2. Similarly, the question-answering system 200 can obtain a plurality of original samples. Each original sample can belong to a target field, or can belong to other fields, and the present specification does not limit this.

[0126] S420: Expand each original sample to obtain at least one enhanced sample by using at least one sample expansion method, so as to obtain a plurality of enhanced samples in total, wherein each enhanced sample includes a question and a reference text.

[0127] The question-answering system 200 can use the same sample expansion method for all original samples, or the question-answering system 200 can use different sample expansion methods for original samples with different characteristics, and the present specification does not limit this.

[0128] The question in the augmented sample can be the original question or an augmented question processed from the original question, and the present specification does not limit the reference text in the augmented sample. The reference text in the augmented sample can be the original reference text or an augmented reference text processed from the original reference text, and the present specification does not limit the reference text in the augmented sample.

[0129] The following takes the original sample #1 as an example to introduce the process of the question and answer system 200 using at least one sample expansion manner to obtain an augmented sample from the original sample #1, and other original samples are similar and will not be described in detail.

[0130] In some embodiments, the question and answer system 200 expands the original sample #1 to obtain at least one augmented sample using at least one of the following sample expansion manners.

[0131] Sample expansion manner 1: expanding the original reference text #1 in the original sample #1 to obtain an augmented reference text #1, and generating an augmented sample #1 based on the augmented reference text #1 and the original question #1;

[0132] Sample expansion manner 2: cutting the original reference text #1 in the original sample #1 to obtain an augmented reference text #2, and generating an augmented sample #2 based on the augmented reference text #2 and the original question #1;

[0133] Sample expansion manner 3: mining the original reference text #1 in the original sample #1 to obtain an augmented reference text #3, and generating an augmented sample #3 based on the augmented reference text #3 and the original question #1; or

[0134] Sample expansion manner 4: decomposing the original question #1 in the original sample #1 to obtain multiple augmented questions, and generating multiple augmented samples based on the multiple augmented questions and the original reference text #1 in the original sample #1.

[0135] It can be understood that if the original reference text #1 in the original sample #1 is short in content, the question and answer system 200 can obtain the augmented sample #1 using the sample expansion manner 1. If the original reference text #1 in the original sample #1 is too long and contains a lot of redundant information, the question and answer system 200 can obtain the augmented sample #2 using the sample expansion manner 2. If the original reference text #1 in the original sample #1 does not contain the content directly answering the original question #1, the question and answer system 200 can obtain the augmented sample #3 using the sample expansion manner 3. If the original question #1 in the original sample #1 is complex, the question and answer system 200 can split the question using the sample expansion manner 4, and then obtain the augmented sample.

[0136] In some embodiments, when the question and answer system 200 configures the sample expansion manner for the original sample, the original sample can be expanded to obtain an enhanced sample with the goal of improving the content correlation between the original question in the original sample and the original reference text. In this way, the number of positive samples and negative samples can be balanced in the subsequent training process, avoiding the problem that the model considers that most of the samples are irrelevant between the reference content and the question during training, and cannot mine features that have a substantial impact on relevance, thereby causing overfitting.

[0137] The positive sample can be understood as a sample in which the relevance between the question (enhanced question or original question) and the reference text (enhanced reference text or original reference text) is higher than or equal to the first preset threshold. The negative sample can be understood as a sample in which the relevance between the question (enhanced question or original question) and the reference text (enhanced reference text or original reference text) is lower than the first preset threshold.

[0138] It should be noted that the question and answer system 200 can adopt a suitable sample expansion manner based on the characteristics of the original sample, thereby generating diverse and covering various types of enhanced samples, increasing the number of positive samples, alleviating the overfitting of the relevance evaluation model, and improving the generalization ability of the relevance evaluation model.

[0139] The following continues to take the original sample #1 as an example to introduce the specific process of the question and answer system 200 obtaining the enhanced sample based on each sample expansion manner.

[0140] Sample expansion manner 1

[0141] For example, as shown in Figure 6 , the question and answer system 200 generates a first guide instruction based on the original reference text #1 (d) and the original question #1 (q) in the original sample #1, inputs the first guide instruction into the second large model, and obtains the enhanced reference text #1 (D1) output by the second large model. The first guide instruction is used to guide the second large model to add expansion content related to the original question #1 in the original reference text #1. After obtaining the enhanced reference text #1, the question and answer system 200 can obtain an enhanced sample #1 containing the enhanced reference text #1 and the original question #1.

[0142] In this example, the question and answer system 200 can use the knowledge of the second large model itself to enrich the original reference text #1, thereby expanding the original reference text #1 to obtain the enhanced reference text #1.

[0143] Sample expansion manner 2

[0144] For example, continuing to refer to Figure 6The question and answer system 200 generates a second guide instruction based on the original reference text 1 and the original question 1 in the original sample 1, inputs the second guide instruction into the second large model, and obtains an enhanced reference text 2 (D2) output by the second large model. The second guide instruction is used to guide the second large model to intercept the part of the original reference text 1 related to the original question 1 and eliminate the part of the original reference text 1 unrelated to the original question 1. After obtaining the enhanced reference text 2, the question and answer system 200 can obtain an enhanced sample 2 containing the enhanced reference text 2 and the original question 1.

[0145] In this example, the question and answer system 200 can use the contextual understanding capability of the second large model to intercept and compress the long original reference text 1 to obtain the enhanced reference text 2, and improve the relevance between the enhanced reference text 2 and the original question 1 in the enhanced reference text 2.

[0146] Sample expansion method 3

[0147] For example, continuing to refer to Figure 6 The question and answer system 200 generates a third guide instruction based on the original reference text 1 and the original question 1 in the original sample 1, inputs the third guide instruction into the second large model, and obtains an enhanced reference text 3 (D3) output by the second large model. The third guide instruction is used to guide the second large model to mine and summarize the content directly related to the original question 1 in the original reference text 1. After obtaining the enhanced reference text 3, the question and answer system 200 can obtain an enhanced sample 3 containing the enhanced reference text 3 and the original question 1.

[0148] In this example, it is assumed that there is no content in the original reference text 1 that can directly answer the original question 1, and then the question and answer system 200 can guide the second large model to mine the hidden content that can answer the original question 1 from the original reference text 1 through the third guide instruction, and improve the relevance between the enhanced reference text 3 and the original question 1 in the enhanced reference text 3. For example, the original question 1 is “What is the medical insurance payment base of employees in company A?” The original reference text 1 includes the medical insurance payment base corresponding to different regions and the business address of company A. It can be seen that the original reference text 1 does not include the direct content that can answer the original question 1. Therefore, it is necessary to use the contextual understanding capability and analysis and summarization capability of the second large model to mine the direct content that can answer the original question 1 from the indirect content related to the original question 1 in the original reference text 1. The direct content is, for example, “The medical insurance payment base of employees in company A is ……”.

[0149] Sample expansion method 4

[0150] For example, continuing to refer to Figure 6The question and answer system 200 generates a fourth guide instruction based on the original question #1 in the original sample #1, inputs the fourth guide instruction into the second large model, and obtains a plurality of enhanced questions output by the second large model. The fourth guide instruction is used to guide the second large model to decompose the original question #1 into a plurality of questions (such as Q1 and Q2). After the question and answer system 200 obtains the plurality of enhanced questions, for each enhanced question, an enhanced sample containing the original reference text #1 and the enhanced question can be obtained, and then a plurality of enhanced samples are obtained. For example, the question and answer system 200 splits the original question #1 into enhanced question #1 (Q1) and enhanced question #2 (Q2), and then the question and answer system 200 can obtain enhanced sample #4 and enhanced sample #5 by using the sample expansion mode 4. The enhanced sample #4 includes the enhanced question #1 and the original reference text #1. The enhanced sample #5 includes the enhanced question #2 and the original reference text #1.

[0151] In this example, the question and answer system 200 can use the second large model to split the relatively complex original question #1 to obtain relatively simple questions. For example, the original question #1 “xx disease how to claim” can be split into the following questions: “xx disease claim, which materials need to be prepared”, “xx disease claim, which institution should be gone to” and “xx disease claim ratio” and the like. Thus, the question and answer system 200 can assemble the split questions and the original reference text #1 to obtain a plurality of enhanced samples. The correlation between the enhanced question and the original reference text is different in different enhanced samples, thereby increasing the diversity of the training samples.

[0152] In the sample expansion mode of the above example, the question and answer system 200 uses the second large model to generate enhanced questions or enhanced reference texts. Therefore, the question and answer system 200 can process different original samples by using the same guide instruction, improve the generation efficiency of the enhanced samples, and use the language understanding ability of the second large model and its own knowledge to improve the quality of the generated enhanced samples, thereby providing a solid foundation for subsequent correlation evaluation model training.

[0153] S430: Determine the annotation information corresponding to each of the plurality of original samples and the plurality of enhanced samples, wherein the annotation information corresponding to each sample represents the content correlation between the question in the sample and the reference text.

[0154] Taking the original sample #1 as an example, the question and answer system 200 can determine the annotation information of the original sample #1. The annotation information of the original sample #1 represents the content correlation between the original question #1 in the original sample #1 and the original reference text #1. The annotation information of the original sample #1 can be a specific numerical value. For example, a numerical value between 1-10 or a numerical value between 1-100, which is not limited in the present specification.

[0155] Exemplarily, the question and answer system 200 can guide the third large model to label the original sample #1 through the fifth guidance instruction, and improve the labeling efficiency. The fifth guidance instruction can represent the original question #1 and the original reference text #1 in the original sample #1, and can also represent the numerical range of the labeling information, and can also represent that the labeling is based on the content correlation degree, and the larger the numerical value is, the higher the content correlation degree is. By analogy, the question and answer system 200 labels each original sample and each enhanced sample to obtain the corresponding labeling information of the plurality of original samples and the plurality of enhanced samples.

[0156] It can be understood that the first large model, the second large model and the third large model can be the same large model or different large models, and the present specification does not limit this. In addition, the model used by the question and answer system 200 to label the sample can be other types of models capable of determining the content correlation degree between different texts, and the present specification does not limit this.

[0157] S440: Fine-tuning the first base large model to obtain a correlation evaluation model by using the plurality of original samples and the corresponding labeling information thereof, and the plurality of enhanced samples and the corresponding labeling information thereof.

[0158] Exemplarily, the question and answer system 200 selects a first number of positive samples with labeling information higher than or equal to a first preset threshold from the plurality of original samples and the plurality of enhanced samples, and selects a second number of negative samples with labeling information lower than the first preset threshold, and tries to make the first number close to the second number. Thus, the question and answer system 200 fine-tunes the first base large model based on the first number of positive samples and the labeling information thereof, and the second number of negative samples and the labeling information thereof to obtain a correlation evaluation model.

[0159] Taking the original sample #1 as an example, the question and answer system 200 inputs the original question #1 and the original reference text #1 into the first base large model to obtain the predicted correlation degree between the original question #1 and the original reference text #1 output by the first base large model. Other samples are similar and will not be described in detail. The question and answer system 200 minimizes the difference between the predicted correlation degree and the labeling information of each sample participating in the training as the training target in the process of fine-tuning the first base large model.

[0160] The base large model generally refers to a deep learning model pre-trained on a large amount of text data, which can capture rich language structures and patterns, and can be used as the basis for retraining models for specific tasks. After fine-tuning, an optimized model suitable for a specific task can be obtained.

[0161] Figure 7 The filtering performance of different models for reference texts is shown.

[0162] Model V0 is a deep learning model with more parameters obtained by replacing T5-Large in the traditional evaluation tool, and its training method is the same as that of the traditional evaluation tool. The accuracy of model V0 in the first round of screening of the multiple reference texts retrieved by the question and answer system 200 is 0.54. The accuracy of the first large model in answering based on the reference texts screened by model V0 is 0.41.

[0163] Model V1 also belongs to a deep learning model with more parameters, and the number of negative samples in the training process is more than that of positive samples, and each sample is labeled by binary classification. The accuracy of model V1 in the first round of screening of the multiple reference texts retrieved by the question and answer system 200 is 0.61. The accuracy of the first large model in answering based on the reference texts screened by model V1 is 0.65.

[0164] Model V2 also belongs to a deep learning model with more parameters, and the range of label information of each sample in the training process is 0-10. The accuracy of model V2 in the first round of screening of the multiple reference texts retrieved by the question and answer system 200 is 0.69. The accuracy of the first large model in answering based on the reference texts screened by model V2 is 0.68.

[0165] Model V3 also belongs to a deep learning model with more parameters, and the training samples are constructed for different fields in the training process, and the number of positive samples is less than that of negative samples. The accuracy of model V3 in the first round of screening of the multiple reference texts retrieved by the question and answer system 200 is 0.73. The accuracy of the first large model in answering based on the reference texts screened by model V3 is 0.80.

[0166] Model V4 is a relevance evaluation model trained by the training method P400 of the present specification. The accuracy of model V4 in the first round of screening of the multiple reference texts retrieved by the question and answer system 200 is 0.90. The accuracy of the first large model in answering based on the reference texts screened by model V4 is 0.82. It can be seen that the screening accuracy of the relevance evaluation model obtained by the above training method P400 is high, and the accuracy of the first large model in answering can be improved.

[0167] In summary, the training method P400 provided by the present specification can train a relevance evaluation model for evaluating the reference value of the reference text from the content relevance dimension. The relevance evaluation model has high accuracy in evaluating the relevance of the reference text. In the actual answering process, the question and answer system 200 can accurately screen the multiple reference texts through the relevance evaluation model, thereby saving the time-consuming of subsequent screening, and improving the answering efficiency of the large model as a whole.

[0168] For the reference degree evaluation model, Figure 8A flowchart of a training method P500 is shown according to an embodiment of the present specification. The training method P500 is executed by the question answering system 200. As shown, the training method P500 includes the following steps. Figure 8

[0169] S510: Obtain a plurality of original samples, each of which includes an original question and an original reference text.

[0170] S510 and S410 are similar, except that the original reference text in the sample trained by the training method P500 needs to include relevant information in multiple dimensions. The relevant information, such as the publication time of the original reference text, is conducive to the reference degree evaluation model to mine features affecting the second reference value from the content timeliness dimension. The relevant information, such as the search data source corresponding to the original reference text, is conducive to the reference degree evaluation model to mine features affecting the second reference value from the content authority dimension. Other similar contents (such as the acquisition method of the original sample, etc.) to S410 can be referred to the description above, and will not be repeated here.

[0171] S520: Determine the annotation information corresponding to each of the plurality of original samples, wherein the annotation information corresponding to each original sample represents the second reference value of the original reference text in the original sample to answer the original question.

[0172] S520 and S430 are different in that the question answering system 200 in S430 only considers the content relevance dimension when labeling, but the question answering system 200 in S520 needs to consider the content relevance dimension, the content timeliness dimension, the content authority dimension and the content completeness dimension when labeling. In addition, other aspects (such as the labeling method, the form of the annotation information, etc.) are similar, and the specific content can be referred to the description above, and will not be repeated here.

[0173] S530: Fine-tune the second base large model using the plurality of original samples and their corresponding annotation information to obtain a reference degree evaluation model.

[0174] S530 and S440 are similar, and the specific content can be referred to the description above, and will not be repeated here. The second base large model and the first base large model can be the same large model, or can be different large models, which are not limited by the present specification.

[0175] In summary, the training method P500 provided by the present specification can train a reference degree evaluation model that evaluates the reference value of a text to answer a question from multiple evaluation dimensions. In the actual answering process, the question answering system 200 considers more comprehensive dimensions through the reference value of each text obtained by the reference degree evaluation model, and the reliability is higher, which can improve the accuracy of the large model to answer the question.

[0176] ​The reference texts mentioned in the specification can be in the form of documents or other forms, and the specification does not limit this.

[0177] In addition, the specification mainly introduces the medical field as an example when introducing related embodiments, so the specific content involved (such as the specific content of the target problem, the type of data source, etc.) is also developed in the medical field, but the specification is not limited to this. When the question and answer method P300 is applied to other fields, the specific content can be adjusted based on the characteristics of other fields, and will not be listed one by one.

[0178] In another aspect of the present specification, a computer-readable non-transitory storage medium storing at least one set of instructions executable by a processor for question answering is provided. When the instructions are executed by the processor, the instructions direct the processor to implement the steps of the question answering method P300, the training method P400 or the training method P500 described in the present specification. In some possible implementation, each aspect of the present specification can also be implemented as a program product in the form of a program code. When the program product is run on the question answering system 200, the program code is used to cause the question answering system 200 to perform the steps of the question answering method P300, the training method P400 or the training method P500 described in the present specification. The program product for implementing the above method can include the program code in a portable compact disc read-only memory (CD-ROM) and can be run on the question answering system 200. However, the program product of the present specification is not limited to this, and in the present specification, the readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system. The program product can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, be but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any suitable combination of the above. More specific examples of the readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. The computer-readable storage medium can include a data signal propagating in a baseband or as part of a carrier wave, in which readable program code is borne. Such a propagating data signal can take on many forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium can also be any readable medium that is not a storage medium and that can transmit, propagate or transport program for use by or in connection with an instruction execution system, apparatus or device. The program code contained in the readable storage medium can be transmitted in any suitable medium, including but not limited to wireless, wired, optical, RF, and the like, or any suitable combination of the above. The program code for performing the operations of the present specification can be written in any combination of one or more programming languages, including an object-oriented programming language such as Java, C++, and the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages.The program code can execute entirely on the question and answer system 200, partly on the question and answer system 200, as a stand-alone software package, partly on the question and answer system 200 and partly on a remote computing device, or entirely on the remote computing device.

[0179] The term "and / or" in the embodiments of the present specification describes an association relationship of associated objects, which means that there can be three relationships. For example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. The character " / " generally represents that the associated objects before and after it are in an "or" relationship.

[0180] The terms "first", "second", and the like in the present specification are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence.

[0181] The term "a plurality of" mentioned in the present specification shall be understood as two or more than two, unless otherwise specified.

[0182] The above describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that in the embodiments and still achieve the desired result. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order to achieve the desired result. In some embodiments, multi-task processing and parallel processing are possible or can be advantageous.

[0183] In summary, after reading the detailed disclosure, those skilled in the art can understand that the foregoing detailed disclosure can be presented only in an exemplary manner and can not be limiting. Although it is not explicitly stated here, those skilled in the art can understand that the present specification requires various reasonable changes, improvements and modifications to the embodiments. These changes, improvements and modifications are intended to be presented by the present specification and are within the spirit and scope of the exemplary embodiments of the present specification.

[0184] In addition, certain terms in the present specification have been used to describe the embodiments of the present specification. For example, "one embodiment", "embodiment" and / or "some embodiments" mean that the specific features, structures or characteristics described in connection with the embodiment can be included in at least one embodiment of the present specification. Therefore, it can be emphasized and should be understood that two or more references to "embodiments" or "one embodiment" or "alternative embodiments" in various parts of the present specification do not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics can be appropriately combined in one or more embodiments of the present specification.

[0185] It is to be understood that the foregoing description of the embodiments of the present specification is intended to be illustrative only and that the application is not limited to the embodiments described herein. In some embodiments, well-known structures, materials, and techniques have not been described in detail in order to avoid obscuring aspects of the application. It is to be understood that the foregoing description is purely illustrative and is not intended to be in any way limiting. For example, while the application has been described in terms of particular embodiments, it is clear that modifications and permutations of the described embodiments are also within the scope of the application. Although specific terms have been employed herein, such terms are intended in a descriptive sense only and not for purposes of limitation. The description itself is an enabling disclosure that also includes any additional claimed apparatuses, systems, compositions of matter, computer

[0186] Each patent, patent application, publication of a patent application, and other material, such as articles, books, specifications, publications, documents, literature, and the like (including any and all related prosecution history documents) cited in this disclosure are hereby incorporated by reference in their entirety for all purposes to the same extent as if each were specifically and individually indicated to be incorporated by reference herein. In the event of inconsistencies between the disclosure of this document and the disclosures of the above- incorporated by reference documents, the disclosure of this document will control.

[0187] Finally, it should be understood that the teachings of the present specification can be employed in a variety of applications. The foregoing description of the embodiments of the application discloses only a few variations. Many other variations are possible by those skilled in the art without departing from the spirit and scope of the application as disclosed in the claims. Accordingly, such variations are contemplated as being within the scope of the application.

Claims

1. A question-answering method, applied to a question-answering system, comprising: Get the target question to be answered; Retrieving a plurality of reference texts from a plurality of data sources based on the target question, wherein the plurality of data sources include an internal data source and an external data source of the question-answering system; Based on the reference values ​​of the multiple reference texts for answering the target question, performing at least two rounds of screening on the multiple reference texts to determine N target texts from the multiple reference texts, wherein the reference value of the reference texts eliminated in each round of screening is lower than the reference value of the reference texts not eliminated, the reference value is evaluated differently in different rounds of screening, and N is an integer greater than or equal to 1; as well as The first large model is guided to answer the target question based on the N target texts to obtain an answer result of the target question.

2. The method according to claim 1, wherein The step of performing at least two rounds of screening on the plurality of reference texts based on the reference value of the plurality of reference texts for answering the target question to determine N target texts from the plurality of reference texts includes: Determining first reference values ​​of each of the plurality of reference texts for answering the target question, and removing reference texts from the plurality of reference texts whose first reference values ​​are lower than a preset threshold to obtain M candidate texts, where M is an integer greater than or equal to N; and Determine the second reference values ​​of the M candidate texts for answering the target question respectively, and determine the top N candidate texts ranked in terms of the second reference value among the M candidate texts as the N target texts, wherein the evaluation dimensions of the second reference value are more than the evaluation dimensions of the first reference value.

3. The method according to claim 2, wherein: The evaluation dimension of the first reference value includes a content-related dimension, and the evaluation dimension of the second reference value includes at least two dimensions of a content-related dimension, a content-timeliness dimension, a content-authority dimension, and a content-completeness dimension.

4. The method according to claim 3, wherein: The first reference value corresponding to each reference text is obtained as follows: Inputting the reference text and the target question into a relevance evaluation model to obtain the relevance between the reference text and the target question, wherein the relevance evaluation model is trained to have the ability to evaluate the relevance between two texts from a content relevance dimension; and The relevance is used as the first reference value corresponding to the reference text.

5. The method according to claim 4, wherein The relevance evaluation model is trained based on the following method: obtaining multiple original samples, each original sample including an original question and an original reference text; At least one sample expansion method is used to expand each original sample to obtain at least one enhanced sample, thereby obtaining a total of multiple enhanced samples, wherein each enhanced sample includes a question and a reference text; Determining the annotation information corresponding to each of the plurality of original samples and the plurality of enhanced samples, wherein the annotation information corresponding to each sample represents the degree of content relevance between the question in the sample and the reference text; and The first base large model is fine-tuned and trained using the multiple original samples and their corresponding annotation information, as well as the multiple enhanced samples and their corresponding annotation information, to obtain the relevance evaluation model.

6. The method according to claim 5, wherein: The method of expanding each original sample to obtain at least one enhanced sample by using at least one sample expansion method includes at least one of the following: Expanding the content of the original reference text in the original sample to obtain an enhanced reference text, and generating an enhanced sample based on the enhanced reference text and the original question in the original sample; Performing content interception on the original reference text in the original sample to obtain an enhanced reference text, and generating an enhanced sample based on the enhanced reference text and the original question in the original sample; Performing content mining on the original reference text in the original sample to obtain an enhanced reference text, and generating an enhanced sample based on the enhanced reference text and the original question in the original sample; or The original question in the original sample is decomposed into a plurality of enhanced questions, and a plurality of enhanced samples are generated based on the plurality of enhanced questions and the original reference text in the original sample.

7. The method according to claim 6, wherein: The step of expanding the content of the original reference text in the original sample to obtain the enhanced reference text includes: generating a first guiding instruction based on the original reference text and the original question in the original sample, wherein the first guiding instruction is used to guide the second large model to add extended content related to the original question to the original reference text; and The first guiding instruction is input into the second large model to obtain the enhanced reference text output by the second large model.

8. The method according to claim 6, wherein: The step of performing content interception on the original reference text in the original sample to obtain the enhanced reference text includes: generating a second guiding instruction based on the original reference text and the original question in the original sample, wherein the second guiding instruction is used to guide the second large model to intercept a portion of the original reference text related to the original question; and The second guiding instruction is input into the second large model to obtain the enhanced reference text output by the second large model.

9. The method according to claim 6, wherein: The performing content mining on the original reference text in the original sample to obtain the enhanced reference text includes: generating a third guiding instruction based on the original reference text and the original question in the original sample, wherein the third guiding instruction is used to guide the second large model to mine and summarize the content directly related to the original question in the original reference text; and The third guidance instruction is input into the second large model to obtain the enhanced reference text output by the second large model.

10. The method according to claim 6, wherein: The original question in the original sample is decomposed into multiple enhanced questions, including: generating a fourth guiding instruction based on the original question in the original sample, wherein the fourth guiding instruction is used to guide the second large model to decompose the original question into multiple questions; and The fourth guidance instruction is input into the second large model to obtain the plurality of enhanced questions output by the second large model.

11. The method according to claim 5, wherein: The at least one sample expansion method is configured to expand the original sample with the goal of improving the content relevance between the original question in the original sample and the original reference text.

12. The method according to claim 3, wherein: The second reference value corresponding to each candidate text is obtained in the following way: The candidate text and the target question are input into a reference evaluation model to obtain a second reference value of the candidate text for answering the target question, wherein the reference evaluation model is trained to have the ability to evaluate the reference value of the current reference text for answering the current question from the at least two dimensions.

13. The method according to claim 12, wherein: The reference evaluation model is trained based on the following method: obtaining multiple original samples, each original sample including an original question and an original reference text; Determining the annotation information corresponding to each of the plurality of original samples, wherein the annotation information corresponding to each original sample represents a second reference value of the original reference text in the original sample for answering the original question; and The second base large model is fine-tuned and trained using the multiple original samples and their corresponding annotation information to obtain the reference evaluation model.

14. The method according to claim 1, wherein The step of retrieving a plurality of reference texts from a plurality of data sources based on the target question includes: Correcting the target question to obtain a corrected question, wherein the correction method includes at least one of the following: modifying erroneous information in the target question, or supplementing information in the target question; and The plurality of reference texts are retrieved from the plurality of data sources based on the revised question.

15. The method according to claim 1, wherein The target problem belongs to a target field, part of the multiple data sources are general data sources, and another part of the data sources are special data sources for the target field.

16. The method according to claim 1, wherein The guiding the first large model to answer the target question based on the N target texts includes: The first large model is guided to answer the target question based on the N target texts and their reference values.

17. A question-answering method, applied to a question-answering system, comprising: Get the target question to be answered; Retrieving a plurality of reference texts from a plurality of data sources based on the target question, wherein the plurality of data sources include an internal data source and an external data source of the question-answering system; Based on the reference value of the multiple reference texts for answering the target question, a round of screening is performed on the multiple reference texts to determine N target texts from the multiple reference texts, wherein the reference value of the reference texts eliminated during the screening process is lower than the reference value of the reference texts not eliminated, and the evaluation dimensions of the reference value during the screening process include at least two of the following dimensions: a content relevance dimension, a content timeliness dimension, a content authority dimension, and a content completeness dimension, where N is an integer greater than or equal to 1; and The first large model is guided to answer the target question based on the N target texts to obtain an answer result of the target question.

18. A question-answering system comprising: at least one storage medium storing at least one instruction set; as well as At least one processor is communicatively connected to the at least one storage medium, wherein the at least one processor reads the at least one instruction set during operation and executes the method according to any one of claims 1 to 17 according to the instructions of the at least one instruction set.

Images