Document question answering method and system, electronic device and storage medium

Abstract

The application relates to the technical field of natural language processing, and provides a document question answering method, device and system, electronic equipment and a storage medium. The method uses a coarse ranking model to obtain a plurality of candidate documents in a target document library, uses a fine ranking model to obtain the similarity of each candidate document and a user question, and determines a target document, and further determines a target answer corresponding to the user question. The coarse ranking model and the fine ranking model are based on a question document pair set in a target field, a first difficult negative sample and a second difficult negative sample, and are obtained through multi-round iterative training of a basic coarse ranking model and a basic fine ranking model. The introduction of the difficult negative sample can effectively improve the sample quality, solve the problems of insufficient sample data and difficult labeling in the target field, reduce the difficulty of model training, and further improve the robustness of the coarse ranking model and the fine ranking model obtained through training, and can also improve the accuracy of the target answer.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and particularly relates to a document question answering method and system, an electronic device and a storage medium. BACKGROUND

[0002] With the development of natural language processing technology, the current document question answering method has certain effect in general fields such as encyclopedic knowledge. However, in life, there are many other fields outside the general field that also need to use the document question answering method, such as intelligent question answering of company regulations and systems, real-time hot information question answering, and in-depth question answering of specific field knowledge.

[0003] In the prior art, the document question answering method used in other fields is generally to migrate the document question answering model applied to the encyclopedic knowledge field to the corresponding field. The method used for model migration is generally to increase the problem document pairs of the new field to be migrated for model training on the basis of the pre-training model.

[0004] However, this method simply increases the sample data required for training, which may have the problem of insufficient sample data in the new field and difficult annotation, thereby not only increasing the difficulty of model training, but also reducing the accuracy of the model question answering result. SUMMARY

[0005] The present application provides a document question answering method, system, electronic device and storage medium to solve the defects in the prior art.

[0006] The present application provides a document question answering method, comprising:

[0007] obtaining a user question in a target field;

[0008] inputting the user question into a coarse ranking model to obtain a plurality of candidate documents retrieved from a target document library in the target field by the coarse ranking model;

[0009] inputting the user question and the plurality of candidate documents into a fine ranking model to obtain a similarity between each candidate document output by the fine ranking model and the user question, and determining a target document with a similarity higher than a preset threshold to the user question;

[0010] determining a target answer corresponding to the user question based on the user question, the target document, and the similarity between the target document and the user question;

[0011] The coarse ranking model and the fine ranking model are obtained by performing multi-round iterative training on a basic coarse ranking model and a basic fine ranking model in a basic question and answer model based on a set of question document pairs in the target field, first difficult negative samples of the basic coarse ranking model, and second difficult negative samples of the basic fine ranking model in the basic question and answer model; and the basic question and answer model is used to determine answers to questions in a basic field different from the target field.

[0012] The first difficult negative sample used in each iteration round is determined based on the basic coarse ranking model and the basic fine ranking model obtained in a previous iteration round, and the second difficult negative sample used in each iteration round is determined based on the basic coarse ranking model obtained in the current iteration round.

[0013] According to the document question and answer method provided in the application, the training steps of the coarse ranking model and the fine ranking model include:

[0014] The basic coarse ranking model obtained in the previous iteration round is trained in the current iteration round based on first question document pairs in the set of question document pairs and the first difficult negative samples of the basic coarse ranking model obtained in the previous iteration round.

[0015] The basic fine ranking model obtained in the previous iteration round is trained in the current iteration round based on second question document pairs in the set of question document pairs and the second difficult negative samples of the basic fine ranking model obtained in the current iteration round.

[0016] The basic coarse ranking model obtained in the current iteration round is taken as the basic coarse ranking model obtained in the previous iteration round, the basic fine ranking model obtained in the current iteration round is taken as the basic fine ranking model obtained in the previous iteration round, and the training in the current iteration round is continued until the multi-round iterative training is completed, and the coarse ranking model and the candidate fine ranking model are obtained.

[0017] The fine ranking model is determined based on the candidate fine ranking model.

[0018] According to the document question and answer method provided in the application, the fine ranking model is determined based on the candidate fine ranking model, which includes:

[0019] The question data outside the target field is obtained, and the question data is input into the coarse ranking model to obtain a plurality of first documents retrieved from a reference document library by the coarse ranking model.

[0020] The question data and the plurality of first documents are input into the candidate fine ranking model to obtain a similarity between each first document output by the candidate fine ranking model and the question data, and a second document with a similarity greater than a preset threshold is determined.

[0021] construct a negative sample based on the problem data and the second document with a similarity within a preset range, and train the candidate fine-ranking model based on the negative sample and the second-class problem document pair to obtain the fine-ranking model.

[0022] According to the document question answering method provided in the application, the second difficult negative sample adopted in each iteration round is determined based on the following steps:

[0023] For any problem sample in the first-class problem document pair adopted in the current iteration round, the any problem sample is input into the basic coarse-ranking model trained in the current iteration round to obtain a plurality of third documents retrieved from the reference document library by the basic coarse-ranking model trained in the current iteration round.

[0024] Based on the any problem sample and the plurality of third documents, the second difficult negative sample adopted in the current iteration round is determined.

[0025] According to the document question answering method provided in the application, the determination of the second difficult negative sample adopted in the current iteration round based on the any problem sample and the plurality of third documents comprises:

[0026] The first preset number of third documents with high ranking are removed from the plurality of third documents, and the second difficult negative sample adopted in the current iteration round is determined based on the any problem sample and the third documents remaining after the removal of the first preset number of third documents.

[0027] According to the document question answering method provided in the application, the training steps of the coarse-ranking model and the fine-ranking model comprise:

[0028] A title document pair corresponding to a reference document library is obtained; the reference document library comprises the target document library or comprises the target document library and a document library of the basic field;

[0029] The basic question answering model is pre-trained based on the title document pair to obtain a pre-training model;

[0030] The initial coarse-ranking model and the initial fine-ranking model are trained in multiple rounds based on the problem document pair set in the target field, the first difficult negative sample of the initial coarse-ranking model in the pre-training model and the second difficult negative sample of the initial fine-ranking model in the pre-training model to obtain the coarse-ranking model and the fine-ranking model.

[0031] According to the document question answering method provided in the application, the first-class problem document pair comprises a correct problem document pair and an incorrect problem document pair;

[0032] For any question sample in the first type of question-document pair used in the current iteration round, the determining step of the question error-document pair corresponding to the any question sample comprises:

[0033] determining the question error-document pair corresponding to the any question sample based on the any question sample and the document samples in the same sample batch except the correct document sample in the question correct-document pair corresponding to the any question sample;

[0034] and / or, determining the question error-document pair corresponding to the any question sample based on the any question sample, the base coarse ranking model trained in the previous iteration round and the base fine ranking model.

[0035] According to the document question answering method provided by the application, the user question is input into the coarse ranking model, and a plurality of candidate documents retrieved from the target document library by the coarse ranking model are obtained, which comprises:

[0036] inputting the user question into the question encoder of the coarse ranking model to obtain a question encoding vector output by the question encoder;

[0037] inputting the question encoding vector and the document encoding vector corresponding to each document in the target document library obtained based on the document encoder of the coarse ranking model into the similarity calculation layer of the coarse ranking model to obtain the similarity between the question encoding vector and the document encoding vector corresponding to each document output by the similarity calculation layer;

[0038] inputting the similarity between the question encoding vector and the document encoding vector corresponding to each document into the output layer to obtain the plurality of candidate documents with high similarity in the each document output by the output layer.

[0039] According to the document question answering method provided by the application, the target answer corresponding to the user question is determined based on the user question, the target document and the similarity between the target document and the user question, which comprises:

[0040] inputting the user question and the target document into the understanding model to obtain an answer paragraph in the target document output by the understanding model;

[0041] inputting the user question, the answer paragraph and the similarity between the target document and the user question into the generation model to obtain the target answer output by the generation model;

[0042] The understanding model is trained based on a first question-answer paragraph pair within the target domain, and the generation model is trained based on a second question-answer paragraph pair within the target domain and the similarity between the document sample containing the answer paragraph sample in the second question-answer paragraph pair and the question sample in the second question-answer paragraph pair.

[0043] The present invention also provides a document question-and-answer device, comprising:

[0044] The acquisition module is used to acquire user questions in the target domain;

[0045] The coarse ranking module is used to input the user question into the coarse ranking model and obtain multiple candidate documents retrieved by the coarse ranking model from the target document library of the target domain;

[0046] The fine-ranking module is used to input the user question and the multiple candidate documents into the fine-ranking model, obtain the similarity between each candidate document output by the fine-ranking model and the user question, and determine the target document whose similarity with the user question is higher than a preset threshold.

[0047] The determination module is used to determine the target answer corresponding to the user question based on the user question, the target document, and the similarity between the target document and the user question;

[0048] The coarse-ranking model and the fine-ranking model are obtained by performing multiple rounds of iterative training on the basic coarse-ranking model and the basic fine-ranking model based on the question document pair set in the target domain, the first difficult negative sample of the basic coarse-ranking model in the basic question-answering model, and the second difficult negative sample of the basic fine-ranking model in the basic question-answering model; the basic question-answering model is used to determine the answer to questions in a basic domain that is different from the target domain.

[0049] The first difficult negative sample used in each iteration round is determined based on the basic coarse-ranking model and basic fine-ranking model trained in the previous iteration round, and the second difficult negative sample used in each iteration round is determined based on the basic coarse-ranking model trained in the current iteration round.

[0050] The present invention also provides a document question-and-answer system, comprising: a voice module and the above-described document question-and-answer device;

[0051] The voice module is connected to the document question-and-answer device;

[0052] The voice module is used to collect user questions and transmit the user questions to the document question-and-answer device.

[0053] The application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the document question-answering method according to any one of the above when executing the program.

[0054] The application further provides a non-transitory computer-readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the document question-answering method according to any one of the above.

[0055] The application further provides a computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the document question-answering method according to any one of the above.

[0056] The application provides a document question-answering method, system, electronic device and storage medium. First, a user question in a target field is obtained. Then, a plurality of candidate documents retrieved from a target document library in the target field by using a coarse ranking model are obtained. The similarity between each candidate document and the user question is obtained by using a fine ranking model, and a target document with a similarity to the user question higher than a preset threshold is determined. Finally, a target answer corresponding to the user question is determined based on the user question, the target document and the similarity between the target document and the user question. The coarse ranking model and the fine ranking model are obtained by performing multi-round iterative training on a basic coarse ranking model and a basic fine ranking model based on a set of question document pairs in the target field, first difficult negative samples and second difficult negative samples. The introduction of the difficult negative samples can effectively improve the sample quality, solve the problems of insufficient sample data and difficult labeling in the target field, reduce the difficulty of model training, and thus improve the robustness of the coarse ranking model and the fine ranking model obtained by training, and improve the accuracy of the target answer. Moreover, the coarse ranking model and the fine ranking model are jointly iteratively trained, which can not only improve the training efficiency and shorten the training period, but also effectively improve the recall effect of the coarse ranking model. BRIEF DESCRIPTION OF DRAWINGS

[0057] In order to more clearly illustrate the technical solutions of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description can also be obtained by those skilled in the art without creative labor.

[0058] Figure 1 is a flowchart of a document question-answering method provided by the application;

[0059] Figure 2 is a structural schematic diagram of a document question-answering device provided by the application;

[0060] Figure 3 is a structural schematic diagram of a document question-answering system provided by the application;

[0061] Figure 4 FIG. 1 is a structural schematic diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0062] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0063] In the prior art, the document question answering method used in different fields is generally applied to a mature question answering model or a question answering model with sufficient training data through domain migration. However, the method used in domain migration is generally to take the mature model or the model with sufficient training data as a pre-training model, and to increase the problem document pairs of the new domain to be migrated for model training on the basis of the pre-training model. This method only simply increases the sample data required for training, and there may be problems such as insufficient sample data in the new domain and difficult annotation, thereby not only increasing the difficulty of model training, but also reducing the accuracy of the question answering result of the model.

[0064] Therefore, the present application provides a document question answering method in the embodiments.

[0065] Figure 1 FIG. 1 is a flowchart of the document question answering method provided in the embodiments of the present application, as shown in the figure, the method comprises the following steps: Figure 1

[0066] S1, obtaining a user question in a target domain;

[0067] S2, inputting the user question into a coarse ranking model to obtain a plurality of candidate documents retrieved from a target document library in the target domain by the coarse ranking model;

[0068] S3, inputting the user question and the plurality of candidate documents into a fine ranking model to obtain a similarity between each candidate document output by the fine ranking model and the user question, and determining a target document with a similarity higher than a preset threshold to the user question;

[0069] S4, determining a target answer corresponding to the user question based on the user question, the target document, and the similarity between the target document and the user question;

[0070] ​The first difficult negative sample and the second difficult negative sample are used to train the basic coarse ranking model and the basic fine ranking model in the basic question and answer model based on the question document pair set in the target field, the first difficult negative sample of the basic coarse ranking model, and the second difficult negative sample of the basic fine ranking model.

[0071] The first difficult negative sample used in each iteration round is determined based on the basic coarse ranking model and the basic fine ranking model obtained through previous iteration rounds, and the second difficult negative sample used in each iteration round is determined based on the basic coarse ranking model obtained through the current iteration round.

[0072] Specifically, the document question and answer method provided in the embodiment of the present application has a document question and answer device as an execution subject, which can be configured in a computer, which can be a local computer or a cloud computer, and the local computer can be a computer, a tablet, etc., which is not specifically limited here.

[0073] Firstly, step S1 is performed to obtain a user question in a target field. The target field can be a company regulation and system question and answer field, a real-time hot information question and answer field, a specific field knowledge in-depth question and answer field, etc., which is not specifically limited here.

[0074] The user question can be obtained through a voice module or directly obtained through an external interface. That is, the user question can be in a voice form or a text form, which is not specifically limited here. It can be understood that the embodiment of the present application is for document question and answer, so if the user question is in a voice form, it needs to be converted into a text form, and the conversion method can adopt a conventional conversion method.

[0075] Then, step S2 is performed, and a document question and answer model can be introduced. The document question and answer model can include a coarse ranking model and a fine ranking model. The user question can be input into the coarse ranking model, the similarity between the user question and each document in the target document library is calculated by the coarse ranking model, and a plurality of candidate documents with high similarity to the user question are retrieved from the target document library. That is, the input of the coarse ranking model can include the user question and each document in the target document library, and the output is a plurality of candidate documents.

[0076] Since the encoding of a large-scale document library requires a lot of resources and time, each document in the target document library can be pre-input into the coarse ranking model, and the feature information of each document can be obtained through offline coding, and the index of each document can be established by using a tool such as faiss. Therefore, the coarse ranking model can be a double-tower structure, that is, it can include two encoders, a question encoder q-encoder and a document encoder p-encoder. The question encoder is online coded, and the document encoder is offline coded. Both encoders can be a bidirectional language model such as BERT. Through the coarse ranking model with a double-tower structure, the recall problem can be solved and the efficiency can be improved.

[0077] The coarse ranking model can first encode the user question to obtain the feature information of the user question, and then calculate the similarity between the user question and each document through the feature information of the user question and the feature information of each document, and then select multiple candidate documents from the target document library according to the similarity. Here, the top K documents with high similarity can be directly selected from the target document library as candidate documents, for example. When selecting candidate documents, we can use a tool such as faiss to retrieve from the pre-established index.

[0078] The target document library can be pre-collected, which can include multiple documents of the target domain, and the length of each document is within a preset length range. The documents in the target document library include documents with original lengths within the preset length range, and documents with lengths within the preset length range obtained by splitting documents with original lengths outside the preset length range. That is, for any document of the target domain, if the length of the any document is outside the preset length range, the any document can be first split into multiple documents with lengths within the preset length range, and then the multiple split documents are added to the target document library.

[0079] Thereafter, step S3 is performed, and the user question and the multiple candidate documents are input into the fine ranking model, and the fine ranking model further calculates the similarity between each candidate document and the user question, and then each candidate document can be sorted in descending order of similarity to the user question, and a candidate document with a similarity greater than a preset threshold is selected from the multiple candidate documents as a target document. That is, the input of the fine ranking model can be the splicing result of the user question and each candidate document, and the output can be the similarity between each candidate document and the user question.

[0080] It can be understood that the target document can be 0, one or more, when the target document is 0, that is, there is no document in the target document library with a similarity to the user question higher than the preset threshold, at this time, the user question is regarded as a non-target field class question, and no answer is provided. When the target document is one, that is, there is only one document in the target document library with a similarity to the user question higher than the preset threshold. When the target document is more, that is, there are multiple documents in the target document library with a similarity to the user question higher than the preset threshold, multiple target documents can be sorted in descending order of similarity to the user question.

[0081] The fine ranking model can be a cross-encoder model, that is, the fine ranking model can include a cross-encoder, which performs deep cross-encoding on the spliced result, and obtains the similarity of each candidate document to the user question according to the encoding result, and outputs the similarity through a fully connected layer. The fine ranking model can also be a bidirectional language model such as BERT, and its loss function can be a classification loss.

[0082] For example, the user question q and the candidate document p can be spliced first to obtain a spliced result. A special word [sep] is inserted between the user question q and the candidate document p during splicing to divide the user question q and the candidate document p. Then the spliced result can be input into the fine ranking model, and the cross-encoder CE in the fine ranking model is used to encode the spliced result (q, [sep], p) into a d-dimensional real value vector CE(q, [sep], p), and then the similarity score of the user question q and the candidate document p is output through the fully connected layer Softmax. That is,

[0083] score = Softmax(W · (CE(q, [sep], p))

[0084] Wherein, W is the weight vector of the fully connected layer Softmax in the fine ranking model.

[0085] The coarse ranking model and the fine ranking model can be obtained by training the basic coarse ranking model and the basic fine ranking model in the basic question and answer model through multiple rounds of iterative training of training samples. The training samples can include a set of question and document pairs in the target field, a first difficult negative sample and a second difficult negative sample used in each iteration round. Each first difficult negative sample and each second difficult negative sample includes a question sample and a document sample.

[0086] The basic question and answer model refers to a mature question and answer model or a question and answer model with sufficient training data. The basic question and answer model corresponds to a basic field, and the basic field is usually different from the target field. The two fields can be similar or not similar. Here, the basic field can include at least one of encyclopedic knowledge, news, social dialogue, and Internet question and answer. For example, the basic field can be an encyclopedic knowledge field with good generality, and the basic question and answer model can be an encyclopedic knowledge question and answer model, which can have good document question and answer capability and lay a good foundation for obtaining a document question and answer model. The basic question and answer model can also include a basic rough ranking model and a basic fine ranking model.

[0087] The question and document pair set in the target field refers to a set containing a plurality of question and document pairs in the target field. The number of question and document pairs contained in the question and document pair set can be set as needed, and is not specifically limited here.

[0088] The question and document pair can include a question and correct document pair and a question and incorrect document pair. The question and correct document pair is a positive sample pair, which is composed of a question sample and a document sample where an answer corresponding to the question sample is located. The question and incorrect document pair is a negative sample pair, which is composed of a question sample and a remaining document sample other than the document sample where the answer corresponding to the question sample is located. Each question sample in the question and document pair is a real question, and the question and document pair can be obtained by manual annotation.

[0089] Since the target field is a new field, its annotation data is relatively less, and the question method is relatively simple. Therefore, in order to improve the efficiency of obtaining the question and document pair set, a series of data enhancement methods can be used to increase the question and paragraph pair of the target field. For example, the initial question and document pair of the target field can be collected or annotated first, and then the initial question sample in the initial question and document pair is subjected to data enhancement by synonym replacement, random insertion, random deletion, random replacement, back translation, etc. Then, the new question and document pair is formed by the data enhanced question sample and the initial document pair corresponding to the initial question sample. Thereafter, the initial question and document pair and the new question and document pair are both added to the question and document pair set.

[0090] In addition, if the target field is similar to the basic field, the question and document pair of the basic field can also be added to the question and document pair set. The question sample in the question and document pair of the basic field can also be subjected to data enhancement, and then the new question and document pair of the basic field is obtained and added to the question and document pair set.

[0091] The first difficult negative sample used in each iteration round of the basic coarse ranking model and the basic fine ranking model is the difficult negative sample of the basic coarse ranking model obtained in the previous iteration round, that is, the difficult negative sample required for training the basic coarse ranking model obtained in the previous iteration round in the current iteration round. The difficult negative sample can be determined by the basic coarse ranking model obtained in the previous iteration round and the basic fine ranking model.

[0092] For example, for any question sample in the first type of question document pair used in the current iteration round, the any question sample can be input into the basic coarse ranking model obtained in the previous iteration round to obtain a plurality of document samples retrieved from the reference document library by the basic coarse ranking model obtained in the previous iteration round. Thereafter, the any question sample and the retrieved plurality of document samples are input into the basic fine ranking model obtained in the previous iteration round to determine the similarity between each document sample and the any question sample, and to determine the document sample with a similarity higher than a preset threshold to the any question sample, which forms the first difficult negative sample used in the current iteration round with the any question sample.

[0093] The second difficult negative sample used in each iteration round is the difficult negative sample of the basic fine ranking model obtained in the current iteration round, that is, the difficult negative sample required for training the basic coarse fine ranking model obtained in the previous iteration round in the current iteration round. The difficult negative sample can be determined by the basic coarse ranking model obtained in the current iteration round.

[0094] For example, for any question sample in the first type of question document pair used in the current iteration round, the any question sample can be input into the basic coarse ranking model obtained in the current iteration round to obtain a plurality of document samples retrieved from the reference document library by the basic coarse ranking model obtained in the current iteration round, and to form the second difficult negative sample used in the current iteration round with the any question sample.

[0095] Finally, step S4 is performed, which can first determine the answer paragraph in the target document by reading the target document according to the user question. Here, the specific value of the preset threshold can be set as needed, which is not limited here.

[0096] After determining the answer paragraph in the target document, it can be judged whether the answer paragraph in the target document can answer the user question in combination with the similarity between the target document and the user question. For the answer paragraph in the target document that can answer the user question, the target answer corresponding to the user question is output.

[0097] The document question answering method provided in the embodiment of the present application first acquires a user question in a target field; then, a plurality of candidate documents retrieved from a target document library of the target field by using a coarse ranking model are used to obtain the similarity of each candidate document to the user question by using a fine ranking model, and a target document with a similarity to the user question higher than a preset threshold is determined; and finally, a target answer corresponding to the user question is determined based on the user question, the target document, and the similarity of the target document to the user question. The coarse ranking model and the fine ranking model are obtained by performing multi-round iterative training on a basic coarse ranking model and a basic fine ranking model based on a question document pair set in the target field, a first difficult negative sample, and a second difficult negative sample. The introduction of the difficult negative sample can effectively improve the sample quality, solve the problems of insufficient sample data and difficult labeling in the target field, reduce the difficulty of model training, and thus improve the robustness of the coarse ranking model and the fine ranking model obtained by training, and improve the accuracy of the target answer. Moreover, the joint iterative training of the coarse ranking model and the fine ranking model can not only improve the training efficiency and shorten the training period, but also effectively improve the recall effect of the coarse ranking model.

[0098] On the basis of the above-mentioned embodiment, the document question answering method provided in the embodiment of the present application comprises the following steps:

[0099] inputting the user question and the target document into an understanding model to obtain an answer paragraph in the target document output by the understanding model;

[0100] inputting the user question, the answer paragraph, and the similarity of the target document to the user question into a generation model to obtain the target answer output by the generation model;

[0101] wherein the understanding model is trained based on a first question answer paragraph pair in the target field, and the generation model is trained based on a second question answer paragraph pair in the target field and the similarity of a document sample in which an answer paragraph sample in the second question answer paragraph pair is located to a question sample in the second question answer paragraph pair.

[0102] Specifically, in the embodiment of the present application, the document question answering model can further comprise an understanding model and a generation model. Correspondingly, the basic question answering model can comprise a basic understanding model and a basic generation model.

[0103] The understanding model is used to determine the answer paragraph in each target document, and the input of the understanding model is the same as that of the fine-ranking model, i.e., the spliced result of the user question and each target document, and the output is the answer paragraph in each target document. The understanding model can also only include an encoder for encoding the spliced result of the user question and each target document, and outputting the answer paragraph obtained according to the encoding result through a fully connected layer. The answer paragraph can be represented by a start position and an end position in the target document.

[0104] The generation model is used to determine whether the answer paragraph in each target document can answer the user question, and output the answer paragraph that can answer the user question, i.e., the target answer corresponding to the user question. The input of the generation model includes the user question, the answer paragraph in each target document, and the similarity between each target document and the user question, and the output is the target answer corresponding to the user question.

[0105] The generation model can also only include an encoder, which can also be a cross-encoder for deep interactive encoding of the spliced result. The generation model can be sorted in descending order of the similarity between the answer paragraph and the user question, or can not be sorted in full, but only select a number of answer paragraphs with high similarity to the user question as target answers for output. For example, the answer paragraph with the highest similarity to the user question can be directly selected for output and provided to the user.

[0106] The understanding model can be trained by the first question and answer paragraph pair in the target field, and the training process can include: inputting the question sample in the first question and answer paragraph pair into the initial understanding model to obtain the output result of the initial understanding model, then calculating the loss function value according to the output result and the answer paragraph sample in the first question and answer paragraph pair, and finally updating the model parameters of the initial understanding model according to the loss function value; iteratively perform the input process and the calculation process until the loss function converges or reaches a preset iteration number to obtain the understanding model. It can be understood that the initial understanding model can be a basic understanding model in a basic question and answer model, or other initial neural network models, which are not limited here.

[0107] The generation model can be trained by a second question-answer paragraph pair in a target field and a similarity between a document sample in which an answer paragraph sample in the second question-answer paragraph pair is located and a question sample in the second question-answer paragraph pair, and the training process can include: inputting the question sample in the second question-answer paragraph pair and the similarity between the document sample in which the answer paragraph sample in the second question-answer paragraph pair is located and the question sample in the second question-answer paragraph pair into an initial generation model to obtain an output result of the initial generation model, then calculating a loss function value according to the output result and the answer paragraph sample in the second question-answer paragraph pair, and finally updating model parameters of the initial generation model according to the loss function value; the input process and the calculation process are iteratively performed until the loss function converges or a preset iteration number is reached, and the generation model is obtained. It can be understood that the initial generation model can be a basic generation model in a basic question-answer model, or can be other initial neural network models, which are not specifically limited here.

[0108] In the embodiment of the application, the target answer corresponding to the user question is determined by combining the understanding model and the generation model, which can improve the determination efficiency of the target answer and effectively improve the accuracy of the target answer.

[0109] On the basis of the above-mentioned embodiment, the training steps of the coarse ranking model and the fine ranking model provided in the embodiment of the application include:

[0110] Based on the first type of question-document pair in the set of question-document pairs and the first difficult negative sample of the basic coarse ranking model trained in the previous iteration round, the basic coarse ranking model trained in the previous iteration round is trained in the current iteration round;

[0111] Based on the second type of question-document pair in the set of question-document pairs and the second difficult negative sample of the basic fine ranking model trained in the current iteration round, the basic fine ranking model trained in the previous iteration round is trained in the current iteration round;

[0112] The basic coarse ranking model trained in the current iteration round is taken as the basic coarse ranking model trained in the previous iteration round, and the basic fine ranking model trained in the current iteration round is taken as the basic fine ranking model trained in the previous iteration round, and the training in the current iteration round is continued until the multi-round iteration training is completed, and the coarse ranking model and the candidate fine ranking model are obtained;

[0113] Based on the candidate fine ranking model, the fine ranking model is determined.

[0114] Specifically, in the embodiment of the application, the coarse ranking model and the fine ranking model can be jointly trained. When jointly training, the set of question-document pairs needs to be used as training samples.

[0115] The set of question document pairs can include first question document pairs and second question document pairs, which can be the same or different, and are only used to distinguish different training processes here. The first question document pairs can be used to train the base coarse ranking model in the base coarse ranking model, and the second question document pairs can be used to train the base fine ranking model in the base coarse ranking model.

[0116] Since random negative samples often have low difficulty in distinguishing, the model cannot learn deep semantic information. Therefore, in the joint training process of the coarse ranking model and the fine ranking model, difficult negative samples are introduced, which can improve the robustness of the coarse ranking model and the fine ranking model obtained by training. First, according to the first question document pairs in the set of question document pairs and the first difficult negative samples of the base coarse ranking model obtained by the previous iteration, the base coarse ranking model obtained by the previous iteration is trained in the current iteration.

[0117] Here, the first question document pairs and the first difficult negative samples of the base coarse ranking model obtained by the previous iteration are both used as training samples to train the base coarse ranking model obtained by the previous iteration in the current iteration. In particular, if the current iteration is the first iteration, the previous iteration is 0, and at this time the first difficult negative sample of the base coarse ranking model obtained by the previous iteration can be the given initial difficult negative sample. The initial difficult negative sample can be constructed by the first question document pair with the help of BM25. Each initial difficult negative sample can include a question sample and a document sample, which is not labeled, and may not contain the answer corresponding to the question sample, but is identified by the base coarse ranking model as containing the answer corresponding to the question sample.

[0118] The training process of the base coarse ranking model obtained by the previous iteration in the current iteration includes: inputting the question sample in the training sample into the initial coarse ranking model to obtain the output result of the base coarse ranking model, then calculating the loss function value according to the output result and the document sample in the training sample, and finally updating the model parameters of the base coarse ranking model according to the loss function value; iteratively perform the input process and the calculation process until the loss function converges or reaches a preset number of iterations, and obtain the base coarse ranking model obtained by training in the current iteration.

[0119] Thereafter, the second question document pairs in the set of question document pairs and the second difficult negative samples of the base coarse ranking model obtained by the current iteration can be used to train the base fine ranking model obtained by the previous iteration in the current iteration.

[0120] Here, the second type of question document pair and the second difficult negative sample of the basic coarse ranking model trained in the current iteration round are used as training samples to train the basic fine ranking model trained in the previous iteration round in the current iteration round.

[0121] The training process of the basic fine ranking model in the current iteration round includes: inputting the question sample in the training sample into the basic fine ranking model to obtain an output result of the basic fine ranking model, then calculating a loss function value according to the output result and the document sample in the training sample, and finally updating the model parameters of the basic fine ranking model according to the loss function value; the above input process and calculation process are iteratively executed until the loss function converges or a preset iteration number is reached, and the basic fine ranking model trained in the current iteration round is obtained.

[0122] Thereafter, the basic coarse ranking model trained in the current iteration round is used as the basic coarse ranking model trained in the previous iteration round, and the basic fine ranking model trained in the current iteration round is used as the basic fine ranking model trained in the previous iteration round, and the training in the current iteration round is continued until the multi-round iteration training is completed, and the coarse ranking model and the candidate fine ranking model are obtained.

[0123] Finally, the fine ranking model can be determined according to the candidate fine ranking model. Here, the candidate fine ranking model can be directly used as the fine ranking model, and the candidate fine ranking model can also be selected and trained with training samples to obtain a fine ranking model with stronger capability, which is not limited here.

[0124] In the embodiment of the application, the first difficult negative sample of the basic coarse ranking model and the second difficult negative sample of the basic fine ranking model obtained by each iteration training are used to perform multi-round iteration training on the basic coarse ranking model and the basic fine ranking model, which can effectively improve the sample quality, and thus the robustness of the coarse ranking model and the fine ranking model obtained by training is improved. Moreover, the coarse ranking model and the fine ranking model are jointly iteratively trained, which not only improves the training efficiency and shortens the training period, but also effectively improves the recall effect of the coarse ranking model.

[0125] Since the coarse ranking model and the candidate fine ranking model have certain retrieval capabilities for the questions in the target field after iteration training on the question document pair set of the target field. However, there are still many non-target field questions or non-question phrases that have not been seen by the coarse ranking model and the candidate fine ranking model, and the coarse ranking model and the candidate fine ranking model may retrieve the corresponding answers with high scores, that is, the coarse ranking model and the candidate fine ranking model have the defect of misabsorption and do not have the ability to reject, which will greatly reduce the accuracy of the document question and answer model in the target field.

[0126] To solve the problem of misabsorption, the candidate fine ranking model needs to have certain recognition ability for the phrases of problems or non-problems in the non-target field, that is, the candidate fine ranking model needs to be changed into a fine ranking model.

[0127] On the basis of the above-mentioned embodiments, the document question answering method provided in the embodiments of the present application comprises:

[0128] The problem data outside the target field is acquired and input into the coarse ranking model to obtain a plurality of first documents retrieved from the reference document library by the coarse ranking model;

[0129] The problem data and the plurality of first documents are input into the candidate fine ranking model to obtain the similarity of each first document output by the candidate fine ranking model to the problem data, and a second document with a similarity greater than a preset threshold is determined;

[0130] Based on the problem data and the second document with a similarity within a preset range, a negative sample is constructed, and the candidate fine ranking model is trained based on the negative sample and the second-class problem document pair to obtain the fine ranking model.

[0131] Specifically, when the fine ranking model is determined according to the candidate fine ranking model, the problem data outside the target field can be acquired first, and the problem data is input into the coarse ranking model to obtain a plurality of first documents retrieved from the reference document library by the coarse ranking model. Thereafter, the problem data and the plurality of first documents can be input into the candidate fine ranking model to obtain the similarity of each first document output by the candidate fine ranking model to the problem data, and a second document with a similarity greater than a preset threshold is selected from the plurality of first documents.

[0132] Since the above-mentioned problem data can have some problems in the target field, the coarse ranking model already has strong retrieval ability for the problems in the target field, and often obtains a second document with high similarity through the candidate fine ranking model. Therefore, the second document with a similarity within a preset range needs to be selected, and the preset range can be a range greater than a specified threshold, and the specified threshold can be greater than the preset threshold, so that the first several second documents with high similarity can be excluded. Thereafter, the negative sample can be constructed by using the problem data and the second document with a similarity within the preset range, and the negative sample and the second-class problem document pair are used as training samples to train the candidate fine ranking model, so that the fine ranking model is obtained.

[0133] In the embodiments of the present application, the problem data outside the target field is introduced, and the negative sample is constructed by using the problem data and the second document with a similarity within a preset range determined by the candidate fine ranking model to train the candidate fine ranking model, so that the obtained fine ranking model has certain recognition ability.

[0134] On the basis of the above-mentioned embodiments, the document question answering method provided in the embodiments of the present application determines the second difficult negative sample adopted in each iteration round based on the following steps:

[0135] For any question sample in the first type of question document pair adopted in the current iteration round, the any question sample is input into the basic coarse ranking model trained in the current iteration round to obtain a plurality of third documents retrieved from the reference document library by the basic coarse ranking model trained in the current iteration round.

[0136] Based on the any question sample and the plurality of third documents, the second difficult negative sample adopted in the current iteration round is determined.

[0137] Specifically, in the embodiments of the present application, when determining the second difficult negative sample adopted in each iteration round, for any question sample in the first type of question document pair adopted in the current iteration round, the any question sample can be input into the basic coarse ranking model trained in the current iteration round to obtain a plurality of third documents retrieved from the reference document library by the basic coarse ranking model trained in the current iteration round, that is, the top K third documents with high similarity to the any question sample from the reference document library.

[0138] Then, according to the any question sample and the plurality of third documents, the second difficult negative sample adopted in the current iteration round, that is, the difficult negative sample of the basic fine ranking model trained in the previous iteration round, can be determined. Wherein, each second difficult negative sample adopted in the current iteration round is constituted by the any question sample and each third sample.

[0139] In the embodiments of the present application, the second difficult negative sample adopted in the current iteration round is determined by the basic coarse ranking model trained in the current iteration round, which can reduce the difficulty of obtaining the second difficult negative sample, and further improve the training efficiency of the basic fine ranking model and reduce the training cost.

[0140] On the basis of the above-mentioned embodiments, the document question answering method provided in the embodiments of the present application determines the second difficult negative sample adopted in the current iteration round based on the following steps:

[0141] From the plurality of third documents, the top pre-set number of third documents are eliminated, and based on the any question sample and the third documents remaining after the pre-set number of third documents are eliminated, the second difficult negative sample adopted in the current iteration round is determined.

[0142] Specifically, in the embodiment of the present application, although the documents with high similarity obtained by the basic coarse ranking model trained by the current iteration round can directly obtain the second difficult negative sample adopted by the current iteration round, due to the large reference document library, there is a certain probability that the documents with high similarity obtained by the alternative coarse ranking model are pseudo-negative examples, and too many pseudo-negative examples will interfere with the distinguishing ability of the model.

[0143] Therefore, the top ranked preset number of third documents need to be removed from the plurality of third documents, that is, the preset number of third documents with high similarity are deleted as pseudo-negative examples from the plurality of third documents. The preset number can be set as needed, for example, it can be 5, 10, etc. The preset number can also be used as a hyperparameter in the training process of the current iteration round of the basic fine ranking model trained in the previous iteration round. Herein, no specific limitation is made.

[0144] Thereafter, the second difficult negative sample adopted by the current iteration round is determined according to the any question sample and the third documents remaining after the removal of the preset number of third documents, that is, the any question sample can form a second difficult negative sample with each remaining third document.

[0145] In addition, to remove pseudo-negative examples, a similarity threshold can also be set, but due to the similarity calculation method used in the training of the coarse ranking model, it is difficult to accurately obtain the second difficult negative sample by using a fixed similarity threshold. To remove pseudo-negative examples, the F1 score and other indicators of the matching of the third document and the correct document sample can also be used to exclude pseudo-negative examples, but since the types of answers to open questions are rich and vary in length, it is also difficult to cut a threshold that can be applied to various types of questions and answers.

[0146] In the embodiment of the present application, by removing the top ranked preset number of third documents from the plurality of third documents, the influence of pseudo-negative examples on the training of the basic fine ranking model trained by each iteration round can be excluded, and the precision of the fine ranking model can be improved.

[0147] On the basis of the above-mentioned embodiment, the document question answering method provided in the embodiment of the present application, the training steps of the coarse ranking model and the fine ranking model include:

[0148] Obtain the title document pair corresponding to the reference document library; the reference document library includes the target document library, or includes the target document library and the document library of the basic field;

[0149] Based on the title document pair, pre-train the basic question answering model to obtain a pre-trained model;

[0150] Based on the set of question document pairs in the target field, the first difficult negative sample of the initial coarse ranking model in the pre-training model, and the second difficult negative sample of the initial fine ranking model in the pre-training model, the initial coarse ranking model and the initial fine ranking model are iteratively trained for multiple rounds to obtain the coarse ranking model and the fine ranking model.

[0151] Specifically, in the embodiments of the present application, when the coarse ranking model and the fine ranking model are jointly trained, a pre-training step can also be introduced.

[0152] Firstly, the title document pairs corresponding to the reference document library can be obtained. The reference document library can only include the target document library, or can include the target document library and the document library of the basic field in the case of similar target field and basic field. The reference document library can also be pre-collected, which can include multiple documents, and each document is also within a preset length range, i.e., the documents in the reference document library include documents with original lengths within the preset length range, and documents with lengths within the preset length range obtained by cutting documents with original lengths outside the preset length range.

[0153] Since each document in the reference document library corresponds to a title, the title of any document in the reference document library can be used as a question sample, and any document can be used as a document sample to form a title document pair.

[0154] Since the title of the document is not a real question, but an equivalent question, it can be called a pseudo question, so the title document pair can be used as a weakly supervised training sample to realize weakly supervised pre-training of the basic question and answer model to obtain a pre-training model. In this way, the pre-training model can fully learn the knowledge in the target field without introducing artificial labeling, which can save model training cost. It can be understood that the pre-training model can include a pre-training coarse ranking model and a pre-training fine ranking model, in addition to a pre-training understanding model and a pre-training generation model.

[0155] Further, based on the set of question document pairs in the target field, the first difficult negative sample of the initial coarse ranking model in the pre-training model, and the second difficult negative sample of the initial fine ranking model in the pre-training model, the initial coarse ranking model and the initial fine ranking model are iteratively trained for multiple rounds to obtain the coarse ranking model and the fine ranking model. Here, the initial coarse ranking model is replaced by the basic coarse ranking model and the initial fine ranking model is replaced by the basic fine ranking model when the basic coarse ranking model and the basic fine ranking model are iteratively trained for multiple rounds in each embodiment, and the details are not repeated here.

[0156] In this embodiment of the invention, a pre-trained model is obtained by utilizing the title document pairs corresponding to the reference document library. This allows the pre-trained model to fully learn knowledge within the target domain without requiring manual annotation, further addressing the problems of insufficient sample data and annotation difficulties in the target domain, thereby reducing model training difficulty, saving training costs, and improving training efficiency.

[0157] Based on the above embodiments, the document question-and-answer method provided in this embodiment of the invention includes, in the first type of question-document pair, question-correct document pairs and question-incorrect document pairs;

[0158] For any question sample in the first type of question document pair used in the current iteration round, the steps for determining the question error document pair corresponding to any question sample include:

[0159] Based on any one of the problem samples and the document samples in the same sample batch other than the correct document samples in the problem correct document pair corresponding to any one of the problem samples, determine the problem error document pair corresponding to any one of the problem samples;

[0160] And / or, based on any problem sample, the basic coarse-ranking model trained in the previous iteration, and the basic fine-ranking model, determine the problem error document pair corresponding to any problem sample.

[0161] Specifically, in this embodiment of the invention, the first type of question-document pairs may include correct question-document pairs and incorrect question-document pairs. Thus, the training of the basic coarse-ranking model can be achieved through adversarial learning, and the loss function used can be an adversarial learning loss function, which can improve training accuracy and training efficiency.

[0162] A correct document pair consists of a question sample and a correct document sample. A correct document sample is one that is marked as containing answers that can answer the question sample. An incorrect document pair consists of a question sample and an incorrect document sample. An incorrect document sample is one that is marked as not containing answers that can answer the question sample.

[0163] For any question sample q in the first type of question document pair i The corresponding correct document sample and m error document samples The loss function used when training the basic coarse-ranking model is:

[0164]

[0165] in, Any problem sample q obtained from the basic coarse-sorting model i With correct document sample similarity, Any problem sample q obtained from the basic coarse-sorting model i With error document sample The similarity.

[0166] For any question sample in the first type of question document pair, determine the corresponding question error document pair, that is, determine the corresponding error document sample. One method is to first determine the correct document sample in the question correct document pair corresponding to the question sample, and then take the document samples in the same sample batch other than the correct document sample as the corresponding error document samples.

[0167] Since the coarse-ranking and fine-ranking models are trained iteratively, the erroneous document samples corresponding to any given problem sample can be updated in each iteration. Therefore, another method can determine the erroneous document samples corresponding to any given document sample using the given problem sample, the basic coarse-ranking model trained in the previous iteration, and the basic fine-ranking model. Specifically, for the current iteration, the given problem sample can be input into the basic coarse-ranking model trained in the previous iteration, and the output of the basic coarse-ranking model trained in the previous iteration can be input into the candidate fine-ranking model trained in the previous iteration. Based on the similarity output of the basic fine-ranking model trained in the previous iteration, documents with similarity scores below a reference threshold are identified as the erroneous document samples corresponding to the given document sample. The reference threshold can be set as needed and is not specifically limited here.

[0168] In this embodiment of the invention, two methods are used to determine the problem-error document pairs, which can increase the diversity of problem-error document pairs and improve the performance of the trained coarse-ranking model.

[0169] Based on the above embodiments, the document question answering method provided in this embodiment of the invention, wherein the user question is input into a coarse-ranking model to obtain multiple candidate documents retrieved by the coarse-ranking model from the target document library, includes:

[0170] The user question is input into the question encoder of the coarse-ranked model to obtain the question encoding vector output by the question encoder.

[0171] The question encoding vector and the document encoding vectors corresponding to each document in the target document library obtained by the document encoder based on the coarse ranking model are input into the similarity calculation layer of the coarse ranking model to obtain the similarity between the question encoding vector output by the similarity calculation layer and the document encoding vectors corresponding to each document.

[0172] The similarity between the problem code vector and the document code vector corresponding to each document is input to an output layer to obtain the multiple candidate documents with high similarity in the output layer.

[0173] Specifically, in the embodiment of the present application, the coarse ranking model can be a double-tower structure, that is, it includes a question encoder q-encoder and a document encoder p-encoder, and further includes a similarity calculation layer and an output layer. The question encoder q-encoder and the document encoder p-encoder are connected with the similarity calculation layer respectively, and the similarity calculation layer is connected with the output layer.

[0174] After the user question query is input to the coarse ranking model, the question encoder q-encoder can be used to perform semantic coding on the user question query to obtain a d-dimensional dense question code vector E q (query), and then the similarity calculation layer is used to calculate the dot product of the question code vector E q (query) and the d-dimensional dense document code vector E p (passage0) corresponding to each document passage0 in the document library, and the dot product is the similarity sim(query, passage0) between the user question query and each document passage0.

[0175] wherein,

[0176] sim(query, passage0) = E q (query)·E p (passage0).

[0177] It can be understood that the question code vector E q (query) and the document code vector E p (passage0) are both the representation vector of the first special word [cls].

[0178] To sum up, the document question answering method provided in the embodiment of the application, the title document pair corresponding to the reference document library is added in the training sample of the document question answering model, the pre-training of the model can be realized, and the recall accuracy will be greatly improved compared with using only the labeled data. The document question answering model used can include a rough sorting model, a fine sorting model, an understanding model and a generation model. In the rough sorting model retrieval, the rough sorting model includes a question encoder and a document encoder. The paragraph encoder can encode each document in the reference document library in advance, and only needs to encode the user question on the question encoder during prediction, and direct retrieval can be performed. The speed is greatly improved compared with cross-encoding, and is suitable for retrieval in a large-scale reference document library. Through the rough sorting model, the most relevant document to the user question can be retrieved in the target document library, and the top-K documents with high similarity are obtained.

[0179] Since the retrieval ability of the rough sorting model is limited, the cross-encoding model can better detect the similarity of the user question and each document in the target document library than the double-encoding model. Therefore, for the multiple candidate documents given by the rough sorting model, reordering by the fine sorting model can greatly improve the retrieval accuracy. The fine sorting model reorders the multiple candidate documents, and takes the candidate document with the highest similarity as the target document. If the similarity of the candidate document is less than a preset threshold, the user question is taken as a non-target field class question, and no answer is provided.

[0180] The fine sorting model is relatively slow in each retrieval because the user question and the target document need to be spliced and encoded, and is not suitable for retrieval in a large-scale target document library. Therefore, the fine sorting model uses the cross-encoding model.

[0181] In the process of training the document question answering model, first, the encyclopedic knowledge field is taken as a basic field, the encyclopedic knowledge data is collected, the basic model is trained using the encyclopedic knowledge data, and the encyclopedic knowledge question answering model is obtained as a basic question answering model, which has certain document question answering ability; then the target field is collected with the title of the document, the title is taken as the question, and in the case of similarity between the target field and the encyclopedic field, a certain title document pair of the encyclopedic knowledge field is added to pre-train the basic question answering model, and a pre-training model is obtained.

[0182] Then, the target field question document pair is collected or labeled, and in the case of similarity between the target field and the encyclopedic field, a certain encyclopedic knowledge field question document pair is mixed. For the labeled questions in the encyclopedic knowledge field and the target field, data enhancement can be performed through synonym replacement, random insertion, random deletion, random replacement and back translation, so as to obtain more question document pairs; the pre-training model is used to screen the documents with high similarity to the question sample to construct difficult negative samples, and the pre-training model is trained using the enhanced data.

[0183] Since the basic question-answering model trained on encyclopedia knowledge data may have difficulty finding difficult negative samples for questions in the target domain, the initial coarse-ranking model and the initial fine-ranking model obtained from each iteration are used to construct difficult negative samples for iterative training, so that the resulting coarse-ranking model and fine-ranking model have better recognition capabilities.

[0184] Finally, we collect everyday questions or casual conversation data as question data outside the target domain. We use the pre-trained coarse ranking model to obtain documents with high similarity to the question data to construct negative samples. We add the second type of question document pairs to the data and train the candidate fine ranking model to obtain a fine ranking model with a certain rejection capability.

[0185] like Figure 2 As shown, based on the above embodiments, this embodiment of the invention provides a document question-and-answer device, including:

[0186] Module 21 is used to acquire user questions in the target domain;

[0187] Answer module 22 is used for the coarse ranking module, which inputs the user question into the coarse ranking model to obtain multiple candidate documents retrieved by the coarse ranking model from the target document library of the target domain;

[0188] The fine-ranking module 23 is used to input the user question and the multiple candidate documents into the fine-ranking model, obtain the similarity between each candidate document output by the fine-ranking model and the user question, and determine the target document whose similarity with the user question is higher than a preset threshold.

[0189] The determining module 24 is used to determine the target answer corresponding to the user question based on the user question, the target document, and the similarity between the target document and the user question;

[0190] The coarse-ranking model and the fine-ranking model are obtained by performing multiple rounds of iterative training on the basic coarse-ranking model and the basic fine-ranking model based on the question document pair set in the target domain, the first difficult negative sample of the basic coarse-ranking model in the basic question-answering model, and the second difficult negative sample of the basic fine-ranking model in the basic question-answering model; the basic question-answering model is used to determine the answer to questions in a basic domain that is different from the target domain.

[0191] The first difficult negative sample used in each iteration round is determined based on the basic coarse-ranking model and basic fine-ranking model trained in the previous iteration round, and the second difficult negative sample used in each iteration round is determined based on the basic coarse-ranking model trained in the current iteration round.

[0192] Based on the above embodiments, the document question-answering device provided in this embodiment of the invention further includes a training module, used for:

[0193] training the base coarse ranking model obtained in the previous iteration round based on the first type of question document pair in the set of question document pairs and the first difficult negative sample of the base coarse ranking model obtained in the previous iteration round of training;

[0194] training the base fine ranking model obtained in the previous iteration round based on the second type of question document pair in the set of question document pairs and the second difficult negative sample of the base coarse ranking model obtained in the current iteration round of training;

[0195] continuing the training in the current iteration round by taking the base coarse ranking model obtained in the current iteration round as the base coarse ranking model obtained in the previous iteration round and taking the base fine ranking model obtained in the current iteration round as the base fine ranking model obtained in the previous iteration round until the multiple iteration rounds of training are completed to obtain the coarse ranking model and the candidate fine ranking model;

[0196] determining the fine ranking model based on the candidate fine ranking model.

[0197] On the basis of the above-mentioned embodiments, the document question answering device provided in the embodiments of the present application comprises a training module, which is specifically configured to:

[0198] obtain question data outside the target field and input the question data into the coarse ranking model to obtain a plurality of first documents retrieved from the reference document library by the coarse ranking model;

[0199] input the question data and the plurality of first documents into the candidate fine ranking model to obtain the similarity between each first document output by the candidate fine ranking model and the question data, and determine a second document with a similarity greater than the preset threshold;

[0200] construct a negative sample based on the question data and the second document with a similarity within a preset range, and train the candidate fine ranking model based on the negative sample and the second type of question document pair to obtain the fine ranking model.

[0201] On the basis of the above-mentioned embodiments, the document question answering device provided in the embodiments of the present application further comprises a sample determination module configured to:

[0202] for any question sample in the first type of question document pair used in the current iteration round, input the any question sample into the base coarse ranking model obtained in the current iteration round of training to obtain a plurality of third documents retrieved from the reference document library by the base coarse ranking model obtained in the current iteration round of training;

[0203] Determine a second difficult negative sample for the current iteration round based on the any question sample and the plurality of third documents.

[0204] Based on the above-mentioned embodiments, the document question answering device provided in the embodiments of the present application, the sample determination module is specifically used for:

[0205] Eliminate a preset number of third documents with high ranking from the plurality of third documents, and determine a second difficult negative sample for the current iteration round based on the any question sample and the third documents remaining after eliminating the preset number of third documents.

[0206] Based on the above-mentioned embodiments, the document question answering device provided in the embodiments of the present application, the training module is specifically used for:

[0207] Obtain a title document pair corresponding to a reference document library; the reference document library includes the target document library, or includes the target document library and a document library of the basic field;

[0208] Pre-train the basic question answering model based on the title document pair to obtain a pre-trained model;

[0209] Based on the question document pair set in the target field, the first difficult negative sample of the initial coarse ranking model in the pre-trained model, and the second difficult negative sample of the initial fine ranking model in the pre-trained model, perform multi-round iteration training on the initial coarse ranking model and the initial fine ranking model to obtain the coarse ranking model and the fine ranking model.

[0210] Based on the above-mentioned embodiments, the document question answering device provided in the embodiments of the present application, the first type of question document pair includes a question correct document pair and a question error document pair; the document question answering device further includes a question error document pair determination module, which is used for:

[0211] Determine a question error document pair corresponding to the any question sample based on the any question sample and the document sample in the same sample batch except the correct document sample in the question correct document pair corresponding to the any question sample;

[0212] And / or, determine a question error document pair corresponding to the any question sample based on the any question sample, the basic coarse ranking model and the basic fine ranking model obtained by the previous iteration round training.

[0213] Based on the above-mentioned embodiments, the document question answering device provided in the embodiments of the present application, the coarse ranking module is specifically used for:

[0214] Input the user question to the question encoder of the coarse ranking model to obtain a question encoding vector output by the question encoder;

[0215] The question encoding vector and the document encoding vectors corresponding to each document in the target document library obtained by the document encoder based on the coarse ranking model are input into the similarity calculation layer of the coarse ranking model to obtain the similarity between the question encoding vector output by the similarity calculation layer and the document encoding vectors corresponding to each document.

[0216] The similarity between the question encoding vector and the document encoding vector corresponding to each document is input to the output layer to obtain the multiple candidate documents with high similarity among the documents output by the output layer.

[0217] Based on the above embodiments, the document question-answering device provided in this embodiment of the invention, wherein the determining module is specifically used for:

[0218] The user question and the target document are input into the understanding model to obtain the answer paragraph within the target document output by the understanding model;

[0219] The user question, the answer paragraph, and the similarity between the target document and the user question are input into the generation model to obtain the target answer output by the generation model;

[0220] The understanding model is trained based on a first question-answer paragraph pair within the target domain, and the generation model is trained based on a second question-answer paragraph pair within the target domain and the similarity between the document sample containing the answer paragraph sample in the second question-answer paragraph pair and the question sample in the second question-answer paragraph pair.

[0221] Specifically, the functions of each module in the document question-and-answer device provided in this embodiment of the invention correspond one-to-one with the operation flow of each step in the above method-like embodiments, and the achieved effects are also the same. For details, please refer to the above embodiments, and this will not be repeated in this embodiment of the invention.

[0222] like Figure 3 As shown, based on the above embodiments, this embodiment of the invention provides a document question-and-answer system, including: a voice module 31 and a document question-and-answer device 32 provided in the above embodiments, wherein the voice module 31 is connected to the document question-and-answer device 32; the voice module 31 is used to collect user questions and transmit user questions to the document question-and-answer device 32.

[0223] Specifically, in this embodiment of the invention, the user can output a user question via voice. The voice signal of the user question can be collected by the voice module 31 and then sent to the document question-and-answer device 32. The document question-and-answer device 32 then converts the voice signal into text and executes the document question-and-answer method to provide the user with the target answer corresponding to the user question.

[0224] Here, the voice module 31 can also directly convert the voice signal into text and send the user question in text form to the document question answering device 32.

[0225] The document question answering system provided in the embodiments of the present application combines a voice module with a document question answering device, so that the document question answering system can realize the function of a voice question answering system, be used for voice question answering, and improve user experience.

[0226] Figure 4 An example of a schematic diagram of the physical structure of an electronic device is shown in Figure 4 The electronic device can include a processor 410, a communications interface 420, a memory 430, and a communications bus 440, wherein the processor 410, the communications interface 420, and the memory 430 can complete mutual communication through the communications bus 440. The processor 410 can invoke a logical instruction in the memory 430 to execute the document question answering method provided in the above embodiments, which includes: obtaining a user question in a target domain; inputting the user question into a coarse ranking model to obtain a plurality of candidate documents retrieved from a target document library in the target domain by the coarse ranking model; inputting the user question and the plurality of candidate documents into a fine ranking model to obtain a similarity between each candidate document output by the fine ranking model and the user question, and determine a target document with a similarity higher than a preset threshold to the user question; determining a target answer corresponding to the user question based on the user question, the target document, and the similarity between the target document and the user question; wherein the coarse ranking model and the fine ranking model are obtained by performing multi-round iterative training on a basic coarse ranking model in a basic question answering model and a basic fine ranking model in the basic question answering model based on a question document pair set in the target domain, a first difficult negative sample of the basic coarse ranking model, and a second difficult negative sample of the basic fine ranking model; the basic question answering model is used to determine an answer to a question in a basic domain different from the target domain; the first difficult negative sample used in each iteration round is determined based on the basic coarse ranking model and the basic fine ranking model obtained by a previous iteration round, and the second difficult negative sample used in each iteration round is determined based on the basic coarse ranking model obtained by a current iteration round.

[0227] Further, the logic instructions in the memory 430 described above can be implemented in the form of software functional units and sold or used as standalone products, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or partially contribute to the prior art, or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0228] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to execute the document question and answer method provided in the above embodiments. The method comprises: obtaining a user question in a target field; inputting the user question into a coarse ranking model to obtain a plurality of candidate documents retrieved from a target document library in the target field by the coarse ranking model; inputting the user question and the plurality of candidate documents into a fine ranking model to obtain a similarity between each candidate document output by the fine ranking model and the user question, and determine a target document with a similarity higher than a preset threshold to the user question; determining a target answer corresponding to the user question based on the user question, the target document, and the similarity between the target document and the user question; wherein the coarse ranking model and the fine ranking model are obtained by performing multi-round iterative training on a basic coarse ranking model in a basic question and answer model and a basic fine ranking model in the basic question and answer model based on a question document pair set in the target field, first difficult negative samples of the basic coarse ranking model, and second difficult negative samples of the basic fine ranking model; the basic question and answer model is used to determine an answer to a question in a basic field different from the target field; the first difficult negative samples used in each iteration round are determined based on the basic coarse ranking model and the basic fine ranking model obtained by a previous iteration round, and the second difficult negative samples used in each iteration round are determined based on the basic coarse ranking model obtained by a current iteration round.

[0229] In yet another aspect, the present application also provides a non-transitory computer-readable storage medium having stored thereon a computer program, which, when executed by a processor, implements a document question-answering method provided in any of the above embodiments, the method comprising: obtaining a user question in a target domain; inputting the user question into a coarse retrieval model to obtain a plurality of candidate documents retrieved from a target document library in the target domain by the coarse retrieval model; inputting the user question and the plurality of candidate documents into a fine retrieval model to obtain a similarity between each candidate document and the user question output by the fine retrieval model, and determining a target document with a similarity to the user question higher than a preset threshold; determining a target answer corresponding to the user question based on the user question, the target document, and the similarity between the target document and the user question; wherein the coarse retrieval model and the fine retrieval model are obtained by performing multi-round iterative training on a basic coarse retrieval model and a basic fine retrieval model in a basic question-answer model based on a question document pair set in the target domain, first difficult negative samples of the basic coarse retrieval model, and second difficult negative samples of the basic fine retrieval model; the basic question-answer model is used to determine answers to questions in a basic domain different from the target domain; the first difficult negative samples used in each iteration round are determined based on the basic coarse retrieval model and the basic fine retrieval model obtained in a previous iteration round, and the second difficult negative samples used in each iteration round are determined based on the basic coarse retrieval model obtained in a current iteration round.

[0230] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0231] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software plus necessary general hardware platforms, and of course can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in the embodiments or some parts of the embodiments.

[0232] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; and although the present application has been described in detail with reference to the foregoing embodiments, it should be appreciated by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features thereof can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A document question-and-answer method, characterized in that, include: Identify user questions within the target domain; The user question is input into the coarse ranking model, which retrieves multiple candidate documents from the target document library of the target domain. The user question and the multiple candidate documents are input into the fine ranking model to obtain the similarity between each candidate document output by the fine ranking model and the user question, and the target document with a similarity to the user question higher than a preset threshold is determined. Based on the user question, the target document, and the similarity between the target document and the user question, determine the target answer corresponding to the user question; The coarse-ranking model and the fine-ranking model are obtained by performing multiple rounds of iterative training on the basic coarse-ranking model and the basic fine-ranking model based on the question document pair set in the target domain, the first difficult negative sample of the basic coarse-ranking model in the basic question-answering model, and the second difficult negative sample of the basic fine-ranking model in the basic question-answering model; the basic question-answering model is used to determine the answer to questions in a basic domain that is different from the target domain. The first difficult negative sample used in each iteration round is determined based on the basic coarse-ranking model and basic fine-ranking model trained in the previous iteration round, and the second difficult negative sample used in each iteration round is determined based on the basic coarse-ranking model trained in the current iteration round.

2. The document question-and-answer method according to claim 1, characterized in that, The training steps for the coarse-ranking model and the fine-ranking model include: Based on the first type of question document pairs in the question document pair set and the first difficult negative sample of the basic coarse-ranking model obtained from the previous iteration, the basic coarse-ranking model obtained from the previous iteration is trained for the current iteration. Based on the second type of question document pairs in the question document pair set, and the second difficult negative sample of the basic coarse-ranking model obtained from the current iteration training, the basic fine-ranking model obtained from the previous iteration training is trained for the current iteration. The basic coarse-ranking model obtained from the current iteration training is used as the basic coarse-ranking model obtained from the previous iteration training, and the basic fine-ranking model obtained from the current iteration training is used as the basic fine-ranking model obtained from the previous iteration training. The training of the current iteration continues until the multi-round iterative training is completed, and a coarse-ranking model and a candidate fine-ranking model are obtained. Based on the candidate ranking models, the ranking model is determined.

3. The document question-and-answer method according to claim 2, characterized in that, The step of determining the refined ranking model based on the candidate refined ranking model includes: Obtain problem data outside the target domain and input the problem data into the coarse ranking model to obtain multiple first documents retrieved by the coarse ranking model from the reference document library; The problem data and the plurality of first documents are input into the candidate ranking model to obtain the similarity between each first document output by the candidate ranking model and the problem data, and the second document with a similarity greater than the preset threshold is determined; Based on the problem data and second documents with similarity within a preset range, negative samples are constructed, and the candidate fine ranking model is trained based on the negative samples and the second type of problem document pairs to obtain the fine ranking model.

4. The document question-and-answer method according to claim 2, characterized in that, The second hard negative sample used in each iteration round is determined based on the following steps: For any question sample in the first type of question document pair used in the current iteration round, input the question sample into the basic coarse ranking model trained in the current iteration round to obtain multiple third documents retrieved from the reference document library by the basic coarse ranking model trained in the current iteration round; Based on any of the problem samples and the multiple third documents, determine the second difficult negative sample to be used in the current iteration round.

5. The document question-and-answer method according to claim 4, characterized in that, The step of determining the second difficult negative sample to use in the current iteration round based on any of the problem samples and the multiple third documents includes: Remove a predetermined number of third documents from the plurality of third documents, and determine the second difficult negative sample to be used in the current iteration round based on any problem sample and the remaining third documents after removing the predetermined number of third documents.

6. The document question-and-answer method according to claim 2, characterized in that, The first type of issue document pairs includes correct issue document pairs and incorrect issue document pairs; For any question sample in the first type of question document pair used in the current iteration round, the steps for determining the question error document pair corresponding to any question sample include: Based on any one of the problem samples and the document samples in the same sample batch other than the correct document samples in the problem correct document pair corresponding to any one of the problem samples, determine the problem error document pair corresponding to any one of the problem samples; And / or, based on any problem sample, the basic coarse-ranking model trained in the previous iteration, and the basic fine-ranking model, determine the problem error document pair corresponding to any problem sample.

7. The document question-and-answer method according to any one of claims 1-6, characterized in that, The training steps for the coarse-ranking model and the fine-ranking model include: Obtain the title document pairs corresponding to the reference document library; the reference document library includes the target document library, or includes the target document library and the document library of the basic domain; Based on the title document pair, the basic question-answering model is pre-trained to obtain a pre-trained model; Based on the set of problem documents in the target domain, the first difficult negative sample of the initial coarse-ranking model in the pre-trained model, and the second difficult negative sample of the initial fine-ranking model in the pre-trained model, the initial coarse-ranking model and the initial fine-ranking model are trained in multiple rounds to obtain the coarse-ranking model and the fine-ranking model.

8. The document question-and-answer method according to any one of claims 1-6, characterized in that, The user question is input into a coarse-ranking model, which retrieves multiple candidate documents from the target document library, including: The user question is input into the question encoder of the coarse-ranked model to obtain the question encoding vector output by the question encoder. The question encoding vector and the document encoding vectors corresponding to each document in the target document library obtained by the document encoder based on the coarse ranking model are input into the similarity calculation layer of the coarse ranking model to obtain the similarity between the question encoding vector output by the similarity calculation layer and the document encoding vectors corresponding to each document. The similarity between the question encoding vector and the document encoding vector corresponding to each document is input to the output layer to obtain the multiple candidate documents with high similarity among the documents output by the output layer.

9. The document question-and-answer method according to any one of claims 1-6, characterized in that, The step of determining the target answer corresponding to the user question based on the user question, the target document, and the similarity between the target document and the user question includes: The user question and the target document are input into the understanding model to obtain the answer paragraph within the target document output by the understanding model; The user question, the answer paragraph, and the similarity between the target document and the user question are input into the generation model to obtain the target answer output by the generation model; The understanding model is trained based on a first question-answer paragraph pair within the target domain, and the generation model is trained based on a second question-answer paragraph pair within the target domain and the similarity between the document sample containing the answer paragraph sample in the second question-answer paragraph pair and the question sample in the second question-answer paragraph pair.

10. A document question-and-answer device, characterized in that, include: The acquisition module is used to acquire user questions in the target domain; The coarse ranking module is used to input the user question into the coarse ranking model and obtain multiple candidate documents retrieved by the coarse ranking model from the target document library of the target domain; The fine-ranking module is used to input the user question and the multiple candidate documents into the fine-ranking model, obtain the similarity between each candidate document output by the fine-ranking model and the user question, and determine the target document whose similarity with the user question is higher than a preset threshold. The determination module is used to determine the target answer corresponding to the user question based on the user question, the target document, and the similarity between the target document and the user question; The coarse-ranking model and the fine-ranking model are obtained by performing multiple rounds of iterative training on the basic coarse-ranking model and the basic fine-ranking model based on the question document pair set in the target domain, the first difficult negative sample of the basic coarse-ranking model in the basic question-answering model, and the second difficult negative sample of the basic fine-ranking model in the basic question-answering model; the basic question-answering model is used to determine the answer to questions in a basic domain that is different from the target domain. The first difficult negative sample used in each iteration round is determined based on the basic coarse-ranking model and basic fine-ranking model trained in the previous iteration round, and the second difficult negative sample used in each iteration round is determined based on the basic coarse-ranking model trained in the current iteration round.

11. A document question-and-answer system, characterized in that, include: The voice module and the document question-answering device as described in claim 10; The voice module is connected to the document question-and-answer device; The voice module is used to collect user questions and transmit the user questions to the document question-and-answer device.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the document question-and-answer method as described in any one of claims 1-9.

13. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the document question-and-answer method as described in any one of claims 1-9.

Images