Negative sample determination method, related device and readable storage medium

Abstract

The present application discloses a negative sample determination method, related equipment and readable storage medium. Based on the semantic similarity between each text in a text library and a sample question, it is possible to determine from the text library a text that has a high semantic similarity with the sample question but does not contain the answer to the sample question as a negative sample of the sample question. The negative sample determined based on this method has a high semantic similarity with the sample question, but does not contain the answer to the sample question, and is highly confusing to the sample question. Furthermore, the negative samples determined based on the above method are used to train the retrieval model and the question-answering model in the intelligent question-answering system, which can improve the training effect. Moreover, when the negative samples determined based on the above method are used to test the robustness of the retrieval model and the question-answering model in the intelligent question-answering system, the effect is better than that of the adversarial samples determined using the existing technology.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a negative sample determination method, related equipment, and readable storage medium. Background Art

[0002] With the development of artificial intelligence (AI), intelligent question-answering systems (such as open-domain and vertical-domain question-answering systems) have gained widespread application. Currently, commonly used intelligent question-answering systems include retrieval models and question-answering models. When a user searches for a question, the retrieval model automatically retrieves relevant text from a text library. The question-answering model then determines the answer based on the text retrieved by the retrieval model and returns the answer to the user.

[0003] When training retrieval and question-answering models in intelligent question-answering systems using sample questions, to improve training effectiveness, text containing the answer to the sample question is typically used as positive samples, while text not containing the answer is used as negative samples. Currently, negative samples for sample questions are typically obtained from a text library using random sampling or the TF-IDF algorithm. However, these negative samples are often poorly convincing and fail to improve training effectiveness.

[0004] Therefore, how to provide a negative sample determination method to obtain highly deceptive negative samples has become a technical problem that needs to be solved urgently by those skilled in the art. Summary of the Invention

[0005] In view of the above problems, this application proposes a negative sample determination method, related equipment and readable storage medium. The specific solution is as follows:

[0006] A negative sample determination method, the method comprising:

[0007] Obtaining sample questions for which negative samples are to be determined and a text library corresponding to the sample questions;

[0008] Calculate the semantic similarity between each text in the text library and the sample question;

[0009] Determining a sample text set from the text library, wherein the sample text set includes a preset number of texts with top-ranked semantic similarities;

[0010] The remaining sample texts after removing the sample texts containing the answers to the sample questions from the sample text set are determined as negative samples of the sample questions.

[0011] Optionally, calculating the semantic similarity between each text in the text library and the sample question includes:

[0012] Inputting the sample question into a question encoder of a semantic similarity calculation model, wherein the question encoder encodes the sample question to obtain a feature vector of the sample question;

[0013] Inputting each text in the text library into the text encoder of the semantic similarity calculation model, wherein the text encoder encodes each text in the text library to obtain a feature vector of each text in the text library;

[0014] The calculation module of the semantic similarity calculation model calculates the similarity between the feature vector of each text in the text library and the feature vector of the sample question as the semantic similarity between each text in the text library and the sample question.

[0015] Optionally, the semantic similarity calculation model is trained as follows:

[0016] Determine a pre-training data set, wherein the pre-training data set includes a plurality of pre-training data, each pre-training data includes a sample question, a positive sample of the sample question, and a negative sample of the sample question;

[0017] Determine a first label of a positive sample of the sample problem and a second label of a negative sample of the sample problem;

[0018] Pre-training the semantic similarity calculation model with the goal that the semantic similarity between each sample question and a positive sample of the sample question approaches the first label, and with the goal that the semantic similarity between the sample question and a negative sample of the sample question approaches the second label, thereby obtaining a pre-trained semantic similarity calculation model;

[0019] The question encoder of the pre-trained semantic similarity calculation model is optimized and trained to obtain the semantic similarity calculation model.

[0020] Optionally, the optimizing and training the question encoder of the pre-trained semantic similarity calculation model to obtain the semantic similarity calculation model includes:

[0021] Obtaining optimization training sample problems;

[0022] Calculating the semantic similarity between each text in the text library and each optimized training sample question based on the pre-trained semantic similarity calculation model;

[0023] Determining negative samples of each optimized training sample problem from the text library based on semantic similarity between each text in the text library and each optimized training sample problem;

[0024] Determining a third label of the negative sample of the optimization training sample problem, wherein the third label is used to represent a probability that the negative sample of the optimization training sample problem is determined to be a sample containing an answer to the optimization training sample problem;

[0025] The pre-trained semantic similarity calculation model is optimized and trained with the similarity between each optimized training sample problem and the negative sample of the optimized training sample problem approaching the third label as a training goal.

[0026] Optionally, the calculating of the semantic similarity between each text in the text library and each optimized training sample question based on the pre-trained semantic similarity calculation model includes:

[0027] Input each text in the text library into the text encoder in the pre-trained semantic similarity calculation model for encoding to generate a text feature vector index library, wherein the text feature vector index library includes multiple text feature vector indexes, each text feature vector index corresponds to a text, and each text feature vector index includes the number of the text in the text library and the feature vector of the text;

[0028] Based on the question encoder in the pre-trained semantic similarity calculation model and the text feature vector index library, the semantic similarity between each text in the text library and each optimized training sample question is calculated.

[0029] Optionally, before determining the third label of the negative sample of the optimization training sample problem, the method further includes:

[0030] Obtaining a positive sample of the optimization training sample problem;

[0031] The retrieval model of the intelligent question answering model is trained with the training objectives that the semantic similarity between the optimized training sample question and the positive sample of the optimized training sample question is close to the first label, and the similarity between the optimized training sample question and the negative sample of the optimized training sample question is close to the second label.

[0032] Optionally, determining the third label of the negative sample of the optimization training sample problem includes:

[0033] The similarity between the optimized training sample question and the negative sample of the optimized training sample question output by the retrieval model of the intelligent question answering model at the end of training is used as the third label of the negative sample of the optimized training sample question.

[0034] Optionally, before determining the third label of the negative sample of the optimization training sample problem, the method further includes:

[0035] Obtaining a positive sample of the optimization training sample problem;

[0036] Determining position labels of answers to the problem of optimizing training samples;

[0037] The retrieval model of the intelligent question-answering model is trained with the probability that the answer to the optimized training sample question is included in the positive sample of the optimized training sample question approaching the first label, the probability that the answer to the optimized training sample question is included in the negative sample of the optimized training sample question approaching the second label, and the position of the answer to the optimized training sample question approaching the position label of the answer to the optimized training sample question as training objectives.

[0038] Optionally, determining the third label of the negative sample of the optimization training sample problem includes:

[0039] The probability that the negative sample of the optimized training sample question output by the question answering model of the intelligent question answering model at the end of training contains the answer to the optimized training sample question is used as the third label of the negative sample of the optimized training sample question.

[0040] Optionally, after determining a negative sample of the sample problem, the method further includes:

[0041] The negative samples of the sample questions are used to detect the robustness of the retrieval model and / or the question-answering model of the intelligent question-answering model.

[0042] A method for training an intelligent question-answering model, comprising:

[0043] Obtaining training samples, where the training samples include a sample problem, a positive sample of the sample problem, and a negative sample of the sample problem, wherein the negative sample of the sample problem is obtained based on the negative sample determination method described above;

[0044] Determine labels of positive samples of the sample problem and labels of negative samples of the sample problem;

[0045] The retrieval model and / or question-answering model of the intelligent question-answering model is trained with the similarity between the sample question and the positive sample of the sample question approaching the label of the positive sample of the sample question, and the similarity between the sample question and the negative sample of the sample question approaching the label of the negative sample of the sample question to obtain the intelligent question-answering model.

[0046] A negative sample determination device, comprising:

[0047] An acquisition unit, configured to acquire a sample question of a to-be-determined negative sample and a text library corresponding to the sample question;

[0048] A calculation unit, used to calculate the semantic similarity between each text in the text library and the sample question;

[0049] A sample text set determining unit, configured to determine a sample text set from the text library, wherein the sample text set includes a preset number of texts with top-ranked semantic similarities;

[0050] The negative sample determination unit is configured to determine the remaining sample texts in the sample text set after removing the sample texts containing the answers to the sample questions as negative samples of the sample questions.

[0051] Optionally, the computing unit includes:

[0052] a sample question encoding unit, configured to input the sample question into a question encoder of a semantic similarity calculation model, wherein the question encoder encodes the sample question to obtain a feature vector of the sample question;

[0053] a text encoding unit, configured to input each text in the text library into a text encoder of the semantic similarity calculation model, wherein the text encoder encodes each text in the text library to obtain a feature vector of each text in the text library;

[0054] The semantic similarity calculation unit is used to calculate the similarity between the feature vector of each text in the text library and the feature vector of the sample question using the calculation module of the semantic similarity calculation model, as the semantic similarity between each text in the text library and the sample question.

[0055] Optionally, the device further comprises: a semantic similarity calculation model training unit;

[0056] The semantic similarity calculation model training unit includes:

[0057] A pre-training data set determining unit, configured to determine a pre-training data set, wherein the pre-training data set includes a plurality of pre-training data, each pre-training data including a sample question, a positive sample of the sample question, and a negative sample of the sample question;

[0058] A first determining unit, configured to determine a first label of a positive sample of the sample question and a second label of a negative sample of the sample question;

[0059] a pre-training unit, configured to pre-train the semantic similarity calculation model with the goal that the semantic similarity between each sample question and a positive sample of the sample question approaches the first label, and the semantic similarity between the sample question and a negative sample of the sample question approaches the second label, thereby obtaining a pre-trained semantic similarity calculation model;

[0060] The optimization training unit is used to optimize the question encoder of the pre-trained semantic similarity calculation model to obtain the semantic similarity calculation model.

[0061] Optionally, the optimization training unit includes:

[0062] An optimization training sample problem acquisition unit, used for acquiring optimization training sample problems;

[0063] A calculation subunit, configured to calculate the semantic similarity between each text in the text library and each optimized training sample question based on the pre-trained semantic similarity calculation model;

[0064] A negative sample determination subunit, configured to determine a negative sample for each optimized training sample problem from the text library based on the semantic similarity between each text in the text library and each optimized training sample problem;

[0065] A second determining unit is configured to determine a third label of the negative sample of the optimization training sample problem, wherein the third label is used to represent a probability that the negative sample of the optimization training sample problem is determined to be a sample containing an answer to the optimization training sample problem;

[0066] The optimization training subunit is used to optimize the pre-trained semantic similarity calculation model by taking the similarity between each optimization training sample problem and the negative sample of the optimization training sample problem approaching the third label as a training goal.

[0067] Optionally, the computing subunit includes:

[0068] a text feature vector index library generating unit, configured to input each text in the text library into a text encoder in the pre-trained semantic similarity calculation model for encoding, thereby generating a text feature vector index library, wherein the text feature vector index library includes a plurality of text feature vector indexes, each text feature vector index corresponding to a text, and each text feature vector index including a number of the text in the text library and a feature vector of the text;

[0069] The semantic similarity calculation subunit is used to calculate the semantic similarity between each text in the text library and each optimized training sample question based on the question encoder in the pre-trained semantic similarity calculation model and the text feature vector index library.

[0070] Optionally, the device further comprises:

[0071] A first positive sample acquisition unit is configured to acquire a positive sample of the optimization training sample problem before determining a third label of the negative sample of the optimization training sample problem;

[0072] The retrieval model training unit is used to train the retrieval model of the intelligent question answering model with the semantic similarity between the optimized training sample problem and the positive sample of the optimized training sample problem approaching the first label, and with the similarity between the optimized training sample problem and the negative sample of the optimized training sample problem approaching the second label as training objectives.

[0073] Optionally, the second determining unit is specifically configured to:

[0074] The similarity between the optimized training sample question and the negative sample of the optimized training sample question output by the retrieval model of the intelligent question answering model at the end of training is used as the third label of the negative sample of the optimized training sample question.

[0075] Optionally, the device further comprises:

[0076] A second positive sample acquisition unit is configured to acquire a positive sample of the optimization training sample problem before determining a third label of the negative sample of the optimization training sample problem;

[0077] an answer position label determination unit, configured to determine a position label of an answer to the problem of optimizing training samples;

[0078] The retrieval model training unit is used to train the retrieval model of the intelligent question-answering model with the probability that the answer to the optimized training sample question is contained in the positive sample of the optimized training sample question approaching the first label, the probability that the answer to the optimized training sample question is contained in the negative sample of the optimized training sample question approaching the second label, and the position of the answer to the optimized training sample question approaching the position label of the answer to the optimized training sample question as training objectives.

[0079] Optionally, the second determining unit is specifically configured to:

[0080] The probability that the negative sample of the optimized training sample question output by the question answering model of the intelligent question answering model at the end of training contains the answer to the optimized training sample question is used as the third label of the negative sample of the optimized training sample question.

[0081] Optionally, the device further comprises:

[0082] A detection unit is used to detect the robustness of the retrieval model and / or question-answering model of the intelligent question-answering model using the negative samples of the sample questions after determining the negative samples of the sample questions.

[0083] A negative sample determination device, comprising a memory and a processor;

[0084] The memory is used to store programs;

[0085] The processor is configured to execute the program to implement the various steps of the negative sample determination method described above.

[0086] A readable storage medium stores a computer program thereon, and when the computer program is executed by a processor, the computer program implements the various steps of the negative sample determination method described above.

[0087] An intelligent question-answering model training device, comprising:

[0088] A training sample acquisition unit, configured to acquire training samples, wherein the training samples include a sample problem, a positive sample of the sample problem, and a negative sample of the sample problem, wherein the negative sample of the sample problem is obtained based on the negative sample determination method described above;

[0089] a label determination unit, configured to determine labels of positive samples of the sample question and labels of negative samples of the sample question;

[0090] A training unit is used to train the retrieval model and / or question-answering model of the intelligent question-answering model with the similarity between the sample question and the positive sample of the sample question approaching the label of the positive sample of the sample question, and the similarity between the sample question and the negative sample of the sample question approaching the label of the negative sample of the sample question, so as to obtain the intelligent question-answering model.

[0091] By leveraging the above technical solution, this application discloses a method for determining negative samples, related devices, and readable storage media. Based on the semantic similarity between each text in a text library and a sample question, texts that have a high semantic similarity to the sample question but do not contain the answer to the sample question can be identified from the text library as negative samples for the sample question. Negative samples determined by this method have a high semantic similarity to the sample question but do not contain the answer to the sample question, making them highly misleading for the sample question.

[0092] Furthermore, using negative samples determined using the above method to train the retrieval model and question-answering model in an intelligent question-answering system can improve training results. Furthermore, using negative samples determined using the above method to test the robustness of the retrieval model and question-answering model in an intelligent question-answering system achieves better results than adversarial samples determined using existing techniques. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present application. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0094] Figure 1 A schematic diagram of the process of determining negative samples disclosed in an embodiment of the present application;

[0095] Figure 2 A schematic diagram of the structure of the retrieval model in the intelligent question-answering system disclosed in the embodiments of this application;

[0096] Figure 3 A schematic diagram of the structure of a semantic similarity calculation model disclosed in an embodiment of the present application;

[0097] Figure 4 A schematic diagram of a process for calculating the semantic similarity between each text in a text library and each optimized training sample question based on the question encoder in the pre-trained semantic similarity calculation model and the text feature vector index library disclosed in an embodiment of the present application;

[0098] Figure 5 This is a schematic diagram of the structure of a negative sample determination device disclosed in an embodiment of the present application;

[0099] Figure 6 This is a hardware structure block diagram of a negative sample determination device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0100] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0101] In order to obtain highly confusing negative samples and use them to train the retrieval model and question-answering model in the intelligent question-answering system to improve the training effect, the inventors of this case conducted research and found that adversarial samples refer to input samples formed by deliberately adding subtle interference to the data set, which causes the machine learning model to output incorrect results with high confidence. At present, in order to test the robustness of the retrieval model and question-answering model in the intelligent question-answering system, adversarial samples of sample questions can be generated. Specifically, there are two commonly used methods for generating adversarial samples of sample questions for testing intelligent question-answering systems. One is a method using heuristic rules, for example, adding a sentence with a large vocabulary overlap with the sample question to the end of a text to generate an adversarial sample of the sample question. The other is a rewriting method, for example, generating adversarial samples of sample questions by adding, deleting, exchanging, or replacing characters or words in the text.

[0102] Based on the above, the inventors of this case initially came up with the idea of ​​using adversarial samples of sample questions generated by the above method as negative samples of sample questions to train the retrieval model and question-answering model in the intelligent question-answering system.

[0103] However, the adversarial samples of sample questions generated by the above method usually contain various errors, such as semantic incoherence errors, word order errors, spelling errors, etc. Using them as negative samples of sample questions to train the retrieval model and question-answering model in the intelligent question-answering system will be easily recognized by the retrieval model and question-answering model, and will be less confusing and unable to improve the training effect. It may even cause the retrieval model and question-answering model to learn the wrong data distribution, which will lead to a decline in training effect.

[0104] In light of the problems with the above approach, the inventors of this case conducted in-depth research and discovered that, for sample questions, high-quality negative samples are typically those that are semantically fluent and highly relevant to the question, but do not contain the answer. Based on this, the inventors of this case proposed a method for determining negative samples. This method, based on the semantic similarity between each text in a text library and the sample question, can identify text from the text library that has a high semantic similarity to the sample question but does not contain the answer to the sample question, serving as negative samples for the sample question. Negative samples determined by this method have a high semantic similarity to the sample question but do not contain the answer to the sample question, making them highly deceptive. Furthermore, these negative samples do not require modification of the text in the text library, reducing the probability of various errors. Furthermore, using these negative samples to train the retrieval model and question-answering model in an intelligent question-answering system can improve training effectiveness. Furthermore, when used to test the robustness of the retrieval model and question-answering model in an intelligent question-answering system, these negative samples achieve superior results compared to adversarial samples determined using existing techniques.

[0105] Next, the negative sample determination method provided by this application is introduced through the following embodiments.

[0106] Reference Figure 1 , Figure 1 This is a flow chart of a negative sample determination method disclosed in an embodiment of the present application. The method may include:

[0107] Step S101: Obtain sample questions to be determined as negative samples and a text library corresponding to the sample questions.

[0108] In this application, the sample questions for determining negative samples can be questions that users may ask in various intelligent question-answering scenarios, or they can be sample questions in a training data set of a retrieval model and / or question-answering model for training an intelligent question-answering system (such as an open domain question-answering system, a vertical field question-answering system, etc.). This application does not impose any restrictions on this. In this application, the text library can be a text library in various intelligent question-answering scenarios. For example, in an open domain question-answering scenario, the text library can be a database composed of massive text data on the Internet. This application does not impose any restrictions on this. It should be noted that in different intelligent question-answering scenarios, the sample questions are different, and the corresponding text libraries are also different.

[0109] Step S102: Calculating the semantic similarity between each text in the text library and the sample question.

[0110] In this application, the semantic similarity between each text in the text library and the sample question is used to characterize the semantic similarity between each text in the text library and the sample question. In this application, the semantic similarity between each text in the text library and the sample question can be calculated based on a neural network model, which will be explained in detail in the following embodiments.

[0111] Step S103: determining a sample text set from the text library, wherein the sample text set includes a preset number of texts corresponding to the top semantic similarities.

[0112] In the present application, after calculating the semantic similarity between each text in the text library and the sample question, the semantic similarities can be sorted in descending order to obtain a preset number of semantic similarities that are ranked at the top. The sample text set can be obtained by obtaining the texts corresponding to the preset number of semantic similarities that are ranked at the top from the text library.

[0113] Step S104: the remaining sample texts after removing the sample texts containing the answers to the sample questions from the sample text set are determined as negative samples of the sample questions.

[0114] It should be noted that the text containing the answer to the sample question generally has a higher semantic similarity with the sample question. Therefore, the text containing the answer to the sample question is usually included in the top preset number of semantic similarities, and the text containing the answer to the sample question can be used as a positive sample of the sample question. Therefore, in this application, the remaining sample text in the sample text set after removing the sample text containing the answer to the sample question is determined as a negative sample of the sample question.

[0115] As one possible implementation method, in this application, the sample text containing the answer to the sample question can be determined by performing string matching between the sample question and each sample text in the sample text set. Specifically, the sample text in the sample text set that successfully matches the string of the sample question is the sample text containing the answer to the sample question. Of course, this can also be implemented using a neural network, and this application does not impose any limitations on this.

[0116] This embodiment discloses a method for determining negative samples. Based on the semantic similarity between each text in a text library and a sample question, this method can identify text from the text library that has a high semantic similarity to the sample question but does not contain the answer to the sample question, serving as negative samples for the sample question. Negative samples determined using this method have a high semantic similarity to the sample question but do not contain the answer to the sample question, making them highly misleading. Furthermore, this method eliminates the need to modify the text in the text library, reducing the probability of various errors.

[0117] Therefore, using negative samples determined using the above method to train the retrieval model and question-answering model in an intelligent question-answering system can improve training results. Furthermore, using negative samples determined using the above method to test the robustness of the retrieval model and question-answering model in an intelligent question-answering system is more effective than using adversarial samples determined using existing techniques.

[0118] Reference Figure 2 , Figure 2 This is a schematic diagram of the structure of the retrieval model in the intelligent question-answering system disclosed in the embodiment of this application. Figure 2As shown, the retrieval model in the intelligent question-answering system includes a question encoder, a text encoder, a computing module, and a retrieval module. The question encoder and the text encoder have the same structure. As an implementation method, both the question encoder and the text encoder can use a pre-trained bidirectional text encoder (Bidirectional Encoder Representations from Transformers, BERT). The BERT model is composed of 24 stacked Transformer layers, and each input word is represented as a feature vector of a specific length (e.g., 1024 dimensions). The input and output of each Transformer layer are both feature vectors of a specific length (e.g., 1024 dimensions). After calculation at each Transformer layer, the feature vector representation of each word will contain more contextual information. For the question encoder using the pre-trained BERT model, its [CLS]-bit output feature vector can be used as the feature vector of the question input by the user. For the sample text encoder using the pre-trained BERT model, its [CLS]-bit output feature vector can be used as the feature vector representation of each text in the text library.

[0119] Given a question input by a user, the question encoder encodes the input question into a feature vector. Similarly, the text encoder encodes each document in the text library and generates its own feature vector. The calculation module calculates the semantic similarity between the question and the text based on the feature vectors of the question and the text. The retrieval module then retrieves text related to the question based on these similarities.

[0120] In this application, it can be based on Figure 2 The structure of the retrieval model in the intelligent question-answering system shown builds a semantic similarity calculation model, and trains the semantic similarity calculation model. The trained semantic similarity calculation model executes step S102, which will be described in detail in the following embodiments.

[0121] Reference Figure 3 , Figure 3 This is a structural diagram of a semantic similarity calculation model disclosed in an embodiment of the present application. The semantic similarity calculation model may include a question encoder, a text encoder, and a calculation module. Figure 3 The semantic similarity calculation model shown is an implementable method. The process of calculating the semantic similarity between each text in the text library and the sample question may include the following steps:

[0122] Step S201: inputting the sample question into a question encoder of a semantic similarity calculation model, wherein the question encoder encodes the sample question to obtain a feature vector of the sample question;

[0123] Step S202: inputting each text in the text library into the text encoder of the semantic similarity calculation model, and the text encoder encodes each text in the text library to obtain a feature vector of each text in the text library;

[0124] Step S203: The calculation module of the semantic similarity calculation model calculates the similarity between the feature vector of each text in the text library and the feature vector of the sample question as the semantic similarity between each text in the text library and the sample question.

[0125] In another embodiment of the present application, Figure 3 The training process of the semantic similarity calculation model shown in FIG is described in detail, and the process may include the following steps:

[0126] Step S301: determining a pre-training data set, wherein the pre-training data set includes a plurality of pre-training data, and each pre-training data includes a sample question, a positive sample of the sample question, and a negative sample of the sample question.

[0127] In this application, if the sample text contains the answer to the sample question, the sample text can be considered as a positive sample of the sample question; if the sample text does not contain the answer to the sample question, the sample text can be considered as a negative sample of the sample question.

[0128] As an implementable method, the pre-training data may be all or part of the data in the training set of the retrieval model in the intelligent question-answering system, and this application does not impose any limitation on this.

[0129] Step S302: Determine a first label of a positive sample of the sample question and a second label of a negative sample of the sample question.

[0130] In this application, the first tag and the second tag have different values. For example, the first tag may be 1, and the second tag may be 0.

[0131] Step S303: Taking the semantic similarity between each sample question and the positive sample of the sample question approaching the first label, and taking the semantic similarity between the sample question and the negative sample of the sample question approaching the second label as training objectives, pre-train the semantic similarity calculation model to obtain a pre-trained semantic similarity calculation model.

[0132] The semantic similarity between each sample question and the positive sample of the sample question is the similarity between the feature vector of each sample question and the feature vector of the positive sample of the sample question, and the semantic similarity between the sample question and the negative sample of the sample question is the similarity between the feature vector of each sample question and the feature vector of the negative sample of the sample question.

[0133] When pre-training the semantic similarity calculation model, binary cross entropy can be used as the loss function, or other loss functions such as mean square error loss (MSE) can be used. This application does not impose any restrictions on this.

[0134] Step S304: Optimizing and training the question encoder of the pre-trained semantic similarity calculation model to obtain the semantic similarity calculation model.

[0135] In another embodiment of the present application, the step S304 is to optimize and train the question encoder of the pre-trained semantic similarity calculation model to obtain the semantic similarity calculation model, including:

[0136] Step S401: Obtain optimization training sample problem.

[0137] In the present application, the problem of optimizing training samples may be a problem of some or all samples in the pre-training dataset, or a problem of samples not included in the pre-training dataset, and the embodiments of the present application do not impose any limitation on this.

[0138] Step S402: Calculating the semantic similarity between each text in the text library and each optimized training sample question based on the pre-trained semantic similarity calculation model;

[0139] In this application, as an implementation method, based on the pre-trained semantic similarity calculation model, the implementation process of calculating the semantic similarity between each text in the text library and each optimized training sample problem can be as follows:

[0140] For each optimized training sample problem, the optimized training sample problem is input into the problem encoder of the pre-trained semantic similarity calculation model, and the problem encoder encodes the optimized training sample problem to obtain a feature vector of the optimized training sample problem;

[0141] Input each text in the text library into the text encoder of the pre-trained semantic similarity calculation model to obtain the feature vector of each text in the text library;

[0142] The semantic similarity between the optimized training sample problem and each text in the text library is calculated by a calculation module in the pre-trained semantic similarity calculation model.

[0143] However, there are multiple problems with optimizing training samples. For each optimized training sample problem, semantic similarity must be calculated with each text in the text library. If a problem encoder is used to encode each text in the text library for each optimized training sample problem, the feature vectors of each text in the text library will be repeatedly calculated. Since the text library contains a large number of texts, calculating the feature vectors of each text in the text library each time will consume a large amount of computing resources. Repeated calculation of feature vectors for each text in the text library will result in a waste of computing resources.

[0144] In order to save computing resources, in this application, as another possible implementation method, based on the pre-trained semantic similarity calculation model, the implementation process of calculating the semantic similarity between each text in the text library and each optimized training sample problem can be as follows:

[0145] Step S4021: Each text in the text library is input into a text encoder in the pre-trained semantic similarity calculation model for encoding, generating a text feature vector index library. The text feature vector index library includes multiple text feature vector indexes, each corresponding to a text, and each text feature vector index includes the text's ID in the text library and the text's feature vector. Based on each text feature vector index in the text library, the feature vector of the text can be quickly obtained without repeatedly calculating the feature vector of the text in the text library.

[0146] Step S4022: Based on the question encoder in the pre-trained semantic similarity calculation model and the text feature vector index library, the semantic similarity between each text in the text library and each optimized training sample question is calculated.

[0147] For details, please refer to Figure 4 , Figure 4 A schematic diagram of a process for calculating the semantic similarity between each text in a text library and each optimized training sample question based on the question encoder in the pre-trained semantic similarity calculation model and the text feature vector index library disclosed in an embodiment of the present application, Figure 4 It can be seen that for each optimized training sample problem, the optimized training sample problem is input into the problem encoder of the pre-trained semantic similarity calculation model, and the problem encoder encodes the optimized training sample problem to obtain the feature vector of the optimized training sample problem; the feature vector of each text in the text library is obtained from the text feature vector index library, and is used to calculate the semantic similarity between each text in the text library and each optimized training sample problem.

[0148] Step S403: Based on the semantic similarity between each text in the text library and each optimized training sample question, a negative sample of each optimized training sample question is determined from the text library.

[0149] In the present application, for each optimized training sample problem, a sample text set of the optimized training sample problem can be determined from the text library, wherein the sample text set includes a preset number of texts corresponding to the top-ranked semantic similarities;

[0150] The remaining sample texts after removing the sample texts containing the answers to the optimized training sample questions from the optimized training sample question sample text set are determined as negative samples of the optimized training sample questions.

[0151] It should be noted that, based on the question encoder in the pre-trained semantic similarity calculation model and the text feature vector index library, the semantic similarity between each text in the text library and each optimized training sample question is calculated. For each optimized training sample question, the specific method of determining the optimized training sample question sample text set from the text library can be as follows:

[0152] A target number set is determined based on a text feature vector index library, wherein the target number set includes numbers corresponding to feature vectors of texts corresponding to a preset number of semantic similarities that are ranked top, and a text set corresponding to the target number set is requested from the text library as a sample text set corresponding to the optimization training sample problem.

[0153] Step S404: determining a third label of the negative sample of the optimization training sample problem, wherein the third label is used to represent the probability that the negative sample of the optimization training sample problem is determined to be a sample containing an answer to the optimization training sample problem.

[0154] As one possible implementation, a positive sample of the optimized training sample question can be obtained; the retrieval model of the intelligent question-answering model is trained with the semantic similarity between the optimized training sample question and the positive sample of the optimized training sample question approaching the first label, and the similarity between the optimized training sample question and the negative sample of the optimized training sample question approaching the second label as the training goal. Determining the third label of the negative sample of the optimized training sample question may include:

[0155] The similarity between the optimized training sample question and the negative sample of the optimized training sample question output by the retrieval model of the intelligent question answering model at the end of training is used as the third label of the negative sample of the optimized training sample question.

[0156] When training the retrieval model, the similarity between each negative sample of the optimized training sample problem and the optimized training sample problem can be regarded as a feedback on the "difficulty" of the negative sample. For example, the value range of similarity is (0, 1). If the score of the negative sample of a certain optimized training sample problem is high (such as 0.9), it means that it has successfully misled the retrieval model, so it is natural to think that the negative sample is an adversarial sample with high deceptiveness. On the contrary, if a negative sample scores very low (such as 0.1), it means that the retrieval model believes with a very high confidence that the negative sample is indeed irrelevant to the problem, so it can be considered that the negative sample is a poor negative sample and is not deceptive. Therefore, at the end of the training of the retrieval model of the intelligent question-answering model, the similarity between the output of the optimized training sample problem and the negative sample of the optimized training sample problem can be used as a supervisory signal to optimize the training of the pre-trained semantic similarity calculation model.

[0157] As another possible implementation method, a positive sample of the optimized training sample question can be obtained; a position label of the answer to the optimized training sample question can be determined; and the retrieval model of the intelligent question-answering model can be trained with the probability that the answer to the optimized training sample question is included in the positive sample of the optimized training sample question being close to the first label, the probability that the answer to the optimized training sample question is included in the negative sample of the optimized training sample question being close to the second label, and the position of the answer to the optimized training sample question being close to the position label of the answer to the optimized training sample question as the training target. Then, determining the third label of the negative sample of the optimized training sample question can include:

[0158] The probability that the negative sample of the optimized training sample question output by the question answering model of the intelligent question answering model at the end of training contains the answer to the optimized training sample question is used as the third label of the negative sample of the optimized training sample question.

[0159] It should be noted that for a difficult negative sample, the question answering model will be more easily misled and think that the negative sample has a high probability of being a positive sample (such as ), and for relatively poor negative samples, the question-answering model will easily determine that the negative sample is irrelevant to the question (such as ). Therefore, in this application, the probability that each negative sample given by the question-answering model contains the answer to the optimized training sample question can be used as a supervisory signal to optimize the training of the pre-trained semantic similarity calculation model.

[0160] Step S405: taking the similarity between each optimized training sample problem and the negative sample of the optimized training sample problem approaching the third label as a training goal, optimizing the pre-trained semantic similarity calculation model.

[0161] It should be noted that, with respect to the case of calculating the semantic similarity between each text in the text library and each optimized training sample question based on the question encoder in the pre-trained semantic similarity calculation model and the text feature vector index library, when optimizing the training of the pre-trained semantic similarity calculation model, only the parameters of the question encoder in the pre-trained semantic similarity calculation model can be adjusted, and the text encoder in the pre-trained semantic similarity calculation model is a fixed parameter to ensure the correlation between the feature vector of the optimized training sample question generated by the question encoder and the feature vector of each text in the text feature vector index library.

[0162] In another embodiment of the present application, after determining the negative sample of the sample question based on the above embodiment, the negative sample of the sample question can also be used to train the intelligent question answering model. Specifically, the intelligent question answering model training method may include the following steps:

[0163] Obtaining training samples, where the training samples include a sample problem, a positive sample of the sample problem, and a negative sample of the sample problem, wherein the negative sample of the sample problem is obtained based on the negative sample determination method described in the above embodiment;

[0164] Determine labels of positive samples of the sample problem and labels of negative samples of the sample problem;

[0165] The retrieval model and / or question-answering model of the intelligent question-answering model is trained with the similarity between the sample question and the positive sample of the sample question approaching the label of the positive sample of the sample question, and the similarity between the sample question and the negative sample of the sample question approaching the label of the negative sample of the sample question to obtain the intelligent question-answering model.

[0166] It should be noted that, based on the above embodiment, it is determined that the negative samples of the sample questions are highly confusing. Therefore, using them to train the retrieval model and / or question-answering model in the intelligent question-answering system can improve the training effect.

[0167] In another embodiment of the present application, after determining the negative samples of the sample questions based on the above embodiment, the negative samples of the sample questions can also be used to detect the robustness of the retrieval model and / or question-answering model of the intelligent question-answering system.

[0168] Taking the detection of the robustness of the question-answering model of the intelligent question-answering system as an example, we can first use the text detected by the retrieval model of the intelligent question-answering system as the input of the question-answering model to obtain the first result output by the question-answering model, and then use the negative sample of the sample question as the input of the question-answering model to obtain the second result output by the question-answering model. Compare the preset evaluation indicators of the two results, such as the exact match rate (EM), F1 score, etc., and determine their decrease values. If the decrease value is less than the preset threshold, it means that the robustness of the question-answering model is good; otherwise, it means that the robustness of the question-answering model is poor.

[0169] It should be noted that, based on the above embodiment, the negative samples of the sample questions are determined to be highly confusing. Therefore, when they are used to detect the robustness of the retrieval model and / or question-answering model of the intelligent question-answering system, the effect is better.

[0170] The negative sample determination device disclosed in the embodiments of the present application is described below. The negative sample determination device described below and the negative sample determination method described above can be referenced to each other.

[0171] Reference Figure 5 , Figure 5 This is a schematic diagram of the structure of a negative sample determination device disclosed in an embodiment of this application. Figure 5 As shown, the negative sample determination device may include:

[0172] An acquisition unit 51 is configured to acquire a sample question to be determined as a negative sample and a text library corresponding to the sample question;

[0173] A calculation unit 52 is used to calculate the semantic similarity between each text in the text library and the sample question;

[0174] A sample text set determining unit 53 is configured to determine a sample text set from the text library, wherein the sample text set includes a preset number of texts with the highest semantic similarity rankings;

[0175] The negative sample determination unit 54 is configured to determine the remaining sample texts in the sample text set after removing the sample texts containing the answers to the sample questions as negative samples of the sample questions.

[0176] As an implementable embodiment, the computing unit includes:

[0177] a sample question encoding unit, configured to input the sample question into a question encoder of a semantic similarity calculation model, wherein the question encoder encodes the sample question to obtain a feature vector of the sample question;

[0178] a text encoding unit, configured to input each text in the text library into a text encoder of the semantic similarity calculation model, wherein the text encoder encodes each text in the text library to obtain a feature vector of each text in the text library;

[0179] The semantic similarity calculation unit is used to calculate the similarity between the feature vector of each text in the text library and the feature vector of the sample question using the calculation module of the semantic similarity calculation model, as the semantic similarity between each text in the text library and the sample question.

[0180] As an implementable embodiment, the device further includes: a semantic similarity calculation model training unit;

[0181] The semantic similarity calculation model training unit includes:

[0182] A pre-training data set determining unit, configured to determine a pre-training data set, wherein the pre-training data set includes a plurality of pre-training data, each pre-training data including a sample question, a positive sample of the sample question, and a negative sample of the sample question;

[0183] A first determining unit, configured to determine a first label of a positive sample of the sample question and a second label of a negative sample of the sample question;

[0184] a pre-training unit, configured to pre-train the semantic similarity calculation model with the goal that the semantic similarity between each sample question and a positive sample of the sample question approaches the first label, and the semantic similarity between the sample question and a negative sample of the sample question approaches the second label, thereby obtaining a pre-trained semantic similarity calculation model;

[0185] The optimization training unit is used to optimize the question encoder of the pre-trained semantic similarity calculation model to obtain the semantic similarity calculation model.

[0186] As an implementable embodiment, the optimization training unit includes:

[0187] An optimization training sample problem acquisition unit, used for acquiring optimization training sample problems;

[0188] A calculation subunit, configured to calculate the semantic similarity between each text in the text library and each optimized training sample question based on the pre-trained semantic similarity calculation model;

[0189] A negative sample determination subunit, configured to determine a negative sample for each optimized training sample problem from the text library based on the semantic similarity between each text in the text library and each optimized training sample problem;

[0190] A second determining unit is configured to determine a third label of the negative sample of the optimization training sample problem, wherein the third label is used to represent a probability that the negative sample of the optimization training sample problem is determined to be a sample containing an answer to the optimization training sample problem;

[0191] The optimization training subunit is used to optimize the pre-trained semantic similarity calculation model by taking the similarity between each optimization training sample problem and the negative sample of the optimization training sample problem approaching the third label as a training goal.

[0192] As an implementable embodiment, the computing subunit includes:

[0193] a text feature vector index library generating unit, configured to input each text in the text library into a text encoder in the pre-trained semantic similarity calculation model for encoding, thereby generating a text feature vector index library, wherein the text feature vector index library includes a plurality of text feature vector indexes, each text feature vector index corresponding to a text, and each text feature vector index including a number of the text in the text library and a feature vector of the text;

[0194] The semantic similarity calculation subunit is used to calculate the semantic similarity between each text in the text library and each optimized training sample question based on the question encoder in the pre-trained semantic similarity calculation model and the text feature vector index library.

[0195] As an implementable embodiment, the device further includes:

[0196] A first positive sample acquisition unit is configured to acquire a positive sample of the optimization training sample problem before determining a third label of the negative sample of the optimization training sample problem;

[0197] The retrieval model training unit is used to train the retrieval model of the intelligent question answering model with the semantic similarity between the optimized training sample problem and the positive sample of the optimized training sample problem approaching the first label, and with the similarity between the optimized training sample problem and the negative sample of the optimized training sample problem approaching the second label as training objectives.

[0198] As an implementable embodiment, the second determining unit is specifically configured to:

[0199] The similarity between the optimized training sample question and the negative sample of the optimized training sample question output by the retrieval model of the intelligent question answering model at the end of training is used as the third label of the negative sample of the optimized training sample question.

[0200] As an implementable embodiment, the device further includes:

[0201] A second positive sample acquisition unit is configured to acquire a positive sample of the optimization training sample problem before determining a third label of the negative sample of the optimization training sample problem;

[0202] an answer position label determination unit, configured to determine a position label of an answer to the problem of optimizing training samples;

[0203] The retrieval model training unit is used to train the retrieval model of the intelligent question-answering model with the probability that the answer to the optimized training sample question is contained in the positive sample of the optimized training sample question approaching the first label, the probability that the answer to the optimized training sample question is contained in the negative sample of the optimized training sample question approaching the second label, and the position of the answer to the optimized training sample question approaching the position label of the answer to the optimized training sample question as training objectives.

[0204] As an implementable embodiment, the second determining unit is specifically configured to:

[0205] The probability that the negative sample of the optimized training sample question output by the question answering model of the intelligent question answering model at the end of training contains the answer to the optimized training sample question is used as the third label of the negative sample of the optimized training sample question.

[0206] As an implementable embodiment, the device further includes:

[0207] A detection unit is used to detect the robustness of the retrieval model and / or question-answering model of the intelligent question-answering model using the negative samples of the sample questions after determining the negative samples of the sample questions.

[0208] Reference Figure 6 , Figure 6 The hardware structure diagram of the negative sample determination device provided in the embodiment of the present application is shown in FIG. Figure 6 ,The hardware structure of the negative sample determination device may include: at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4;

[0209] In the embodiment of the present application, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 communicate with each other through the communication bus 4;

[0210] The processor 1 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention;

[0211] The memory 3 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0212] The memory stores a program, and the processor can call the program stored in the memory, wherein the program is used to:

[0213] Obtaining sample questions for which negative samples are to be determined and a text library corresponding to the sample questions;

[0214] Calculate the semantic similarity between each text in the text library and the sample question;

[0215] Determining a sample text set from the text library, wherein the sample text set includes a preset number of texts with top-ranked semantic similarities;

[0216] The remaining sample texts after removing the sample texts containing the answers to the sample questions from the sample text set are determined as negative samples of the sample questions.

[0217] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0218] The present application also provides a readable storage medium, which may store a computer program suitable for execution by a processor, wherein the computer program is used to:

[0219] Obtaining sample questions for which negative samples are to be determined and a text library corresponding to the sample questions;

[0220] Calculate the semantic similarity between each text in the text library and the sample question;

[0221] Determining a sample text set from the text library, wherein the sample text set includes a preset number of texts with top-ranked semantic similarities;

[0222] The remaining sample texts after removing the sample texts containing the answers to the sample questions from the sample text set are determined as negative samples of the sample questions.

[0223] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0224] The present application also provides an intelligent question-answering model training device, comprising:

[0225] A training sample acquisition unit, configured to acquire training samples, wherein the training samples include a sample problem, a positive sample of the sample problem, and a negative sample of the sample problem, wherein the negative sample of the sample problem is obtained based on the negative sample determination method described in the above embodiment;

[0226] a label determination unit, configured to determine labels of positive samples of the sample question and labels of negative samples of the sample question;

[0227] A training unit is used to train the retrieval model and / or question-answering model of the intelligent question-answering model with the similarity between the sample question and the positive sample of the sample question approaching the label of the positive sample of the sample question, and the similarity between the sample question and the negative sample of the sample question approaching the label of the negative sample of the sample question, so as to obtain the intelligent question-answering model.

[0228] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0229] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0230] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A negative sample determination method, characterized in that: The method comprises: Obtaining sample questions for which negative samples are to be determined and a text library corresponding to the sample questions; Calculate the semantic similarity between each text in the text library and the sample question; Determining a sample text set from the text library, wherein the sample text set includes a preset number of texts with top-ranked semantic similarities; The remaining sample texts after removing the sample text containing the answer to the sample question from the sample text set are determined as negative samples of the sample question. The negative samples have high semantic similarity with the sample question but do not contain the answer to the sample question.

2. The method according to claim 1, characterized in that The calculating of the semantic similarity between each text in the text library and the sample question includes: Inputting the sample question into a question encoder of a semantic similarity calculation model, wherein the question encoder encodes the sample question to obtain a feature vector of the sample question; Inputting each text in the text library into the text encoder of the semantic similarity calculation model, wherein the text encoder encodes each text in the text library to obtain a feature vector of each text in the text library; The calculation module of the semantic similarity calculation model calculates the similarity between the feature vector of each text in the text library and the feature vector of the sample question as the semantic similarity between each text in the text library and the sample question.

3. The method according to claim 2, characterized in that The training method of the semantic similarity calculation model is as follows: Determine a pre-training data set, wherein the pre-training data set includes a plurality of pre-training data, each pre-training data includes a sample question, a positive sample of the sample question, and a negative sample of the sample question; Determine a first label of a positive sample of the sample problem and a second label of a negative sample of the sample problem; Pre-training the semantic similarity calculation model with the goal that the semantic similarity between each sample question and a positive sample of the sample question approaches the first label, and with the goal that the semantic similarity between the sample question and a negative sample of the sample question approaches the second label, thereby obtaining a pre-trained semantic similarity calculation model; The question encoder of the pre-trained semantic similarity calculation model is optimized and trained to obtain the semantic similarity calculation model.

4. The method according to claim 3, characterized in that The step of optimizing and training the question encoder of the pre-trained semantic similarity calculation model to obtain the semantic similarity calculation model includes: Obtaining optimization training sample problems; Calculating the semantic similarity between each text in the text library and each optimized training sample question based on the pre-trained semantic similarity calculation model; Determining negative samples of each optimized training sample problem from the text library based on semantic similarity between each text in the text library and each optimized training sample problem; Determining a third label of the negative sample of the optimization training sample problem, wherein the third label is used to represent a probability that the negative sample of the optimization training sample problem is determined to be a sample containing an answer to the optimization training sample problem; The pre-trained semantic similarity calculation model is optimized and trained with the similarity between each optimized training sample problem and the negative sample of the optimized training sample problem approaching the third label as a training goal.

5. The method according to claim 4, characterized in that The semantic similarity calculation model based on the pre-training is used to calculate the semantic similarity between each text in the text library and each optimized training sample question, including: Input each text in the text library into the text encoder in the pre-trained semantic similarity calculation model for encoding to generate a text feature vector index library, wherein the text feature vector index library includes multiple text feature vector indexes, each text feature vector index corresponds to a text, and each text feature vector index includes the number of the text in the text library and the feature vector of the text; Based on the question encoder in the pre-trained semantic similarity calculation model and the text feature vector index library, the semantic similarity between each text in the text library and each optimized training sample question is calculated.

6. The method according to claim 4, characterized in that Before determining the third label of the negative sample of the optimization training sample problem, the method further includes: Obtaining a positive sample of the optimization training sample problem; The retrieval model of the intelligent question answering model is trained with the training objectives that the semantic similarity between the optimized training sample question and the positive sample of the optimized training sample question is close to the first label, and the similarity between the optimized training sample question and the negative sample of the optimized training sample question is close to the second label.

7. The method according to claim 6, characterized in that Determining the third label of the negative sample of the optimization training sample problem includes: The similarity between the optimized training sample question and the negative sample of the optimized training sample question output by the retrieval model of the intelligent question answering model at the end of training is used as the third label of the negative sample of the optimized training sample question.

8. The method according to claim 4, characterized in that Before determining the third label of the negative sample of the optimization training sample problem, the method further includes: Obtaining a positive sample of the optimization training sample problem; Determining position labels of answers to the problem of optimizing training samples; The retrieval model of the intelligent question-answering model is trained with the probability that the answer to the optimized training sample question is included in the positive sample of the optimized training sample question approaching the first label, the probability that the answer to the optimized training sample question is included in the negative sample of the optimized training sample question approaching the second label, and the position of the answer to the optimized training sample question approaching the position label of the answer to the optimized training sample question as training objectives.

9. The method according to claim 4, characterized in that Determining the third label of the negative sample of the optimization training sample problem includes: The probability that the negative sample of the optimized training sample question output by the question answering model of the intelligent question answering model at the end of training contains the answer to the optimized training sample question is used as the third label of the negative sample of the optimized training sample question.

10. The method according to claim 1, characterized in that After determining a negative sample of the sample problem, the method further includes: The negative samples of the sample questions are used to detect the robustness of the retrieval model and / or the question-answering model of the intelligent question-answering model.

11. A method for training an intelligent question-answering model, characterized in that: The method comprises: Obtaining training samples, where the training samples include a sample problem, a positive sample of the sample problem, and a negative sample of the sample problem, wherein the negative sample of the sample problem is obtained based on the negative sample determination method according to any one of claims 1 to 10; Determine labels of positive samples of the sample problem and labels of negative samples of the sample problem; The retrieval model and / or question-answering model of the intelligent question-answering model is trained with the similarity between the sample question and the positive sample of the sample question approaching the label of the positive sample of the sample question, and the similarity between the sample question and the negative sample of the sample question approaching the label of the negative sample of the sample question to obtain the intelligent question-answering model.

12. A negative sample determination device, characterized in that: The device comprises: An acquisition unit, configured to acquire a sample question of a to-be-determined negative sample and a text library corresponding to the sample question; A calculation unit, used to calculate the semantic similarity between each text in the text library and the sample question; A sample text set determining unit, configured to determine a sample text set from the text library, wherein the sample text set includes a preset number of texts with top-ranked semantic similarities; The negative sample determination unit is used to determine the remaining sample texts in the sample text set after removing the sample text containing the answer to the sample question as negative samples of the sample question, where the negative samples have high semantic similarity with the sample question but do not contain the answer to the sample question.

13. A negative sample determination device, characterized in that: including memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement each step of the negative sample determination method according to any one of claims 1 to 10.

14. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the negative sample determination method according to any one of claims 1 to 10 is implemented.

15. An intelligent question-answering model training device, characterized in that: The device comprises: a training sample acquisition unit, configured to acquire training samples, the training samples comprising a sample problem, a positive sample of the sample problem, and a negative sample of the sample problem, wherein the negative sample of the sample problem is obtained based on the negative sample determination method according to any one of claims 1 to 10; a label determination unit, configured to determine labels of positive samples of the sample question and labels of negative samples of the sample question; A training unit is used to train the retrieval model and / or question-answering model of the intelligent question-answering model with the similarity between the sample question and the positive sample of the sample question approaching the label of the positive sample of the sample question, and the similarity between the sample question and the negative sample of the sample question approaching the label of the negative sample of the sample question, so as to obtain the intelligent question-answering model.

Images