Method, device and electronic device for sorting candidate sentences

Abstract

The present invention provides a method, device and electronic device for sorting candidate sentences, which obtain a candidate sentence set corresponding to a target question text, wherein the candidate sentence set includes at least one candidate sentence, determines feature information of the candidate sentence, calls a preset sorting model to process the feature information of the candidate sentence, obtains the relevance between the target question text and the candidate sentence, and screens out the candidate sentence with the greatest relevance to the target question text from the candidate sentence set. The preset sorting model in the present invention is composed of a pre-training module, a convolution module and an attention module connected in sequence, and is obtained through training based on training samples. The preset sorting model in the present invention is obtained through training with a large number of training samples, and the model has a high accuracy. Then, through the relevance output by the model, the accuracy of screening out the candidate sentence with the greatest relevance to the target question text will also be high, thereby making the accuracy of the answer returned to the user high, thereby improving the user experience.

Description

Technical Field

[0001] The present invention relates to the field of text sorting, and more specifically, to a method, device and electronic device for sorting candidate sentences. Background Art

[0002] At present, when conducting search-based intelligent question-answering, users can enter a question, and the intelligent question-answering device will output an answer corresponding to the question. For example, if a user enters "How is the weather today", the intelligent question-answering device will answer "It is sunny today".

[0003] When determining a reply, the intelligent question-and-answer device obtains the user's input, then selects several questions that are semantically related to the user's input from the question library as candidate questions, performs weighted calculations on the feature values ​​of the candidate questions to obtain a score, selects the optimal candidate question based on the score, and returns the answer to the optimal candidate question to the user.

[0004] The above step of determining the best candidate question from the candidate questions based on the feature value has a low accuracy in determining the best candidate question, so that the answer returned to the user has a low accuracy and the user experience is poor. Summary of the invention

[0005] In view of this, the present invention provides a method, device and electronic device for sorting candidate sentences to solve the problem of low accuracy in determining the best candidate questions and poor user experience.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] A method for ranking candidate sentences, comprising:

[0008] Obtaining a candidate sentence set corresponding to the target question text; the candidate sentence set includes at least one candidate sentence;

[0009] Determining feature information of the candidate sentence;

[0010] Calling a preset sorting model to process the feature information of the candidate sentence to obtain the relevance between the target question text and the candidate sentence; the preset sorting model is composed of a pre-training module, a convolution module, and an attention module connected in sequence; the feature information is used as input to the pre-training module and the convolution module respectively; the preset sorting model is obtained by training based on training samples; the training samples include: feature information of a sentence sample composed of a question sample and a candidate sample corresponding to the question sample, and the relevance between the candidate sample and the question sample;

[0011] A candidate sentence with the greatest relevance to the target question text is selected from the candidate sentence set.

[0012] Optionally, determining feature information of the candidate sentence includes:

[0013] Combining the target question text with the candidate sentences to obtain an initial sentence to be processed, and adjusting the length of the sentence to be processed based on the length of the sentence to be processed to obtain a target sentence to be processed;

[0014] Performing position encoding on the characters in the candidate sentence to obtain a position encoding result of the candidate sentence;

[0015] Performing character encoding on the words in the candidate sentence to obtain a character encoding result of the candidate sentence;

[0016] The target sentence to be processed, the position encoding result, and the character encoding result are combined to obtain feature information of the candidate sentence.

[0017] Optionally, the generation process of the preset sorting model includes:

[0018] Obtaining a problem sample, a candidate sample corresponding to the problem sample, and a target correlation between the candidate sample and the problem sample;

[0019] Determining feature information of the candidate sample;

[0020] The preset sorting model is trained using the target correlation between the candidate sample and the problem sample and the feature information of the candidate sample until the training is stopped when a preset training stop condition is met.

[0021] Optionally, get a sample of questions, including:

[0022] Acquire an initial sample set; the initial sample set includes a plurality of initial samples;

[0023] Based on the text vectors of the initial samples, clustering the multiple initial samples to obtain clustering results;

[0024] When the number of initial samples included in the clustering result is greater than a preset threshold, performing similarity analysis on the initial samples in the clustering result to obtain similarities between the initial samples;

[0025] One of the initial samples in the initial sample pair whose similarity is greater than a preset similarity threshold is used as a problem sample, and an initial sample whose similarity to any other initial sample is not greater than the preset similarity threshold is used as a problem sample;

[0026] The initial samples in the clustering results whose number of included initial samples is not greater than a preset threshold are taken as problem samples.

[0027] Optionally, using the target correlation between the candidate sample and the problem sample and the feature information of the candidate sample to train a preset sorting model until the training is stopped when a preset training stop condition is met, comprises:

[0028] Arrange and combine candidate samples corresponding to the problem samples in pairs to obtain a combination result;

[0029] Screen out a combination result including a first candidate sample whose target relevance corresponding to the first candidate sample is greater than a second candidate sample whose target relevance corresponding to the second candidate sample, and use it as the target combination result;

[0030] Inputting the feature information corresponding to the two candidate samples in the target combination result into a preset sorting model respectively to obtain the output correlation between the two candidate samples and the problem sample;

[0031] Calculating the difference of the output correlations, and determining the difference, the difference of the target correlations between the two candidate samples in the target combination result and the problem sample, and the corresponding loss function;

[0032] Based on the value of the loss function, the preset sorting model is trained until the loss function value is less than a preset threshold.

[0033] Optionally, determining the difference, the difference in target relevance between two candidate samples in the target combination result and the problem sample, and a corresponding loss function includes:

[0034] Calculating a first normalized discounted cumulative gain of the target combination result based on a difference in target relevance between two candidate samples in the target combination result and the problem sample;

[0035] The candidate samples in the target combination result are swapped, and based on the difference in target relevance between two candidate samples in the target combination result and the problem sample, a second normalized discounted cumulative gain of the target combination result after the swap is calculated;

[0036] A loss function is calculated according to the first normalized discounted cumulative gain, the second normalized discounted cumulative gain, and the difference.

[0037] A candidate sentence sorting device, comprising:

[0038] A set acquisition module, used to acquire a candidate sentence set corresponding to the target question text; the candidate sentence set includes at least one candidate sentence;

[0039] An information determination module, used to determine feature information of the candidate sentence;

[0040] A relevance determination module is used to call a preset sorting model to process the feature information of the candidate sentence to obtain the relevance between the target question text and the candidate sentence; the preset sorting model is composed of a pre-training module, a convolution module, and an attention module connected in sequence; the feature information is used as input to the pre-training module and the convolution module respectively; the preset sorting model is obtained by training based on training samples; the training samples include: feature information of a sentence sample composed of a question sample and a candidate sample corresponding to the question sample, and the relevance between the candidate sample and the question sample;

[0041] The screening module is used to screen out the candidate sentence with the greatest relevance to the target question text from the candidate sentence set.

[0042] Optionally, the information determination module includes:

[0043] A length adjustment submodule, used for combining the target question text with the candidate sentence to obtain an initial sentence to be processed, and performing a length adjustment operation on the sentence to be processed based on the length of the sentence to be processed to obtain a target sentence to be processed;

[0044] A position encoding submodule, used for performing position encoding on the characters in the candidate sentence to obtain a position encoding result of the candidate sentence;

[0045] A character encoding submodule, used for performing character encoding on the words in the candidate sentence to obtain a character encoding result of the candidate sentence;

[0046] The combining submodule is used to combine the target sentence to be processed, the position encoding result, and the character encoding result to obtain feature information of the candidate sentence.

[0047] Optionally, a model generation module is further included, and the model generation module includes:

[0048] A data acquisition submodule, used to acquire a problem sample, a candidate sample corresponding to the problem sample, and a target correlation between the candidate sample and the problem sample;

[0049] A feature determination submodule, used to determine feature information of the candidate sample;

[0050] The training submodule is used to train the preset sorting model using the target correlation between the candidate sample and the problem sample and the feature information of the candidate sample until the training is stopped when a preset training stop condition is met.

[0051] An electronic device comprising: a memory and a processor;

[0052] Wherein, the memory is used to store programs;

[0053] The processor calls the program and is used to execute the above-mentioned sorting method of candidate statements.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] The present invention provides a candidate sentence sorting method, device and electronic device, which obtain a candidate sentence set corresponding to a target question text; the candidate sentence set includes at least one candidate sentence, determines the feature information of the candidate sentence, calls a preset sorting model to process the feature information of the candidate sentence, obtains the relevance between the target question text and the candidate sentence, and screens out the candidate sentence with the greatest relevance to the target question text from the candidate sentence set. The preset sorting model in the present invention is composed of a pre-training module, a convolution module, and an attention module connected in sequence; the feature information is respectively used as the input of the pre-training module and the convolution module; the preset sorting model is obtained by training based on training samples; the training samples include: feature information of a sentence sample composed of a question sample and a candidate sample corresponding to the question sample, and the relevance between the candidate sample and the question sample. Since the preset sorting model in the present invention is obtained by training a large number of training samples, the accuracy of the model is high, and the accuracy of screening out the candidate sentence with the greatest relevance to the target question text through the relevance output by the model will also be high, so that the accuracy of the answer returned to the user is high, and the user experience is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0057] Figure 1 A method flow chart of a method for sorting candidate sentences provided by an embodiment of the present invention;

[0058] Figure 2 A method flow chart of another method for sorting candidate sentences provided by an embodiment of the present invention;

[0059] Figure 3 A schematic diagram of a scenario of a preset sorting model provided by an embodiment of the present invention;

[0060] Figure 4 A method flow chart of another method for sorting candidate sentences provided by an embodiment of the present invention;

[0061] Figure 5 A method flow chart of another method for sorting candidate sentences provided by an embodiment of the present invention;

[0062] Figure 6 A method flow chart of a fifth candidate sentence sorting method provided by an embodiment of the present invention;

[0063] Figure 7 A schematic diagram of training a preset sorting model provided in an embodiment of the present invention;

[0064] Figure 8 A schematic diagram of the structure of a candidate sentence sorting device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] When determining a reply, the intelligent question-and-answer device obtains the user's input, then selects several questions that are semantically related to the user's input from the question library as candidate questions, performs weighted calculations on the feature values ​​of the candidate questions to obtain a score, selects the optimal candidate question based on the score, and returns the answer to the optimal candidate question to the user.

[0067] In the above step of determining the best candidate question from the candidate questions based on the feature value, if the feature selection is inappropriate or the number of feature selection is small, the accuracy of the determined best candidate question will be low, thereby making the accuracy of the answer returned to the user low and the user experience poor.

[0068] In order to solve this technical problem, the inventors found that a neural network model can be used to implement a solution for determining the optimal candidate question among candidate questions.

[0069] However, the neural network model requires a large amount of domain text annotation data, which is difficult and costly to obtain. In addition, the number of texts in unpopular fields is small, the coverage is small, and the distributed representation of text semantics is not comprehensive enough, which makes the accuracy of the neural network model low.

[0070] In order to solve the problem that training requires a large amount of labeled data, and the number of texts in obscure fields is small and the coverage is small, the inventors found that the pre-trained model + convolutional model + attention mechanism model can be used to train the sorting model based on the pre-trained model, which can achieve similar results on less labeled text. In addition, the sorting model based on the pre-trained model has better sorting effect on obscure and rare texts than the existing sorting model.

[0071] The use of a method based on a pre-trained model can greatly reduce the amount of domain annotated data required in training the sorting model. Compared with using a deep learning model, training a neural network model from scratch to achieve the same effect requires at least millions of labeled data. If a pre-trained model is used to train and achieve the same effect, only 10,000 pieces of data are needed. This method can reduce the need for labeled data and save the cost of obtaining labeled data.

[0072] In addition, the use of pre-trained model training methods can cover a larger language range. In sorting cases that occur less frequently (unpopular fields), the sorting effect based on the pre-trained sorting model greatly exceeds that of the ordinary neural network model.

[0073] Specifically, the present invention provides a candidate sentence sorting method, device and electronic device, obtaining a candidate sentence set corresponding to a target question text; the candidate sentence set includes at least one candidate sentence, determining the feature information of the candidate sentence, calling a preset sorting model to process the feature information of the candidate sentence, obtaining the relevance between the target question text and the candidate sentence, and filtering out the candidate sentence with the greatest relevance to the target question text from the candidate sentence set and outputting it. The preset sorting model in the present invention is composed of a pre-training module, a convolution module, and an attention module connected in sequence; the feature information is respectively used as the input of the pre-training module and the convolution module; the preset sorting model is obtained by training based on training samples; the training samples include: feature information of a sentence sample composed of a question sample and a candidate sample corresponding to the question sample, and the relevance between the candidate sample and the question sample. Since the preset sorting model in the present invention is obtained by training a large number of training samples, the accuracy of the model is high, and the accuracy of filtering out the candidate sentence with the greatest relevance to the target question text through the relevance output by the model will also be high, so that the accuracy of the answer returned to the user is high, and the user experience is improved.

[0074] Based on the above content, an embodiment of the present invention provides a method for ranking candidate sentences, which can be applied to intelligent question-answering devices, such as intelligent robots. Figure 1 , the ranking methods of candidate statements may include:

[0075] S11. Obtain a set of candidate sentences corresponding to the target question text.

[0076] The candidate sentence set includes at least one candidate sentence.

[0077] Specifically, the target question text is input by the user, such as "What's the weather like today?", and then the intelligent question-answering device can determine questions semantically related to the target question text, each question as a candidate sentence, and the obtained candidate sentences constitute a candidate sentence set.

[0078] Candidate sentences could be:

[0079] How is the weather today?

[0080] What is the weather like?

[0081] Will it rain today...etc.

[0082] In this embodiment, the candidate sentences are sentences pre-stored in the intelligent question-answering device and have corresponding answers. In this embodiment, sentences semantically related to the target question text are screened out from the multiple sentences pre-stored in the intelligent question-answering device and used as candidate sentences.

[0083] S12: Determine feature information of the candidate sentence.

[0084] In practical applications, the feature information includes position encoding results, character encoding results, and mask adding results.

[0085] Reference Figure 2 , step S12 may include:

[0086] S21. Combining the target question text with the candidate sentences to obtain an initial sentence to be processed, and based on the length of the sentence to be processed, performing a length adjustment operation on the sentence to be processed to obtain a target sentence to be processed.

[0087] In this embodiment, the standard question input length is set to a maximum length of 64. If the input length is less than 64, a mask (usually 0) needs to be added to reach the standard input length of 64. If the input length is greater than 64, only 64 bits are retained and the rest are deleted.

[0088] When adding mask data, the target question text is first combined with the candidate sentence. When combined, the two can be directly connected to obtain the initial sentence to be processed. In this embodiment, the two are combined for the purpose of subsequent analysis of the correlation between the two.

[0089] If there are five candidate sentences corresponding to a target question text, the target question text is respectively matched with each candidate sentence to obtain five initial sentences to be processed.

[0090] After obtaining the initial statement to be processed, a mask addition or deletion operation is performed on it to obtain the target statement to be processed.

[0091] S22: Perform position encoding on the characters in the candidate sentence to obtain a position encoding result of the candidate sentence.

[0092] The position encoding refers to the position of each word in the candidate sentence, which is usually a vector with the same length as the subsequent character encoding token Embedding, usually a vector of length 256, and zeros are added if it is less.

[0093] S23: Perform character encoding on the words in the candidate sentence to obtain a character encoding result of the candidate sentence.

[0094] Token Embedding represents the vector carrier of text semantics, usually a vector of length 256.

[0095] S24: Combine the target sentence to be processed, the position encoding result, and the character encoding result to obtain feature information of the candidate sentence.

[0096] The feature information of the candidate sentence can be obtained by adding the target sentence to be processed, the position encoding result, and the character encoding result.

[0097] S13: calling a preset sorting model to process the feature information of the candidate sentences to obtain the relevance between the target question text and the candidate sentences.

[0098] Specifically, by inputting the feature information of each candidate sentence into a preset sorting model, the relevance of each candidate sentence to the target question text can be obtained, and the relevance is expressed as a numerical value.

[0099] In this embodiment, refer to Figure 3 The preset sorting model is composed of a pre-training module, a convolution module, and an attention module connected in sequence. The feature information is used as the input of the pre-training module and the convolution module respectively.

[0100] Among them, the convolution module can be composed of multiple convolution units. Figure 3 In the textCNN (convolution kernel 768), textCNN (convolution kernel 256) and textCNN (convolution kernel 128), textCNN (convolution kernel 128) are composed.

[0101] The pre-training module converts the input text into a better vector expression through powerful text vector representation capabilities. The pre-trained model is a model that has been trained using massive data. The pre-trained model contains a vast amount of language knowledge and can greatly enhance the text semantic representation capabilities.

[0102] There are already many pre-trained models in the field of natural language processing. Pre-trained models usually refer to transformer-based neural network models. This model is trained using massive, easily accessible unsupervised data. A well-trained pre-trained model can use distributed expression semantic information, which also lays the foundation for training sorting models based on pre-trained models.

[0103] The basic architecture of TextCNN is a convolutional neural network, which can further improve the quality of vector representation by further reducing the dimensions (768-256-128) of the sentence vector representation output by the pre-trained model.

[0104] The connection between Token Embedding and TextCNN is set up because the semantic expression ability of feature information will be reduced after the pre-training model is used. The semantic expression ability is maintained by introducing feature information and directly adding it.

[0105] The attention module focuses on learning the relationship between different characters in a sentence and optimizing the sorting features.

[0106] In this embodiment, the feature information of the candidate sentences is input into the pre-training module. The output of the pre-training module and the feature information of the candidate sentences are used as the input of the convolution module. After three layers of convolution units, the output of the last layer of convolution units is input into the attention module to obtain the relevance between the target question text and the candidate sentences. Figure 3 The fraction 1 represents .

[0107] In another implementation of the present invention, the preset sorting model is obtained by training based on training samples. The training samples include:

[0108] Feature information of a sentence sample consisting of a question sample and a candidate sample corresponding to the question sample, and the correlation between the candidate sample and the question sample.

[0109] S14: Filter out the candidate sentence with the greatest relevance to the target question text from the candidate sentence set.

[0110] Specifically, the candidate sentence with the greatest relevance is screened out. The candidate sentence is the sentence most relevant to the target question text input by the user and stored in the intelligent question-answering device in this embodiment. For example, the user inputs:

[0111] "How's the weather today?"

[0112] The candidate sentences selected by the intelligent question-answering device are:

[0113] How is the weather today?

[0114] The intelligent question-answering device then determines the answer corresponding to the candidate sentence and outputs the answer to complete the interaction with the user.

[0115] In this embodiment, a candidate sentence set corresponding to the target question text is obtained; the candidate sentence set includes at least one candidate sentence, the feature information of the candidate sentence is determined, the preset sorting model is called to process the feature information of the candidate sentence, the relevance between the target question text and the candidate sentence is obtained, and the candidate sentence with the greatest relevance to the target question text is screened out from the candidate sentence set. The preset sorting model in the present invention is composed of a pre-training module, a convolution module, and an attention module connected in sequence; the feature information is respectively used as the input of the pre-training module and the convolution module; the preset sorting model is obtained by training based on training samples; the training samples include: feature information of a sentence sample composed of a question sample and a candidate sample corresponding to the question sample, and the relevance between the candidate sample and the question sample. Since the preset sorting model in the present invention is obtained by training a large number of training samples, the accuracy of the model is high, and the accuracy of screening the candidate sentence with the greatest relevance to the target question text through the relevance output by the model will also be high, so that the accuracy of the answer returned to the user is high, and the user experience is improved.

[0116] In addition, the use of methods based on pre-trained models can greatly reduce the amount of domain annotation data required in training sorting models and save the cost of acquiring annotation data.

[0117] In addition, the use of pre-trained model training methods can cover a larger language range. In sorting cases that occur less frequently (unpopular fields), the sorting effect based on the pre-trained sorting model greatly exceeds that of the ordinary neural network model.

[0118] The above embodiment mentions the preset sorting model, and now the generation process thereof is introduced, referring to Figure 4 , the generation process of the preset sorting model includes:

[0119] S41, obtaining a problem sample, a candidate sample corresponding to the problem sample, and a target correlation between the candidate sample and the problem sample.

[0120] In this embodiment, the question sample is similar to the target question text mentioned above. In actual application, refer to Figure 5 , obtaining sample questions may include:

[0121] S51. Obtain an initial sample set.

[0122] The initial sample set includes multiple initial samples.

[0123] Specifically, an initial sample set containing a preset number of questions, such as 5000 questions, may be prepared in advance. Each question in the initial sample set is an initial sample.

[0124] In order to prevent the content in the question from containing sensitive fields, in this embodiment, a keyword method is used to remove inappropriate content such as insults and abuse.

[0125] S52: Clustering the multiple initial samples based on the text vectors of the initial samples to obtain clustering results.

[0126] Specifically, in practical applications, there may be duplication in the initial samples, and processing the duplicate data will occupy unnecessary resources.

[0127] Therefore, in this embodiment, a corresponding text vector is generated for each initial sample, and Kemans clustering is performed on the initial sample based on the text vector to obtain a clustering result.

[0128] S53: When the number of initial samples included in the clustering result is greater than a preset threshold, perform similarity analysis on the initial samples in the clustering result to obtain similarities between the initial samples.

[0129] In this embodiment, for each clustering result, the number of initial samples included therein may be different. In order to ensure a balanced distribution of samples, a preset threshold is set, such as 50. If the number of initial samples included therein is greater than 50, the initial samples are deduplicated.

[0130] At this time, similarity analysis is performed on the initial samples in the clustering result to obtain the similarity between the initial samples. The similarity can be calculated using a set similarity calculation method.

[0131] S54: taking one of the initial samples in the initial sample pair whose similarity is greater than a preset similarity threshold as a problem sample, and taking the initial sample whose similarity to any other initial sample is not greater than the preset similarity threshold as a problem sample.

[0132] Specifically, a preset similarity threshold is set, such as 0.98. If the similarity is greater than 0.98, it means that the two are highly similar. Only one of the initial samples is retained as the problem sample to ensure the diversity of the training data set.

[0133] If, in the clustering result, the similarity between some initial samples and any other initial samples is not greater than 0.98, they are directly regarded as problem samples.

[0134] S55. The initial samples in the clustering results whose number of initial samples is not greater than a preset threshold are taken as problem samples.

[0135] Specifically, if the number of initial samples included therein is not greater than 50, in order to ensure the data volume of the clustering result, each initial sample in the clustering result is directly used as a problem sample.

[0136] Through the above steps S51-S55, the question samples in this embodiment can be obtained.

[0137] After determining the problem sample, the next step is to determine the candidate sample corresponding to the problem sample. Specifically, the problem sample can be searched using search software, and multiple candidate samples with different relevance can be selected from different pages of search results.

[0138] For example, for each data query on Baidu Zhidao, 10 questions are randomly selected on page 1 and page 2 as candidate samples with a relevance of 2. 5 questions are randomly selected on page 25 and page 26 as candidate samples with a relevance of 1. 5 questions are randomly selected from the candidate samples of other questions as candidate samples with a relevance of 0.

[0139] When selecting candidate samples, the target correlation between the candidate samples and the problem samples is manually set, such as 2, 1, and 0. The candidate samples can refer to Table 1.

[0140] Table 1

[0141]

[0142]

[0143] S42: Determine feature information of the candidate sample.

[0144] Specifically, determining the feature information of the candidate sample is similar to determining the feature information of the candidate sentence described above. Please refer to the corresponding description above.

[0145] S43: Use the target correlation between the candidate sample and the problem sample and the feature information of the candidate sample to train the preset sorting model until the preset training stop condition is met.

[0146] Specifically, refer to Figure 6 , step S43 may include:

[0147] S61, arranging and combining candidate samples corresponding to the problem sample in pairs to obtain a combination result.

[0148] Specifically, for a problem sample, the corresponding candidate samples are arranged and combined in pairs. When combining, the candidate samples are arranged in a certain order, such as:

[0149] (The first candidate sample, the second candidate sample) and (the second candidate sample, the first candidate sample) belong to different combination results.

[0150] Through this step, multiple combination results corresponding to the same problem sample can be obtained. When combining, candidate samples with different target relevance can be combined at will.

[0151] S62. Filter out a combination result including a first candidate sample whose target relevance corresponding to the first candidate sample is greater than a second candidate sample whose target relevance corresponding to the second candidate sample, and use the combination result as the target combination result.

[0152] Specifically, when candidate samples are sorted, there is a priority order, in which candidate samples with a greater relevance to the problem sample are ranked in the front, and candidate samples with a smaller relevance to the problem sample are ranked in the back.

[0153] Therefore, in this embodiment, the combination results including the target correlation corresponding to the first candidate sample is greater than the target correlation corresponding to the second candidate sample are screened out. For example, the target correlation of the first candidate sample is 2, and the target correlation of the second candidate sample is 1. Then keep

[0154] (the first candidate sample, the second candidate sample) this combination, delete (the second candidate sample, the first candidate sample) this combination.

[0155] S63: Input feature information corresponding to the two candidate samples in the target combination result into a preset sorting model respectively to obtain output correlation between the two candidate samples and the problem sample.

[0156] Specifically, a target combination result includes two candidate samples, and the corresponding feature information is respectively input into the above-mentioned preset sorting model, referring to Figure 7 , Figure 7 The architecture is a twin network structure.

[0157] The preset sorting model outputs score 1 and score 2, where score 1 represents the output correlation between one of the candidate samples in the target combination result and the problem sample, and score 2 represents the output correlation between another candidate sample in the target combination result and the problem sample.

[0158] S64, calculating the difference of the output correlation, and determining the difference, the difference of the target correlation between the two candidate samples in the target combination result and the problem sample, and the corresponding loss function.

[0159] Specifically, the difference in output relevance is score 1-score 2, and score 1-score 2 represents the difference in relevance between the two candidate samples output by the model and the problem sample.

[0160] The target relevance between the two candidate samples in the target combination result and the problem sample is pre-marked, and the difference between the two is calculated. In this embodiment, the difference in target relevance needs to be made closer to the above score 1-score 2 to complete the model training.

[0161] During training, a loss function is used as a basis for determining whether to stop training. The loss function is determined based on the difference in output relevance and the difference in target relevance between the two candidate samples in the target combination result and the problem sample.

[0162] In practical applications, the difference between the difference value and the target correlation between the two candidate samples in the target combination result and the problem sample is determined, and the corresponding loss function may include:

[0163] 1) Based on the difference in target relevance between two candidate samples in the target combination result and the problem sample, a first normalized discounted cumulative gain of the target combination result is calculated.

[0164] The normalized discounted cumulative gain NDCG, when calculated, uses the target relevance difference between the two candidate samples in the target combination result and the problem sample.

[0165] 2) The candidate samples in the target combination result are swapped, and based on the difference in target relevance between two candidate samples in the target combination result and the problem sample, a second normalized discounted cumulative gain of the target combination result after the swap is calculated.

[0166] Specifically, assuming that the candidate samples in the target combination result are (first candidate sample, second candidate sample), at this time, the order of the candidate samples is swapped, that is, (second candidate sample, first candidate sample), and the NDCG is calculated again, which is called the second normalized discounted cumulative gain in this embodiment.

[0167] 3) Calculating a loss function based on the first normalized discounted cumulative gain, the second normalized discounted cumulative gain, and the difference.

[0168] Specifically, the loss function is the cross entropy loss function, and the formula of the loss function is:

[0169]

[0170] Among them, s i is the fraction 1, s j is the above fraction 2, σ=1, ΔNDCG=the difference between the first normalized discounted cumulative gain and the second normalized discounted cumulative gain. r(i) is the candidate sample ranked in the front in the target combination result, and r(j) is the candidate sample ranked in the back in the target combination result.

[0171] Calculate L corresponding to all target combination results as the loss function.

[0172] S65. Based on the value of the loss function, train the preset sorting model until the loss function value is less than a preset threshold.

[0173] In this embodiment, the above-mentioned ΔNDCG is used as the loss weight, and the relevance and ranking are used as the target, and the cross entropy loss is used for training. The parameters in the preset ranking model are continuously adjusted so that the value of the loss function changes continuously. When it is less than the preset threshold, the training can be stopped to obtain the preset ranking model that can be finally used online. After obtaining the preset ranking model, the model can be used for ranking inference.

[0174] In this embodiment, the training method of this model, compared with the traditional method of training the ranking model, achieves the same effect, requires less labeled data and shorter training time, and has better generalization performance.

[0175] Optionally, based on the above embodiment of the method for ranking candidate sentences, another embodiment of the present invention provides a device for ranking candidate sentences, referring to Figure 8 , which may include:

[0176] A set acquisition module 11 is used to acquire a candidate sentence set corresponding to the target question text; the candidate sentence set includes at least one candidate sentence;

[0177] An information determination module 12, used to determine feature information of the candidate sentence;

[0178] The relevance determination module 13 is used to call a preset sorting model to process the feature information of the candidate sentence to obtain the relevance between the target question text and the candidate sentence; the preset sorting model is composed of a pre-training module, a convolution module, and an attention module connected in sequence; the feature information is used as input to the pre-training module and the convolution module respectively; the preset sorting model is obtained by training based on training samples; the training samples include: feature information of a sentence sample composed of a question sample and a candidate sample corresponding to the question sample, and the relevance between the candidate sample and the question sample;

[0179] The screening module 14 is used to screen out the candidate sentence with the greatest relevance to the target question text from the candidate sentence set.

[0180] Further, the information determination module includes:

[0181] A length adjustment submodule, used for combining the target question text with the candidate sentence to obtain an initial sentence to be processed, and performing a length adjustment operation on the sentence to be processed based on the length of the sentence to be processed to obtain a target sentence to be processed;

[0182] A position encoding submodule, used for performing position encoding on the characters in the candidate sentence to obtain a position encoding result of the candidate sentence;

[0183] A character encoding submodule, used for performing character encoding on the words in the candidate sentence to obtain a character encoding result of the candidate sentence;

[0184] The combining submodule is used to combine the target sentence to be processed, the position encoding result, and the character encoding result to obtain feature information of the candidate sentence.

[0185] Furthermore, a model generation module is included, and the model generation module includes:

[0186] A data acquisition submodule, used to acquire a problem sample, a candidate sample corresponding to the problem sample, and a target correlation between the candidate sample and the problem sample;

[0187] A feature determination submodule, used to determine feature information of the candidate sample;

[0188] The training submodule is used to train the preset sorting model using the target correlation between the candidate sample and the problem sample and the feature information of the candidate sample until the training is stopped when a preset training stop condition is met.

[0189] Furthermore, the data acquisition submodule includes:

[0190] A sample acquisition unit, configured to acquire an initial sample set; the initial sample set includes a plurality of initial samples;

[0191] A clustering unit, used for clustering the multiple initial samples based on the text vectors of the initial samples to obtain a clustering result;

[0192] A similarity determination unit, configured to perform similarity analysis on the initial samples in the clustering result to obtain similarities between the initial samples when the number of initial samples included in the clustering result is greater than a preset threshold;

[0193] The sample determination unit is used to take one of the initial samples in the initial sample pair whose similarity is greater than a preset similarity threshold as a problem sample, and to take the initial sample whose similarity with any other initial sample is not greater than the preset similarity threshold as a problem sample, and to take the initial sample in the clustering result whose number of initial samples is not greater than the preset threshold as a problem sample.

[0194] Furthermore, the training submodule includes:

[0195] A combining unit, used for arranging and combining candidate samples corresponding to the problem samples in pairs to obtain a combination result;

[0196] A screening unit, used to screen out a combination result including a first candidate sample whose target relevance corresponding to the first candidate sample is greater than a second candidate sample whose target relevance corresponding to the second candidate sample, and use the combination result as the target combination result;

[0197] A correlation determination unit, used to input feature information corresponding to two candidate samples in the target combination result into a preset sorting model respectively, to obtain an output correlation between the two candidate samples and the problem sample;

[0198] A data calculation unit, used to calculate the difference of the output correlation, and determine the difference, the difference of the target correlation between the two candidate samples in the target combination result and the problem sample, and the corresponding loss function;

[0199] The training unit is used to train the preset sorting model based on the value of the loss function until the loss function value is less than a preset threshold.

[0200] Furthermore, the data calculation unit includes:

[0201] A first calculation subunit, configured to calculate a first normalized discounted cumulative gain of the target combination result based on a difference in target relevance between two candidate samples in the target combination result and the problem sample;

[0202] A second calculation subunit is used to swap the positions of the candidate samples in the target combination result, and calculate a second normalized discounted cumulative gain of the target combination result after the position swap based on the difference in target relevance between the two candidate samples in the target combination result and the problem sample;

[0203] The third calculation subunit is used to calculate a loss function according to the first normalized discounted cumulative gain, the second normalized discounted cumulative gain, and the difference.

[0204] In this embodiment, a candidate sentence set corresponding to the target question text is obtained; the candidate sentence set includes at least one candidate sentence, the feature information of the candidate sentence is determined, the preset sorting model is called to process the feature information of the candidate sentence, the relevance between the target question text and the candidate sentence is obtained, and the candidate sentence with the greatest relevance to the target question text is screened out from the candidate sentence set. The preset sorting model in the present invention is composed of a pre-training module, a convolution module, and an attention module connected in sequence; the feature information is respectively used as the input of the pre-training module and the convolution module; the preset sorting model is obtained by training based on training samples; the training samples include: feature information of a sentence sample composed of a question sample and a candidate sample corresponding to the question sample, and the relevance between the candidate sample and the question sample. Since the preset sorting model in the present invention is obtained by training a large number of training samples, the accuracy of the model is high, and the accuracy of screening the candidate sentence with the greatest relevance to the target question text through the relevance output by the model will also be high, so that the accuracy of the answer returned to the user is high, and the user experience is improved.

[0205] It should be noted that, for the working process of each module, sub-module, unit and sub-unit in this embodiment, please refer to the corresponding description in the above embodiment, which will not be repeated here.

[0206] Optionally, based on the above embodiments of the method and device for ranking candidate sentences, another embodiment of the present invention provides an electronic device, which may be the above intelligent question-answering device, and includes: a memory and a processor;

[0207] Wherein, the memory is used to store programs;

[0208] The processor calls the program and is used to execute the above-mentioned sorting method of the candidate statements.

[0209] In this embodiment, a candidate sentence set corresponding to the target question text is obtained; the candidate sentence set includes at least one candidate sentence, the feature information of the candidate sentence is determined, the preset sorting model is called to process the feature information of the candidate sentence, the relevance between the target question text and the candidate sentence is obtained, and the candidate sentence with the greatest relevance to the target question text is screened out from the candidate sentence set. The preset sorting model in the present invention is composed of a pre-training module, a convolution module, and an attention module connected in sequence; the feature information is respectively used as the input of the pre-training module and the convolution module; the preset sorting model is obtained by training based on training samples; the training samples include: feature information of a sentence sample composed of a question sample and a candidate sample corresponding to the question sample, and the relevance between the candidate sample and the question sample. Since the preset sorting model in the present invention is obtained by training a large number of training samples, the accuracy of the model is high, and the accuracy of screening the candidate sentence with the greatest relevance to the target question text through the relevance output by the model will also be high, so that the accuracy of the answer returned to the user is high, and the user experience is improved.

[0210] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for ranking candidate sentences, characterized in that: include: Obtain a set of candidate sentences corresponding to the target question text; The candidate sentence set includes at least one candidate sentence; Determining feature information of the candidate sentence; Calling a preset sorting model to process the feature information of the candidate sentence to obtain the relevance between the target question text and the candidate sentence; The preset sorting model consists of a pre-training module, a convolution module, and an attention module connected in sequence; The feature information is used as input of the pre-training module and the convolution module respectively; The preset sorting model is obtained by training based on training samples; The training sample includes: feature information of a sentence sample consisting of a question sample and a candidate sample corresponding to the question sample, and a correlation between the candidate sample and the question sample; Filtering out the candidate sentence with the greatest relevance to the target question text from the candidate sentence set; Wherein, determining the characteristic information of the candidate sentence includes: Combining the target question text with the candidate sentences to obtain an initial sentence to be processed, and adjusting the length of the sentence to be processed based on the length of the sentence to be processed to obtain a target sentence to be processed; Performing position encoding on the characters in the candidate sentence to obtain a position encoding result of the candidate sentence; Performing character encoding on the words in the candidate sentence to obtain a character encoding result of the candidate sentence; The target sentence to be processed, the position encoding result, and the character encoding result are combined to obtain feature information of the candidate sentence.

2. The method for ranking candidate sentences according to claim 1, characterized in that: The generation process of the preset sorting model includes: Obtaining a problem sample, a candidate sample corresponding to the problem sample, and a target correlation between the candidate sample and the problem sample; Determining feature information of the candidate sample; The preset sorting model is trained using the target correlation between the candidate sample and the problem sample and the feature information of the candidate sample until the training is stopped when a preset training stop condition is met.

3. The method for ranking candidate sentences according to claim 2, characterized in that: Get sample questions, including: Acquire an initial sample set; the initial sample set includes a plurality of initial samples; Based on the text vectors of the initial samples, clustering the multiple initial samples to obtain clustering results; When the number of initial samples included in the clustering result is greater than a preset threshold, performing similarity analysis on the initial samples in the clustering result to obtain similarities between the initial samples; One of the initial samples in the initial sample pair whose similarity is greater than a preset similarity threshold is used as a problem sample, and an initial sample whose similarity to any other initial sample is not greater than the preset similarity threshold is used as a problem sample; The initial samples in the clustering results whose number of included initial samples is not greater than a preset threshold are taken as problem samples.

4. The method for ranking candidate sentences according to claim 2, characterized in that: Using the target correlation between the candidate sample and the problem sample and the feature information of the candidate sample, training the preset sorting model until the preset training stop condition is met, including: Arrange and combine candidate samples corresponding to the problem samples in pairs to obtain a combination result; Screen out a combination result including a first candidate sample whose target relevance corresponding to the first candidate sample is greater than a second candidate sample whose target relevance corresponding to the second candidate sample, and use it as the target combination result; Inputting the feature information corresponding to the two candidate samples in the target combination result into a preset sorting model respectively to obtain the output correlation between the two candidate samples and the problem sample; Calculating the difference of the output correlations, and determining the difference, the difference of the target correlations between the two candidate samples in the target combination result and the problem sample, and the corresponding loss function; Based on the value of the loss function, the preset sorting model is trained until the loss function value is less than a preset threshold value; Wherein, determining the difference, the difference in target relevance between the two candidate samples in the target combination result and the problem sample, and the corresponding loss function include: Calculating a first normalized discounted cumulative gain of the target combination result based on a difference in target relevance between two candidate samples in the target combination result and the problem sample; The candidate samples in the target combination result are swapped, and based on the difference in target relevance between two candidate samples in the target combination result and the problem sample, a second normalized discounted cumulative gain of the target combination result after the swap is calculated; Calculate a loss function according to the first normalized discounted cumulative gain, the second normalized discounted cumulative gain, and the difference; The formula of the loss function is: Among them, s i Indicates the output correlation between one of the candidate samples in the target combination result and the problem sample, s j represents the output correlation between another candidate sample in the target combination result and the problem sample, σ=1, ΔNDCG is equal to the difference between the first normalized discounted cumulative gain and the second normalized discounted cumulative gain, r(i) is the candidate sample ranked in front in the target combination result, and r(j) is the candidate sample ranked in the back in the target combination result.

5. A candidate sentence ranking device, characterized in that: include: A set acquisition module is used to obtain a set of candidate sentences corresponding to the target question text; The candidate sentence set includes at least one candidate sentence; An information determination module, used to determine feature information of the candidate sentence; A relevance determination module, used to call a preset ranking model to process the feature information of the candidate sentence to obtain the relevance between the target question text and the candidate sentence; The preset sorting model consists of a pre-training module, a convolution module, and an attention module connected in sequence; The feature information is used as input of the pre-training module and the convolution module respectively; The preset sorting model is obtained by training based on training samples; The training sample includes: feature information of a sentence sample consisting of a question sample and a candidate sample corresponding to the question sample, and a correlation between the candidate sample and the question sample; A screening module, used to screen out the candidate sentence with the greatest relevance to the target question text from the candidate sentence set; Wherein, the information determination module includes: A length adjustment submodule, used for combining the target question text with the candidate sentence to obtain an initial sentence to be processed, and performing a length adjustment operation on the sentence to be processed based on the length of the sentence to be processed to obtain a target sentence to be processed; A position encoding submodule, used for performing position encoding on the characters in the candidate sentence to obtain a position encoding result of the candidate sentence; A character encoding submodule, used for performing character encoding on the words in the candidate sentence to obtain a character encoding result of the candidate sentence; The combining submodule is used to combine the target sentence to be processed, the position encoding result, and the character encoding result to obtain feature information of the candidate sentence.

6. The candidate sentence ranking device according to claim 5, characterized in that: It also includes a model generation module, the model generation module includes: A data acquisition submodule, used to acquire a problem sample, a candidate sample corresponding to the problem sample, and a target correlation between the candidate sample and the problem sample; A feature determination submodule, used to determine feature information of the candidate sample; The training submodule is used to train the preset sorting model using the target correlation between the candidate sample and the problem sample and the feature information of the candidate sample until the training is stopped when a preset training stop condition is met.

7. An electronic device, characterized in that: include: Memory and processor; Wherein, the memory is used to store programs; The processor calls the program and is used to execute the candidate statement sorting method as described in any one of claims 1 to 4.

Images