Document sorting method, sorting device, electronic device and storage medium

Abstract

The present application provides a document sorting method, sorting device, electronic device, and storage medium. The sorting method includes: after receiving a query statement input by a user, recalling multiple candidate documents related to the query statement; each candidate document includes associated statements related to the query statement; for each associated statement in the candidate document, splicing the query statement and the associated statements in the candidate document to obtain a spliced ​​corpus; inputting the spliced ​​corpus into a pre-trained similarity score prediction model to obtain a similarity score for the candidate document; the similarity score represents the similarity between the associated statements in the candidate document and the query statement; and determining the document sorting results of the multiple candidate documents based on the similarity score of each candidate document. The sorting method and sorting device can accurately sort documents and improve user experience.

Description

Technical Field

[0001] The present application relates to the field of information retrieval technology, and in particular to a document sorting method, a sorting device, an electronic device, and a storage medium. Background Art

[0002] With the development of electronic technology, more and more users are becoming accustomed to using electronic devices to search for various questions on the web. Search engines retrieve relevant documents from the index based on the user's query q, rank the documents by relevance, and return the ranked documents to the user. When using search engines for information retrieval, ranking algorithms are often used to sort relevant documents.

[0003] Currently, the existing method for sorting related documents uses machine learning to solve the sorting problem, namely Learning To Rank (LTR). LTR is mainly divided into three categories: PointWise (single point annotation), PairWise (pairwise annotation), and ListWise (list annotation). Existing search engines generally use the RankNet algorithm for sorting. The RankNet algorithm is a PairWise algorithm that solves the sorting problem from a probabilistic perspective. However, when using the RankNet algorithm for sorting, there is still the problem of inaccurate document sorting, which reduces the user experience. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a document sorting method, a sorting device, an electronic device and a storage medium, which can accurately sort documents and improve the user experience.

[0005] In a first aspect, an embodiment of the present application provides a document sorting method, the sorting method comprising:

[0006] After receiving a query statement input by a user, multiple candidate documents related to the query statement are recalled; each candidate document includes an associated statement related to the query statement;

[0007] For each associated sentence in the candidate document, the query sentence and the associated sentence in the candidate document are concatenated to obtain a concatenated corpus;

[0008] Inputting the concatenated corpus into a pre-trained similarity score prediction model to obtain a similarity score for the candidate document; the similarity score represents the similarity between the associated sentence in the candidate document and the query sentence;

[0009] A document ranking result of the plurality of candidate documents is determined according to the similarity score of each candidate document.

[0010] Optionally, the similarity score prediction model is obtained by training through the following steps:

[0011] Get the corpus training dataset;

[0012] An initial similarity score prediction model is trained based on the corpus training data set to obtain a similarity score prediction model.

[0013] Optionally, the corpus training dataset is obtained by the following steps:

[0014] Obtaining original single sentence corpus, and preprocessing the original single sentence corpus to obtain single sentence corpus;

[0015] Obtaining an initial corpus training data set based on the single sentence corpus; the initial corpus training data set includes a plurality of corpus pairs with set semantic labels;

[0016] A corpus training data set is obtained according to the initial corpus training data set; the corpus training data set includes multiple groups of corpus samples corresponding to each target corpus; each group of corpus samples includes multiple corpus samples with set relevance labels.

[0017] Optionally, obtaining an initial corpus training data set based on the single sentence corpus includes:

[0018] Randomly combining the single sentence corpora in pairs to obtain multiple corpus pairs;

[0019] For each corpus pair, a similarity model is used to determine the semantic relationship between the two single sentences in the corpus pair; the semantic relationship includes a similarity relationship and a dissimilarity relationship;

[0020] According to the semantic relationship, a semantic tag is set for the corpus pair to obtain a corpus pair with a set semantic tag; the categories of the semantic tag include similarity and dissimilarity;

[0021] The set of all corpus pairs with semantic labels is determined as the initial corpus training data set.

[0022] Optionally, obtaining the corpus training dataset according to the initial corpus training dataset includes:

[0023] For each single sentence in the initial corpus training data set, based on the sentence structure of the single sentence, a replicated corpus with the same sentence structure as the single sentence is selected from the initial corpus training data set, and the single sentence and the replicated corpus are used as target corpuses, and the corpus pair containing the target corpus is used as the target corpus pair;

[0024] According to the semantic labels of the target corpus pair, similar corpora with semantic similarity to the target corpus and dissimilar corpora with semantic dissimilarity to the target corpus are respectively screened out from the target corpus pair;

[0025] The target corpus, the similar corpus, and the dissimilar corpus are arranged and combined to obtain multiple groups of corpus samples corresponding to the target corpus; each corpus sample in each group of corpus samples includes the target corpus, a first corpus, and a second corpus; the first corpus is the first corpus after the target corpus, and the second corpus is the second corpus after the target corpus;

[0026] For each group of corpus samples, a relevance label is set for each corpus sample in the group according to the semantic relationship between the first corpus and the second corpus in the group of corpus samples, so as to obtain a plurality of corpus samples with set relevance labels; the categories of the relevance labels include: a first label; a second label; and a third label; the first label indicates that the first corpus is more relevant to the target corpus than the second corpus, the second label indicates that the first corpus and the second corpus have the same relevance to the target corpus, and the third label indicates that the second corpus is more relevant to the target corpus than the first corpus;

[0027] A set of all corpus samples with relevance labels set is determined as a corpus training data set.

[0028] Optionally, the training of the initial similarity score prediction model based on the corpus training dataset to obtain the similarity score prediction model includes:

[0029] For each corpus sample in the corpus training data set that is set with a relevance label, concatenating the target corpus and the first corpus in the corpus sample to obtain a first concatenated corpus, and concatenating the target corpus and the second corpus in the corpus sample to obtain a second concatenated corpus;

[0030] Inputting the first concatenated corpus and the second concatenated corpus into the initial similarity score prediction model to obtain a first similarity score output by the initial similarity score prediction model for the first concatenated corpus and a second similarity score output by the initial similarity score prediction model for the second concatenated corpus;

[0031] Determining a relevance probability of the corpus sample with the relevance label set according to the first similarity score and the second similarity score; the relevance probability represents a probability that the relevance between the first corpus and the target corpus is greater than the relevance between the second corpus and the target corpus;

[0032] According to the relevance probability and the relevance label of the corpus sample, the network parameters in the initial similarity score prediction model are continuously adjusted until the cross entropy loss function reaches a convergence state, thereby obtaining a similarity score prediction model.

[0033] Optionally, the corpus training dataset is obtained by the following steps:

[0034] Obtaining original single sentence corpus, and preprocessing the original single sentence corpus to obtain single sentence corpus;

[0035] Obtaining an initial corpus training data set based on the single sentence corpus; the initial corpus training data set includes a plurality of corpus pairs with set semantic labels;

[0036] A corpus training data set is obtained according to the initial corpus training data set; the corpus training data set includes multiple groups of corpus samples corresponding to each target corpus; each group of corpus samples includes multiple corpus samples with set relevance labels.

[0037] In a second aspect, an embodiment of the present application provides a document sorting device, the sorting device comprising:

[0038] The recall module is used to recall multiple candidate documents related to the query statement after receiving the query statement input by the user; each candidate document includes an associated statement related to the query statement;

[0039] A splicing module is used to splice the query sentence with the relevant sentences in each candidate document to obtain a spliced ​​corpus;

[0040] A calculation module is used to input the concatenated corpus into a pre-trained similarity score prediction model to obtain a similarity score of the candidate document; the similarity score represents the similarity between the associated sentence in the candidate document and the query sentence;

[0041] The ranking module is used to determine the document ranking result of the multiple candidate documents according to the similarity score of each candidate document.

[0042] Optionally, the sorting device further includes a training module, wherein the training module is configured to:

[0043] Get the corpus training dataset;

[0044] An initial similarity score prediction model is trained based on the corpus training data set to obtain a similarity score prediction model.

[0045] In a third aspect, an embodiment of the present application provides an electronic device comprising: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory through the bus, and the processor executes the machine-readable instructions to perform the steps of the document sorting method described in any one of the first aspects.

[0046] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of any document sorting method described in the first aspect are executed.

[0047] The document sorting method, sorting device, electronic device and storage medium provided in the embodiments of the present application combine the pre-trained language model Bert with the sorting algorithm RankNet to sort documents. In this way, documents can be accurately sorted and the user experience can be improved.

[0048] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 A flowchart of a document sorting method provided in an embodiment of the present application is shown;

[0051] Figure 2 A schematic diagram of the structure of an initial similarity score prediction model provided in an embodiment of the present application is shown;

[0052] Figure 3 A schematic diagram showing the process of training the initial similarity score prediction model provided in an embodiment of the present application is shown;

[0053] Figure 4 One of the structural diagrams of a document sorting device provided in an embodiment of the present application is shown;

[0054] Figure 5 The second structural diagram of a document sorting device provided in an embodiment of the present application is shown;

[0055] Figure 6 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for which protection is claimed, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work falls within the scope of protection of the present application.

[0057] Prior to the present application, the existing document ranking method used the RankNet algorithm. The RankNet algorithm is a PairWise algorithm that solves the ranking problem from a probabilistic perspective. However, the RankNet algorithm can still cause inaccurate document ranking, thereby reducing the user experience.

[0058] Based on this, embodiments of the present application provide a document sorting method, a sorting device, an electronic device, and a storage medium.

[0059] To facilitate understanding of the embodiments of the present application, a document sorting method disclosed in the embodiments of the present application is first introduced in detail.

[0060] See also Figure 1 , Figure 1 This is a flowchart of a document sorting method provided in an embodiment of the present application. Figure 1 As shown in , the document sorting method provided by the embodiment of the present application includes the following steps:

[0061] S101: After receiving a query statement input by a user, recall multiple candidate documents related to the query statement; each candidate document includes an associated statement related to the query statement;

[0062] S102: For each associated sentence in the candidate document, concatenate the query sentence with the associated sentence in the candidate document to obtain a concatenated corpus;

[0063] S103: input the concatenated corpus into a pre-trained similarity score prediction model to obtain a similarity score of the candidate document;

[0064] S104: Determine a document ranking result of the plurality of candidate documents according to the similarity score of each candidate document.

[0065] As an example, in step S101, after receiving a query statement input by a user, multiple candidate documents related to the query statement can be recalled using any method in the prior art. For example, keyword technology can be used to recall candidate documents that include keywords in the query statement, or semantic analysis technology can be used to recall candidate documents that are semantically similar to the query statement.

[0066] Here, each candidate document includes a topic question and an answer, wherein the topic question in each candidate document is a related sentence of the candidate document.

[0067] For example, assuming the query is "Can a paper policy be mailed to me for insurance purchased online?", the candidate documents that are recalled may be ["Will the insurance policy purchased from your company be mailed to me?" and "Yes"], ["Will a paper policy be mailed to me for insurance purchased online?" and "Mail"], ["Can a paper policy be mailed to me for insurance purchased online?" and "Yes"]". The topic questions, or associated statements, in each candidate document are "Will the insurance policy purchased from your company be mailed to me?", "Will a paper policy be mailed to me for insurance purchased online?", and "Can a paper policy be mailed to me for insurance purchased online?". The answers in each candidate document are "Yes", "Mail", and "Yes" respectively.

[0068] As an example, after recalling multiple candidate documents related to the query statement, in step S102, the query statement and the associated statement in each candidate document can be first spliced ​​to obtain a spliced ​​sub-corpus; then a first separator symbol is added at the starting position of the query statement, a second separator symbol is added between the query statement and the associated statement, and a second separator symbol is added at the end position of the associated statement to obtain a spliced ​​corpus.

[0069] In a specific example, assuming that the associated sentence of the candidate document is "Can the insurance policy bought from your company be mailed to me?", the query sentence "Can the insurance bought online be mailed as a paper policy?" and the associated sentence "Will the insurance policy bought from your company be mailed to me?" are spliced ​​to obtain the spliced ​​sub-corpus "Can the insurance bought online be mailed as a paper policy? Will the insurance policy bought from your company be mailed to me?", then add the first separator symbol [CLS] at the starting position of the query sentence "Can the insurance bought online be mailed as a paper policy?", add the second separator symbol [SEP] between the query sentence "Can the insurance bought online be mailed as a paper policy?" and the associated sentence "Will the insurance policy bought from your company be mailed to me?", and add the second separator symbol [SEP] at the end position of the associated sentence to obtain the spliced ​​corpus "[CLS]Can the insurance bought online be mailed as a paper policy? [SEP]Will the insurance policy bought from your company be mailed to me? [SEP]".

[0070] After obtaining the concatenated corpus for the candidate document, in step S103, the concatenated corpus is input into a similarity score prediction model to obtain a similarity score for the candidate document. Here, the similarity score prediction model is a similarity score prediction model that incorporates the pre-trained neural network model BERT. The similarity score represents the similarity between the associated statements in the candidate document and the query statement.

[0071] After obtaining the similarity score of the candidate document, the above steps S102 and S103 may be repeatedly performed to obtain the similarity score of each candidate document.

[0072] After obtaining the similarity score of each candidate document, as an example, in step S104, the plurality of candidate documents can be sorted in descending order of similarity score, and the sorted plurality of candidate documents are used as the document sorting result. After obtaining the sorted plurality of candidate documents, the user can browse the sorted plurality of candidate documents.

[0073] Here, specifically, the similarity score prediction model can be trained through the following steps:

[0074] S1. Obtain the corpus training dataset.

[0075] S2. Training an initial similarity score prediction model based on the corpus training data set to obtain a similarity score prediction model.

[0076] As an example, step S1, the step of obtaining a corpus training data set, may include steps S11, S12, and S13:

[0077] S11. Obtain original single sentence corpus, and preprocess the original single sentence corpus to obtain single sentence corpus.

[0078] As an example, the original single sentence corpus can be compiled from a frequently asked questions dataset in an existing knowledge base. Alternatively, the original single sentence corpus can be compiled based on online user search queries and recalled questions within a preset time period. Here, as an example, the frequently asked questions dataset includes historical user search queries and corpus compiled by the business party.

[0079] As an example, the preprocessing may include at least one of the following: normalization of the original single sentence corpus, pronunciation conversion of the original single sentence corpus, and format conversion of the original single sentence corpus. Here, normalization of the original single sentence corpus may include normalization of common punctuation marks, pronunciation conversion of the original single sentence corpus may include pronunciation conversion of numbers, percent signs, addition, subtraction, multiplication, and division, and other contents with Chinese pronunciations, and format conversion of the original single sentence corpus may include removal of invalid characters such as line breaks and spaces.

[0080] S12. Obtain an initial corpus training data set based on the single sentence corpus.

[0081] As an example, in step S12, the single sentence corpora can be first randomly combined in pairs to obtain multiple corpus pairs; then, for each corpus pair, the semantic relationship between the two single sentence corpora in the corpus pair is determined using a similarity model, and then, based on the semantic relationship, a semantic label is set for the corpus pair to obtain a corpus pair with a set semantic label; finally, a set consisting of all corpus pairs with a set semantic label is determined as the initial corpus training data set.

[0082] Here, the similarity model may be a BM25 model, which is an unsupervised model for evaluating similarity, that is, a similarity scoring mechanism.

[0083] The semantic relationship in the present invention includes similarity relationship and dissimilarity relationship. The categories of the semantic tags include similarity and dissimilarity.

[0084] The following is a specific example of how to obtain the initial corpus training dataset.

[0085] In a specific example, suppose the obtained single sentence corpus is "Can a paper policy be mailed to me for insurance bought online?", "Is the insurance I bought online the same as the insurance I bought in the store?", "How do you deliver the policy I bought online?" and "Will the insurance policy I bought in your company be mailed to me?", these single sentence corpora are randomly combined to obtain multiple corpus pairs [Can a paper policy be mailed to me for insurance bought online? Is the insurance I bought online the same as the insurance I bought in the store?", [Can a paper policy be mailed to me for insurance bought online? How do you deliver the policy I bought online?" and [Can a paper policy be mailed to me for insurance bought online? Will the insurance policy be mailed to me?" Then, for each corpus pair, the semantic relationship between the two single sentences in the corpus pair is determined. For example, the semantic relationship between the two single sentences in "Can insurance purchased online be mailed as a paper policy? Is the insurance I bought online the same as the insurance I bought in the store?" is a dissimilar relationship, the semantic relationship between the two single sentences in "Can insurance purchased online be mailed as a paper policy? How do you deliver the insurance policy purchased online?" is a dissimilar relationship, and the semantic relationship between the two single sentences in "Can insurance purchased online be mailed as a paper policy? Will the insurance policy purchased from your company be mailed to me?" is a similar relationship. Then, based on the semantic relationship, a semantic label is assigned to the corpus pair, that is, a similar label is assigned to corpus pairs with similar semantic relationships, and a dissimilar label is assigned to corpus pairs with dissimilar semantic relationships. After obtaining the corpus pairs with set semantic labels, the set consisting of all corpus pairs with set semantic labels is determined as the initial corpus training dataset.

[0086] S13. Obtain a corpus training dataset based on the initial corpus training dataset.

[0087] As an example, step S13 may include step S131, step S132, step S133, step S134, and step S135:

[0088] S131. For each single sentence in the initial corpus training data set, based on the sentence structure of the single sentence, filter out a replica corpus with the same sentence structure as the single sentence from the initial corpus training data set, and use the single sentence and the replica as target corpus, and use the corpus pair containing the target corpus as the target corpus pair.

[0089] Here, the sentence structure of a single sentence corpus refers to the sentence form of the single sentence corpus. As an example, the sentence structure of a single sentence corpus may include the number of words and characters in the sentences of the single sentence corpus. As an example, completely identical sentence structure means that the number of words and characters in a sentence of one single sentence corpus is completely identical to the number of words and characters in a sentence of another single sentence corpus.

[0090] In a specific example, assuming that the single sentence corpus is "Can a paper policy be mailed for insurance purchased online?", based on the sentence structure of the single sentence corpus, a replicated corpus with exactly the same sentence structure as the single sentence corpus is screened out from the initial corpus training data set, that is, other corpora that are also "Can a paper policy be mailed for insurance purchased online?", and "Can a paper policy be mailed for insurance purchased online?" is used as the target corpus, and the corpus pair containing the target corpus "Can a paper policy be mailed for insurance purchased online?" is used as the target corpus pair.

[0091] S132 , based on the semantic labels of the target corpus pair, respectively screening out similar corpora that are semantically similar to the target corpus and dissimilar corpora that are semantically dissimilar to the target corpus from the target corpus pair.

[0092] As an example, in this step, it is determined whether the semantic labels of the target corpus pair are similar labels. If the semantic labels are similar labels, the other corpus in the target corpus pair excluding the target corpus is regarded as a similar corpus; if the semantic labels are dissimilar labels, the other corpus in the target corpus pair excluding the target corpus is regarded as a dissimilar corpus.

[0093] According to the above steps, using the above specific example, for the target corpus "Can paper policies for insurance purchased online be mailed to me?", similar corpus obtained may include: "Will the insurance policies purchased from your company be mailed to me?", and dissimilar corpus obtained may include "Is the insurance I bought online the same as the insurance I bought in-store?" and "How do you deliver the policies I bought online?"

[0094] S133: Arrange and combine the target corpus, the similar corpus, and the dissimilar corpus to obtain multiple groups of corpus samples corresponding to the target corpus.

[0095] Here, each corpus sample in each group of corpus samples includes a target corpus, a first corpus, and a second corpus; the first corpus is the first corpus after the target corpus, and the second corpus is the second corpus after the target corpus; the first corpus and the second corpus in each group of corpus samples are different. As an example, the multiple groups of corpus samples may include three groups, where the first corpus in the first group of corpus samples may be a similar corpus and the second corpus may be a dissimilar corpus; the first corpus in the second group of corpus samples may be a similar corpus and the second corpus may be a similar corpus; and the first corpus in the third group of corpus samples may be a dissimilar corpus and the second corpus may be a similar corpus.

[0096] According to the above steps, using the above specific example, for the target corpus "Can a paper policy be mailed for insurance purchased online?", three groups of corpus samples corresponding to the target corpus are obtained. The three groups of corpus samples can be shown in Table 1 below:

[0097] Table 1

[0098]

[0099] S134. For each group of corpus samples, according to the semantic relationship between the first corpus and the second corpus in the group of corpus samples, set a relevance label for each corpus sample in the group of corpus samples to obtain a plurality of corpus samples with set relevance labels;

[0100] Here, the categories of the relevance labels include: a first label; a second label and a third label; the first label indicates that the first corpus is more relevant to the target corpus than the second corpus, the second label indicates that the first corpus and the second corpus have the same relevance to the target corpus, and the third label indicates that the second corpus is more relevant to the target corpus than the first corpus.

[0101] S135: Determine a set of corpus samples that are all assigned relevance labels as a corpus training data set.

[0102] After the corpus training dataset is obtained, in step S2, an initial similarity score prediction model may be trained based on the corpus training dataset.

[0103] As an example, step S2 may include: step S21, step S22, step S23 and step S24.

[0104] S21. For each corpus sample in the corpus training data set that is set with a relevance label, concatenate the target corpus and the first corpus in the corpus sample to obtain a first concatenated corpus, and concatenate the target corpus and the second corpus in the corpus sample to obtain a second concatenated corpus.

[0105] As an example, in this step, for each corpus sample in the corpus training data set that is set with a relevance label, the target corpus and the first corpus in the corpus sample can be spliced ​​to obtain a first spliced ​​sub-corpus, and then a first separator symbol is added to the starting position of the target corpus in the first spliced ​​sub-corpus, a second separator symbol is added between the target corpus and the first corpus, and a second separator symbol is added to the end position of the second corpus to obtain the first spliced ​​corpus.

[0106] At the same time, the target corpus and the second corpus in the corpus sample are spliced ​​to obtain a second spliced ​​sub-corpus, and then a first separator is added at the starting position of the target corpus in the second spliced ​​sub-corpus, a second separator is added between the target corpus and the second corpus, and a second separator is added at the end position of the second corpus to obtain a second spliced ​​corpus.

[0107] For example, for the first corpus sample in the corpus training dataset shown in Table 1, the first concatenated corpus obtained may be "[CLS] Can a paper policy be mailed to me for insurance purchased online? [SEP] Will the insurance policy purchased from your company be mailed to me? [SEP]".

[0108] The second concatenated corpus can be "[CLS] Can a paper policy be mailed to me for insurance purchased online? [SEP] Is the insurance I purchased online the same as the insurance I purchased in a store? [SEP]".

[0109] S22: Input the first concatenated corpus and the second concatenated corpus into the initial similarity score prediction model to obtain a first similarity score output by the initial similarity score prediction model for the first concatenated corpus and a second similarity score output by the initial similarity score prediction model for the second concatenated corpus.

[0110] Specifically, see Figure 2 , Figure 2 A schematic structural diagram of an initial similarity score prediction model provided in an embodiment of the present application is shown.

[0111] like Figure 2 As shown, the initial similarity score prediction model includes an encoding layer and a fully connected layer. As an example, the encoding layer can be a Bert Encoder layer in a Bert model.

[0112] As an example, in step S22, the first concatenated corpus and the second concatenated corpus can be simultaneously input into the encoding layer for encoding to obtain the context vector of the first concatenated corpus and the context vector of the second concatenated corpus, and then the context vector of the first concatenated corpus and the context vector of the second concatenated corpus are input into the fully connected layer for linear transformation to obtain the first similarity score output by the initial similarity score prediction model for the first concatenated corpus and the second similarity score output for the second concatenated corpus.

[0113] Here, the context vector of the first concatenated corpus is V a CLS , the context vector of the second concatenated corpus is V b CLS . V a CLS and V b CLS The default feature dimension is 1×768. Here, since the [CLS] symbol is inserted into the first concatenated corpus, this symbol without obvious semantic information will more "fairly" integrate the semantic information of each word or phrase in the text compared with other words or phrases in the text, so the context vector V can be used. a CLS Represents the first concatenated corpus, using the context vector V b CLS Represents the second concatenated corpus.

[0114] Linear transformation refers to adjusting the size of the output size in the fully connected layer to shrink V a CLS and V b CLS As an example, the output size is 1. Through linear transformation, the first similarity score and the second similarity score can be obtained. Here, the feature dimensions of the first similarity score and the second similarity score are 1×1 respectively.

[0115] S23: Determine the relevance probability of the corpus sample with the relevance label set according to the first similarity score and the second similarity score.

[0116] Here, the relevance probability indicates the probability that the relevance between the first corpus and the target corpus is greater than the relevance between the second corpus and the target corpus.

[0117] For example, the correlation probability P can be determined by the following formula ab

[0118]

[0119] Among them, S aS is the first similarity score output by the initial similarity score prediction model for the first concatenated corpus; b The second similarity score is output by the initial similarity score prediction model for the second concatenated corpus.

[0120] S24. Continuously adjusting the network parameters in the initial similarity score prediction model according to the relevance probability and the relevance label of the corpus sample until the cross entropy loss function reaches a convergence state, thereby obtaining a similarity score prediction model.

[0121] Here, the cross entropy loss function is the sigmoid function.

[0122] For example, the function value Loss of the cross entropy loss function sigmod can be determined using the following formula:

[0123] Loss=-[y i LogP ab +(1-y i )Log(1-P ab )]; Formula 2

[0124] Here, y i is the true label value of the i-th corpus sample; P ab is the relevance probability of the i-th corpus sample.

[0125] The following will illustrate the specific process of step S2, training the initial similarity score prediction model based on the corpus training dataset to obtain the similarity score prediction model, with reference to a specific example.

[0126] Please refer to the figure, Figure 3 A schematic diagram of the process of training the initial similarity score prediction model provided in an embodiment of the present application is shown.

[0127] like Figure 3As shown in the figure, the corpus sample with the relevance label is [Can a paper policy be mailed to me for insurance purchased online? Will the insurance policy purchased from your company be mailed to me? Is the insurance I bought online the same as the insurance I bought in the store?, the first label]. For this corpus sample, the sentences "Can a paper policy be mailed to me for insurance purchased online?" and "Will the insurance policy purchased from your company be mailed to me" in the corpus sample are spliced ​​to obtain the first spliced ​​corpus "[CLS] Can a paper policy be mailed to me for insurance purchased online? [SEP] Will the insurance policy purchased from your company be mailed to me?" The first and second concatenated corpus are input into the encoding layer (Bert Encoder layer) of the initial similarity score prediction model for encoding, and the context vector V of the first concatenated corpus is obtained. a CLS and the context vector V of the second concatenated corpus b CLS , then the context vector V of the first concatenated corpus is a CLS and the context vector V of the second concatenated corpus b CLS Input into the fully connected layer for linear transformation to obtain the first similarity score S output by the initial similarity score prediction model for the first concatenated corpus a And the second similarity score S output for the second concatenated corpus b , and then according to the first similarity score S a and the second similarity score S b , use formula 1 to determine the relevance probability P of the corpus sample with the relevance label set ab , when the cross entropy loss function sigmod reaches a convergence state, the similarity score prediction model is obtained.

[0128] The document sorting method, sorting device, electronic device and storage medium provided in the embodiments of the present application combine the pre-trained language model Bert with the sorting algorithm RankNet to sort documents. In this way, documents can be accurately sorted and the user experience can be improved.

[0129] Based on the same inventive concept, an embodiment of the present application also provides a document sorting device corresponding to the document sorting method.

[0130] See also Figure 4 As shown, Figure 4 This is a schematic diagram of the structure of a document sorting device provided in an embodiment of the present application. The document sorting device 400 includes:

[0131] The recall module 401 is configured to recall a plurality of candidate documents related to a query statement after receiving the query statement input by the user; each candidate document includes an associated statement related to the query statement;

[0132] A splicing module 402 is configured to splice the query statement with the associated statements in each candidate document to obtain a spliced ​​corpus;

[0133] The calculation module 403 is used to input the concatenated corpus into a pre-trained similarity score prediction model to obtain a similarity score of the candidate document; the similarity score represents the similarity between the associated sentence in the candidate document and the query sentence;

[0134] The ranking module 404 is configured to determine a document ranking result of the plurality of candidate documents according to the similarity score of each candidate document.

[0135] See also Figure 5 , Figure 5 The second structural diagram of a document sorting device provided in an embodiment of the present application is shown;

[0136] like Figure 5 As shown, in a possible implementation manner, the sorting device 400 further includes a training module 405, wherein the training module 405 is configured to:

[0137] Get the corpus training dataset;

[0138] An initial similarity score prediction model is trained based on the corpus training data set to obtain a similarity score prediction model.

[0139] In a possible implementation, the training module 405 is specifically configured to:

[0140] Obtaining original single sentence corpus, and preprocessing the original single sentence corpus to obtain single sentence corpus;

[0141] Obtaining an initial corpus training data set based on the single sentence corpus; the initial corpus training data set includes a plurality of corpus pairs with set semantic labels;

[0142] A corpus training data set is obtained according to the initial corpus training data set; the corpus training data set includes multiple groups of corpus samples corresponding to each target corpus; each group of corpus samples includes multiple corpus samples with set relevance labels.

[0143] In a possible implementation, the training module 405 is specifically configured to:

[0144] Randomly combining the single sentence corpora in pairs to obtain multiple corpus pairs;

[0145] For each corpus pair, a similarity model is used to determine the semantic relationship between the two single sentences in the corpus pair; the semantic relationship includes a similarity relationship and a dissimilarity relationship;

[0146] According to the semantic relationship, a semantic tag is set for the corpus pair to obtain a corpus pair with a set semantic tag; the categories of the semantic tag include similarity and dissimilarity;

[0147] The set of all corpus pairs with semantic labels is determined as the initial corpus training data set.

[0148] In a possible implementation, the training module 405 is specifically configured to:

[0149] For each single sentence in the initial corpus training data set, based on the sentence structure of the single sentence, a replicated corpus with the same sentence structure as the single sentence is selected from the initial corpus training data set, and the single sentence and the replicated corpus are used as target corpuses, and the corpus pair containing the target corpus is used as the target corpus pair;

[0150] According to the semantic labels of the target corpus pair, similar corpora with semantic similarity to the target corpus and dissimilar corpora with semantic dissimilarity to the target corpus are respectively screened out from the target corpus pair;

[0151] The target corpus, the similar corpus, and the dissimilar corpus are arranged and combined to obtain multiple groups of corpus samples corresponding to the target corpus; each corpus sample in each group of corpus samples includes the target corpus, a first corpus, and a second corpus; the first corpus is the first corpus after the target corpus, and the second corpus is the second corpus after the target corpus;

[0152] For each group of corpus samples, a relevance label is set for each corpus sample in the group according to the semantic relationship between the first corpus and the second corpus in the group of corpus samples, so as to obtain a plurality of corpus samples with set relevance labels; the categories of the relevance labels include: a first label; a second label; and a third label; the first label indicates that the first corpus is more relevant to the target corpus than the second corpus, the second label indicates that the first corpus and the second corpus have the same relevance to the target corpus, and the third label indicates that the second corpus is more relevant to the target corpus than the first corpus;

[0153] A set of all corpus samples with relevance labels set is determined as a corpus training data set.

[0154] In a possible implementation, the training module 405 is specifically configured to:

[0155] For each corpus sample in the corpus training data set that is set with a relevance label, concatenating the target corpus and the first corpus in the corpus sample to obtain a first concatenated corpus, and concatenating the target corpus and the second corpus in the corpus sample to obtain a second concatenated corpus;

[0156] Inputting the first concatenated corpus and the second concatenated corpus into the initial similarity score prediction model to obtain a first similarity score output by the initial similarity score prediction model for the first concatenated corpus and a second similarity score output by the initial similarity score prediction model for the second concatenated corpus;

[0157] Determining a relevance probability of the corpus sample with the relevance label set according to the first similarity score and the second similarity score; the relevance probability represents a probability that the relevance between the first corpus and the target corpus is greater than the relevance between the second corpus and the target corpus;

[0158] According to the relevance probability and the relevance label of the corpus sample, the network parameters in the initial similarity score prediction model are continuously adjusted until the cross entropy loss function reaches a convergence state, thereby obtaining a similarity score prediction model.

[0159] The document sorting device provided in the embodiment of the present application combines the pre-trained language model Bert with the sorting algorithm RankNet to sort documents. In this way, documents can be sorted accurately and the user experience can be improved.

[0160] See also Figure 6 , Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 6 As shown in FIG, the electronic device 600 includes a processor 610 , a memory 620 and a bus 630 .

[0161] The memory 620 stores machine-readable instructions executable by the processor 610. When the electronic device 600 is running, the processor 610 communicates with the memory 620 through the bus 630. When the machine-readable instructions are executed by the processor 610, the steps of the document sorting method in the above-mentioned method embodiment can be executed. The specific implementation method can be found in the method embodiment and will not be repeated here.

[0162] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the document sorting method in the above method embodiment can be executed. The specific implementation method can be found in the method embodiment and will not be repeated here.

[0163] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0164] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. There may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be through some communication interface, indirect coupling or communication connection of devices or units, which may be electrical, mechanical or other forms.

[0165] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0166] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0167] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0168] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The scope of protection of the present application is not limited thereto. Although the present application has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-mentioned embodiments within the technical scope disclosed in the present application, or perform equivalent replacements for some of the technical features thereof. These modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A document sorting method, characterized in that: The sorting method includes: After receiving a query statement input by a user, multiple candidate documents related to the query statement are recalled; each candidate document includes an associated statement related to the query statement; For each associated sentence in the candidate document, the query sentence and the associated sentence in the candidate document are concatenated to obtain a concatenated corpus; Inputting the concatenated corpus into a pre-trained similarity score prediction model to obtain a similarity score for the candidate document; the similarity score represents the similarity between the associated sentence in the candidate document and the query sentence; Determining a document ranking result of the plurality of candidate documents based on the similarity score of each candidate document; The similarity score prediction model is obtained by training through the following steps: Get the corpus training dataset; Training an initial similarity score prediction model based on the corpus training data set to obtain a similarity score prediction model; Obtain the corpus training dataset by following the steps below: Obtaining original single sentence corpus, and preprocessing the original single sentence corpus to obtain single sentence corpus; Obtaining an initial corpus training data set based on the single sentence corpus; the initial corpus training data set includes a plurality of corpus pairs with set semantic labels; According to the initial corpus training data set, a corpus training data set is obtained; the corpus training data set includes multiple groups of corpus samples corresponding to each target corpus; each group of corpus samples includes multiple corpus samples with set relevance labels; The step of obtaining a corpus training data set according to the initial corpus training data set comprises: For each single sentence in the initial corpus training data set, based on the sentence structure of the single sentence, a replicated corpus with the same sentence structure as the single sentence is selected from the initial corpus training data set, and the single sentence and the replicated corpus are used as target corpuses, and the corpus pair containing the target corpus is used as the target corpus pair; According to the semantic labels of the target corpus pair, similar corpora with semantic similarity to the target corpus and dissimilar corpora with semantic dissimilarity to the target corpus are respectively screened out from the target corpus pair; The target corpus, the similar corpus, and the dissimilar corpus are arranged and combined to obtain multiple groups of corpus samples corresponding to the target corpus; each corpus sample in each group of corpus samples includes the target corpus, a first corpus, and a second corpus; the first corpus is the first corpus after the target corpus, and the second corpus is the second corpus after the target corpus; For each group of corpus samples, a relevance label is set for each corpus sample in the group according to the semantic relationship between the first corpus and the second corpus in the group of corpus samples, so as to obtain a plurality of corpus samples with set relevance labels; the categories of the relevance labels include: a first label; a second label; and a third label; the first label indicates that the first corpus is more relevant to the target corpus than the second corpus, the second label indicates that the first corpus and the second corpus have the same relevance to the target corpus, and the third label indicates that the second corpus is more relevant to the target corpus than the first corpus; A set of all corpus samples with relevance labels set is determined as a corpus training data set.

2. The sorting method according to claim 1, wherein: The step of obtaining an initial corpus training data set based on the single sentence corpus includes: Randomly combining the single sentence corpora in pairs to obtain multiple corpus pairs; For each corpus pair, a similarity model is used to determine the semantic relationship between the two single sentences in the corpus pair; the semantic relationship includes a similarity relationship and a dissimilarity relationship; According to the semantic relationship, a semantic tag is set for the corpus pair to obtain a corpus pair with a set semantic tag; the categories of the semantic tag include similarity and dissimilarity; The set of all corpus pairs with semantic labels is determined as the initial corpus training data set.

3. The sorting method according to claim 1, wherein: The step of training the initial similarity score prediction model based on the corpus training dataset to obtain the similarity score prediction model includes: For each corpus sample in the corpus training data set that is set with a relevance label, concatenating the target corpus and the first corpus in the corpus sample to obtain a first concatenated corpus, and concatenating the target corpus and the second corpus in the corpus sample to obtain a second concatenated corpus; Inputting the first concatenated corpus and the second concatenated corpus into the initial similarity score prediction model to obtain a first similarity score output by the initial similarity score prediction model for the first concatenated corpus and a second similarity score output by the initial similarity score prediction model for the second concatenated corpus; Determining a relevance probability of the corpus sample with the relevance label set according to the first similarity score and the second similarity score; the relevance probability represents a probability that the relevance between the first corpus and the target corpus is greater than the relevance between the second corpus and the target corpus; According to the relevance probability and the relevance label of the corpus sample, the network parameters in the initial similarity score prediction model are continuously adjusted until the cross entropy loss function reaches a convergence state, thereby obtaining a similarity score prediction model.

4. A document sorting device, characterized in that: The sorting device comprises: The recall module is used to recall multiple candidate documents related to the query statement after receiving the query statement input by the user; each candidate document includes an associated statement related to the query statement; A splicing module is used to splice the query sentence with the relevant sentences in each candidate document to obtain a spliced ​​corpus; A calculation module is used to input the concatenated corpus into a pre-trained similarity score prediction model to obtain a similarity score of the candidate document; the similarity score represents the similarity between the associated sentence in the candidate document and the query sentence; A ranking module, configured to determine a document ranking result of the plurality of candidate documents based on a similarity score of each candidate document; The sorting device further includes a training module, which is configured to: Get the corpus training dataset; Training an initial similarity score prediction model based on the corpus training data set to obtain a similarity score prediction model; The training module is specifically used for: Randomly combine single sentence corpora in pairs to obtain multiple corpus pairs; For each corpus pair, a similarity model is used to determine the semantic relationship between the two single sentences in the corpus pair; the semantic relationship includes a similarity relationship and a dissimilarity relationship; According to the semantic relationship, a semantic tag is set for the corpus pair to obtain a corpus pair with a set semantic tag; the categories of the semantic tag include similarity and dissimilarity; The set of all semantically labeled corpus pairs is determined as the initial corpus training data set; When the training module is used to obtain the corpus training data set based on the initial corpus training data set, the training module is further specifically used to: For each single sentence in the initial corpus training data set, based on the sentence structure of the single sentence, a replicated corpus with the same sentence structure as the single sentence is selected from the initial corpus training data set, and the single sentence and the replicated corpus are used as target corpuses, and the corpus pair containing the target corpus is used as the target corpus pair; According to the semantic labels of the target corpus pair, similar corpora with semantic similarity to the target corpus and dissimilar corpora with semantic dissimilarity to the target corpus are respectively screened out from the target corpus pair; The target corpus, the similar corpus, and the dissimilar corpus are arranged and combined to obtain multiple groups of corpus samples corresponding to the target corpus; each corpus sample in each group of corpus samples includes the target corpus, a first corpus, and a second corpus; the first corpus is the first corpus after the target corpus, and the second corpus is the second corpus after the target corpus; For each group of corpus samples, a relevance label is set for each corpus sample in the group according to the semantic relationship between the first corpus and the second corpus in the group of corpus samples, so as to obtain a plurality of corpus samples with set relevance labels; the categories of the relevance labels include: a first label; a second label; and a third label; the first label indicates that the first corpus is more relevant to the target corpus than the second corpus, the second label indicates that the first corpus and the second corpus have the same relevance to the target corpus, and the third label indicates that the second corpus is more relevant to the target corpus than the first corpus; A set of all corpus samples with relevance labels set is determined as a corpus training data set.

5. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate via the bus, and the machine-readable instructions are executed by the processor to execute the steps of the document sorting method as described in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the document sorting method according to any one of claims 1 to 3 are executed.

Images