Training set construction method and device, classification model training method and device and document classification method and device

Abstract

The invention provides a training set construction method and device, a classification model training method and device and a document classification method and device, and the training set construction method comprises the steps that a training set and target corpora are obtained, each training corpus in the training set carries a category label, and the target corpus does not carry the category label; each training corpus and the target corpus are respectively input into a pre-training model to obtain a first text vector of each training corpus and a second text vector of the target corpus, and the pre-training model is obtained by pre-training a preset natural language model based on the training set; performing similarity judgment on each first text vector and the second text vector; and if the similarity judgment result reaches a preset condition, adding a target category label to the target corpus according to the similarity judgment result, and adding the target corpus added with the target category label to the training set to obtain an updated training set.

Description

[0001] This application claims the priority of the Chinese patent application filed with the China Patent Office on August 16, 2023, with application number "202311036640.3" and invention name "Training set construction, classification model training and document classification method and device", all contents of which are recorded in this application by reference. Technical Field

[0002] The present application relates to the field of data processing technology, and in particular to a training set construction method. The present application also relates to a classification model training method, a document classification method, a training set construction device, a classification model training device, a document classification device, a computing device, and a computer-readable storage medium. Background Art

[0003] For deep learning models trained in a supervised manner, a large amount of labeled training corpus is generally required for training. However, the training corpus is usually relatively small. Model training based on a small amount of training corpus will lead to suboptimal model training results.

[0004] In the prior art, unlabeled corpora are usually manually annotated to obtain more training corpora to expand the training set. However, this method not only consumes a lot of manpower and material resources, but also reduces the speed of building the training set, thereby reducing the training efficiency of the model. Summary of the invention

[0005] In view of this, the embodiment of the present application provides a training set construction method to solve the technical defects existing in the prior art. The embodiment of the present application also provides a classification model training method, a document classification method, a training set construction device, a classification model training device, a document classification device, a computing device, and a computer-readable storage medium.

[0006] According to a first aspect of an embodiment of the present application, a training set construction method is provided, comprising:

[0007] Obtaining a training set and a target corpus, wherein each training corpus in the training set carries a category label, and the target corpus does not carry a category label;

[0008] Inputting each training corpus and the target corpus into a pre-training model respectively to obtain a first text vector of each training corpus and a second text vector of the target corpus, wherein the pre-training model is obtained by pre-training a preset natural language model based on the training set;

[0009] Performing similarity judgment on each of the first text vectors and the second text vector;

[0010] If the similarity judgment result meets the preset condition, a target category label is added to the target corpus according to the similarity judgment result, and the target corpus with the target category label is added to the training set to obtain an updated training set.

[0011] According to a second aspect of an embodiment of the present application, a classification model training method is provided, comprising:

[0012] Obtain a training set, wherein the training set includes a plurality of training corpora, each training corpus carries a category label, wherein the training set is obtained by the training set construction method described in the first aspect above;

[0013] Inputting the plurality of training corpora into a classification model to obtain a predicted category for each training corpus;

[0014] Determine the loss value based on the predicted category and category label of each training corpus;

[0015] The classification model is trained based on the loss value until a training stop condition is reached.

[0016] According to a third aspect of an embodiment of the present application, a document classification method is provided, comprising:

[0017] Get the documents to be classified;

[0018] The document to be classified is input into a classification model to determine the category of the document to be classified, wherein the classification model is obtained by the classification model training method described in the second aspect above.

[0019] According to a fourth aspect of an embodiment of the present application, a training set construction device is provided, including:

[0020] A first acquisition module is configured to acquire a training set and a target corpus, wherein each training corpus in the training set carries a category label, and the target corpus does not carry a category label;

[0021] a text vector determination module, configured to input the training corpora and the target corpora into a pre-training model respectively, to obtain a first text vector of the training corpora and a second text vector of the target corpus, wherein the pre-training model is obtained by pre-training a preset natural language model based on the training set;

[0022] a similarity judgment module, configured to perform similarity judgment on each of the first text vectors and the second text vector;

[0023] The adding module is configured to add a target category label to the target corpus according to the similarity judgment result if the similarity judgment result meets the preset condition, and add the target corpus with the target category label added to the training set to obtain the updated training set.

[0024] According to a fifth aspect of an embodiment of the present application, a classification model training device is provided, comprising:

[0025] A second acquisition module is configured to acquire a training set, wherein the training set includes a plurality of training corpora, each training corpus carries a category label, wherein the training set is obtained by the training set construction method described in the first aspect above;

[0026] A prediction module is configured to input the plurality of training corpora into a classification model to obtain a predicted category of each training corpus;

[0027] A first determination module is configured to determine a loss value based on the predicted category and category label of each training corpus;

[0028] The training module is configured to train the classification model based on the loss value until a training stop condition is reached.

[0029] According to a sixth aspect of an embodiment of the present application, there is provided a document classification device, comprising:

[0030] A third acquisition module is configured to acquire a document to be classified;

[0031] The second determination module is configured to input the document to be classified into a classification model to determine the category of the document to be classified, wherein the classification model is obtained by the classification model training method described in the second aspect above.

[0032] According to a seventh aspect of an embodiment of the present application, there is provided a computing device, including:

[0033] Memory and processor;

[0034] The memory is used to store computer-executable instructions, and when the processor executes the computer-executable instructions, it can implement the steps of the training set construction method described in the first aspect above, or implement the steps of the classification model training method described in the second aspect above, or implement the steps of the document classification method described in the third aspect above.

[0035] According to an eighth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores computer-executable instructions. When the instructions are executed by a processor, the steps of the training set construction method described in the first aspect are implemented, or the steps of the classification model training method described in the second aspect are implemented, or the steps of the document classification method described in the third aspect are implemented.

[0036] According to the fifth aspect of the embodiment of the present application, a chip is provided, which stores a computer program. When the computer program is executed by the chip, the steps of the training set construction method described in the first aspect are implemented, or the steps of the classification model training method described in the second aspect are implemented, or the steps of the document classification method described in the third aspect are implemented.

[0037] The training set construction method provided in the present application obtains a training set and a target corpus, wherein each training corpus in the training set carries a category label, and the target corpus does not carry a category label; each training corpus and the target corpus are respectively input into a pre-training model to obtain a first text vector of each training corpus and a second text vector of the target corpus, wherein the pre-training model is obtained by pre-training a preset natural language model based on the training set; a similarity judgment is performed on each first text vector and the second text vector; if the similarity judgment result meets a preset condition, a target category label is added to the target corpus according to the similarity judgment result, and the target corpus with the added target category label is added to the training set to obtain an updated training set. In the above method, since the pre-trained model is obtained by training with a training set, the first text vector and the second text vector determined based on the pre-trained model are more accurate. Based on the first text vector and the second text vector, a similarity comparison is performed on the training corpus and the target corpus, and the target category label is annotated for the target corpus according to the similarity comparison result, thereby achieving the effect of automatic labeling, eliminating the need for manual labeling, reducing the consumption of manpower and material resources, and increasing the speed of constructing the training set, thereby improving the training efficiency of the model. In addition, training the model based on an updated training set including more training corpora can improve the performance of the trained model. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a structural diagram of a training set construction method provided in one embodiment of the present application;

[0039] Figure 2 is a flowchart of a training set construction method provided in one embodiment of the present application;

[0040] Figure 3 It is a flowchart of a training method of a pre-training model provided in one embodiment of the present application;

[0041] Figure 4is a flowchart of a character mask processing method provided by an embodiment of the present application;

[0042] Figure 5 is a flow chart of a method for determining a predicted character provided by an embodiment of the present application;

[0043] Figure 6 is a flow chart of a method for determining a loss value provided by an embodiment of the present application;

[0044] Figure 7 is a flow chart of a similarity determination method provided by an embodiment of the present application;

[0045] Figure 8 is a flow chart of a method for adding a target category label to a target corpus provided by an embodiment of the present application;

[0046] Fig. 9 It is a processing flow chart of a training set construction method applied to a classification model provided in one embodiment of the present application;

[0047] Fig.10 is a flow chart of a classification model training method provided by an embodiment of the present application;

[0048] Fig.11 is a flowchart of a document classification method provided by an embodiment of the present application;

[0049] Fig.12 It is a structural schematic diagram of a training set construction device provided in one embodiment of the present application;

[0050] Fig.13 It is a structural schematic diagram of a classification model training device provided in one embodiment of the present application;

[0051] Fig.14 It is a structural schematic diagram of a document classification device provided by an embodiment of the present application;

[0052] Fig.15 It is a structural block diagram of a computing device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0053] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0054] The terms used in one or more embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present application. The singular forms of "a", "said" and "the" used in one or more embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of the present application refers to and includes any or all possible combinations of one or more associated listed items.

[0055] It should be understood that, although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present application, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present application, the first may also be referred to as the second, and similarly, the second may also be referred to as the first.

[0056] First, the terms involved in one or more embodiments of the present application are explained.

[0057] Pre-trained model: used to vectorize the input text and obtain the text vector of the input text.

[0058] Classification model: used to classify the input documents and determine the category to which the input documents belong.

[0059] Mask: MASK, some characters in the text are replaced with the [mask] symbol, and the replaced characters are predicted by the model to train the model.

[0060] Next, the application scenario of the training corpus construction method provided in this application is explained.

[0061] Document classification is to input documents into a classification model to determine the category of the document. Currently, document classification is usually implemented through a classification model, and the classification model is trained in a supervised way, which requires a large number of training sets with category labels. However, manual labeling will incur high costs, so automatic expansion of training sets is of great significance. The current methods of automatic expansion of training sets mainly include the following three:

[0062] Rule-based method: manually design rules and expand the training corpus through rule matching; data enhancement method: process the training set through replacement, deletion, reordering, back translation and other techniques to expand the training corpus; multi-model fusion-based method: train multiple different models and expand data by fusing multiple recognition results.

[0063] However, the rule-based method requires dedicated design of a large number of rules, which has high construction costs and is time-consuming and labor-intensive. The data enhancement-based method is only different on the surface of the data, and is essentially the original training set without adding new training corpus. The multi-model fusion-based method requires training multiple different models, which increases cost and complexity.

[0064] To this end, this application provides a training set construction method that can solve the above technical problems. Figure 1 , Figure 1 It is a structural diagram of a training set construction method provided according to an embodiment of the present application.

[0065] Taking the execution subject as server 102 as an example, the terminal 104 uploads the training set and the target corpus, and the server 102 obtains the training set and the target corpus through the acquisition unit 102-1. Each training corpus in the training set carries a category label, and the target corpus does not carry a category label, that is, the server 102 obtains the training set and the target corpus, and then, the text vector determination unit 102-2 of the server 102 inputs each training corpus and the target corpus into the pre-training model respectively to obtain the first text vector of each training corpus and the second text vector of the target corpus. The pre-training model is obtained by pre-training a preset natural language model based on the training set; then the similarity judgment unit 102-3 of the server 102 judges the similarity between each first text vector and the second text vector; then, if the similarity judgment result meets the preset condition, the adding unit 102-4 of the server 102 adds the target category label to the target corpus according to the similarity judgment result, and adds the target corpus with the added target category label to the training set to obtain an updated training set.

[0066] Compared with the rule-based method, the above-mentioned training corpus construction method does not need to design a large number of rules, saving a lot of manpower and material resources; compared with the method based on data enhancement, what is added to the training set is the training corpus that was not originally in the training set, truly realizing the expansion of the training set; compared with the method based on multi-model fusion, only one pre-training model needs to be trained, which saves cost and is simple and convenient.

[0067] In the present application, a training corpus construction method is provided. The present application also relates to a classification model training method, a document classification method, a training corpus construction device, a classification model training device, a document classification device, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0068] Figure 2 This is a flowchart of a training set construction method provided according to an embodiment of the present application, which specifically includes the following steps:

[0069] Step 202: Obtain a training set and a target corpus, wherein each training corpus in the training set carries a category label, and the target corpus does not carry a category label.

[0070] The training set is a collection of training corpora used to train the classification model, and the target corpus is the corpus used to expand the training set.

[0071] In an embodiment of the present application, each training corpus in the training set carries a category label, and the training set constructed in the present application can be used to train a classification model, which can be used to process text classification tasks to determine the category of the input text.

[0072] In some embodiments, the training set can be obtained from an existing training corpus, or the training set can be obtained by labeling the training corpus with category labels according to the content of the training corpus. In addition, the training corpus in the training set can carry the same category labels or different category labels. The target corpus can be obtained from a corpus without annotated category labels.

[0073] As an example, the training corpus may include training corpora of the resume category and training corpora of the non-resume category (such as training corpora of the contract, finance, law, etc. categories). If the training corpus is training corpora of the resume category, the content of the training corpus may be the document content of the resume, or the key fields in the resume, or the brief content of the resume, etc.; if the training corpus is training corpora of the contract category, the content of the training corpus may be the document content of the contract, or the key fields in the contract, or the brief content of the contract, etc.

[0074] In the embodiment of the present application, since the training set needs to be expanded, it is necessary to obtain the training set and the target corpus for expanding the training set, which provides a data basis for the subsequent expansion of the training set.

[0075] Step 204: Input each training corpus and the target corpus into the pre-training model respectively to obtain a first text vector of each training corpus and a second text vector of the target corpus, wherein the pre-training model is obtained by pre-training a preset natural language model based on the training set.

[0076] The first text vector is a text vector obtained after text processing of the training corpus, which can be understood as a vectorized representation of the training corpus.

[0077] The second text vector is a text vector obtained after text processing of the target corpus, which can be understood as a vectorized representation of the target corpus.

[0078] As an example, the preset natural language model may be an untrained model for processing natural language, which can be trained into a pre-trained model with different functions based on different task requirements. The pre-trained model in the embodiment of the present application can be used to vectorize the input text and obtain the text vector of the input text, then the pre-trained model may be a model with the function of vectorizing the text. For example, the pre-trained model may be a classification model.

[0079] In some embodiments, the pre-trained model is obtained by training a preset natural language model based on a training set, which can be understood as a pre-trained model specific to the training set. The pre-trained model learns the characteristics of the training corpus in the training set, and then the training corpus in the training set is input into the pre-trained model, and the first text vector of each training corpus obtained is more accurate; and the target corpus is used to expand the training set of the training classification model. The target corpus generally belongs to the same category as some training corpus in the training set. Since the pre-trained model is obtained by training based on the training set, the pre-trained model learns the characteristics of the training corpus belonging to the same category as the target corpus during the training stage. Then the target corpus is input into the pre-trained model, and the accuracy of the output second text vector of the target corpus will be higher.

[0080] As an example, the pre-trained model may be a Bert model, or various variant models of Bert, or other models that can vectorize text, for example, the pre-trained model may also be a Doc2vec model, etc.

[0081] Taking the Bert model as an example, the pre-trained model may include a self-attention layer. Based on the attention mechanism in the self-attention layer, when determining the word vector of a single character in the input corpus, the word vectors of other characters can be combined, that is, the context features are combined. The obtained word vector is more accurate, and the obtained text vector expresses the text semantics more accurately. That is, the Bert model can obtain richer semantic features than other models, and thus obtain a more accurate text vector.

[0082] In an embodiment of the present application, each training corpus and target corpus is respectively input into a pre-trained model that has been trained. Since the pre-trained model is obtained based on training of the training set, the first text vector and the second text vector determined based on the pre-trained model are more accurate and can better reflect the semantics of the training corpus and the target corpus.

[0083] Step 206: Perform similarity determination on each first text vector and the second text vector.

[0084] That is, the similarity between each first text vector and the second text vector is determined.

[0085] In a possible implementation of the present application, the specific implementation of similarity judgment between each first text vector and the second text vector may include:

[0086] For a first text vector of any training corpus, similarity calculation is performed between the first text vector and the second text vector to determine the similarity between the training corpus and the target corpus.

[0087] In some embodiments, the similarity between the training corpus and the target corpus can be determined by calculating the cosine similarity, Euclidean distance, Mahalanobis distance, Hamming distance, etc. between the first text vector and the second text vector.

[0088] Taking the calculation of cosine similarity as an example, the cosine distance between the first text vector and the second text vector can be calculated, and the cosine distance is proportional to the cosine similarity. The larger the cosine distance, the greater the cosine similarity between the first text vector and the second text vector, that is, the higher the similarity between the training corpus corresponding to the first text vector and the target corpus corresponding to the second text vector, the greater the probability that the training corpus corresponding to the first text vector and the target corpus corresponding to the second text vector belong to the same category of corpus; the smaller the cosine distance, the smaller the cosine similarity between the first text vector and the second text vector, that is, the lower the similarity between the training corpus corresponding to the first text vector and the target corpus corresponding to the second text vector, the smaller the probability that the training corpus corresponding to the first text vector and the target corpus corresponding to the second text vector belong to the same category.

[0089] In an embodiment of the present application, the training corpus and the target corpus are converted into vector representations, and the similarity is calculated by using the first text vector and the second text vector to determine the similarity between the training corpus and the target corpus. This can obtain a more accurate similarity result, and because the similarity can be used to characterize the possibility that the training corpus and the target corpus belong to the same category of corpus, the determination of the similarity provides a basis for determining whether to add the target corpus to the training set to expand the training corpus.

[0090] Step 208: If the similarity judgment result meets the preset condition, a target category label is added to the target corpus according to the similarity judgment result, and the target corpus with the target category label is added to the training set to obtain an updated training set.

[0091] The preset condition is used to determine whether the degree of similarity between the training corpus and the target corpus is sufficient for the target corpus to be added to the training set.

[0092] As an example, for each training corpus, a similarity judgment result corresponding to it can be obtained by the above method, and the similarity judgment result can be the similarity between the training corpus and the target corpus. For example, the similarity can be a similarity score or a similarity level, which is not limited in the embodiment of the present application.

[0093] In some embodiments, the similarity determination result meeting a preset condition may include:

[0094] If there is a training corpus with a similarity greater than a preset threshold, determining that the similarity judgment result meets the preset comparison condition; or,

[0095] If there are training corpora with a similarity greater than a preset threshold, and the number of the training corpora is greater than a preset number, it is determined that the similarity judgment result meets the preset comparison condition; or,

[0096] If there are multiple training corpora with similarity greater than a preset threshold, and the similarity between the multiple training corpora is greater than the preset threshold, it is determined that the similarity judgment result meets the preset comparison condition; or,

[0097] If there are multiple training corpora with similarities greater than a preset threshold, and the number of the multiple training corpora is greater than a preset number, and the similarities between the multiple training corpora are greater than the preset threshold, it is determined that the similarity judgment result meets the preset comparison condition.

[0098] The preset threshold and the preset number can be set by the user according to actual needs, or can be set by the device by default, which is not limited in the present embodiment. For example, the preset threshold can be 95%, the preset number can be 10, and so on.

[0099] In the first example, as long as there is a training corpus with a similarity greater than a preset threshold, it means that there is a training corpus with a high degree of similarity to the target corpus in the training set, and the target corpus can be added to the training set, and it is determined that the similarity judgment result meets the preset comparison condition. If there is no training corpus with a similarity greater than the preset threshold, that is, the training corpus in the training set has a low degree of similarity to the target corpus, then there is no need to add the target corpus to the training set.

[0100] This method adds the target corpus to the training set when there is training corpus with a similarity greater than a preset threshold. This can add more target corpora with a high degree of similarity to the training corpus to the training set, fully enriching the training corpus.

[0101] In the second example, if there is a training corpus with a similarity greater than a preset threshold, it means that there may be a training corpus with a high degree of similarity to the target corpus in the training set. However, if the calculation result of the similarity is inaccurate, adding the target corpus to the training set may affect the quality of the training corpus in the training set, and even affect the model training. Therefore, it can be judged whether the number of training corpora is greater than the preset number. If so, it is considered that there are more training corpora in the training set with a high degree of similarity to the target corpus, and the target corpus can be used to train the model together with these training corpora. Therefore, the target corpus can be added to the training set, that is, it is determined that the similarity judgment result meets the preset comparison condition. If the number of training corpora is less than or equal to the preset number, it means that there are relatively few training corpora with a high degree of similarity to the target corpus in the training set, and the training corpus in the training set may not be of the same category as the target corpus. Therefore, the target corpus may not be added to the corpus, that is, it is determined that the similarity judgment result does not meet the preset comparison condition.

[0102] In this way, the similarity between the target corpus added to the training set and the training corpus can be improved, which not only enriches the training corpus, but also improves the quality of the training corpus in the training set, thereby improving the performance of the trained model.

[0103] In the third example, if there are multiple training corpora with similarities greater than a preset threshold, it means that there may be training corpora with a high degree of similarity to the target corpus in the training set, but there may also be a problem of inaccurate similarity calculation occasionally. In this case, the similarity between multiple training corpora with similarities to the target corpus greater than the preset threshold can be calculated. Because if the similarity between multiple training corpora is relatively high, and the similarities between the target corpus and these training corpora are all greater than the preset threshold, then it can be understood that the calculated similarities are relatively accurate, or the possibility of inaccurate similarity calculation in this case is relatively low. Therefore, it can be determined that the multiple training corpora are indeed training corpora with a high degree of similarity to the target corpus, that is, if it is determined that the similarity judgment result meets the preset comparison condition, then the target corpus can be added to the training set. However, if the similarities between multiple training corpora are relatively low, while the similarities between the target corpus and these training corpora are relatively high, this is rather unreasonable. Therefore, it can be considered that inaccurate calculations have occurred in these similarities, or that the possibility of inaccurate similarity calculations in this case is relatively high. Therefore, it can be considered that there are not many training corpora that are highly similar to the target corpus, and the target corpus will not be added to the training set, that is, it is determined that the similarity judgment result does not meet the preset comparison conditions.

[0104] This method not only considers the similarity between the target corpus and the training corpus, but also considers the similarity between the training corpora in order to avoid the problem of inaccurate similarity calculation. The target corpus is added to the training set only when the similarity between the training corpora is relatively high. The target corpus added to the training set is a corpus with a high degree of similarity to most of the training corpora in the training set. This not only expands the training set, but also improves the quality of the training corpus.

[0105] In the fourth example, if there are multiple training corpora with similarities greater than a preset threshold, it means that there may be training corpora with a high degree of similarity to the target corpus in the training set, but there may also be occasional problems with inaccurate similarity calculations. Therefore, it can be determined whether the number of multiple training corpora is greater than the preset number. If so, when it is determined that the similarities between the multiple training corpora are greater than the preset threshold, it can be considered that the similarities between the multiple training corpora are relatively high, and the possibility of inaccurate similarity calculations is relatively low. Therefore, it can be considered that the multiple training corpora are indeed training corpora with a high degree of similarity to the target corpus, and the target corpus can be added to the training set, that is, it is determined that the similarity judgment result meets the preset comparison conditions.

[0106] This method first determines whether the number of multiple training corpora with similarities greater than a preset threshold is sufficient, and then determines whether the similarities between the multiple training corpora are large enough. Only when the number is large and the similarities are large, it is determined that the preset comparison conditions are met, and the target corpus is added to the training set. This can not only expand the training set, but also improve the similarity between the training corpora in the expanded training set, making the expanded training set more conducive to model training, thereby improving the performance of the trained model.

[0107] In some embodiments, after determining that the similarity judgment result meets the preset conditions, it means that the target corpus can be added to the corpus set, but the target corpus does not carry a category label. The model training in this scheme is labeled training. Therefore, it is necessary to add a target category label to the target corpus based on the similarity judgment result.

[0108] As an example, if the training corpus included in the training set all belongs to the same category, the category label of any training corpus in the training set can be determined as the target category label, and the target category label can be added to the target corpus. For example, if the training set includes training corpus of resume type, the target category label "resume" can be added to the target corpus.

[0109] As another example, if the training set is training corpora of different categories, the category label of the training corpora with the greatest similarity among the training corpora whose similarity judgment results meet preset conditions can be determined as the target category label, and the target category label can be added to the target corpus.

[0110] Exemplarily, after adding a target category label to the target corpus, the target corpus can be added to a training set, and an updated training set can be obtained.

[0111] The training set construction method provided in the present application obtains a training set and a target corpus, wherein each training corpus in the training set carries a category label, and the target corpus does not carry a category label; each training corpus and the target corpus are respectively input into a pre-training model to obtain a first text vector of each training corpus and a second text vector of the target corpus, wherein the pre-training model is obtained by pre-training a preset natural language model based on the training set; a similarity judgment is performed on each first text vector and the second text vector; if the similarity judgment result meets a preset condition, a target category label is added to the target corpus according to the similarity judgment result, and the target corpus with the added target category label is added to the training set to obtain an updated training set. In the above method, since the pre-trained model is obtained by training with a training set, the first text vector and the second text vector determined based on the pre-trained model are more accurate. Based on the first text vector and the second text vector, a similarity comparison is performed on the training corpus and the target corpus, and the target category label is annotated for the target corpus according to the similarity comparison result, thereby achieving the effect of automatic labeling, eliminating the need for manual labeling, reducing the consumption of manpower and material resources, and increasing the speed of constructing the training set, thereby improving the training efficiency of the model. In addition, training the model based on an updated training set including more training corpora can improve the performance of the trained model.

[0112] Figure 3 is a flowchart of a training method for a pre-training model provided according to an embodiment of the present application, see Figure 3 The pre-trained model provided in the embodiment of the present application can be trained through the following steps.

[0113] Step 302: Perform character masking on each training corpus in the training set to obtain input corpus.

[0114] In implementation, before using the pre-trained model, it is necessary to first train the preset natural language processing model to obtain the pre-trained model. In an embodiment of the present application, the Mask LM task of the Bert model can be used to train the preset natural language processing model. The core idea of ​​this training method is to predict the input shielded part in some form based on other parts other than the shielded part. Therefore, before inputting the training corpus, some characters in the training corpus need to be shielded, that is, the training corpus needs to be subjected to character masking.

[0115] In some embodiments, character masking processing refers to masking one or some characters in a single training corpus. When the preset natural language model is input, the single training corpus includes unknown characters. The preset natural language model predicts the masked characters and adjusts the model parameters of the preset natural language model according to the loss between the predicted results and the actual results, so as to train the preset natural language model and obtain the effect of a pre-trained model.

[0116] It should be noted that model training is an iterative process. In the model application stage, the input corpus is complete, that is, the model application is vectorized based on the complete corpus. If the model training stage has always been based on incomplete training corpus, it may affect the accuracy of the text vector output when the model is applied. Therefore, the masked characters cannot be shielded throughout the entire training process of the model training, but can be shielded in some stages. Usually, for a training corpus, in the entire model training process, the masked characters can be shielded in 90% of the iterative training process, and the masked characters can be not shielded in 10% of the iterative training process, so that the model can learn more complete semantic features of the training corpus.

[0117] As an example, if the preset natural language processing model is the Bert model, 15% of the characters in the training corpus can be determined as masked characters; alternatively, masked characters can be selected from the training corpus according to user needs, which is not limited in the embodiments of the present application.

[0118] In the embodiment of the present application, by performing character masking processing on the training corpus, the model is enabled to determine the masked characters during the model training process, which can improve the performance of the model.

[0119] Step 304: Input the input corpus into a preset natural language model, determine the word vector corresponding to the position of the masked character in each input corpus, and obtain multiple word vectors.

[0120] In some embodiments, the preset natural language model can be a Bert model, which includes at least a self-attention layer. Through the self-attention layer, the word vector corresponding to the position of the masked character can be determined based on the word vectors of non-masked characters in the input corpus and the association relationship between non-masked characters.

[0121] As an example, for any input corpus, the influence weight of each non-masked character in the input corpus on the masked character can be determined based on the self-attention layer, and the word vector of the masked character can be determined based on the word vector and influence weight of each non-masked character.

[0122] Step 306: Based on the multiple word vectors and the preset word vector table, determine the predicted character at the position of each masked character.

[0123] In some embodiments, similarity calculations can be performed between multiple word vectors and word vectors in a preset word vector table to determine word vectors in the preset word vector table that have a high degree of similarity to each word vector, and the preset characters corresponding to the word vectors with a high degree of similarity are determined as the predicted characters at the position of the masked character.

[0124] Step 308: Determine a loss value based on the plurality of predicted characters and the plurality of masked characters.

[0125] In one or more embodiments of the present application, the loss value can be determined based on multiple predicted characters and multiple masked characters through methods such as a cross entropy loss function and a logarithmic loss function.

[0126] As an example, a loss value can be determined through a loss function based on each predicted character and its corresponding masked character, so that multiple loss values ​​can be obtained, and the average of the multiple loss values ​​is determined as the loss value that can be used as a basis for adjusting model parameters.

[0127] Step 310: Adjust the parameters of the preset natural language model based on the loss value to obtain a pre-trained model.

[0128] In some embodiments, if the loss value is greater than the loss threshold, the parameters of the preset natural language model can be adjusted according to the loss value, and then the preset natural language model can be trained based on the training set, and the loss value can be re-determined until the determined loss value is less than or equal to the loss threshold. It can be determined that the model training is completed, and the model parameters at this time are determined as the parameters of the pre-trained model, and the preset natural language model at this time is determined as the pre-trained model.

[0129] In this way, when it is determined that the loss value is less than or equal to the loss threshold, it can be considered that the result determined based on the current parameters of the model is relatively accurate, so the training of the preset natural language model can be stopped.

[0130] In other embodiments, if the loss value is greater than the loss threshold, the parameters of the preset natural language model can be adjusted according to the loss value, and the number of iterative training is recorded as 1, and then the preset natural language model is continued to be trained based on the training set, and the loss value is re-determined. When the loss value is not less than the loss threshold, the parameters of the preset natural language model are adjusted according to the loss value, and the number of iterative training is recorded plus 1, and then the preset natural language model is continued to be trained based on the training set until the number of iterative training is greater than or equal to the preset number threshold. It can be considered that the training of the preset natural language model has been completed, and the training of the preset natural language model is stopped, and the preset natural language model at this time is determined as the pre-trained model.

[0131] In this way, when the number of iterative training reaches a certain number, it can be considered that the parameters of the model will basically not change anymore, and continued training will not help the performance of the model much, so the training of the preset natural language model can be stopped.

[0132] In the embodiment of the present application, the preset natural language model is trained using the Mask LM task of Bert, and the word vector corresponding to the position of the masked character is similarly calculated with the word vector (embedding) in the preset word vector table to obtain the predicted character for the masked character, and then the loss value is determined according to the masked character and the predicted character, and the model parameters of the preset natural language model are adjusted according to the loss value to obtain the pre-trained model. By shielding the masked characters in the training corpus and allowing the preset natural language model to predict the masked characters, the ability of the predictive natural language model to determine the word vector can be improved, that is, the ability of the pre-trained model obtained by training to vectorize the text can be improved, so that the text vector obtained based on the pre-trained model obtained by training is more accurate.

[0133] Figure 4 is a flowchart of a character mask processing method provided according to an embodiment of the present application, see Figure 4 The specific implementation of performing character masking processing on each training corpus in the training set, i.e. step 302, may include the following steps.

[0134] Step 3022: For any training corpus, determine any character in the training corpus as a masked character.

[0135] In some embodiments, taking a training corpus as an example, characters can be randomly selected from the training corpus as masked characters, and the number of masked characters can be one or more. Alternatively, the proportion of masked characters in the training corpus can be set, and then characters are randomly selected as masked characters according to the proportion.

[0136] For example, assuming that the training corpus is my dog ​​is hai ry, the last word hai ry can be determined as the masked character.

[0137] Step 3024: Replace the masked characters with preset mask characters.

[0138] The preset mask character is a character used to replace the masked character.

[0139] As an example, the preset mask character may be a random character, such as a randomly selected word, text, letter, etc., or may be a mask symbol, such as the symbol [masked].

[0140] For example, assuming that the training corpus is my dog ​​is hairy, the last word hairy can be determined as a masked character, and the masked character can be replaced with a mask symbol [masked], and the obtained input corpus is mydog is[masked]; or, the masked character can be replaced with a random character, and the obtained input corpus can be my dog ​​is[random word]. For example, if the random character is Sam, the obtained input corpus can be my dog ​​isSam.

[0141] In an embodiment of the present application, the masked characters in the training corpus are masked by replacing the characters in the training corpus with preset mask characters. Since the semantics of the training corpus can usually be inferred from some characters in the training corpus, the masked characters that have little impact on the semantics are masked. Allowing the model to predict the masked characters can enhance the model's ability to extract semantic features of the training corpus, thereby improving the model's ability to perform text vectorization processing on the input text.

[0142] It should be noted that during the entire model training process, 15% of the characters can be determined as masked characters, that is, 15% of the characters are masked, and on this basis, the masked characters can be replaced with masking symbols in 80% of the iterative training process, the masked characters are not masked in 10% of the iterative training process, and the masked characters are replaced with random characters in 10% of the iterative training process. In this way, the model can learn the semantic features of the complete training corpus and predict the masked characters, which not only improves the model's ability to extract semantic features, but also improves the model's ability to predict masked characters based on semantics.

[0143] Figure 5 is a flowchart of a method for determining a predicted character according to an embodiment of the present application, see Figure 5 The preset word vector table may include multiple preset characters and the word vector of each preset character. On this basis, based on the multiple word vectors and the preset word vector table, the specific implementation of determining the predicted character at the position of each masked character, that is, step 306, may include the following steps.

[0144] Step 3062: For any masked character, the word vector corresponding to the position of the masked character is calculated similarity with the word vector in the preset word vector table to obtain the similarity between each preset character and the masked character.

[0145] In some embodiments, the similarity between characters corresponding to two word vectors can be determined by calculating the cosine similarity, Euclidean distance, Mahalanobis distance, Hamming distance, etc. between the two word vectors.

[0146] As an example, for the first masked character, the cosine similarity between the word vector corresponding to the position of the first masked character and the word vector in the preset word vector table can be determined, and the cosine similarity between each preset character and the first masked character can be obtained. For example, the preset word vector table includes N preset characters, and the similarity between the word vector of the first masked character and the word vector of the N preset characters can be calculated respectively, and N similarities can be obtained. If the training set includes M masked characters, each masked character is calculated in the above manner, and M×N similarities can be obtained.

[0147] Step 3064: Determine a preset character in the preset word vector table whose similarity is greater than a preset threshold as the predicted character at the position where the masked character is located.

[0148] The preset threshold value may be set by the user according to actual needs, or may be set by the device by default, which is not limited in the present embodiment. For example, the preset threshold value may be 95%.

[0149] In some embodiments, for the first masked character, the similarity between the first masked character and each preset character in the preset word vector table is determined through step 3062. If there is a similarity greater than a preset threshold among these similarities, and the number of similarities greater than the preset threshold is one, then the preset character corresponding to the similarity greater than the preset threshold is determined as the predicted character at the position of the masked character; if the number of similarities greater than the preset threshold is multiple, then the preset character corresponding to the maximum similarity among the multiple similarities greater than the preset threshold is determined as the predicted character at the position of the masked character.

[0150] Exemplarily, assuming that the first masked character is A, the preset word vector table includes preset characters E, F, R, and T, the similarity between the first masked character A and the preset character E is 90%, the similarity between the first masked character A and the preset character F is 97%, the similarity between the first masked character A and the preset character R is 96%, and the similarity between the first masked character A and the preset character T is 99%. Assuming that the preset threshold is 95%, it can be determined that there are three similarities greater than the preset threshold, that is, the preset characters F, R, and T are all characters whose similarity with the first masked character is greater than the preset threshold. Since the similarity between the preset character T and the first masked character A is the largest among the three similarities, the preset character T can be determined as the predicted character at the position where the first masked character A is located.

[0151] In an embodiment of the present application, by calculating the similarity between the word vector at the position of the masked character and the word vector in a preset word vector table, the original character at the position of the masked character (i.e., the predicted character) can be predicted, and the masked character can be predicted by the non-masked character. Since the prediction is based on the word vector determined by the preset natural language processing model, the higher the accuracy of the word vector, the higher the accuracy of the predicted character. The predicted character can be used to adjust the parameters of the model to achieve the effect of model training.

[0152] Figure 6 is a flowchart of a method for determining a loss value according to an embodiment of the present application, see Figure 6 The specific implementation of determining the loss value based on the multiple predicted characters and the multiple masked characters, that is, step 308, may include the following steps:

[0153] Step 3082: Determine a sub-loss value based on each masked character and the predicted character at the position of the masked character to obtain multiple sub-loss values.

[0154] As an example, for any masked character, the masked character and the predicted character at the position of the masked character can be input into the loss function, and then a sub-loss value can be obtained. After the above method is used for each masked character and predicted character, multiple sub-loss values ​​can be obtained.

[0155] Exemplarily, the loss function may be a function capable of determining a loss value, such as a cross entropy loss function, a logarithmic loss function, or an exponential loss function.

[0156] Step 3084: Determine a loss value based on multiple sub-loss values.

[0157] In some embodiments, the sum of multiple sub-loss values ​​can be determined as the loss value; or, the average of the sum of multiple sub-loss values ​​can be determined as the loss value; or, the loss value can be determined based on the weight corresponding to each sub-loss value and multiple sub-loss values.

[0158] As an example, for each sub-loss value, the weight of each sub-loss value is multiplied by the sub-loss value to obtain multiple products, and the sum of the multiple products can be determined as the loss value; alternatively, the average of the sum of the multiple products can be determined as the loss value.

[0159] As an example, the weight corresponding to each sub-loss value can be determined according to the similarity between the predicted character and the masked character corresponding to the sub-loss value, and the higher the similarity, the greater the weight. Exemplarily, the similarities between the predicted character and the masked character corresponding to multiple sub-loss values ​​can be added to obtain the total similarity, and then the proportion of each similarity in the total similarity is determined, and the proportion corresponding to each similarity is determined as the sub-loss value of the predicted character and the masked character corresponding to the similarity.

[0160] For example, suppose there are three masked characters E, F, and T, and the predicted characters at the positions of the three masked characters are A, B, and C, respectively. The similarity 1 between the masked character E and the predicted character A is 98%, the similarity 2 between the masked character F and the predicted character B is 99%, and the similarity 3 between the masked character T and the predicted character C is 98%. The three similarities are added together to obtain a total similarity of 295%, and it is determined that the proportions of similarity 1 and similarity 3 in the total similarity are both 0.33, and the proportion of similarity 2 in the total similarity is 0.34. Therefore, it can be determined that the weight corresponding to the sub-loss value of the masked character E and the predicted character A is 0.33, the weight corresponding to the sub-loss value of the masked character F and the predicted character B is 0.34, and the weight corresponding to the sub-loss value of the masked character T and the predicted character C is 0.33.

[0161] In an embodiment of the present application, a sub-loss value can be determined based on each masked character and its corresponding predicted character, and the loss value can be determined by summing, averaging, weighted summing or weighted averaging the sub-loss values. A more accurate loss value can be determined, and the model parameters can be adjusted based on the highly accurate loss value. The model parameters can be quickly adjusted to appropriate parameters, thereby improving the efficiency of model training.

[0162] Figure 7 is a flowchart of a similarity determination method provided according to an embodiment of the present application, see Figure 7 For the first text vector of any training corpus, similarity calculation is performed between the first text vector and the second text vector to determine the similarity between the training corpus and the target corpus. The specific implementation of step 206 may include the following steps.

[0163] Step 2062: For a first text vector of any training corpus, based on the category vector in the first text vector and the category vector in the second text vector, determine the category similarity corresponding to the training corpus.

[0164] In an embodiment of the present application, a text vector may include a category vector and a word vector.

[0165] In some embodiments, taking the pre-trained model as a Bert model, the Bert model includes at least a self-attention layer, based on which the category vector (CLS vector) of the input training corpus can be determined, and the word vector of each character in the training corpus can be determined, and the word vector of the training corpus can be determined based on the word vectors of multiple characters in the training corpus, and the category vector and the word vector are combined to obtain the first text vector of the training corpus. Similarly, the second text vector of the target corpus can be determined.

[0166] As an example, the sum of the word vectors of multiple characters in the training corpus can be determined as the word vector of the training corpus; or, the average of the sum of the word vectors of multiple characters in the training corpus can be determined as the word vector of the training corpus; or, the influence weight of each character relative to the training corpus can be determined based on the self-attention layer, and the word vector of the training corpus can be obtained by performing weighted summation based on the influence weight and word vector of each character.

[0167] In an optional embodiment of the present application, cosine similarity calculation may be performed based on the category vector in the first text vector and the category vector in the second text vector of any training corpus to obtain the category similarity corresponding to the training corpus.

[0168] Step 2064: Based on the word vectors in the first text vector and the word vectors in the second text vector, determine the semantic similarity corresponding to the training corpus.

[0169] In an optional embodiment of the present application, cosine similarity calculation can be performed based on word vectors in the first text vector and word vectors in the second text vector of any training corpus to obtain the semantic similarity corresponding to the training corpus.

[0170] Step 2066: Based on the category similarity and semantic similarity corresponding to the training corpus, determine the similarity between the training corpus and the target corpus.

[0171] In some embodiments, the sum of the category similarity and the semantic similarity can be determined as the similarity between the training corpus and the target corpus; alternatively, preset weights for the category similarity and the semantic similarity can be preset respectively, and the category similarity and the semantic similarity can be weightedly summed based on the preset weights to obtain the similarity between the training corpus and the target corpus; alternatively, the mean of the sum of the category similarity and the semantic similarity can be determined as the similarity between the training corpus and the target corpus; alternatively, the mean of the weighted sum of the category similarity and the semantic similarity can be determined as the similarity between the training corpus and the target corpus.

[0172] In an embodiment of the present application, the similarity between the first text vector and the second text vector is calculated through the two dimensions of category dimension and semantic dimension, so as to realize the similarity judgment between the training corpus and the target corpus from two dimensions, which can improve the accuracy of the similarity judgment result and further improve the accuracy of the target category label annotated for the target corpus.

[0173] Figure 8 is a flowchart of a method for adding target category labels to target corpus according to an embodiment of the present application, see Figure 8 According to the similarity judgment result, the specific implementation of adding the target category label to the target corpus, i.e. step 208, may include:

[0174] Step 2082: If there are multiple training corpora whose similarity judgment results meet the preset conditions, count the number of training corpora corresponding to various category labels in the multiple training corpora.

[0175] In some embodiments, if there are multiple training corpora whose similarity judgment results meet the preset conditions, and the category labels of the multiple training corpora are different, the types of category labels corresponding to the multiple training corpora are first determined, and then the number of training corpora corresponding to each category label is counted respectively.

[0176] Exemplarily, the number of training corpora whose similarity judgment results meet the preset conditions is N, and based on the category labels corresponding to the N training corpora, it can be determined that the N training corpora can be divided into three categories according to category: resume category, contract category and paper category. Then, the number of training corpora corresponding to the resume category, the number of training corpora corresponding to the contract category, and the number of training corpora corresponding to the paper category can be determined.

[0177] Step 2084: Based on the number of training corpora corresponding to various category labels, determine a target category label from the various category labels and add it to the target corpus.

[0178] In some embodiments, the category label with the largest number among the training corpora corresponding to various category labels may be determined as the target category label, and the target category label may be added to the target corpus.

[0179] Continuing with the above example, assuming that the number of training corpora corresponding to the resume category is 5, the number of training corpora corresponding to the contract category is 3, and the number of training corpora corresponding to the paper category is 2, it can be determined that the target category label of the target corpus is the resume category.

[0180] In an embodiment of the present application, when there are multiple training corpora whose similarity judgment results meet the preset conditions, the number of training corpora corresponding to each category label can be counted, and the category label with the largest number of training corpora can be determined as the target category label of the target corpus. The category label that best matches the category of the target corpus can be determined, thereby improving the accuracy of the target category label added to the target corpus.

[0181] The following combination Fig. 9 Taking the application of the training set construction method provided in this application in the classification model as an example, the training set construction method is further explained. Fig. 9 It is a processing flow chart of a training set construction method for a classification model provided in accordance with an embodiment of the present application, which specifically includes the following steps:

[0182] Step 902: Obtain a training set and a target corpus, wherein each training corpus in the training set carries a category label, and the target corpus does not carry a category label.

[0183] Step 904: pre-training a preset natural language model based on the training set to obtain a pre-trained model.

[0184] Step 906: Input each training corpus and target corpus into the pre-training model respectively to obtain the first text vector of each training corpus and the second text vector of the target corpus.

[0185] The first text vector includes a category vector and a word vector, and the second text vector includes a category vector and a word vector.

[0186] Step 908: For a first text vector of any training corpus, based on the category vector in the first text vector and the category vector in the second text vector, determine the category similarity corresponding to the training corpus.

[0187] Step 910: Determine the semantic similarity corresponding to the training corpus based on the word vectors in the first text vector and the word vectors in the second text vector.

[0188] Step 912: Based on the category similarity and semantic similarity corresponding to the training corpus, determine the similarity between the training corpus and the target corpus.

[0189] Exemplarily, the similarity can be understood as a similarity judgment result.

[0190] Step 914: If there are multiple training corpora whose similarity judgment results meet the preset conditions, count the number of training corpora corresponding to various category labels in the multiple training corpora.

[0191] Step 916: Based on the amount of training corpora corresponding to various category labels, determine a target category label from the various category labels and add it to the target corpus.

[0192] Step 918: If the number of training corpora that meet the preset condition in the similarity judgment result is one, the category label of the training corpus is added to the target corpus as the target category label.

[0193] Step 920: Add the target corpus with the target category label added to the training set to obtain an updated training set.

[0194] The training set construction method provided in the present application obtains a training set and a target corpus, wherein each training corpus in the training set carries a category label, and the target corpus does not carry a category label; each training corpus and the target corpus are respectively input into a pre-training model to obtain a first text vector of each training corpus and a second text vector of the target corpus, wherein the pre-training model is obtained by pre-training a preset natural language model based on the training set; a similarity judgment is performed on each first text vector and the second text vector; if the similarity judgment result meets a preset condition, a target category label is added to the target corpus according to the similarity judgment result, and the target corpus with the added target category label is added to the training set to obtain an updated training set. In the above method, since the pre-trained model is obtained by training with a training set, the first text vector and the second text vector determined based on the pre-trained model are more accurate. Based on the first text vector and the second text vector, a similarity comparison is performed on the training corpus and the target corpus, and the target category label is annotated for the target corpus according to the similarity comparison result, thereby achieving the effect of automatic labeling, eliminating the need for manual labeling, reducing the consumption of manpower and material resources, and increasing the speed of constructing the training set, thereby improving the training efficiency of the model. In addition, training the model based on an updated training set including more training corpora can improve the performance of the trained model.

[0195] Fig.10 is a flow chart of a classification model training method provided according to an embodiment of the present application. Fig.10 ,The classification model training method includes the following steps.

[0196] Step 1002: Obtain a training set, where the training set includes multiple training corpora, each of which carries a category label.

[0197] The training set is obtained by the training set construction method described in the above embodiment.

[0198] As an example, the training corpus may belong to the same category or different categories.

[0199] Step 1004: Input multiple training corpora into the classification model to obtain the predicted category of each training corpus.

[0200] Among them, the classification model can be used to process text classification tasks to determine the category of the input text.

[0201] As an example, the classification model may be a Bert model, and the classification model may include an embedding layer, an attention layer, and a fully connected layer. After a plurality of training corpora are input into the classification model, each training corpus is subjected to word embedding processing through the embedding layer to obtain an embedding vector of each training corpus, and then the word embedding vectors of the plurality of training corpora are input into the attention layer for attention calculation to obtain an attention vector of each training corpus, and then the attention vector of each training corpus is input into the fully connected layer to obtain a plurality of prediction probabilities corresponding to each training corpus, the number of the plurality of prediction probabilities is the same as the number of categories to which the training corpus may belong, and based on the plurality of prediction probabilities, the prediction category of the training corpus may be determined.

[0202] Step 1006: Determine a loss value based on the predicted category and category label of each training corpus.

[0203] In some embodiments, the predicted category and category label of each training corpus can be used as input to the loss function, and the output of the loss function is the loss value. As an example, for any training corpus, the predicted category and category label of the training corpus can be input into the loss function to obtain a sub-loss value. After the above method is used for each training corpus, multiple sub-loss values ​​can be obtained.

[0204] In some embodiments, the sum of multiple sub-loss values ​​can be determined as the loss value; or, the average of the sum of multiple sub-loss values ​​can be determined as the loss value; or, the loss value can be determined based on the weight corresponding to each sub-loss value and multiple sub-loss values.

[0205] As an example, for each sub-loss value, the weight of each sub-loss value is multiplied by the sub-loss value to obtain multiple products, and the sum of the multiple products can be determined as the loss value; alternatively, the average of the sum of the multiple products can be determined as the loss value.

[0206] As an example, the weight corresponding to each sub-loss value can be determined according to the similarity between the predicted character and the masked character corresponding to the sub-loss value, and the higher the similarity, the greater the weight. Exemplarily, the similarities between the predicted character and the masked character corresponding to multiple sub-loss values ​​can be added to obtain the total similarity, and then the proportion of each similarity in the total similarity is determined, and the proportion corresponding to each similarity is determined as the sub-loss value of the predicted character and the masked character corresponding to the similarity.

[0207] Step 1008: Train the classification model based on the loss value until a training stop condition is reached.

[0208] In some embodiments, if the loss value is greater than the loss threshold, the parameters of the classification model can be adjusted according to the loss value, and then the classification model can be trained based on the training set, and the loss value can be re-determined until the determined loss value is less than or equal to the loss threshold. It can be determined that the model training is completed, and the model parameters at this time are determined as the parameters of the classification model to obtain a trained classification model. In this way, when it is determined that the loss value is less than or equal to the loss threshold, it can be considered that the result determined based on the current parameters of the model is relatively accurate, so the training of the classification model can be stopped.

[0209] In other embodiments, if the loss value is greater than the loss threshold, the parameters of the classification model can be adjusted according to the loss value, and the number of iterative training is recorded as 1, and then the classification model is continued to be trained based on the training set, and the loss value is re-determined. When the loss value is not less than the loss threshold, the parameters of the classification model are adjusted according to the loss value, and the number of iterative training is recorded plus 1, and then the classification model is continued to be trained based on the training set until the number of iterative training is greater than or equal to the preset number threshold. It can be considered that the training of the classification model has been completed, and the training of the classification model is stopped to obtain a trained classification model. In this way, when the number of iterative training reaches a certain number, it can be considered that the parameters of the model will basically not change, and continuing training will not help the performance of the model much, so the training of the classification model can be stopped.

[0210] The classification model training method provided in the embodiment of the present application obtains a training set, the training set includes multiple training corpora, each training corpus carries a category label, the training set is obtained by the training set construction method described in the above embodiment, multiple training corpora are input into the classification model, the predicted category of each training corpus is obtained, the loss value is determined based on the predicted category and category label of each training corpus, and the classification model is trained based on the loss value until the training stop condition is reached. In the classification model training method, since the training set used by the training model is obtained by the above training set construction method and includes more training corpora, training the classification model based on the training set can improve the classification accuracy of the classification model obtained by training.

[0211] Fig.11 is a flowchart of a document classification method provided according to an embodiment of the present application. Fig.11 ,The document classification method includes the following steps.

[0212] Step 1102: Obtain documents to be classified.

[0213] Step 1104: Input the document to be classified into the classification model to determine the category of the document to be classified.

[0214] Wherein, the classification model is obtained through the classification model training method described in the above embodiment.

[0215] In some embodiments, the classification model may be a Bert model, and the classification model may include an embedding layer, an attention layer, and a fully connected layer. The document to be classified is input into the classification model, and the document to be classified is word-embedded through the embedding layer to obtain an embedding vector of the document to be classified, and then the word embedding vector of the document to be classified is input into the attention layer for attention calculation to obtain an attention vector of the document to be classified, and then the attention vector of the document to be classified is input into the fully connected layer to obtain multiple prediction probabilities corresponding to the document to be classified, the number of the multiple prediction probabilities is the same as the number of categories to which the document to be classified may belong, and based on the multiple prediction probabilities, the predicted category of the document to be classified can be determined.

[0216] For example, the category corresponding to the maximum prediction probability may be determined as the predicted category of the document to be classified.

[0217] The document classification method provided in the present application obtains the document to be classified, inputs the document to be classified into a classification model, and determines the category of the document to be classified. Since the classification model used is trained by the above-mentioned classification model training method, the classification accuracy of the classification model is higher, and the category of the document to be classified is more accurate.

[0218] Corresponding to the above training set construction method embodiment, the present application also provides a training set construction device embodiment, Fig.12 is a schematic diagram of the structure of a training set construction device provided according to an embodiment of the present application. Fig.12 As shown, the device comprises:

[0219] A first acquisition module 1202 is configured to acquire a training set and a target corpus, wherein each training corpus in the training set carries a category label, and the target corpus does not carry a category label;

[0220] The text vector determination module 1204 is configured to input the training corpus and the target corpus into a pre-training model to obtain a first text vector of the training corpus and a second text vector of the target corpus, wherein the pre-training model is obtained by pre-training a preset natural language model based on the training set;

[0221] A similarity determination module 1206 is configured to perform similarity determination on each of the first text vectors and the second text vectors;

[0222] The adding module 1208 is configured to add a target category label to the target corpus according to the similarity judgment result if the similarity judgment result meets a preset condition, and add the target corpus with the target category label added to the training set to obtain an updated training set.

[0223] In a possible implementation of the present application, the similarity determination module 1206 is further configured to:

[0224] For a first text vector of any training corpus, similarity calculation is performed between the first text vector and the second text vector to determine the similarity between the training corpus and the target corpus.

[0225] In a possible implementation of the present application, the adding module 1208 is configured to:

[0226] If there are multiple training corpora whose similarity judgment results meet the preset conditions, count the number of training corpora corresponding to various category labels in the multiple training corpora;

[0227] Based on the amount of training corpora corresponding to various category labels, a target category label is determined from the various category labels and added to the target corpus.

[0228] In a possible implementation of the present application, the text vector includes a category vector and a word vector;

[0229] The similarity determination module 1206 is further configured to:

[0230] For a first text vector of any training corpus, based on a category vector in the first text vector and a category vector in the second text vector, determining a category similarity corresponding to the training corpus;

[0231] Determining the semantic similarity corresponding to the training corpus based on the word vector in the first text vector and the word vector in the second text vector;

[0232] Based on the category similarity and semantic similarity corresponding to the training corpus, the similarity between the training corpus and the target corpus is determined.

[0233] In a possible implementation of the present application, the apparatus further includes a training module configured to:

[0234] Performing character masking processing on each training corpus in the training set to obtain input corpus;

[0235] Inputting the input corpus into the preset natural language model, determining the word vector corresponding to the position of the masked character in each input corpus, and obtaining multiple word vectors;

[0236] Determine a predicted character at the position of each masked character based on the multiple word vectors and the preset word vector table;

[0237] determining a loss value based on the plurality of predicted characters and the plurality of masked characters;

[0238] The parameters of the preset natural language model are adjusted based on the loss value to obtain the pre-trained model.

[0239] In a possible implementation of the present application, the training module is further configured as follows:

[0240] For any training corpus, determining any character in the training corpus as a masked character;

[0241] The masked characters are replaced with preset mask characters.

[0242] In a possible implementation of the present application, the preset word vector table includes a plurality of preset characters and a word vector of each preset character; and the training module is configured to:

[0243] For any masked character, the word vector corresponding to the position of the masked character is calculated similarity with the word vector in the preset word vector table to obtain the similarity between each preset character and the masked character;

[0244] The preset characters in the preset word vector table whose similarity is greater than a preset threshold are determined as the predicted characters at the position where the masked character is located.

[0245] In a possible implementation of the present application, the training module is configured as follows:

[0246] Determine a sub-loss value based on each masked character and a predicted character at the position of the masked character to obtain a plurality of sub-loss values;

[0247] The loss value is determined based on the plurality of sub-loss values.

[0248] The training set construction device provided by the present application obtains a training set and a target corpus, wherein each training corpus in the training set carries a category label, and the target corpus does not carry a category label; each training corpus and the target corpus are respectively input into a pre-training model to obtain a first text vector of each training corpus and a second text vector of the target corpus. Since the pre-training model is obtained by training the training set, the first text vector and the second text vector determined based on the pre-training model are more accurate. Then, based on the first text vector and the second text vector, a similarity comparison is performed on the training corpus and the target corpus, and a target category label is labeled for the target corpus according to the similarity comparison result, thereby achieving the effect of automatic labeling, without the need for manual labeling, reducing the consumption of manpower and material resources, and increasing the speed of constructing the training set, thereby improving the training efficiency of the model. In addition, training the model based on an updated training set including more training corpora can improve the performance of the trained model.

[0249] The above is a schematic scheme of a training set construction device of this embodiment. It should be noted that the technical scheme of the training set construction device and the technical scheme of the training set construction method described above belong to the same concept, and the details not described in detail in the technical scheme of the training set construction device can be found in the description of the technical scheme of the training set construction method described above.

[0250] Corresponding to the above classification model training method embodiment, the present application also provides a classification model training device embodiment, Fig.13 Schematic diagram of a classification model training device provided according to an embodiment of the present application. Fig.13 As shown, the device comprises:

[0251] The second acquisition module 1302 is configured to acquire a training set, wherein the training set includes a plurality of training corpora, each training corpus carries a category label, wherein the training set is obtained by the training set construction method described above;

[0252] The prediction module 1304 is configured to input the plurality of training corpora into a classification model to obtain a predicted category of each training corpus;

[0253] A first determination module 1306 is configured to determine a loss value based on the predicted category and category label of each training corpus;

[0254] The training module 1308 is configured to train the classification model based on the loss value until a training stop condition is reached.

[0255] The classification model training device provided in the embodiment of the present application inputs multiple training corpora into the classification model to obtain the predicted category of each training corpus; determines the loss value based on the predicted category and category label of each training corpus; and trains the classification model based on the loss value until the training stop condition is reached. Since the training set used by the training model is obtained through the above-mentioned training set construction method, including more training corpora, training the classification model based on the training set can improve the classification accuracy of the trained classification model.

[0256] The above is a schematic scheme of a classification model training device of this embodiment. It should be noted that the technical scheme of the classification model training device and the technical scheme of the classification model training method described above belong to the same concept, and the details not described in detail in the technical scheme of the classification model training device can be found in the description of the technical scheme of the classification model training method described above.

[0257] Corresponding to the above document classification method embodiment, the present application also provides a document classification device embodiment, Fig.14 Schematic diagram of a document classification device according to an embodiment of the present application. Fig.14 As shown, the device comprises:

[0258] The third acquisition module 1402 is configured to acquire a document to be classified;

[0259] The second determination module 1404 is configured to input the document to be classified into a classification model to determine the category of the document to be classified, wherein the classification model is obtained by the classification model training method described above.

[0260] The document classification method provided in the present application obtains the document to be classified, inputs the document to be classified into a classification model, and determines the category of the document to be classified. Since the classification model used is trained by the above-mentioned classification model training method, the classification accuracy of the classification model is higher, and the determined category of the document to be classified is more accurate.

[0261] The above is a schematic scheme of a document classification device of this embodiment. It should be noted that the technical scheme of the document classification device and the technical scheme of the document classification method described above are of the same concept, and the details not described in detail in the technical scheme of the document classification device can be found in the description of the technical scheme of the document classification method described above.

[0262] In addition, the components in the device embodiment should be understood as the functional modules that must be established to implement the steps of the program flow or the steps of the method, and the functional modules are not actually functionally divided or separated. The device claim defined by such a group of functional modules should be understood as the functional module architecture of the solution mainly implemented by the computer program recorded in the specification, and should not be understood as the physical device that mainly implements the solution by hardware.

[0263] Fig.15 15 is a block diagram of a computing device according to an embodiment of the present application. The components of the computing device 1500 include but are not limited to a memory 1510 and a processor 1520. The processor 1520 is connected to the memory 1510 via a bus 1530, and the database 1550 is used to store data.

[0264] The computing device 1500 also includes an access device 1540 that enables the computing device 1500 to communicate via one or more networks 1560. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1540 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0265] In one embodiment of the present application, the above components of the computing device 1500 and Fig.15 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig.15 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of the present application. Those skilled in the art may add or replace other components as needed.

[0266] Computing device 1500 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. Computing device 1500 may also be a mobile or stationary server.

[0267] The processor 1520 is used to execute computer executable instructions for the training set construction, classification model training and document classification method.

[0268] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical schemes of the training set construction, classification model training and document classification method mentioned above belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical schemes of the training set construction, classification model training and document classification method mentioned above.

[0269] An embodiment of the present application also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, are used for training set construction, classification model training and document classification methods.

[0270] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the training set construction, classification model training and document classification method mentioned above belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the training set construction, classification model training and document classification method mentioned above.

[0271] An embodiment of the present application also provides a chip, which stores a computer program, and when the computer program is executed by the chip, the steps of training set construction, classification model training and document classification method are implemented.

[0272] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0273] The computer instructions include computer program codes, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0274] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0275] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0276] The preferred embodiments of the present application disclosed above are only used to help explain the present application. The optional embodiments do not describe all the details in detail, nor do they limit the application to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the present application. The present application selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present application, so that those skilled in the art can understand and use the present application well. The present application is only limited by the claims and their full scope and equivalents.

Claims

1. A training set construction method, characterized in that: include: Obtaining a training set and a target corpus, wherein each training corpus in the training set carries a category label, and the target corpus does not carry a category label; Inputting each training corpus and the target corpus into a pre-training model respectively to obtain a first text vector of each training corpus and a second text vector of the target corpus, wherein the pre-training model is obtained by pre-training a preset natural language model based on the training set; Performing similarity judgment on each of the first text vectors and the second text vector; If the similarity judgment result meets the preset condition, a target category label is added to the target corpus according to the similarity judgment result, and the target corpus with the target category label is added to the training set to obtain an updated training set.

2. The method according to claim 1, characterized in that Performing similarity judgment on each of the first text vectors and the second text vector includes: For a first text vector of any training corpus, similarity calculation is performed between the first text vector and the second text vector to determine the similarity between the training corpus and the target corpus.

3. The method according to claim 1 or 2, characterized in that According to the similarity judgment result, adding a target category label to the target corpus includes: If there are multiple training corpora whose similarity judgment results meet the preset conditions, count the number of training corpora corresponding to various category labels in the multiple training corpora; Based on the amount of training corpora corresponding to various category labels, a target category label is determined from the various category labels and added to the target corpus.

4. The method according to claim 2, characterized in that The text vector includes a category vector and a word vector; for a first text vector of any training corpus, similarity calculation is performed on the first text vector and the second text vector to determine the similarity between the training corpus and the target corpus, including: For a first text vector of any training corpus, based on a category vector in the first text vector and a category vector in the second text vector, determining a category similarity corresponding to the training corpus; Determining the semantic similarity corresponding to the training corpus based on the word vector in the first text vector and the word vector in the second text vector; Based on the category similarity and semantic similarity corresponding to the training corpus, the similarity between the training corpus and the target corpus is determined.

5. The method according to claim 1, 2 or 4, characterized in that: The pre-trained model is trained in the following way: Performing character masking processing on each training corpus in the training set to obtain input corpus; Inputting the input corpus into the preset natural language model, determining the word vector corresponding to the position of the masked character in each input corpus, and obtaining multiple word vectors; Determine a predicted character at the position of each masked character based on the multiple word vectors and the preset word vector table; determining a loss value based on the plurality of predicted characters and the plurality of masked characters; The parameters of the preset natural language model are adjusted based on the loss value to obtain the pre-trained model.

6. The method according to claim 5, characterized in that Performing character masking processing on each training corpus in the training set includes: For any training corpus, determining any character in the training corpus as a masked character; The masked characters are replaced with preset mask characters.

7. The method according to claim 5, characterized in that The preset word vector table includes a plurality of preset characters and a word vector of each preset character; Determining the predicted character at the position of each masked character based on the multiple word vectors and the preset word vector table includes: For any masked character, the word vector corresponding to the position of the masked character is calculated similarity with the word vector in the preset word vector table to obtain the similarity between each preset character and the masked character; The preset characters in the preset word vector table whose similarity is greater than a preset threshold are determined as the predicted characters at the position where the masked character is located.

8. The method according to claim 5, characterized in that A loss value is determined based on a plurality of predicted characters and a plurality of masked characters, including: Determine a sub-loss value based on each masked character and a predicted character at the position of the masked character, and obtain multiple sub-loss values; The loss value is determined based on the plurality of sub-loss values.

9. A classification model training method, characterized in that: include: Obtain a training set, wherein the training set includes a plurality of training corpora, each training corpus carries a category label, wherein the training set is obtained by the training set construction method according to any one of claims 1 to 8 above; Inputting the plurality of training corpora into a classification model to obtain a predicted category for each training corpus; Determine the loss value based on the predicted category and category label of each training corpus; The classification model is trained based on the loss value until a training stop condition is reached.

10. A document classification method, characterized in that: include: Get the documents to be classified; The document to be classified is input into a classification model to determine the category of the document to be classified, wherein the classification model is obtained by the classification model training method described in claim 9 above.

11. A training set construction device, characterized in that: include: A first acquisition module is configured to acquire a training set and a target corpus, wherein each training corpus in the training set carries a category label, and the target corpus does not carry a category label; a text vector determination module, configured to input the training corpora and the target corpora into a pre-training model respectively, to obtain a first text vector of the training corpora and a second text vector of the target corpus, wherein the pre-training model is obtained by pre-training a preset natural language model based on the training set; a similarity judgment module, configured to perform similarity judgment on each of the first text vectors and the second text vector; The adding module is configured to add a target category label to the target corpus according to the similarity judgment result if the similarity judgment result meets the preset condition, and add the target corpus with the target category label added to the training set to obtain the updated training set.

12. A classification model training device, characterized in that: include: A second acquisition module is configured to acquire a training set, wherein the training set includes a plurality of training corpora, each training corpus carries a category label, wherein the training set is obtained by the training set construction method according to any one of claims 1 to 8 above; A prediction module is configured to input the plurality of training corpora into a classification model to obtain a predicted category of each training corpus; A first determination module is configured to determine a loss value based on the predicted category and category label of each training corpus; The training module is configured to train the classification model based on the loss value until a training stop condition is reached.

13. A document classification device, characterized in that: include: A third acquisition module is configured to acquire a document to be classified; The second determination module is configured to input the document to be classified into a classification model to determine the category of the document to be classified, wherein the classification model is obtained by the classification model training method described in claim 9 above.

14. A computing device, characterized in that include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the steps of the training set construction method described in any one of claims 1 to 8, or to implement the steps of the classification model training method described in claim 9, or to implement the steps of the document classification method described in claim 10.

15. A computer-readable storage medium storing computer instructions, characterized in that: When the instruction is executed by the processor, the steps of the training set construction method described in any one of claims 1 to 8 are implemented, or the steps of the classification model training method described in claim 9 are implemented, or the steps of the document classification method described in claim 10 are implemented.

Images