Confidential text recognition model training method, system and device based on knowledge distillation

Abstract

The present invention proposes a method, system and device for training a confidential text recognition model based on knowledge distillation, the method comprising: preparing an unlabeled corpus A in a confidentiality field; constructing a text label hierarchy tree according to confidential business data, and annotating to obtain a labeled data set B, preparing an unlabeled data set C; performing text enhancement on the text in the unlabeled data set C; performing knowledge distillation through the unlabeled corpus A, so that the IDCNN model learns the semantic feature extraction capability from the Bert model; constructing a label path classification model Bert‑clf based on the Bert model, and performing supervised classification training on the Bert model through the labeled data set B to obtain the label path classification model Bert‑clf; constructing a label path classification model Idcnn‑clf based on the IDCNN model; performing knowledge distillation on the label path classification model Idcnn‑clf through the labeled data set B and the unlabeled data set C; and saving the label path classification model Idcnn‑clf. The present invention can effectively improve the prediction speed and classification accuracy of the confidential text recognition model.

Description

Technical Field

[0001] The present invention relates to the field of text recognition technology, and more specifically to a confidential text recognition model training method, system and device based on knowledge distillation. Background Art

[0002] The continuous progress of science and technology has led to the rapid development of productivity and profound changes in society. A large amount of data has been generated in various industries, fields, and scenarios. As a special form of asset, the security of data assets is as important as the security of data content. Especially in special industries, the leakage of confidential information will bring huge losses. The identification of confidential texts is of great significance to information security.

[0003] Existing confidential text recognition mainly uses rule-based methods. In rule-based methods, business personnel usually build a large number of rules and use regular matching and other methods to determine whether the text is leaked. Relying on business personnel to build a large number of rules and use regular matching and other methods to make judgments has problems such as poor generalization ability and inconsistent logic of rules.

[0004] Moreover, the text classification model used in confidential text recognition relies on a large amount of manually annotated data. Based on the large-scale pre-trained language model Bert, a higher accuracy rate can be achieved with less training data. However, there are defects such as the large size of the pre-trained model and the slow online prediction speed. Summary of the invention

[0005] In view of the above problems, the purpose of the present invention is to provide a confidential text recognition model training method, system and device based on knowledge distillation, which can effectively improve the prediction speed and classification accuracy of the confidential text recognition model.

[0006] To achieve the above object, the present invention is implemented through the following technical solutions: a confidential text recognition model training method based on knowledge distillation, comprising:

[0007] The steps include:

[0008] S1: Prepare the unlabeled corpus A in the confidentiality domain;

[0009] S2: Build a text label hierarchy tree based on confidential business data, annotate it to obtain a labeled dataset B, and prepare an unlabeled dataset C;

[0010] S3: Perform text enhancement on the text in the unlabeled dataset C;

[0011] S4: Knowledge distillation is performed through the unlabeled corpus A, so that the IDCNN model can learn the semantic feature extraction ability from the Bert model;

[0012] S5: Construct a label path classification model Bert-clf based on the Bert model, perform supervised classification training on the Bert model through the labeled data set B, use the label path probability distribution output by the Bert model and the cross loss entropy of the text annotation label as the loss function, and obtain the label path classification model Bert-clf;

[0013] S6: Construct a label path classification model Idcnn-clf based on the IDCNN model;

[0014] S7: Perform knowledge distillation on the label path classification model Idcnn-clf through the labeled dataset B and the unlabeled dataset C;

[0015] S8: Save the label path classification model Idcnn-clf and use it as a confidential text recognition model.

[0016] Furthermore, each node of the label hierarchy tree is provided with a label, the label of each child node of the label hierarchy tree belongs to the label of the parent node, the category label corresponding to the text in the label data set B is the leaf node label of the label hierarchy tree, and each leaf node label represents the label path where the node is located.

[0017] Further, the step S3 specifically includes:

[0018] The text in the unlabeled dataset C is enhanced through synonym dictionary replacement, word vector similarity-based replacement, back-translation-based replacement, and Masked language model-based replacement.

[0019] Further, the step S4 specifically includes:

[0020] Start model training with the pre-trained Bert model as the Teacher model and the IDCNN model as the Student model;

[0021] The prediction target of the IDCNN model is set as the vector corresponding to the characters of the text after the last output layer [CLS] of the Bert model. This vector is used as the semantic feature extraction of the text by the Bert model.

[0022] The IDCNN model is trained through the mean square error loss function, so that the IDCNN model can learn the semantic feature extraction ability of Bert.

[0023] Furthermore, the label path classification model Bert-clf based on the Bert model and the label path classification model Idcnn-clf based on the IDCNN model are both constructed using a label hierarchical tree, and the construction process includes:

[0024] Use the label hierarchy tree to build the output layer of the Bert model or IDCNN model, so that the model can learn the information between labels and calculate the global optimal label path;

[0025] When calculating the global optimal label path, the text is used to output the probability value of each label through the Bert model or IDCNN model, and then the probability value of each label path is calculated through the probability calculation formula of the label path. The leaf node of the label path with the largest probability value is the classification result of the model.

[0026] Furthermore, the probability calculation formula of the label path is as follows:

[0027]

[0028]

[0029] in, represents the probability of label path i in the label hierarchy tree, represents the sum of the probabilities from the start node to the leaf node on the label path i, i and j both represent label paths, represents the number of nodes in label path i, k represents the kth node in label path i, X represents the set of probability values ​​of each node in the label hierarchy tree, Indicates the probability value of the kth node in the label path i in the set The subscript in Represents the probability value corresponding to the kth node.

[0030] Further, the step S7 specifically includes:

[0031] The label path classification model Bert-clf is used as the Teacher model, and the label path classification model Idcnn-clf is used as the Student model;

[0032] Through the labeled dataset B, the learning target of the label path classification model Idcnn-clf is set to the category label of the text annotation in B;

[0033] Through the labeled data set B, let the label path classification model Idcnn-clf learn the label probability distribution of the text in the labeled data set B output by the label path classification model Bert-clf;

[0034] Through the unlabeled dataset C, let the label path classification model Bert-clf output the path probability distribution of the text in the unlabeled dataset C and the path probability distribution of the enhanced version corresponding to the text, and calculate the KL divergence between the two distributions: , the label path classification model Idcnn-clf model outputs the path probability distribution of the two texts and calculates the KL divergence as , the learning goal is to minimize and The square of the difference;

[0035] Construct a joint loss function through the above three learning objectives Training label path classification model Idcnn-clf;

[0036] The calculation formula of the joint loss function is as follows:

[0037]

[0038]

[0039]

[0040]

[0041] in, It represents the cross loss entropy between the probability distribution of the text label path output by the label path classification model Idcnn-clf and the category label corresponding to the text, and N represents the number of all label paths; Indicates whether the annotation category of the text is label path i, if so, the value is 1, otherwise 0; Indicates the probability value corresponding to the label path i output by the label path classification model Idcnn-clf; Represents the squared difference loss between the text path probability distribution output by the label path classification model Idcnn-clf and the text path probability output by the label path classification model Bert-clf; Indicates the probability value corresponding to the label path i output by the label path classification model Bert-clf; Indicates the probability value corresponding to the label path i output by the label path classification model Idcnn-clf; express and The square of the difference between .

[0042] Correspondingly, the present invention also discloses a confidential text recognition model training system based on knowledge distillation, comprising:

[0043] A corpus preparation unit, used to prepare an unlabeled corpus A in the confidentiality domain;

[0044] The data set preparation unit is used to construct a text label hierarchical tree based on the confidential business data, annotate it to obtain a labeled data set B, and prepare an unlabeled data set C;

[0045] A text enhancement unit, used to enhance the text in the unlabeled dataset C;

[0046] The first knowledge distillation unit is used to perform knowledge distillation through the unlabeled corpus A, so that the IDCNN model can learn the semantic feature extraction ability from the Bert model;

[0047] A supervised classification training unit is used to construct a label path classification model Bert-clf based on the Bert model, and supervise the classification training of the Bert model through a labeled data set B. The label path probability distribution output by the Bert model and the cross loss entropy of the text annotation label are used as the loss function to obtain the label path classification model Bert-clf;

[0048] A classification model building unit, used to build a label path classification model Idcnn-clf based on the IDCNN model;

[0049] The second knowledge distillation unit is used to perform knowledge distillation on the label path classification model Idcnn-clf through the labeled dataset B and the unlabeled dataset C;

[0050] The classification model storage unit is used to save the label path classification model Idcnn-clf and use it as a confidential text recognition model.

[0051] Accordingly, the present invention discloses a confidential text recognition model training device based on knowledge distillation, comprising:

[0052] A memory, used for storing a confidential text recognition model training program based on knowledge distillation;

[0053] A processor is used to implement the steps of the confidential text recognition model training method based on knowledge distillation as described in any of the above items when executing the confidential text recognition model training program based on knowledge distillation.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] 1. The present invention constructs a label hierarchy structure tree of the text based on confidential knowledge and divides the text. By training a deep learning model, the model automatically classifies the label path of the text and determines whether the text is confidential based on the label path.

[0056] 2. The present invention constructs a label hierarchy structure tree of text through business knowledge, which can better characterize the text, be closer to the business, and can be used to decide whether the text is confidential. Through the label hierarchy structure tree, the model can learn the global optimal label path, learn the relationship between labels and hierarchical structure information, and improve the classification accuracy of the model.

[0057] 3. The present invention uses knowledge distillation, with Bert as the Teacher model and the lightweight model IDCNN as the Student model. Knowledge distillation is performed through unlabeled data and labeled data to transfer the knowledge in the Teacher model to the Student model. The text classification model based on IDCNN has the advantages of being lightweight and having a fast prediction speed. Through secondary knowledge distillation, the knowledge in Bert can be fully transferred to the IDCNN model, thereby improving the accuracy of the IDCNN model while retaining the prediction speed of IDCNN. The classification model Idcnn-clf based on IDCNN has a prediction speed that is more than 30 times that of the classification model Bert-clf based on Bert, and the accuracy after distillation is improved by 8.8 percentage points compared to before distillation.

[0058] 4. The present invention not only uses this distillation method, but also distills the semantic feature extraction ability of the Bert model through large-scale confidential field corpus, and distills the text classification prediction ability of Bert using unlabeled industry text classification datasets.

[0059] It can be seen that compared with the prior art, the present invention has outstanding substantive features and significant progress, and the beneficial effects of its implementation are also obvious. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0061] Attached Figure 1 It is a method flow chart of a specific implementation mode of the present invention.

[0062] Attached Figure 2 It is a classification model diagram based on a label hierarchy structure tree according to a specific implementation mode of the present invention.

[0063] Attached Figure 3 It is a system structure diagram of a specific implementation mode of the present invention.

[0064] In the figure, 1 is the corpus preparation unit; 2 is the data set preparation unit; 3 is the text enhancement unit; 4 is the first knowledge distillation unit; 5 is the supervised classification training unit; 6 is the classification model construction unit; 7 is the second knowledge distillation unit; and 8 is the classification model storage unit. DETAILED DESCRIPTION

[0065] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0066] Embodiment 1:

[0067] like Figure 1 As shown, this embodiment provides a confidential text recognition model training method based on knowledge distillation, comprising the following steps:

[0068] S1: Prepare an unlabeled corpus A in the confidential domain.

[0069] The purpose of this step is to prepare a large-scale unlabeled confidential domain corpus for subsequent knowledge distillation.

[0070] S2: Build a text label hierarchy tree based on confidential business data, annotate it to obtain a labeled dataset B, and prepare an unlabeled dataset C.

[0071] Among them, each node of the label hierarchy tree is equipped with a label, the label of each child node of the label hierarchy tree belongs to the label of the parent node, and the category label corresponding to the text in the label data set B is the leaf node label of the label hierarchy tree.

[0072] Specifically, this step constructs a label hierarchy structure tree based on confidentiality business knowledge. Each node of the tree has a corresponding label, and the label of each child node of the tree belongs to the label of the parent node (with a containment relationship). The label hierarchy structure tree is used to divide the text as a basis for determining whether the text is confidential. The labeled dataset B obtained by annotating the label hierarchy tree is the leaf node of the label hierarchy structure tree.

[0073] S3: Perform text augmentation on the text in the unlabeled dataset C.

[0074] The specific text enhancement methods are: synonym dictionary replacement, word vector similarity-based replacement, back translation-based replacement, and Masked language model-based replacement.

[0075] S4: Through the unlabeled corpus A, knowledge distillation is performed to enable the IDCNN model to learn the semantic feature extraction ability from the Bert model.

[0076] This step achieves the first knowledge distillation. Specifically, the pre-trained Bert model is used as the Teacher model, and the IDCNN model is used as the Student model. The prediction target of the IDCNN model is the vector corresponding to the character '[CLS]' of the last output layer of the Bert model, which is used as the semantic feature extraction of the text by the Bert model. The IDCNN model is trained by the mean square error loss function to enable the IDCNN model to learn the semantic feature extraction ability of Bert.

[0077] S5: Construct a label path classification model Bert-clf based on the Bert model, perform supervised classification training on the Bert model through the labeled data set B, use the label path probability distribution output by the Bert model and the cross loss entropy of the text annotation label as the loss function, and obtain the label path classification model Bert-clf.

[0078] S6: Construct a label path classification model Idcnn-clf based on the IDCNN model.

[0079] It should be noted that the above steps S5 and S6 both construct classification models based on Bert and IDCNN through label hierarchical trees. Figure 2 As shown in the figure, the output layer of the model is constructed by using the label hierarchy structure tree, so that the model can learn the information between labels and calculate the global optimal label path. The probability value of each label is output through the Bert or IDCNN model, and then the probability value of each label path (that is, the path from the root node of the tree to the leaf node) is calculated through the probability calculation formula of the label path. The probability of each label path is the sum of the probability values ​​of all nodes on the path.

[0080] The probability calculation formula of the label path is as follows:

[0081]

[0082]

[0083] in, represents the probability of label path i in the label hierarchy tree, represents the sum of the probabilities from the start node to the leaf node on the label path i, i and j both represent label paths, represents the number of nodes in label path i, k represents the kth node in label path i, X represents the set of probability values ​​of each node in the label hierarchy tree, Indicates the probability value of the kth node in the label path i in the set The subscript in Represents the probability value corresponding to the kth node.

[0084] S7: Knowledge distillation of the label path classification model Idcnn-clf is performed through the labeled dataset B and the unlabeled dataset C.

[0085] This step realizes the second knowledge distillation. The specific process is as follows:

[0086] The label path classification model Bert-clf is used as the Teacher model, and the label path classification model Idcnn-clf is used as the Student model.

[0087] Through the labeled dataset B, the learning target of the label path classification model Idcnn-clf is set to the category label of the text annotation in B.

[0088] Through the labeled data set B, the label path classification model Idcnn-clf is allowed to learn the label probability distribution of the text in the labeled data set B output by the label path classification model Bert-clf.

[0089] Through the unlabeled dataset C, let the label path classification model Bert-clf output the path probability distribution of the text in the unlabeled dataset C and the path probability distribution of the enhanced version corresponding to the text, and calculate the KL divergence between the two distributions: , the label path classification model Idcnn-clf model outputs the path probability distribution of the two texts and calculates the KL divergence as , the learning goal is to minimize and The square of the difference.

[0090] Construct a joint loss function through the above three learning objectives Train the label path classification model Idcnn-clf.

[0091] The calculation formula of the joint loss function is as follows:

[0092]

[0093]

[0094]

[0095]

[0096] in, It represents the cross loss entropy between the probability distribution of the text label path output by the label path classification model Idcnn-clf and the category label corresponding to the text, and N represents the number of all label paths; Indicates whether the annotation category of the text is label path i, if so, the value is 1, otherwise 0; Indicates the probability value corresponding to the label path i output by the label path classification model Idcnn-clf; Represents the squared difference loss between the text path probability distribution output by the label path classification model Idcnn-clf and the text path probability output by the label path classification model Bert-clf; Indicates the probability value corresponding to the label path i output by the label path classification model Bert-clf; Indicates the probability value corresponding to the label path i output by the label path classification model Idcnn-clf; express and The square of the difference between .

[0097] S8: Save the label path classification model Idcnn-clf and use it as a confidential text recognition model.

[0098] Embodiment 2:

[0099] Based on the first embodiment, Figure 3 As shown, the present invention also discloses a confidential text recognition model training system based on knowledge distillation, including: a corpus preparation unit 1, a data set preparation unit 2, a text enhancement unit 3, a first knowledge distillation unit 4, a supervised classification training unit 5, a classification model construction unit 6, a second knowledge distillation unit 7 and a classification model storage unit 8.

[0100] Corpus preparation unit 1, used to prepare unlabeled corpus A in the confidentiality field;

[0101] The data set preparation unit 2 is used to construct a text label hierarchy tree based on the confidential business data, annotate it to obtain a labeled data set B, and prepare an unlabeled data set C.

[0102] The text enhancement unit 3 is used to perform text enhancement on the text in the unlabeled dataset C.

[0103] The first knowledge distillation unit 4 is used to perform knowledge distillation through the unlabeled corpus A, so that the IDCNN model can learn the semantic feature extraction ability from the Bert model.

[0104] The supervised classification training unit 5 is used to construct a label path classification model Bert-clf based on the Bert model, and supervised classification training is performed on the Bert model through a labeled data set B. The label path probability distribution output by the Bert model and the cross loss entropy of the text annotation label are used as the loss function to obtain the label path classification model Bert-clf.

[0105] The classification model building unit 6 is used to build a label path classification model Idcnn-clf based on the IDCNN model.

[0106] The second knowledge distillation unit 7 is used to perform knowledge distillation on the label path classification model Idcnn-clf through the labeled data set B and the unlabeled data set C.

[0107] The classification model storage unit 8 is used to store the label path classification model Idcnn-clf and use it as a confidential text recognition model.

[0108] Embodiment three:

[0109] This embodiment discloses a confidential text recognition model training device based on knowledge distillation, comprising a processor and a memory; wherein the processor implements the following steps when executing the confidential text recognition model training program based on knowledge distillation stored in the memory:

[0110] 1. Prepare an unlabeled corpus A in the confidentiality field.

[0111] 2. Build a text label hierarchy tree based on confidential business data, annotate it to obtain a labeled data set B, and prepare an unlabeled data set C.

[0112] 3. Perform text enhancement on the text in the unlabeled dataset C.

[0113] 4. Perform knowledge distillation through the unlabeled corpus A, so that the IDCNN model can learn the semantic feature extraction ability from the Bert model.

[0114] 5. Construct a label path classification model Bert-clf based on the Bert model, perform supervised classification training on the Bert model through the labeled data set B, use the label path probability distribution output by the Bert model and the cross loss entropy of the text annotation label as the loss function, and obtain the label path classification model Bert-clf.

[0115] 6. Build a label path classification model Idcnn-clf based on the IDCNN model.

[0116] 7. Perform knowledge distillation on the label path classification model Idcnn-clf through the labeled dataset B and the unlabeled dataset C.

[0117] 8. Save the label path classification model Idcnn-clf and use it as a confidential text recognition model.

[0118] Furthermore, the confidential text recognition model training device based on knowledge distillation in this embodiment may also include:

[0119] The input interface is used to obtain the confidential text recognition model training program based on knowledge distillation imported from the outside, and save the obtained confidential text recognition model training program based on knowledge distillation to the memory. It can also be used to obtain various instructions and parameters transmitted by external terminal devices and transmit them to the processor so that the processor can use the above various instructions and parameters to carry out corresponding processing. In this embodiment, the input interface can specifically include but is not limited to a USB interface, a serial interface, a voice input interface, a fingerprint input interface, a hard disk reading interface, etc.

[0120] The output interface is used to output various data generated by the processor to the terminal device connected thereto, so that other terminal devices connected to the output interface can obtain various data generated by the processor. In this embodiment, the output interface may specifically include but is not limited to a USB interface, a serial interface, etc.

[0121] A communication unit is used to establish a remote communication connection between the confidential text recognition model training device based on knowledge distillation and an external server, so that the confidential text recognition model training device based on knowledge distillation can mount the image file to the external server. In this embodiment, the communication unit may specifically include but is not limited to a remote communication unit based on wireless communication technology or wired communication technology.

[0122] The keyboard is used to obtain various parameter data or instructions input by the user by tapping the keycaps in real time.

[0123] The display is used to display the relevant information of the server power supply line short circuit locating process in real time.

[0124] The mouse can be used to assist users in inputting data and simplify user operations.

[0125] In summary, the label hierarchy structure tree constructed by the present invention through business knowledge can better describe the text, be closer to the business, and can be used to decide whether the text is confidential. Through the label hierarchy structure tree, the model can learn the global optimal label path, the relationship between labels and the hierarchical structure information, and effectively improve the preset speed and classification accuracy of the model.

[0126] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the method disclosed in the embodiment, since it corresponds to the system disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0127] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0128] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0129] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0130] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0131] The above is a detailed introduction to the confidential text recognition model training method, system, and device based on knowledge distillation provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A confidential text recognition model training method based on knowledge distillation, characterized in that: The steps include: S1: Prepare the unlabeled corpus A in the confidentiality domain; S2: Build a text label hierarchy tree based on confidential business data, annotate it to obtain a labeled dataset B, and prepare an unlabeled dataset C; S3: Perform text enhancement on the text in the unlabeled dataset C; S4: Knowledge distillation is performed through the unlabeled corpus A, so that the IDCNN model can learn the semantic feature extraction ability from the Bert model; Specifically include: Start model training with the pre-trained Bert model as the Teacher model and the IDCNN model as the Student model; The prediction target of the IDCNN model is set as the vector corresponding to the characters of the text after the last output layer [CLS] of the Bert model. This vector is used as the semantic feature extraction of the text by the Bert model. The IDCNN model is trained by the mean square error loss function to enable the IDCNN model to learn the semantic feature extraction ability of Bert; S5: Construct a label path classification model Bert-clf based on the Bert model, perform supervised classification training on the Bert model through the labeled data set B, use the label path probability distribution output by the Bert model and the cross loss entropy of the text annotation label as the loss function, and obtain the label path classification model Bert-clf; S6: Construct a label path classification model Idcnn-clf based on the IDCNN model; S7: Perform knowledge distillation on the label path classification model Idcnn-clf through the labeled dataset B and the unlabeled dataset C; specifically including: The label path classification model Bert-clf is used as the Teacher model, and the label path classification model Idcnn-clf is used as the Student model; Through the labeled dataset B, the learning target of the label path classification model Idcnn-clf is set to the category label of the text annotation in B; Through the labeled data set B, let the label path classification model Idcnn-clf learn the label probability distribution of the text in the labeled data set B output by the label path classification model Bert-clf; Through the unlabeled dataset C, let the label path classification model Bert-clf output the path probability distribution of the text in the unlabeled dataset C and the path probability distribution of the enhanced version corresponding to the text, and calculate the KL divergence between the two distributions: , the label path classification model Idcnn-clf model outputs the path probability distribution of the two texts and calculates the KL divergence as , the learning goal is to minimize and The square of the difference; Construct a joint loss function through the above three learning objectives Training label path classification model Idcnn-clf; The calculation formula of the joint loss function is as follows: in, It represents the cross loss entropy between the probability distribution of the text label path output by the label path classification model Idcnn-clf and the category label corresponding to the text, and N represents the number of all label paths; Indicates whether the annotation category of the text is label path i, if so, the value is 1, otherwise 0; Indicates the probability value corresponding to the label path i output by the label path classification model Idcnn-clf; Represents the squared difference loss between the text path probability distribution output by the label path classification model Idcnn-clf and the text path probability output by the label path classification model Bert-clf; Indicates the probability value corresponding to the label path i output by the label path classification model Bert-clf; express and The square of the difference between S8: Save the label path classification model Idcnn-clf and use it as a confidential text recognition model.

2. The confidential text recognition model training method based on knowledge distillation according to claim 1 is characterized in that: Each node of the label hierarchy tree is provided with a label, the label of each child node of the label hierarchy tree belongs to the label of the parent node, the category label corresponding to the text in the label data set B is the leaf node label of the label hierarchy tree, and the label path of the node is recorded in each leaf node label of the label hierarchy tree.

3. The confidential text recognition model training method based on knowledge distillation according to claim 1 is characterized in that: The step S3 specifically includes: The text in the unlabeled dataset C is enhanced through synonym dictionary replacement, word vector similarity-based replacement, back-translation-based replacement, and Masked language model-based replacement.

4. The confidential text recognition model training method based on knowledge distillation according to claim 2 is characterized in that: The label path classification model Bert-clf based on the Bert model and the label path classification model Idcnn-clf based on the IDCNN model are both constructed using a label hierarchical tree, and the construction process includes: The output layer of the Bert model is constructed using the label hierarchy tree, so that the model can learn the information between labels and calculate the global optimal label path. When calculating the global optimal label path, the probability value of each label is output through the Bert model using text. Or use the label hierarchy tree to construct the output layer of the IDCNN model, so that the model can learn the information between labels and calculate the global optimal label path; when calculating the global optimal label path, use the text to output the probability value of each label through the IDCNN model; Then, the probability value of each label path is calculated through the probability calculation formula of the label path, and the leaf node of the label path with the largest probability value is the classification result of the model.

5. The confidential text recognition model training method based on knowledge distillation according to claim 4 is characterized in that: The probability calculation formula of the label path is as follows: in, represents the probability of label path i in the label hierarchy tree, represents the sum of the probabilities from the start node to the leaf node on the label path i, i and j both represent label paths, represents the number of nodes in label path i, k represents the kth node in label path i, X represents the set of probability values ​​of each node in the label hierarchy tree, Indicates the probability value of the kth node in the label path i in the set The subscript in Represents the probability value corresponding to the kth node.

6. A confidential text recognition model training system based on knowledge distillation, characterized in that: When the system is executed, the steps of implementing the confidential text recognition model training method based on knowledge distillation as described in any one of claims 1 to 5 include: A corpus preparation unit, used to prepare an unlabeled corpus A in the confidentiality domain; The data set preparation unit is used to construct a text label hierarchical tree based on the confidential business data, annotate it to obtain a labeled data set B, and prepare an unlabeled data set C; A text enhancement unit, used to enhance the text in the unlabeled dataset C; The first knowledge distillation unit is used to perform knowledge distillation through the unlabeled corpus A, so that the IDCNN model can learn the semantic feature extraction ability from the Bert model; A supervised classification training unit is used to construct a label path classification model Bert-clf based on the Bert model, and supervise the classification training of the Bert model through a labeled data set B. The label path probability distribution output by the Bert model and the cross loss entropy of the text annotation label are used as the loss function to obtain the label path classification model Bert-clf; A classification model building unit, used to build a label path classification model Idcnn-clf based on the IDCNN model; The second knowledge distillation unit is used to perform knowledge distillation on the label path classification model Idcnn-clf through the labeled dataset B and the unlabeled dataset C; The classification model storage unit is used to save the label path classification model Idcnn-clf and use it as a confidential text recognition model.

7. A confidential text recognition model training device based on knowledge distillation, characterized in that: include: A memory, used for storing a confidential text recognition model training program based on knowledge distillation; A processor is used to implement the steps of the confidential text recognition model training method based on knowledge distillation as described in any one of claims 1 to 5 when executing the confidential text recognition model training program based on knowledge distillation.

Images