A fusion local key information and pre-training text classification method

Abstract

The application provides a text classification method fusing local key information and pre-training, comprising: selecting a pre-training model with the highest matching degree with task data as a first pre-training model; further pre-training the first pre-training model using the task data to obtain a second pre-training model; extracting a keyword sequence from the text using the second pre-training model; setting n kinds of local attributes of the text; setting m kinds of text classifications; regarding each text classification under each local attribute as a local key category, thereby obtaining n*m local key categories; performing vector merging on a full-text representation vector obtained through the second pre-training model and local representation vectors under all local attributes to obtain a final representation vector of the text; evaluating the final representation vector of the text in terms of the labeling quality of the task data; and classifying the final representation vector of the text after the evaluation. The application improves the classification efficiency and accuracy.

Description

Technical Field

[0001] This application relates to the field of vulnerability classification in cybersecurity, specifically to a text classification method that integrates local key information and pre-trained data. Background Technology

[0002] With the comprehensive development of technological intelligence and the in-depth advancement of digital industrialization, various types of network software are constantly emerging. System software is accelerating its development towards open source, networking, and semantic markup. The diverse types of software and large amounts of interactive data within these systems have led to a continuous increase in the number of security vulnerabilities, resulting in a surge of network security incidents based on these vulnerabilities. If these numerous and varied vulnerabilities are not patched in a timely and effective manner, they may be exploited, increasing the probability of attacks and causing significant security risks. Because the actual environments and needs vary across different industries, networks, and platforms, the key vulnerability classifications vary. It is necessary to extract key information from vulnerability reports and, in conjunction with the specific environment and scenario, select the vulnerabilities of primary concern. In the application of information mining and classification of vulnerability reports, early methods primarily used manual annotation or rule matching. With the rapid development of Natural Language Processing (NLP), various models and methods have been applied to vulnerability information extraction. For example, the patent "An Information Security Vulnerability Classification Method Based on an Improved BERT Model" (application number: CN202111227292.9) uses a pre-trained language model, but the pre-trained model has a complex structure and a large number of parameters. For vulnerability classification tasks, data annotation often only labels the final vulnerability type, failing to capture the crucial information of the local categories within each type in the text. This lack of fine-grained information prevents the model from learning from hidden but important information sources. If a classification error occurs, the absence of fine-grained information makes correction difficult. Furthermore, the quality of the training set's data annotation significantly impacts model training performance. If the quality assessment is inadequate, re-annotation is necessary until the data quality meets the requirements of the real-world scenario. Since verifying each vulnerability individually is too costly and small-scale sampling may introduce survivorship bias, a high-quality data quality assessment process is essential.

[0003] Therefore, in vulnerability information classification, how to conduct in-depth learning of vulnerability reports in threat intelligence scenarios to effectively extract core information and how to effectively assess data quality are urgent problems to be solved in improving cybersecurity capabilities. Summary of the Invention

[0004] To address the aforementioned problems, this application provides a text classification method that integrates local key information and pre-trained data, comprising the following steps:

[0005] S1. From the pre-trained models that can obtain the overall semantics of the full text, select the pre-trained model that matches the task data the most closely as the first pre-trained model.

[0006] The task data includes text and annotations, wherein the annotations include the category to which the text belongs and a sequence of keywords representing key information of the text;

[0007] S2, use task data to further pre-train the first pre-trained model to obtain a second pre-trained model, which can obtain a full-text representation vector of the overall semantics of the representative text based on the task data;

[0008] S3, by learning to extract local key information from the second pre-trained model, the second pre-trained model is used to extract keyword sequences representing local key information from the text.

[0009] There are n types of local attributes for the text and m types of text categories. If each text category under each local attribute is treated as a local key category, then n*m types of local key categories can be obtained.

[0010] Extract word sequences belonging to the same local key category from the text as keyword sequences for that local key category, and use all keyword sequences in that local key category as local key information for that local key category;

[0011] S4, train word vectors for each word in the text to obtain word vectors that represent the semantic vector representation of each word itself;

[0012] S5. Based on the word vector of each word in the keyword sequence contained in the local key category, the average vector of each keyword sequence in the local key category is obtained by averaging the word vectors. Then, the average vector of all keyword sequences is averaged to obtain the local key vector corresponding to the local key information of the local key category.

[0013] S6: Concatenate all local key vectors under the same local attribute to obtain the local representation vector of that local attribute.

[0014] S7, merge the full-text representation vector obtained through the second pre-trained model with the local representation vectors under all local attributes to obtain the final text representation vector;

[0015] S8 evaluates the quality of the task data annotation on the final representation vector of the text.

[0016] S9 classifies the final representation vector of the text after the evaluation is passed.

[0017] Preferably, in step S1, the method for obtaining the pre-trained model that best matches the task data includes the following steps:

[0018] From several candidate pre-trained models based on relevant corpus data in this field, the candidate pre-trained model with the highest matching score between the candidate pre-trained model and the task data is selected as the first pre-trained model by calculating the matching score between the candidate pre-trained model and the task data.

[0019] Wherein, the matching score is calculated as a1 * coverage_rate + a2 * diversity_rate.

[0020] Where a1 and a2 are hyperparameters, coverage_rate is the coverage metric, and diversity_rate is the diversity metric;

[0021] Coverage rate metric: coverage_rate = 1 - YXL <unk> / RW <c>Among them, YXL <unk>To address the issue of unknown words appearing after segmenting the training corpus using candidate pre-trained models. <unk>The sum of the number of times, RW <c>The sum of the occurrences of each word after segmenting the task data using the candidate pre-trained model;

[0022] The richness metric diversity_rate = YXL <qc> / RW <qc>Among them, YXL <qc>RW represents the total number of words in the dictionary provided by the candidate pre-trained model. <qc>The total number of words after word segmentation and deduplication when the task data is input into the candidate pre-trained model.

[0023] Preferably, in step S2, during the process of obtaining the second pre-trained model by training the first pre-trained model, the randomly selected keyword sequence is first masked, and then some words are randomly selected from the remaining content to mask, so that the second pre-trained model can predict the masked keyword sequence and random words; wherein, the loss function during training is Loss = r1*Loss1 + r2*Loss2; where r1 and r2 are hyperparameters, Loss1 is the cross-entropy loss after randomly selecting some words at a certain masking rate and masking the selected words, and Loss2 is the cross-entropy loss after randomly selecting a keyword sequence and masking the selected keyword sequence.

[0024] Preferably, the method for extracting local key information from the second training model is as follows: the full-text representation vector representing the overall semantics of the text is obtained through the second training model, and it is used as the input of the BiLSTM-CRF model. LSTM is used to extract contextual information from both forward and reverse order perspectives. Conditional Random Field (CRF) is used to obtain the category labeled for each word in the text. The word sequence belonging to the same local key category is obtained as the keyword sequence of that local key category. All keyword sequences in that local key category are used as the local key information of that local key category.

[0025] Preferably, after word segmentation, the word2vec model is used to train word vectors to obtain the word vector of each word in the text.

[0026] Preferably, step S5 includes: setting the local key category corresponding to the j-th text classification under the i-th local attribute as Rij, and setting that a total of c keyword sequences are extracted from the local key category Rij, wherein the z-th keyword sequence consists of p words;

[0027] The average word vector of the z-th keyword sequence is obtained by averaging the word vectors of the p words in the z-th keyword sequence.

[0028] Obtain the intra-group average word vector of all c keyword sequences in the local key category Rij, and then average the c intra-group average word vectors to obtain the local key vector of the local key category Rij.

[0029] Preferably, in step S6, the local attributes are set to include verb attributes and noun attributes, and all local key vectors under the verb attributes and noun attributes are concatenated to obtain the local representation vector of the verb attributes and the local representation vector of the noun attributes.

[0030] Preferably, step S7 includes: concatenating the full-text representation vector of the text obtained through the second pre-trained model with the local representation vectors of verb attributes and noun attributes to obtain the final representation vector of the text.

[0031] Preferably, in step S8, the step of evaluating the quality of data annotation includes:

[0032] S81, use the K-MEANS clustering algorithm to obtain the center point of the corresponding text classification category, and at the same time obtain the clustering result of each text; obtain the basic seed sample RI0 that is closest to the center point according to a certain ratio;

[0033] S82, calculate the classification accuracy Q0 of the basic seed sample RI0, set the basic verification threshold W0. When Q0 < W0, the labeling quality is unqualified and the data is re-labeled; when Q0 ≥ W0, the verification is qualified and proceed to step S83.

[0034] S83, obtain samples with a certain similarity to the basic seed sample RI0 as first-order samples RI1, calculate the classification accuracy Q1 of first-order samples RI1, set the first-order verification threshold W1. When Q1 < W1, the labeling quality is unqualified and the data is re-labeled; when Q1 ≥ W1, the verification is qualified and proceed to step S84.

[0035] S84. Obtain samples with a certain similarity to the first-order sample RI1 as second-order samples RI2, calculate the classification accuracy Q2 of the second-order sample RI2, and set the second-order verification threshold W2. When Q2 < W2, the labeling quality is unqualified and the data is re-labeled; when Q2 ≥ W2, the verification is qualified and the labeled data quality assessment is passed.

[0036] Preferably, in step S81, the sample closest to the center particle is obtained using a sampling rate α_1 (0<α_1<1) as the basic seed sample;

[0037] In step S83, the sample with the highest similarity to the basic seed sample is obtained using a sampling rate α_2 (0<α_2<1) as the first-order sample;

[0038] In step S84, the sample with the highest similarity to the first-order sample is obtained using a sampling rate α_3 (0<α_3<1) as the second-order sample;

[0039] Among them, α_1<α_2<α_3; preferably, α_2=sin(α_1*π / 2) and α_3=sin(α_2*π / 2) are set.

[0040] The beneficial effects achieved by this application are as follows:

[0041] This application, based on general network security vulnerability classification methods, adds in-depth learning of local key information in information security texts and designs an evaluation method for the annotation quality of task data. The method provided in this application not only enables classification prediction from an overall semantic perspective but also adds support for local semantics, thereby improving the accuracy and interpretability of the classification results. This application extracts keyword sequences of local key categories under different attributes in the text for different classification categories, and further learns local representation vectors of local key information and global information representations based on pre-trained models, fusing these vectors to achieve more accurate classification of vulnerability report categories. By designing a data quality evaluation module that uses the learned representation vectors as input, it avoids evaluating each piece of data individually and reduces the bias of small-part sampling, lowering data evaluation costs and forming an efficient and high-quality annotation data quality evaluation. This application can be applied to threat intelligence production and other scenarios, and has positive significance for improving network security capabilities. Attached Figure Description

[0042] To more clearly illustrate the present invention, the accompanying drawings will be described and explained below. Obviously, the drawings described below only illustrate certain aspects of some exemplary embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0043] Figure 1 This is a diagram illustrating word sequences in local key categories of text in the embodiments.

[0044] Figure 2 This is a schematic diagram of the module functional structure of the module system in this application.

[0045] Figure 3 This is a flowchart illustrating the text classification method that integrates local key information and pre-trained data in this application.

[0046] Figure 4 This is a flowchart of the pre-training steps.

[0047] Figure 5 A schematic diagram of the functional structure of the data annotation quality assessment module.

[0048] Figure 6 Flowchart of the steps for assessing the quality of data annotation. Detailed Implementation

[0049] The following is for reference only. Figures 1-6 Various exemplary embodiments of this disclosure are described in detail. The descriptions of exemplary embodiments are merely illustrative and are in no way intended to limit this disclosure or its application or use. This disclosure may be implemented in many different forms and is not limited to the embodiments described herein. These embodiments are provided so that this disclosure will be thorough and complete, and will fully express the scope of this disclosure to those skilled in the art. It should be noted that, unless otherwise stated, the relative arrangement of components and steps, numerical expressions, and values ​​set forth in these embodiments should be interpreted as merely exemplary and not as limiting.

[0050] like Figure 2 and Figure 3 The diagram shown is a flowchart of the module system and method of this application. The module system used in this application includes a pre-training module, a local key information extraction module, a word vector training module, a local representation vector module, an annotation quality evaluation module, and a classification module.

[0051] Pre-trained modules: such as Figure 4 The diagram illustrates the workflow of the pre-training module. First, a custom matching metric is used to select the pre-trained model that best matches the task data as the first pre-trained model. Based on the first pre-trained model, further pre-training is performed using the task data to generate a second pre-trained model adapted to the task data. The task data consists of labeled text sequences, with annotations including the text's category and a sequence of keywords representing key information.

[0052] By inputting text into the second pre-trained model, a full-text representation vector of the overall semantics of the representative text based on task data can be obtained.

[0053] In this application, the task data consists of text sequences in the field of information security. Each text is labeled with its classification category in the classification task and a sequence of keywords representing key information. For example, in a text classification task, there are three categories: Denial of Service (DOS), Privilege Escalation, and Information Disclosure. One text in the task data that forms the basis of the second pre-trained model is: "vulnerability in junos os allow remote attacker to craft message to cause process crash and information leak." The keyword sequence annotated for this text is remoteattacker, cause process crash, information leak, and the text belongs to the category "information disclosure."

[0054] In the pre-trained model, the specific method for selecting the pre-trained model with the highest degree of matching with the task data as the first pre-trained model using a custom matching index is as follows: select multiple candidate pre-trained models constructed from cybersecurity corpus data (such as Advanced Persistent Threat (APT) analysis reports and security vulnerability reports), such as SecBERT, CySecBERT, and CyBERT. Score the degree of matching between each candidate pre-trained model and the task data by designing a custom matching index, and select the candidate pre-trained model with the highest score as the first training model.

[0055] The formula for calculating the matching score between the candidate pre-trained model and the task data is: score = a1 * coverage_rate + a2 * diversity_rate; where a1 and a2 are hyperparameters, coverage_rate is the coverage index, and diversity_rate is the diversity index.

[0056] Coverage rate metric: coverage_rate = 1 - YXL <unk> / RW <c>Among them, YXL <unk>To address the issue of unknown words appearing after segmenting the training corpus using candidate pre-trained models. <unk>The sum of the number of times, RW <c>The sum of the occurrences of each word after segmenting the task data using the candidate pre-trained model.

[0057] The richness metric diversity_rate = YXL <qc> / RW <qc>Among them, YXL <qc>RW represents the total number of words in the dictionary provided by the candidate pre-trained model. <qc>The total number of words after word segmentation and deduplication when the task data is input into the candidate pre-trained model.

[0058] Based on the above calculation formula, the candidate pre-trained model with the highest matching score is selected as the first pre-trained model. Using task data as the training basis, the first pre-trained model is further pre-trained to generate a second pre-trained model based on the task data. Specifically, the masked language model is used for pre-training on task data for the target task. However, unlike the first pre-trained model, considering the ability to simultaneously learn both the overall semantic information and local key information of the text, the keyword sequences representing local key information are first randomly masked, and then a portion of words randomly selected from the remaining content are masked. This allows the model to eventually learn to predict these masked local key information and words after further training.

[0059] The second pre-trained model obtained through the above pre-training method can further optimize the semantic learning of text in task data and has a deeper understanding of the semantic relationship between keyword sequences and words in the text and the overall semantics.

[0060] The loss function during the training of the second pre-trained model is r1*Loss1+r2*Loss2.

[0061] Where r1 and r2 are hyperparameters; Loss1 is the random masking loss of text words, which is the cross-entropy loss calculated after randomly selecting some words in the input sentence of the model with a certain masking rate and masking them; Loss2 is the masking loss of keyword sequence. The difference is that Loss1 randomly selects some words in the whole sentence and masks them, while Loss2 randomly selects a keyword sequence and masks the selected keyword sequence.

[0062] Local key information extraction module: Fine-tuning learning of the second pre-trained model for extracting local key information. The fine-tuning learning method is to use the second pre-trained model + BiLSTM + CRF neural network to extract keyword sequences representing local key information from the text.

[0063] There are n types of local attributes for the input text and m types of text categories. Each text category under each local attribute is treated as a local key category, resulting in n*m types of local key categories. Keyword sequences are extracted for each local key category, and all extracted keyword sequences under the same local key category are used as local key information for that local key category.

[0064] Taking a text classification task about the impact of a vulnerability as an example, the classification categories for this task include three types: Denial of Service (DOS), Privilege Escalation, and Information Disclosure. There are two types of local attributes for the text: noun phrases and verb phrases. Therefore, the local key categories for this text classification task F include 2*3 types: noun phrase - Denial of Service, noun phrase - Privilege Escalation, noun phrase - Information Disclosure, verb phrase - Denial of Service, verb phrase - Privilege Escalation, and verb phrase - Information Disclosure. In each text, local key information may be extractable from only some of these local key categories, while it may not be extractable from others.

[0065] The specific steps for the second training model to extract local key information are as follows: The second training model obtains a full-text representation vector representing the overall semantics of the text, which is then used as input to the BiLSTM-CRF model. LSTM is used to extract contextual information from both forward and reverse order perspectives. Conditional Random Field (CRF) is used to obtain the category of each word in the text. Word sequences from the above six local key categories are extracted as keyword sequences representing the local key information of the corresponding local key category. For example, in an information security text "the kernel-mode drivers in Windows Server 2008 SP2 and Windows Server 2016 allow a local authenticated attack to obtain information, or in Windows 7 and later, cause denial of service, aka Win32k information disclosure vulnerability.", such as... Figure 1 As shown, through the extraction of local key information, we finally obtained one keyword sequence "obtain information" contained in the local key category "[verb phrase - information leakage]", one keyword sequence "cause denial of service" contained in the local key category "[verb phrase - denial of service]", and one keyword sequence "information disclosure vulnerability" contained in the local key category "[noun phrase - information leakage]".

[0066] Word vector training module: After word segmentation of the text, the word2vec model is used to train word vectors to obtain the word vector of each word in the text.

[0067] Local Representation Vector Module: Based on the word vector of each word in the keyword sequence contained in the local key category, the local key vector corresponding to the local key information of the local key category is obtained by averaging the word vectors; all local key vectors under the same local attribute are concatenated to obtain the local representation vector of the local attribute; the full-text representation vector obtained by the second pre-trained model and the local representation vectors under all local attributes are merged to obtain the final representation vector of the input text.

[0068] by Figure 1 Taking the information security text shown as an example, the method for calculating the local key vector of the local key category [verb phrase - information leakage] is as follows: Based on the local key category [verb phrase - information leakage] extracted in the previous step, which contains one keyword sequence, obtain information, the word vectors of the two words "obtain" and "information" included in the keyword sequence are averaged to obtain the intra-group average vector. Since there is only one keyword sequence, the intra-group average vector is used as the local key vector of the local key category [verb phrase - information leakage]. If the local key category [verb phrase - information leakage] contains two or more keyword sequences, the intra-group average vectors are averaged again to obtain the inter-group average vector, which is used as the local key vector of the local key category [verb phrase - information leakage].

[0069] The local key vectors of other local key categories are obtained in the same way. For local key categories where no keyword sequence is extracted from the text, such as the local key category [verb phrase - elevate authority], its local key vector is represented by the zero vector.

[0070] Using the above method, the local key vectors of the three local key categories under the verb attribute [verb phrase - information leakage], [verb phrase - denial of service], and [verb phrase - escalation of privileges] are concatenated to obtain the local representation vector under the verb attribute.

[0071] The local key vectors of the three local key categories under the noun attribute [noun phrase - information leakage], [noun phrase - denial of service], and [noun phrase - privilege escalation] are concatenated to obtain the local representation vector under the noun attribute.

[0072] The full-text representation vector, the local representation vector under the verb attribute, and the local representation vector under the noun attribute obtained through the second pre-trained model are merged to obtain the final representation vector of the text.

[0073] Annotation Quality Assessment Module: This module is used to assess the quality of the labeled data in the final representation vectors. For example... Figure 5 , Figure 6 As shown, the quality assessment steps include: For the final text representation vector, the K-MEANS clustering algorithm is used to obtain a small proportion of basic seed samples RI0 that are close to the center point samples, with this small proportion set as α_1 (0 < α_1 < 1); the classification accuracy Q0 of the basic seed samples RI0 is calculated, and a basic verification threshold W0 is set. When Q0 < W0, the annotation quality is unqualified, and the dataset needs to be re-annotated; when Q0 ≥ W0, the verification is qualified, and the next step of verification is performed; using the basic seed samples RI0 as the benchmark samples, the cosine similarity is calculated, and a certain proportion of samples with the highest similarity to the basic seed samples RI0 are obtained as first-order samples RI1, with the proportion set as α_2 (0 < α_2 < 1). First, calculate the consistency Q1 of the first-order sample RI1 and set a first-order verification threshold W1. When Q1 < W1, the annotation quality is unqualified, and the dataset needs to be re-annotated. When Q1 ≥ W1, the verification is qualified, and further verification continues. Using the first-order sample RI1 as the baseline sample, obtain a certain proportion of samples with the highest similarity to the first-order sample RI1 as the second-order sample RI2. The proportion can be set to α_3 (0 < α_3 < 1). Calculate the consistency Q2 of the second-order sample RI2 and set a second-order verification threshold W2. When Q2 < W2, the annotation quality is unqualified, and the dataset needs to be re-annotated. When Q2 ≥ W2, the verification is qualified, the annotation data quality assessment is passed, and the final text representation vector obtained from such annotated data is usable. The relationship between α_1, α_2, and α_3 is α_1 < α_2 < α_3, α_2 = sin(α_1 * π / 2), α_3 = sin(α_2 * π / 2). For example, if α_1 = 10%, then α_2 ≈ 15.6% and α_3 ≈ 24.3%.

[0074] The classification module takes the final text representation vector, which has passed the data labeling quality assessment module, and inputs it into a simple neural network for classification. Specifically, it takes the final text representation vector and its corresponding label as input, followed by two fully connected layers and a dropout layer, and finally a softmax layer, thus obtaining the classification of the text into three types: denial of service, privilege escalation, and information disclosure.

[0075] As used in this disclosure, the words "including" or "contains" or similar terms mean that the element preceding the word covers the element listed after the word, and do not exclude the possibility that it may also cover other elements.

[0076] All terms used in this disclosure (including technical or scientific terms) have the same meaning as understood by one of ordinary skill in the art to which this disclosure pertains, unless otherwise specifically defined. It should also be understood that terms defined in general dictionaries should be understood to have meanings consistent with their meanings in the context of the relevant art, and not to be interpreted with idealized or highly formalized meanings, unless expressly defined herein.

[0077] For components, specific model numbers and other parameters of components not described in detail in this section, the interrelationships between components and control circuits, these may be considered as techniques, methods and devices known to those skilled in the art, but where appropriate, such techniques, methods and devices should be considered part of the specification.

[0078] It should be understood that the specific embodiments described above are only used to explain the present invention, and the scope of protection of the present invention is not limited thereto. Any modifications, substitutions, or combinations made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.< / qc> < / qc> < / qc> < / qc> < / c> < / unk> < / unk> < / c> < / unk> < / qc> < / qc> < / qc> < / qc> < / c> < / unk> < / unk> < / c> < / unk>

Claims

1. A method of fusing local key information and pre-trained text classification, characterized in that, The method comprises the following steps: S1, selecting a pre-training model with the highest matching degree with task data as a first pre-training model from pre-training models capable of obtaining the overall semantics of the full text of the text; The task data comprises a text and a label, and the label comprises a category to which the text belongs and a keyword sequence representing key information of the text; S2, further pre-training the first pre-training model using the task data to obtain a second pre-training model capable of obtaining a full-text representation vector representing the overall semantics of the full text of the text based on the task data; S3, extracting a keyword sequence representing local key information from the text by performing local key information extraction learning on the second pre-training model using the second pre-training model; Wherein, the local attributes of the text are set to n, the text categories are set to m, and each text category under each local attribute is regarded as a local key category, so that n*m local key categories can be obtained; The word sequence belonging to the same local key category is extracted from the text as the keyword sequence of the local key category, and all keyword sequences in the local key category are regarded as the local key information of the local key category; S4, performing word vector training on each word in the text to obtain a word vector representing the semantic vector representation of each word itself; S5, obtaining an average vector of each keyword sequence in the local key category by performing word vector average calculation according to the word vector of each word in the keyword sequence contained in the local key category, and then performing average calculation on the average vectors of all keyword sequences to obtain a local key vector corresponding to the local key information of the local key category; S6, performing vector splicing on all local key vectors under the same local attribute to obtain a local representation vector of the local attribute; S7, performing vector merging on the full-text representation vector obtained by the second pre-training model and the local representation vectors under all local attributes to obtain a final representation vector of the text; S8, evaluating the final representation vector of the text in terms of the label quality of the task data; S9, classifying the final representation vector of the text after the evaluation passes.

2. The method of claim 1, wherein the local key information and pre-trained text classification method is characterized by, In step S1, the method for obtaining the pre-training model with the highest matching degree with the task data comprises the following steps: From several candidate pre-training models based on related corpus data in the field, a candidate pre-training model with the highest score is selected as the first pre-training model by calculating the matching degree score of the candidate pre-training model and the task data; Wherein, the matching degree score score=a1*coverage_rate+a2*diversity_rate; Wherein, a1 and a2 are hyperparameters, coverage_rate is a coverage rate index, and diversity_rate is a diversity index; The coverage indicator coverage rate = 1 - YXL <unk> / RW <c>; wherein YXL <unk>To use a candidate pre-trained model to segment a training corpus, unknown words appear after segmentation <unk>the sum of the number of times, RW <c>The sum of the number of times each word appears after the task data is segmented using the candidate pre-training model;< / c> < / unk> < / unk> < / c> < / unk> The richness indicator diversity_rate = YXL <qc> / RW <qc>; wherein YXL <qc>Total number of words in the dictionary that comes with the candidate pre-trained model, RW <qc>The total number of words after the task data is segmented and deduplicated after being input into the candidate pre-training model.< / qc> < / qc> < / qc> < / qc> 3.The method of claim 1, wherein, In step S2, in the process of obtaining the second pre-training model by training the first pre-training model, the randomly selected keyword sequence is first masked, and then part of the words in the remaining content are randomly selected and masked, so that the second pre-training model can predict the masked keyword sequence and the random words; wherein the loss function Loss during training is Loss = r1*Loss1 + r2*Loss2; wherein r1 and r2 are hyperparameters, Loss1 is the cross-entropy loss after randomly selecting part of the words at a certain mask rate and masking the selected words, and Loss2 is the cross-entropy loss of randomly selecting a keyword sequence and masking the selected keyword sequence.

4. The method of claim 1, wherein the local key information and pre-trained text classification method is characterized by, The method for extracting and learning the local key information of the second training model is: obtaining the full-text representation vector representing the overall semantics of the text by the second training model, taking the full-text representation vector as the input of the BiLSTM-CRF model, using LSTM to capture context information from two angles of positive sequence and reverse sequence, using conditional random field CRF to obtain the category of each word in the text, obtaining the word sequence belonging to the same local key category as the keyword sequence of the local key category, and taking all the keyword sequences in the local key category as the local key information of the local key category.

5. The method of claim 1, wherein the local key information and pre-trained text classification method is characterized by, After the text is segmented, the word2vec model is used to train the word vector to obtain the word vector of each word in the text.

6. The method of claim 1, wherein the local key information and pre-trained text classification method is characterized by, In step S5, the following steps are included: setting the local key category corresponding to the jth text classification under the ith local attribute as Rij, and setting c keyword sequences extracted in the local key category Rij, wherein the zth keyword sequence is composed of p words; The word vectors of the p words in the zth keyword sequence are averaged to obtain the intra-group average word vector of the zth keyword sequence; The intra-group average word vectors of the c keyword sequences in the local key category Rij are obtained, and the c intra-group average word vectors are averaged to obtain the local key vector of the local key category Rij.

7. The method of claim 1, wherein the local key information and pre-trained text classification method is characterized by, In step S6, the local attributes include verb attributes and noun attributes, and the vector splicing is performed on all local key vectors under the verb attributes and the noun attributes respectively to obtain the local representation vector of the verb attributes and the local representation vector of the noun attributes.

8. The method of claim 7, wherein the local key information is fused with the pre-trained text classification method by, In step S7, the following steps are included: the full-text representation vector of the text obtained by the second pre-training model and the local representation vector of the verb attributes and the local representation vector of the noun attributes are spliced to obtain the final representation vector of the text.

9. The method of claim 1, wherein the local key information and pre-trained text classification method is characterized by, In step S8, the steps of evaluating the data labeling quality include: S81, using the K-MEANS clustering algorithm to obtain the center point corresponding to the text classification category, and obtaining the clustering result corresponding to each text; a certain proportion of basic seed samples RIO closest to the center point are obtained; S82, the classification accuracy Q0 of the basic seed sample RIO is calculated, and the basic verification threshold W0 is set. When Q0 < W0, the labeling quality is unqualified, and the data labeling is performed again; when Q0 ≥ W0, the verification is qualified, and step S83 is entered; S83, obtain a sample with a certain similarity to the basic seed sample RIO as a first-order sample RII, calculate the classification accuracy Q1 of the first-order sample RII, set a first-order verification threshold W1, when Q1W1, the labeling quality is unqualified, and data labeling is performed again; when Q1≥W1, the verification is qualified, and step S84 is entered; S84, obtain a sample with a certain similarity to the first-order sample RII as a second-order sample RII, calculate the classification accuracy Q2 of the second-order sample RII, set a second-order verification threshold W2, when Q2W2, the labeling quality is unqualified, and data labeling is performed again; when Q2≥W2, the verification is qualified, and the labeling data quality evaluation passes.

10. The method of claim 9, wherein the local key information and pre-trained text classification method is characterized by ； In step S81, a sampling rate α_1 (0<α_1<1) is used to obtain a sample closest to the center point as a basic seed sample; In step S83, a sampling rate α_2 (0<α_2<1) is used to obtain a sample with the highest similarity to the basic seed sample as a first-order sample; In step S84, a sampling rate α_3 (0<α_3<1) is used to obtain a sample with the highest similarity to the first-order sample as a second-order sample; Wherein, α_1<α_2<α_3; set α_2=sin(α_1*π / 2), α_3=sin(α_2*π / 2).

Images