A text classification method and device based on word weight statistics and pre-training model

Through the combination of word frequency-inverse text frequency algorithm and pre-trained BERT model, the problem of background information and noise interference in long text classification is solved, the accuracy and efficiency of text classification are improved, and the development of natural language processing technology is promoted.

CN118535743BActive Publication Date: 2024-11-05ZHEJIANG LAB
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411004264.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2024-11-05
Estimated Expiration
2044-07-25

AI Technical Summary

Technical Problem

The prior art is difficult to effectively process long texts in text classification, especially when there is a large amount of background information or noise, resulting in low classification accuracy, high rule-making cost, and limitations in machine learning algorithms in feature extraction and semantic understanding.

Method used

The word frequency-inverse text frequency algorithm based on word weight statistics is used to calculate the importance of each word in the vocabulary table relative to each label, combined with the pre-trained BERT model and attention mechanism, and text semantic vectors are integrated to improve the classification accuracy of the model.

Benefits of technology

Through the generation word-label weight dictionary and the use attention mechanism, the text classification model's understanding and representation ability of text semantics is improved, the contribution of text information is dynamically adjusted, and the classification accuracy and efficiency are improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118535743B_ABST
    Figure CN118535743B_ABST
Patent Text Reader

Abstract

The present invention discloses a text classification method and device based on word weight statistics and pre-training model, which mainly includes the following steps: obtaining text corpus and storing it in a text database; performing data cleaning on the text corpus to obtain a text data set; using the word frequency-inverse label set frequency algorithm to generate a word-label weight dictionary; using the word-label weight dictionary to generate a text weight score vector, combining the pre-processed text with the text classification model to obtain a text classification prediction result, and iteratively training the model; using the pre-trained text classification model to classify new input text. The present invention takes the importance of each word relative to each label as prior knowledge, and then uses the attention mechanism in the text classification model to integrate the prior knowledge into the text semantic representation output by the pre-trained BERT model, which can further improve the accuracy and efficiency of text classification.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of text classification, and in particular relates to a text classification method and device based on word weight statistics and a pre-training model. Background Art

[0002] Text classification is an extremely basic and important technology in the field of natural language processing (NLP), and it is also the focus and hotspot of current research. With the rapid development of informatization today, the massive amount of text data, such as posts on social media, news articles, academic papers, and user comments, is growing at an alarming rate. How to efficiently and accurately process, understand, and classify these texts is not only related to the efficiency of information retrieval, but also the key to promoting the development of advanced applications such as intelligent question answering, sentiment analysis, and public opinion monitoring.

[0003] Traditional text classification methods mainly rely on statistical analysis and rule-making. These methods can achieve certain results when dealing with short texts or texts with relatively simple structures. However, when faced with long text datasets, especially those containing a lot of background information or irrelevant paragraphs, it is difficult to distinguish which information is the core content and which is redundant or noise. Rule-making relies on expert knowledge bases, which are not only costly to build, but also difficult to cover all possible text patterns, and have low classification accuracy.

[0004] With the rise of machine learning technology, researchers have tried to apply statistical learning algorithms to text classification tasks. However, there are often long texts in the numerous and complex text data, and these texts contain a large proportion of background fields, that is, paragraphs that do not contribute significantly to the label to which the text belongs. These paragraphs even affect the model's misleading judgment of the label. On the one hand, machine learning algorithms rely on a large amount of labeled data for training, and the cost of long text labeling is high and time-consuming; on the other hand, due to the high proportion of background fields in long texts, these fields have a weak correlation with text labels and may even mislead the model, resulting in a decrease in classification accuracy. In addition, machine learning algorithms often have limitations in feature extraction and representation learning, making it difficult to fully capture the semantic information and contextual relationships in the text, resulting in a relatively low accuracy rate for text classification, which can no longer meet usage requirements.

[0005] In recent years, with the rapid development of deep learning technology, text classification models based on neural networks have gradually become a research hotspot. With its powerful nonlinear modeling ability and automatic feature extraction ability, neural networks have achieved remarkable results in image recognition, speech recognition and other fields, and have also brought new hope to text classification. However, the application of neural networks to long text classification also faces many challenges. First, the redundant information and noise fields in long texts interfere with the learning process of neural network models. Although neural networks can automatically learn text features, too much useless information will cause the model to overfit and reduce generalization ability. Secondly, the semantic structure and contextual relationship of long texts are more complex, requiring the model to have stronger understanding and reasoning capabilities. However, the current neural network model still has shortcomings in dealing with long-distance dependencies and it is difficult to fully capture the deep semantic information in the text. Therefore, it is urgent to develop more advanced neural network models and algorithms to achieve efficient and accurate text classification. Summary of the invention

[0006] In view of the above, the purpose of the present invention is to provide a text classification method and device based on word weight statistics and pre-trained models, using an improved word frequency-inverse text frequency algorithm to successively count the prior knowledge of each word in the vocabulary relative to each label, and then using an attention mechanism to integrate the prior knowledge into the text semantic vector output by the pre-trained model, thereby improving the accuracy of model text classification.

[0007] In order to achieve the above-mentioned invention object, the technical solution provided by the present invention is as follows:

[0008] In a first aspect, an embodiment of the present invention provides a text classification method based on word weight statistics and a pre-trained model, comprising the following steps:

[0009] Obtain text corpus and store it in a text database;

[0010] Perform data cleaning on the text corpus to obtain a text data set;

[0011] Divide the text dataset into several subsets according to the text classification labels, segment each text in the subset and generate a vocabulary of the text dataset, calculate the importance value of each word in the vocabulary relative to each label and generate a word-label weight dictionary;

[0012] Construct a text classification model, where the weight score calculation layer uses the word-label weight dictionary to generate the weight score vector of each text in the subset, the pre-trained BERT model layer extracts the semantic representation of each text in the subset, the attention mechanism layer combines the weight score vector and semantic representation of each text to extract the text semantic representation intermediate result, the classification layer outputs the text classification prediction result according to the text semantic representation intermediate result, and the loss function is used to pre-train the text classification model;

[0013] Use the pre-trained text classification model to classify new input text.

[0014] Preferably, the text data set is divided into several subsets according to the text classification labels, each text in the subset is segmented and a vocabulary of the text data set is generated, including:

[0015] The text dataset The text corpus after data cleaning is divided into subsets according to the text classification labels , For the subsets, texts with the same label are grouped into the same subset , For the In the subset a text;

[0016] Use the word segmentation tool to segment each text in the subset Perform word segmentation to generate word sequence representation of text , For the Subset The first words;

[0017] Traverse the word sequence representation of each text in each subset , count the frequency of each word, delete words with a frequency less than the set threshold, and generate a vocabulary list for the text dataset , The first word.

[0018] Preferably, the step of calculating the importance value of each word in the vocabulary relative to each tag and generating a word-tag weight dictionary includes:

[0019] The word frequency-inverse tag set frequency algorithm is used to calculate the importance value of each word in the vocabulary relative to each tag. The calculation formula of the word frequency-inverse tag set frequency algorithm is as follows:

[0020] ,

[0021] In the formula, Words Relative to the The importance value of the text classification label, For the subset Chinese words The number of occurrences, For subset The total number of words in is the number of subsets divided, which is also the number of labels. To contain words The number of subsets of Words Length, is a constant;

[0022] According to the calculated importance value Generate word-label weight dictionary:

[0023] ,

[0024] In the formula, the word-label weight dictionary represents the word The weight relative to each label.

[0025] Preferably, the step of generating a weight score vector for each text in the subset using a word-tag weight dictionary in the weight score calculation layer includes:

[0026] Calculate the weight score of each text in the subset relative to each tag based on the word-tag weight dictionary , and generate a text-label weight score vector :

[0027] ,

[0028] ,

[0029] In the formula, For the Subset Text Relative to the The weight score of the label, For Text The word sequence representation generated after word segmentation The words, Words Relative to the The importance value of the text classification label, For the The text-label weight score vector for the text.

[0030] Preferably, extracting the semantic representation of each text in the subset in the pre-trained BERT model layer includes:

[0031] The text you enter The word sequence representation generated after word segmentation Vectorize the text to get:

[0032] ,

[0033] In the formula, For the The text vector of the text, for Middle word vectors;

[0034] Encoder of the pre-trained BERT model to encode the text vector Extract semantic representation of text:

[0035] ,

[0036] In the formula, For the The semantic representation of a text, Encoder for the pre-trained BERT model.

[0037] Preferably, extracting the text semantic representation intermediate result by combining the weight score vector and semantic representation of each text at the attention mechanism layer includes:

[0038] Use the attention mechanism to transform the score vector of the text Incorporating semantic representation into text middle:

[0039] ,

[0040] In the formula, The text semantics extracted by the attention mechanism layer represents the intermediate result. For the splicing symbol, To get the first row of the matrix, For custom projection matrix.

[0041] Preferably, outputting the text classification prediction result according to the text semantic representation intermediate result at the classification layer includes:

[0042] The text semantics output of the attention mechanism layer represents the intermediate result Input classification layer to get input text The probability score relative to each label :

[0043] ,

[0044] In the formula, is a linear function;

[0045] Select The label category corresponding to the probability with the largest score is taken as the text classification result.

[0046] In a second aspect, in order to achieve the above-mentioned purpose of the invention, an embodiment of the present invention further provides a text classification device based on word weight statistics and a pre-trained model, comprising: a text database module, a text preprocessing module, a word weight statistics module, a text modeling module and a text classification module;

[0047] The text database module is used to obtain text corpus and store it in a text database;

[0048] The text preprocessing module is used to perform data cleaning on the text corpus to obtain a text data set;

[0049] The word weight statistics module is used to divide the text data set into several subsets according to the text classification labels, segment each text in the subset and generate a word list of the text data set, calculate the importance value of each word in the word list relative to each label and generate a word-label weight dictionary;

[0050] The text modeling module is used to build a text classification model, wherein a weight score vector of each text in a subset is generated using a word-label weight dictionary in a weight score calculation layer, a semantic representation of each text in a subset is extracted in a pre-trained BERT model layer, an intermediate result of text semantic representation is extracted in an attention mechanism layer in combination with the weight score vector and semantic representation of each text, a text classification prediction result is output according to the intermediate result of text semantic representation in a classification layer, and a text classification model is pre-trained using a loss function;

[0051] The text classification module is used to classify new input text using a pre-trained text classification model.

[0052] On the third aspect, in order to achieve the above-mentioned purpose of the invention, an embodiment of the present invention also provides a text classification device based on word weight statistics and pre-trained models, including a memory and one or more processors, the memory is used to store computer programs, and the processor is used to implement the above-mentioned text classification method based on word weight statistics and pre-trained models when executing the computer program.

[0053] In a fourth aspect, in order to achieve the above-mentioned purpose of the invention, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computer, the above-mentioned text classification method based on word weight statistics and pre-trained model is implemented.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] (1) The present invention innovatively proposes a word frequency-inverse label set frequency algorithm, which calculates the importance of each word relative to each label in a statistical way and finally generates a word-label weight dictionary. The dictionary contains a wealth of prior knowledge required for text classification. Applying this prior knowledge in the model text classification process can improve the accuracy of text classification.

[0056] (2) When using the model for text classification, the present invention uses the word-label weight dictionary as prior knowledge, and then calculates the importance of the entire text to each label. It combines the pre-trained BERT model to extract the semantic representation of the text context information, improves the model's understanding and representation capabilities of the text semantics, and inputs all the text information into a model containing an attention mechanism for training and classification. It dynamically adjusts the contribution of different parts of the text to the classification results, thereby improving the accuracy and efficiency of the model's text classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0058] Figure 1 It is a flowchart of a text classification method based on word weight statistics and a pre-training model provided by an embodiment of the present invention;

[0059] Figure 2 It is a schematic diagram of a process of calculating a word-label weight dictionary provided by an embodiment of the present invention;

[0060] Figure 3 It is a schematic diagram of the pre-training process of the text classification model provided by an embodiment of the present invention;

[0061] Figure 4 It is a flowchart of text classification using a pre-trained text classification model provided by an embodiment of the present invention;

[0062] Figure 5 It is a structural schematic diagram of a text classification device based on word weight statistics and a pre-training model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0063] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0064] The inventive concept of the present invention is: in view of the problem of low accuracy of text classification methods in the prior art, the embodiments of the present invention provide a text classification method and device based on word weight statistics and a pre-trained model, and innovatively propose a word frequency-inverse label set frequency algorithm based on word frequency-inverse document frequency. The algorithm successively counts the importance of each word in the vocabulary relative to each label as prior knowledge, and then uses the attention mechanism to integrate the prior knowledge into the text semantic representation output by the pre-trained BERT model, so that the output text semantic information is expressed more richly, thereby improving the accuracy and efficiency of model text classification and promoting the further development and application of natural language processing technology.

[0065] Figure 1 FIG. 1 is a flow chart of a text classification method based on word weight statistics and a pre-training model provided by an embodiment of the present invention. Figure 1 As shown, the embodiment provides a text classification method based on word weight statistics and a pre-trained model, comprising the following steps:

[0066] S1, obtain text corpus and store it in the text database.

[0067] In the embodiment, a large amount of Chinese text corpus is obtained and stored in a text database.

[0068] S2, clean the text corpus to obtain a text dataset.

[0069] In the embodiment, cleaning the text corpus in the text database mainly includes: unifying the capitalization of English letters, unifying Chinese and English punctuation marks, converting traditional Chinese to simplified Chinese, and deleting garbled characters and unprintable characters.

[0070] S3, generates a word-tag weight dictionary using the word frequency-inverse tag set frequency algorithm.

[0071] In the embodiment, Figure 2 As shown, the text dataset is divided into several subsets according to the text classification labels, each text in the subset is segmented and a vocabulary of the text dataset is generated, the importance value of each word in the vocabulary relative to each label is calculated and a word-label weight dictionary is generated.

[0072] S3.1, divide the text dataset into subsets. The text corpus after data cleaning is divided into subsets according to the text classification labels , For the subsets, texts with the same label are grouped into the same subset , For the In the subset A text.

[0073] S3.2, Tokenize the text in the subset. Use a tokenization tool to tokenize each piece of text in the subset to generate a word sequence representation of the text , is the th word in the th piece of text in the th subset.

[0074] S3.3, Calculate the weights using the term frequency - inverse label set frequency algorithm. Traverse the word sequence representation of each piece of text in each subset , count the term frequency of each word, delete the words with a term frequency less than a set threshold. In the embodiment, the words with a term frequency less than or equal to 2 are deleted to generate a vocabulary of the text data set , is the th word in the vocabulary. Use the term frequency - inverse label set frequency algorithm to calculate the importance value of each word in the vocabulary relative to each label. The formula of the term frequency - inverse label set frequency algorithm is as follows:

[0075] ,

[0076] In the formula, is the importance value of word relative to the th text classification label, is the number of times word appears in subset , is the total number of words in subset , is the number of subsets divided, which is also the number of labels, is the number of subsets containing word , is the length of word , is a constant, i.e., a hyperparameter that needs to be adjusted through experiments. Introducing serves to take the length of word as a factor of the weight of word in the text, which can reduce the attention of the model to some high - frequency words such as "is", "of", etc. that have little impact on semantics.

[0077] S3.4, Generate a word - label weight dictionary. Generate a word - label weight dictionary according to the calculated importance value :

[0078] ,

[0079] In the formula, the word - label weight dictionary represents word The weight relative to each label.

[0080] S4, using the word-label weight dictionary to generate a text weight score vector and combining it with the preprocessed text through the text classification model to obtain the text classification prediction result, and iteratively train the model.

[0081] In the embodiment, Figure 3 As shown, a text classification model is constructed, wherein the word-label weight dictionary is used in the weight score calculation layer to generate a weight score vector for each text in the subset, the semantic representation of each text in the subset is extracted in the pre-trained BERT model layer, the weight score vector and semantic representation of each text are combined in the attention mechanism layer to extract the intermediate result of the text semantic representation, the text classification prediction result is output according to the intermediate result of the text semantic representation in the classification layer, the text classification model is pre-trained using the loss function, and the model parameters are optimized through iteration.

[0082] S4.1, calculate the weight score of each text in the subset relative to each tag based on the word-tag weight dictionary , and generate a text-label weight score vector :

[0083] ,

[0084] ,

[0085] In the formula, For the Subset Text Relative to the The weight score of the label, For Text The word sequence representation generated after word segmentation The words, Words Relative to the The importance value of the text classification label, For the The text-label weight score vector for the text.

[0086] S4.2, input text The word sequence representation generated after word segmentation Vectorize the text to get:

[0087] ,

[0088] In the formula, For the The text vector of the text, for Middle word vectors.

[0089] S4.3, using the encoder of the pre-trained BERT model to encode the text vector Extract semantic representation of text:

[0090] ,

[0091] In the formula, For the The semantic representation of a text, Encoder for the pre-trained BERT model.

[0092] S4.4, use the attention mechanism to transform the score vector of the text Incorporating semantic representation into text middle:

[0093] ,

[0094] In the formula, The text semantics extracted by the attention mechanism layer represents the intermediate result. For the splicing symbol, To get the first row of the matrix, For custom projection matrix.

[0095] S4.5, the text semantic representation intermediate result output by the attention mechanism layer Input classification layer to get input text The probability score relative to each label , and get the probability distribution of text-label:

[0096] ,

[0097] In the formula, is a linear function;

[0098] Select The label category corresponding to the probability with the largest score is taken as the text classification result.

[0099] S4.6, using the cross entropy loss function crossentropy Calculate predicted values With the true label Loss , and provide feedback to complete a round of training for the text classification model:

[0100] ,

[0101] The text dataset is divided into training set and test set. The training set is used to iteratively train the text classification model, and the test set is used to verify the model performance.

[0102] S5, classify the new input text using the pre-trained text classification model.

[0103] In the embodiment, Figure 4 As shown in the figure, after the newly input text to be classified is preprocessed, the text-label weight score vector is calculated according to the word-label weight dictionary, and the text-label probability distribution is predicted using the pre-trained text classification model, and the category with the highest probability score is selected as the final text classification result.

[0104] In summary, a text classification method based on word weight statistics and pre-trained models generates a word-label weight dictionary by using an innovative word frequency-inverse label set frequency algorithm, and applies the rich prior knowledge required for text classification in the dictionary to the model text classification process, which can improve the accuracy of text classification. At the same time, the semantic representation of text context information is extracted in combination with the pre-trained BERT model to improve the model's understanding and representation capabilities of text semantics, and all text information is input into a model containing an attention mechanism for training and classification, and the contribution of different parts of the text to the classification results is dynamically adjusted, thereby improving the accuracy and efficiency of the model text classification. Using the pre-trained text classification model of the present invention, it will be able to better handle the complex structure and noise interference in long texts, and realize more intelligent and personalized text classification services. At the same time, with the integration of cross-domain knowledge and the continuous expansion of application scenarios, the present invention is expected to play an important role in more fields and promote the further development and application of natural language processing technology.

[0105] Based on the same inventive concept, Figure 5 As shown, an embodiment of the present invention also provides a text classification device 500 based on word weight statistics and a pre-trained model, including: a text database module 510, a text preprocessing module 520, a word weight statistics module 530, a text modeling module 540 and a text classification module 550.

[0106] The text database module 510 is used to obtain text corpus and store it in a text database.

[0107] The text preprocessing module 520 is used to perform data cleaning on the text corpus to obtain a text data set.

[0108] The word weight statistics module 530 is used to divide the text data set into several subsets according to the text classification labels, segment each text in the subset and generate a vocabulary of the text data set, calculate the importance value of each word in the vocabulary relative to each label and generate a word-label weight dictionary.

[0109] The text modeling module 540 is used to construct a text classification model, wherein the weight score calculation layer uses the word-label weight dictionary to generate a weight score vector for each text in the subset, the pre-trained BERT model layer extracts the semantic representation of each text in the subset, the attention mechanism layer combines the weight score vector and the semantic representation of each text to extract the intermediate result of the text semantic representation, the classification layer outputs the text classification prediction result based on the intermediate result of the text semantic representation, and the text classification model is pre-trained using the loss function.

[0110] The text classification module 550 is used to classify new input text using a pre-trained text classification model.

[0111] Based on the same inventive concept, an embodiment of the present invention also provides a text classification device based on word weight statistics and pre-trained models, including a memory and one or more processors, the memory is used to store computer programs, and the processor is used to implement the above-mentioned text classification method based on word weight statistics and pre-trained models when executing the computer program.

[0112] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computer, the above-mentioned text classification method based on word weight statistics and pre-training model is implemented.

[0113] It should be noted that the text classification device based on word weight statistics and pre-trained model, the text classification equipment based on word weight statistics and pre-trained model and the computer-readable storage medium provided in the above embodiments all belong to the same inventive concept as the text classification method based on word weight statistics and pre-trained model. The specific implementation process is detailed in the embodiment of the text classification method based on word weight statistics and pre-trained model, which will not be repeated here.

[0114] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A text classification method based on word weight statistics and pre-training model, characterized in that: The following steps are involved: Obtain text corpus and store it in a text database; Perform data cleaning on the text corpus to obtain a text data set; Divide the text dataset into several subsets according to the text classification labels, segment each text in the subset and generate a vocabulary of the text dataset, including: The text corpus after data cleaning is divided into subsets according to the text classification labels , For the subsets, texts with the same label are grouped into the same subset , For the In the subset texts, and use the word segmentation tool to segment each text in the subset Perform word segmentation to generate word sequence representation of text , For the Subset The first words, traverse the word sequence representation of each text in each subset , count the frequency of each word, delete words with a frequency less than the set threshold, and generate a vocabulary list for the text dataset , The first words; then, the importance value of each word in the vocabulary relative to each tag is calculated and a word-tag weight dictionary is generated, including: using the word frequency-inverse tag set frequency algorithm to calculate the importance value of each word in the vocabulary relative to each tag, and the calculation formula of the word frequency-inverse tag set frequency algorithm is as follows: , In the formula, Words Relative to the The importance value of the text classification label, For the subset Chinese words The number of occurrences, For subset The total number of words in is the number of subsets divided, which is also the number of labels. To contain words The number of subsets of Words Length, is a constant, and the importance value is calculated based on Generate word-label weight dictionary: , In the formula, the word-label weight dictionary represents the word The weight relative to each label; Construct a text classification model, where the weight score calculation layer uses the word-label weight dictionary to generate the weight score vector of each text in the subset, the pre-trained BERT model layer extracts the semantic representation of each text in the subset, the attention mechanism layer combines the weight score vector and semantic representation of each text to extract the text semantic representation intermediate result, the classification layer outputs the text classification prediction result according to the text semantic representation intermediate result, and the loss function is used to pre-train the text classification model; Use the pre-trained text classification model to classify new input text.

2. The text classification method based on word weight statistics and pre-training model according to claim 1 is characterized in that: The weight score calculation layer generates a weight score vector for each text in the subset using the word-label weight dictionary, including: Calculate the weight score of each text in the subset relative to each tag based on the word-tag weight dictionary , and generate a text-label weight score vector : , , In the formula, For the Subset Text Relative to the The weight score of the label, For text The word sequence representation generated after word segmentation The words, Words Relative to the The importance value of the text classification label, For the The text-label weight score vector for the text.

3. The text classification method based on word weight statistics and pre-training model according to claim 2 is characterized in that: The semantic representation of each text in the subset extracted in the pre-trained BERT model layer includes: The text you enter The word sequence representation generated after word segmentation Vectorize the text to get: , In the formula, For the The text vector of the text, for Middle word vectors; Encoder of the pre-trained BERT model to encode the text vector Extract semantic representation of text: , In the formula, For the The semantic representation of a text, Encoder for the pre-trained BERT model.

4. The text classification method based on word weight statistics and pre-training model according to claim 3 is characterized in that: The intermediate result of extracting the text semantic representation by combining the weight score vector and semantic representation of each text at the attention mechanism layer includes: Use the attention mechanism to transform the score vector of the text Incorporating semantic representation into text middle: , In the formula, The text semantics extracted by the attention mechanism layer represents the intermediate result. For the splicing symbol, To get the first row of the matrix, For custom projection matrix.

5. The text classification method based on word weight statistics and pre-training model according to claim 4 is characterized in that: Outputting the text classification prediction result according to the text semantic representation intermediate result at the classification layer includes: The text semantics output of the attention mechanism layer represents the intermediate result Input classification layer to get input text The probability score relative to each label : , In the formula, is a linear function; Select The label category corresponding to the probability with the largest score is taken as the text classification result.

6. A text classification device based on word weight statistics and pre-training model, characterized in that: include: Text database module, text preprocessing module, word weight statistics module, text modeling module and text classification module; The text database module is used to obtain text corpus and store it in a text database; The text preprocessing module is used to perform data cleaning on the text corpus to obtain a text data set; The word weight statistics module is used to divide the text data set into several subsets according to the text classification labels, segment each text in the subset and generate a word list of the text data set, including: The text corpus after data cleaning is divided into subsets according to the text classification labels , For the subsets, texts with the same label are grouped into the same subset , For the In the subset texts, and use the word segmentation tool to segment each text in the subset Perform word segmentation to generate word sequence representation of text , For the Subset The first words, traverse the word sequence representation of each text in each subset , count the frequency of each word, delete words with a frequency less than the set threshold, and generate a vocabulary list for the text dataset , The first words; then, the importance value of each word in the vocabulary relative to each tag is calculated and a word-tag weight dictionary is generated, including: using the word frequency-inverse tag set frequency algorithm to calculate the importance value of each word in the vocabulary relative to each tag, and the calculation formula of the word frequency-inverse tag set frequency algorithm is as follows: , In the formula, Words Relative to the The importance value of the text classification label, For the subset Chinese words The number of occurrences, For subset The total number of words in is the number of subsets divided, which is also the number of labels. To contain words The number of subsets of Words Length, is a constant, and the importance value is calculated based on Generate word-label weight dictionary: , In the formula, the word-label weight dictionary represents the word The weight relative to each label; The text modeling module is used to build a text classification model, wherein a weight score vector of each text in a subset is generated using a word-label weight dictionary in a weight score calculation layer, a semantic representation of each text in a subset is extracted in a pre-trained BERT model layer, an intermediate result of text semantic representation is extracted in an attention mechanism layer in combination with the weight score vector and semantic representation of each text, a text classification prediction result is output according to the intermediate result of text semantic representation in a classification layer, and a text classification model is pre-trained using a loss function; The text classification module is used to classify new input text using a pre-trained text classification model.

7. A text classification device based on word weight statistics and pre-trained models, comprising a memory and one or more processors, wherein the memory is used to store a computer program, characterized in that: The processor is used to implement the text classification method based on word weight statistics and pre-training model described in any one of claims 1-5 when executing the computer program.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a computer, the text classification method based on word weight statistics and a pre-trained model described in any one of claims 1 to 5 is implemented.