Data classification method and system based on keyword weight matching

Abstract

The present application relates to the technical field of data processing, and discloses a data classification method and system based on keyword weight matching, the method comprising preprocessing input target data to obtain key features; calculating trace information of the key features in the target data, and matching the key features with corresponding key weights using a first algorithm according to the trace information; inputting the key features into a neural network model to extract generalized features of the target data; selecting classification features from the generalized features using a second algorithm according to the key weights and the trace information; inputting the key features and the corresponding classification features into a classifier, and using the classifier to classify the target data to obtain classification results; introducing a key weight matching mechanism for keyword features, and assigning different feature weights to each key feature according to the importance of the key features; automatically extracting features of the target data; and having strong adaptability without requiring a large amount of training data, and being applicable to various data classification scenarios.

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and in particular to a data classification method and system based on keyword weight matching. Background Art

[0002] With the rapid development of the Internet, the amount of text data has exploded. How to extract valuable information from this massive amount of text data has become an urgent problem to be solved. Article classification, as a common text processing method, can organize and classify a large amount of text according to certain standards, so that users can quickly find the information they need.

[0003] Existing article classification methods mainly include traditional machine learning-based methods and deep learning-based methods. Among them, the traditional machine learning-based methods require manual feature extraction, which is time-consuming and labor-intensive, and the classification effect is affected by the quality of feature extraction; although the deep learning-based methods can automatically extract features, they require a large amount of training data and the model training time is long.

[0004] Therefore, a data classification method that can reduce time consumption is needed. Summary of the invention

[0005] In order to reduce the time consumption of data classification, the present application provides a data classification method and system based on keyword weight matching.

[0006] In a first aspect, the present application provides a data classification method based on keyword weight matching, which adopts the following technical solution:

[0007] A data classification method based on keyword weight matching includes the following steps:

[0008] Preprocess the input target data to obtain key features;

[0009] Calculating the trace information of the key feature in the target data, and matching the key feature with a corresponding key weight using a first algorithm according to the trace information;

[0010] Inputting the key features into a Transformer neural network model to extract generalized features of the target data;

[0011] Selecting classification features from the general features using a second algorithm according to the key weights and the trace information;

[0012] Inputting the key features and the corresponding classification features into a classifier, and using the classifier to classify the target data to obtain a classification result;

[0013] Among them, the classification results corresponding to the target data have three-level indicators. The classification sub-items of the first-level indicators include whether the keywords are related to or unrelated to trade unions; the classification sub-items of the second-level indicators include ideological guidance, rights protection services, achievements, self-construction, and content that does not belong to trade unions; the classification sub-items of the third-level indicators are the classification contents in each classification sub-item in the second-level indicators.

[0014] By adopting the above technical solution, a key weight matching mechanism for keyword features is introduced, and different feature weights are assigned to each key feature according to its importance, making the classification of target data more accurate. The Transformer neural network model is used to automatically extract the features of target data and improve classification efficiency. It does not require a large amount of training data, has strong adaptability, and can be applied to various data classification scenarios.

[0015] Optionally, the preprocessing method comprises the following steps:

[0016] If the content of the target data is not a character, converting the content of the target data into a character;

[0017] Performing a word segmentation operation on the target data;

[0018] The word sequence of the text obtained after the word segmentation operation is used as the key feature.

[0019] By adopting the above technical solution, the non-character contents in the target data are converted into characters, and then the characters are segmented to obtain the required key features, namely keywords, which is beneficial to subsequent classification operations.

[0020] Optionally, the method further comprises the following steps:

[0021] According to the total character quantity of the target data, the word groups of the word segmentation are adjusted in a positive correlation, the more the total character quantity is, the more the word groups are, and the fewer the total character quantity is, the fewer the word groups are;

[0022] According to the total punctuation mark amount in the target data, the maximum character amount of the word group is adjusted inversely, the more the total punctuation mark amount, the smaller the maximum character amount, and the smaller the total character amount, the larger the maximum character amount;

[0023] The word groups are combinations of words of different lengths.

[0024] By adopting the above technical solution, the word segmentation rules are adaptively adjusted according to the amount of characters or punctuation marks in the target data, so that the word segmentation effect of the target data can be improved.

[0025] Optionally, the preprocessing method comprises the following steps:

[0026] Performing a stop word removal operation on the target data or the key feature;

[0027] The word sequence of the text obtained after the stop word operation is used as the key feature.

[0028] By adopting the above technical solution, the stop word operation is performed on the characters, so that the key features, namely the keywords, can be further optimized, and meaningless words can be removed, which is further beneficial to the subsequent classification operation.

[0029] Optionally, the method further comprises the following steps:

[0030] The word removal group of the word removal is adjusted inversely according to the total punctuation amount of the target data or the key feature. The more the total punctuation amount, the fewer the word removal group, and the fewer the total punctuation amount, the more the word removal group; the maximum character amount of the word removal group is adjusted in positive correlation according to the total character amount in the target data or the key feature. The more the total character amount, the more the maximum character amount, and the fewer the total character amount, the fewer the maximum character amount;

[0031] The word groups are word combinations of different lengths.

[0032] By adopting the above technical solution, the stop word removal rule is adaptively adjusted according to the number of characters or punctuation marks in the target data or key features, which can improve the stop word removal effect of the target data.

[0033] Optionally, the first algorithm comprises the following steps:

[0034] Obtaining an optimal data set in the trace information, wherein each of the key features corresponds to an optimal data set;

[0035] The key weight corresponding to each optimal data is calculated according to the proportion of the optimal data in the optimal data set.

[0036] By adopting the above technical solution, the key weight is calculated through the trace information of the key feature in the target data, so that the key weight is strongly associated with the key feature in the target data, which is conducive to ensuring the effective classification of the target data through the key weight.

[0037] Optionally, the method comprises the following steps:

[0038] Calculate the occurrence frequency value of each of the key features, where the occurrence frequency value = the amount of each of the key features * 100 / the total amount of all the key features;

[0039] Calculating the inverse document frequency of each of the key features;

[0040] The set of occurrence frequency values ​​or the set of inverse document frequencies is used as the optimal data set.

[0041] By adopting the above technical solution and using the occurrence frequency value or inverse stable frequency of the key feature as the optimal data set, the representativeness of the key feature to the characteristics of the target data can be highlighted.

[0042] Optionally, the method comprises the following steps:

[0043] Calculate the average word count of the key features;

[0044] If the average word number is greater than a preset reference word number, the set of occurrence frequency values ​​is used as the optimal data set; otherwise, the set of inverse document frequencies is used as the optimal data set.

[0045] By adopting the above technical solution, the rule of adaptively selecting the optimal data set according to the number of characters in the key feature can improve the representative effect of the key feature.

[0046] Optionally, the method comprises the following steps:

[0047] The reference word count is adjusted inversely according to the number of types of the key features; the more the number of types of the key features is, the fewer the reference word count is; the fewer the number of types of the key features is, the more the reference word count is.

[0048] By adopting the above technical solution, the number of reference words is adaptively adjusted according to the number of types of key features, which is conducive to selecting the optimal data set that represents the best effect.

[0049] Optionally, the second algorithm comprises the following steps:

[0050] The classification feature is obtained by calculating the product of the number of times each key feature appears in the target data and the key weight.

[0051] By adopting the above technical solution, accurate classification features are obtained, and articles can be accurately classified without a lot of training.

[0052] In the second aspect, the present application provides a data classification system based on keyword weight matching, which adopts the following technical solution:

[0053] A data classification system based on keyword weight matching includes a processor in which a program of any one of the above-mentioned data classification methods based on keyword weight matching is run.

[0054] In summary, the present application includes at least one of the following beneficial technical effects: introducing a key weight matching mechanism for keyword features, assigning different feature weights to each key feature according to the importance of the key feature, so that the target data classification is more accurate; using the Transformer neural network model to automatically extract the features of the target data and improve classification efficiency; it does not require a large amount of training data, has strong adaptability, and can be applied to various data classification scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a method step diagram of a data classification method based on keyword weight matching.

[0056] Figure 2 It is a step diagram of the preprocessing method.

[0057] Figure 3 This is a diagram of the steps for adjusting the maximum number of characters in a word group according to the total number of punctuation marks in the target data. DETAILED DESCRIPTION

[0058] Embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings.

[0059] In the description of this specification, the description with reference to the terms "certain embodiments", "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0060] The present application embodiment discloses a data classification method based on keyword weight matching, referring to Figure 1 , including the following steps:

[0061] The input target data is preprocessed to obtain key features. The target data is text, articles, or data containing text or articles, or data with Chinese character codes, wherein the Chinese character codes can be converted into text characters or punctuation marks. If the content of the target data is not a character (such as a number, special code, etc.), or contains non-required character codes (such as Chinese character codes that need to be converted into text characters or punctuation marks), then the code conversion is performed. For Chinese character codes, a coding library (such as Python's codecs or chardet library) can usually be used to detect and convert the code. The key features are keywords or key phrases, which can represent the core content or category of the data. The content of the keyword is identified to determine whether the keyword is related to the union. The determination method can use the template matching method, that is, there is a union word library for the keywords related to the union. If it can be matched from the union word library, it is related to the union. If it cannot be matched from the union word library, it is not related to the union.

[0062] Reference Figure 2 , the preprocessing is to remove noise, invalid characters, special symbols, etc. in the target data. The preprocessing method in this embodiment includes the following steps:

[0063] If the content of the target data is not a character, convert the content of the target data into a character. Convert all text to lowercase (if necessary), remove punctuation marks, numbers, etc., and only retain key words. In addition, if the data cannot be converted, delete it directly. Perform word segmentation on the target data; use the word sequence of the text obtained after the word segmentation operation as the key feature. The word segmentation operation can be implemented by using a natural language processing (NLP) library (such as Python's jieba word segmentation library). For example, 2 characters are a word, or 3, or 4 characters are a word.

[0064] You can use algorithms such as TF-IDF (term frequency-inverse document frequency) and TextRank to extract key features and calculate their weights. Based on the extracted key features, calculate the weight of each keyword or phrase. The weight is usually calculated based on the frequency of occurrence of the keyword in the document (TF) and the rarity of the keyword in the entire document set (IDF).

[0065] Reference Figure 3 , the word groups of the word segmentation are adjusted according to the total character volume of the target data. The more total characters, the more word groups, and the fewer total characters, the fewer word groups. If the total character volume is large, more word groups (i.e., finer-grained word segmentation results) tend to be generated to capture the information of the text more accurately. On the contrary, if the total character volume is small, fewer word groups can be generated (i.e., coarser-grained word segmentation results) because the amount of information in the text itself is small.

[0066] For example,

[0067] Text with many total characters: "Beijing is the capital of China, with a long history and rich cultural heritage. Tiananmen Square is the center of Beijing, visited by tens of thousands of tourists every day. The Forbidden City was the imperial palace of the Ming and Qing dynasties, and is now a museum that displays royal life and artworks from ancient China.".

[0068] After using Chinese word segmentation tools (such as Jieba) to segment words, the possible word groups are:

[0069] Beijing is the capital of China, and has a long history and rich cultural heritage. Tiananmen Square is the center of Beijing, and thousands of tourists come to visit every day. The Forbidden City was the imperial palace of the Ming and Qing dynasties, and now it is a museum that displays the royal life and works of art of ancient China.

[0070] Text with a small total number of characters:

[0071] "Beijing is very big."

[0072] After using the Chinese word segmentation tool to segment words, the word groups obtained are:

[0073] "Beijing", "very big", "."

[0074] The maximum character count of the word group is adjusted inversely according to the total punctuation amount in the target data. The more total punctuation, the smaller the maximum character count, and the smaller the total character count, the larger the maximum character count; the word group is a combination of words of different lengths. Punctuation marks are usually used to separate sentences or phrases in text, so they can be used as a reference for word boundaries. If the total punctuation amount is large, it means that the text contains more sentences or phrases, so the maximum character count of the word group tends to be set smaller, so that the text is cut more frequently during word segmentation and more word groups are generated. On the contrary, if the total punctuation amount is small, the maximum character count can be set larger to reduce the cutting frequency.

[0075] For example,

[0076] Text A with less punctuation:

[0077] "Beijing is the capital of China, with a profound cultural heritage and rich historical heritage."

[0078] Number of punctuation marks: 2 (commas and periods);

[0079] Text B with more punctuation marks:

[0080] "Beijing is the capital of China, with famous attractions such as Tiananmen Square and the Forbidden City. Tiananmen Square is the center of Beijing, attracting a large number of tourists every day; the Forbidden City was the imperial palace of the Ming and Qing dynasties and has now become a museum."

[0081] Number of punctuation marks: 7 (comma, period and semicolon);

[0082] The maximum number of characters for adjusting word groups is:

[0083] Since text A has fewer punctuation marks and a relatively simple sentence structure, a larger maximum character count can be set so that the word segmentation results tend to retain longer word groups. For example, the maximum character count can be set to 20, and the word segmentation results can be:

[0084] Beijing is the capital of China and has a profound cultural heritage and rich historical heritage.

[0085] For text B, since there are many punctuation marks and the sentence structure is relatively complex, a smaller maximum character count can be set so that the word segmentation result tends to cut out shorter word groups. For example, the maximum character count can be set to 10, and the word segmentation result can be:

[0086] Beijing is the capital of China, and it has many famous attractions, such as Tiananmen Square, the Forbidden City, etc. Among them, Tiananmen Square is the center of Beijing, attracting a large number of tourists to visit every day; while the Forbidden City was the imperial palace of the Ming and Qing dynasties, and now it has become a museum.

[0087] Converting non-character content in the target data into characters and then performing word segmentation on the characters can obtain the required key features, namely keywords, which is beneficial for subsequent classification operations. Adaptively adjusting the word segmentation rules according to the number of characters or punctuation marks in the target data can improve the word segmentation effect of the target data.

[0088] The preprocessing method further includes the following steps: performing a stop-word removal operation on the target data or key features; using the word sequence of the text obtained after the stop-word removal operation as the key feature. Remove common words in the text (such as "of", "is", "in", etc.), which appear frequently in the text but contribute little to the meaning of the text. The specific steps and methods of the stop-word removal operation: Prepare a stop-word list: The stop-word list is a collection of common words that need to be removed. A stop-word list can be prepared in advance, and these stop-words can be selected and adjusted according to the actual application scenarios and requirements. The stop-word list can be created manually or obtained from existing resources, such as the stop-word list provided by the NLTK library.

[0089] Select the method of removing stop words: Based on the stop-word list: directly search for the words in the stop-word list in the text and delete them from the text. Based on word frequency: sort the words according to their occurrence frequency in the text and remove the high-frequency words. This method usually sets a threshold, and the words with a frequency exceeding this threshold are regarded as stop words. Based on information entropy: calculate the information entropy of each word in the text, and the words with lower information entropy are considered as stop words. This method identifies stop words by measuring the importance of words to the text. Based on part of speech: filter according to the part of speech of the word, for example, remove stop words of specific parts of speech such as nouns and verbs. Based on machine learning: use machine learning algorithms to train a classification model to classify stop words and non-stop words, and then use this model to classify the words in the text and remove the stop words.

[0090] Implement the stop-word removal operation:

[0091] In Python, the NLTK library or a custom script can be used to implement the stop-word removal operation. First, load the stop-word list, and then traverse each word in the text. If the word is in the stop-word list, it is deleted from the text. For Chinese text, the jieba library can be used for word segmentation and combined with the stop-word list to remove stop words. The jieba library provides functions such as word segmentation and part-of-speech tagging, which can conveniently process Chinese text.

[0092] Among them, the stop-word group for word removal is adjusted inversely according to the total number of punctuation marks in the target data or key features. The more the total number of punctuation marks, the fewer the stop-word groups; the fewer the total number of punctuation marks, the more the stop-word groups. The maximum character amount of the stop-word group is adjusted positively according to the total number of characters in the target data or key features. The more the total number of characters, the more the maximum character amount; the fewer the total number of characters, the fewer the maximum character amount. The word group is a combination of words for removal with different lengths.

[0093] For example, text A (with fewer punctuation marks and moderate character amount):

[0094] "I like watching movies, especially science fiction movies."

[0095] After removing stop words: "I", "like", "watch", "movie", "especially", "science fiction", "movie"

[0096] Remove phrases (due to the small amount of punctuation and moderate character count, a larger maximum character count may be set): "I like watching movies", "Especially science fiction movies"

[0097] Text B (more punctuation and longer characters):

[0098] "Yesterday I went to the cinema and watched two movies. One was a science fiction movie called Interstellar, and the other was a comedy called Kung Fu Panda."

[0099] After removing stop words: "yesterday", "I", "go", "yes", "cinema", "watch", "yes", "two", "movie", "science fiction", "interstellar", "comedy", "Kung Fu Panda"

[0100] Remove phrases (due to the large number of punctuation marks, the character count is long, but a smaller maximum character count may be set to maintain sentence integrity): "I went to the cinema yesterday", "I watched two movies", "Interstellar, a science fiction film", "Kung Fu Panda, a comedy film"

[0101] Performing stop word removal on characters can further optimize key features, namely keywords, and remove meaningless words, which is further beneficial to subsequent classification operations. Adaptively adjusting the stop word removal rules based on the number of characters or punctuation marks in the target data or key features can improve the stop word removal effect of the target data.

[0102] Calculate the trace information of the key features in the target data, and use the first algorithm to match the key features with corresponding key weights according to the trace information. The trace information usually refers to a certain metric or indicator that can reflect the importance of keywords or key features in the text. This information can be calculated based on various factors such as the frequency, position, and contextual relationship of the keywords in the text.

[0103] The first algorithm includes the following steps:

[0104] Get the optimal data set in the trace information. Each key feature has an optimal data set. Calculate the occurrence frequency value of each key feature. The occurrence frequency value = the amount of each key feature * 100 / the total amount of all key features; calculate the inverse document frequency of each key feature; and take the set of occurrence frequency values ​​or the set of inverse document frequencies as the optimal data set.

[0105] For example, suppose there is a dataset containing 100 movie reviews covering different types of movies. In this case, two key features are selected for analysis: "Science Fiction" and "Comedy".

[0106] Calculate the frequency of occurrence value:

[0107] “Science fiction” appears 30 times in all reviews;

[0108] “Comedy” appears 20 times in all reviews;

[0109] The total number of all key characteristics (here only "Science Fiction" and "Comedy") is 50 times (30+20);

[0110] The frequency of occurrence of "science fiction" = (30*100) / 50 = 60%;

[0111] The frequency value of "comedy" = (20*100) / 50 = 40%.

[0112] Calculate the inverse document frequency (IDF):

[0113] Suppose there are 50 reviews that mention "science fiction" and 50 reviews that mention "comedy" (here it is simplified to mean that each feature appears in exactly half of the documents);

[0114] The total number of documents in the dataset D = 100;

[0115] The inverse document frequency of “science fiction” is IDF = log(100 / 50) = 1;

[0116] The inverse document frequency of "comedy" is IDF = log(100 / 50) = 1 (Note: in this simplified example, the IDFs of the two features are the same, but in real applications they may be different);

[0117] Among them, the inverse document frequency (IDF) is a measure of the general importance of a word, usually calculated as log(D / Dt), where D is the total number of documents and Dt is the number of documents containing the word. It is used to evaluate the importance of a word in a document collection. The fewer documents a word appears in a document collection, the larger the IDF value, indicating that the word is more representative or unique in the document collection.

[0118] Calculate the average word count of the key features. If the average word count is greater than the preset reference word count, the set of occurrence frequency values ​​is used as the optimal data set; otherwise, the set of inverse document frequencies is used as the optimal data set. The reference word count is adjusted inversely according to the number of key feature types; the more key feature types there are, the fewer reference words there are; the fewer key feature types there are, the more reference words there are. The reference word count is the number of keywords, which is generally 2, 3, 4, or 5 (relatively few).

[0119] Calculate the average word count:

[0120] "Sword" is 1 word;

[0121] "Apple" is 2 words;

[0122] "Cat" is 1 word;

[0123] "Ball" is 1 word;

[0124] Average number of words = (1+2+1+1) / 4 = 1.25 words.

[0125] A basic reference word count can be set to 3 words, and then the set of inverse document frequencies is used as the optimal data set. If the reference word count is set to 1 word, then the set of occurrence frequency values ​​is used as the optimal data set.

[0126] Assuming that the basic reference number of characters is 3 characters, the basic reference number of characters is adjusted according to the number of types. If the number of types is large (for example, greater than 10), the reference number of characters is reduced (for example, to 2 characters); if the number of types is small (for example, less than 5), the reference number of characters is increased (for example, to 4 characters). The specific value can be adjusted according to actual conditions and needs.

[0127] Using the occurrence frequency value or inverse stable frequency of key features as the optimal data set can highlight the representativeness of the key features to the characteristics of the target data. Adaptively selecting the rule of the optimal data set based on the number of characters in the key features can improve the representative effect of the key features. Adaptively adjusting the number of reference words based on the number of types of key features is conducive to selecting the optimal data set with the best representative effect.

[0128] The key weight corresponding to each optimal data is calculated based on the proportion of the optimal data in the optimal data set. The key weight is calculated through the trace information of the key feature in the target data, so that the key weight is strongly associated with the key feature in the target data, which is conducive to ensuring the effective classification of the target data through the key weight. For example, a movie review data set, the goal is to classify these reviews into two categories: "positive reviews" and "negative reviews". After a series of processing, the key features (such as "wonderful", "bad", "highly recommended", etc.) are extracted, and an "optimal data set" is determined.

[0129] First, count the number of times each key feature appears in the optimal data set and calculate their respective proportions. This proportion can be used as a preliminary basis for weight allocation. For example, in the optimal data set, the word "wonderful" appears 100 times in positive reviews, accounting for 10% of the total positive reviews; and the word "terrible" appears 80 times in negative reviews, accounting for 20% of the total negative reviews. Then, the preliminary weight of "wonderful" can be set to 0.1, and the preliminary weight of "terrible" can be set to 0.2.

[0130] Secondly, consider the specific performance of these key features in the target data, that is, their "trace information" in the target data. These "trace information" can include the frequency of occurrence of key features, context, co-occurrence relationship with other features, etc. For example, in the target data set, although the word "wonderful" does not appear many times, each time it appears, it is accompanied by strong positive emotional expressions, and rarely co-occurs with negative words. This shows that "wonderful" is highly representative of positive reviews. Therefore, you can consider increasing the weight of "wonderful". On the contrary, if the word "bad" has a complex co-occurrence relationship with other words in the target data set, and sometimes even appears in positive reviews, then it may not be so representative of negative reviews. Therefore, you can consider reducing the weight of "bad".

[0131] After the above two steps, the final weight of each key feature in the classification of target data can be determined. This weight will be used in subsequent machine learning models or classification algorithms to guide the model to classify target data and improve classification accuracy.

[0132] The key features are input into the Transformer neural network model to extract the generalized features of the target data. The generalized features can be the grammatical structure, semantic information, contextual relationship, etc. of the text. The classification features are selected from the generalized features using the second algorithm based on the key weights and trace information. Among them, the Transformer neural network model is a deep learning model based on the self-attention mechanism, which is mainly used to process sequence data, especially in the field of natural language processing (NLP), and has achieved remarkable results. Compared with traditional recurrent neural networks (RNNs) and convolutional neural networks (CNNs), the Transformer model can better capture long-term dependencies in sequences and supports parallel computing, thereby improving training efficiency and model performance.

[0133] The second algorithm includes the following steps:

[0134] The classification feature is obtained by calculating the product of the number of times each key feature appears in the target data and the key weight. With accurate classification features, the article can be accurately classified without a lot of training.

[0135] Steps for calculating classification features: Count the number of times each key feature appears in the target data; multiply the number of occurrences of the key feature by its corresponding key weight to obtain the classification feature score of the key feature; aggregate the classification feature scores of all key features (such as summing, averaging, etc.) to obtain the final classification feature vector.

[0136] The key features and the corresponding classification features are input into the classifier, and the classifier is used to classify the target data to obtain the classification results. Among them, the key features are characters and the classification features are numerical values. The classifier can use various machine learning algorithms, such as logistic regression, support vector machine, random forest, etc., to classify the target data according to the input features.

[0137] After inputting multiple union-related and union-irrelevant contents, a three-level indicator or a three-level system is obtained. The classification sub-items of the first-level indicator include the keywords related to and unrelated to the union, that is, the union-related content and the union-irrelevant content are divided into two classification sub-items. Then the union-related content is classified into secondary indicators. The classification sub-items of the secondary indicators include ideological guidance, rights protection services, meritorious service, and self-construction. The union-irrelevant content is classified into the classification sub-item of the secondary indicator that does not belong to the union content. The classification sub-items of the third-level indicator are the classification contents in each classification sub-item of the secondary indicator, namely ideological guidance, rights protection services, meritorious service, self-construction, and the lower-level keyword classification content or detailed content that does not belong to the union content.

[0138] A key weight matching mechanism for keyword features is introduced to assign different feature weights to each key feature according to its importance, making the target data classification more accurate. The Transformer neural network model is used to automatically extract the features of the target data and improve classification efficiency. It does not require a large amount of training data, has strong adaptability, and can be applied to various data classification scenarios.

[0139] An embodiment of the present application further discloses a data classification system based on keyword weight matching, including a processor, in which a program of any one of the above-mentioned data classification methods based on keyword weight matching is running.

[0140] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in the field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A data classification method based on keyword weight matching, characterized in that: The steps include: Preprocess the input target data to obtain key features; Calculating the trace information of the key feature in the target data, and matching the key feature with a corresponding key weight using a first algorithm according to the trace information; wherein the trace information generally refers to a certain metric or indicator that can reflect the importance of a keyword or key feature in a text; Inputting the key features into a Transformer neural network model to extract generalized features of the target data; Selecting classification features from the general features using a second algorithm according to the key weights and the trace information; Inputting the key features and the corresponding classification features into a classifier, and using the classifier to classify the target data to obtain a classification result; The classification result corresponding to the target data has three-level indicators. The classification sub-items of the first-level indicator include whether the keyword is related to the trade union or not; the classification sub-items of the second-level indicator include ideological guidance, rights protection services, achievements, self-construction, and content not belonging to the trade union; the classification sub-items of the third-level indicator are the classification contents in each classification sub-item of the second-level indicator; The first algorithm comprises the following steps: Obtaining an optimal data set in the trace information, wherein each of the key features corresponds to an optimal data set; The key weight corresponding to each optimal data is calculated according to the proportion of the optimal data in the optimal data set; The second algorithm comprises the following steps: Calculate the product of the number of times each of the key features appears in the target data and the key weight to obtain the classification feature; Calculate the occurrence frequency value of each of the key features, where the occurrence frequency value = the amount of each of the key features * 100 / the total amount of all the key features; Calculating the inverse document frequency of each of the key features; Taking the set of occurrence frequency values ​​or the set of inverse document frequencies as the optimal data set; Calculate the average word count of the key features; If the average word number is greater than a preset reference word number, the set of occurrence frequency values ​​is used as the optimal data set; otherwise, the set of inverse document frequencies is used as the optimal data set.

2. The data classification method based on keyword weight matching according to claim 1 is characterized in that: The pretreatment method comprises the following steps: If the content of the target data is not a character, converting the content of the target data into a character; Performing a word segmentation operation on the target data; The word sequence of the text obtained after the word segmentation operation is used as the key feature.

3. The data classification method based on keyword weight matching according to claim 2 is characterized in that: The method further comprises the steps of: According to the total character quantity of the target data, the word groups of the word segmentation are adjusted in a positive correlation, the more the total character quantity is, the more the word groups are, and the fewer the total character quantity is, the fewer the word groups are; According to the total punctuation mark amount in the target data, the maximum character amount of the word group is adjusted inversely, the more the total punctuation mark amount is, the less the maximum character amount is, and the less the total punctuation mark amount is, the more the maximum character amount is; The word groups are combinations of words of different lengths.

4. The data classification method based on keyword weight matching according to claim 1 or 2, characterized in that: The pretreatment method comprises the following steps: Performing a stop word removal operation on the target data or the key feature; The word sequence of the text obtained after the stop word operation is used as the key feature.

5. The data classification method based on keyword weight matching according to claim 4 is characterized in that: The method further comprises the steps of: The word removal group of the word removal is adjusted inversely according to the total punctuation amount of the target data or the key feature. The more the total punctuation amount, the fewer the word removal group, and the fewer the total punctuation amount, the more the word removal group; the maximum character amount of the word removal group is adjusted in positive correlation according to the total character amount in the target data or the key feature. The more the total character amount, the more the maximum character amount, and the fewer the total character amount, the fewer the maximum character amount; The removed word groups are removed word combinations of different lengths.

6. The data classification method based on keyword weight matching according to claim 1 is characterized in that: The method comprises the following steps: The reference word count is adjusted inversely according to the number of types of the key features; the more the number of types of the key features is, the fewer the reference word count is; the fewer the number of types of the key features is, the more the reference word count is.

7. A data classification system based on keyword weight matching, characterized in that: It comprises a processor, in which runs a program of the data classification method based on keyword weight matching as described in any one of claims 1 to 6.

Images