A text classification method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application disclose a text classification method and device, electronic equipment and a storage medium, wherein the method comprises: obtaining a to-be-classified corpus having a network new word; performing word segmentation processing on the to-be-classified corpus to obtain a first feature word corresponding to the network new word; updating a weight of the first feature word based on an information amount of the first feature word in the to-be-classified corpus and an association degree between the network new word and a classification category; and performing text classification on the to-be-classified corpus based on the updated weight. The weight of the network new word is updated based on the information amount of the network new word in the to-be-classified corpus and the association degree between the network new word and the classification category, so that the accuracy of classifying the to-be-classified corpus having the network new word is improved.

Description

Technical Field

[0001] This application relates to the field of information technology, and in particular to a text classification method, apparatus, electronic device, and storage medium. Background Technology

[0002] In related technologies, when classifying texts containing new internet terms, the special characteristics of these new terms are usually only addressed and specially processed during the word segmentation stage. This makes the feature extraction and text representation stages in the text classification process susceptible to the influence of the special characteristics of these new internet terms, which in turn reduces the accuracy of the text classification. Summary of the Invention

[0003] To address the aforementioned technical problems, embodiments of this application aim to provide a text classification method, apparatus, electronic device, and storage medium. By updating the weight of new online words in the corpus to be classified, based on the amount of information in the new online words and the correlation between the new online words and the classification category, the accuracy of classifying corpora containing new online words can be improved.

[0004] To achieve the above objectives, the technical solution of this application is implemented as follows:

[0005] This application provides a text classification method, the method comprising:

[0006] Obtain corpora containing novel internet terms to be classified;

[0007] The corpus to be classified is segmented to obtain the first feature word corresponding to the new network word;

[0008] The weight of the first feature word is updated based on the amount of information in the corpus to be classified and the correlation between the new network word and the classification category.

[0009] Based on the updated weights, the text to be classified is classified.

[0010] This application embodiment also provides a text classification device, which includes: an acquisition module, a determination module, an update module, and a classification module; wherein:

[0011] The acquisition module is used to acquire corpora to be classified that contain new online words;

[0012] The determining module is used to perform word segmentation on the corpus to be classified to obtain the first feature word corresponding to the new network word;

[0013] The update module is used to update the weight of the first feature word based on the information content of the first feature word in the corpus to be classified and the correlation between the new network word and the classification category.

[0014] The classification module is used to perform text classification on the corpus to be classified based on updated weights.

[0015] This application also provides an electronic device, which includes: a processor, a memory, and a communication bus;

[0016] The communication bus is used to realize the communication connection between the processor and the memory;

[0017] The processor is used to execute the program in the memory to implement any of the text classification methods described above.

[0018] Accordingly, embodiments of this application also provide a computer-readable storage medium storing one or more programs, which can be executed by one or more processors to implement the steps of the above-described text classification method.

[0019] The text classification method, apparatus, electronic device, and storage medium provided in this application embodiment firstly acquire corpus containing new internet terms; secondly, perform word segmentation on the corpus to be classified to obtain first feature words corresponding to the new internet terms; thirdly, update the weight of the first feature words based on the information content of the first feature words in the corpus to be classified and the correlation between the new internet terms and the classification category; finally, perform text classification on the corpus to be classified based on the updated weights. Thus, by updating the weights of the new internet terms based on the information content of the new internet terms in the corpus to be classified and the correlation between the new internet terms and the classification category, the accuracy of classifying corpus containing new internet terms can be improved. Attached Figure Description

[0020] Figure 1 A flowchart illustrating a text classification method provided in an embodiment of this application;

[0021] Figure 2 A flowchart illustrating another text classification method provided in this application embodiment;

[0022] Figure 3 A flowchart illustrating yet another text classification method provided in this application embodiment;

[0023] Figure 4 A flowchart illustrating an improved text classification method based on display semantic features, provided in an embodiment of this application;

[0024] Figure 5This is a schematic diagram of the structure of a text classification device provided in an embodiment of this application;

[0025] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0027] It should be understood that the phrases "embodiments of this application" or "foreign embodiments" throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "embodiments of this application" or "in the foreign embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0028] Unless otherwise specified, any step in the embodiments of this application performed by the information processing device may be executed by the processor of the information processing device. It is also worth noting that the embodiments of this application do not limit the order in which the information processing device performs the following steps. Furthermore, the methods used to process data in different embodiments may be the same or different methods. It should also be noted that any step in the embodiments of this application can be executed independently by the information processing device; that is, when the information processing device performs any step in the following embodiments, it may not depend on the execution of other steps.

[0029] The internet has become the primary channel for users to obtain information, but the explosive growth of information makes it difficult for users to obtain the information they need from massive amounts of data. Therefore, in order to improve user experience, effective classification of online information has become increasingly important. In related technologies, classification algorithms can be used to process documents of unknown categories and determine their predefined categories. Among these, classification algorithms are mainly based on the Vector Space Model (VSM), and the quality of the feature weighting algorithm directly affects the representation quality of the text VSM, thereby further affecting the accuracy of text classification. Meanwhile, in the field of machine learning, the most commonly used feature weighting algorithm is the Term Frequency-Inverse Document Frequency (TF-IDF) algorithm.

[0030] The improvements to the existing TF-IDF algorithm mainly include the following aspects: 1. Improving the TF-IDF algorithm based on the distribution of features between and within classes; 2. Improving the TF-IDF algorithm based on the positional distribution of feature words in the text; 3. Improving the TF-IDF algorithm based on the different identification capabilities of the same feature word in different categories. Among these, the TF-IDF formula treats all feature terms uniformly, while VSM does not consider the semantic relationships between contexts and potential conceptual structures. That is, the process of classifying Chinese documents based on TF-IDF and VSM lacks consideration of Chinese semantics.

[0031] Based on the above problems, this application provides a text classification method for use in electronic devices, as shown in the following embodiments. Figure 1 The steps shown are explained below:

[0032] Step 101: Obtain the corpus of texts containing new internet terms to be classified.

[0033] In the embodiments of this application, the electronic device can be any device with data processing capabilities.

[0034] The methods by which electronic devices acquire corpora containing new online terms include, but are not limited to: inputting the corpora into the electronic device by a user operating the electronic device using the device's input device; or sending them to the electronic device from other electronic devices capable of interacting with the electronic device. This application does not limit the specific source of the corpora to be classified.

[0035] It should be noted that new internet slang can refer to internet language, that is, emerging words that have semantic information and are not included in the corpus of commonly used words in the Internet environment; in other words, they are usually informal language that is popular on the Internet; many of them are homophones, misspellings, or pictographic words.

[0036] In one feasible implementation, new internet slang could include: "virus", "Buddhist-style", "Aoli Gei", "Lemon Spirit", "Versailles", and "a werewolf", etc.

[0037] In the embodiments of this application, the corpus to be classified containing new internet words can be text information to be classified, that is, text to be classified containing new internet words; wherein, the corpus to be classified containing new internet words can be a piece of text information, a sentence, or an article, and the specific reference of the corpus to be classified in this application is not limited in any way.

[0038] In one feasible implementation, the corpus to be classified containing new internet terms could be: "At least three different types of mutated viruses have been discovered in a certain country".

[0039] In another feasible approach, the corpus to be classified containing network information could be: "Research has found that the pandemic has caused children to spend too much time looking at screens, resulting in a huge gap in their extracurricular activities. At the same time, children's screen time has increased accordingly, and addiction to watching TV may lead to binge eating behavior, which is due to the ease with which one loses self-control when distracted."

[0040] It should be noted that the specific quantity, form of expression, language used, and content of the corpus to be classified are not limited in this application; in the other embodiments of this application below, the corpus to be classified containing new internet terms is expressed in Chinese.

[0041] Step 102: Perform word segmentation on the corpus to be classified to obtain the first feature word corresponding to the new network word.

[0042] In this embodiment of the application, the electronic device performs word segmentation processing on the corpus to be classified, and obtains at least the first feature word corresponding to the new network word; wherein, there is a mapping relationship between the first feature word and the new network word, and the two can be the same, different, or partially the same.

[0043] In one feasible implementation, the electronic device performs word segmentation on the phrase "a country has discovered at least three or more variant viruses" to obtain "a country", "discovered", "at least", "three or more", "of", and "variant virus"; among them, "variant virus" is the first feature word, while the other new network words "virus" are different.

[0044] In another feasible implementation, the electronic device performs word segmentation on "How do you evaluate Versailles" to obtain "you", "to", "Versailles", "how", and "evaluation"; among them, "Versailles" is the first feature word, which is the same as the new online word "Versailles".

[0045] In the following other embodiments of this application, the example is that the new network word is the same as the first feature word.

[0046] It should be noted that the electronic device performs word segmentation on the corpus to be classified, and obtains, but is not limited to, the first feature words corresponding to new online words, the second feature words corresponding to commonly used words, and stop words; among them, the commonly used words may be the same as or different from the second feature words, and this application does not impose any restrictions on this.

[0047] In this embodiment of the application, word segmentation refers to the process of using a word segmentation algorithm to cut the text to be classified into individual characters, words, or phrases; wherein, there are no gaps between the characters, words, and phrases in the Chinese text, and they are presented in the form of continuous strings.

[0048] In the embodiments of this application, the electronic device may employ relevant word segmentation tools, such as "Jieba word segmentation tool and Stanford word segmenter, etc." This application does not impose any limitation on the method or tool used for word segmentation.

[0049] It should be noted that in the embodiments of this application, the first feature word refers to the text information displayed as a phrase. The first feature word can be one, two or more, and this application does not impose any limitation on it.

[0050] Step 103: Update the weight of the first feature word based on the information content of the first feature word in the corpus to be classified and the correlation between the new network words and the classification category.

[0051] In this embodiment of the application, the electronic device first needs to obtain the information content of the first feature word, i.e., the new network word, in the corpus to be classified, as well as the correlation between the new network word and the classification category; secondly, based on the information content of the first feature word in the corpus to be classified, and the correlation between the new network word, i.e., the first feature word, and the classification category, the weight of the first feature word is updated.

[0052] It should be noted that the information content of the first feature word in the corpus to be classified is a measure of how much the first feature word appears in the corpus to be classified. For example, it can refer to the relevant frequency information of the first feature word in the corpus to be classified.

[0053] In one feasible implementation, the information content of the first feature word in the corpus to be classified can refer to the number of times the first feature word appears in the corpus to be classified.

[0054] In another feasible implementation, the information content of the first feature word in the corpus to be classified can refer to the frequency of occurrence of the first feature word in the corpus to be classified and the frequency of reverse documents, etc.

[0055] It should be noted that the correlation between new online terms and category can refer to the degree of association between the new online terms and category, where the degree of association can be strong, moderate, or weak; at the same time, this degree of association can be expressed numerically, and its value range can be [0, 1], or initially represented as [0, 100]; in one feasible implementation, when the degree of association is strong, the corresponding value is 1, and correspondingly, when the degree of association is weak, the corresponding value is 0.2, etc.

[0056] The category refers to the name of the category when classifying related texts, such as: geographical terms, architectural terms, clothing terms, and food terms.

[0057] In one feasible implementation, the new internet term "Buddhist-style" has a weak correlation with geographical terms (0.1) and a strong correlation with everyday life terms (0.5).

[0058] In another feasible implementation, the new online term is "virus," which has a strong correlation with the medical category (correlation score of 0.98) and a relatively strong correlation with food-related terms (correlation score of 0.5).

[0059] It should be noted that the correlation between new online terms and their categories can be determined by calculations based on the initial correlation between the new online terms and each category.

[0060] In the embodiments of this application, electronic devices can determine the correlation between new online words and classification categories through methods such as Chi-Square Statistic (CHI), Information Gain (IG), and Mutual Information (MI). This application does not impose any limitations on this.

[0061] It should be noted that weight represents the importance of a factor or indicator relative to a thing. It is different from the general proportion. It reflects not only the percentage of a factor or indicator, but also the relative importance of the factor or indicator, tending to the degree of contribution or importance. Usually, weight can be judged and calculated by dividing indicators into multiple levels. Therefore, the weight of the first feature word can refer to the importance of the first feature word in the corpus to be classified.

[0062] Correspondingly, when the number of first feature words is two or more, the number of weights of the corresponding first feature words is also two or more, that is, there is a mapping relationship between the first feature words and the weights.

[0063] It should be noted that, with the information content of the first feature word remaining constant, the higher its relevance, the higher its corresponding weight; conversely, with the information content of the first feature word remaining constant, the lower its relevance, the lower its corresponding weight.

[0064] Step 104: Based on the updated weights, perform text classification on the corpus to be classified.

[0065] In the embodiments of this application, the electronic device performs text classification based on the updated weights of the corpus to be classified.

[0066] The electronic device can generate a weight matrix corresponding to the corpus to be classified based on the updated weights, and realize the text representation of the corpus to be classified based on the weight matrix, thereby enabling text classification of the corpus to be classified.

[0067] In one feasible implementation, the electronic device first performs word segmentation on the corpus to be classified to obtain at least one feature word and calculates the weight of at least one feature word; then, based on at least one feature word and its corresponding weight, it expresses the weight matrix corresponding to the corpus to be classified and performs text classification on the corpus to be classified based on the weight matrix.

[0068] In the embodiments of this application, the electronic device can perform text classification based on traditional machine learning algorithms or deep learning algorithms. This application does not limit the specific method of text classification.

[0069] It should be noted that, in the embodiments of this application, when the electronic device performs text classification on the corpus to be classified containing new online words, it can update the weight of the first feature word by the amount of information of the new online word, i.e., the first feature word, in the corpus to be classified, as well as the correlation between the new online word and the classification category; in this way, the weight of the first feature word depends not only on the amount of relevant information, but also on the correlation, thereby improving the accuracy of text classification on the corpus to be classified containing new online words.

[0070] The text classification method provided in this application firstly acquires corpus containing new internet terms; secondly, it performs word segmentation on the corpus to be classified to obtain first feature words corresponding to the new internet terms; thirdly, it updates the weight of the first feature words based on the information content of the first feature words in the corpus to be classified and the correlation between the new internet terms and the classification category; finally, it performs text classification on the corpus to be classified based on the updated weights. In this way, by updating the weights of the new internet terms based on the information content of the new internet terms in the corpus to be classified and the correlation between the new internet terms and the classification category, the accuracy of classifying corpus containing new internet terms can be improved.

[0071] Based on the foregoing embodiments, this application also provides a text classification method, applied to electronic devices, with reference to... Figure 1 and Figure 2 The steps shown are described below. The text classification method provided in this application embodiment may include step 101 above, steps 201 to 204 below, and step 104 above:

[0072] Step 201: Obtain the target lexicon, which includes a lexicon of new online words and a lexicon of commonly used words.

[0073] In this embodiment of the application, the electronic device acquires a target lexicon, which includes a lexicon of new online words and a lexicon of commonly used words; wherein, the method by which the electronic device acquires the target lexicon is not limited in this application.

[0074] Among them, the new online word lexicon is the first lexicon formed by new online words; it is dynamically updated and changed based on online information; correspondingly, the common word lexicon is the second lexicon formed by words frequently used in daily life, which is basically fixed.

[0075] In this embodiment of the application, the target lexicon is composed of a new online word lexicon and a commonly used word lexicon in any form. In other words, the vocabulary in the target lexicon includes new online words and commonly used words.

[0076] It should be noted that while online new word databases and commonly used word databases may contain the same vocabulary, their intended meanings may differ.

[0077] In one feasible implementation, the online neologism database includes terms such as "Versailles," "lemon spirit," and "Martian," while the commonly used word database includes terms such as "dictionary," "robot," and "learning."

[0078] In the embodiments of this application, the specific storage format and presentation of the target lexicon are not limited in this application, and can be documents, tables, etc.

[0079] Step 202: Based on the target lexicon, perform word segmentation on the corpus to be classified to obtain the first feature word.

[0080] In this embodiment of the application, the electronic device performs word segmentation on the corpus to be classified based on the target lexicon to obtain the first feature word.

[0081] For electronic devices, the first step is to segment the corpus to be classified based on a target lexicon that includes both new and commonly used online words. This process yields the first feature words corresponding to new online words, the second feature words corresponding to commonly used words, and related stop words. Then, the first feature words corresponding to new online words are selected from the obtained lexicon.

[0082] It should be noted that, in this embodiment of the application, the electronic device performs word segmentation on the corpus to be classified based on a target lexicon that includes new online words and commonly used words, in order to obtain the first feature word. In this way, new online words are taken into account during the word segmentation process, so that the missegmentation caused by the semantic specialness of new online words can be avoided during the word segmentation process, thereby improving the accuracy of the word segmentation process of the corpus to be classified. At the same time, it can improve the accuracy of the electronic device in performing the next step of feature weight calculation and subsequent text representation and text classification.

[0083] In one feasible implementation, the electronic device can perform step 202 by performing steps 202a to 202c in addition to step 202a:

[0084] Step 202a: Based on the target lexicon, perform word segmentation on the corpus to be classified to obtain the feature word set.

[0085] In this embodiment of the application, the electronic device performs word segmentation on the corpus to be classified based on the target lexicon to obtain a feature word set; wherein, the feature word set may include at least one feature word. At the same time, when there is only one feature word, it may be the first feature word corresponding to a new online word; when there are two or more feature words, it may be the first feature word corresponding to a new online word and the second feature word corresponding to a common word, or it may be the first feature word corresponding to a new online word and a stop word.

[0086] In this application embodiment, the feature word set includes, but is not limited to: feature words corresponding to new online words, feature words corresponding to commonly used words, and stop words; wherein, the number of the above-mentioned feature words and their distribution in the feature word set are not limited in this application.

[0087] The feature word set can be represented in any form, such as a document or a table, and this application makes no restrictions on it.

[0088] Step 202b: Filter the feature word set based on stop words to obtain the target feature word set.

[0089] In this embodiment of the application, the electronic device acquires stop words and filters the feature word set based on the stop words to obtain a target feature word set that does not contain stop words.

[0090] Specifically, electronic devices can filter and select feature words corresponding to stop words in the feature word set based on stop words to obtain the target feature word set.

[0091] It should be noted that stop words can generally be divided into two categories. One category consists of function words contained in human language. These function words are extremely common and have no actual meaning compared to other words, such as: these, which, this or that, etc. The other category consists of words that, when used in search engines, cannot guarantee that they will provide truly relevant search results, are difficult to help narrow down the search scope, and will also reduce search efficiency.

[0092] Step 202c: Determine the first feature word from the target feature word set.

[0093] In this embodiment of the application, the electronic device determines the first feature word corresponding to the new network word from the determined target feature word set.

[0094] In one feasible implementation, the electronic device filters and selects feature words corresponding to common words from the target feature word set to determine the first feature word.

[0095] In another feasible implementation, the target feature word set consists entirely of first feature words.

[0096] In the embodiments of this application, the specific number of the first feature words in the target feature word set is not limited in this application.

[0097] It should be noted that, in the embodiments of this application, the electronic device performs word segmentation on the corpus to be classified to obtain a feature word set, and filters the feature word set based on stop words to obtain a target feature word set, and then obtains the first feature word. In this way, filtering stop words can improve the efficiency of the electronic device in subsequent weight calculation, text representation and text classification, and at the same time avoid information interference caused by stop words, thereby improving the accuracy of classification of the corpus to be classified.

[0098] Accordingly, the electronic device can also update the weight of the first feature word in step 103, which is based on the information content of the first feature word in the corpus to be classified and the correlation between the new network word and the classification category, through the following steps 203 and 204:

[0099] Step 203: Based on the amount of information, determine the frequency of the first word and the frequency of the first inverse document in the corpus to be classified.

[0100] Among them, the first inverse document frequency represents the prevalence of the first feature word in the corpus to be classified.

[0101] In this embodiment of the application, the electronic device calculates and determines the first word frequency and the first inverse document frequency of the first feature word in the corpus to be classified based on the amount of information.

[0102] Among them, the first word frequency refers to the frequency of occurrence of the first feature word in the corpus to be classified; correspondingly, the first inverse document frequency refers to the prevalence of the first feature word in the corpus to be classified, which can refer to the correlation ratio information between the number of documents with the first feature word in the corpus to be classified and the total number of documents in the corpus to be classified.

[0103] Step 204: Update the weight of the first feature word based on the first word frequency, the first inverse document frequency, and the relevance.

[0104] In this embodiment of the application, the electronic device updates the weight of the first feature word to the first weight based on the first word frequency, the first reverse document frequency, and the relevance.

[0105] In one feasible implementation, the electronic device can calculate and determine the weight of the first feature word based on the first word frequency, the first inverse document frequency, and the relevance.

[0106] It should be noted that, in this embodiment, the electronic device updates the weight of the first feature word based on the first word frequency, the first reverse document frequency, and the relevance. In this way, the determination of the weight of the first feature word not only depends on the relevant information content of the first feature word in the corpus to be classified, i.e., the relevant frequency information, but also involves the relevance of the first feature word to the classification category. Thus, the parameters involved in the weight of the first feature word determined by the electronic device are more comprehensive, thereby ensuring the weight ratio of the first feature word, i.e., the new network word, and improving the accuracy of the text representation and text classification of the corpus to be classified by the electronic device in the later stages.

[0107] In one feasible implementation, the electronic device may also perform the following step A before performing step 204:

[0108] Step A: Obtain the conversion coefficient corresponding to the correlation value.

[0109] In this embodiment of the application, the electronic device obtains the conversion coefficient corresponding to the value of the correlation degree.

[0110] In one feasible implementation, the higher the correlation value, the higher the corresponding conversion coefficient.

[0111] In another feasible implementation, the higher the correlation value, the lower the corresponding conversion coefficient.

[0112] It should be noted that the conversion coefficient corresponding to the correlation value can be determined by the electronic device based on preset rules, or it can be user-defined by the user operating the electronic device. This application does not limit the specific source of the conversion coefficient.

[0113] Accordingly, the electronic device can also perform step 204, which involves updating the weight of the first feature word based on the first word frequency, the first inverse document frequency, and the relevance, through the following steps 204a and 204b:

[0114] Step 204a: Determine the gain corresponding to the correlation degree based on the conversion coefficient and correlation degree.

[0115] In this embodiment of the application, the electronic device determines the gain corresponding to the correlation degree based on the conversion coefficient and the correlation degree; specifically, the electronic device may calculate and determine the corresponding gain based on the values ​​of the conversion coefficient and the correlation degree.

[0116] In one feasible implementation, the electronic device adds the values ​​of the conversion coefficient and the correlation degree to obtain the gain corresponding to the correlation degree.

[0117] In another feasible implementation, the electronic device multiplies the values ​​of the conversion coefficient and the correlation degree to obtain the gain corresponding to the correlation degree.

[0118] It should be noted that the method by which the electronic device calculates the conversion coefficient and correlation degree to obtain the corresponding gain is not limited in the embodiments of this application.

[0119] In the embodiments of this application, the gain is expressed in numerical form, which may be expressed as a percentage, decimal, or fraction, etc. This application does not impose any limitations on this.

[0120] Step 204b: Based on the first word frequency, the first inverse document frequency, and the gain, update the weight of the first feature word to the first weight.

[0121] In this embodiment of the application, the electronic device updates the weight of the first feature word to a first weight based on the first word frequency, the first inverse document frequency, and the gain; specifically, it can be calculated based on the above three types of parameters to determine the weight of the first feature word.

[0122] Accordingly, when there are two or more first feature words, the electronic device calculates and determines the first weight of each first feature word based on the first word frequency, the first inverse document frequency, and the gain corresponding to each feature word.

[0123] It should be noted that, in this embodiment, the electronic device updates the weight of the first feature word based on the first word frequency, the first inverse document frequency, and the relevance. This ensures that the electronic device determines the weight of the first feature word not only based on the information content of the first feature word in the corpus to be classified, i.e., the relevant frequency information, but also on the relevance between the first feature word and the classification category. Simultaneously, the electronic device can assign high gain to new network words with high similarity to the classification category, and correspondingly, assign low gain to new network words with low similarity to the classification category. This ensures that new network words maintain a certain weight during the classification process of the corpus to be classified, thereby improving the accuracy of text classification for the corpus to be classified.

[0124] The text classification method provided in this application, based on a target lexicon including new online words and commonly used words, performs word segmentation on the corpus to be classified. This avoids missegmentation caused by the semantic specialness of new online words during the word segmentation process, thereby improving the accuracy of word segmentation of the corpus to be classified. Simultaneously, the electronic device updates the weight of the first feature word based on the first word frequency, the first inverse document frequency, and the relevance. This ensures that the electronic device determines the weight of the first feature word not only based on the information content of the first feature word in the corpus to be classified, i.e., the relevant frequency information, but also on the relevance of the first feature word to the classification category. This ensures that new online words maintain a certain weight during the classification process of the corpus to be classified, thereby improving the accuracy of text classification for the corpus to be classified.

[0125] Based on the foregoing embodiments, this application also provides a text classification method, applied to electronic devices, with reference to... Figures 1 to 3 The steps shown are described below. The text classification method provided in this application embodiment may include steps 101, 201 to 204 as described above, and steps 301 to 304 as described below:

[0126] If the target feature word set also includes a second feature word, wherein the second feature word represents feature words other than the first feature word, the electronic device may also perform the following steps 301 and 302:

[0127] Step 301: Obtain the second word frequency and the second inverse document frequency of the second feature word in the corpus to be classified.

[0128] Among them, the second inverse document frequency represents the prevalence of the second feature word in the corpus to be classified.

[0129] In this embodiment, the target feature word set is the information obtained in step 202b of the above embodiment. For the specific implementation of how to obtain the target feature word set, please refer to the description in the above embodiment, which will not be repeated here.

[0130] In this embodiment of the application, when the target feature word set also includes a second feature word, wherein the second feature word represents a feature word other than the first feature word, the electronic device can obtain the second word frequency and the second inverse document frequency of the second feature word in the corpus to be classified based on the amount of information of the second feature word in the corpus to be classified.

[0131] The second feature word refers to the feature word corresponding to the commonly used word; correspondingly, the second feature word may be the same as or different from the commonly used word, and this application does not impose any restrictions on this.

[0132] Accordingly, the number of the second feature word in the target feature word set, i.e. the corpus to be classified, can be one, two or more. This application does not impose any restrictions on the specific number and distribution of the second feature word in the corpus to be classified.

[0133] In one feasible implementation, the electronic device calculates and determines the second word frequency and the second inverse document frequency of the second feature word in the corpus to be classified based on the number of times or occurrences of the feature words corresponding to the common words in the corpus to be classified.

[0134] It should be noted that the second word frequency and the second reverse document frequency correspond to the first word frequency and the first reverse document frequency, respectively. For specific descriptions, please refer to the explanations of the first word frequency and the first reverse document frequency in the above embodiments, which will not be repeated here.

[0135] Step 302: Based on the second word frequency and the second inverse document frequency, determine the weight of the second feature word as the second weight.

[0136] In this embodiment of the application, the electronic device determines the weight of the second feature word as the second weight based on the second word frequency and the second inverse document frequency. That is, when the electronic device determines the weight of the feature word corresponding to the common word, it only needs to consider the corresponding word frequency and inverse document frequency.

[0137] It should be noted that, in the embodiments of this application, the electronic device calculates and determines the weight of the second feature word based on the second word frequency and the second inverse document frequency of the second feature word, i.e., the second word frequency of the common word; in this way, the accuracy of determining the weight of the common word is high, thereby improving the accuracy of the electronic device in performing the next step of operation, such as the text representation of the text corpus to be classified and the text classification process.

[0138] Accordingly, in performing step 104, that is, the electronic device performs text classification on the corpus to be classified based on the updated weights, the electronic device may also perform the following steps 303 and 304:

[0139] Step 303: Determine the weight matrix of the corpus to be classified based on the first weight and the second weight.

[0140] In this embodiment of the application, the electronic device constructs a weight matrix of the corpus to be classified based on the first weight and the second weight, wherein each weight corresponds to its corresponding feature word, that is, the first weight corresponds to the first feature word and the second weight corresponds to the second feature word.

[0141] In the embodiments of this application, the number of first feature words can be one, two or more, and this application does not limit this. Correspondingly, the number of second feature words can also be one, two or more, and this application does not limit this. Then, the electronic device calculates a first weight based on each first feature word and a second weight based on each second feature word. In other words, the first feature word and the second feature word are respectively associated with their corresponding weights.

[0142] Electronic devices can use this weight matrix to represent the text of the corpus to be classified.

[0143] Step 304: Based on the weight matrix, perform text classification on the corpus to be classified.

[0144] In this embodiment of the application, the electronic device performs text classification on the corpus to be classified based on the weight matrix. That is, the electronic device performs text representation on the corpus to be classified based on the weight matrix, and then classifies the text based on the relevant text classifier.

[0145] The electronic device can perform text classification on the corpus to be classified based on transmission machine learning algorithms such as Bayesian or SVM, or on deep learning algorithms such as fastText or Convolutional Neural Networks (CNN). This application does not impose any limitations on this.

[0146] It should be noted that, in the embodiments of this application, the electronic device uses a weight matrix formed by the first weight and the second weight as the text representation of the corpus to be classified, so as to achieve text classification of the corpus to be classified; thus, the process of the electronic device classifying the corpus to be classified is more specific and more accurate.

[0147] In one feasible implementation, when the electronic device performs step 304, it may also perform the following step B.

[0148] Step B: Based on the acquired sample corpus, train the classifier used for text classification to obtain the trained classifier.

[0149] In this embodiment of the application, the electronic device first acquires sample corpus; wherein, the sample corpus may be text information including common words, new internet words and stop words, or text information including common words and stop words, or even text information including internet information words and stop words. This application does not limit the specific reference or specific content of the sample corpus.

[0150] In this embodiment, the electronic device inputs the acquired sample corpus into a classifier for text classification to predict the classification result of each sample corpus. Simultaneously, it compares the label of each sample corpus with the predicted classification result of each sample corpus to obtain an output result. Based on the output result, it generates the loss function of the classifier, and then adjusts the network parameters of the classifier based on the loss function. The sample corpus is then input into the classifier again to obtain the corresponding loss function. This process is repeated until the final loss function satisfies the preset convergence condition. At this point, the classifier is considered a trained classifier.

[0151] In this embodiment of the application, the specific references to classifiers and corresponding trained classifiers are not limited.

[0152] Accordingly, the electronic device performs step 340, that is, the electronic device performs text classification on the corpus to be classified based on the weight matrix. This can also be achieved by performing the following step 304a:

[0153] Step 304a: Based on the trained classifier and weight matrix, perform text classification on the corpus to be classified.

[0154] In this embodiment of the application, the electronic device first performs text representation on the corpus to be classified based on the weight matrix, and then inputs the text representation into the trained classifier to achieve text classification of the corpus to be classified.

[0155] In the embodiments of this application, the electronic device trains the text classification of the corpus to be classified, and performs text classification on the corpus to be classified based on the trained classifier, thereby improving the accuracy of text classification of the corpus to be classified.

[0156] The text classification method provided in this application, based on a target lexicon including new online words and commonly used words, performs word segmentation on the corpus to be classified. This avoids missegmentation caused by the semantic specialness of new online words during the word segmentation process, thereby improving the accuracy of word segmentation on the corpus to be classified. Simultaneously, the electronic device updates the weight of the first feature word based on the first word frequency, the first inverse document frequency, and the relevance. This allows the electronic device to determine the weight of the first feature word not only based on the information content of the first feature word in the corpus to be classified, i.e., the relevant frequency information, but also on the relevance between the first feature word and the classification category. Furthermore, by classifying the text based on the determined weight matrix and the trained classifier, the accuracy of classification can be further improved.

[0157] Based on this, this application embodiment also provides a flowchart of a text classification method improved based on display semantic features; wherein, the electronic device acquires the corpus to be classified, i.e., executes 401, wherein the corpus to be classified is the corpus to be classified that includes new online words; then executes 402, i.e., performs text segmentation on the corpus to be classified, and obtains 403, a feature word set; after obtaining the feature word set, the feature words are judged, i.e., executes 404: whether it is the first feature word, i.e., the first feature word corresponding to the new online word; if so, executes 405, determining the correlation, i.e., determining the correlation between the new online word and the classification category, and calculating the corresponding weight based on the correlation and related frequency information. If not, calculate the corresponding weights based on the relevant frequency information: That is, 406; then, based on two different weights, the corresponding weight matrix is ​​obtained, which is 407; then, it is input into the classifier of 409 for text classification to obtain the classification result, which is 410; where the classifier of 409 is based on the classifier trained in 408.

[0158] Specifically, the electronic device can calculate the corresponding weight based on the commonly used feature weighting algorithm: TF-IDF algorithm, that is, the second weight of the second feature word in the embodiment of this application; wherein, the TF-IDF weighting algorithm consists of two parts: Term Frequency (TF) and Inverse Document Frequency (IDF); TF considers the frequency of feature terms in the text, and IDF considers the distribution of feature terms in the text. The commonly used TF-IDF weight calculation method is shown in formula (1).

[0159] (1);

[0160] in, Representing characteristic terms, This represents the corpus to be classified. express exist The weights in for exist Frequency of occurrence in For the corpus The total number, For text corpora containing The number of texts, For text middle The number of.

[0161] Accordingly, Corresponding to the second word frequency in the embodiments of this application, This corresponds to the second reverse document frequency in the embodiments of this application.

[0162] Among them, 405 determines the degree of correlation, which corresponds to the degree of correlation between new network words and classification categories in the embodiments of this application. The degree of correlation can be calculated by the IG algorithm, and can be referred to the following formula (2) or (3).

[0163] (2);

[0164] (3);

[0165] in, It is a characteristic term The resulting information gain It is a category Information entropy In the feature terms Classification under conditions Information entropy It is the total number of categories. It belongs to the category in the corpus to be classified The proportion of documents in the document category. It contains feature terms The proportion of documents in the corpus to be classified. It contains feature terms Also belongs to the category The proportion of documents in the document category. It does not contain The probability of the document appearing. It does not contain But belongs to The proportion of documents belonging to each class.

[0166] Accordingly, This corresponds to the correlation between the new online words and the classification categories provided in the embodiments of this application; at the same time, the electronic device also provides the conversion coefficient corresponding to the value of the correlation, that is, the corresponding conversion function. The significance of the correlation conversion function is to measure the role of the new online words in classifying the corpus to be classified. You can refer to formula (4).

[0167] (4);

[0168] in, This is the gain corresponding to the correlation degree; the specific parameter conversion coefficients can be determined based on the actual application, and this application provides an example.

[0169] Furthermore, electronic devices can be based on gain Equation (1) is updated to obtain Equation (5), which is then used to calculate the weight of the new network words in the corpus to be classified, i.e., the weight of the first feature word. For a detailed explanation of the parameters, please refer to formula (1).

[0170] (5);

[0171] Accordingly, the electronic device constructs a weight matrix for the corpus to be classified based on the obtained first and second weights. , as in formula (6).

[0172] (6);

[0173] It should be noted that electronic devices are based on a weight matrix. The corpus to be classified is converted into vector operations in a vector space, and the similarity in space is used to represent the semantic similarity. After the documents are converted into vectors in the VSM model, the similarity between documents can be measured by calculating the similarity between vectors.

[0174] Table 1: Text Vector Space Model

[0175]

[0176] Correspondingly, as shown in Table 1, the characteristic terms t1, t2, ..., t n It can be viewed as an n-dimensional coordinate system, with weights w1, w2, ..., w n This represents the corresponding coordinate value, d for each document. i Mapped to a feature vector V(d) in the vector coordinate space i )=(t1, w i1 ;t2,w i2 ;…;t n w in ).

[0177] It should be noted that it is assumed that all new words can be correctly identified and XC(t)=100; the following is a comparison table of weight calculations for the same new words on the Internet using formula (1) and formula (5) respectively.

[0178] Table 2: Weight Comparison of New Internet Terms

[0179]

[0180] Based on the foregoing embodiments, this application also provides a text classification device 5, which can be applied to... Figures 1 to 3 In a text classification method provided in a corresponding embodiment, referring to Figure 5As shown, the text classification device 5 includes: an acquisition module 51, a determination module 52, an update module 53, and a classification module 54; wherein:

[0181] The acquisition module 51 is used to acquire corpora to be classified that contain new online words;

[0182] The determination module 52 is used to perform word segmentation on the corpus to be classified to obtain the first feature word corresponding to the new network word;

[0183] The update module 53 is used to update the weight of the first feature word based on the information content of the first feature word in the corpus to be classified and the correlation between the new network words and the classification category.

[0184] Classification module 54 is used to classify texts in the corpus to be classified based on updated weights.

[0185] In other embodiments of this application, the determining module 52 is further configured to obtain a target lexicon including a network new word lexicon and a common word lexicon; and based on the target lexicon, to perform word segmentation on the corpus to be classified to obtain the first feature word.

[0186] In other embodiments of this application, the determining module 52 is further configured to perform word segmentation on the corpus to be classified based on the target lexicon to obtain a feature word set; filter the feature word set based on stop words to obtain a target feature word set; and determine a first feature word from the target feature word set.

[0187] In other embodiments of this application, the update module 53 is further configured to determine the first word frequency and the first inverse document frequency of the first feature word in the corpus to be classified based on the amount of information; wherein the first inverse document frequency represents the prevalence of the first feature word in the corpus to be classified; and update the weight of the first feature word based on the first word frequency, the second inverse document frequency and the relevance.

[0188] In other embodiments of this application, the acquisition module 51 is further configured to acquire the conversion coefficient corresponding to the numerical value of the relevance; correspondingly, the update module 53 is further configured to determine the gain corresponding to the relevance based on the conversion coefficient and the relevance; and update the weight of the first feature word to the first weight based on the first word frequency, the first reverse document frequency and the gain.

[0189] In other embodiments of this application, when the target feature word set further includes a second feature word, wherein the second feature word represents feature words other than the first feature word, the acquisition module 51 is further configured to acquire the second word frequency and the second inverse document frequency of the second feature word in the corpus to be classified; wherein the second inverse document frequency represents the prevalence of the second feature word in the corpus to be classified; correspondingly, the determination module 52 is further configured to determine the weight of the second feature word as a second weight based on the second word frequency and the second inverse document frequency.

[0190] In other embodiments of this application, the classification module 54 is further configured to determine the weight matrix of the corpus to be classified based on the first weight and the second weight; and to perform text classification on the corpus to be classified based on the weight matrix.

[0191] In other embodiments of this application, the text classification device 5 further includes a training module 55, wherein the training module 55 is further configured to train a classifier for text classification based on the acquired sample corpus to obtain a trained classifier; correspondingly, the classification module 54 is further configured to perform text classification on the corpus to be classified based on the trained classifier and the weight matrix.

[0192] It should be noted that the specific implementation process of the steps performed by each module in this embodiment can be referred to Figures 1 to 3 The implementation process of the text classification method provided in the corresponding embodiment will not be described in detail here.

[0193] The text classification apparatus provided in this application performs word segmentation on the corpus to be classified based on a target lexicon including new online words and commonly used words. This avoids missegmentation caused by the semantic specialness of new online words during the word segmentation process, thereby improving the accuracy of word segmentation of the corpus to be classified. At the same time, the weight of the first feature word is updated based on the first word frequency, the first inverse document frequency, and the relevance. This ensures that the determination of the weight of the first feature word depends not only on the information content of the first feature word in the corpus to be classified, i.e., the relevant frequency information, but also on the relevance of the first feature word to the classification category. Furthermore, by performing text classification on the corpus to be classified based on the determined weight matrix and the trained classifier, the accuracy of classification can be further improved.

[0194] Based on the foregoing embodiments, embodiments of this application also provide an electronic device 6, which can be applied to... Figures 1 to 3 In a text classification method provided in a corresponding embodiment, referring to Figure 6 As shown, the electronic device 6 may include: a processor 61, a memory 62, and a communication bus 63, wherein:

[0195] The communication bus 63 is used to realize the communication connection between the processor 61 and the memory 62.

[0196] Processor 61 is used to execute a program for a text classification method stored in memory 62, in order to implement a reference... Figures 1 to 3 The corresponding implementation provides a text classification method.

[0197] The electronic device provided in this application embodiment performs word segmentation on the corpus to be classified based on a target lexicon including new online words and commonly used words. This avoids missegmentation caused by the semantic specialness of new online words during the word segmentation process, thereby improving the accuracy of word segmentation on the corpus to be classified. At the same time, the electronic device updates the weight of the first feature word based on the first word frequency, the first inverse document frequency, and the relevance. This allows the electronic device to determine the weight of the first feature word not only based on the information content of the first feature word in the corpus to be classified, i.e., the relevant frequency information, but also on the relevance of the first feature word to the classification category. In addition, the text classification of the corpus to be classified based on the determined weight matrix and the trained classifier can further improve the accuracy of classification.

[0198] Based on the foregoing embodiments, embodiments of this application provide a computer-readable storage medium storing one or more programs, which can be executed by one or more processors to achieve the following: Figures 1 to 3 The corresponding implementation provides the steps of the text classification method.

[0199] It should be noted that the aforementioned computer-readable storage media can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM), etc.; or it can be various electronic devices including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0200] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0201] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0202] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0203] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0204] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0205] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0206] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A text classification method, characterized in that, The method includes: Obtain corpora containing novel internet terms to be classified; The corpus to be classified is segmented to obtain the first feature word corresponding to the new network word; The weight of the first feature word is updated based on the amount of information in the corpus to be classified and the correlation between the new network word and the classification category. Based on the updated weights, the text to be classified is classified. The step of updating the weight of the first feature word based on the information content of the first feature word in the corpus to be classified and the correlation between the new network word and the classification category includes: Based on the information content, the first word frequency and the first inverse document frequency of the first feature word in the corpus to be classified are determined; wherein, the first inverse document frequency characterizes the prevalence of the first feature word in the corpus to be classified. Obtain the conversion coefficient corresponding to the value of the correlation degree; The weight of the first feature word is updated based on the first word frequency, the first reverse document frequency, and the relevance. The step of updating the weight of the first feature word based on the first word frequency, the first inverse document frequency, and the relevance includes: Based on the conversion coefficient and the correlation degree, determine the gain corresponding to the correlation degree; Based on the first word frequency, the first inverse document frequency, and the gain, the weight of the first feature word is updated to the first weight.

2. The method according to claim 1, characterized in that, The step of segmenting the corpus to be classified to obtain the first feature word corresponding to the new network word includes: Obtain the target lexicon, which includes a lexicon of new online words and a lexicon of commonly used words; Based on the target lexicon, the corpus to be classified is segmented to obtain the first feature word.

3. The method according to claim 2, characterized in that, The first feature word is obtained by segmenting the corpus to be classified based on the target vocabulary to obtain the first feature word, including: Based on the target vocabulary, the corpus to be classified is segmented to obtain a feature word set; The feature word set is filtered based on stop words to obtain the target feature word set; The first feature word is determined from the target feature word set.

4. The method according to claim 1, characterized in that, When the target feature word set also includes a second feature word, wherein the second feature word represents feature words other than the first feature word, the method further includes: Obtain the second word frequency and the second inverse document frequency of the second feature word in the corpus to be classified; wherein, the second inverse document frequency represents the prevalence of the second feature word in the corpus to be classified; Based on the second word frequency and the second inverse document frequency, the weight of the second feature word is determined as the second weight.

5. The method according to claim 4, characterized in that, The text classification of the corpus to be classified based on the updated weights includes: Based on the first weight and the second weight, the weight matrix of the corpus to be classified is determined; Based on the weight matrix, text classification is performed on the corpus to be classified.

6. The method according to claim 5, characterized in that, Before performing text classification on the corpus to be classified based on the weight matrix, the method further includes: Based on the acquired sample corpus, the classifier used for text classification is trained to obtain the trained classifier. Accordingly, the step of classifying the text based on the weight matrix includes: Based on the trained classifier and the weight matrix, the text to be classified is classified.

7. A text classification device, characterized in that, The text classification device includes: an acquisition module, a determination module, an update module, and a classification module; wherein: The acquisition module is used to acquire corpora to be classified that contain new online words; The determining module is used to perform word segmentation on the corpus to be classified to obtain the first feature word corresponding to the new network word; The update module is used to update the weight of the first feature word based on the information content of the first feature word in the corpus to be classified and the correlation between the new network word and the classification category; wherein, the update module is specifically used to: determine the first word frequency and the first inverse document frequency of the first feature word in the corpus to be classified based on the information content; obtain the conversion coefficient corresponding to the value of the correlation; determine the gain corresponding to the correlation based on the conversion coefficient and the correlation; update the weight of the first feature word to a first weight based on the first word frequency, the first inverse document frequency and the gain; the first inverse document frequency represents the prevalence of the first feature word in the corpus to be classified; The classification module is used to perform text classification on the corpus to be classified based on updated weights.

8. An electronic device, characterized in that, The electronic device includes: a processor, a memory, and a communication bus; The communication bus is used to realize the communication connection between the processor and the memory; The processor is used to execute the program in the memory to implement the text classification method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the text classification method as described in any one of claims 1 to 6.

Images