An automatic text classification method based on automobile community

Abstract

The application relates to an automatic text classification method based on a car community, which comprises the following steps: obtaining a data set of car community texts; obtaining word vectors and text feature vectors as input values of a double-layer clustering model; performing clustering calculation on the word vectors and the text feature vectors to generate word classification and text classification respectively, so as to form the double-layer clustering model. When a new text enters: calculating the word vectors and the text feature vectors of the new text; calculating the word classification of each word vector of the new text and the word frequency under each classification; calculating the text classification of the new text; performing dynamic analysis according to the word classification, the word frequency and the text classification of each word vector of the new text; when the number of words and the word frequency of the new text generated outside the existing word classification reach a threshold value, the double-layer clustering model is updated. Therefore, the application can realize automatic classification of car community texts in the whole process, improve the classification accuracy and efficiency, and form a closed-loop management.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and more specifically to an automatic text classification method based on a car community. Background Technology

[0002] Text classification in automotive communities is characterized by its high level of specialization (numerous technical terms), strong practicality (extractability of vehicle-related information), significant variations in text length, and high similarity in some texts. With increasingly fierce competition among automakers, major companies are gradually focusing on the operation of private domain communities. How to efficiently and accurately achieve automatic classification of community content for ease of business management and application has become a crucial business issue. Currently, text classification in automotive communities primarily employs keyword-based classification or traditional clustering algorithms. Keyword-based classification mainly relies on business experience and data analysis to manually identify classification keywords. Traditional clustering algorithms include, for example, k-means.

[0003] In summary, existing classification methods have the following problems: 1. Keyword rule-based classification methods mainly rely on manual judgment, which has the problems of incomplete classification and difficulty in quickly adjusting the classification according to the latest changes in the text.

[0004] 2. Traditional clustering algorithms such as k-means are susceptible to the effects of imbalanced data, outliers, and noise, and the optimal solution may only be applicable to local data.

[0005] Currently, the official patent search and analysis website lacks content related to text classification in automotive communities, only offering patents related to car reviews and sentiment analysis. Therefore, it is necessary to provide an automatic text classification method based on automotive communities, capable of automatically classifying automotive community text throughout the entire process, improving classification accuracy and efficiency, and forming a closed-loop management system. Summary of the Invention

[0006] The purpose of this invention is to provide a method and system for automatic text classification based on automotive communities, which can realize automatic classification of automotive community texts throughout the entire process, improve classification accuracy and efficiency, and form a closed-loop management.

[0007] To achieve the above objectives, the technical solution adopted by this invention is as follows: an automatic text classification method based on automotive communities, comprising the following steps: Data preprocessing: Obtain the dataset of text from the automotive community and perform preprocessing; Obtaining word vectors and text feature vectors: The preprocessed dataset is trained using a word vector model and a text vector model to obtain the word vector of each word and the text feature vector of each text, which are used as input values ​​for the two-layer clustering model; Construct a two-layer clustering model: perform clustering calculations on word vectors and text feature vectors to generate word classification and text classification respectively, thus obtaining word classification model and text classification model.

[0008] The new text is processed and analyzed using a two-layer clustering model: The trained word vector model and text vector model are used to calculate the word vectors and text feature vectors of the new text; the vocabulary classification model is applied to calculate the vocabulary classification of each word vector in the new text, and the word frequency under each vocabulary classification is calculated; the trained text classification model is applied to calculate the text classification of the new text; the vocabulary classification, word frequency, and text classification of each word vector in the new text are dynamically analyzed, and the two-layer clustering model is updated when the number and frequency of words outside the existing vocabulary classifications generated by the new text reach a threshold.

[0009] Furthermore, the specific steps for constructing the two-layer clustering model are as follows: The first step is to initialize cluster centers: using the Chameleon clustering algorithm, some words / text vectors in the trained dataset are randomly selected as multiple initial word / text vector cluster centers, and each word / text vector cluster center represents a category; The second step is to calculate the similarity: use the centered cosine similarity algorithm to calculate the similarity between each word / text vector and the cluster center of each word / text vector; The third step is to assign cluster centers: each word / text vector is assigned to the cluster center of the word / text vector that is most similar to it; The fourth step is to update the cluster centers: For each word / text vector cluster, calculate the average value of all its member word / text vectors and use it as the new word / text vector cluster center; The fifth step is to obtain the two-layer clustering model: repeat steps two through four until the word / text vector cluster centers no longer change or the maximum number of iterations is reached, thus obtaining the two-layer clustering model.

[0010] Furthermore, constructing the two-layer clustering model also includes using the silhouette coefficient, cohesion index, and separation index to evaluate the quality of the clustering results of the two-layer clustering model.

[0011] Furthermore, the trained word vector model and text vector model are obtained by continuously training the word vector model and text vector model on the preprocessed dataset, respectively.

[0012] Furthermore, the specific process of the new text entering the two-layer clustering model for processing and analysis is as follows: The preprocessed new text is used to calculate the word vectors and text feature vectors of the new text using the trained word vector model and text vector model, respectively. The vocabulary classification model is used to calculate the vocabulary classification of each word vector in the new text, and the word frequency under each classification is calculated. The trained text classification model is used to classify the text feature vectors of new texts. Cross-validation is performed on the text classification and vocabulary classification of new texts to check whether there is a contradiction between the meaning of the vocabulary classification with the highest word frequency in the text classification and the meaning of the text classification. When the number and frequency of words outside the existing vocabulary classification generated by the new text reach the threshold, the two-layer clustering model is updated.

[0013] Furthermore, the preprocessing steps are as follows: The first step is data cleaning, which includes rules such as removing invalid text; removing duplicate text; deleting excessively long or short text; removing meaningless text consisting solely of English letters, numbers, punctuation marks, and special characters; and correcting typos. The second step is text segmentation: the cleaned text is segmented into words to obtain the original vocabulary; The third step is stop word processing: stop words in the original vocabulary are removed using a pre-built library of commonly used Chinese stop words.

[0014] Furthermore, the preprocessing step also includes adding the automotive industry and brand-related proper nouns that were sorted out during word segmentation to a pre-set custom dictionary.

[0015] Furthermore, the word segmentation process uses the jieba Chinese word segmentation tool.

[0016] Furthermore, the word vector model and the text vector model are respectively the Word2vec model and the Doc2vec model.

[0017] Furthermore, there are two ways to obtain datasets of automotive community text: collecting automotive community text data within the private domain of car manufacturers, and crawling long automotive text information from major automotive forums and websites using web crawling tools.

[0018] The present invention provides an automatic text classification method based on automotive communities, which has the following advantages: 1. This invention uses word vector model and text vector model to train the car community text, and obtains word vector and text vector respectively. This facilitates the management and analysis of word frequency and the classification of text. Then, the Chameleon clustering algorithm is applied to cluster the car community text data from two dimensions: word vector and text vector. The classification effect is better and is not limited by the shape of the data. 2. This invention optimizes the similarity calculation method in the Chameleon clustering algorithm to the centered cosine similarity algorithm. Since the centered cosine similarity calculation method is faster, improves the accuracy and robustness of clustering, and takes into account the offset, it calculates the similarity of text vectors faster and more accurately. 3. This invention adds the organized automotive industry and brand-related proper nouns to a pre-set custom dictionary during word segmentation, avoiding segmentation errors caused by failure to recognize proper nouns and increasing segmentation accuracy; 4. This invention initially acquires a large amount of text data from automotive communities, and then continuously trains text vector and word vector models. It further combines the chameleon clustering algorithm and the centered cosine similarity algorithm to continuously train word vectors and text feature vectors, resulting in a vocabulary classification model and a text classification model. Therefore, when new text enters the system, it can more quickly calculate the text feature vectors, vocabulary classification, word frequency, and text classification of the new text. This saves working time and improves work efficiency. When the number of words and word frequency outside the existing vocabulary categories generated by new text reaches a threshold, the two-layer clustering model is updated. 5. This invention achieves end-to-end text management by preprocessing data, calculating text feature vectors and word vectors for new text, and then calculating lexical classification, word frequency, and text classification. Finally, it performs dynamic analysis on lexical classification, word frequency, and text classification. Attached Figure Description

[0019] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0020] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0021] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0022] This embodiment proposes an automatic text classification method based on automotive communities, including the following steps: I. Data Preprocessing: Obtain the dataset of car community texts and perform preprocessing.

[0023] Specifically, the steps include the following: (1) Data Collection. Collect text data from the private domains of car companies' automotive communities, and use the Selenium web crawler to crawl relevant long text information from major automotive forums and websites. Since the main purpose is to classify text, short texts may only be phrases or long words, which do not meet the requirements of text classification. Therefore, it is necessary to emphasize that it is long text, and then integrate these data into a dataset.

[0024] (2) Data cleaning. The rules include removing invalid text; removing duplicate text; deleting excessively long or short text; removing meaningless text consisting solely of English, numbers, punctuation marks, and special characters; and correcting typos.

[0025] (3) Text Segmentation: The cleaned text is segmented to obtain the original vocabulary. In this embodiment, the jieba Chinese word segmentation tool can be used. In addition, to improve the accuracy of word segmentation, the organized automotive industry and brand-related professional terms should be added to the pre-set custom dictionary. The professional dictionary needs to be added to the custom dictionary during word segmentation because the dictionary may not have enough automotive professional terms. This way, as many automotive professional terms as possible can be identified during word segmentation.

[0026] (4) Stop word processing: Use a pre-built library of commonly used Chinese stop words to remove stop words from the original vocabulary.

[0027] 2. Obtaining word vectors and text feature vectors: The preprocessed dataset is trained using word vector model and text vector model to obtain the word vector of each word and the text feature vector of each text, which are used as input values ​​for the two-layer clustering model.

[0028] In this embodiment, the word vector model can be trained using the Word2vec (Skip-gram) model on the processed dataset to obtain the word vector for each word; the text vector model can be trained using the Doc2vec (PV-DBOW) model on the processed dataset to obtain the feature vector for each text.

[0029] III. Constructing a Two-Layer Clustering Model: The similarity calculation method in the Chameleon clustering algorithm is optimized to a centered cosine similarity algorithm. Clustering calculations are performed on word vectors and text feature vectors to generate word classifications and text classifications respectively, thus forming a two-layer clustering model: a word classification model and a text classification model. The purpose of optimizing the similarity calculation method in the Chameleon clustering algorithm to a centered cosine similarity algorithm is to improve the accuracy and robustness of clustering. The quality of the clustering results of the two-layer clustering model is evaluated using silhouette coefficient, cohesion, and separation indices. For the silhouette coefficient, the closer it is to 1, the better the clustering effect; the higher the separation and the lower the cohesion, the better the clustering effect.

[0030] Here's a brief introduction to the centered cosine similarity algorithm. It uses the cosine of the angle between two vectors in a vector space as a measure of the difference between the two individuals. A cosine value closer to 1 and an angle approaching 0 indicates that the two vectors are more similar, while a cosine value closer to 0 and an angle approaching 90 degrees indicates that the two vectors are less similar.

[0031] The specific steps for constructing a two-layer clustering model are as follows: The first step is to initialize cluster centers: randomly select some word / text vectors from the trained dataset as multiple initial word / text vector cluster centers, with each word / text vector cluster center representing a category; The second step is to calculate the similarity: use the centered cosine similarity algorithm to calculate the similarity between each word / text vector and the cluster center of each word / text vector; The third step is to assign cluster centers: each word / text vector is assigned to the cluster center of the word / text vector that is most similar to it; The fourth step is to update the cluster centers: For each word / text vector cluster, calculate the average value of all its member word / text vectors and use it as the new word / text vector cluster center; The fifth step is to obtain the two-layer clustering model: repeat steps two through four until the word / text vector cluster centers no longer change or the maximum number of iterations is reached, thus obtaining the two-layer clustering model.

[0032] In particular, because steps two through four require repeated calculations, and the cluster centers are updated after each calculation, the first calculation is the initial cluster center, and from the second calculation onwards it is each cluster center.

[0033] Specifically, the results of the vocabulary classification are back-mapped onto the original vocabulary to obtain a mapping table between the original vocabulary and the vocabulary classification.

[0034] The process of processing and analyzing new text entering the two-layer clustering model is as follows: The first step is new text preprocessing. This involves cleaning the new text and then using the jieba Chinese word segmentation tool and a pre-built library of commonly used Chinese stop words to perform word segmentation and stop word processing.

[0035] The second step involves using a trained word vector model and a text vector model to calculate the word vectors and text feature vectors of the preprocessed new text. The vocabulary classification model is used to calculate the vocabulary category of each word vector in the new text, and the word frequency within each category is calculated. The trained text classification model is then used to classify the text feature vectors of the new text. In other words, after calculating the feature vectors of the newly entered text using the trained text classification model, the distance between the text feature vectors and each cluster center is calculated using the centered cosine similarity algorithm, and the new text is assigned to the text vector category represented by the cluster center closest to it. In this embodiment, the number and frequency of words outside the existing vocabulary categories can also be calculated simultaneously, and the results of each calculation are retained. This method is beneficial for quickly checking whether the model data source needs to be updated (an increase in words in the newly entered text that do not belong to any category indicates that the original model is not fully covered and needs to be updated) and word frequency calculation.

[0036] The third step involves dynamic analysis of the vocabulary classification and word frequency of the new text, as well as the text classification of the already classified new text.

[0037] After the above process is completed, each text will correspond to one text classification result and multiple vocabulary classification results, along with the corresponding word frequencies under each vocabulary category. This will ultimately generate a statistical table of text classification × vocabulary category. For each text classification, all word frequencies within each vocabulary category are summed. By summarizing each text classification and vocabulary category, the text meaning (e.g., vehicle issues) is determined, and cross-validation of text and vocabulary classifications is performed. Manual verification is conducted to check whether there are any contradictions between the meanings of vocabulary categories with high word frequencies in the text classification and the meanings of the text classifications. In other words, the purpose of dynamic analysis is to help cross-validate the accuracy of the classification and, through the two-level classification results, quickly locate the vehicle issue category within the community text.

[0038] The fourth step is to update the two-layer clustering model. When the number and frequency of words outside the existing word categories generated by the new text reach a threshold (1‱ of the historical training samples), the two-layer clustering model is updated.

[0039] All of the above models were trained on massive amounts of data. Once new text enters, they can automatically segment the text, calculate word vectors, text feature vectors, and classify the words and text.

[0040] The present invention provides an automatic text classification method based on automotive communities, which has the following advantages: 1. This invention uses word vector model and text vector model to train car community text to obtain word vectors and text vectors, which facilitates word frequency management and analysis and text classification. Then, the Chameleon clustering algorithm is applied to cluster the car community text data from two dimensions: word vectors and text vectors. The classification effect is better and is not limited by the shape of the data.

[0041] 2. This invention optimizes the similarity calculation method in the Chameleon clustering algorithm to the centered cosine similarity algorithm. Since the centered cosine similarity calculation method is faster, improves the accuracy and robustness of clustering, and takes into account the offset, it calculates the similarity of text vectors faster and more accurately.

[0042] 3. This invention adds the organized automotive industry and brand-related proper nouns to a pre-set custom dictionary during word segmentation, avoiding segmentation errors caused by failure to recognize proper nouns and increasing segmentation accuracy.

[0043] 4. This invention initially acquires a large amount of text data from automotive communities, and then continuously trains text vector and word vector models. It further combines the chameleon clustering algorithm and the centered cosine similarity algorithm to continuously train word vectors and text feature vectors, resulting in a vocabulary classification model and a text classification model. Therefore, when new text enters the system, it can more quickly calculate the text feature vectors, vocabulary classification, word frequency, and text classification of the new text. This saves working time and improves work efficiency. When the number of words and word frequency outside the existing vocabulary categories generated by new text reaches a threshold, the two-layer clustering model is updated. 5. This invention achieves end-to-end text management by preprocessing data, calculating text feature vectors and word vectors for new text, and then calculating lexical classification, word frequency, and text classification. Finally, it performs dynamic analysis on lexical classification, word frequency, and text classification.

[0044] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for automatic text classification based on car communities, characterized in that, Includes the following steps: Data preprocessing: Obtain the dataset of text from the automotive community and perform preprocessing; Obtaining word vectors and text feature vectors: The preprocessed dataset is trained using word vector model and text vector model to obtain the word vector of each word and the text feature vector of each text, which are used as input values ​​for the two-layer clustering model; Constructing a two-layer clustering model: Clustering calculations are performed on word vectors and text feature vectors to generate word classification and text classification respectively, resulting in word classification models and text classification models, specifically: The first step is to initialize cluster centers: using the Chameleon clustering algorithm, some words / text vectors in the trained dataset are randomly selected as multiple initial word / text vector cluster centers, and each word / text vector cluster center represents a category; The second step is to calculate the similarity: use the centered cosine similarity algorithm to calculate the similarity between each word / text vector and the cluster center of each word / text vector; The third step is to assign cluster centers: each word / text vector is assigned to the cluster center of the word / text vector that is most similar to it; The fourth step is to update the cluster centers: For each word / text vector cluster, calculate the average value of all its member word / text vectors and use it as the new word / text vector cluster center; The fifth step is to obtain the two-layer clustering model: repeat steps two through four until the word / text vector cluster centers no longer change or the maximum number of iterations is reached, thus obtaining the two-layer clustering model; New text is processed and analyzed using a two-layer clustering model: For the pre-processed new text, a trained word vector model and a text vector model are used to calculate the word vectors and text feature vectors of the new text, respectively; the vocabulary classification model is used to calculate the vocabulary classification of each word vector in the new text, and the word frequency under each classification is calculated; a trained text classification model is used to classify the text feature vectors of the new text; the text classification and vocabulary classification of the new text are cross-validated to check whether there is a contradiction between the meaning of the vocabulary classification with the highest word frequency in the text classification and the meaning of the text classification. When the number and frequency of words outside the existing vocabulary classification generated by the new text reach a threshold, the two-layer clustering model is updated.

2. The automatic text classification method based on car communities according to claim 1, characterized in that, The construction of the two-layer clustering model also includes using the silhouette coefficient, cohesion index, and separation index to evaluate the quality of the clustering results of the two-layer clustering model.

3. The automatic text classification method based on car communities according to claim 1, characterized in that, The trained word vector model and text vector model are obtained by continuously training the word vector model and text vector model on the preprocessed dataset, respectively.

4. The automatic text classification method based on car communities according to claim 1, characterized in that, The preprocessing steps are as follows: The first step, data cleaning: its rules include, Remove invalid text; remove duplicate text; delete excessively long or short text; remove meaningless text consisting solely of English letters, numbers, punctuation marks, and special characters; and correct typos. The second step is text segmentation: the cleaned text is segmented into words to obtain the original vocabulary; The third step is stop word processing: stop words in the original vocabulary are removed using a pre-built library of commonly used Chinese stop words.

5. The automatic text classification method based on car communities according to claim 4, characterized in that, The preprocessing steps also include adding the automotive industry and brand-related proper nouns that were sorted out during word segmentation to a pre-set custom dictionary.

6. The automatic text classification method based on car communities according to claim 5, characterized in that, The word segmentation process used is the jieba Chinese word segmentation tool.

7. The automatic text classification method based on car communities according to any one of claims 1 to 6, characterized in that, The word vector model and text vector model are respectively the Word2vec model and the Doc2vec model.

8. The automatic text classification method based on car communities according to claim 1, characterized in that, There are two ways to obtain datasets of automotive community text: collecting automotive community text data within the private domains of car manufacturers and crawling long-form automotive text information from various automotive forums and websites using web crawling tools.

Images