Data mining method based on user comments

Abstract

The invention relates to the technical field of information data processing, in particular to a data mining method based on user comments. The method is particularly suitable for data mining of comments published by users in web portals, e-commerce platforms and the like. According to the method, a text mining method combining a TF-IDF algorithm and a cosine similarity function is utilized to perform keyword extraction on user comments and realize short sentence classification of comment information, so that preferences, interests and attitudes of users are known, and an important basis is provided for personalized recommendation of the users.

Description

Technical Field

[0001] The present invention relates to the technical field of information data processing, and relates to a data mining method based on user comments, and is particularly suitable for data mining of comments posted by users of portal websites, e-commerce platforms, etc. Background Art

[0002] With the rapid popularization of the Internet and the development of social networking platforms, the amount of data generated by users has exploded. Information such as comments, opinions, and evaluations posted on these social networks has become a valuable data resource, containing rich information such as emotions, preferences, and attitudes. On the other hand, companies are eager to understand users' preferences, interests, and attitudes in order to better meet user needs, improve products or services, and make accurate personalized recommendations. However, the traditional way of understanding user needs is relatively simple and cannot obtain users' invisible needs. Moreover, with the rapid increase in the scale of information, manual processing alone is no longer sufficient for this task. Therefore, a research hotspot in the field of natural language processing has been promoted, namely user comment data mining. Analyzing user comment information can help companies understand users' evaluation and feedback on products, services, activities, etc., thereby guiding business decisions. Summary of the invention

[0003] The purpose of the present invention is to solve the shortcomings of existing data mining methods and provide a text mining method combining the TF-IDF algorithm and the cosine similarity function to extract keywords from user comments, realize short sentence classification of comment information, and then understand the user's preferences, interests and attitudes, providing an important basis for personalized recommendations for users.

[0004] A data mining method based on user comments comprises the following steps: S1. Obtain keywords of user comments based on TF-IDF algorithm; S2, using the cosine similarity function to calculate the similarity in the comments and classify them; S3, extract keywords to classify short sentences in comments; In S1, the keywords that users are interested in are extracted through the TF-IDF algorithm. TF-IDF consists of two parts: one is TF term frequency and the other is IDF inverse document frequency. TF refers to the frequency of keywords appearing in the text, and its calculation formula is (1) in is the number of times the keyword appears in the text, and the denominator is the sum of the number of times all the keywords appear in the record; IDF is the inverse document frequency. The stronger a word's ability to represent a text is, the greater its weight and the greater its importance. IDF is used to reflect the importance of this word. IDF is obtained by dividing the total number of documents by the number of documents containing the word, and then taking the logarithm of the quotient. The basic formula of IDF is formula (2) Where |D| is the total number of texts in the text library. |{j:t i ∈d f}| is the number of texts containing the keyword t; in case the keyword is not in the vocabulary, 1+|{j:t i ∈d f}|; The TF-IDF calculation formula is as follows (3): TF refers to the word frequency of a keyword in the current text, and IDF is the inverse document frequency. The TF-IDF algorithm is used to extract words that users pay more attention to from the comment information.

[0005] The keywords extracted in S3 are used to classify short comment sentences, and the TF-IDF value is calculated for each word in the comment information; the word with the highest TF-IDF value is selected as the keyword; the comment information is segmented according to semantic words to establish a comment information set D, where D={d1, d2d3, d4,…}, where d1, d2d3, d4,… are specific word segments.

[0006] Extract the keywords of the specific word segmentation of the comment and calculate the TF value of each word segmentation; then, we calculate the TF and IDF values ​​of each word in the entire comment information set D; calculate the TF-IDF value of each word, and select the word with a high TF-IDF value as the keyword; extract multiple keywords and select them in order from high to low according to the TF-IDF value.

[0007] Beneficial effects of the present invention: The text mining method combining the TF-IDF algorithm and the cosine similarity function is used to extract keywords from user comments and classify short sentences in comment information, thereby understanding user preferences, interests and attitudes, providing an important basis for personalized recommendations for users. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is a schematic diagram of the mathematical model of cosine similarity; Figure 2 Data similarity calculation flow chart; Figure 3 Data classification flow chart. DETAILED DESCRIPTION

[0009] A data mining method based on user comments includes the following steps: S1. Obtain keywords of user comments based on TF-IDF algorithm S2. Use the cosine similarity function to calculate the similarity in the comments and classify them S3. Extract keywords to classify short sentences in comments The keywords that users are interested in are extracted through the TF-IDF algorithm. TF-IDF mainly consists of two parts, one is TF (term frequency) and the other is IDF (inverse document frequency).

[0010] TF refers to the frequency of a keyword appearing in a text. Its calculation formula is as follows: (1).

[0011] in It is the number of times the keyword appears in the text, and the denominator is the total number of times all the keywords appear in the record.

[0012] IDF is the inverse document frequency. The stronger a word's ability to represent a text is, the greater its weight and the greater its importance. IDF is used to reflect the importance of this word. IDF is obtained by dividing the total number of documents by the number of documents containing the word, and then taking the logarithm of the quotient. The basic formula of IDF is formula (2) Where |D| is the total number of texts in the text library. |{j:t i ∈d f}| is the number of texts containing the keyword t. In case the keyword is not in the vocabulary, 1+|{j:t i ∈d f}|.

[0013] The TF-IDF calculation formula is as follows (3): TF refers to the frequency of a keyword in the current text, and IDF is the inverse document frequency. The TF-IDF algorithm can be used to extract words that users pay more attention to from the comment information. As follows: (1) Calculate the TF-IDF value for each word in the comment information.

[0014] (2) Select the word with the highest TF-IDF value as the keyword For example, if we have the following review information set D: Extract comments d 1 keyword, first we calculate d1. Then, we calculate the TF and IDF values ​​of each word in the entire review information set D. Finally, we calculate the TF-IDF value of each word and select the highest ones as keywords.

[0015] Assuming we only consider the words "product" and "very", using formula (1) Calculate the TF value of "product": TF("product", D) = 4 (because it appears in 4 reviews in review information set D) Calculate the TF value of "very": TF("very", D) = 2 (because it appears in 2 comments in the comment information set D) Then, we calculate the d 1. Assume that the product d 1 appears once, "very" appears once, and N d 1 Yes d 1 The total number of entries (assuming d 1 has 10 words in total). Then: TF("Product", d 1)= TF("very", d 1)= Next, we use formula (2) to calculate the IDF value: IDF("Product", D) = =0 IDF(“very”, D)= = =0.301 Calculate TF-IDF using formula (3) TF-IDF("Product", d 1, D) = TF("Product", d 1) ×IDF("Product", D)= ×0=0 TF-IDF("very", d 1, D) = TF("very", d 1) ×IDF(“very”, D)= ×0.301=0.0301 In this example, "very" has a positive TF-IDF value, while "product" has a TF-IDF value of 0 because its IDF value is 0. This indicates that "very" is more popular in reviews. d 1 is a more important keyword.

[0016] Use the space vector model (VSM) to represent it as a feature vector to prepare for the similarity calculation in step S2 Step 1: Extract the key words in the text, form feature items and represent them as T (T1, T2, T3, ..., T n ); Step 2: Use TF-IDF to calculate the weight of each keyword in the text, expressed as W (W1, W2, W3, ..., W n ) Step 3: Simplify the characteristic terms to: D = W (W1, W2, W3, ..., W n ); Example of feature vector: Consider a comment as a space vector. Each word in the comment represents a dimension in the vector space. The number of times the word appears in the comment represents a vector. For example, if the content of a text is "This product is very good, I like it very much", then the result of word segmentation is "This / product / is / very / good / , / I / like / it / very / much". This text can be regarded as a vector in a seven-dimensional space. Its vector representation is: The numbers in the corresponding rows are the number of times "this", "product", "good", "I", "," "very", and "like" appear.

[0017] Further, the step 2 specifically includes the following: In step 1, the vector space representation of the text has been obtained, and the cosine similarity function is used to calculate the similarity between the vectors of different weighted texts.

[0018] The mathematical model of cosine similarity is as follows: Figure 1 shown.

[0019] The cosine similarity calculation formula is as follows: If two text space vectors are expressed as formula (6).

[0020] Among them, Wi,k, Wj,k are the weight representations of text feature items.

[0021] Then, by substituting it into the calculation formula (5), we can obtain the similarity calculation formula (7) of the text: The text similarity calculation process is as follows Figure 2In this case, the cosine similarity function is used to calculate the similarity in the comments and classify them. Classification by similarity is more targeted. The calculation of the cosine similarity function in this step is a conventional public technology. This case is only applied in conjunction with steps S1 and S3.

[0022] Calculation Example Assume the feature vector of the first comment is: Assume the feature vector of the second comment is: Assume the feature vector of the second comment is: Then use formula (7) to calculate the similarity between the three comments, Similarity between the first and second items: Similarity between the first and third items: Similarity between Article 2 and Article 3: From the calculation results, we can see that the similarity between the first and second items is the highest, and the similarity between the second and third items is the lowest. Furthermore, the step 3 specifically includes the following: The keywords extracted based on the TF-IDF algorithm are used as the basis for classifying the comment text. The process is as follows: Figure 3 In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the present invention, that is, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Example

[0023] We use crawler technology to obtain user comments from Taobao and store them in the database, and randomly select 5,000 comments from them. Some user comments crawled for a certain brand of electric kettle are as follows: Table 1 Some collected user comments Some keywords extracted in step 1 and their weights Table 2 Some keywords and their weights The user's comments on the kettle stored in the database are used as the extraction source, and the keywords are extracted. The corresponding relationship between the keywords and the evaluation words is sorted out, as shown in Table 3. Table 3 Correspondence between some extracted keywords and evaluation words Through step 2 Figure 2 The text similarity calculation process, the calculation results of some keyword similarities are shown in Table 4. The similarities between comment sentences calculated based on keyword similarity are shown in Table 5.

[0024] Table 4 Similarity calculation of some keywords Table 5 Similarity calculation of some comment sentences Through step 3 Figure 3 Text classification flow chart,classification is performed by matching the classification word list and manual processing.,Part of the classification results are shown in Table 6.

[0025] Table 6. Correspondence table of some keywords and their comment sentences Glossary TF-IDF (term frequency–inverse document frequency) is a commonly used weighting technique for information retrieval and data mining. TF stands for term frequency, and IDF stands for inverse document frequency.

[0026] Vector Space Model (VSM) is a model commonly used in information retrieval and text mining.

Claims

1. A data mining method based on user comments, characterized in that The steps include: S1. Obtain keywords of user comments based on TF-IDF algorithm; S2, using the cosine similarity function to calculate the similarity in the comments and classify them; S3, extract keywords to classify short sentences in comments; In S1, the keywords that users are interested in are extracted through the TF-IDF algorithm. TF-IDF consists of two parts: one is TF term frequency and the other is IDF inverse document frequency. TF refers to the frequency of keywords appearing in the text, and its calculation formula is (1) （1） where n i, j is the number of times the keyword appears in the text, and the denominator is the sum of the number of times all the keywords appear in the record; IDF is the inverse document frequency. The stronger a word's ability to represent a text is, the greater its weight and the greater its importance. IDF is used to reflect the importance of this word. IDF is obtained by dividing the total number of documents by the number of documents containing the word, and then taking the logarithm of the quotient. The basic formula of IDF is formula (2) （2） Where |D| is the total number of texts in the text library; |{j:t i ∈d f }| is the number of texts containing the keyword t; in case the keyword is not in the vocabulary, 1+|{j:t i ∈d f }|; The TF-IDF calculation formula is as follows (3): (3) TF refers to the word frequency of a keyword in the current text, and IDF is the inverse document frequency. The TF-IDF algorithm is used to extract words that users pay more attention to from the comment information.

2. A data mining method based on user comments according to claim 1, characterized in that The keywords extracted in S3 are used to classify the short sentences in the comments, and the TF-IDF value is calculated for each word in the comment information; the word with the highest TF-IDF value is selected as the keyword; the comment information is segmented according to semantic words to establish a comment information set D, where D={d1, d2 d3, d4,…}, where d1, d2 d3, d4,… are specific segmentations; Extract the keywords of the specific word segmentation of the comment and calculate the TF value of each word segmentation; then, we calculate the TF and IDF value of each word in the entire comment information set D; calculate the TF-IDF value of each word, and select the word with the highest TF-IDF value as the keyword; Extract multiple keywords and select them in order from high to low according to TF-IDF values.

Images