Three-dimensional customer demand dynamic mining method and system based on social media big data

Abstract

The present application relates to the technical field of data processing, in particular to a three-dimensional customer demand dynamic mining method and system based on social media big data. The method has: S1, data preprocessing; S2, constructing a feature dictionary of the target product; S3, obtaining evaluation tendency data of each attribute feature of the target product in the comment data; S4, establishing a demand classification model. The system is used to realize the above method. The present application can preferably mine the customer demand in actual production, so that the manufacturer can comprehensively consider the customer's attention, satisfaction and the importance of product features to customers, and provide suggestions for the design and investment of the next generation of products.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically, to a method and system for dynamic mining of three-dimensional customer needs based on social media big data. Background Technology

[0002] In a highly competitive and challenging market environment, product manufacturers must not only enrich product functionality and improve product quality, but also meet customer needs. Whether a designed product meets the actual needs of the target customer group plays a crucial role in product sales and new product design. However, customer needs are often vague and implicit. Customer needs contained in questionnaires and online reviews are implicit and difficult to directly assist in the design and manufacturing of next-generation products. Therefore, how to clearly and meticulously uncover customer needs from multiple perspectives based on massive amounts of online reviews has become an urgent problem to solve. Furthermore, customer needs are variable and diverse. Customer needs often change with product iterations and may be influenced by various factors, such as product category, the social status and economic strength of the target customer group. In addition, the degree to which customers pay attention to different product features also changes over time. Manufacturers must not only accurately grasp the preference differences between different customer groups, but also pay attention to changes in their needs in the context of the overall market environment. Therefore, how to accurately and timely identify customer needs and analyze the patterns of change in the needs of the target customer group has become a difficult problem for manufacturers to solve. Summary of the Invention

[0003] This invention provides a method for dynamically mining three-dimensional customer needs based on social media big data, which can overcome some or all the defects of existing technologies.

[0004] The method for dynamic mining of three-dimensional customer needs based on social media big data according to the present invention is used to analyze customer needs for target products, and specifically includes the following steps:

[0005] S1, Data Preprocessing

[0006] In this step, the first step is to obtain the review data of the target product, then the review words in the review data, and finally the word vectors of the review words.

[0007] S2. Construct the feature dictionary of the target product

[0008] In this step, the feature dictionary includes multiple feature sets, with different feature sets corresponding to different attribute features of the target product. Each feature set has at least one feature word, which is selected from the comment words.

[0009] S3. Obtain the evaluation tendency data for each attribute feature of the target product from the review data.

[0010] In this step, the review data is divided into review clauses according to the product feature dictionary, and each review clause corresponds to one and only one related attribute feature; by analyzing the review clauses, evaluation tendency data is obtained.

[0011] S4. Establish a demand classification model

[0012] In this step, the established demand classification model includes the importance of attribute features, the rate of change in satisfaction with attribute features of different two generations of target products, and the rate of change in attention to attribute features of different two generations of target products.

[0013] The method of this invention can extract comment data of target products from social media big data (step S1), extract relevant feature words from the comment data (step S2), and summarize the target products through attribute features. Then, by obtaining the evaluation tendency data of each attribute feature in the comment data (step S3), it is possible to compare two different generations of target products (step S4), thereby realizing the establishment of a demand classification model. This can effectively explore the customer needs of specific products, monitor changes in customer needs, and is of great significance for the redesign of new products.

[0014] Preferably, step S2 includes the following steps.

[0015] S21. Extract original feature words from the self-comment words;

[0016] S22. Cluster the original feature words, and each category after clustering is a feature set;

[0017] S23. Expand the feature set by adding expanded feature words.

[0018] Based on the above, it is possible to better extract the product attribute features that customers care about from the review data.

[0019] Preferably, in step S21, comment words with a length not less than a set word length threshold, a part of speech of nouns, and a frequency of occurrence in the comment data not less than a set word frequency value are selected as the original feature words. Therefore, the extraction of original feature words can be achieved better.

[0020] As a preferred approach, the similarity between original feature words is measured based on the cosine similarity value between word vectors, and the original feature words are clustered using a hierarchical clustering method. Therefore, it is possible to better summarize each class based on the clustering results, forming a feature vocabulary set (i.e., feature set) of multiple categories.

[0021] Preferably, for any feature set, relevant comment words with a maximum similarity not lower than a set text similarity threshold and a noun part of speech are selected from the comment words as expanded feature words. This allows for a better expansion of the feature set, and thus a better enrichment of the feature dictionary.

[0022] Preferably, in step S3, the comment data is divided into comment clauses based on the product feature dictionary, using the comment words obtained in S1. Each relevant comment word is sorted according to the sentence order of its corresponding single comment. For each single comment, the similarity between each relevant comment word and each feature set is calculated. If the similarity is greater than a set feature set similarity threshold, the relevant single comment is segmented at the punctuation mark closest to the comment word to obtain the corresponding comment clause. Therefore, this method can better achieve the acquisition of evaluation tendency data.

[0023] As a preferred option, the importance of attribute features is obtained based on the following formula. Among them, IoPF m That is, the importance of the m-th attribute feature. Let n be the number of times the p-th feature word in the m-th feature set corresponding to the m-th attribute appears in the comment data. m Let M be the total number of feature words in the m-th feature set, and M represent the total number of feature sets.

[0024] Based on the above, we can obtain the total number of occurrences of each feature word in each feature set in the comment data. By normalizing this total number, we can better use the obtained data as the importance of the attribute features.

[0025] As a preferred option, the formula for calculating the rate of change in attribute feature satisfaction is as follows: in, Let m be the rate of change in customer satisfaction between the g+1 generation product and the g generation product on attribute feature m. and That is, the satisfaction level of the m-th attribute feature of the g-th and g+1-th generation products;

[0026] For any generation of products, the formula for calculating customer satisfaction is: in, This represents the satisfaction level of the m-th attribute feature of the g-th generation product. This represents the number of positive comment clauses that match the attribute characteristics of the g-th product with those of the m-th product. This represents the number of neutral comment clauses that match the features of the g-th generation product with those of the m-th product. This represents the number of comment clauses that match the attribute characteristics of the g-th product with those of the m-th product.

[0027] Therefore, it can better analyze the changes in satisfaction with sexual characteristics between two different generations of products.

[0028] As a preferred option, the formula for calculating the rate of change in attribute feature attention is as follows: in, This refers to the rate of change in customer attention regarding attribute feature m between the (g+1)th generation product and the gth generation product. and This refers to the attention given to the m-th attribute feature of the g-th and g+1-th generation products;

[0029] For any generation of product, the formula for calculating its attention is: in, This indicates the level of attention given to the m-th attribute feature of the g-th generation product. This represents the number of comment clauses that match the attribute characteristics of the g-th product with those of the m-th product. This represents the maximum value of the comment clauses in the g-th generation product that match the relevant product attribute characteristics.

[0030] Therefore, it is able to better analyze the changes in attention to sexual characteristics between two different generations of products.

[0031] Furthermore, this invention also provides a three-dimensional customer demand dynamic mining system based on social media big data, used to implement any of the above methods, comprising:

[0032] The acquisition unit is used to implement step S1;

[0033] A building unit, which is used to implement step S2;

[0034] Processing unit, which is used to implement step S3; and

[0035] The display unit is used to implement step S4.

[0036] Therefore, by better understanding customer needs in actual production, manufacturers can comprehensively consider customer attention, satisfaction, and the importance of product features to customers, and provide suggestions for the design and investment of next-generation products. Attached Figure Description

[0037] Figure 1 This is a flowchart illustrating the method in Example 1;

[0038] Figure 2 This is a schematic diagram of the deep neural network architecture in Example 1;

[0039] Figure 3 This is a schematic diagram illustrating the classification model in the embodiment. Detailed Implementation

[0040] To further understand the content of this invention, a detailed description of the invention is provided in conjunction with embodiments. It should be understood that the embodiments are merely illustrative and not limiting of the invention.

[0041] Example 1

[0042] Seen in Figure 1 This embodiment provides a three-dimensional dynamic mining method for customer needs based on social media big data. It can realize multi-dimensional mining and analysis of the potential needs of customer groups based on online customer review data, thereby enabling product manufacturers to obtain relevant guidance data from the changing patterns of customer needs.

[0043] The method provided in this embodiment is used to analyze customer needs for a target product, and specifically includes the following steps:

[0044] S1, Data Preprocessing

[0045] In this step, the first step is to obtain the review data of the target product, then the review words in the review data, and finally the word vectors of the review words.

[0046] S2. Construct the feature dictionary of the target product

[0047] In this step, the feature dictionary includes multiple feature sets, with different feature sets corresponding to different attribute features of the target product. Each feature set has at least one feature word, which is selected from the comment words.

[0048] S3. Obtain the evaluation tendency data for each attribute feature of the target product from the review data.

[0049] In this step, the review data is divided into review clauses according to the product feature dictionary, and each review clause corresponds to one and only one related attribute feature; by analyzing the review clauses, evaluation tendency data is obtained.

[0050] S4. Establish a demand classification model

[0051] In this step, the established demand classification model includes the importance of attribute features, the rate of change in satisfaction with attribute features of different two generations of target products, and the rate of change in attention to attribute features of different two generations of target products.

[0052] The method in this embodiment can extract review data of target products from social media big data (step S1), extract relevant feature words from the review data (step S2), and summarize the target products through attribute features. Then, by obtaining the evaluation tendency data of each attribute feature in the review data (step S3), it is possible to compare two different generations of target products (step S4), thereby realizing the establishment of a demand classification model. This can effectively explore the customer needs of specific products, monitor changes in customer needs, and is of great significance for the redesign of new products.

[0053] Step S1 allows for the efficient acquisition of online customer review data in text form, leveraging social media big data. It's understood that online customer review data typically includes three types of data: "evaluation bias data," "attribute features," and other data. Specifically, online customer reviews often use phrases like "a certain function or component of the product is good, bad, or average." In these phrases, "a certain function or component of the product" represents "attribute features," and "good, bad, or average" represents "evaluation bias data." By acquiring the review terms in step S1, the "evaluation bias data" and "attribute features" can be extracted from the online customer review data. Vectorizing these review terms allows for better subsequent algorithmic processing.

[0054] Through step S2, based on the data obtained in S1, the attribute features of the target product can be mined and the feature dictionary can be constructed, thereby enabling better discovery of the attribute features (such as performance, appearance and other attributes) that customers are concerned about regarding the target product.

[0055] Step S3 allows for the segmentation of online customer reviews, ensuring that each review clause contains both an "evaluation preference data" and an "attribute feature," thus enabling better collection of customer evaluation preferences for products.

[0056] Step S4 allows us to categorize the customer needs of the target customer group into customer attention to product features (rate of change in attribute feature attention), customer satisfaction (rate of change in attribute feature satisfaction), and relative importance (importance of attribute features), thereby analyzing the dynamic changes in customer needs of the target customer group between the two generations of products.

[0057] In step S1,

[0058] The aforementioned acquisition of review data for the target product can be obtained from social media big data. Specifically, it can be achieved by using Python-based web crawlers to collect online reviews from social web pages. For example, lightweight review data can be crawled using Python libraries such as bs4, requests, and selenium, and by utilizing XPath, CSS expressions, and regular expressions.

[0059] It is understandable that the comment data is a collection of multiple individual comments, each of which is expressed in text format;

[0060] The method for obtaining comment words from comment data involves first cleaning the comment data, then segmenting and tagging each individual comment. During data cleaning, noise such as URLs, special symbols, contact information, and HTML elements are removed from each individual comment. Then, a sentence length threshold is set, and comments shorter than the threshold are deleted, thus effectively filtering and retaining valid individual comments. Furthermore, the Harbin Institute of Technology (HIT) LTP language toolkit is used for segmentation of the processed comments, and the HIT stop word list is used to remove stop words from the comments, further enhancing the ability to obtain comment words for each individual comment.

[0061] The process of obtaining word vectors for comment words involves vectorizing each comment word, which facilitates subsequent data processing. Specifically, the CBOW model within Word2vec (word embedding model) can be used to obtain the word vectors for comment words.

[0062] In step S2, the following steps can be followed:

[0063] S21. Extract original feature words from the self-comment words;

[0064] S22. Cluster the original feature words, and each category after clustering is a feature set;

[0065] S23. Expand the feature set by adding expanded feature words.

[0066] Based on the above, it is possible to better extract the product attribute features that customers care about from the review data.

[0067] In step S21, comment words with a length not less than a set word length threshold (in this embodiment, the word length threshold is 2 characters), a part of speech of nouns, and a frequency of occurrence in the comment data not less than a set word frequency value (the word frequency value can be determined based on the amount of comment data, such as being set to 1 / 100 of the total number of comments in a single comment) are selected as the original feature words. Therefore, the extraction of original feature words can be achieved better.

[0068] In step S22, the similarity between original feature words can be measured based on the cosine similarity value between word vectors, and the original feature words can be clustered based on hierarchical clustering. Therefore, each class can be better summarized according to the clustering results, forming a feature vocabulary set (i.e., feature set) of multiple categories.

[0069] In step S23, for any feature set, relevant comment words with a maximum similarity not lower than a set text similarity threshold and a noun part of speech are selected from the comment words as expanded feature words. This allows for a better expansion of the feature set, and thus a better enrichment of the feature dictionary.

[0070] The maximum similarity is calculated by taking the cosine similarity value between each relevant comment word and each original feature word in the corresponding feature set, and the maximum cosine similarity value is taken as the maximum similarity. The text similarity threshold can be set based on the similarity between classes in the hierarchical clustering method.

[0071] In the specific implementation of step S3:

[0072] The method of dividing comment data into comment clauses based on the product feature dictionary can be based on the comment words obtained in S1. Specifically, each relevant comment word can be sorted according to the sentence order of the corresponding single comment. For each single comment, the similarity between each relevant comment word and each feature set is calculated one by one. If the similarity is greater than the set feature set similarity threshold, the relevant single comment is segmented at the punctuation mark closest to the comment word to obtain the corresponding comment clause, and the corresponding comment clause can be associated with the corresponding attribute feature.

[0073] In the above process, similarity can be evaluated based on the cosine similarity value of word vectors. Considering that each feature set may have multiple feature words, the maximum or average value of the similarity value between the relevant comment word and each feature word in each feature set can be used as the similarity between each relevant comment word and each feature set.

[0074] In the specific implementation of step S3:

[0075] The aforementioned acquisition of evaluation tendency data can be based on deep neural networks, thus enabling better analysis of each comment clause and thereby acquiring evaluation tendency data for the corresponding attribute features.

[0076] Seen in Figure 2In this embodiment, the deep neural network can sequentially have an input layer, a BERT word embedding layer, a Bi-LSTM layer, a CNN layer, a fully connected layer, and an output layer. The evaluation sentiment data in this embodiment can be defined as "positive", "negative", or "neutral". After the deep neural network is built, each comment clause is first labeled manually, that is, each comment clause is labeled as "positive", "negative", or "neutral". Then, a sample set consisting of comment clauses and associated labeled data is obtained. After that, by dividing the sample set into a training set, a test set, and a validation set, the deep neural network can be trained better.

[0077] The architecture of the deep neural network in this embodiment can adopt a conventional architecture, which will not be described in detail here.

[0078] In step S4, the importance of the attribute features can be obtained based on the following formula:

[0079]

[0080] Among them, IoPF m That is, the importance of the m-th attribute feature. Let n be the number of times the p-th feature word in the m-th feature set corresponding to the m-th attribute appears in the comment data. m Let M be the total number of feature words in the m-th feature set, and M represent the total number of feature sets.

[0081] Based on the above, we can obtain the total number of occurrences of each feature word in each feature set in the comment data. By normalizing this total number, we can better use the obtained data as the importance of the attribute features.

[0082] In addition, in order to obtain the change rate of attribute feature satisfaction and the change rate of attribute feature attention for different two generations of target products, relevant data of different two generations of target products can be obtained according to steps S1-S3. By simultaneously obtaining the number of comment clauses associated with the feature set and the number of different evaluation tendencies in the comment clauses (i.e., the number of "positive", "negative" or "neutral" evaluation tendencies, which constitute the overall evaluation tendency data) in step S3, the change rate of attribute feature satisfaction and the change rate of attribute feature attention can be obtained better.

[0083] Specifically, when obtaining the number of comment clauses associated with a feature set, for the current feature set, if there are comment words in the related comment clauses with a similarity greater than the set feature set similarity threshold, then count 1; otherwise, count 0. The number can be obtained by accumulating these numbers.

[0084] Specifically, when obtaining the number of different evaluation tendencies in relevant comment clauses, the deep neural network in step S3 can be used to determine and count them separately.

[0085] Specifically, for any generation of products, the formula for calculating customer satisfaction can be:

[0086]

[0087] in, This represents the satisfaction level of the m-th attribute feature of the g-th generation product. This represents the number of positive comment clauses that match the attribute characteristics of the g-th product with those of the m-th product. This represents the number of neutral comment clauses that match the features of the g-th generation product with those of the m-th product. This represents the number of comment clauses that match the attribute characteristics of the g-th product with those of the m-th product.

[0088] Specifically, the formula for calculating the rate of change in attribute feature satisfaction can be:

[0089]

[0090] in, That is, the rate of change in customer satisfaction with attribute feature m between the (g+1)th generation product and the gth generation product; This refers to the satisfaction level of the m-th attribute feature of the g+1-th generation product.

[0091] Specifically, for any generation of product, the formula for calculating its attention level can be:

[0092]

[0093] in, This indicates the level of attention given to the m-th attribute feature of the g-th generation product. This represents the number of comment clauses that match the attribute characteristics of the g-th product with those of the m-th product. This represents the maximum value of the comment clauses in the g-th generation product that match the relevant product attribute characteristics.

[0094] Specifically, the formula for calculating the rate of change in attribute feature attention can be:

[0095]

[0096] in, This refers to the rate of change in customer attention regarding attribute feature m between the (g+1)th generation product and the gth generation product. This refers to the attention level of the m-th attribute feature of the g+1-th generation product.

[0097] Seen in Figure 3 In step S4, a classification model can be established and displayed with the change rate of attribute feature attention on the x-axis, the change rate of attribute feature satisfaction on the y-axis, and the importance of attribute features on the y-axis. Therefore, it is possible to better assess the dynamic changes in customer needs from three dimensions.

[0098] The method of the present invention enables:

[0099] 1. By mining customer needs in actual production, it enables manufacturers to fully consider customer attention, satisfaction, and the importance of product features to customers, providing suggestions for the design and investment of next-generation products.

[0100] 2. By constructing a feature set corresponding to the attribute features, it can fully take into account situations such as synonyms of product attribute features in the evaluation data, thus achieving higher accuracy;

[0101] 3. It can analyze customer needs based on text comments;

[0102] 4. By constructing a classification model, it is possible to achieve dynamic monitoring and display of relevant data.

[0103] Furthermore, based on the method of this embodiment, this embodiment also provides a three-dimensional customer demand dynamic mining system based on social media big data, which can have the following functions:

[0104] The acquisition unit is used to implement step S1;

[0105] A building unit, which is used to implement step S2;

[0106] Processing unit, which is used to implement step S3; and

[0107] The display unit is used to implement step S4.

[0108] It is readily understood that those skilled in the art can combine, split, or reorganize the embodiments provided in this application to obtain other embodiments, all of which do not exceed the protection scope of this application.

[0109] The present invention and its embodiments have been described above illustratively. This description is not restrictive, and the embodiments shown are only part of the embodiments of the present invention. The actual structure is not limited thereto. Therefore, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the present invention, they should all fall within the protection scope of the present invention.

Claims

1. A three-dimensional dynamic mining method for customer needs based on social media big data is used to analyze customer needs for target products. The method includes the following steps: S1, Data Preprocessing In this step, the first step is to obtain the review data of the target product, then the review words in the review data, and finally the word vectors of the review words. S2. Construct the feature dictionary of the target product In this step, the feature dictionary includes multiple feature sets, each corresponding to different attribute features of the target product. Each feature set has at least one feature word, which is selected from the review terms. This step specifically includes: S21. Extract original feature words from the self-comment words; S22. Cluster the original feature words, and each category after clustering is a feature set; S23. Expand the feature set by adding expanded feature words; S3. Obtain the evaluation tendency data for each attribute feature of the target product from the review data. In this step, the review data is divided into review clauses according to the product feature dictionary, and each review clause corresponds to one and only one related attribute feature; by analyzing the review clauses, evaluation tendency data is obtained. Specifically, the process involves dividing the comment data into comment clauses based on the product feature dictionary, using the comment words obtained in S1. Each relevant comment word is sorted according to the sentence order of its corresponding single comment. For each single comment, the similarity between each relevant comment word and each feature set is calculated. If the similarity is greater than a set feature set similarity threshold, the relevant single comment is segmented at the punctuation mark closest to the comment word to obtain the corresponding comment clause. The maximum or average similarity value between the relevant comment word and each feature word in each feature set is used as the similarity between each relevant comment word and each feature set. S4. Establish a demand classification model In this step, the established demand classification model includes the importance of attribute features, the rate of change in satisfaction with attribute features of different two generations of target products, and the rate of change in attention to attribute features of different two generations of target products.

2. The method for dynamic mining of three-dimensional customer needs based on social media big data according to claim 1, characterized in that: In step S21, comment words with a length not less than the set word length threshold, a part of speech of nouns, and a frequency of occurrence in the comment data not less than the set word frequency value are selected as original feature words.

3. The method for dynamic mining of three-dimensional customer needs based on social media big data according to claim 2, characterized in that: The cosine similarity between word vectors is used to measure the similarity between original feature words, and hierarchical clustering is used to cluster the original feature words.

4. The method for dynamic mining of three-dimensional customer needs based on social media big data according to claim 3, characterized in that: For any feature set, select relevant comment words from the comment words whose maximum similarity is not lower than the set text similarity threshold and whose part of speech is noun as expanded feature words.

5. The method for dynamic mining of three-dimensional customer needs based on social media big data according to claim 1, characterized in that: The importance of attribute features is obtained based on the following formula: ;in, That is, the importance of the m-th attribute feature. This represents the number of times the p-th feature word in the m-th feature set corresponding to the m-th attribute appears in the comment data. Let M be the total number of feature words in the m-th feature set, and M represent the total number of feature sets.

6. The method for dynamic mining of three-dimensional customer needs based on social media big data according to claim 1, characterized in that: The formula for calculating the rate of change in attribute feature satisfaction is as follows: ;in, , where is the rate of change in customer satisfaction between the g+1 generation product and the g generation product on attribute feature m; and That is, the satisfaction level of the m-th attribute feature of the g-th and g+1-th generation products; For any generation of products, the formula for calculating customer satisfaction is: ;in, This represents the satisfaction level of the m-th attribute feature of the g-th generation product. Indicates the first g The second generation of products and the first m The number of positive review clauses that match the product attribute characteristics. Indicates the first g The second generation of products and the first m The number of neutral comment clauses that match each product feature. Indicates the first g The second generation of products and the first m The number of comment clauses that match the product attribute characteristics.

7. The method for dynamic mining of three-dimensional customer needs based on social media big data according to claim 1, characterized in that: The formula for calculating the rate of change in attribute feature attention is as follows: ;in, This represents the rate of change in customer attention regarding attribute feature m between the (g+1)th generation product and the gth generation product. and This refers to the attention given to the m-th attribute feature of the g-th and g+1-th generation products; For any generation of product, the formula for calculating its attention is: ;in, This indicates the level of attention given to the m-th attribute feature of the g-th generation product. Indicates the first g The second generation of products and the first m The number of comment clauses that match each product attribute feature Indicates the first g The maximum value of comment clauses in the product that match the relevant product attribute features, where M represents the total number of feature sets.

8. A three-dimensional customer demand dynamic mining system based on social media big data, used to implement the method described in any one of claims 1-7, comprising: The acquisition unit is used to implement step S1; A building unit, which is used to implement step S2; A processing unit is used to implement step S3; as well as The display unit is used to implement step S4.

Images