A method for analyzing skin care product market trends based on big data social media

Abstract

The present invention discloses a skin care product market trend analysis method based on big data, which is characterized by comprising the following steps: S1: obtaining text information related to skin care products on social media; S2: preprocessing the text information; S3: extracting category information from the preprocessed text information and marking it; S4: classifying the preprocessed text information according to the marking; S5: integrating the classified text information to obtain new skin care product information; S6: performing trend analysis on the new skin care product information to obtain skin care product market trend analysis results. The present application analyzes the skin care product market trend by processing skin care product data on social media.

Description

Technical Field

[0001] The present invention relates to the field of big data skin care product analysis, and designs a skin care product market trend analysis method based on big data social media. Background Art

[0002] With the rapid development of the Internet and mobile Internet, people are increasingly inclined to buy skin care products online and share and discuss their purchase and use experiences on social media. Since these experiences can well reflect market demand and trends, using big data social media data to analyze skin care market trends has become an important topic for researchers and companies.

[0003] Nowadays, many users share skin care product trends on social networking service platforms such as Weibo, WeChat, Xiaohongshu, and Twitter every day. However, due to the complexity of these data types and quantities, how to effectively use these data for market trend analysis is a very challenging problem. Therefore, a reliable method is needed to process these data and extract relevant skin care product information to analyze the skin care product market trends. Summary of the invention

[0004] In view of the above-mentioned defects of the prior art, the present invention proposes a skin care product market trend analysis method based on big data. The technical solution steps designed by the present invention include:

[0005] S1: Obtain text information related to skin care products on social media;

[0006] S2: preprocessing the text information;

[0007] S3: extracting and marking category information from the preprocessed text information;

[0008] S4: classifying the preprocessed text information according to the identifier;

[0009] S5: integrating the classified text information to obtain new skin care product information;

[0010] S6: Perform trend analysis on the new skin care product information to obtain skin care product market trend analysis results.

[0011] Preferably, S1 includes: using the SpaCyNER tool to extract countries and regions from social media, using GeoNames as a data source, matching place names from multi-source global place name data, and expanding the scope of information acquisition; then using the Twitter data center official API, using "skin care" as a keyword, to obtain text information related to skin care products, wherein the text information includes but is not limited to Twitter text, number of likes, number of comments, number of reposts, purchase records, and product information.

[0012] Preferably, the preprocessing of S2 includes: cleaning and denoising, word splitting, stop word removal and word form restoration; the cleaning and denoising includes removing interference information and unnecessary information of the text information; the word splitting includes using the basic word segmentation function of SpaCy to identify common vocabulary in the text information after cleaning and denoising, and then splitting the words according to spaces; the stop word removal includes using SpaCy to remove stop words in the text information after word splitting, and the stop words include but are not limited to adverbs, articles and pronouns; the word form restoration includes restoring the words in the text information after the stop words are removed to the most basic words.

[0013] Preferably, S3 includes: vectorizing the preprocessed text information to convert it into vector data, then inputting the vector data into a BTM model to extract category information, and marking the category information of each piece of the vector data.

[0014] Preferably, the BTM model includes: the BTM model is implemented by bitern in Python language to extract category information from the vector data, biterm is an open source third-party Python tool, and the BTM model formula is:

[0015]

[0016] Preferably, the identification includes: 9 product category identifications, including skin care creams, lotions, oils, toners, body refreshers, bath products, eye skin care products, facial masks, and facial cleansers; 4 functional category identifications, including cleansing products, moisturizing products, whitening products, and anti-aging products.

[0017] Preferably, S4 includes: classifying the preprocessed text information according to the identification; using the identified text information as training samples, and then using the random forest algorithm training samples to classify the identified text information based on the "Scikit-learn" library in Python;

[0018] The training steps are as follows:

[0019] S41: Perform random sampling with replacement on the training set to obtain N samples as a new training set;

[0020] S42: extracting m features from the new training set to form a subset;

[0021] S43: training a decision tree using the subset;

[0022] S44: Then repeat the above steps S41-S43 until K decision trees are trained;

[0023] S45: Finally, the test samples to be classified are classified into each decision tree and the results are counted;

[0024] Among them, the training set contains N samples, M feature attributes, m≤M, and the result formula is:

[0025]

[0026] In the formula, H(x) is the model of the classifier, h i is a single decision tree, Y is the output variable, and I is the indicator function.

[0027] Preferably, S5 includes: integrating the classified text information to obtain new skin care product information; the classified text information includes commodity category information and function category information, and integrating the commodity category information and function category information, the formula is as follows:

[0028]

[0029] In the formula, a(m) is the product category information, b(m) is the function category information, E a and E b are the energies of a(m) and b(m), are constants, and ω ab is the relevant parameter for integration, and its value depends on Y ab To measure the correlation between a(m) and b(m);

[0030]

[0031] Wherein, N is the total number of data, n = 0, 1, 2...k;

[0032] Signal a i (m), a j (m) related energy E ij for:

[0033]

[0034] E i for a i(m) and b i (m) The sum of the energy associated with the data, as follows:

[0035]

[0036] Weight α i satisfy:

[0037] α 1 +α 2 +α 2 +α 3 +…+α m =1

[0038] α 1 : α 2 : α 3 :…:α m =E 1 :E 2 :E 3 ∶…∶e m

[0039] Get the final integrated new skin care product information A:

[0040] A=α 1 a 1 +α 2 a 2 +α 3 a 3 +…+α m a m .

[0041] Preferably, S6 includes: performing Spearman correlation analysis on the new skin care product information to obtain skin care product market trend analysis results; the Spearman correlation analysis is to register the correlation coefficient, a non-parametric indicator for measuring the dependence between two variables, and use a monotone equation to evaluate the correlation between two statistical variables, rank the two variables X and Y, and the ranks of the two variables are marked with p and q respectively, and then substituted into the following formula:

[0042]

[0043] Where p and q are the ranks of random variables X and Y, respectively. p and q is the standard deviation of random variables X and Y, and n is the sample size;

[0044] When the correlation coefficient |r| is between 0.8-1.0, it is an extremely strong relationship; between 0.6-0.8, it is a strong relationship; between 0.4-0.6, it is a moderate relationship; between 0.2-0.4, it is a weak relationship; and between 0.0-0.2, it is an extremely weak relationship.

[0045] Preferably, the skin care product market trend analysis results are converted into graphics or images through a data visualization library in Python and presented on the screen, and then interactively processed and visualized.

[0046] Beneficial effects:

[0047] 1. The present invention uses the SpaCyNER tool to extract information from social media, which can analyze the differences in skin care product reviews between different countries, expand the scope of market analysis, and meet the needs of different users;

[0048] 2. The integration method used in the present invention utilizes the optimal method to fully integrate complementary redundant information in different dimensions, utilizes the functional category characteristics of different commodity categories, improves the accuracy of the integrated results to a certain extent, and can also make full use of massive data and simplify the data volume;

[0049] 3. The present invention finally performs Spearman correlation analysis on the data and performs visual expression. The corresponding result data under any combination of commodities and functions can be quickly predicted, and information can be obtained simply and conveniently, which is easy to operate. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a flow chart of a preferred embodiment of the present invention;

[0051] Figure 2 It is a schematic diagram of a BTM model algorithm of a preferred embodiment of the present invention;

[0052] Figure 3 It is a training flow diagram of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0053] The embodiments of the present invention are described in detail below. The following embodiments are implemented on the premise of the technical solution of the present invention, and detailed implementation methods and specific operation processes are given, but the protection scope of the present invention is not limited to the following embodiments.

[0054] The present invention designs a technical solution comprising the following steps: Figure 1 As shown, specifically including:

[0055] S1: Obtain text information related to skin care products on social media;

[0056] S2: preprocessing text information;

[0057] S3: extracting and marking category information from the preprocessed text information;

[0058] S4: classify the preprocessed text information according to the identifier;

[0059] S5: integrating the classified text information to obtain new skin care product information;

[0060] S6: Perform trend analysis on new skin care product information to obtain skin care product market trend analysis results.

[0061] Specifically, S1 includes: obtaining text information related to skin care product reviews on social media (Weibo, WeChat, FourSquare, Twitter, etc.), and completing data cleaning preprocessing of punctuation marks, numbers, URLs, etc. of the text information based on natural language processing technology. (The SpaCyNER tool is used to extract countries and regions from social media, and GeoNames is used as the data source to match place names from multi-source global place name data to obtain data from multiple source regions. It can also analyze the differences in skin care product reviews between different countries, expand the scope of market analysis, and meet the needs of different users.) The present invention uses the official API of the Twitter data center and uses "skin care" as the keyword to obtain Twitter data. The data mainly includes Twitter text, likes, comments, reposts, purchase records, product information, etc.

[0062] Specifically, S2 includes: cleaning and denoising. The collected Twitter text contains punctuation, HTML tags, hashtags, addresses and other interfering or unnecessary information. In order to eliminate these useless information, the efficiency and accuracy of subsequent information extraction are improved; including word splitting, using SpaCy's basic word segmentation function to identify common words in the text, and then splitting words according to spaces to achieve the result of self-segmentation; removing stop words (stop words have no specific meaning in the text, including adverbs, articles, pronouns, etc., and removing stop words can reduce storage space). Use SpaCy to quickly and effectively delete stop words from the text; word form restoration, restore English words to the most basic words.

[0063] Specifically, S3 and S4 include: constructing a skin care product subject extraction and classification framework based on the subject model and machine learning methods to achieve information extraction, using the BTM subject model combined with the random forest algorithm, and dividing the information distribution into 9 product categories and 4 functional categories to extract and classify useful information. Skin care creams, lotions, oils, toners, body toners, baths, eye skin care, masks, facial washes; cleansing, moisturizing, whitening, and anti-aging categories.

[0064] Information extraction: Figure 2As shown, based on the BTM model, category extraction is performed to generate category identifiers for each data; some data is selected from the processed data as sample topics for generating categories, and the topics correspond to 13 categories one by one; the btm.exe file is generated, and the pre-processed data is vectorized and converted into vector data; then the vector data is input into the BTM model to generate the category identifier for each Twitter text. Specifically, the BTM model implements Twitter topic extraction for biterm in the Python language. Biterm is an open source third-party Python tool. The final BTM extraction model formula is:

[0065]

[0066] Information classification: The marked data is finally classified based on the random forest algorithm; the marked data is used as training samples, and the random forest algorithm is used to train the samples, and the marked data is finally classified based on the "Scikit-learn" library in Python. Specifically, the training set contains N samples and M feature attributes. The training set is randomly sampled with replacement to obtain N samples as a new training set; then m features are extracted from the new training set to form a subset (m≤M), and a decision tree is trained using the subset (the decision tree is not pruned, and it will grow to the maximum extent); then the above steps are repeated until K decision trees are trained; finally, the test samples to be classified are classified for each decision tree and the results are counted (the category recognized by the most decision trees will be the final classification result), and the result formula is as follows:

[0067]

[0068] In the formula, H(x) is the model of the classifier, h i is a single decision tree, Y is the output variable, and I is the indicator function;

[0069] Preferably, if Figure 3 As shown, the training steps are as follows:

[0070] S41: Perform random sampling with replacement on the training set to obtain N samples as a new training set;

[0071] S42: extract m features from the new training set to form a subset;

[0072] S43: Train a decision tree using the subset;

[0073] S44: Then repeat the above steps S41-S43 until K decision trees are trained;

[0074] S45: Finally, the test samples to be classified are classified into each decision tree and the results are counted;

[0075] Specifically, S5 includes: integrating the product category and function category information into a method for combining local information by computer to achieve the purpose of systematically reflecting the integrity of information. Because the information extracted in this application is single, the product category information or function category information alone cannot fully express the whole, so the classified text information is integrated to obtain new skin care product information; the classified text information includes product category information and function category information, and the product category information and function category information are integrated. The formula is as follows:

[0076]

[0077] In the formula, a(m) is the product category information, b(m) is the function category information, E a and E b are the energies of a(m) and b(m), are constants, and ω ab is the relevant parameter for integration, and its value depends on Y ab To measure the correlation between a(m) and b(m);

[0078]

[0079] Wherein, N is the total number of data, n = 0, 1, 2...k;

[0080] Signal a i (m), a j (m) related energy E ij for:

[0081]

[0082] E i for a i (m) and b i (m) The sum of the energy associated with the data, as follows:

[0083]

[0084] Weight α i satisfy:

[0085] α 1 +α 2 +α 2 +α 3 +…+α m =1

[0086] α 1 : α 2 : α 3 :…:α m =E 1 :E 2:E 3 ∶…∶E m

[0087] Get the final integrated new skin care product information A:

[0088] A=α 1 a 1 +α 2 a 2 +α 3 a 3 +…+α m a m .

[0089] Preferably, S6 includes: performing Spearman correlation analysis on the new skin care product information to obtain the skin care product market trend analysis results; Spearman correlation analysis is to register the correlation coefficient, a non-parametric indicator for measuring the dependence between two variables, and use a monotone equation to evaluate the correlation between two statistical variables, rank the two variables X and Y, and the ranks of the two variables are marked with p and q respectively, and then substituted into the following formula:

[0090]

[0091] Where p and q are the ranks of random variables X and Y, respectively. p and q is the standard deviation of random variables X and Y, and n is the sample size;

[0092] When the correlation coefficient |r| is between 0.8-1.0, it is an extremely strong relationship; between 0.6-0.8, it is a strong relationship; between 0.4-0.6, it is a moderate relationship; between 0.2-0.4, it is a weak relationship; and between 0.0-0.2, it is an extremely weak relationship.

[0093] Specifically, Spearman correlation analysis is performed on Twitter text data with labels; data visualization libraries in Python (which convert data into graphics or images and present them on the screen, and then perform interactive processing to visualize the results of trend analysis), such as matplotlib and seaborn, are used to visualize the analysis results.

[0094] The preferred specific embodiments of the present invention are described in detail above. It should be understood that ordinary technicians in the field can make many modifications and changes based on the concept of the present invention without creative work. Therefore, all technical solutions that can be obtained by technicians in the technical field based on the concept of the present invention through logical analysis, reasoning or limited experiments on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A skin care product market trend analysis method based on big data, characterized in that: The following steps are involved: S1: Obtain text information related to skin care products on social media; S2: preprocessing the text information; S3: extracting and marking category information from the preprocessed text information; S4: classifying the preprocessed text information according to the identifier; S5: integrating the classified text information to obtain new skin care product information; S6: Performing trend analysis on the new skin care product information to obtain skin care product market trend analysis results; The S3 includes: Performing vectorization processing on the preprocessed text information to convert it into vector data, then inputting the vector data into a BTM model to extract category information, and marking the category information of each piece of the vector data; The BTM model includes: The BTM model is implemented by bitern in Python language to extract category information from the vector data. Biterm is an open source third-party Python tool. The BTM model formula is: In the formula, P(z|z -b , B, α, β) is the conditional probability of topic z, α and β are hyperparameters, z -b is other information besides the current sample, B is the vocabulary set, n(w i |z) is the vocabulary w under topic z i The frequency of occurrence, n(w j |z) is the vocabulary w under topic z j The frequency of occurrence, ∑ w n(w|z) is the total frequency of all words under topic z, and M is the total number of words in the vocabulary set; The S5 includes: The classified text information is integrated to obtain new skin care product information; the classified text information includes commodity category information and function category information, and the commodity category information and function category information are integrated, and the formula is as follows: In the formula, a(m) is the product category information, b(m) is the function category information, E a and E b are the energies of a(m) and b(m), are constants, and ω ab is the relevant parameter for integration, and its value depends on Y ab To measure the correlation between a(m) and b(m); Wherein, N is the total number of data, n = 0, 1, 2...k; Signal a i (m), a j (m) related energy E ij for: E i for a i (m) and b i (m) The sum of the energy associated with the data, as follows: Weight α i satisfy: α1+α2+α2+α3+…+α m ＝1 a1: a2: a3:…: a m =E1:E2:E3:…:E m Get the final integrated new skin care product information A: A=α1a1+α2a2+α3a3+…+α m a m ; The S6 includes: The new skin care product information is subjected to Spearman correlation analysis to obtain the skin care product market trend analysis results; the Spearman correlation analysis is to register the correlation coefficient, a non-parametric indicator for measuring the dependence between two variables, and to evaluate the correlation between two statistical variables using a monotone equation. The two variables X and Y are ranked, and the ranks of the two variables are marked with p and q respectively, and then substituted into the following formula: Where p and q are the ranks of random variables X and Y, respectively. p and q is the standard deviation of random variables X and Y, and n is the sample size; When the correlation coefficient |r| is between 0.8-1.0, it is an extremely strong relationship; between 0.6-0.8, it is a strong relationship; between 0.4-0.6, it is a moderate relationship; between 0.2-0.4, it is a weak relationship; and between 0.0-0.2, it is an extremely weak relationship; The skin care product market trend analysis results are converted into graphics or images through the data visualization library in Python and presented on the screen, and then interactively processed and visualized.

2. The method for analyzing the market trend of skin care products based on big data according to claim 1, characterized in that: The S1 includes: The SpaCyNER tool was used to extract countries and regions from social media. GeoNames was used as the data source to match place names from multi-source global place name data to expand the scope of information acquisition. Then, through the official API of social media, "skin care" was used as the keyword to obtain text information related to skin care products, including but not limited to text, number of likes, number of comments, number of reposts, purchase records and product information.

3. The method for analyzing skin care product market trends based on big data according to claim 1, characterized in that: The preprocessing of S2 includes: Cleaning and denoising, word splitting, stop word removal and word form restoration; the cleaning and denoising includes removing interference information and unnecessary information from the text information; the word splitting includes using the basic word segmentation function of SpaCy to identify common vocabulary in the text information after cleaning and denoising, and then splitting the words according to spaces; the stop word removal includes using SpaCy to remove stop words in the text information after word splitting, and the stop words include but are not limited to adverbs, articles and pronouns; the word form restoration includes restoring the words in the text information after the stop words are removed to the most basic words.

4. The method for analyzing skin care product market trends based on big data according to claim 1, characterized in that: The identification includes: 9 product category labels, including skin care creams, lotions, oils, toners, body refreshers, bath products, eye skin care products, facial masks, and facial cleansers; 4 functional category labels, including cleansing products, moisturizing products, whitening products, and anti-aging products.

5. The method for analyzing skin care product market trends based on big data according to claim 1, characterized in that: The S4 includes: Classifying the preprocessed text information according to the identification; using the identified text information as training samples, and then using the random forest algorithm training samples to classify the identified text information based on the "Scikit-learn" library in Python; The training steps are as follows: S41: Perform random sampling with replacement on the training set to obtain N samples as a new training set; S42: extracting m features from the new training set to form a subset; S43: training a decision tree using the subset; S44: Then repeat the above steps S41-S43 until K decision trees are trained; S45: Finally, the test samples to be classified are classified into each decision tree and the results are counted; Among them, the training set contains N samples, M feature attributes, m≤M, and the result formula is: In the formula, H(x) is the model of the classifier, h i is a single decision tree, Y is the output variable, and I is the indicator function.

Images