Consumer policy perception analysis method and system based on big data text mining

Abstract

The present invention discloses a method and system for mining consumer policy perception based on big data text mining, including: collecting policy text data in specific fields and consumer comment text data on relevant government websites and social platforms; preprocessing the collected text data, including deduplication, noise reduction, and short sentence processing; in-depth mining of professional vocabulary and construction of policy corpus based on machine recognition and expert knowledge corpus; mining and analyzing consumer policy perception based on consumer comments and in combination with policy corpus. The present invention makes full use of online public opinion information, mines consumer comment texts from the perspective of consumer policy perception, and establishes an expert knowledge corpus for mining consumer policy perception, thereby greatly improving the text segmentation effect, and combining natural language processing and other technologies to more systematically and accurately mine and analyze consumers' perception of policies, and provide suggestions for optimizing policy supply and improving policy accuracy and effectiveness.

Description

Technical Field

[0001] The present invention relates to the field of data mining technology, and in particular to consumer policy perception mining based on big data text mining, and specifically to a consumer policy perception analysis method and system based on big data text mining. Background Art

[0002] Studying consumers’ perception of policies has great practical significance. To study consumers’ policy perception is to study consumers’ attention content, attention degree, evaluation of policy rationality and necessity, and emotional attitudes.

[0003] With the popularization of Internet technology, more and more consumers are posting their comments online, forming a large amount of consumer-generated content (UGC). These large amounts of text data generated on social media platforms largely reflect the true thoughts of consumers. Analyzing them can help us understand consumers' concerns and emotional attitudes. At the same time, the growing policy text data on government websites, as an open and accessible information resource, also contains a lot of information. In-depth mining and analysis of them is an important way to trace policy intentions and grasp the attitudes of decision makers. In terms of policy research, traditional content analysis methods mainly rely on experts and scholars to study the content of policy texts, which is labor-intensive, narrow in coverage, and inefficient. With the continuous development of big data technology, quantitative analysis methods have occupied a place in policy text mining and analysis research.

[0004] Regarding the study of consumers' policy perception, although relevant researchers have also proposed many solutions in policy mining. However, the existing policy mining research is relatively small in number at the "government-consumer" level, and rarely involves solutions for consumer policy perception mining, and there is room for improvement in terms of completeness. For example, the Chinese patent with application number 202011260570.6 disclosed on February 19, 2021, a corporate policy matching method based on label similarity. This solution completes the matching of enterprises and policies by constructing corporate labels and policy labels and calculating the similarity between the two. This solution matches the supply side and the demand side of policies, but the research is based on the "government-enterprise" level, ignoring the optimization role of consumer public opinion in policy formulation and implementation, and there is room for improvement.

[0005] In the existing solutions for mining policy perception based on text mining technology, few researchers have used online public opinion data, such as a matching recommendation method and system based on specific urban populations and related policies disclosed on October 9, 2017 by Chinese patent application number 201710934706.9. This solution clusters specific populations, extracts labels for each classification demand, analyzes relevant policies adopted for specific populations, matches populations with policies, and recommends policies to designated consumers. However, the information source of this solution mainly comes from the information system data of various commissions and bureaus and questionnaire survey data for specific populations. It does not make full use of online public opinion information and ignores the perceived value contained in consumer public opinion. The questionnaire survey method used has the problems of unguaranteed recovery rate and efficiency, insufficient population coverage, and the authenticity of the answer results is difficult to guarantee.

[0006] In addition, the word segmentation tools used in existing solutions lack a policy corpus combined with an expert knowledge corpus to improve the identification of policy-related vocabulary in consumer comments, and the accuracy of policy text segmentation needs to be improved. For example, the Chinese patent with application number 202110204049.9 disclosed on May 14, 2021 a government big data analysis method and device for government decision-making. This solution constructs a multidimensional data mining model based on the hierarchical division indicators and classification summary indicators related to government data, and performs multidimensional mining and analysis on government data. However, since policy texts have different language styles and characteristics from ordinary texts, the expression of policy texts is strictly standardized and has specific official document terms, while the policy-related vocabulary in consumer comments is relatively rich. Existing solutions do not construct a policy corpus for this, and the accuracy of word segmentation results needs to be improved.

[0007] It can be seen that the existing policy perception research methods have the following problems:

[0008] Consumer perception data mostly comes from questionnaire surveys, the response rate and efficiency of which cannot be guaranteed, the coverage of the population is not wide enough, and the authenticity of the answers is difficult to guarantee;

[0009] Existing text mining-based research often ignores the perceived value contained in consumer reviews;

[0010] Existing word segmentation tools lack a policy corpus for identifying consumer reviews based on an expert knowledge corpus. The accuracy of policy text segmentation needs to be improved, and manual annotation methods are generally used. Therefore, building a policy corpus requires a lot of manpower. Summary of the invention

[0011] In view of the problems existing in the prior art, the present invention provides a consumer policy perception analysis method and system based on big data text mining, which is used to obtain more accurate text mining analysis results and policy perception differences among different consumer groups.

[0012] According to one aspect of the present invention, a method for analyzing consumer policy perception based on big data text mining is provided, comprising:

[0013] Obtain policy text data and consumer review text data and perform preprocessing;

[0014] Build a policy corpus based on the preprocessed text data;

[0015] Analyzing consumer policy perception based on policy corpus.

[0016] The above technical solution makes full use of network information, mines consumer comment texts from the perspective of consumer perception, and establishes a policy corpus to improve the text segmentation effect. It combines deep learning, expert knowledge corpus and other technologies to more systematically and accurately mine and analyze consumers' perception of policies.

[0017] The above technical solution can conduct a comparative analysis of the differences in policy perceptions among different categories of consumer groups, thereby providing suggestions for optimizing policy supply and improving policy accuracy and effectiveness.

[0018] As a further technical solution, the steps of obtaining policy text data and consumer review data further include:

[0019] Identify the data source;

[0020] Use web crawler tools to collect policy text data and corresponding consumer review text data;

[0021] Randomly sample the collected data sources and match and verify them with the crawled data;

[0022] If the crawled data passes the verification, the corresponding data will be stored locally persistently.

[0023] Specifically, the data source may be relevant government websites and social networking platforms.

[0024] When using web crawler tools for collection, you can customize the collection rules to obtain the required policy text data and corresponding consumer review text data.

[0025] As a further technical solution, preprocessing further includes deduplication, denoising and deletion of short sentences.

[0026] Deduplication means deleting duplicate data in text data to reduce the interference of redundant information.

[0027] Denoising is the process of deleting special characters such as emoticons and URLs in text data. These characters lack practical meaning and research value, and will affect subsequent word segmentation and other text mining analysis results. Therefore, denoising is required to make the text features more focused on vocabulary and semantics.

[0028] Text short sentence deletion refers to deleting data with too few words in the text data. From an analytical perspective, fewer words means that the comment information contains less information value, and it is very likely that it is randomly generated by consumers or the platform's default evaluation, which has no feedback significance and research value.

[0029] As a further technical solution, constructing a policy text corpus based on the preprocessed text data further includes:

[0030] Perform word splitting on the pre-processed policy text and consumer review text data, and count the word frequencies of all generated words;

[0031] Screening of mutual information between points based on word frequency statistics;

[0032] The left and right information entropy are screened based on the mutual information screening results between points;

[0033] Based on the left and right information entropy screening results, stop words and common general words are removed to obtain policy professional vocabulary;

[0034] The obtained policy professional vocabulary is screened by the expert knowledge corpus, and the vocabulary related to policy perception in consumer comments is improved and identified. The screened and improved vocabulary is divided into four categories according to policy classification to form the final policy text corpus.

[0035] The above technical solution first screens based on point-to-point mutual information and left-right information entropy, then removes stop words and common general vocabulary, mines professional vocabulary in policy texts and policy-related vocabulary in consumer comment texts, and constructs a policy corpus; then, based on machine recognition, it combines expert knowledge corpus to screen and supplement the policy corpus to obtain the final policy corpus.

[0036] The specific method of point-to-point mutual information (PMI) screening is as follows: according to the results of word frequency, the solidification degree of each word fragment is calculated (for the binary word string AB, its solidification degree is PMI(A,B); for the ternary word string ABC, its solidification degree is min[PMI(A,BC),PMI(AB,C)] and so on.), different thresholds are set for word fragments of different lengths, and screening is performed according to the threshold to obtain a set G containing fragments with higher solidification degrees.

[0037] The left-right information entropy screening is specifically as follows: all possible left-right neighboring words of each segment in the set G in the text are counted, the left-right information entropy of each segment is calculated, and the word segments are sorted according to the left-right information entropy to obtain the set F.

[0038] Remove stop words and some common general words from the words in set F to obtain policy professional vocabulary.

[0039] The professional vocabulary obtained through machine recognition will be screened and supplemented by the expert knowledge corpus, meaningless words will be eliminated, and the words related to policy perception in consumer comments will be identified. The words will be classified into four types according to policy: supply-driven, demand-driven, environmental regulation, and environmental support, to form the final policy corpus.

[0040] As a further technical solution, performing consumer policy perception mining analysis further includes:

[0041] Categorize consumers into different categories;

[0042] Mining review data from different categories of consumers, including word frequency intensity analysis, topic identification, semantic network analysis, and sentiment tendency analysis;

[0043] By comparing the mining and analysis results of different categories of consumers, we can obtain the differences in the perceptions of different groups towards different categories of policies.

[0044] The above technical solution conducts text mining on consumer comments based on the policy corpus, divides consumers into different categories according to their portraits and regions, mines comment data among consumers in different categories, compares the results of word frequency intensity, topic identification, semantic network analysis, and sentiment tendency analysis, and obtains the differences in the perceptions of different consumer groups on different categories of policies.

[0045] As a further technical solution, the word frequency intensity analysis further includes: importing the constructed policy corpus and related field cell vocabulary into the custom dictionary, using jieba to segment the consumer review text, and counting the word frequency intensity; dividing the word frequency results according to policy categories, and generating corresponding word cloud diagrams based on the word frequency intensity.

[0046] As a further technical solution, topic identification further includes:

[0047] Store the segmented consumer review text in a list format for each document;

[0048] Convert the text into a sparse vector expression corresponding to the bag-of-words model and construct a word frequency matrix;

[0049] Calculate the perplexity of models with different numbers of topics and determine the optimal number of model topics;

[0050] The optimal number of topics evaluated by the perplexity method is used as a parameter to train the LDA model and analyze the topics of concern in consumer reviews.

[0051] As a further technical solution, semantic network analysis further includes:

[0052] According to the policy corpus, select the comments related to policies from the consumer comments;

[0053] Count the number of times high-frequency words in the review text appear in pairs, and find out the closeness between these words;

[0054] Count the frequencies of co-occurring word pairs and construct a high-frequency word co-occurrence matrix based on the results;

[0055] Based on the high-frequency word co-occurrence matrix, NetDraw is used to draw a semantic network diagram to obtain the hierarchy and close relationship between the nodes in the semantic network diagram, so as to understand consumers' perception of the policy.

[0056] As a further technical solution, sentiment analysis further includes:

[0057] Extract policy-related fragments from consumer reviews based on the policy corpus, segment each review document, and find out the sentiment words, negative words, and degree adverbs in the document;

[0058] Determine whether there is a negation word or degree adverb before each sentiment word. If there is a negation word or degree adverb, divide the sentiment word and the previous negation word or degree adverb into a group, and multiply the sentiment score of the sentiment word by the weight coefficient corresponding to the group; the weight coefficient of the negation word is set to -1, and the degree adverb has different weight coefficients according to the semantics;

[0059] Determine whether each sentence in the comment document is an exclamatory sentence or a rhetorical question. If so, increase the sentiment score.

[0060] Add up all the sentiment scores in the comment to get the final sentiment analysis score of the comment document.

[0061] According to another aspect of the present invention specification, there is provided a consumer policy perception analysis system based on big data text mining, comprising: a data acquisition module for acquiring policy text data and consumer comment text data; a data preprocessing module for preprocessing the acquired text data; a corpus construction module for constructing a policy corpus based on the preprocessed text data; and a policy perception mining module for conducting consumer policy perception analysis based on the policy corpus.

[0062] Compared with the prior art, the beneficial effects of the present invention include:

[0063] (1) The present invention constructs a policy corpus based on the method of machine recognition of professional vocabulary and manual semantic screening of expert knowledge corpus. The constructed policy corpus can be used as a word segmentation dictionary to improve the text segmentation effect. Then, based on text mining, the word frequency intensity, topic identification, semantic network, and sentiment analysis results of consumer comments are compared. On the basis of existing research, dimensions such as consumer portraits and regions are added to pay attention to the differences in policy perceptions among different categories of consumer groups, thereby providing suggestions for optimizing policy supply and improving policy accuracy and benefit nature.

[0064] (2) The present invention adopts natural language processing technology, uses machines to identify professional vocabulary in the policy field, constructs a policy corpus, improves information screening efficiency, and saves a lot of manpower.

[0065] (3) The policy corpus constructed by the present invention serves as a word segmentation dictionary, which improves the text segmentation effect and makes the subsequent text mining analysis results more accurate.

[0066] (4) The present invention is based on big data text mining, uses multi-source data for mining and analysis, makes full use of network information, and compares the policy perception differences of specific consumer groups through analysis of consumer reviews of different categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is a flowchart of a method for analyzing consumer policy perception based on big data text mining according to a first embodiment of the present invention;

[0068] Figure 2 This is a schematic diagram of the process of data collection in step S1 in the first embodiment of the present invention;

[0069] Figure 3 A schematic diagram of the process of constructing a policy corpus in step S3 of the first embodiment of the present invention;

[0070] Figure 4 This is a flow chart of consumer policy perception mining and analysis in step S4 in the first embodiment of the present invention;

[0071] Figure 5 This is a flowchart of topic identification in sub-step S43 of step S4 in the first embodiment of the present invention;

[0072] Figure 6 This is a schematic diagram of the process of semantic network analysis of sub-step S44 in step S4 in the first embodiment of the present invention;

[0073] Figure 7 Schematic diagram of the process of sentiment tendency analysis of sub-step S45 in step S4 in the first embodiment of the present invention. DETAILED DESCRIPTION

[0074] The following will clearly and completely describe the technical solutions of various embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0075] like Figure 1 As shown, this embodiment provides a new energy vehicle policy perception analysis method based on consumer review text mining, taking the analysis of new energy vehicle policy as an example, including the following steps:

[0076] S1, collect policy text data and consumer comment data in specific areas from relevant government websites and social platforms.

[0077] S2, preprocess the collected policy text and consumer review text data.

[0078] S3, constructs a policy corpus based on the combination of machine recognition and expert knowledge corpus.

[0079] S4, Mining and analysis of new energy vehicle consumers’ policy perception.

[0080] Specifically, step S1 is as follows: Figure 2 As shown, specifically including:

[0081] S11, determine the data source. In this embodiment, the data source is China Industrial Information Network, the Ministry of Industry and Information Technology and other relevant government official websites, legal and regulatory databases, and new energy vehicle vertical portals such as iCar.com and Autohome.com.

[0082] S12, use Python to write a web crawler program or use Houyi Collector, Octopus and other collection tools, customize the collection rules, and collect policy text data and corresponding consumer review text data.

[0083] S13, randomly sample the collection sources and match and verify them with the crawled data to eliminate incomplete or erroneous data caused by errors in the crawling process.

[0084] S14, after ensuring that the crawled data passes the verification, the collected data is stored locally and persistently.

[0085] In this embodiment, after the data collection in step S1 is completed, the data preprocessing in step S2 should be performed. The main purpose of data preprocessing is to remove useless information in the text. Specifically, it is mainly divided into three aspects of operations, namely, deduplication, denoising, and text short sentence deletion.

[0086] S21, deduplication, is to delete duplicate data in text data to reduce the interference of redundant information.

[0087] S22, denoising, deletes some special characters such as emoticons and URLs in the text data. These contents lack practical meaning and research value, and will affect the subsequent word segmentation and other text mining analysis results. Therefore, denoising operations are required to make the text features more focused on vocabulary and semantics themselves.

[0088] S23, text short sentence deletion, refers to deleting data with too few words in the text data. From an analytical perspective, fewer words means that the comment information contains less information value, and it is very likely that it is randomly generated by consumers or the platform's default evaluation, which has no feedback significance and research value.

[0089] The process of step S3 of constructing the policy corpus in this embodiment is as follows: Figure 3 As shown in the figure, firstly, based on the mutual information between points and the left and right information entropy, we filter, then remove stop words and common general words, and mine the policy-related words in the policy texts and consumer comment texts to build the policy corpus. Based on machine recognition, we combine the expert knowledge corpus to filter and supplement the policy corpus to obtain the final policy corpus. The details are as follows:

[0090] S31, using N-gram to split the incoming text into words, and count the frequency of all generated words.

[0091] S32, pointwise mutual information (PMI) screening. Pointwise mutual information (PMI) can be used to measure the correlation between two variables, and here it can be used to reflect the closeness between adjacent characters or words. Its calculation formula is as follows:

[0092]

[0093] Among them: x, y are adjacent character strings; xy is a word composed of x and y; P(x), P(y), P(xy) are the probabilities of x, y, and xy appearing in the full text of the corpus. The larger the PMI, the higher the co-occurrence frequency of adjacent character strings, that is, the more likely x and y are to form a fixed word.

[0094] The solidification degree of each word segment is calculated based on the word frequency (for the binary word string AB, its solidification degree is PMI(A,B); for the trigram word string ABC, its solidification degree is min[PMI(A,BC),PMI(AB,C)], and so on.) Different thresholds are set for word segments of different lengths, and the segments are screened based on the thresholds. At this time, a set G containing segments with higher solidification degrees is obtained.

[0095] S33, left and right information entropy screening. Left and right information entropy is used to determine whether a word string has rich left and right collocations. The calculation formulas are as follows:

[0096]

[0097]

[0098] Among them, H l (x) and H r (x) is the left and right information entropy of word string x; s l With s r is the left and right neighboring word set of word string x; P(w l x|x) is the word string x when its left neighbor is w l The conditional probability of r x|x) is the word string x, when it appears, its right neighbor is w r The conditional probability of .

[0099] The higher the left and right information entropy of a word string, the more uncertain its left and right adjacent characters are, indicating that the word string is more likely to be an independent word. After the word string is obtained by filtering the inter-point mutual information, the boundary information entropy is calculated to determine whether the word string is an independent word.

[0100] In this step, it is necessary to count all possible left and right neighboring words of each segment in the set G in the text, calculate the left and right information entropy of each segment, sort the word segments according to the left and right information entropy, and obtain the set F.

[0101] S34, remove stop words and some common general words from the words in set F to obtain policy professional vocabulary.

[0102] S35, the professional vocabulary obtained through machine recognition will be screened and supplemented by the expert knowledge corpus, meaningless words will be eliminated, and the recognition of words related to policy perception in consumer comments will be improved. The words will be classified into four types according to policy: supply-driven, demand-driven, environmental regulation, and environmental support, to form the final policy corpus.

[0103] (4) Consumer policy perception mining, the process is as follows Figure 4 As shown in the figure. Based on the policy corpus, we conduct text mining on consumer comments, divide consumers into different categories according to their portraits and regions, mine comment data from consumers in different categories, compare word frequency intensity, topic identification results, semantic network, and sentiment analysis results, and obtain the differences in perceptions of different groups on different categories of policies.

[0104] S41, Consumer Classification

[0105] The automobile word-of-mouth information collected by iCar.com includes consumer nicknames, car models, car purchase locations, car purchase purposes, bare car prices, membership levels, and comments on various dimensions of automobiles. Consumer information is extracted from it, and consumer portraits are constructed based on membership levels, spending power (bare car prices), car models, and car purchase purposes. Consumers are classified according to consumer portraits and their regions. Consumer comment data is then filtered and classified according to consumer categories, so that further comparative analysis can be conducted on the perception differences of different categories of consumers.

[0106] S42, word frequency intensity analysis

[0107] Jieba is used to segment the text, and the constructed policy corpus and related field cell word library are imported into the custom dictionary for segmentation, and then the word frequency intensity is counted. The word frequency results are divided according to policy classification, and the corresponding word cloud chart can be generated according to the word frequency intensity, so as to more intuitively compare the differences in consumers' concerns.

[0108] S43, Topic Identification

[0109] The LDA (Latent Dirichlet Allocation) model is used to extract the topics of consumer reviews and analyze the topics of concern in consumer reviews. Figure 5 As shown, the steps include:

[0110] S431, storing the segmented text in a format of one list per document.

[0111] S432, convert the text into a sparse vector expression corresponding to the bag-of-words model, and construct a word frequency matrix.

[0112] S433, determining the optimal number of model topics by evaluating the perplexity of models with different numbers of topics.

[0113] S434, using the optimal number of topics obtained through perplexity as a parameter, training the LDA model, and analyzing to obtain the topics of concern in consumer comments.

[0114] The text storage in sub-step S431 must meet the requirements of the gensim library. (The input format required by gensim is: ['new energy vehicles', 'policy', 'subsidy', ...], each document is a list, and the elements are words).

[0115] The method for determining the number of topics of the LDA model by the perplexity method in sub-step S433 is specifically as follows:

[0116] a, calculate the probability of word occurrence, the formula is as follows:

[0117] P(w)=P(L|M)P(w|L)

[0118] b. Calculate the perplexity under different model topic numbers according to the perplexity calculation formula:

[0119]

[0120] Among them, N represents the total number of words in the text, P(w) represents the probability of word w appearing, P(L|M) represents the topic probability of topic word L in text M, and P(w|L) represents the word probability of word w in topic L. The lower the perplexity value, the smaller the uncertainty, and the better the final clustering result.

[0121] c. Draw a line graph of topic number-perplexity based on the perplexity under different model topic numbers. According to the elbow method, the inflection point is the optimal model topic number.

[0122] S44, semantic network analysis

[0123] By conducting semantic network analysis on consumer comment texts, we can understand consumers’ overall perception of new energy vehicle policies. The process is as follows: Figure 6 As shown, the following steps are included:

[0124] S441, based on the policy corpus, filter out the comment texts that mention policy-related issues from the consumer comment texts.

[0125] S442, counting the number of times high-frequency words in the text appear in pairs, and obtaining the closeness between these words.

[0126] S443, counting the frequencies of co-occurring word pairs, and constructing a high-frequency word co-occurrence matrix based on the results.

[0127] S444, based on the high-frequency word co-occurrence matrix, use NetDraw to draw a semantic network diagram, and obtain the hierarchy and close relationship between the nodes in the semantic network diagram to understand consumers' perception of the policy.

[0128] S45, Sentiment Analysis

[0129] Based on the policy corpus, we extract policy-related fragments from consumer reviews and perform sentiment analysis on them. This paper analyzes text sentiment based on sentiment dictionaries. The process is as follows: Figure 7 As shown, the following steps are included:

[0130] S451, before doing sentiment analysis, you need to segment the document based on the policy corpus to find the sentiment words, negative words and degree adverbs in the document.

[0131] S452, degree word and negative word processing. Determine whether there is a negative word or degree adverb before each sentiment word. If there is a negative word or degree adverb, divide the sentiment word and the previous negative word or degree adverb into a group, and multiply the sentiment score of the sentiment word by the weight coefficient corresponding to the group. The weight coefficient of the negative word is set to -1; different weight coefficients are set for degree adverbs according to semantics. Common degree adverbs such as "very", "especially", "a little", "not too", etc., are assigned different weight coefficients.

[0132] S453, special sentence processing. Determine whether it is an exclamatory sentence or a rhetorical question. Because exclamatory sentences and rhetorical questions will strengthen the tone, they need special processing. Here, you only need to determine whether the punctuation mark at the end of the sentence is an exclamation mark or a question mark. If there is, increase a certain sentiment score value.

[0133] S454, adding up all sentiment scores in the comment document to obtain the final sentiment analysis score of the comment document.

[0134] S455, the comment data is divided into two groups (positive and negative) according to the sentiment score. The number 0 is used as the sentiment boundary value, that is, positive sentiment is scored above 0, and negative sentiment is scored below 0.

[0135] S46, Results Comparison

[0136] Finally, by comparing the above text mining analysis results of different categories of consumers, we can obtain the differences in policy perceptions among different consumer groups.

[0137] The present invention is described with reference to flowcharts and / or block diagrams of methods, platforms (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0138] According to another aspect of the present invention, a consumer policy perception analysis system based on big data text mining is also provided, including: a data acquisition module for acquiring policy text data and consumer text data; a data preprocessing module for preprocessing the acquired text data; a corpus construction module for constructing a policy corpus based on the preprocessed text data; and a policy perception mining module for performing consumer policy perception analysis based on the policy corpus.

[0139] The modules involved in the above system can be implemented by computer chips or entities, or by products with certain functions. The present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A consumer policy perception analysis method based on big data text mining, characterized by: include: Obtain policy text data and consumer review text data and perform preprocessing; Constructing a policy corpus based on the preprocessed text data further includes: performing word splitting on the preprocessed policy text and consumer comment text data, and counting the word frequencies of all generated words; performing point-to-point mutual information screening based on the word frequency statistical results; performing left-right information entropy screening based on the point-to-point mutual information screening results; removing stop words and common general words based on the left-right information entropy screening results to obtain policy professional words; screening the obtained policy professional words with the expert knowledge corpus, and improving the identification of words related to policy perception in consumer comments, and classifying the screened and improved words into four categories according to policy classification to form the final policy corpus; The analysis of consumer policy perception based on the policy corpus further includes: dividing consumers into different categories; mining comment data among consumers of different categories, including word frequency intensity analysis, topic identification, semantic network analysis, and sentiment tendency analysis; comparing the mining and analysis results of consumers of different categories to obtain the differences in perceptions of different consumer groups on different categories of policies; The word frequency intensity analysis further includes: importing the constructed policy corpus and related field cell lexicon into the custom dictionary, using Jieba to segment the consumer review text, and counting the word frequency intensity; dividing the word frequency results according to policy classification, and generating corresponding word cloud diagrams according to the word frequency intensity; Topic identification further includes: storing the consumer review text after word segmentation in the format of a list per document; converting the text into a sparse vector expression corresponding to the bag-of-words model and constructing a word frequency matrix; calculating the perplexity of models with different numbers of topics and determining the optimal number of model topics; using the optimal number of topics evaluated by the perplexity method as a parameter, training the LDA model, and analyzing the topics of concern in consumer reviews; Semantic network analysis further includes: based on the policy corpus, select the comment texts that mention policies from the consumer comment texts; count the number of times high-frequency words in the comment texts appear together, and obtain the closeness between these words; count the frequency of co-occurring word pairs, and construct a high-frequency word co-occurrence matrix based on the results; based on the high-frequency word co-occurrence matrix, use NetDraw to draw a semantic network diagram, and obtain the hierarchy and close relationship between the nodes in the semantic network diagram, so as to understand consumers' perception of the policy; The sentiment tendency analysis further includes: extracting policy-related fragment documents from consumer reviews based on the policy corpus, segmenting each review document, and finding the sentiment words, negative words and degree adverbs in the document; judging whether each sentiment word is preceded by a negative word or a degree adverb, and if so, dividing the sentiment word and the previous negative word or degree adverb into a group, and multiplying the sentiment score of the sentiment word by the weight coefficient corresponding to the group; the weight coefficient of the negative word is set to -1, and different weight coefficients are set for the degree adverb according to the semantics; judging whether each sentence in the review document is an exclamation sentence or an interrogative sentence, and if so, increasing the sentiment score value; adding up all the sentiment scores in the review document is the final sentiment analysis score of the review document.

2. The method for analyzing consumer policy perception based on big data text mining according to claim 1 is characterized in that: The steps of obtaining policy text data and consumer review text data further include: Identify the data source; Use web crawler tools to collect policy text data and corresponding consumer review text data; Randomly sample the collected data sources and match and verify them with the crawled data; If the crawled data passes the verification, the corresponding data will be stored locally persistently.

3. The method for analyzing consumer policy perception based on big data text mining according to claim 1 is characterized in that: Preprocessing further includes deduplication, denoising and text short sentence removal.

4. A consumer policy perception analysis system based on big data text mining, implemented based on the method described in any one of claims 1 to 3, characterized in that: include: Data collection module, used to obtain policy text data and consumer comment text data; A data preprocessing module, used for preprocessing the acquired text data; The policy corpus construction module is used to construct a policy corpus based on the preprocessed text data; The policy perception mining module is used to analyze consumers’ policy perception based on the policy corpus.

Images