A platform message text mining method and system

Abstract

The present invention discloses a platform message text mining method and system, including data tagging of historical platform message information to obtain corresponding tag categories; training and constructing a long short-term memory network model based on historical platform message information and corresponding tag categories; classifying the platform message information to be classified with the trained long short-term memory network model to obtain the tag category to which the platform message information to be classified belongs; merging the message number and message subject of the platform message information under any tag category into a merged column, clustering the merged column to obtain the message cluster under the current tag category; and obtaining the hot message cluster under the current tag category based on the heat of the message cluster. For the user message data in the online government inquiry platform, it is helpful to timely discover hot issues, deal with them in a targeted manner, and improve service efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of data analysis and mining, and in particular to a method and system for mining platform message texts. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In recent years, as online government inquiry platforms such as the Sunshine Hotline, Weibo, Mayor's Mailbox, and WeChat have gradually become important channels for the government to understand public opinion, gather public wisdom, and gather public sentiment, the amount of text data related to various social conditions and public opinions has continued to rise, which has brought great challenges to the previous manual work of dividing messages and sorting out hot issues. At the same time, with the development of technologies such as artificial intelligence, big data, and cloud computing, the establishment of a smart government affairs system based on natural language processing technology has become a new trend in the innovative development of social governance, which has greatly promoted the improvement of government management level and municipal efficiency.

[0004] Data mining is a process of automatically discovering useful information in large data repositories, which generally includes data preprocessing, data mining, and post-processing. At present, there are many applications of data mining in related fields. However, based on the characteristics of public messages and government replies in the past, there are no mature results and explanations for the methods of mining public messages and evaluating government replies.

[0005] With the rapid increase in the amount of text data reflecting social conditions and public opinion and the faster pace of people's lives, the government's municipal efficiency must also be greatly improved. For the message information collected by various government inquiry platforms, the traditional method is to manually judge which department should give each message to reply. With people's increasing requirements for efficiency and the rapid increase in the amount of message data, the traditional manual classification method can no longer meet the needs of the times; in addition, after the message division is completed, the problems reflecting a specific place or a specific group of people in a certain period of time are called hot issues. The first thing the department needs to solve is the hot issue, because hot issues are often urgent or serious problems, so it is necessary to extract hot issues; for the extraction of hot issues, if data mining methods are used instead of traditional manual processing, the efficiency will be greatly improved. In summary, there is still a lack of effective solutions for dividing the messages of the masses and mining hot messages using data mining technology. Summary of the invention

[0006] In order to solve the above problems, the present invention proposes a platform message text mining method and system, which targets user message data in the online government inquiry platform, performs data tagging on them, obtains corresponding tag categories, classifies the message information according to the classification model, clusters them according to their categories, and mines hot messages according to the popularity of message clusters; this is conducive to timely discovering hot issues, targeted processing, and improving service efficiency.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a platform message text mining method, comprising:

[0009] Data tagging is performed on the acquired historical platform message information to obtain the corresponding tag category;

[0010] A long short-term memory network model constructed by training based on historical platform message information and corresponding tag categories;

[0011] The trained long short-term memory network model is used to classify the message information on the to-be-classified platform, and the label category to which the message information on the to-be-classified platform belongs is obtained;

[0012] The message number and message subject of the platform message information under any tag category are merged into a merged column, and the merged column is clustered to obtain the message cluster under the current tag category. According to the popularity of the message cluster, the hot platform message information corresponding to the hot message cluster under the current tag category is obtained.

[0013] In a second aspect, the present invention provides a platform message text mining system, comprising:

[0014] The marking module is used to mark the data of the acquired historical platform message information and obtain the corresponding marking category;

[0015] The training module is used to train the long short-term memory network model constructed based on the historical platform message information and the corresponding tag categories;

[0016] A classification module is used to classify the message information on the to-be-classified platform using the trained long short-term memory network model to obtain the label category to which the message information on the to-be-classified platform belongs;

[0017] The clustering module is used to merge the message number and message subject of the platform message information under any tag category into a merged column, cluster the merged column to obtain the message cluster under the current tag category, and obtain the hot platform message information corresponding to the hot message cluster under the current tag category according to the popularity of the message cluster.

[0018] In a third aspect, the present invention provides an electronic device comprising a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method described in the first aspect is completed.

[0019] In a fourth aspect, the present invention provides a computer-readable storage medium for storing computer instructions, wherein when the computer instructions are executed by a processor, the method described in the first aspect is performed.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] The present invention marks and classifies historical platform message information, performs data analysis, data cleaning and feature extraction from the original message data, mines useful information and standardizes the data set, performs subspace clustering based on the classified message information combined with its eigenvalues, and extracts hot issues, which is conducive to timely discovery of hot issues, has high accuracy and execution efficiency, and helps relevant departments to carry out targeted processing and improve service efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0023] Figure 1 A flow chart of a platform message text mining method provided in Example 1 of the present invention;

[0024] Figure 2 A flowchart of historical message details data preprocessing and feature extraction provided in Example 1 of the present invention;

[0025] Figure 3 A structural diagram of the CBOW model provided in Example 1 of the present invention;

[0026] Figure 4 LSTM model structure diagram provided for Example 1 of the present invention;

[0027] Figure 5 LSTM model training flow chart provided in Example 1 of the present invention;

[0028] Figure 6 The accuracy flow chart of LSTM model training and testing provided in Example 1 of the present invention;

[0029] Figure 7 A flowchart of cleaning, vectorization, TF-IDF preprocessing, standardization and dimensionality reduction of merged columns provided in Example 1 of the present invention;

[0030] Figure 8This is a cluster scatter plot provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0031] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0032] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0033] It should be noted that the terms used herein are only for describing specific embodiments, and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "include" and "have" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0035] Example 1

[0036] With the development and application of the Internet of Things and cloud computing, especially the popularization of mobile Internet, the use of e-government platforms has become popular. Publishing information through e-government platforms has become an important way for the public to express their opinions and views on public issues. The rapid increase in online message data has brought difficulties to the government's response work. The online messages on the e-government platform cannot be separated from data mining technology; Figure 1 As shown, this embodiment provides a platform message text mining method, which is applied to a smart government platform or other government platforms or a network platform where messages and comments can be posted, etc., and classifies the messages of the masses and mines hot issues according to the relevant data in the message information of the masses, specifically including:

[0037] S1: Mark the acquired historical platform message information to obtain the corresponding mark category;

[0038] S2: A long short-term memory network model constructed by training based on historical platform message information and corresponding tag categories;

[0039] S3: Classify the message information on the to-be-classified platform using the trained long short-term memory network model to obtain the label category to which the message information on the to-be-classified platform belongs;

[0040] S4: merge the message number and message subject of the platform message information under any tag category into a merged column, cluster the merged column, and obtain the message cluster under the current tag category;

[0041] S5: Obtaining the hot platform message information corresponding to the hot message cluster under the current mark category according to the popularity of the message cluster.

[0042] In step S1, the acquired historical platform message information is preprocessed, including cleaning and word segmentation of the historical platform message details information, and removing stop words; Figure 2 As shown, including:

[0043] S101: Clean up missing values: The historical platform message information and tagged category labels obtained may have missing values. Calculate the missing value ratio and determine the range of missing values;

[0044] Different processing strategies are adopted according to the missing ratio: for features with low missing rates in the records, they are filled in through experience or business knowledge; for features with high missing rates in the records, the records are deleted.

[0045] S102: Clear symbols: Delete all symbols except letters, numbers and Chinese characters in the message details.

[0046] S103: Word segmentation and removal of stop words: In order to save storage space and improve search efficiency, stop words should be removed before processing natural language; this embodiment adopts the jieba word segmentation method, which is mainly based on a statistical dictionary to construct a prefix dictionary; then the prefix dictionary is used to segment the input sentence to obtain all segmentation possibilities, and a directed acyclic graph is constructed according to the segmentation position; the maximum probability path is calculated through a dynamic programming algorithm to obtain the final segmentation form; after the message details information is segmented using the precise mode of the jieba word segmentation, the stop words are removed from the segmented message details information.

[0047] In step S2, the historical platform message information is divided into a training set and a test set, and feature extraction is performed on the historical platform message information. The specific steps are:

[0048] This embodiment extracts features from the platform message information list after word segmentation. The message features include the message number, the message user, the message details, the message time, the number of objections and the number of likes. The LSTM neural network algorithm is used to map the platform message information to data that can be processed by LSTM, and the feature vector is selected according to the feature extraction algorithm. This embodiment uses the Tokenizer of keras to implement it. The specific steps are as follows:

[0049] S201: Processing the platform message information into a single word index sequence, and the correspondence between the word and the sequence number depends on the word index table;

[0050] S202: All classification labels are represented by one-hot codes, and a list of classification label mappings is generated. A dictionary is generated by mapping with the one-hot codes, which is {classification name: one-hot code} for easy mapping and search;

[0051] S203: directly convert and map the platform message information into single words to generate a word-based mapping dictionary;

[0052] S204: Mapping words to numbers;

[0053] S205: pre-train word2vec vector;

[0054] In step S205, word2vec uses a distributed word vector representation method to map each word into a real vector of K dimensions through a training model, and then determines the semantic similarity between words by the distance between them.

[0055] Word2vec has CBOW model and Skip-gram model. This embodiment uses CBOW model to train word vector. CBOW model includes three layers: input layer, output layer and projection layer. It predicts the current word wt based on the known w1 context wt-2, wt-2, wt+1, wt+2. The CBOW model processing structure is as follows: Figure 3 shown.

[0056] In word2vec, hierarchical Softmax function and negative sampling are used for approximate calculations, with the aim of reducing training time, increasing calculation speed and improving the quality of word vectors after training. Compared with the hierarchical Softmax function, NEG no longer uses a complex Huffman tree, but instead adopts a random negative sampling method to reduce the probability of negative sampling while increasing the probability of positive samples.

[0057] In the CBOW model, the context of word w is known to be Context(w), and word w needs to be predicted. Therefore, for a given Context(w), word w is a positive sample, and other words are negative samples. For a given sample (Context(w), w), maximize it as much as possible.

[0058]

[0059] Among them, NEG(w) represents the negative sample set, the positive sample label is 1, and the negative sample label is 0. The probability calculation formula is as follows:

[0060]

[0061] This embodiment first uses the word2vec word vector trained with specific corpus data to perform text classification tasks, which can improve the expressive power of the word vector to a certain extent; then uses Scikit-learn to divide the data into training data and test data in a ratio of 9:1.

[0062] In step S2, the constructed long short-term memory network LSTM model includes:

[0063] This embodiment adopts an LSTM sequence model. The first layer of the model is an embedding layer, which uses a vector of length 100 to represent each word. The SpatialDropout1D layer randomly sets the input unit ratio to 0.2 each time it is updated during training, which helps prevent overfitting. The LSTM layer contains 100 memory units, and the output layer is a fully connected layer containing 10 categories.

[0064] In step S2, training the long short-term memory network LSTM model includes: in the historical message details training stage, building the LSTM model according to the LSTM algorithm idea, processing the text into the embedding layer of the vector, and each message detail is processed into a two-dimensional vector of word_dict×250, where word_dict is the length of each message detail, 250 is the length of each line, and the whole represents the word vector of the word in the space. The operation process is as follows:

[0065] S201: LSTM layer outputs a 1-dimensional vector of length 250;

[0066] S202: The initial parameter of the Dropout layer is set to 0.2 to avoid overfitting;

[0067] S203: Dense (fully connected layer) shrinks the vector length to 7, corresponding to 7 message categories.

[0068] The LSTM training model structure is as follows Figure 4-5 As shown in the figure, since the training result is multi-classification, the activation function softmax, the loss function cross-entropy, and the optimizer adam are used simultaneously. The batch_size in the training process is set to 64, and the epoch is set to 6. The actual training round takes about 1 minute. After 6 rounds, the loss gradually decreases, and the accuracy becomes higher and higher. The accuracy finally reaches 0.9757.

[0069] S211: Test the LSTM classification model and use F1-Score as the indicator for evaluating the classification model, that is, the harmonic mean of precision and recall, with a maximum of 1 and a minimum of 0:

[0070]

[0071] Among them, precision is the accuracy rate and recall is the recall rate.

[0072] In this embodiment, the accuracy of the test message data is 0.833, and the accuracy of the training message data and the test message data is as follows: Figure 6 shown.

[0073] In the step S3, the test message details to be classified are obtained, the cleaned and vectorized message details to be classified are input into the long short-term memory network LSTM, and the tag category to which the test message details belong is output.

[0074] In step S4, after classification, a message information record of a certain category within a period of time is intercepted, and the record includes: message number, message details, number of objections and number of likes, and the message number and message subject are merged into a merged column, and the information of the merged column is cleaned and word segmented; the specific steps are:

[0075] S401: Cleaning the merged column information: removing Chinese symbols and English symbols in the text;

[0076] S402: word segmentation: This embodiment uses the Jieba word segmentation method, see step S103 for details, which will not be repeated here.

[0077] In step S4, the merged columns are vectorized, TF-IDF preprocessed and standardized, such as Figure 7 As shown, the specific steps are:

[0078] S411: Vectorization: This embodiment adopts the bag-of-words model. After word segmentation, the word-based features of the text are obtained by counting the number of times each word appears in the text. If the words of each text sample and the corresponding word frequency are put together, it is vectorized.

[0079] S412: TF-IDF preprocessing: After the text is segmented and vectorized, the word vectors formed by each word in the vocabulary in each text are obtained, and then TF-IDF preprocessing is performed;

[0080] For example, the following four short texts were counted for word frequency:

[0081] corpus=["I come to China to travel","This is a car polupar inChina","I love tea and Apple","The work is to write some papers in science"]

[0082] Without considering stop words, the word vector obtained after processing is as follows: [[0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 2 1 0 0] [0 0 1 1 0 1 1 0 0 1 0 0 0 0 1 0 0 0 0] [1 1 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0] [0 0 0 0 0 1 1 0 1 0 1 1 0 1 0 1 0 1 1]]

[0087] If the 19-dimensional features after word frequency statistics are directly used as input for text classification, problems will be found: for example, in the first text, "come", "China" and "Travel" appear once each, while "to" appears twice, indicating that the text is more closely related to the "to" feature. However, in fact, "to" is a very common word and is used in almost all texts. Therefore, although its word frequency is 2, its importance is lower than that of "China" and "Travel" with a word frequency of 1. If the vectorized features are only represented by word frequency, this cannot be reflected. Therefore, further preprocessing is required to reflect this feature of the text, namely TF-IDF.

[0088] TF-IDF (Term Frequency-Inverse Document Frequency) consists of two parts: TF and IDF.

[0089] TF stands for word frequency. Vectorization means counting the frequency of occurrence of each word in the text and using it as a text feature.

[0090] IDF, or "Inverse Document Frequency", in the above example, "to", which appears in almost all texts, has a high frequency, but its importance should be lower than that of "China" and "Travel" with low frequencies. IDF reflects the importance of this word, and then corrects the word feature value expressed only by frequency.

[0091] That is, IDF reflects the frequency of a word appearing in all texts. If a word appears in many texts, its IDF value is low, such as "to". On the contrary, if a word appears in relatively few texts, its IDF value should be high, such as some professional nouns such as "Machine Learning", which have high IDF values. If a word appears in all texts, its IDF value is 0.

[0092] Quantitatively analyze the IDF of a word and give the formula for the IDF of a word x as follows:

[0093]

[0094] Where N represents the total number of texts in the corpus, and N(x) represents the total number of texts in the corpus that contain word x.

[0095] If a rare word does not exist in the corpus, the denominator of the above formula is 0, so the commonly used IDF needs to be smoothed so that the words that do not appear in the corpus can also get a suitable IDF value; the formula for smoothed IDF used in this embodiment is:

[0096]

[0097] Calculate the TF-IDF value of a word:

[0098] TF-IDF(x)=TF(x)*IDF(x)

[0099] Where TF(x) refers to the frequency of word x in the current text.

[0100] S413: Perform dimensionality reduction processing on the merged columns after TF-IDF preprocessing. The specific steps are as follows:

[0101] In large-scale text processing, since the dimension of the feature corresponds to the size of the word segmentation vocabulary, dimensionality reduction is required when the dimension is large. This embodiment adopts the principal component analysis method to synthesize correlated high-dimensional variables into linearly independent low-dimensional variables, called principal components. The new low-dimensional data set will retain the variables of the original data as much as possible.

[0102] S414: Clustering the merged columns after dimensionality reduction. The specific steps are as follows:

[0103] This embodiment uses the K-means clustering algorithm to divide the vector space of the sample into multiple regions Each region has a regional center For each sample, a mapping q(x) from the sample to the center of the region is established:

[0104]

[0105] Among them, 1() is an indicator function, which represents whether the sample x belongs to region S.

[0106] The main difference between different partition-based clustering algorithms lies in how to establish the corresponding mapping method q(x). In K-means, the mapping is established by minimizing the sum of squared errors between samples and centers. The main implementation steps of the K-means algorithm are:

[0107] S414-1: Initialize cluster centers Select the first K samples or randomly select K samples;

[0108] S414-2: Allocate each sample x j To the closest cluster set, the sample allocation is based on:

[0109]

[0110] Where i = 1, 2, …, k, p ≠ j;

[0111] S414-3: Update the cluster center according to the allocation result:

[0112]

[0113] S414-4: If the number of iterations reaches the maximum number of iterations or the difference between the previous and next iterations is less than the set threshold ε, Then it ends, otherwise repeat step S414-2; the cluster scatter diagram is as follows Figure 8 shown.

[0114] The step S5 calculates the popularity of the message cluster, and the specific steps are:

[0115] S501: Each cluster represents a question, and the number of dislikes and likes corresponding to the element records in the message cluster are extracted.

[0116] S502: sum up all antilogarithms and like numbers of elements in the cluster respectively to obtain the antilogarithm and like number of the cluster.

[0117] S503: Calculate the popularity of the cluster using the Reddit comment ranking algorithm:

[0118]

[0119] Wilson's scoring range is as follows:

[0120]

[0121] in, is the rate of favorable comments; n is the total number of votes; Z(1-α / 2) represents the z statistic corresponding to a certain confidence level, which is a constant and can be obtained by looking up the table. Generally, at a confidence level of 95%, the value of the z statistic is 1.96.

[0122] S504: ranking each cluster from high to low according to the popularity of each cluster, and the message contents corresponding to the top N message clusters are hot issues; it can be understood that the value of N can be customized.

[0123] Example 2

[0124] This embodiment provides a platform message text mining system, including:

[0125] The marking module is used to mark the data of the acquired historical platform message information and obtain the corresponding marking category;

[0126] The training module is used to train the long short-term memory network model constructed based on the historical platform message information and the corresponding tag categories;

[0127] A classification module is used to classify the message information on the to-be-classified platform using the trained long short-term memory network model to obtain the label category to which the message information on the to-be-classified platform belongs;

[0128] A clustering module is used to merge the message number and message subject of the platform message information under any tag category into a merged column, cluster the merged column, and obtain the message cluster under the current tag category;

[0129] The popularity module is used to obtain the hot platform message information corresponding to the hot message cluster under the current tag category according to the popularity of the message cluster.

[0130] It should be noted that the above modules correspond to steps S1 to S5 in Example 1, and the examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above Example 1. It should be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer executable instructions.

[0131] In further embodiments, there is also provided:

[0132] An electronic device includes a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method described in Embodiment 1 is performed. For the sake of brevity, it will not be described in detail here.

[0133] It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0134] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0135] A computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the method described in Example 1 is completed.

[0136] The method in Example 1 can be directly embodied as a hardware processor, or a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it is not described in detail here.

[0137] Those skilled in the art will appreciate that the units, i.e., algorithm steps, of the various examples described in conjunction with this embodiment can be implemented in electronic hardware or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0138] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0139] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A platform message text mining method, characterized in that: include: Data tagging is performed on the acquired historical platform message information to obtain the corresponding tag category; A long short-term memory network model constructed by training based on historical platform message information and corresponding tag categories; The trained long short-term memory network model is used to classify the message information on the to-be-classified platform, and the label category to which the message information on the to-be-classified platform belongs is obtained; Merge the message number and message subject of the platform message information under any tag category into a merged column, cluster the merged column to obtain the message cluster under the current tag category, and obtain the hot platform message information corresponding to the hot message cluster under the current tag category according to the popularity of the message cluster; The popularity of the message cluster is obtained by the following method, including: Extract the number of dissent and likes corresponding to the message information on the message cluster platform, add up the number of dissent and likes of all the message information on the platform in the message cluster, and get the number of dissent and likes of the message cluster; use the Reddit comment ranking algorithm to calculate the popularity of the message cluster: Wilson's scoring range is as follows: in, is the favorable rate; n is the total number of votes; Represents the z statistic corresponding to a certain confidence level, which is a constant; According to the popularity of each cluster, the clusters are ranked from high to low, and the message information corresponding to the top N message clusters is the hot issues; Pre-process the acquired historical platform message information, including cleaning, word segmentation, and removal of stop words; specifically, including: Clean up missing values: The historical platform message information and tag categories obtained may have missing values. Calculate the missing value ratio and determine the range of missing values. According to the missing ratio, adopt different processing strategies: For features with low missing rates in the records, fill them in through experience or business knowledge; For features with high missing rates in the records, delete the record. Clear symbols: Delete all symbols except letters, numbers and Chinese characters in the historical platform messages; Word segmentation and stop word removal: Jieba word segmentation method is used to construct a prefix dictionary. The historical platform message information is segmented according to the prefix dictionary. A directed acyclic graph is constructed according to the segmentation position. The maximum probability path is calculated through a dynamic programming algorithm to obtain the final word segmentation form. Stop words are removed from the historical platform message information after word segmentation.

2. A platform message text mining method as claimed in claim 1, characterized in that: After performing word segmentation on the merged column, the bag-of-words model is used to count the number of times each word appears, obtain word-based features, merge the word with the corresponding word frequency, complete vectorization, and obtain the word vector formed by each word in the message information of each historical platform, and then perform TF-IDF preprocessing; The principal component analysis method is used to reduce the dimension of the merged columns after TF-IDF preprocessing; The K-means clustering algorithm is used to cluster the merged columns after dimensionality reduction.

3. A platform message text mining system, characterized in that: include: The marking module is used to mark the data of the acquired historical platform message information and obtain the corresponding marking category; The training module is used to train the long short-term memory network model constructed based on the historical platform message information and the corresponding tag categories; A classification module is used to classify the message information on the to-be-classified platform using the trained long short-term memory network model to obtain the label category to which the message information on the to-be-classified platform belongs; A clustering module is used to merge the message number and message subject of the platform message information under any tag category into a merged column, cluster the merged column to obtain the message cluster under the current tag category, and obtain the hot platform message information corresponding to the hot message cluster under the current tag category according to the popularity of the message cluster; The popularity of the message cluster is obtained by the following method, including: Extract the number of dissent and likes corresponding to the message information on the message cluster platform, add up the number of dissent and likes of all the message information on the platform in the message cluster, and get the number of dissent and likes of the message cluster; use the Reddit comment ranking algorithm to calculate the popularity of the message cluster: Wilson's scoring range is as follows: in, is the rate of favorable comments; n is the total number of votes Represents the z statistic corresponding to a certain confidence level, which is a constant; According to the popularity of each cluster, the clusters are ranked from high to low, and the message information corresponding to the top N message clusters is the hot issues; Pre-process the acquired historical platform message information, including cleaning, word segmentation, and removal of stop words; specifically, including: Clean up missing values: The historical platform message information and tag categories obtained may have missing values. Calculate the missing value ratio and determine the range of missing values. According to the missing ratio, adopt different processing strategies: For features with low missing rates in the records, fill them in through experience or business knowledge; For features with high missing rates in the records, delete the record. Clear symbols: Delete all symbols except letters, numbers and Chinese characters in the historical platform messages; Word segmentation and stop word removal: Jieba word segmentation method is used to construct a prefix dictionary. The historical platform message information is segmented according to the prefix dictionary. A directed acyclic graph is constructed according to the segmentation position. The maximum probability path is calculated through a dynamic programming algorithm to obtain the final word segmentation form. Stop words are removed from the historical platform message information after word segmentation.

4. An electronic device, characterized in that: The method comprises a memory and a processor and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method according to any one of claims 1 to 2 is completed.

5. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed by a processor, complete the method described in any one of claims 1-2.

Images