Method for automatically analyzing forum website data

Abstract

The invention discloses a method for automatically analyzing forum website data, and relates to the technical field of forum website data analysis. The method is realized based on a forum website data automatic analysis system, and the forum website data automatic analysis system comprises a data acquisition module, a data processing module, a data processing module, a data processing module and a data processing module, and the data acquisition module is responsible for capturing related data from a forum website and comprises post titles and content information; the data preprocessing module is used for preprocessing the collected data so as to facilitate subsequent keyword extraction and weight calculation; the keyword extraction module is used for extracting keywords according to the preprocessed data; and the topic classification module is used for automatically classifying the posts according to the keywords and the weight and context information thereof. According to the method, data of the forum website can be automatically analyzed, and the working efficiency is improved; keywords of a forum website are extracted through a TF-IDF algorithm, and a user is helped to rapidly know theme content; and weight analysis is carried out on each keyword to determine the importance of the keyword, so that a user can conveniently screen information.

Description

Technical Field

[0001] The present invention relates to the technical field of forum website data analysis, and in particular to a method for automatically analyzing forum website data. Background Art

[0002] With the development of the Internet, forum websites have become an important platform for people to exchange ideas and share experiences. However, due to the large number of forum posts, users often need to spend a lot of time to filter and find the content they are interested in; and for administrators, managing such a large amount of information is even more laborious. Therefore, there is an urgent need for a method to automatically parse forum website data to complete the parsing of forum data and provide convenience for relevant personnel.

[0003] After searching, the Chinese patent application number CN201910793205.2 discloses a forum data management classification system and method, which involves the field of big data processing technology. The system and method collect original forum data from multiple data sources in real time through the data acquisition module; and cache the original forum data through the message queue unit and then pass it to the data preprocessing module for data preprocessing; the forum data in Json format output from the data preprocessing module is indexed and stored; the forum data is intelligently classified in the algorithm engine module so that each type of forum data corresponds to a type of administrator; the forum administrator enters the system through the identity authentication module to analyze the forum data within the scope of authority. The forum data management classification system in the above patent has the following shortcomings: it cannot split the forum information according to the editable keywords and their weights, and the flexibility is insufficient, which needs to be improved. Summary of the invention

[0004] The purpose of the present invention is to solve the shortcomings of the prior art and to propose a method for automatically parsing forum website data.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A method for automatically parsing forum website data is implemented based on an automatic parsing system for forum website data, wherein the automatic parsing system for forum website data comprises:

[0007] Data collection module: responsible for grabbing relevant data from forum websites, including post titles and content information;

[0008] Data preprocessing module: preprocess the collected data for subsequent keyword extraction and weight calculation;

[0009] Keyword extraction module: extract keywords based on preprocessed data;

[0010] Topic classification module: automatically classifies posts based on keywords, their weights, and contextual information;

[0011] Result display module: displays the analysis results in charts or other forms.

[0012] Preferably: the method for automatically parsing forum website data comprises the following steps:

[0013] S1: The data collection module grabs relevant data information from the forum website;

[0014] S2: The data preprocessing module cleans and standardizes the collected data;

[0015] S3: The keyword extraction module extracts keywords based on the preprocessed data;

[0016] S4: The topic classification module automatically classifies posts based on keywords, keyword weights, and context information;

[0017] S5: The result display module displays the analysis results in the form of charts or other forms.

[0018] Preferably: the forum website data automatic parsing system further includes a keyword setting module, and the keyword setting module includes:

[0019] Keyword management unit: used to add, modify or delete set keywords;

[0020] Backup unit: After completing the keyword setting, the backup unit will automatically save the setting and back it up.

[0021] Preferably: the forum website data automatic parsing system further includes a weight value allocation module, and the weight value allocation module includes:

[0022] Weight value allocation unit: used to allocate a weight value to each keyword and sort them according to priority;

[0023] Scheme saving unit: After completing the weight value allocation, the scheme saving unit will automatically save the current setting scheme.

[0024] Preferably: the method for automatically parsing forum website data comprises the following steps:

[0025] S11: The user sets keywords through the keyword management unit;

[0026] S12: assigning a weight value to each keyword through a weight value assigning unit, and sorting them according to priority;

[0027] S13: The data collection module captures relevant data information from the forum website;

[0028] S14: The data preprocessing module cleans and standardizes the collected data;

[0029] S15: The keyword extraction module extracts keywords based on the preprocessed data;

[0030] S16: The topic classification module automatically classifies posts based on the set keywords and weighting scheme;

[0031] S17: The result display module displays the analysis results in a chart or other form.

[0032] Preferably: the data preprocessing module includes:

[0033] Text cleaning unit: removes HTML tags and special characters from text;

[0034] Word segmentation processing unit: divides the text into words or phrases to facilitate subsequent keyword extraction;

[0035] Stop word filtering unit: removes common words in the text and reduces meaningless information;

[0036] Stemming unit: converts words in the text into their base form;

[0037] Lemmatization unit: restores words in the text to their original form;

[0038] Synonym expansion unit: automatically expands synonyms in the text based on context information to increase keyword coverage;

[0039] Word frequency counting unit: counts the frequency of occurrence of each word in the text.

[0040] Preferably: the subject classification module includes:

[0041] Clustering classification unit: clusters and classifies texts by topic based on keywords and weight values ​​set by the user;

[0042] Classifier Classification Unit: Use machine learning algorithms to classify texts, and perform training and prediction based on keywords and weight values ​​set by users;

[0043] Rule classification unit: classifies text according to the rules and conditions set by the user.

[0044] Preferably: the forum website data automatic parsing system further includes an automatic parsing module, and the push analysis module includes:

[0045] Automatic recommendation unit: Based on the user's tag data and historical behavior, the platform can automatically recommend posts and users that match the user's interests;

[0046] Data analysis unit: The platform can analyze users’ tagged data to understand their interests, hobbies and needs, thereby providing more personalized services.

[0047] Preferably, the method for automatically parsing forum website data adopts the TF-IDF algorithm when extracting keywords. The specific formula is as follows:

[0048]

[0049] Among them, t represents a word, D represents a document set, tf D (t) represents the number of times the word appears in the document set, IDF(t) represents the inverse document frequency of the word in the entire corpus; N represents the number of all words in the document set, and α and β are two weight parameters.

[0050] Preferably, the method for automatically parsing forum website data adopts the Word2Vec algorithm when extracting keywords. The specific formula is as follows:

[0051]

[0052] Among them, ω i and ω j represents two words, and N represents the number of all words in the corpus.

[0053] This formula is used to calculate the similarity between two words. Specifically, it divides the Euclidean distance between the two word vectors by a constant. Then take the exponent and finally divide by N to get their similarity.

[0054] Preferably, when extracting keywords, the TextRank algorithm is used, and the specific formula is as follows:

[0055]

[0056] Among them, w1 and w2 represent two words, w3 represents the third word, and α and β are two weight parameters.

[0057] The beneficial effects of the present invention are:

[0058] 1. The present invention can automatically parse the data of forum websites and improve work efficiency; extract the keywords of forum websites through TF-IDF algorithm to help users quickly understand the subject content; perform weight analysis on each keyword to determine its importance, so as to facilitate users to filter information.

[0059] 2. The present invention displays the analysis results in the form of charts, which are intuitive and easy to understand; the system is suitable for different types of forum websites and has strong adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 The present invention provides a flow chart of a method for automatically parsing forum website data. DETAILED DESCRIPTION

[0061] The technical solution of this patent is further described in detail below in conjunction with specific implementation methods.

[0062] Embodiment 1:

[0063] A method for automatically parsing forum website data is implemented based on an automatic parsing system for forum website data, wherein the automatic parsing system for forum website data comprises:

[0064] Data collection module: responsible for grabbing relevant data from forum websites, including post titles and content information;

[0065] Data preprocessing module: preprocess the collected data for subsequent keyword extraction and weight calculation;

[0066] Keyword extraction module: extract keywords based on preprocessed data using methods such as TF-IDF algorithm or TextRank algorithm;

[0067] Topic classification module: automatically classifies posts based on keywords, their weights, and contextual information;

[0068] Result display module: displays the analysis results in charts or other forms.

[0069] The method for automatically parsing forum website data comprises the following steps:

[0070] S1: The data collection module grabs relevant data information from the forum website;

[0071] S2: The data preprocessing module cleans and standardizes the collected data;

[0072] S3: The keyword extraction module extracts keywords based on the preprocessed data;

[0073] S4: The topic classification module automatically classifies posts based on keywords, keyword weights, and context information;

[0074] S5: The result display module displays the analysis results in the form of charts or other forms.

[0075] Wherein, the data preprocessing module includes:

[0076] Text cleaning unit: Remove interfering factors such as HTML tags and special characters from the text;

[0077] Word segmentation processing unit: Split the text into words or phrases to facilitate subsequent keyword extraction;

[0078] Stop word filtering unit: Remove common words in the text, such as "de" (of), "le" (particle indicating completion), etc., to reduce meaningless information;

[0079] Stemming unit: Convert words in the text to their basic forms, e.g., convert "running" to "run";

[0080] Lemmatization unit: Restore words in the text to their original forms, e.g., convert "am" to "be";

[0081] Synonym expansion unit: Automatically expand synonyms in the text according to context information to increase keyword coverage;

[0082] Word frequency statistics unit: Count the frequency of each word in the text for subsequent keyword extraction and weight assignment.

[0083] Among them, the theme classification module includes:

[0084] Clustering classification unit: Cluster and classify the text by theme according to keywords and weight values set by the user;

[0085] Classifier classification unit: Use machine learning algorithms (Naive Bayes, Support Vector Machine, etc.) to classify the text, and perform training and prediction according to keywords and weight values set by the user;

[0086] Rule classification unit: Classify the text according to rules and conditions set by the user; for example, it can be classified according to keywords, sentiment colors, etc. appearing in the text.

[0087] Among them, the automatic parsing system for forum website data further includes an automatic parsing module, and the push analysis module includes:

[0088] Automatic recommendation unit: According to the user's marked data and historical behavior, the platform can automatically recommend posts and users that match the user's interests;

[0089] Data analysis unit: The platform can analyze the user's marked data to understand the user's interests and needs, so as to provide more personalized services.

[0090] Among them, when extracting keywords in the method for automatically parsing forum website data, the TF-IDF algorithm is adopted, and the specific formula is as follows:

[0091]

[0092] Among them, t represents a word, D represents a document set, tf D (t) represents the number of times the word appears in the document set, IDF(t) represents the inverse document frequency of the word in the entire corpus; N represents the number of all words in the document set, α and β are two weight parameters, which are adjusted according to the specific application scenario;

[0093] Through the above formula, the word frequency, inverse document frequency and weight parameters are combined to better measure the importance of a word in the text; specifically, for a word t, if it appears more times in a document and less times in the entire corpus, then its TF-IDF value will be higher, indicating that it is more important in the document; conversely, if it appears less times in a document and more times in the entire corpus, then its TF-IDF value will be lower, indicating that it is less important in the document. At the same time, the weight parameters α and β can control the weights of different types of words in the calculation. For example, for some common nouns or proper nouns, their weights may be higher.

[0094] Embodiment 2:

[0095] A method for automatically parsing forum website data is implemented based on an automatic parsing system for forum website data, wherein the automatic parsing system for forum website data comprises:

[0096] Data collection module: responsible for grabbing relevant data from forum websites, including post titles and content information;

[0097] Data preprocessing module: preprocess the collected data for subsequent keyword extraction and weight calculation;

[0098] Keyword extraction module: extract keywords based on preprocessed data using methods such as TF-IDF algorithm or TextRank algorithm;

[0099] Topic classification module: automatically classifies posts based on keywords, their weights, and contextual information;

[0100] Result display module: displays the analysis results in charts or other forms.

[0101] The method for automatically parsing forum website data comprises the following steps:

[0102] S1: The data collection module grabs relevant data information from the forum website;

[0103] S2: The data preprocessing module cleans and standardizes the collected data;

[0104] S3: The keyword extraction module extracts keywords based on the preprocessed data;

[0105] S4: The topic classification module automatically classifies posts based on keywords, keyword weights, and context information;

[0106] S5: The result display module displays the analysis results in the form of charts or other forms.

[0107] The forum website data automatic parsing system further includes a keyword setting module, which includes:

[0108] Keyword management unit: used to add, modify or delete set keywords:

[0109] Backup unit: After completing the keyword setting, the backup unit will automatically save the setting and back it up.

[0110] The forum website data automatic parsing system further includes a weight value allocation module, which includes:

[0111] Weight value allocation unit: used to allocate a weight value to each keyword and sort them according to priority;

[0112] Scheme saving unit: After completing the weight value allocation, the scheme saving unit will automatically save the current setting scheme.

[0113] The method for automatically parsing forum website data comprises the following steps:

[0114] S11: The user sets keywords through the keyword management unit;

[0115] S12: assigning a weight value to each keyword through a weight value assigning unit, and sorting them according to priority;

[0116] S13: The data collection module captures relevant data information from the forum website;

[0117] S14: The data preprocessing module cleans and standardizes the collected data;

[0118] S15: The keyword extraction module extracts keywords based on the preprocessed data;

[0119] S16: The topic classification module automatically classifies posts based on the set keywords and weighting scheme;

[0120] S17: The result display module displays the analysis results in a chart or other form.

[0121] Among them, the data preprocessing module includes:

[0122] Text cleaning unit: Remove interference factors such as HTML tags and special characters in the text;

[0123] Word segmentation processing unit: Split the text into words or phrases to facilitate subsequent keyword extraction;

[0124] Stop word filtering unit: Remove common words in the text, such as "de" (of), "le" (particle indicating completion), etc., to reduce meaningless information;

[0125] Stem extraction unit: Convert words in the text to their basic forms, such as converting "running" to "run";

[0126] Lemmatization unit: Restore words in the text to their original forms, such as converting "am" to "be";

[0127] Synonym expansion unit: Automatically expand synonyms in the text according to context information to increase keyword coverage;

[0128] Word frequency statistics unit: Statistically calculate the frequency of each word appearing in the text for subsequent keyword extraction and weight assignment.

[0129] Among them, the theme classification module includes:

[0130] Clustering classification unit: Classify the text by theme according to the keywords and weight values set by the user;

[0131] Classifier classification unit: Use machine learning algorithms (such as Naive Bayes, Support Vector Machine, etc.) to classify the text, and perform training and prediction according to the keywords and weight values set by the user;

[0132] Rule classification unit: Classify the text according to the rules and conditions set by the user; for example, it can be classified according to keywords, sentiment colors, etc. appearing in the text.

[0133] Among them, the forum website data automatic parsing system further includes an automatic parsing module, and the push analysis module includes:

[0134] Automatic recommendation unit: According to the user's marked data and historical behavior, the platform can automatically recommend posts and users that meet the user's interests;

[0135] Data analysis unit: The platform can analyze the user's marked data to understand the user's interests and needs, so as to provide more personalized services.

[0136] Among them, for the method of automatically parsing forum website data, when performing keyword extraction, the Word2Vec algorithm is adopted, and the specific formula is as follows:

[0137]

[0138] Among them, ω i and ω j represents two words, and N represents the number of all words in the corpus.

[0139] This formula is used to calculate the similarity between two words. Specifically, it divides the Euclidean distance between the two word vectors by a constant. Then take the exponent and finally divide by N to get their similarity.

[0140] Embodiment 3:

[0141] This embodiment is based on Embodiment 1 or 2: the method for automatically parsing forum website data adopts the TextRank algorithm when extracting keywords. The specific formula is as follows:

[0142]

[0143] Among them, w1 and w2 represent two words, w3 represents the third word, α and β are two weight parameters, which can be adjusted according to the specific application scenario; the above formula calculates the similarity of two words as the dot product between them divided by the product of their lengths, plus a weight parameter α and a weight parameter β; multiplied by their similarity with the third word respectively. In this way, words with higher similarity have higher weights in the calculation, thereby better reflecting the relationship between them.

[0144] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for automatically parsing forum website data, characterized in that: It is implemented based on the automatic parsing system of forum website data, and the automatic parsing system of forum website data includes: Data collection module: responsible for grabbing relevant data from forum websites, including post titles and content information; Data preprocessing module: preprocess the collected data for subsequent keyword extraction and weight calculation; Keyword extraction module: extract keywords based on preprocessed data; Topic classification module: automatically classifies posts based on keywords, their weights, and contextual information; Result display module: displays the analysis results in charts or other forms.

2. A method for automatically parsing forum website data according to claim 1, characterized in that: The method for automatically parsing forum website data comprises the following steps: S1: The data collection module grabs relevant data information from the forum website; S2: The data preprocessing module cleans and standardizes the collected data; S3: The keyword extraction module extracts keywords based on the preprocessed data; S4: The topic classification module automatically classifies posts based on keywords, keyword weights, and context information; S5: The result display module displays the analysis results in the form of charts or other forms.

3. A method for automatically parsing forum website data according to claim 1, characterized in that: The forum website data automatic parsing system also includes a keyword setting module, which includes: Keyword management unit: used to add, modify or delete set keywords; Backup unit: After completing the keyword setting, the backup unit will automatically save the setting and back it up.

4. A method for automatically parsing forum website data according to claim 3, characterized in that: The forum website data automatic parsing system further includes a weight value allocation module, which includes: Weight value allocation unit: used to allocate a weight value to each keyword and sort them according to priority; Scheme saving unit: After completing the weight value allocation, the scheme saving unit will automatically save the current setting scheme.

5. A method for automatically parsing forum website data according to claim 4, characterized in that: The method for automatically parsing forum website data comprises the following steps: S11: The user sets keywords through the keyword management unit; S12: assigning a weight value to each keyword through a weight value assigning unit, and sorting them according to priority; S13: The data collection module captures relevant data information from the forum website; S14: The data preprocessing module cleans and standardizes the collected data; S15: The keyword extraction module extracts keywords based on the preprocessed data; S16: The topic classification module automatically classifies posts based on the set keywords and weighting scheme; S17: The result display module displays the analysis results in a chart or other form.

6. A method for automatically parsing forum website data according to claim 1, characterized in that: The data preprocessing module comprises: Text cleaning unit: removes HTML tags and special characters from text; Word segmentation processing unit: divides the text into words or phrases to facilitate subsequent keyword extraction; Stop word filtering unit: removes common words in the text and reduces meaningless information; Stemming unit: converts words in the text into their base form; Lemmatization unit: restores words in the text to their original form; Synonym expansion unit: automatically expands synonyms in the text based on context information to increase keyword coverage; Word frequency counting unit: counts the frequency of occurrence of each word in the text.

7. A method for automatically parsing forum website data according to claim 1, characterized in that: The subject classification module includes: Clustering classification unit: clusters and classifies texts by topic based on keywords and weight values ​​set by the user; Classifier Classification Unit: Use machine learning algorithms to classify texts, and perform training and prediction based on keywords and weight values ​​set by users; Rule classification unit: classifies text according to the rules and conditions set by the user; The forum website data automatic parsing system also includes an automatic parsing module, and the push analysis module includes: Automatic recommendation unit: Based on the user's tag data and historical behavior, the platform can automatically recommend posts and users that match the user's interests; Data analysis unit: The platform can analyze users’ tagged data to understand their interests, hobbies and needs, thereby providing more personalized services.

8. A method for automatically parsing forum website data according to claim 2, characterized in that: The method for automatically parsing forum website data adopts the TF-IDF algorithm when extracting keywords. The specific formula is as follows: Among them, t represents a word, D represents a document set, tf D (t) represents the number of times the word appears in the document set, IDF(t) represents the inverse document frequency of the word in the entire corpus; N represents the number of all words in the document set, and α and β are two weight parameters.

9. A method for automatically parsing forum website data according to claim 2, characterized in that: The method for automatically parsing forum website data adopts the Word2Vec algorithm when extracting keywords. The specific formula is as follows: Among them, ω i and ω j represents two words, and N represents the number of all words in the corpus. This formula is used to calculate the similarity between two words. Specifically, it divides the Euclidean distance between the two word vectors by a constant. Then take the exponent and finally divide by N to get their similarity.

10. The method for automatically parsing forum website data according to claim 2, characterized in that: The method for automatically parsing forum website data adopts the TextRank algorithm when extracting keywords. The specific formula is as follows: Among them, w1 and w2 represent two words, w3 represents the third word, and α and β are two weight parameters.

Images