User note identification system based on network big data
Through the user notes recognition system based on network big data, keywords, cleaning and preprocessing data are automatically extracted, and note content is recognized in combination with algorithms, the problems of inaccurate keyword extraction and insufficient data processing capabilities in the existing system are solved, and efficient , accurate note identification and analysis services.
Patent Information
- Application Number
- CN202411344635.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-25
- Publication Date
- 2024-12-20
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing user note recognition system has problems such as inaccurate keyword extraction, insufficient data processing capabilities, and insufficient recognition algorithms. It is difficult to effectively reflect the core content and intentions of user notes, and it is impossible to cope with large-scale complex data sets.
A user note recognition system based on network big data was designed, including a note keyword extraction module, a network data acquisition and storage module, a data processing module, a feature extraction module, a note recognition module and a data analysis and mining module, and automatically extracts keywords. , clean and preprocess data, extract feature information, combine algorithms to identify notes, and conduct in-depth analysis.
It improves the efficiency and accuracy of information extraction, ensures the comprehensiveness and diversity of data, significantly improves data quality, and provides efficient, accurate and comprehensive note recognition and analysis services.
Smart Images
Figure CN119166751A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of natural language processing, and in particular to a user note recognition system based on network big data. Background Art
[0002] With the rapid development of the Internet, network big data has become an important resource for all walks of life. The text data such as notes, comments, and articles generated by users on the Internet contain rich information and knowledge, which is of great significance for understanding user needs, analyzing market trends, and optimizing product design. However, facing the massive amount of network data, how to efficiently extract, process, analyze and utilize this information has become an urgent problem to be solved. The traditional user note recognition method mainly relies on manual reading and analysis. This method is not only inefficient, but also easily affected by subjective factors, resulting in inaccurate and unstable recognition results. In addition, with the continuous increase in data volume, manual processing methods can no longer meet actual needs. With the continuous advancement of natural language processing technology, methods based on big data and machine learning provide new ideas for user note recognition. Most of the existing user note recognition systems have the following problems: First, the keyword extraction is inaccurate and cannot effectively reflect the core content of user notes; second, the data processing capacity is insufficient and it is difficult to cope with large-scale and complex data sets; third, the recognition algorithm is not smart enough and cannot accurately recognize the intention and emotion of user notes. In this regard, we propose a user note recognition system based on network big data. Summary of the invention
[0003] In order to solve the above technical problems, a user note recognition system based on network big data is provided. This technical solution solves the above problems.
[0004] In order to achieve the above purpose, the technical solution adopted by the present invention is:
[0005] User note recognition system based on network big data, the user note recognition system includes:
[0006] Note keyword extraction module, which is used to extract keywords from the provided user notes;
[0007] A network data collection and storage module, the network data collection and storage module is electrically connected to the note keyword extraction module, and the network data collection and storage module is used to collect text data from a variety of network sources according to the extracted keywords, and store the collected data;
[0008] A data processing module, the data processing module is electrically connected to the network data acquisition and storage module, and the data processing module is used to clean and pre-process the collected data;
[0009] A feature extraction module, the feature extraction module is electrically connected to the data processing module, and the feature extraction module is used to extract key feature information from the processed data;
[0010] A note recognition module, the note recognition module is electrically connected to the feature extraction module, and the note recognition module is used to recognize the content of the user's note provided according to the extracted feature information in combination with an algorithm;
[0011] A data analysis and mining module is electrically connected to the note recognition module, and is used to analyze the recognized user notes to obtain valid information therein.
[0012] Preferably, the note keyword extraction module includes:
[0013] A text preprocessing unit, which is used to process the input user notes, including: removing HTML tags, removing punctuation marks, removing stop words, and performing text segmentation;
[0014] A keyword candidate generation unit, which is used to generate a keyword candidate set by counting word frequencies based on the result of text preprocessing;
[0015] The keyword screening and sorting unit is used to screen out the final keywords from the candidate keywords through an algorithm and sort the keywords;
[0016] The keyword output unit is used to output the finally determined keywords to the network data collection and storage module.
[0017] Preferably, the network data collection and storage module includes:
[0018] A web crawler unit, which is responsible for automatically searching and crawling relevant text data on the Internet according to keywords;
[0019] Data filtering and deduplication unit: The data filtering and deduplication unit is used to perform preliminary filtering on the crawled data to remove duplicate data, irrelevant data or low-quality data;
[0020] A data storage unit, the data storage unit is used to store the filtered data in a storage medium and a cloud;
[0021] The interface and interaction unit is used to provide data interaction and sharing through the data access interface.
[0022] Preferably, the data processing module includes:
[0023] A data receiving unit, the data receiving unit is used to receive data transmitted by the network data acquisition and storage module;
[0024] A data cleaning unit, the data cleaning unit is used to perform data cleaning processing on the received data, the data cleaning processing includes: removing noise, correcting errors, and processing missing values;
[0025] The data standardization unit is used to perform standardization processing on the cleaned data, and the standardization processing includes: unifying the data format, converting the data type, and normalizing the data range.
[0026] Preferably, the feature extraction module includes:
[0027] A text feature extraction unit, which is used to extract language features from text data and convert the text data into a structured feature vector;
[0028] Feature selection and optimization unit: The feature selection and optimization unit is used to select the most important features through feature selection algorithms to reduce feature dimensions, and optimize feature sets to remove redundant and noisy features;
[0029] Feature conversion and encoding unit,The feature conversion and encoding unit is used to convert and encode the extracted features.
[0030] Preferably, the note recognition module includes:
[0031] Model loading unit, the model selection and loading unit is used to load the pre-trained note recognition model parameters;
[0032] A feature input unit, the feature input unit is used to receive feature information from the feature extraction module;
[0033] The recognition execution unit is used to perform recognition processing on the input user note features using the loaded model, and the recognition processing includes: classification, clustering, and labeling;
[0034] The result output unit and the result output and interpretation unit are used to output the recognition results to the user and subsequent modules.
[0035] Preferably, the data analysis and mining module:
[0036] Statistical analysis unit: The statistical analysis unit is used to use statistical methods to conduct descriptive statistics and inferential statistics on data;
[0037] A text analysis and mining unit, which is used to analyze text-type user note data using natural language processing technology, including sentiment analysis, topic extraction, keyword clustering, and entity recognition;
[0038] The association analysis and pattern recognition unit is used to analyze the association relationship between user note data and identify potential patterns and trends in the data;
[0039] The visualization display unit is used to visualize the results of data analysis and mining in the form of charts.
[0040] Preferably, the method of filtering keywords by algorithm and sorting them is:
[0041] Based on the candidate keyword set, the TF-IDF value of each word is calculated, where the TF-IDF value calculation method is:
[0042] TF-IDF(t,d)=TF(t,d)×IDF(t)
[0043] In the formula, TF represents term frequency, IDF represents inverse document frequency, t represents word, d represents document, TF-IDF(t,d) represents the TF-IDF value of term t in document d, TF(t,d) represents the term frequency of term t in document d, and IDF(t) represents the inverse document frequency of term t.
[0044] Sort in descending order according to TF-IDF values, and remove phrases below the threshold from the set by setting a threshold.
[0045] Preferably, the word frequency and inverse document frequency are calculated as follows:
[0046] The calculation method of word frequency is:
[0047]
[0048] Where N t,d N, L represents the number of times word t appears in document d d Represents the total number of words in document d;
[0049] The calculation method of inverse document frequency is:
[0050]
[0051] In the formula, log() represents the log function, I represents the total number of documents, and I t Represents the number of documents containing word t.
[0052] Preferably, the method for the note recognition module to recognize and process the input user note features is:
[0053] Based on the acquired feature data, the data is divided into a training set and a test set;
[0054] Based on RNN, the model is trained using the training set data, and the model performance is evaluated using the test set data. After the training is completed, the note recognition model is obtained;
[0055] Save the trained model parameters to a file and provide the model loading function in the note recognition module.
[0056] Compared with the prior art, the present invention has the following beneficial effects:
[0057] The user note recognition system proposed in the present invention uses a note keyword extraction module, and the system can automatically extract keywords from user notes, thereby quickly locating the core content that users care about, improving the efficiency and accuracy of information extraction; the network data acquisition and storage module can collect text data from a variety of network sources based on the extracted keywords, which not only enriches the data sources but also ensures the comprehensiveness and diversity of the data; the data processing module cleans and preprocesses the collected data, removes noise, corrects errors, processes missing values, and performs standardization, significantly improving the quality and availability of the data; the data analysis and mining module performs in-depth analysis of the identified user notes, providing users with more efficient, accurate, and comprehensive note recognition and analysis services. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] Figure 1 This is the framework diagram of the user note recognition system based on network big data;
[0059] Figure 2 This is the structure diagram of the note keyword extraction module;
[0060] Figure 3 It is the structural diagram of the network data acquisition and storage module;
[0061] Figure 4 It is the structure diagram of data processing module;
[0062] Figure 5 This is the structural diagram of the feature extraction module;
[0063] Figure 6 This is the structural diagram of the note recognition module;
[0064] Figure 7 This is the structure diagram of the data analysis and mining module. DETAILED DESCRIPTION
[0065] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art may think of other obvious variations.
[0066] Reference Figure 1As shown, the user note recognition system based on network big data includes:
[0067] Through the collaborative work of multiple modules, intelligent recognition and analysis of user notes are realized. The note keyword extraction module extracts keywords from user notes; the network data acquisition and storage module collects and stores relevant data from network sources based on these keywords; the data processing module cleans and preprocesses the collected data; the feature extraction module further extracts key features from the processed data; the note recognition module uses these features combined with algorithms to identify the content of user notes; the data analysis and mining module conducts in-depth analysis of the recognition results and mines valuable information. The entire system aims to provide users with personalized note recognition and analysis services through big data and intelligent technology.
[0068] Reference Figure 2 As shown, the note keyword extraction module includes:
[0069] The text preprocessing unit is used to process the input user notes, including removing HTML tags, punctuation marks, stop words, and performing text segmentation, aiming to purify the text content and reduce noise, thus laying a foundation for subsequent word frequency statistics and keyword extraction;
[0070] A keyword candidate generation unit, which is used to generate a keyword candidate set by counting word frequencies based on the result of text preprocessing;
[0071] Keyword screening and ranking unit: The keyword screening and ranking unit is used to select the final keywords from the candidate keywords through an algorithm and to rank the keywords, which involves scoring the words using the TF-IDF algorithm to assess their importance and relevance to the text topic;
[0072] The keyword output unit is used to output the finally determined keywords to the network data collection and storage module.
[0073] Reference Figure 3 As shown, the network data acquisition and storage module includes:
[0074] The network data acquisition and storage module completes the entire process from Internet crawling, filtering, storage to providing data access through the collaborative work of four units. The web crawler unit is responsible for searching and crawling relevant data; the data filtering and deduplication unit performs preliminary screening and purification on the crawled data; the data storage unit stores the processed data safely and reliably; the interface and interaction unit provides a data access interface to support data interaction and sharing. This module ensures the comprehensiveness, accuracy, security and ease of use of the data.
[0075] Reference Figure 4As shown, the data processing module includes:
[0076] The data processing module is responsible for receiving data from the network data acquisition and storage module, removing noise, correcting errors, and processing missing values through the data cleaning unit, and then using the data standardization unit to unify the data format, convert data types, and normalize the data range to ensure the cleanliness, consistency, and comparability of the data, providing a high-quality data foundation for subsequent analysis and processing.
[0077] Reference Figure 5 As shown, the feature extraction module includes:
[0078] The text feature extraction unit uses a variety of technologies to mine language features from text data and convert unstructured text data into structured feature vectors. These feature vectors can reflect the essential information of the text content and provide a basis for subsequent analysis.
[0079] In order to improve the efficiency and performance of the model, the feature selection and optimization unit uses the feature selection algorithm to select the most important features and reduce the feature dimensions. At the same time, it also optimizes the feature set, removes redundant and noisy features, and ensures that the features input into the model are both effective and concise;
[0080] The feature conversion and encoding unit is responsible for performing necessary conversion and encoding on the features before inputting them into the machine learning or deep learning model. The conversion operations include normalization, standardization, and discretization to eliminate the dimensional differences and outlier effects between different features. The encoding operations involve one-hot encoding, label encoding, and binary encoding to convert category features into numerical forms that can be recognized by the model.
[0081] Reference Figure 6 As shown, the note recognition module includes:
[0082] The note recognition module has four main units: model loading unit, feature input unit, recognition execution unit and result output unit. The note recognition module realizes efficient and accurate recognition and analysis of user notes by loading pre-trained models, receiving feature inputs, performing recognition processing and outputting results to users and subsequent modules.
[0083] Reference Figure 7 As shown, data analysis and mining module:
[0084] Statistical analysis unit: The statistical analysis unit is used to use statistical methods to conduct descriptive statistics and inferential statistics on data;
[0085] A text analysis and mining unit, which is used to analyze text-type user note data using natural language processing technology, including sentiment analysis, topic extraction, keyword clustering, and entity recognition;
[0086] The association analysis and pattern recognition unit is used to analyze the association relationship between user note data and identify potential patterns and trends in the data;
[0087] The visualization display unit is used to visualize the results of data analysis and mining in the form of charts.
[0088] The method of filtering keywords through algorithms and sorting them is as follows:
[0089] Based on the candidate keyword set, the TF-IDF value of each word is calculated, where the TF-IDF value calculation method is:
[0090] TF-IDF(t,d)=TF(t,d)×IDF(t)
[0091] In the formula, TF represents term frequency, IDF represents inverse document frequency, t represents word, d represents document, TF-IDF(t,d) represents the TF-IDF value of term t in document d, TF(t,d) represents the term frequency of term t in document d, and IDF(t) represents the inverse document frequency of term t.
[0092] Sort in descending order according to TF-IDF values, and remove phrases below the threshold from the set by setting a threshold.
[0093] The word frequency and inverse document frequency are calculated as follows:
[0094] The word frequency is calculated as follows:
[0095]
[0096] Where N t,d N, L represents the number of times word t appears in document d d Represents the total number of words in document d;
[0097] The calculation method of inverse document frequency is:
[0098]
[0099] In the formula, log() represents the log function, I represents the total number of documents, and I t Represents the number of documents containing word t.
[0100] The method by which the note recognition module recognizes and processes the input user note features is as follows:
[0101] Based on the acquired feature data, the data is divided into a training set and a test set;
[0102] Based on RNN, the model is trained using the training set data, and the model performance is evaluated using the test set data. After the training is completed, the note recognition model is obtained;
[0103] Save the trained model parameters to a file and provide the model loading function in the note recognition module.
[0104] The use process of the present invention is:
[0105] The note keyword extraction module extracts keywords from user notes;
[0106] The network data collection and storage module collects and stores text data from various network sources based on keywords;
[0107] The data processing module cleans and preprocesses the collected data;
[0108] The feature extraction module extracts key feature information from the processed data;
[0109] The note recognition module recognizes the content of the user's notes based on the extracted feature information and the algorithm;
[0110] The data analysis and mining module analyzes the identified user notes to obtain effective information;
[0111] The system displays the analysis results to the user in the form of charts through a visual display unit.
[0112] To sum up, the advantages of the present invention are: by automatically extracting note keywords, the system improves the efficiency and accuracy of information extraction; the network data acquisition and storage module broadens the scope of data collection, ensuring the comprehensiveness and diversity of the data; the data processing module effectively cleans and preprocesses the data, improves the data quality; the feature extraction module accurately extracts key features and optimizes them to provide high-quality data for identification; the note recognition module uses pre-trained models to improve recognition accuracy and efficiency; in addition, the system deepens data analysis and mining, provides comprehensive insights, and makes the results intuitive and easy to understand through visual display, providing users with efficient, accurate and comprehensive note recognition and analysis services.
[0113] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.
Claims
1. A user note recognition system based on network big data, characterized by: The user note recognition system includes: Note keyword extraction module, which is used to extract keywords from the provided user notes; A network data collection and storage module, the network data collection and storage module is electrically connected to the note keyword extraction module, and the network data collection and storage module is used to collect text data from a variety of network sources according to the extracted keywords, and store the collected data; A data processing module, the data processing module is electrically connected to the network data acquisition and storage module, and the data processing module is used to clean and pre-process the collected data; A feature extraction module, the feature extraction module is electrically connected to the data processing module, and the feature extraction module is used to extract key feature information from the processed data; A note recognition module, the note recognition module is electrically connected to the feature extraction module, and the note recognition module is used to recognize the content of the user's note provided according to the extracted feature information in combination with an algorithm; A data analysis and mining module is electrically connected to the note recognition module, and is used to analyze the recognized user notes to obtain valid information therein.
2. The user note recognition system based on network big data according to claim 1 is characterized in that: The note keyword extraction module includes: A text preprocessing unit, which is used to process the input user notes, including: removing HTML tags, removing punctuation marks, removing stop words, and performing text segmentation; A keyword candidate generation unit, which is used to generate a keyword candidate set by counting word frequencies based on the result of text preprocessing; The keyword screening and sorting unit is used to screen out the final keywords from the candidate keywords through an algorithm and sort the keywords; The keyword output unit is used to output the finally determined keywords to the network data collection and storage module.
3. The user note recognition system based on network big data according to claim 1 is characterized in that: The network data collection and storage module includes: A web crawler unit, which is responsible for automatically searching and crawling relevant text data on the Internet according to keywords; Data filtering and deduplication unit: The data filtering and deduplication unit is used to perform preliminary filtering on the crawled data to remove duplicate data, irrelevant data or low-quality data; A data storage unit, the data storage unit is used to store the filtered data in a storage medium and a cloud; The interface and interaction unit is used to provide data interaction and sharing through the data access interface.
4. The user note recognition system based on network big data according to claim 1 is characterized in that: The data processing module includes: A data receiving unit, the data receiving unit is used to receive data transmitted by the network data acquisition and storage module; A data cleaning unit, the data cleaning unit is used to perform data cleaning processing on the received data, the data cleaning processing includes: removing noise, correcting errors, and processing missing values; The data standardization unit is used to perform standardization processing on the cleaned data, and the standardization processing includes: unifying the data format, converting the data type, and normalizing the data range.
5. The user note recognition system based on network big data according to claim 1 is characterized in that: The feature extraction module includes: A text feature extraction unit, which is used to extract language features from text data and convert the text data into a structured feature vector; Feature selection and optimization unit: The feature selection and optimization unit is used to select the most important features through feature selection algorithms to reduce feature dimensions, and optimize feature sets to remove redundant and noisy features; Feature conversion and encoding unit,The feature conversion and encoding unit is used to convert and encode the extracted features.
6. The user note recognition system based on network big data according to claim 1 is characterized in that: The note recognition module includes: Model loading unit, the model selection and loading unit is used to load the pre-trained note recognition model parameters; A feature input unit, the feature input unit is used to receive feature information from the feature extraction module; The recognition execution unit is used to perform recognition processing on the input user note features using the loaded model, and the recognition processing includes: classification, clustering, and labeling; The result output unit and the result output and interpretation unit are used to output the recognition results to the user and subsequent modules.
7. The user note recognition system based on network big data according to claim 1 is characterized in that: Data analysis and mining module: Statistical analysis unit: The statistical analysis unit is used to use statistical methods to conduct descriptive statistics and inferential statistics on data; A text analysis and mining unit, which is used to analyze text-type user note data using natural language processing technology, including sentiment analysis, topic extraction, keyword clustering, and entity recognition; The association analysis and pattern recognition unit is used to analyze the association relationship between user note data and identify potential patterns and trends in the data; The visualization display unit is used to visualize the results of data analysis and mining in the form of charts.
8. The user note recognition system based on network big data according to claim 2 is characterized in that: The method of filtering keywords through algorithms and sorting them is as follows: Based on the candidate keyword set, the TF-IDF value of each word is calculated, where the TF-IDF value calculation method is: TF-IDF(t,d)=TF(t,d)×IDF(t) In the formula, TF represents term frequency, IDF represents inverse document frequency, t represents word, d represents document, TF-IDF(t,d) represents the TF-IDF value of term t in document d, TF(t,d) represents the term frequency of term t in document d, and IDF(t) represents the inverse document frequency of term t. Sort in descending order according to TF-IDF values, and remove phrases below the threshold from the set by setting a threshold.
9. The user note recognition system based on network big data according to claim 8, characterized in that: The word frequency and inverse document frequency are calculated as follows: The calculation method of word frequency is: Where N t,d N, L represents the number of times word t appears in document d d Represents the total number of words in document d; The calculation method of inverse document frequency is: In the formula, log() represents the log function, I represents the total number of documents, and I t Represents the number of documents containing word t.
10. The user note recognition system based on network big data according to claim 1, characterized in that: The method by which the note recognition module recognizes and processes the input user note features is as follows: Based on the acquired feature data, the data is divided into a training set and a test set; Based on RNN, the model is trained using the training set data, and the model performance is evaluated using the test set data. After the training is completed, the note recognition model is obtained; Save the trained model parameters to a file and provide the model loading function in the note recognition module.