An intelligent method for automatic acquisition and classification of unstructured data
By using automated tools and algorithms to collect and classify unstructured data, the problem of unstructured data being difficult to process is solved, and efficient and intelligent data management is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- FUZHOU UNIV
- Filing Date
- 2026-03-06
- Publication Date
- 2026-06-05
Smart Images
Figure CN122153587A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing technology, specifically to an intelligent method for automatic collection and classification of unstructured data. Background Technology
[0002] Unstructured data includes text documents, emails, audio, video, social media posts, etc. Unstructured data is difficult to store and manage using traditional database tables because it lacks a clear structure and standardized format. Furthermore, unstructured data often contains rich information, but it is also more difficult to analyze and utilize. It lacks a predefined data model, has an irregular or incomplete structure, and is difficult to represent using two-dimensional logical tables in relational databases. It does not follow fixed format specifications and cannot be directly processed by traditional structured query languages. It is one of the most important data types in the era of big data. Currently, in the process of processing unstructured data, since the types of unstructured data include but are not limited to text, audio, images or videos, these unstructured data are often difficult to process directly or effectively analyze due to the uncertainty of their format and structure, resulting in the data not being used effectively. Furthermore, traditional rule set-based data collection and classification methods are also unable to effectively collect and classify such unstructured data, thereby increasing the cost of data management. Summary of the Invention
[0003] This invention provides an intelligent method for automatic collection and classification of unstructured data, which can effectively solve the problem mentioned in the background art that in the current process of processing unstructured data, since the types of unstructured data include but are not limited to text, audio, images or videos, these unstructured data are often difficult to process or analyze effectively due to the uncertainty of their format and structure, resulting in the ineffective use of data. Furthermore, traditional rule set-based data collection and classification methods are also unable to effectively collect and classify such unstructured data, thereby increasing the cost of data management.
[0004] To achieve the above objectives, the present invention provides the following technical solution: an intelligent method for automatic collection and classification of unstructured data, which uses automated tools and algorithms to automatically collect and classify unstructured data, thereby achieving automated data collection and intelligent data classification, including the following steps: Step 1: Automatically identify external public web pages, social media data sources, and internal business systems to determine whether the data belongs to the type of text, image, or audio, and obtain the corresponding legal access permissions based on access control policies; Step 2: Automated data collection is performed through web crawlers, API calls, or direct database connections. The collection frequency is dynamically adjusted based on the data source update characteristics. During the collection phase, the data is initially parsed, deduplicated, and noise filtered. Step 3: Perform data cleaning, format standardization, anomaly detection and missing value repair, and segment and filter the text. Perform encoding unification and necessary feature enhancement on images and audio to construct high-quality input data. Step 4: Generate learnable vector representations from deep feature extraction methods for different modalities, and build a stable classification model through model training, parameter optimization and regularization strategies to improve the model's generalization ability under large-scale heterogeneous data; Step 5: Perform prediction on the input data and output the classification results and confidence scores. If the confidence scores are insufficient, trigger manual review. At the same time, combine clustering algorithms to automatically identify potential data structures, thereby achieving intelligent data classification and management.
[0005] According to the above technical solution, step one prepares for subsequent automated data collection. When identifying data sources, all potential data sources are identified by using network scanning tools, domain name resolution, and API documentation analysis. Specifically, this includes identifying and determining external and internal data sources.
[0006] According to the above technical solution, step one confirms the specific data type based on the identified data source, which facilitates the adoption of different collection and processing methods in the future. The data types specifically include text, image, audio and video. When obtaining permissions, it is necessary to communicate and negotiate with the owner of the data source to obtain access rights. At the same time, it is necessary to clarify the scope of data use and compliance requirements, and establish a data exchange protocol to clarify the transmission format, encoding standards and interface specifications.
[0007] According to the above technical solution, in step two, target data is collected from the identified and confirmed data sources using an automated data acquisition tool; Before collecting target data, a data collection strategy needs to be designed. The collection strategy includes the collection frequency and the collection range. The collection frequency is set according to the data update speed to determine the collection cycle, and the collection range is the boundary that needs to be clearly defined. When choosing automated data collection tools, specific options include web crawlers, API calls, and direct database connections.
[0008] According to the above technical solution, in step two, when acquiring data through automated data acquisition tools, a collection script is written or configured to set filtering conditions in order to achieve data capture, parsing and preliminary noise reduction. After the data is acquired, it is stored in a database or a distributed file system to ensure data security and accessibility. Specifically, the database is MySQL and the distributed file system is HDFS.
[0009] According to the above technical solution, step three involves processing the collected raw data to convert it into a standardized format that is easy to analyze, specifically including data cleaning and format conversion. Data cleaning includes removing HTML tags, advertising code, special characters, invalid records, and duplicate data, as well as filling in missing values according to business logic, identifying and handling outliers that are obviously illogical, and format conversion specifically includes unifying text format, resampling images and audio, and performing natural language processing operations on the text.
[0010] According to the above technical solution, after data processing, step three also requires data annotation to provide labeled data for subsequent model training. The annotation process includes manual annotation and semi-automatic annotation. Manual annotation involves building an annotation platform to provide clear annotation guidelines and labeling the data. Semi-automatic annotation uses existing business rules, dictionaries, or basic models to generate preliminary labels, which are then manually verified.
[0011] According to the above technical solution, in step four, when extracting features, a deep learning method is specifically used to extract features from text data and media data, so as to convert unstructured raw data into structured numerical vectors that computers can understand. Media data includes image data and audio data. When extracting features from text data, using pre-trained models to generate word vectors and sentence vectors can better capture semantic information. When extracting features from image data, the intermediate layer output of a pre-trained convolutional neural network is used as the feature vector. When extracting features from audio data, the audio data is converted into a spectrogram and then treated as an image for feature extraction.
[0012] According to the above technical solution, in step four, when constructing and training the classification model, a suitable classification algorithm is selected based on the task and data type. For text classification models, a deep learning model can be selected; for image classification models, the mainstream convolutional neural network (CNN) can be selected; and for audio classification models, RNN or LSTM can be selected.
[0013] According to the above technical solution, when the classification is achieved through model application in step five, new, unclassified unstructured data is specifically input into the trained classification model after undergoing the same preprocessing and feature extraction process to achieve automatic classification of the new unstructured data, and the model outputs the category label and its corresponding confidence score.
[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. By clearly defining the data source, we can prepare for subsequent automated data collection. When identifying the data source, by identifying all potential data sources, it is convenient to use automated data collection tools to accurately collect target data from the clearly defined data sources, while ensuring the integrity and consistency of the collected data. Processing improves data quality, eliminates noise interference, and facilitates the conversion of the collected raw data into a standardized format that is easy to analyze. Extracting features from the data and building and training classification models facilitates the use of deep learning methods to extract features from text and media data. Furthermore, when using the model to implement classification, it is convenient to automatically classify new unstructured data and output category labels.
[0015] 2. By automatically collecting and processing unstructured data from various sources and classifying the data through intelligent algorithms, it facilitates the automatic collection and intelligent management of unstructured data. It realizes the entire process from data source identification and collection to data cleaning, feature extraction, model training, and classification. By introducing automated tools and intelligent algorithms, it significantly improves the efficiency and accuracy of data processing, while taking into account data security and compliance. Compared with traditional rule set-based data collection and classification methods, it has a higher level of automation and intelligence, as well as stronger data processing capabilities, and can better handle complex and volatile data environments. Attached Figure Description
[0016] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.
[0017] In the attached diagram: Figure 1 This is a flowchart of the steps of the data acquisition and classification method of the present invention. Detailed Implementation
[0018] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.
[0019] Example: Figure 1 As shown, this invention provides a technical solution: an intelligent method for automatic collection and classification of unstructured data. Through automated tools and algorithms, it automatically collects and classifies unstructured data to achieve automated data collection and intelligent data classification, thereby improving the efficiency and accuracy of data processing. The method includes the following steps: Step 1: Clarify the data source, confirm the data type, and access permissions; Step two: Efficiently and compliantly obtain target data from the data source; Step 3: Improve data quality and eliminate noise interference through processing; Step four: Extract features from the data and build and train a classification model; Step 5: Apply the model to automatically classify and cluster the new data.
[0020] Based on the above technical solution, step one mainly prepares for subsequent automated data collection. When identifying data sources, network scanning tools, domain name resolution, and API documentation analysis are used to identify all potential data sources, specifically including identifying and determining external and internal data sources. External data sources include public web pages, social media, and public databases. Public web pages include news portals, blogs, Baidu Tieba, and Zhihu, which are obtained through web crawlers. Social media includes Weibo, WeChat official accounts, Douyin, Twitter, and Facebook. Some provide official APIs, while others require web crawling. Public databases refer to government open data and academic datasets. Internal data sources include internal enterprise systems, file servers, and email. Internal enterprise systems include content management systems (CMS), customer relationship management systems (CRM), and enterprise resource planning systems (ERP), containing a large amount of customer feedback, product descriptions, and support tickets. The file servers store a large number of Word documents, PDF reports, and PPT presentations. Emails include internal business communications and external customer inquiries.
[0021] Based on the above technical solution, the first step is to confirm the specific data type based on the identified data source, so as to facilitate the adoption of different collection and processing methods in the future. The specific data types include text, image, audio and video. Text categories include HTML pages, JSON data streams, plain text files, and PDF documents; image categories include JPEG and PNG images; audio categories include customer service call recordings, meeting minutes, and podcasts; and video categories include surveillance footage, promotional videos, and online courses. When obtaining permissions, it is necessary to communicate and negotiate with the owner of the data source to obtain access rights. At the same time, it is necessary to clarify the scope of data use and compliance requirements, and establish a data exchange protocol to clarify the transmission format, encoding standards and interface specifications.
[0022] Based on the above technical solution, in step two, target data is collected from the identified and confirmed data sources using automated data collection tools, while ensuring the integrity and consistency of the collected data. Before collecting target data, a data collection strategy needs to be designed. The collection strategy includes collection frequency and collection scope. The collection frequency is set according to the data update speed. News websites need to be crawled on a minute-by-minute basis, while enterprise document libraries only need to be synchronized on a daily and weekly basis. The collection scope is the boundary that needs to be clearly defined. Specifically, it includes: URLs of specific columns, posts that match specific keywords, and data after a certain point in time. When choosing automated data collection tools, specific options include web crawlers, API calls, and direct database connections. Web crawlers are used to crawl public web pages, and specific tools include Scrapy, Selenium, and Apache Nutch. Scrapy is a high-performance framework, Selenium handles dynamic JavaScript rendering of pages, and Apache Nutch is a large-scale distributed crawler. API calls are the preferred method due to their stability and higher level of structure. Specifically, the requests library is used to make HTTP requests, handle authentication and rate limiting, and direct database connections are used for internal databases, using JDBC to directly query them.
[0023] Based on the above technical solution, in step two, when acquiring data through automated data acquisition tools, a collection script is written to set filtering conditions in order to achieve data capture, parsing and preliminary noise reduction. The filtering conditions include keywords and time ranges, and the parsing process includes HTML, JSON and XML. After the data is acquired, it is stored in a database to ensure data security and accessibility. The database in question is MySQL.
[0024] Based on the above technical solution, step three involves processing the collected raw data to convert it into a standardized format that is easy to analyze, specifically including data cleaning and format conversion. Data cleaning includes removing HTML tags, advertising code, special characters, invalid records, and duplicate data, as well as filling in missing values according to business logic, identifying and handling obviously illogical outliers, and format conversion specifically includes unifying text format, resampling images and audio, and performing natural language processing operations on the text. Unifying text format includes converting to lowercase and removing spaces, while natural language processing operations include word segmentation, stop word removal, stemming, and lemmatization.
[0025] Based on the above technical solution, in step three, after data processing, it is also necessary to label the data to provide labeled data for subsequent model training. The labeling process includes manual labeling and semi-automatic labeling. Manual labeling involves building a labeling platform to provide clear labeling guidelines and labeling the data to ensure labeling consistency. Semi-automatic labeling uses existing business rules to generate preliminary labels, which are then manually checked to significantly improve labeling efficiency.
[0026] Based on the above technical solution, in step four, when extracting features, a deep learning method is specifically used to extract features from text data and media data, so as to convert unstructured raw data into structured numerical vectors that computers can understand. Media data includes image data and audio data. In the process of multimodal feature fusion, in order to achieve a unified semantic representation of different types of data, this invention adopts a nonlinear joint embedding mapping method to integrate the primary features of text, image, and audio modalities. Projecting onto a unified feature space, the mapping function is expressed as follows: in, , , σ(⋅) is a cross-modal mapping matrix, and σ(⋅) is a nonlinear activation function. This mapping enables different modal data to be fused and represented in a unified feature space, thereby improving the model's generalization ability under heterogeneous data.
[0027] When extracting features from text data, using a pre-trained model to generate word vectors and sentence vectors can better capture semantic information. The pre-trained model is BERT. When extracting features from image data, the output of the intermediate layer of a pre-trained convolutional neural network (CNN) is used as the feature vector. The CNN model is ResNet. When extracting features from audio data, the audio data is converted into a spectrogram and then treated as an image for feature extraction. The spectrogram is specifically a spectrogram.
[0028] Based on the above technical solution, in step four, when building and training the classification model, the appropriate classification algorithm is selected according to the task and data type. For text classification models, deep learning models can be selected. Deep learning models are suitable for scenarios with large amounts of data and complex features, including RNN, LSTM, and Transformer. For image classification models, the mainstream convolutional neural network CNN can be selected. For audio classification models, LSTM can be selected. After selecting a model, labeled data is used to train the model, and cross-validation and grid search are used to fine-tune the model parameters to minimize prediction error. Finally, the model's performance is evaluated by setting metrics, including precision, recall, and F1 score. Precision refers to the proportion of correct classifications overall, recall measures the accuracy and completeness of predictions for each class, and F1 score is the harmonic mean of precision and recall, which is a comprehensive metric.
[0029] To further enhance the discriminative power of multimodal features, an attention mechanism is introduced in the fusion stage to adaptively weight the importance of different modalities. The formula for calculating the attention weights is as follows: Where v is the query vector, For feature weighting mapping matrix, These are the dynamic weight coefficients for each modality. Through this attention mechanism, the model can automatically learn the key features to focus on in different task scenarios, achieving optimal information fusion.
[0030] Based on the above technical solution, in step five, when classifying through model application, new, unclassified unstructured data is specifically input into the trained classification model after undergoing the same preprocessing and feature extraction process to automatically classify the new unstructured data, and the model outputs the category label and its corresponding confidence score. At the same time, a confidence threshold is set. Only results with a confidence level higher than this threshold are directly adopted, while results with a confidence level lower than the threshold need to be transferred to the manual review queue and reviewed manually. In cluster analysis, unsupervised learning methods are mainly used to group data and discover potential structures. The specific unsupervised learning method is K-Means. In addition, it is necessary to review the classification results to ensure their accuracy, and to regularly check the classification results to identify misclassified samples and analyze the reasons for the errors. At the same time, user feedback should be collected to establish a feedback loop mechanism.
[0031] To improve the stability and self-learning ability of classification, this invention introduces an adaptive confidence threshold function during the classification stage to dynamically adjust the review strategy. The specific calculation formula is as follows: in, λ is the initial threshold, λ is the adjustment coefficient, and Var(⋅) represents the variance of the prediction distribution. When the confidence level of the model prediction fluctuates greatly, the threshold is automatically increased to prevent misclassification, thereby achieving a dynamic balance between classification accuracy and manual review efficiency.
[0032] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. An intelligent method for automatic collection and classification of unstructured data, characterized in that: Automated tools and algorithms are used to automatically collect and classify unstructured data, achieving automated data collection and intelligent data classification. This includes the following steps: Step 1: Automatically identify external public web pages, social media data sources, and internal business systems to determine whether the data belongs to the type of text, image, or audio, and obtain the corresponding legal access permissions based on access control policies; Step 2: Automated data collection is performed through web crawlers, API calls, or direct database connections. The collection frequency is dynamically adjusted based on the data source update characteristics. During the collection phase, the data is initially parsed, deduplicated, and noise filtered. Step 3: Perform data cleaning, format standardization, anomaly detection and missing value repair, and segment and filter the text. Perform encoding unification and necessary feature enhancement on images and audio to construct high-quality input data. Step 4: Generate learnable vector representations from deep feature extraction methods for different modalities, and build a stable classification model through model training, parameter optimization and regularization strategies to improve the model's generalization ability under large-scale heterogeneous data; Step 5: Perform prediction on the input data and output the classification results and confidence scores. If the confidence scores are insufficient, trigger manual review. At the same time, combine clustering algorithms to automatically identify potential data structures, thereby achieving intelligent data classification and management.
2. The intelligent method for automatic acquisition and classification of unstructured data according to claim 1, characterized in that: Step one prepares for subsequent automated data collection. When identifying data sources, network scanning tools, domain name resolution, and API documentation analysis are used to identify all potential data sources, specifically including the identification and determination of external and internal data sources.
3. The intelligent method for automatic acquisition and classification of unstructured data according to claim 2, characterized in that: Step one involves identifying the specific data type based on the identified data source, which facilitates the adoption of different collection and processing methods in the future. The specific data types include text, image, audio, and video. When obtaining permissions, it is necessary to communicate and negotiate with the owner of the data source to obtain access rights. At the same time, it is necessary to clarify the scope of data use and compliance requirements, and establish a data exchange protocol to clarify the transmission format, encoding standards and interface specifications.
4. The intelligent method for automatic acquisition and classification of unstructured data according to claim 2, characterized in that: In step two, target data is collected from the identified and confirmed data sources using automated data acquisition tools; Before collecting target data, a data collection strategy needs to be designed. The collection strategy includes the collection frequency and the collection range. The collection frequency is set according to the data update speed to determine the collection cycle, and the collection range is the boundary that needs to be clearly defined. When choosing automated data collection tools, specific options include web crawlers, API calls, and direct database connections.
5. The intelligent method for automatic acquisition and classification of unstructured data according to claim 4, characterized in that: In step two, when acquiring data through automated data acquisition tools, the acquisition script is written or configured to set filtering conditions in order to achieve data capture, parsing and preliminary noise reduction. After the data is acquired, it is stored in a database or a distributed file system to ensure data security and accessibility. Specifically, the database is MySQL and the distributed file system is HDFS.
6. The intelligent method for automatic acquisition and classification of unstructured data according to claim 4, characterized in that: Step three involves processing the collected raw data to convert it into a standardized format that is easy to analyze, specifically including data cleaning and format conversion. Data cleaning includes removing HTML tags, advertising code, special characters, invalid records, and duplicate data, as well as filling in missing values according to business logic, identifying and handling outliers that are obviously illogical, and format conversion specifically includes unifying text format, resampling images and audio, and performing natural language processing operations on the text.
7. The intelligent method for automatic acquisition and classification of unstructured data according to claim 6, characterized in that: Step three, after data processing, also requires data annotation to provide labeled data for subsequent model training. The annotation process includes manual annotation and semi-automatic annotation. Manual annotation involves building an annotation platform to provide clear annotation guidelines and labeling the data. Semi-automatic annotation uses existing business rules, dictionaries, or basic models to generate preliminary labels, which are then manually verified.
8. The intelligent method for automatic acquisition and classification of unstructured data according to claim 6, characterized in that: In step four, when extracting features, deep learning methods are specifically used to extract features from text data and media data, so as to convert unstructured raw data into structured numerical vectors that computers can understand. Media data includes image data and audio data. When extracting features from text data, using pre-trained models to generate word vectors and sentence vectors can better capture semantic information. When extracting features from image data, the intermediate layer output of a pre-trained convolutional neural network is used as the feature vector. When extracting features from audio data, the audio data is converted into a spectrogram and then treated as an image for feature extraction.
9. The intelligent method for automatic acquisition and classification of unstructured data according to claim 8, characterized in that: In step four, when constructing and training the classification model, the appropriate classification algorithm is selected based on the task and data type. For text classification models, deep learning models can be selected; for image classification models, the mainstream convolutional neural network (CNN) can be selected; and for audio classification models, RNN or LSTM can be selected.
10. The intelligent method for automatic acquisition and classification of unstructured data according to claim 8, characterized in that: In step five, when classification is achieved through model application, new, unclassified unstructured data is processed through the same preprocessing and feature extraction process and then input into the trained classification model to automatically classify the new unstructured data. The model outputs category labels and their corresponding confidence scores.