A network data information intelligent extraction system

Abstract

The application relates to the technical field of network data extraction, and discloses a network data information intelligent extraction system which comprises a collection module and a data feature extraction system and is jointly constructed into a data storage library for screening and classification after gathering, calculating, filtering and extracting common features of file data sources, stream data and crawler data; the collection module is used for collecting network original data, and sending the original data to the data feature extraction system for processing after conversion, preprocessing and judgment; the matching similarity classification value comprises format data source classification, row and column matching data source classification, keyword data source matching classification and repetition rate matching classification. The application has the advantages that metadata can be intelligently extracted and classified, associated data can be classified in the same category, and the user can obtain complete and associated data requirements during retrieval.

Description

Technical Field

[0001] This invention relates to the field of network data extraction technology, specifically to an intelligent network data information extraction system. Background Technology

[0002] Network data can be broadly categorized into four characteristics: First, massive amounts of network data, measured in petabytes (PB) increments, with a large amount of stored content; second, high speed, requiring rapid acquisition and analysis to ensure information reaches as many people as possible within a short timeframe; third, diversity, with data sourced from various channels, including text, images, and videos, resulting in a wide variety of data types; and fourth, value, encompassing not only intrinsic informational value but also commercial value, including structured, semi-structured, and unstructured data from databases and user-generated data.

[0003] Online data is diverse, including news articles and user-generated files in various formats such as Word, DOC, and EXL. Due to this diversity, it's crucial to extract value from these different formats and content to obtain high-frequency data that meets the needs of a broad user base. This requires manually defined keywords for searching and filtering. Currently, online data extraction and retrieval rely on a single selection method, using reports to integrate and extract data for searching. However, the metadata is often overly complex, potentially leading to data duplication and crashes. Furthermore, during the current extraction and retrieval process, very little truly relevant data is retrieved, failing to meet actual user needs. Summary of the Invention

[0004] This invention provides an intelligent network data information extraction system, which facilitates the intelligent extraction and classification of metadata and merges related data into the same category, making it easier for users to obtain complete and relevant data during retrieval. It solves the problems mentioned in the background technology, such as the metadata being too complex, which may lead to data duplication and crashes, and the fact that in the current data extraction and retrieval process, very little actual related data can be retrieved, which cannot actually meet the needs of users.

[0005] This invention provides the following technical solution: a network data information intelligent extraction system, including a collection module and a data feature extraction system. The collection module and the data feature extraction system jointly construct a data storage repository, which is used for data storage and calculation, aggregation, calculation, filtering and extraction of common features of file data sources, streaming data and crawler data, and then screening and classification. The acquisition module is used to acquire raw network data, and after converting, preprocessing, and judging the raw data, it sends it to the data feature extraction system for further processing. The data feature extraction system is used to filter the raw data and / or converted data sent by the acquisition module, and extract common features according to the matching similarity value to improve intelligent classification and extraction. The matching similarity classification values ​​include format data source classification, row and column matching data source classification, keyword data source matching classification, and repetition rate matching classification.

[0006] As an optional solution to the intelligent network data information extraction system of the present invention, the data feature extraction system further includes a matching module and an extraction module; The matching module is used to match the original data with the preset template file format based on similarity, and to classify regions based on the matching similarity classification value. The extraction module is used to extract and select data by inputting tags, search terms, etc.

[0007] As an optional solution to the intelligent network data information extraction system of the present invention, the matching module includes a similarity classification unit and a conversion unit; The similarity classification unit is used to calculate the matching similarity classification value. The matching steps of the similarity classification unit are as follows: S1. Obtain the collected message data; S2. Recognize the format and content of the message data; S3. Extract common network traffic features from the identified message data; S4. Extract frequent words from the message rows and columns; S5. Calculate the number of occurrences of frequent words extracted from the message rows; S6. Obtain keywords, search terms, and frequency values; S7. Write the keywords and frequency values ​​into the message tag; S8. Link related data information.

[0008] As an optional solution of the intelligent network data information extraction system described in this invention, in step S5, when the frequency of frequent words is calculated, if the extracted frequent words are sensitive words, they are marked with a sensitive label. If the frequent words are related to place names, events, or personal names, they are classified according to person, event, or place, and a specific classification label is added.

[0009] As an optional solution of the intelligent network data information extraction system of the present invention, wherein: in step S1, the acquired messages are specifically subjected to format recognition and format classification, and the data feature extraction system further includes a format recognition unit and a format classification unit; The format recognition unit is used to receive data collected by the acquisition module and recognize data formats such as FPG / DOC / PDF / EXL / XSD; The format classification unit is used to classify and store the identified data and data with the same format into the database.

[0010] As an optional solution of the intelligent network data information extraction system of the present invention, the acquisition module includes a real-time database unit, a historical database unit, a monitoring unit, and multiple data acquisition ports; The real-time database unit is used to store real-time data collected from multiple data acquisition ports in real time. The historical database unit is used to automatically transfer unprocessed data collected in the real-time database unit more than 24 hours ago to the historical database unit for storage. The monitoring unit is used to monitor the data collected in real time, and to monitor the risks of data viruses and firewalls.

[0011] As an optional solution of the intelligent network data information extraction system described in this invention, the data collection port is used to collect browsing data, search data and interaction data of users from several IPs in / APP / webpage, and convert the browsing data, search data and interaction data into files in real time and save them as access files. The data acquisition port is configured with a user-defined template that includes at least one target table. Data acquired from the acquisition port is processed using this user-defined template to generate DOC / PDF / EXL files, etc.

[0012] As an optional solution of the intelligent network data information extraction system described in this invention, the browsing data is mainly used for consumer behavior analysis in the e-commerce field. The browsing data reflects each user's access steps, further characterizes the user's access path, and analyzes different pages and jump probabilities, etc. The search data mainly refers to the time-series data of keyword search frequency recorded by the search engine, which can reflect the interests, concerns, and intentions of hundreds of millions of users. The search represents the amplitude. The interaction data mainly consists of data from Weibo and social networking sites, reflecting users' preferences and emotional factors.

[0013] As an optional solution to the intelligent network data information extraction system described in this invention, the extraction module includes a full extraction unit and an incremental extraction unit. The full extraction unit is used to extract the full amount of data from business tables that are not very large each day. The incremental extraction unit is used to extract, label, and classify large amounts of data by time-segmenting.

[0014] As an optional solution of the intelligent network data information extraction system described in this invention, the extraction module further includes an extraction unit. The extraction unit allows users to retrieve multiple data sets, correlate and merge them to generate a new form, and then perform queries or further correlate and merge with other data based on the new form.

[0015] The present invention has the following beneficial effects: 1. This intelligent network data information extraction system collects network data in real time through an acquisition module, and then extracts common features from the obtained network data according to the matching similarity value through a data feature extraction system, thereby achieving intelligent extraction and reducing the situation of classification difficulties and system crashes caused by data duplication and excessive data volume; By classifying raw data into four categories—format data source classification, row and column matching data source classification, keyword data source matching classification, and repetition rate matching classification—the intelligence of common feature extraction is improved, and the duplication of raw data extraction is reduced.

[0016] 2. This intelligent network data information extraction system uses a matching module to match the original data with preset template file formats based on similarity and classify them by region. Then, an extraction module extracts the data using three methods: full extraction, incremental extraction, and extraction, generating a new form. Based on the new form, it queries and merges with other related data. When the merged data is related, it is easier to collect and store it in the same category, improving the intelligent extraction and classification effect after collection. When extracting related data, related data can be viewed at the same time, making the data presented to the user more relevant and complete.

[0017] 3. This intelligent network data information extraction system uses identification, feature extraction, keywords, and frequent words in steps S1-S8 of the similarity classification unit matching process to extract data. The keywords and frequency values ​​are written into the message tags, and the original data is associated with related information in the database. This facilitates better retrieval and extraction of data through keywords. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the system flow of the present invention.

[0019] Figure 2 This is a flowchart illustrating the matching steps of the similarity classification unit in this invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Example 1: Online data is diverse, including news articles and user-posted documents in various formats such as Word, DOC, and EXL. Due to this diversity, it's necessary to extract value from different formats and content to obtain high-frequency data that meets the needs of a broad user base. This requires manually defining keywords for searching and filtering. Currently, online data extraction and retrieval uses a single selection mode, integrating and extracting data through reports for retrieval. However, the metadata is often overly complex, potentially leading to data duplication and crashes. Furthermore, during the current data extraction and retrieval process, very little relevant data is retrieved, failing to truly meet user needs.

[0022] This invention provides the following technical solution: an intelligent network data information extraction system. Please refer to [link / reference]. Figure 1 - Figure 2 It includes a data acquisition module and a data feature extraction system. The data acquisition module and the data feature extraction system together form a data storage repository, which is used for data storage and computation, aggregation, calculation, filtering and extraction of common features of file data sources, streaming data and crawler data, and then screening and classification. The acquisition module is used to acquire raw network data, and after converting, preprocessing, and judging the raw data, it sends it to the data feature extraction system for further processing. The data feature extraction system is used to filter the raw data and / or converted data sent by the acquisition module, and extract common features according to the matching similarity value to improve intelligent classification and extraction. The matching similarity classification values ​​include format data source classification, row and column matching data source classification, keyword data source matching classification, and repetition rate matching classification.

[0023] Formatted data source classification is the intelligent classification of data in different formats; Row and column matching data source classification specifically refers to the import of batches of raw data. Based on row and column filtering rules, it supports filtering principles such as batch equal, batch not equal, and batch similar, and will generate data that only includes the specified columns. For example: The system supports Excel files in 2003 format. If it is in 2007 or a higher format, save it as an Excel 97-2003 workbook. Set the data tabs, set the row where the data header is located. The header can only be a simple structure at present. Set the data range, mainly the start row and start column. The end row and end column are optional. Column conversion is a small conversion task that converts the columns displayed as letters in Excel into number form. The keyword data source matching and classification specifically involves classifying batches of raw data by keywords. For example, for business types, relevant keywords are selected based on business, transaction, and product; for consultation types, service types, and other keywords, intelligent matching and classification are performed. The repetition rate matching classification involves obtaining format, type, and keywords after the first three classifications, then extracting repetitions, filtering out unrepresentative data, and selecting representative and valuable data for reference.

[0024] In this embodiment, the network data is collected in real time by the acquisition module, and then the data feature extraction system extracts common features from the obtained network data according to the matching similarity value, so as to realize intelligent extraction and reduce the situation of classification difficulties and system crashes caused by data duplication and excessive data volume. By classifying raw data into four categories—format data source classification, row and column matching data source classification, keyword data source matching classification, and repetition rate matching classification—the intelligence of common feature extraction is improved, and the duplication of raw data extraction is reduced.

[0025] Example 2 is an explanation based on Example 1. For details, please refer to [link / reference]. Figure 1 - Figure 2 The data feature extraction system further includes a matching module and an extraction module. The matching module is used to match the original data with the preset template file format based on similarity, and to classify regions based on the matching similarity classification value. The extraction module is used to extract and select data by inputting tags, search terms, etc.

[0026] Wherein: the extraction module includes a full extraction unit and an incremental extraction unit. The full extraction unit is used to extract the full amount of data from business tables that are not very large each day. The incremental extraction unit is used to extract, label, and classify large amounts of data by time-segmenting.

[0027] The extraction module further includes an extraction unit, which retrieves multiple data sets, correlates and merges them to generate a new form, and then performs queries or further correlates and merges them with other data based on the new form.

[0028] In this embodiment, the matching module matches the original data with the preset template file format to classify the regions based on similarity. Then, the extraction module extracts the data using three methods: full, incremental, and extraction, generating a new form. Based on the new form, it queries and merges with other related data. When the merged data is related, it is easier to collect and store it in the same category, improving the intelligent extraction and classification effect after collection. When extracting related data, related data can be viewed at the same time, making the data presented to the user more relevant and complete.

[0029] Example 3 is an explanation based on Example 2. For details, please refer to [link / reference]. Figure 1 - Figure 2 The matching module includes a similarity classification unit and a conversion unit. The similarity classification unit is used to calculate the matching similarity classification value. The matching steps of the similarity classification unit are as follows: S1. Obtain the collected message data; In step S1, the acquired messages are specifically subjected to format recognition and format classification. The data feature extraction system further includes a format recognition unit and a format classification unit. The format recognition unit is used to receive data collected by the acquisition module and recognize data formats such as FPG / DOC / PDF / EXL / XSD; The format classification unit is used to classify and store the identified data and data with the same format into the database.

[0030] S2. Recognize the format and content of the message data; S3. Extract common network traffic features from the identified message data; S4. Extract frequent words from the message rows and columns; S5. Calculate the number of occurrences of frequent words extracted from the message rows; In step S5, during the process of calculating the occurrence count of frequent words, if a frequent word is found to be a sensitive word, it is marked with a sensitive label. If the frequent word is related to place names, events, or personal names, it is classified according to person, event, or place, and a specific classification label is added.

[0031] S6. Obtain keywords, search terms, and frequency values; S7. Write the keywords and frequency values ​​into the message tag; S8. Link related data information.

[0032] In this embodiment, the specific matching steps are explained in detail. Through steps S1-S8, identification, feature extraction, keywords and frequent words are used for extraction. The keywords and frequency values ​​are written into the message tag, and the original data is associated with the relevant information in the database. This makes it easier to retrieve and extract data by keywords.

[0033] Example 4 is an explanation based on Example 1. For details, please refer to [link / reference]. Figure 1 - Figure 2 The acquisition module includes a real-time database unit, a historical database unit, a monitoring unit, and multiple data acquisition ports. The real-time database unit is used to store real-time data collected from multiple data acquisition ports in real time. The historical database unit is used to automatically transfer unprocessed data collected in the real-time database unit more than 24 hours ago to the historical database unit for storage. The monitoring unit is used to monitor the data collected in real time, and to monitor the risks of data viruses and firewalls.

[0034] In this embodiment, the collected data is stored separately from real-time data and historical data, which facilitates better classification of the raw data. Especially when there is a lot of historical data, the large amount of historical data makes the retrieval buffer time too long when real-time data needs to be retrieved. The original data collection time of the database is stored separately, which facilitates reasonable storage of the original data. In addition, the monitoring unit monitors the data online in real time, which reduces the risk of the original data being stolen or written with viruses. Example 5 is an explanation based on Example 1. For details, please refer to [link / reference]. Figure 1 - Figure 2 The data collection port is used to collect browsing data, search data and interaction data of users from several IPs in / APP / webpage, and convert the browsing data, search data and interaction data into files in real time and save them as access files. The data acquisition port is configured with a user-defined template that includes at least one target table. Data acquired from the acquisition port is processed using this user-defined template to generate DOC / PDF / EXL files, etc.

[0035] The browsing data is mainly used for consumer behavior analysis in the e-commerce field. The browsing data reflects each user's browsing steps, further characterizes the user's browsing path, and analyzes different pages and jump probabilities, etc. The search data mainly refers to the time-series data of keyword search frequency recorded by the search engine, which can reflect the interests, concerns, and intentions of hundreds of millions of users. The search represents the amplitude. The interaction data mainly consists of data from Weibo and social networking sites, reflecting users' preferences and emotional factors.

[0036] In this embodiment, the collected data is classified into browsing data, search data, and interaction data. This classification facilitates the extraction of data later and better reflects the value of the network data. For example, interaction data reflects user preferences, making it easier to set up relevant products based on those preferences.

[0037] All user data, network data, and data collection and processing operations involved in the above embodiments are carried out under the conditions permitted by national laws and regulations and with the consent of the users.

[0038] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0039] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A network data information intelligent extraction system, comprising a data acquisition module and a data feature extraction system, characterized in that: The acquisition module and the data feature extraction system together form a data repository, which is used for data storage and computation, aggregation, calculation, filtering and extraction of common features from file data sources, streaming data and crawler data, and then screening and classification. The acquisition module is used to acquire raw network data, and after converting, preprocessing, and judging the raw data, it sends it to the data feature extraction system for further processing. The data feature extraction system is used to filter the converted data sent by the acquisition module and extract common features according to the matching similarity value, thereby improving intelligent classification and extraction. The matching similarity classification values ​​include format data source classification, row and column matching data source classification, keyword data source matching classification, and repetition rate matching classification. The data feature extraction system also includes a matching module and an extraction module; The matching module is used to match the original data with the preset template file format based on similarity, and to classify regions based on the matching similarity classification value. The extraction module is used to extract and select data by inputting tags and search terms; The matching module includes a similarity classification unit and a conversion unit; The similarity classification unit is used to calculate the matching similarity classification value. The matching steps of the similarity classification unit are as follows: S1. Obtain the collected message data; S2. Recognize the format and content of the message data; S3. Extract common network traffic features from the identified message data; S4. Extract frequent words from the message rows and columns; S5. Calculate the number of occurrences of frequent words extracted from the message rows; S6. Obtain keywords, search terms, and frequency values; S7. Write the keywords and frequency values ​​into the message tag; S8. Link related data information; In step S1, the acquired messages are specifically subjected to format recognition and format classification. The data feature extraction system further includes a format recognition unit and a format classification unit. The format recognition unit is used to receive the data collected by the acquisition module and recognize the data format as FPG / DOC / PDF / EXL / XSD; The format classification unit is used to classify and store the identified data and data with the same format into the database. The data acquisition port is used to collect user browsing data, search data, and interaction data from the APP / webpage, and convert the browsing data, search data, and interaction data into files in real time and save them as accessible files; The data acquisition port is configured to include a user-defined template with at least one target table, and the data acquired from the acquisition port is used to generate DOC / PDF / EXL files from the user-defined template. The browsing data is used for consumer behavior analysis in the e-commerce field. The browsing data reflects each user's browsing steps, further characterizes the user's browsing path, and analyzes different pages and jump probabilities. The search data refers to the time-series data of keyword search frequency recorded by the search engine, which can reflect the interests, concerns, and intentions of hundreds of millions of users. The interaction data refers to data from Weibo and social networking sites, which reflects users' access data based on their tendencies and emotional factors. The extraction module also includes an extraction unit, which retrieves multiple data sets, correlates and merges them to generate a new form, and then performs queries or further correlates and merges them with other data based on the new form.

2. The intelligent network data information extraction system according to claim 1, characterized in that: In step S5, during the process of calculating the occurrence count of frequent words, if a frequent word is found to be a sensitive word, it is marked with a sensitive label. If the frequent word is related to place names, events, or personal names, it is classified according to person, event, or place, and a specific classification label is added.

3. The intelligent network data information extraction system according to claim 1, characterized in that: The acquisition module includes a real-time database unit, a historical database unit, a monitoring unit, and multiple data acquisition ports; The real-time database unit is used to store real-time data collected from multiple data acquisition ports in real time. The historical database unit is used to automatically transfer unprocessed data collected in the real-time database unit more than 24 hours ago to the historical database unit for storage. The monitoring unit is used to monitor the data collected in real time, and to monitor the risks of data viruses and firewalls.

4. The intelligent network data information extraction system according to claim 1, characterized in that: The extraction module includes a full extraction unit and an incremental extraction unit.

Images