Information acquisition and analysis method and system based on big data

Through the combined distributed acquisition rules and intelligent filtering mechanism combined with deep learning technology, the problem of inefficient big data acquisition and analysis is solved, efficient and accurate data acquisition and analysis is achieved, and the real-time and security of data processing is improved. Data applications provide all-round support.

CN119066674APending Publication Date: 2024-12-03XIAMEN CENTURY LIANHENG TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411047844.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The existing big data information acquisition and analysis system has problems such as insufficient real-time data acquisition, uneven data quality, and low analysis efficiency, which affects the accuracy of data analysis results and the in-depth application of big data technology.

Method used

Data acquisition is collected using distributed acquisition rules and intelligent filtering mechanisms, combining deep learning and visual presentation to ensure safe data transmission and storage, and improve the accuracy and efficiency of data analysis through data preprocessing and multiple analytical methods.

Benefits of technology

It realizes efficient and accurate collection and analysis of big data, ensures the purity and security of data, improves data processing efficiency, meets the needs of large-scale data processing, and provides users with more accurate and intuitive data support services.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119066674A_ABST
    Figure CN119066674A_ABST
Patent Text Reader

Abstract

The invention belongs to the field of data processing, and particularly relates to an information collection and analysis method and system based on big data, and the method mainly comprises the following steps: a data collection module carries out the real-time data collection according to a preset distributed collection rule, employs an intelligent filtering mechanism to filter the valuable data, carries out the encryption and verification of the data, and carries out the collection and verification of the data; safe transmission and storage of the data are ensured, and the processed original data are output; the data preprocessing module cleans the data, removes noise, abnormal values and missing values in the data, and ensures the accuracy and integrity of the data; the method comprises the following steps that: firstly, a data processing module is used for processing data, then the data is standardized, the data analysis module is used for classifying and clustering the data through a machine learning algorithm and extracting potential information and laws in the data, and finally, a result display module is used for selecting a proper visual chart and a report template to output an analysis result. And powerful support is provided for development of big data application.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of data processing, and specifically relates to a method and system for collecting and analyzing information based on big data. Background Art

[0002] With the advent of the big data era, the collection and analysis of massive data resources has become an important demand for enterprises, scientific research institutions and even the entire society, and is the core resource of today's society. However, the existing big data information collection and analysis systems generally have problems such as insufficient real-time data collection, uneven data quality, and low analysis efficiency. These problems not only affect the accuracy of data analysis results, but also restrict the in-depth application of big data technology in various fields. The ability to efficiently analyze massive data is of great significance to improving data processing capabilities and promoting the development of big data applications.

[0003] Therefore, how to extract valuable information from massive data, conduct efficient analysis, and make accurate decisions is a technical problem that needs to be solved urgently. The present invention provides an efficient and accurate big data information collection and analysis system to provide users with comprehensive and high-quality data support services, and promote the wide application and deep integration of big data technology in more fields. Summary of the invention

[0004] In view of the defects of the prior art, the present invention provides a method and system for information collection and analysis based on big data, which optimizes the data collection method and introduces an intelligent filtering mechanism to ensure the safe transmission and storage of data, and effectively improves the accuracy and efficiency of data collection and processing through deep learning and visual display of results.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows: a method for collecting and analyzing information based on big data, comprising the following steps:

[0006] (1) Data collection: According to the preset distributed collection rules, real-time data collection is carried out and the valueless data is filtered out by intelligent filtering mechanism. The data is then encrypted and verified to ensure the secure transmission and storage of data, and the processed raw data is output;

[0007] (2) Data preprocessing: First, clean the data to remove noise, outliers, and missing values ​​to ensure data accuracy and completeness. Then, standardize the data to convert data from different sources and formats into a unified format and standard to eliminate inconsistencies and incomparability between data. Then, reduce the complexity of the data and improve the efficiency of subsequent analysis by performing dimensionality reduction and feature extraction.

[0008] (3) Data analysis: Classify and cluster data through machine learning algorithms to extract potential information and patterns in the data. Then, combine time series analysis and association rule mining with multiple analysis methods to provide users with more comprehensive and in-depth analysis results.

[0009] (4) Result presentation: Select appropriate visualization charts and report templates to output analysis results, including but not limited to charts, tables, and reports, support user interaction, gain in-depth understanding of data details, and share or export analysis results into common formats.

[0010] Furthermore, the distributed collection rule is to automatically collect data from data sources at all levels, including databases and the Internet, divide the data collection task into multiple subtasks, and execute them in parallel by multiple data collection nodes, namely multi-user + multi-server websites.

[0011] Furthermore, the specific execution process of the distributed collection rule is as follows:

[0012] 1) According to the type, quantity and complexity of data sources, the entire collection task D I Divide into multiple subtasks D i {i∈I};

[0013] 2) Select appropriate data collection nodes N{N∈i}, which should have stable network connections and sufficient computing resources;

[0014] 3) Assign subtasks to each node, and let the nodes execute data collection tasks in parallel;

[0015] 4) Merge and organize the data collected by each node to form a complete data set.

[0016] Furthermore, the intelligent filtering mechanism screens and filters the data according to the rules or conditions set by the user, so as to ensure that the collected data has a high degree of purity and availability.

[0017] Furthermore, the sh i ro improved algorithm is used for the data encryption and verification process, and the specific process is as follows:

[0018] 1) Configure Sh i ro: specify the management and use of encryption keys, generate an encryption key, or obtain a key from a secure key storage, which is used to encrypt and decrypt data;

[0019] 2) Encrypt data: Encrypt the data, use Sh i ro's encryption component to take the data as input, encrypt it using the key, and generate encrypted data;

[0020] 3) Storing encrypted data: Store the encrypted data in an appropriate location to ensure that only authorized users or systems can access the data;

[0021] 4) Decrypted data: When the original data needs to be accessed, the same key and Sh i ro’s decryption component are used to decrypt the encrypted data. The decryption process will restore the original data.

[0022] Furthermore, for missing data, linear interpolation can be used to repair missing data with small time intervals.

[0023] Furthermore, if the variation range of the sampled data at a certain moment is greater or less than 10% of the monitoring values ​​before and after it, the sampled data is considered to be distorted and is processed using the mean smoothing method.

[0024] Furthermore, the machine learning algorithm first uses the preprocessed data as a training set, and uses a CNN+LSTM neural network to evaluate and optimize the performance of the algorithm, and uses the data output by the CNN+LSTM model as input data for model adjustment to complete subsequent analysis.

[0025] A system based on a big data information collection and analysis method is provided, comprising a data collection module, a data preprocessing module, a data analysis module, and a result display module, wherein the data collection module collects data using distributed collection rules, and transmits the data to the data preprocessing module after intelligent filtering and encryption, and the data preprocessing module cleans, standardizes, and extracts features of the data; the data analysis module receives and processes the preprocessed data transmitted by the data preprocessing module using an algorithm, and the result display module receives the final analysis result output by the data analysis module.

[0026] Furthermore, the result display module also provides result sharing and exporting functions, and users can share the analysis results with others or export them into common file formats for easy viewing and use on different platforms or devices.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] (1) The present invention can implement a distributed collection strategy + intelligent filtering mechanism for big data, perform real-time analysis and identification of data, automatically filter out duplicate, invalid or low-value data, and ensure that the collected data has high purity and availability.

[0029] (2) The present invention adopts the improved sh i ro algorithm to realize the encryption of collected data, ensuring that the collected data has high security and preventing the data from being tampered with or leaked during transmission and storage.

[0030] (3) The present invention adopts CNN+LSTM deep learning technology and a visualization method of the result display module to improve the efficiency of data processing and analysis, reduce the system operation cost, meet the needs of large-scale data processing, and provide users with more accurate and intuitive data support services. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a flow chart of the big data information collection and analysis method of the present invention;

[0032] Figure 2 This is a structural diagram of the encryption process of the present invention. DETAILED DESCRIPTION

[0033] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0034] like Figure 1 As shown, a method for collecting and analyzing information based on big data includes the following steps:

[0035] (1) Data collection: According to the preset distributed collection rules, real-time data collection is carried out and the valueless data is filtered out by intelligent filtering mechanism. The data is then encrypted and verified to ensure the secure transmission and storage of data, and the processed raw data is output;

[0036] Distributed collection strategy is an efficient data collection method. In traditional data collection methods, single-machine crawling is often used, that is, all data collection tasks are completed by a single website (single user + single server website); however, with the explosive growth of data volume, the single-machine crawling method can no longer meet the needs of large-scale data collection. Therefore, the present invention adopts distributed collection rules to automatically collect data from data sources at all levels, including databases and the Internet, and divides the data collection task into multiple subtasks, which are executed in parallel by multiple data collection nodes, namely multiple users + multiple server websites. The specific execution process is:

[0037] 1) According to the type, quantity and complexity of data sources, the entire collection task D I Divide into multiple subtasks D i {i∈I};

[0038] 2) Select appropriate data collection nodes N{N∈i}, which should have stable network connections and sufficient computing resources;

[0039] 3) Assign subtasks to each node, and let the nodes execute data collection tasks in parallel;

[0040] 4) Merge and organize the data collected by each node to form a complete data set.

[0041] The intelligent filtering mechanism will analyze the collected data in real time, identify duplicate, invalid or low-value data, and filter them out. At the same time, the mechanism can also further screen and filter the data according to the rules or conditions set by the user to ensure that the collected data has a high degree of purity and availability.

[0042] In the process of data collection, attention should also be paid to the integrity and security of the data. Therefore, data verification and encryption technology are used to ensure that the collected data is not tampered with or leaked during transmission and storage. In addition, the module also provides a flexible data collection interface, supports access to multiple data sources and formats, and facilitates users to collect data according to actual needs.

[0043] In the data collection process, the Shiro improved algorithm is used, and its encryption structure is as follows: Figure 2 As shown:

[0044] The specific process is as follows:

[0045] 1) Configure Shiro: specify the management and use of encryption keys, generate an encryption key, or obtain a key from a secure key storage, which is used to encrypt and decrypt data;

[0046] 2) Encrypt data: Encrypt the data, use Sh i ro's encryption component to take the data as input, encrypt it using the key, and generate encrypted data;

[0047] 3) Storing encrypted data: Store the encrypted data in an appropriate location to ensure that only authorized users or systems can access the data;

[0048] 4) Decrypted data: When the original data needs to be accessed, the same key and Sh i ro’s decryption component are used to decrypt the encrypted data. The decryption process will restore the original data.

[0049] (2) Data preprocessing: First, clean the data to remove noise, outliers, and missing values ​​to ensure data accuracy and completeness. Then, standardize the data to convert data from different sources and formats into a unified format and standard to eliminate inconsistencies and incomparability between data. Then, reduce the complexity of the data and improve the efficiency of subsequent analysis by performing dimensionality reduction and feature extraction.

[0050] In the process of sampling data initialization, the data needs to be denoised and outliers and missing values ​​repaired to eliminate interfering data. For missing data, linear interpolation is used to repair missing data with a small time interval:

[0051]

[0052] Where: x t and x t+ j is the valid data collected at time t and time t+j; x t+1 The missing parameter at t+i.

[0053] For distorted data, since the required indicator data is continuously sampled and transmitted back in sequence, the data sampled in adjacent periods will not change suddenly. If the range of sampled data at a certain moment is greater than or less than 10% of the monitored value before and after, the sampled data is considered distorted; the mean smoothing method is used for processing:

[0054]

[0055] In the formula, and are the thresholds of adjacent sampling data errors.

[0056] (3) Data analysis: Classify and cluster data through machine learning algorithms to extract potential information and patterns in the data. Then, combine time series analysis and association rule mining with multiple analysis methods to provide users with more comprehensive and in-depth analysis results.

[0057] The model used in machine learning is the CNN+LSTM model: first, the preprocessed data is used as a training set, and the CNN+LSTM neural network is used to evaluate and optimize the performance of the algorithm. The data output by the CNN+LSTM model is used as input data for model adjustment to complete subsequent analysis.

[0058] The specific process is as follows: Assume that the network input x is M-dimensional: x = [x1, x2, ..., x M ] T The hidden layer has a total of K nodes, the output y is L-dimensional, the length of the input and output sample pairs is N, and the action function of the hidden layer nodes of the radial basis network is the Gaussian basis function:

[0059]

[0060] The input data vector is mapped to the hidden layer, and the output of the hidden layer node j is:

[0061]

[0062] Where: δ is the standardization constant of the hidden layer nodes, c is the center vector of the Gaussian function of the hidden layer nodes, c j =[c j1 ,c j2, ...,c jM ] T

[0063] The hidden layer to output layer of the RBF network realizes a linear mapping of the conversion dimension, that is, the output of the output layer node k is:

[0064]

[0065] Where: jk is the adjustment weight from the hidden layer to the output layer; θ k is the bias of the output layer node k. k As a response to the corresponding input signal, it is output to the workspace as an important regulatory variable according to different backgrounds.

[0066] The input x above is the preprocessed data, and the output y is the input data after model adjustment.

[0067] (4) Result presentation: Select appropriate visualization charts and report templates to output analysis results, including but not limited to charts, tables, and reports. Support user interaction to gain an in-depth understanding of data details, and share or export analysis results into common formats for easy viewing and use on different platforms or devices.

[0068] A big data information collection and analysis system, including a data collection module, a data preprocessing module, a data analysis module, and a result display module.

[0069] 1. Data collection module: start data collection task

[0070] Distributed collection strategy: Divide the task into multiple subtasks, which are executed in parallel by multiple data collection nodes;

[0071] Real-time data filtering: Analyze data and automatically filter invalid, duplicate or low-value data;

[0072] Data encryption and verification: ensure secure data transmission and storage;

[0073] Output: cleaned raw data.

[0074] 2. Data preprocessing module: receiving the output of the data acquisition module

[0075] Data cleaning: remove noise, outliers and missing values;

[0076] Data standardization: unify data formats and standards;

[0077] Data dimensionality reduction and feature extraction: reduce data complexity and extract key features;

[0078] Visualization: Use visualization tools and techniques to display data distribution and characteristics;

[0079] Output: preprocessed data.

[0080] 3. Data analysis module: receives the output of the data preprocessing module

[0081] Machine learning analysis: classification, clustering, regression and other tasks;

[0082] Deep learning mining: discovering complex relationships and patterns;

[0083] Time series analysis and association rule mining: discovering time trends and association relationships;

[0084] Model optimization and evaluation: adjust model parameters and evaluate model performance;

[0085] Output: Analysis results.

[0086] 4. Result display module: receiving the output of the data analysis module

[0087] Choose appropriate visualization charts and report templates;

[0088] Results presentation: Present the analysis results in the form of charts, tables, reports, etc.;

[0089] Interactive exploration: supports user interactive operations and in-depth understanding of data details;

[0090] Result sharing and export: Share or export analysis results to common formats.

[0091] In summary, the present invention constructs an efficient and accurate big data information collection and analysis system through the organic combination of data acquisition module, data preprocessing module, data analysis module and result display module. The system can realize real-time collection, preprocessing, in-depth analysis and result display of big data, and provide users with a full range of data support services. Compared with the prior art, the present invention has higher real-time performance, accuracy and efficiency, and can better meet the needs of practical applications. At the same time, it also has strong scalability and flexibility, and can be customized and optimized according to the needs of different users, providing strong support for the development of big data applications.

[0092] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A method for collecting and analyzing information based on big data, It is characterized in that The following steps are involved: (1) Data collection: According to the preset distributed collection rules, real-time data collection is carried out and the intelligent filtering mechanism is used to filter out worthless data. The data is then encrypted and verified to ensure secure data transmission and storage, and the processed raw data is output; (2) Data preprocessing: First, clean the data to remove noise, outliers, and missing values ​​to ensure data accuracy and completeness. Then, standardize the data to convert data from different sources and formats into a unified format and standard to eliminate inconsistencies and incomparability between data. Then, reduce the complexity of the data and improve the efficiency of subsequent analysis by performing dimensionality reduction and feature extraction. (3) Data analysis: Classify and cluster data through machine learning algorithms to extract potential information and patterns in the data. Then, combine time series analysis and association rule mining with multiple analysis methods to provide users with more comprehensive and in-depth analysis results. (4) Result presentation: Select appropriate visualization charts and report templates to output analysis results, including but not limited to charts, tables, and reports, support user interaction, gain in-depth understanding of data details, and share or export analysis results into common formats.

2. According to the method for collecting and analyzing big data information according to claim 1, It is characterized in that The distributed collection rule is to automatically collect data from data sources at all levels, including databases and the Internet, divide the data collection task into multiple subtasks, and execute them in parallel by multiple data collection nodes, namely multi-user + multi-server websites.

3. According to the method for collecting and analyzing big data information according to claim 2, It is characterized in that The specific execution process of the distributed collection rule is as follows: 1) According to the type, quantity and complexity of data sources, the entire collection task D I Divide into multiple subtasks D i {i∈I}; 2) Select appropriate data collection nodes N{N∈i}, which should have stable network connections and sufficient computing resources; 3) Assign subtasks to each node, and let the nodes execute data collection tasks in parallel; 4) Merge and organize the data collected by each node to form a complete data set.

4. According to the method for collecting and analyzing big data information according to claim 1, It is characterized in that The intelligent filtering mechanism is to screen and filter the data according to the rules or conditions set by the user to ensure that the collected data has high purity and availability.

5. According to the method for collecting and analyzing big data information according to claim 1, It is characterized in that The Shiro improved algorithm is used for data encryption and verification. The specific process is as follows: 1) Configure Shiro: specify the management and use of encryption keys, generate an encryption key, or obtain a key from a secure key storage, which is used to encrypt and decrypt data; 2) Encrypted data: Encrypt the data, use Shiro's encryption component to take the data as input, encrypt it with a key, and generate encrypted data; 3) Storing encrypted data: Store the encrypted data in an appropriate location to ensure that only authorized users or systems can access the data; 4) Decrypted data: When you need to access the original data, use the same key and Shiro's decryption component to decrypt the encrypted data. The decryption process will restore the original data.

6. According to the method for collecting and analyzing big data information according to claim 1, It is characterized in that For missing data, linear interpolation can be used to repair missing data with small time intervals.

7. According to the method for collecting and analyzing big data information according to claim 1, It is characterized in that If the variation range of the sampling data at a certain moment is greater or less than 10% of the monitoring value before and after it, the sampling data is considered to be distorted and the mean smoothing method is used for processing.

8. According to the method for collecting and analyzing big data information according to claim 1, It is characterized in that The machine learning algorithm first uses the preprocessed data as a training set, and uses a CNN+LSTM neural network to evaluate and optimize the performance of the algorithm, and uses the data output by the CNN+LSTM model as input data for model adjustment to complete subsequent analysis.

9. A system for implementing a method for collecting and analyzing big data information according to any one of claims 1 to 8, It is characterized in that It includes a data acquisition module, a data preprocessing module, a data analysis module, and a result display module, wherein the data acquisition module adopts distributed acquisition rules to collect data, and transmits the data to the data preprocessing module after intelligent filtering and encryption. The data preprocessing module cleans, standardizes and extracts features of the data; the data analysis module receives and uses an algorithm to process the preprocessed data transmitted by the data preprocessing module, and the result display module accepts the final analysis results output by the data analysis module.

10. A system based on a big data information collection and analysis method according to claim 9, It is characterized in that The result display module also provides result sharing and exporting functions, and users can share the analysis results with others or export them into common file formats for easy viewing and use on different platforms or devices.