Big data intelligent storage system

Abstract

The invention relates to the technical field of big data storage, and discloses a big data intelligent storage system which comprises a data storage unit used for storing data, a compression and deduplication unit used for data processing, an auditing and classifying unit used for data preprocessing and a data mining unit used for deep collection. The data extraction unit is connected with a verification unit, the verification unit is connected with a user platform, the auditing and classifying unit and the data mining unit are connected with the same data encryption unit, the data encryption unit is connected with the compression and deduplication unit, and the verification unit and the data mining unit are both connected with the data encryption unit. And the auditing and classifying unit comprises image-text classification, digital classification, information classification and audio classification. According to the method, the information can be effectively preprocessed and normalized, the storage efficiency is improved, data leakage and loss are prevented, and the safety of data storage is ensured.

Description

Technical Field

[0001] The present invention relates to the field of big data storage technology, and in particular to a big data intelligent storage system. Background Art

[0002] Big data intelligent storage system refers to a system specifically used to store and manage large-scale data. With the advent of the big data era, various industries and organizations are facing the needs of processing and storing massive amounts of data. Traditional storage systems can no longer meet these needs. Therefore, big data intelligent storage system came into being.

[0003] The current big data storage system needs to receive a large amount of data information when collecting data, and then process the data, which reduces the efficiency of data storage and consumes a lot of processing resources and time. At the same time, the current big data storage security is not stable, and security problems such as information leakage may occur. For this reason, a big data intelligent storage system is proposed. Summary of the invention

[0004] The purpose of the present invention is to provide a big data intelligent storage system to solve the problems raised in the above-mentioned background technology.

[0005] To achieve the above-mentioned objectives, the present invention provides the following technical solution: comprising a data storage unit for storing data, a compression and deduplication unit for data processing, an audit and classification unit for data preprocessing, and a data mining unit for deep acquisition, wherein the data storage unit is connected to a data extraction unit, the data extraction unit is connected to a verification unit, the verification unit is connected to a user platform, the audit and classification unit and the data mining unit are connected to the same data encryption unit, and the data encryption unit is connected to the compression and deduplication unit.

[0006] Preferably, the verification unit and the data mining unit are both connected to the data encryption unit.

[0007] Preferably, the review classification unit includes graphic classification, digital classification, information classification and audio classification.

[0008] Preferably, the data encryption unit is provided with an access rights module, a defense system, a security audit mechanism, an anonymization unit and an encryption module.

[0009] Preferably, the encryption module is connected with the audit classification and data mining, and the verification unit and the user platform are connected with the permission module, the defense system, the security audit mechanism and the anonymization unit.

[0010] Preferably, the audit classification module and the data mining module are connected to uncollected data sets.

[0011] Compared with the prior art, the present invention has the following beneficial effects:

[0012] By pre-reviewing and classifying the data, the information can be effectively pre-processed and organized to improve storage efficiency. The collected and stored data can be encrypted through the data encryption unit, and multiple verifications can be performed on the user login direction to prevent data leakage and ensure the security of data storage. In addition, the data will be automatically compressed and deduplicated before storage, which can effectively reduce storage space occupancy and increase data storage capacity. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a schematic diagram of the overall module structure of a big data intelligent storage system of the present invention;

[0014] Figure 2 This is a schematic diagram of a specific connection structure of a data encryption module in a big data intelligent storage system of the present invention;

[0015] Figure 3 This is a structural diagram of an audit classification module in a big data intelligent storage system of the present invention. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] See also Figure 1-3 , the present invention provides a technical solution: including a data storage unit for storing data, a compression and deduplication unit for data processing, an audit and classification unit for data preprocessing, and a data mining unit for deep acquisition, the data storage unit is connected to a data extraction unit, the data extraction unit is connected to a verification unit, the verification unit is connected to a user platform, the audit and classification unit and the data mining unit are connected to the same data encryption unit, and the data encryption unit is connected to the compression and deduplication unit;

[0018] Both the verification unit and the data mining unit are connected to the data encryption unit. The user logs in through the user platform and then passes the basic information verification through the verification unit. During this operation, the user's operation information will be read and received by the data mining unit, and encrypted through the data encryption module to desensitize the user's private information and store it. The data encryption module uses column storage, compression algorithm, deduplication algorithm and storage compression module to automatically compress and deduplicate data, save storage space and increase data transmission speed.

[0019] The audit classification unit includes graphic classification, digital classification, information classification and audio classification. It can classify a large amount of uncollected information or information collected by the data mining module according to type and store them in a classified manner to facilitate data management and improve efficiency. At the same time, the audit classification module will perform data cleaning and data security audits.

[0020] The access permission module in the data encryption unit monitors user operations when the user logs in. The defense system and data storage unit both use blockchain technology to increase the security and credibility of data and prevent data from being tampered with and forged;

[0021] The security audit mechanism includes the following steps:

[0022] Define audit strategy: First, you need to clearly define the scope and objectives of the audit, determine the types of data access and operations that need to be recorded, including read, write, modify, delete, etc., and determine which key information needs to be recorded, such as user ID, timestamp, IP address, operation type, accessed data object, etc.;

[0023] Configure audit logs: Enable the audit log function in the big data system and configure the appropriate log level and log format according to the defined audit policy to ensure that the audit log can record enough detailed information to support subsequent audit analysis;

[0024] Record audit logs: When users access and operate data, the system should automatically record the corresponding audit logs. This can be achieved by adding audit log recording code to key components, services or interfaces of the system, ensuring that the recorded logs contain sufficient contextual information for subsequent audit analysis and investigation;

[0025] Storing and protecting audit logs: Audit logs should be stored in a secure location, and necessary measures should be taken to protect their integrity and availability. Audit logs can be stored in a dedicated audit log server or log management platform, and encryption, access control, and backup methods should be used to protect log data.

[0026] Analyze audit logs: Regularly analyze and check stored audit logs to detect abnormal activities or potential security risks. Use log analysis tools, machine learning techniques, or custom scripts to extract useful information and create reports, alerts, or dashboards to support real-time monitoring and rapid response.

[0027] Audit tracking and investigation: After discovering anomalies or security incidents, audit logs can serve as an important source of evidence to support the security team in investigation and tracking. By analyzing key information such as the timestamp, user ID, and operation type of the audit logs, the process of the incident can be restored and the relevant responsible persons can be identified;

[0028] Compliance of audit logs: Ensure that audit logs meet regulatory and compliance requirements, such as GDPR, PCI DSS, etc. For certain sensitive data or data in specific industries, additional measures may need to be taken, such as data encryption, access approval processes, etc. to ensure data security and compliance and anonymization units and encryption modules.

[0029] The encryption module is connected with the audit classification and data mining, and the verification unit and the user platform are connected with the permission module, the defense system, the security audit mechanism and the anonymization unit. The anonymization unit processes sensitive data to desensitize personal identity information to prevent sensitive information from being maliciously used. For example, the hash algorithm, desensitization algorithm and other technologies are used to convert the original data into irreversible anonymous data.

[0030] The audit classification module and the data mining module are connected with uncollected data sets. The data mining module can collect data in a deeper and larger range in the data set to ensure the real-time data collection effect and help the audit classification module to appropriately reduce the workload.

[0031] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0032] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A big data intelligent storage system, comprising a data storage unit for storing data, a compression and deduplication unit for data processing, an audit and classification unit for data preprocessing, and a data mining unit for deep acquisition, characterized in that: The data storage unit is connected to a data extraction unit, the data extraction unit is connected to a verification unit, the verification unit is connected to a user platform, the audit classification unit and the data mining unit are connected to the same data encryption unit, and the data encryption unit is connected to a compression and deduplication unit.

2. The big data intelligent storage system according to claim 1, characterized in that: The verification unit and the data mining unit are both connected to the data encryption unit.

3. The big data intelligent storage system according to claim 1, characterized in that: The review classification unit includes picture and text classification, digital classification, information classification and audio classification.

4. The big data intelligent storage system according to claim 1, characterized in that: The data encryption unit is provided with an access rights module, a defense system, a security audit mechanism, an anonymization unit and an encryption module.

5. The big data intelligent storage system according to claim 4, characterized in that: The encryption module is connected with the audit classification and data mining, and the verification unit and the user platform are connected with the permission module, the defense system, the security audit mechanism and the anonymization unit.

6. The big data intelligent storage system according to claim 1, characterized in that: The audit classification module and the data mining module are connected with uncollected data sets.

Images