Data storage system for information security

Abstract

The application discloses an information security data storage system and relates to the technical field of data storage.The application solves the technical problem of avoiding the problem of resource waste or performance bottleneck caused by one-size-fits-all storage by intelligently selecting local storage or distributed storage mode according to the type, capacity size and delay requirement of data.The application dynamically selects local or distributed storage based on data capacity and delay requirement, stores small-capacity and low-delay data locally to reduce network overhead, stores large-capacity and high-delay data in a distributed manner to improve scalability, improves resource utilization, automatically deletes temporary data, combines high-frequency / low-frequency data differentiation processing, takes into account data security and space optimization, dynamically adapts different capacity data through a dynamic sharding strategy, ensures balanced distribution of data through consistent hashing and virtual nodes, and realizes dynamic load adjustment through cold / hot data layering and node performance grading, thereby improving access speed and system stability.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, specifically to a data storage system for information security. Background Technology

[0002] With the rapid development of information technology, the scale of data is growing explosively, and the types of data are becoming increasingly diversified. Different types of data have significantly different requirements for storage capacity, latency, and security.

[0003] The current data storage system has the following characteristics:

[0004] The storage options are limited, often using fixed patterns of local or distributed storage, making it difficult to dynamically adapt to data characteristics. Local storage management lacks a refined classification strategy, and core data is mixed with temporary data, leading to security risks and inefficiency.

[0005] In distributed storage, fixed data sharding strategies and weak node load balancing capabilities can easily lead to overload of some nodes or uneven data distribution. At the same time, the lack of tiered management of hot and cold data, coupled with the sharing of storage resources between high-frequency and low-frequency data, results in a decline in access performance.

[0006] The storage system lacks a global coordination mechanism, and the management of local storage and distributed storage is fragmented, making it difficult to dynamically optimize storage strategies according to the data lifecycle and failing to meet the dual requirements of information security and efficient utilization. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a data storage system for information security, which solves the problem of intelligently selecting local or distributed storage methods based on data type, capacity, and latency requirements, thus avoiding resource waste or performance bottlenecks caused by a one-size-fits-all approach to storage.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a data storage system for information security, comprising:

[0009] The classification storage analysis module performs local or distributed storage analysis on the acquired classification data and generates local or distributed storage signals.

[0010] The local storage analysis module analyzes the acquired local storage signals, performs secondary classification on the classified data and stores it to generate preliminary storage information, and performs secondary analysis on the preliminary storage information based on the data usage frequency to generate local data storage information.

[0011] The distributed storage analysis module analyzes the acquired distributed storage signals, obtains distributed storage data and dynamically shards it to obtain shard data, calculates its hash value with the storage node, matches it based on the same hash value to determine the matching storage node, and classifies the shard data into hot data and cold data according to the frequency of data use, and transmits it to the comprehensive storage analysis module.

[0012] The comprehensive storage analysis module manages the storage of hot and cold data obtained from the classification. It compares the total amount of fragmented data of matching storage nodes with the maximum load. If the load is too high, a data adjustment signal is generated; otherwise, distributed storage information is generated. The adjustment signal is analyzed to determine the nodes to be distributed. Priority is given to distributing the less numerous categories of hot and cold data. The nodes are also divided into high-performance and low-performance nodes according to their processing speed. If the node to be distributed is high-performance, cold data is distributed to low-load nodes first. At the same time, hot data from low-load nodes is transferred to that node to fill the idle load, generating distributed storage information. If it is low-performance, its hot data is transferred to other nodes in round-robin order of node performance from high to low, generating distributed storage information.

[0013] As a further embodiment of the present invention, it also includes a data classification and processing module and a storage management information output module;

[0014] The data classification and processing module is used to acquire all information data and classify it according to the data content to obtain structured data, unstructured data, and semi-structured data, and then transmit it to the classification data storage and analysis module.

[0015] The storage management information output module is used to store data based on local data storage information and distributed storage information.

[0016] As a further aspect of the present invention, the specific method by which the classification and storage analysis module generates local storage signals or distributed storage signals is as follows:

[0017] Obtain the data categories and denote them as i, where i = 1, 2, ..., j, and j represents the data category. At the same time, obtain the data capacity and latency requirements corresponding to data category i.

[0018] For data types with small data volume and low latency requirements, local storage is selected, and local storage processing signals are generated. For data types with large data volume and high latency requirements, distributed storage is selected, and distributed storage processing signals are generated.

[0019] As a further aspect of the present invention, the local storage analysis module generates local data storage information in the following specific manner:

[0020] Acquire local storage data and storage media, classify them into core, general, and temporary data, and process them differently: core data is backed up in multiple copies on highly reliable devices, general data is stored in a single copy and redundancy is cleared periodically, and temporary data is set with automatic deletion rules to form preliminary storage information;

[0021] Based on preliminary information and secondary analysis, the storage medium is dynamically updated, and the data is divided into low-frequency and high-frequency categories according to the usage frequency within time t, generating local data storage information.

[0022] As a further aspect of the present invention, the specific method by which the distributed storage analysis module determines the matching storage node is as follows:

[0023] All distributed storage nodes are labeled and their hash values ​​are calculated to obtain distributed storage data. The data is dynamically sharded according to capacity to obtain shard data. After calculating the hash value of the shard data, the first matching node is searched clockwise on the hash ring as the storage node. The matching storage nodes for all shards are determined in turn.

[0024] As a further aspect of the present invention, the specific method for obtaining fragmented data by dynamic fragmentation according to capacity is as follows:

[0025] For large data files / objects > 1GB, 128MB / block fragments are used, which can be adjusted to 64MB or 256MB via configuration options. For small to medium data files / objects ≤ 1GB, 4MB / block fragments are used, which can be adjusted to 1MB or 8MB.

[0026] As a further aspect of the present invention, the specific method for classifying the fragmented data into hot data and cold data based on the frequency of data usage is as follows:

[0027] Retrieve all shard data from the matching storage node and label them as n, where n = 1, 2, ..., m, and m represents the type of shard data. Simultaneously, obtain the number of times shard data n is used within time t and calculate its corresponding usage frequency. Then, compare the usage frequency with the classification threshold, and the specific value of the classification threshold is set by the operator. Shard data with a usage frequency greater than the classification threshold is marked as hot data, while shard data with a usage frequency less than the classification threshold is marked as cold data.

[0028] As a further aspect of the present invention, the specific method by which the comprehensive storage and analysis module manages the storage of the classified hot and cold data is as follows:

[0029] The system retrieves the matching storage node and its total data volume. It then compares the total data volume with the maximum load of the matching storage node, where the maximum load represents the maximum storage capacity. If the total data volume is greater than the maximum load, the matching storage node needs to perform data adjustment processing, and a data adjustment signal is generated. Otherwise, no adjustment is required, and distributed storage information is generated.

[0030] As a further aspect of the present invention, the specific method for processing the data adjustment signal is as follows:

[0031] The corresponding matching storage nodes are obtained and recorded as nodes to be distributed. At the same time, the quantities of hot data and cold data are obtained, and the quantities of the two are compared. The distribution and matching are prioritized based on the category data with smaller quantities. Then, the performance of the nodes corresponding to the matching storage nodes is analyzed. The node performance is specifically represented by the average processing speed within the time period t. Nodes with an average processing speed greater than the speed threshold are classified as high-performance nodes, and those with an average processing speed less than the speed threshold are classified as low-performance nodes. The matching is then performed based on the different performance nodes obtained from the classification.

[0032] As a further aspect of the present invention, the specific method for performing calculation and matching based on the different performance nodes obtained from the classification is as follows:

[0033] If the node to be distributed is a high-performance node, cold data is transferred to low-load nodes first, while hot data from low-load nodes is transferred to that node to fill idle load and generate distributed storage information.

[0034] If it is a low-performance node, its hot data is transferred and stored in round-robin order of performance of other nodes from high to low, generating distributed storage information.

[0035] This invention provides a data storage system for information security. Compared with existing technologies, it has the following advantages:

[0036] This invention dynamically selects local or distributed storage based on data capacity and latency requirements. Local storage of small-capacity, low-latency data reduces network overhead, while distributed storage of large-capacity, high-latency data improves scalability and resource utilization. Through multi-copy backup of core data, redundancy cleanup of general data, and automatic deletion of temporary data, combined with differentiated processing of high-frequency / low-frequency data, it balances data security and space optimization. Dynamic sharding strategies adapt to different data capacities, consistent hashing and virtual nodes ensure balanced data distribution, deduplication and redundancy control reduce space waste, and hot / cold data stratification and node performance grading enable dynamic load adjustment, improving access speed and system stability. Attached Figure Description

[0037] Figure 1 This is a system block diagram of the present invention. Detailed Implementation

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] First Embodiment

[0040] Please see Figure 1 This application provides a data storage system for information security, including a data classification and processing module, a classification storage and analysis module, a local storage and analysis module, a distributed storage and analysis module, and a storage management information output module, and in conjunction with... Figure 1 It can be seen that the information between the above functional modules is transmitted in one direction only.

[0041] The data classification and processing module is used to acquire all information data and classify it according to its content into structured data, unstructured data, and semi-structured data. Specifically, structured data refers to data with a fixed format and organizational structure, usually stored in tabular form, and follows a strict data model, such as user information, order records, and financial statement data. Unstructured data refers to data without a fixed format or structure, with diverse and irregular formats, such as text files, images, audio, video, emails, and log files. Semi-structured data refers to data between structured and unstructured data, with some structure but not strict, usually marked with tags or key-value pairs, such as documents in JSON, XML, CSV, and NoSQL databases. At the same time, the classified and processed data is transmitted to the classification storage and analysis module.

[0042] The classification storage analysis module performs storage analysis based on the acquired classification data. It identifies the data categories and denotes them as i, where i = 1, 2, ..., j, and j represents the data category. For example, i = 1 represents basic user information (structured data), i = 2 represents high-definition video files (unstructured data), and i = 3 represents real-time sensor data (semi-structured data). The module also acquires the data capacity and latency requirements corresponding to data category i. Here, data capacity represents the average capacity range under normal conditions for the corresponding data category, and latency requirement is the maximum acceptable data read / write response time for the business. Combining data capacity and latency requirements, the module selects the appropriate storage method. Specifically, for data categories with small data capacity and low latency requirements, local storage is chosen; for data categories with large data capacity and high latency requirements, distributed storage is chosen. Local storage is suitable for high-frequency, low-latency access scenarios (such as database indexes and real-time caching) to avoid distributed network overhead, while distributed storage is suitable for massive data storage and expansion (such as video files and log archives) by distributing the load across multiple nodes.

[0043] Further, the storage methods for different data types are determined, and local storage processing signals and distributed storage processing signals are generated, while both are transmitted to the corresponding analysis modules.

[0044] The local storage analysis module analyzes the acquired local storage processing signals, obtains local storage data and corresponding storage media, and performs secondary classification to obtain core data, general data, and temporary data. Specific core data includes customer information and core code, general data includes daily communication documents, and temporary data includes downloaded installation packages and cache files. Different storage processing is applied to the local storage data obtained from the secondary classification to generate preliminary storage information. For core data, multiple copies are backed up and stored on high-performance, high-reliability devices. For general data, single-copy storage is performed and redundant data is cleaned up periodically. For temporary data, automatic deletion rules are set.

[0045] Secondary storage analysis is performed based on the generated preliminary storage information. The storage media is dynamically updated, specifically through periodic updates and deletions. Deletion includes removing duplicate files, expired logs, and temporary files. Simultaneously, local storage data is acquired, and the usage frequency of different local storage data within time t is calculated. Local storage data with a usage frequency below a frequency threshold is classified as low-frequency data, while data with a usage frequency above the threshold is classified as high-frequency data. Low-frequency data is compressed, while high-frequency data is backed up using a 3-2-1 backup principle: three data copies (one original and two backups), two different storage media (e.g., hard drive and USB drive), and one offline backup (located away from the local device, such as locked in a safe). Local data storage information is generated and transmitted to the storage management information output module.

[0046] The storage management information output module is used to perform corresponding storage processing based on the acquired local storage information.

[0047] Second Embodiment

[0048] As a second embodiment of the present invention, it is implemented based on the first embodiment, and differs from the first embodiment in the following ways:

[0049] The distributed storage analysis module analyzes the acquired distributed storage processing signals, obtains all distributed storage nodes and labels them as 'a', where a = 1, 2, ..., b, and b represents the number of distributed storage nodes. It also calculates the hash value corresponding to distributed storage node 'a', which can be based on unique identifiers such as node IP and hostname. Next, it acquires distributed storage data and dynamically adjusts the fragment size based on the data capacity to obtain different fragment data. Specifically, for large-capacity data (single file / object > 1GB), a 128MB / block fragment is used, which can be adjusted to 64MB or 256MB via configuration options. For small-to-medium-capacity data (single file / object ≤ 1GB), a 4MB / block fragment is used, which can be adjusted to 1MB or 8MB.

[0050] Simultaneously, the hash value corresponding to the sharded data is calculated using the SHA-256 algorithm, which is derived from the node hash. The input is the shard content + metadata, mapped to the integer position of the hash ring. The hash value of the sharded data is searched clockwise within the integer range. The node corresponding to the first encountered node hash value is the storage node for that sharded data. For example, imagine the hash value range as a ring, with data shards and nodes distributed on the ring. Data is always stored on the node closest to it in the clockwise direction. According to the redundancy strategy, continue to search clockwise for the next two virtual nodes and their corresponding physical nodes as replica nodes, and record them as matching storage nodes. This process is repeated to obtain the matching storage nodes corresponding to all sharded data.

[0051] Next, duplicate data in the matching storage nodes is identified and deleted. If deleting duplicate data has no impact on the overall data, it is deleted directly; otherwise, the duplicate data is retained. Specifically, a global hash index table is built based on the shard hash value to record the list of storage nodes corresponding to each hash value. If a new shard hash value already exists in the index table (i.e., it is determined to be duplicate data), a redundancy check is triggered. If it already exists on ≥N nodes (N is the number of replicas required by the redundancy strategy), the existing shard is directly referenced and not stored repeatedly. Only the reference count of the index table is updated. If the number of existing duplicate shards is less than N, they are supplemented and stored on the missing nodes to ensure that the redundancy requirements are met.

[0052] For duplicate shards with a reference count of 0 (i.e., no file references), they are marked as deletable and deleted during periods of low node load. For duplicate shards with a reference count > 0 but exceeding redundancy requirements (e.g., the policy requires 3 replicas but there are actually 5), the redundant shards on the node with the highest load are deleted first.

[0053] Based on this, the sharded data in the matching storage nodes is classified, and the specific classification method is as follows:

[0054] All fragment data in the matching storage nodes are obtained and labeled as n, where n = 1, 2, ..., m, and m represents the type of fragment data. At the same time, the number of times fragment data n is used within time t is obtained, and its corresponding usage frequency is calculated. The specific value of time t here is the same as the value of time t in the first embodiment. Then, the usage frequency is compared with the classification threshold, and the specific value of the classification threshold is set by the operator. Fragment data with a usage frequency greater than the classification threshold is marked as hot data, and conversely, fragment data with a usage frequency less than the classification threshold is marked as cold data. This process is repeated for all matching storage nodes to obtain the corresponding hot data and cold data. At the same time, the hot data and cold data obtained by classification are transmitted to the comprehensive storage analysis module.

[0055] The comprehensive storage analysis module is used to manage the storage of hot and cold data obtained from matching storage nodes. It obtains the matching storage nodes and the total amount of fragmented data in them. At the same time, it compares the total amount of fragmented data with the maximum load of its matching storage node, where the maximum load represents the corresponding maximum storage capacity. If the total amount of fragmented data is greater than the maximum load, it means that the corresponding matching storage node needs to perform data adjustment processing, and a data adjustment signal is generated. Otherwise, it means that no adjustment is needed, and distributed storage information is generated.

[0056] The generated data adjustment signals are analyzed to obtain the corresponding matching storage nodes, which are recorded as nodes to be distributed. Simultaneously, the quantities of hot and cold data are obtained, and their magnitudes are compared. Distribution and matching are prioritized for the category with the smaller quantity. Next, the performance of the nodes corresponding to the matching storage nodes is analyzed, specifically the average processing speed within a time period t. Nodes with an average processing speed greater than a speed threshold are classified as high-performance nodes, while those with a lower average speed are classified as low-performance nodes. Nodes are then classified into high-performance and low-performance nodes. Matching is then performed based on the different performance levels obtained from the classification. The specific processing method is as follows:

[0057] If the nodes to be distributed are classified as high-performance nodes, then the priority of distributing and matching with small amounts of classified data is abandoned. Instead, the cold data is directly distributed and matched. Then, all the cold data is transferred to low-load matching storage nodes and stored thereon. At the same time, the hot data in the low-load matching storage nodes is obtained and transferred to the nodes to be distributed. Here, it is assumed that the transferred hot data can meet the idle load of the nodes to be distributed, thus generating distributed storage information.

[0058] If the nodes to be distributed are classified as low-performance nodes, the existing hot data will be matched and transferred to obtain the required number of hot data to be transferred. The hot data will be sorted from highest to lowest performance according to the matching storage nodes, and the obtained hot data will be transferred to the matching storage nodes in a round-robin manner. At the same time, the transferred hot data will be stored to generate distributed storage information.

[0059] Next, the distributed storage information is transmitted to the storage management information output module.

[0060] The storage management information output module is used to store corresponding data based on the acquired distributed storage information.

[0061] Third Embodiment

[0062] As a third embodiment of the present invention, the focus is on implementing the first and second embodiments in combination.

[0063] The data in the above formulas are all calculated using numerical values, without substituting the parameter units. In addition, the contents not described in detail in this specification are all prior art known to those skilled in the art.

[0064] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A data storage system for information security, characterized in that, include: The classification storage analysis module performs local or distributed storage analysis on the acquired classification data and generates local or distributed storage signals. The local storage analysis module analyzes the acquired local storage signals, performs secondary classification on the classified data and stores it to generate preliminary storage information, and performs secondary analysis on the preliminary storage information based on the data usage frequency to generate local data storage information. The distributed storage analysis module analyzes the acquired distributed storage signals, obtains distributed storage data and dynamically shards it to obtain shard data, calculates its hash value with the storage node, matches it based on the same hash value to determine the matching storage node, and classifies the shard data into hot data and cold data according to the frequency of data use, and transmits it to the comprehensive storage analysis module. The comprehensive storage analysis module manages the storage of hot and cold data obtained from the classification. It compares the total amount of fragmented data of matching storage nodes with their maximum load. If the load is overloaded, a data adjustment signal is generated; otherwise, distributed storage information is generated. The adjustment signal is analyzed to determine the nodes to be distributed. Priority is given to distributing the less abundant categories of hot and cold data to these nodes. Nodes are also categorized into high-performance and low-performance nodes based on their processing speed. If the node to be distributed is high-performance, cold data is distributed to low-load nodes first, while hot data from low-load nodes is transferred to that node to fill idle load, generating distributed storage information. If the node is low-performance, its hot data is transferred to other nodes in round-robin order of node performance from highest to lowest, generating distributed storage information. The specific processing method is as follows: Obtain the matching storage node and the total amount of fragmented data in it. At the same time, compare the total amount of fragmented data with the maximum load of its matching storage node, where the maximum load represents the corresponding maximum storage capacity. If the total amount of fragmented data is greater than the maximum load, it means that the corresponding matching storage node needs to perform data adjustment processing, and a data adjustment signal is generated. Otherwise, it means that no adjustment is needed, and distributed storage information is generated. Obtain the corresponding matching storage node and record it as the node to be distributed. At the same time, obtain the quantity of hot data and cold data in it, compare the two quantities, and prioritize the distribution matching of the data with smaller quantities. Then, analyze the performance of the node corresponding to the matching storage node. The node performance is specifically represented by the average processing speed within the time period t. Nodes with an average processing speed greater than the speed threshold are classified as high-performance nodes, and those with an average processing speed less than the speed threshold are classified as low-performance nodes. At the same time, perform distribution matching based on the different performance nodes obtained from the classification. If the node to be distributed is a high-performance node, cold data is transferred to low-load nodes first, while hot data from low-load nodes is transferred to that node to fill idle load and generate distributed storage information. If it is a low-performance node, its hot data is transferred and stored in round-robin order of performance of other nodes from high to low, generating distributed storage information.

2. The data storage system for information security according to claim 1, characterized in that, It also includes a data classification and processing module and a storage management information output module; The data classification and processing module is used to acquire all information data and classify it according to the data content to obtain structured data, unstructured data, and semi-structured data, and then transmit it to the classification data storage and analysis module. The storage management information output module is used to store data based on local data storage information and distributed storage information.

3. The data storage system for information security according to claim 1, characterized in that, The specific method by which the classification and storage analysis module generates local storage signals or distributed storage signals is as follows: Obtain the data categories and denote them as i, where i = 1, 2, ..., j, and j represents the data category. At the same time, obtain the data capacity and latency requirements corresponding to data category i. For data types with small data volume and low latency requirements, local storage is selected, and local storage processing signals are generated. For data types with large data volume and high latency requirements, distributed storage is selected, and distributed storage processing signals are generated.

4. The data storage system for information security according to claim 1, characterized in that, The specific method by which the local storage analysis module generates local data storage information is as follows: Acquire local storage data and storage media, classify them into core, general, and temporary data, and process them differently: core data is backed up in multiple copies on highly reliable devices, general data is stored in a single copy and redundancy is cleared periodically, and temporary data is set with automatic deletion rules to form preliminary storage information; Based on preliminary information and secondary analysis, the storage medium is dynamically updated, and the data is divided into low-frequency and high-frequency categories according to the usage frequency within time t, generating local data storage information.

5. The data storage system for information security according to claim 1, characterized in that, The specific method by which the distributed storage analysis module determines the matching storage node is as follows: All distributed storage nodes are labeled and their hash values ​​are calculated to obtain distributed storage data. The data is dynamically sharded according to capacity to obtain shard data. After calculating the hash value of the shard data, the first matching node is searched clockwise on the hash ring as the storage node. The matching storage nodes for all shards are determined in turn.

6. The data storage system for information security according to claim 5, characterized in that, The specific method for obtaining fragmented data through dynamic fragmentation based on capacity is as follows: For large data files / objects > 1GB, 128MB / block fragments are used, which can be adjusted to 64MB or 256MB via configuration options. For small to medium data files / objects ≤ 1GB, 4MB / block fragments are used, which can be adjusted to 1MB or 8MB.

7. The data storage system for information security according to claim 1, characterized in that, The specific method for classifying fragmented data into hot and cold data based on data usage frequency is as follows: Retrieve all shard data from the matching storage node and label them as n, where n = 1, 2, ..., m, and m represents the type of shard data. Simultaneously, obtain the number of times shard data n is used within time t and calculate its corresponding usage frequency. Then, compare the usage frequency with the classification threshold, and the specific value of the classification threshold is set by the operator. Shard data with a usage frequency greater than the classification threshold is marked as hot data, while shard data with a usage frequency less than the classification threshold is marked as cold data.

Images