Distributed network security data storage system
By designing a distributed network security data storage system, using technical means such as security detection, data partitioning, encrypted storage and access control, the defects in the existing technology in data storage security are solved, and a more efficient and reliable network security data storage is achieved.
Patent Information
- Application Number
- CN202411646887.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-18
- Publication Date
- 2025-03-04
AI Technical Summary
The existing distributed storage technology has shortcomings in the security of data storage, and it is difficult to effectively ensure the integrity and privacy of network security data.
A distributed network security data storage system is designed, including security detection module, distributed architecture module, data partition module, security storage module and access control module. The system ensures the security of the data to be stored before, during and during the stored procedure by performing security verification, data partitioning, encrypted storage and fine-grained access control on the received data to be stored.
By comprehensively applying data security verification, distributed computing, big data processing technology and distributed storage, the system can better adapt to the complex and changeable network security environment, provide more reliable service guarantees, and ensure data security and reliability.
Smart Images
Figure CN119557905A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data security technology, and in particular to a distributed network security data storage system. Background Art
[0002] Network security data refers to various information and data related to security in a network environment. These data are important for detecting and preventing network attacks and protecting systems from threats. Therefore, network security data needs to be properly stored to avoid data loss or damage.
[0003] Common data storage methods include local storage, cloud storage, distributed file systems, database storage, cache, data warehouse, backup and archiving, and edge storage. Among them, cloud storage, distributed file systems, database storage, and edge storage have higher security. For existing distributed storage technologies, their purpose is mostly to ensure that data can be stored efficiently. However, there are still certain defects in the security of data storage. For this reason, the present application discloses a distributed network security data storage system. Summary of the invention
[0004] The purpose of this application is to provide a distributed network security data storage system to solve the technical problems raised in the above background technology.
[0005] To achieve the above-mentioned purpose, the present application discloses the following technical solutions: a distributed network security data storage system, including a security detection module, a distributed architecture module, a data partitioning module, a security storage module and an access control module;
[0006] The security detection module is configured to: perform security verification on the received data to be stored, and identify whether the data to be stored has security threats;
[0007] The data partitioning module is configured to: partition the data to be stored to obtain a plurality of pieces of data;
[0008] The distributed architecture module is configured as follows: having a multi-node distributed architecture, and each node is used to store partitioned slice data;
[0009] The security storage module is configured to: encrypt the data to be stored;
[0010] The access control module is configured to perform fine-grained control on access to stored data and record all operation logs on the data.
[0011] Preferably, the security verification of the received data to be stored specifically includes:
[0012] Collect and transmit data to be stored through real-time message queues;
[0013] Use a stream processing framework to perform real-time analysis on the collected data to be stored to identify potential security threats, where the potential security threats are security threats that were not discovered when the data to be stored was defined as network security data;
[0014] When no potential security threat is found, the security check is passed. Otherwise, the data to be stored is returned and a recheck warning is issued to prompt the data to be stored to re-identify the threat.
[0015] Preferably, the collecting and transmitting of the data to be stored through the real-time message queue specifically includes:
[0016] The data to be stored is temporarily stored through a ring buffer;
[0017] Use UDP protocol for data transmission.
[0018] Preferably, the data to be stored is partitioned to obtain multiple pieces of data, specifically including:
[0019] Define a partition function f:D×T→P, where D is the data item, T is the timestamp, and P is the partition identifier set;
[0020] Preliminarily divide the data to be transmitted according to time windows;
[0021] In the same time window, advanced segmentation is performed according to the spatial attributes of the data to be transmitted to obtain the slice data;
[0022] Regularly evaluate the data access frequency and data size of each partition, increase the space partition granularity for partitions with high access frequency or large data size, and reduce the space partition granularity for partitions with low access frequency or small data size.
[0023] Preferably, the data partitioning module is further configured to: encode each slice data block and generate additional redundant blocks;
[0024] The distributed architecture module is also configured to: store the encoded data blocks and the redundant blocks in different nodes in a dispersed manner.
[0025] Preferably, the data encryption for the data to be stored specifically includes:
[0026] Choose a random initial condition x0 as part of the key;
[0027] Choose a fixed control parameter r as part of the key;
[0028] The chaotic sequence {xn}, x n = r × x n-1 ×(1-x n-1 ), where x n is the iteration variable;
[0029] Extract a sequence from the chaotic sequence as the key sequence K;
[0030] The data to be encrypted is XORed with the key sequence K to obtain the encrypted data.
[0031] Preferably, the fine-grained control of access to stored data specifically includes:
[0032] Constructing an interaction relationship graph, wherein the interaction relationship graph includes a plurality of relationship nodes and edges, wherein each relationship node represents a user, and each edge represents an interaction relationship between users;
[0033] Use the shortest path algorithm to evaluate the trust relationship between users and determine the user's access rights;
[0034] Dynamically adjust the interaction diagram based on the user's real-time behavior.
[0035] Preferably, when the user is allowed access, the access control module sends a decryption instruction to the secure storage module; the secure storage module is also configured to: upon receiving the decryption instruction, perform an XOR operation on the encrypted data and the key sequence K to restore the original data.
[0036] As a preferred embodiment, the logs recording all operations on the data specifically include:
[0037] configuring a blockchain, wherein each block on the blockchain is linked to a previous block via a hash pointer;
[0038] Configure consensus algorithm for each block;
[0039] A set of operation logs is recorded in each block, and the content of the operation logs includes one or more of the operation type, operation time, operation user and operation object.
[0040] Beneficial effects: The distributed network security data storage system of the present application utilizes data security verification, distributed computing, big data processing technology and distributed storage to provide comprehensive technical support for security checks of stored data before storage, efficiency during storage, and security assurance during the storage process. It can better adapt to the complex and changing network security environment and provide users with more reliable service guarantees. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.
[0042] Figure 1 A structural block diagram of a distributed network security data storage system provided in an embodiment of the present application. DETAILED DESCRIPTION
[0043] The technical solutions in the embodiments of the present application are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.
[0044] In this article, the term "comprising" is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "comprising..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0045] This embodiment provides a Figure 1 The distributed network security data storage system shown includes a security detection module, a distributed architecture module, a data partitioning module, a security storage module and an access control module.
[0046] Next, we will give a detailed introduction to the various modules that make up the system.
[0047] 1. The security detection module is configured to: perform security verification on the received data to be stored, and identify whether there is a security threat to the data to be stored.
[0048] The security check of the received data to be stored can be implemented by the following steps:
[0049] Collect and transmit data to be stored through real-time message queues;
[0050] Use the stream processing framework to perform real-time analysis on the collected data to be stored to identify potential security threats. This process can use any detection technology in the existing technology. Potential security threats can indicate whether the data to be stored has security threats or whether there are undiscovered potential threats in the threat events. For example, when judging whether the data has undiscovered potential threats, different technical content (such as different detection models or different discrimination thresholds, etc.) should be used to implement the security identification of the data to be stored. It is feasible that the method for identifying potential security threats can be but is not limited to: Let μ be the mean of the data points in the window, σ be the standard deviation, and x be the current data point, then the anomaly score S can be expressed as: The threshold value B used in the judgment is set, and the relationship between the threshold value B and the threshold value A used in the previous judgment is B<A. At this time, when the newly calculated abnormal score S satisfies B<S<A, it means that there is no potential security threat to the data to be stored. If the newly calculated abnormal score S satisfies S≤B, it means that there is a potential security threat to the data to be stored;
[0051] When no potential security threat is found, the security check is passed. Otherwise, the data to be stored is returned and a re-inspection warning is issued to prompt the data to be stored to re-identify the threat. At this time, a detection technology different from the first threat identification (such as a different detection model, etc.) should be used for threat identification.
[0052] The real-time message queue is used to collect and transmit the data to be stored, including:
[0053] The data to be stored is temporarily stored in a ring buffer. The ring buffer has a fixed size. When the data written reaches the end, the old data at the head is automatically overwritten. The producer-consumer model is adopted. The producer writes the data into the ring buffer, and the consumer reads the data from the buffer for processing.
[0054] The UDP protocol is used for data transmission to ensure low latency. In terms of data format, a lightweight message format is used as the standard, including a message header and a message body. The message header contains metadata such as the message type, timestamp, and source node ID, and the message body contains the actual data content. In order to ensure reliability, a simple retransmission mechanism is introduced. When the receiving end detects packet loss, a retransmission request is sent.
[0055] 2. The data partitioning module is configured as follows: the data to be stored is partitioned to obtain multiple pieces of data.
[0056] The data to be stored is partitioned to obtain multiple pieces of data. The following solutions can be used:
[0057] Define a partition function f:D×T→P, where D is the data item, T is the timestamp, and P is the partition identifier set;
[0058] Preliminarily divide the data to be transmitted according to time windows;
[0059] In the same time window, advanced segmentation is performed according to the spatial attributes of the data to be transmitted to obtain the slice data;
[0060] Regularly evaluate the data access frequency and data size of each partition, and increase the spatial partition granularity for partitions with high access frequency or large data volume, and reduce the spatial partition granularity for partitions with low access frequency or small data volume. Specifically, let di be the th data item, ti be its timestamp, and pi be the corresponding partition identifier. The partition function can be expressed as: pi = f(di, ti), where f(di, ti) = hash(time_bucket(ti), spatial_feature(di)) mod N, where N is the total number of partitions, time_bucket(ti) is the mapping of timestamps to specific time windows, spatial_feature(di) is the spatial feature of the extracted data item, and hash represents the hash function used for hash calculation.
[0061] When performing data partitioning, dynamic adjustments can also be made based on the actual workload of each node to avoid overloading some nodes while other nodes are idle, and to reduce the risk of service interruption due to node failure. The specific implementation methods can be:
[0062] Specific implementation
[0063] (1) Initial weight assignment: Assign an initial weight value to each node Usually set to 1;
[0064] (2) Workload monitoring: Continuously monitor the workload of each node, including CPU utilization, memory usage, disk I / O, etc.
[0065] (3) Weight update rule: Regularly adjust the weight value according to the workload of the node. If the load of a node is too high, its weight will be appropriately reduced; otherwise, its weight will be increased.
[0066] (4) Request allocation strategy: Use weighted polling or random selection algorithm to allocate client requests. Nodes with higher weights have a higher probability of being selected.
[0067] Among them, the following methods can be used to update the weight:
[0068] set up is the weight value of node i after the nth adjustment, L iis the workload index of node i (such as CPU utilization), and α is the adjustment factor (0<α<1). The weight update formula is:
[0069] In order to ensure the security of data storage and avoid the impact of data loss, it is also necessary to ensure that the lost data can be quickly restored in the event of failure of one or more nodes, and minimize the impact on normal services during the data recovery process. To this end, in this embodiment, the data partitioning module is also configured to: encode each slice data block and generate additional redundant blocks;
[0070] The distributed architecture module is also configured to: store the encoded data blocks and the redundant blocks in different nodes. When the number of original data blocks is k and the number of redundant blocks is m, a total of k+m blocks are generated. When storing, the encoded data blocks are stored in different nodes in a distributed manner. Even if some nodes fail, data can be recovered from the remaining nodes. Once a node failure is detected, the data recovery process is immediately started to reconstruct the lost data using the redundant blocks on other nodes. Assuming that the original data block set is S={s1,s2,...,sk} and the redundant block set is R={r1,r2,...,rm}, the encoding process can be expressed as: E(S)=S∪R, where E is the encoding function and the decoding process is to use some blocks in S and R to restore the complete original data set S.
[0071] 3. The distributed architecture module is configured as follows: having a multi-node distributed architecture, and each node is used to store partitioned slice data.
[0072] 4. The security storage module is configured to encrypt the data to be stored.
[0073] The data encryption to be stored specifically includes:
[0074] Choose a random initial condition x0 as part of the key;
[0075] Choose a fixed control parameter r as part of the key;
[0076] The chaotic sequence {x n}, x n = r × x n-1 ×(1-x n-1 ), where x n is the iteration variable;
[0077] Extract a sequence from the chaotic sequence as the key sequence K;
[0078] The data to be encrypted is XORed with the key sequence K to obtain the encrypted data.
[0079] That is, let P be the plaintext, C be the ciphertext, and K be the key sequence, then the encryption process can be expressed as: C i =P i ⊕K i , where ⊕ represents the bitwise XOR operation.
[0080] 5. The access control module is configured to: perform fine-grained control on access to stored data and record all operation logs on the data.
[0081] Among them, the following methods can be used to perform fine-grained control on access to stored data:
[0082] Constructing an interaction relationship graph, wherein the interaction relationship graph includes a plurality of relationship nodes and edges, wherein each relationship node represents a user, and each edge represents an interaction relationship between users;
[0083] Use the shortest path algorithm to evaluate the trust relationship between users and determine the user's access rights;
[0084] Dynamically adjust the interaction diagram based on the user's real-time behavior.
[0085] And, when the user is allowed to access, the access control module sends a decryption instruction to the secure storage module; the secure storage module is also configured to: when receiving the decryption instruction, perform an XOR operation on the encrypted data and the key sequence K to restore the original data.
[0086] In addition, in this embodiment, all operation logs on the data are recorded, including:
[0087] configuring a blockchain, wherein each block on the blockchain is linked to a previous block via a hash pointer;
[0088] Configure consensus algorithms for each block, such as the Practical Byzantine Fault Tolerance algorithm;
[0089] A set of operation logs is recorded in each block, and the content of the operation logs includes one or more of the operation type, operation time, operation user and operation object.
[0090] In summary, the distributed network security data storage system of this embodiment uses data security verification, distributed computing, big data processing technology and distributed storage to provide comprehensive technical support for security checks on stored data before storage, efficiency during storage, and security assurance during the storage process. It can better adapt to the complex and changeable network security environment and provide users with more reliable service guarantees.
[0091] In the embodiments provided in the present application, it should be understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, code or any appropriate combination thereof. For hardware implementation, the processor can be implemented in one or more of the following units: application specific integrated circuit (ASIC), digital signal processor (DSP), digital signal processing device (DSPD), programmable logic device (PLD), field programmable gate array (FPGA), processor, controller, microcontroller, microprocessor, other electronic units designed to implement the functions described herein or their combination. For software implementation, part or all of the flow of the embodiment can be completed by instructing the relevant hardware through a computer program. When implemented, the above program can be stored in a computer-readable storage medium or transmitted as one or more instructions or codes on a computer-readable storage medium. Computer-readable storage media include computer storage media and communication media, wherein the communication medium includes any medium that is convenient for transmitting a computer program from one place to another. The storage medium can be any available medium that a computer can access. The computer-readable storage medium can include but is not limited to RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer.
[0092] Finally, it should be noted that the above is only a preferred embodiment of the present application and is not intended to limit the present application. Although the present application has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.
Claims
1. A distributed network security data storage system, characterized in that: It includes a security detection module, a distributed architecture module, a data partitioning module, a security storage module and an access control module; The security detection module is configured to: perform security verification on the received data to be stored, and identify whether the data to be stored has security threats; The data partitioning module is configured to: partition the data to be stored to obtain a plurality of pieces of data; The distributed architecture module is configured as follows: having a multi-node distributed architecture, and each node is used to store partitioned slice data; The security storage module is configured to: encrypt the data to be stored; The access control module is configured to perform fine-grained control on access to stored data and record all operation logs on the data.
2. The distributed network security data storage system according to claim 1, characterized in that: The security verification of the received data to be stored specifically includes: Collect and transmit data to be stored through real-time message queues; Use the stream processing framework to analyze the collected data to be stored in real time to identify potential security threats; When no potential security threat is found, the security check is passed. Otherwise, the data to be stored is returned and a recheck warning is issued to prompt the data to be stored to re-identify the threat.
3. The distributed network security data storage system according to claim 2, characterized in that: The collection and transmission of the data to be stored through the real-time message queue specifically includes: The data to be stored is temporarily stored through a ring buffer; Use UDP protocol for data transmission.
4. The distributed network security data storage system according to claim 1, characterized in that: The data to be stored is partitioned to obtain multiple pieces of data, specifically including: Define a partition function f:D×T→P, where D is the data item, T is the timestamp, and P is the partition identifier set; Preliminarily divide the data to be transmitted according to time windows; In the same time window, advanced segmentation is performed according to the spatial attributes of the data to be transmitted to obtain the slice data; Regularly evaluate the data access frequency and data size of each partition, increase the space partition granularity for partitions with high access frequency or large data size, and reduce the space partition granularity for partitions with low access frequency or small data size.
5. The distributed network security data storage system according to claim 4, characterized in that: The data partitioning module is further configured to: encode each slice data block and generate additional redundant blocks; The distributed architecture module is also configured to: store the encoded data blocks and the redundant blocks in different nodes in a dispersed manner.
6. The distributed network security data storage system according to claim 1, characterized in that: The data encryption for the data to be stored specifically includes: Choose a random initial condition x0 as part of the key; Choose a fixed control parameter r as part of the key; The chaotic sequence {x n }, x n = r × x n-1 ×(1-x n-1 ), where x n is the iteration variable; Extract a sequence from the chaotic sequence as the key sequence K; Perform an XOR operation on the data to be encrypted and the key sequence K to obtain the encrypted data.
7. The distributed network security data storage system according to claim 1, characterized in that: The fine-grained control of access to stored data specifically includes: Constructing an interaction relationship graph, wherein the interaction relationship graph includes a plurality of relationship nodes and edges, wherein each relationship node represents a user, and each edge represents an interaction relationship between users; Use the shortest path algorithm to evaluate the trust relationship between users and determine the user's access rights; Dynamically adjust the interaction diagram based on the user's real-time behavior.
8. The distributed network security data storage system according to claim 6, characterized in that: When the user is allowed to access, the access control module sends a decryption instruction to the secure storage module; the secure storage module is also configured to: when receiving the decryption instruction, perform an XOR operation on the encrypted data and the key sequence K to restore the original data.
9. The distributed network security data storage system according to claim 1, characterized in that: The above-mentioned log records all operations on the data, including: configuring a blockchain, wherein each block on the blockchain is linked to a previous block via a hash pointer; Configure consensus algorithm for each block; A set of operation logs is recorded in each block, and the content of the operation logs includes one or more of the operation type, operation time, operation user and operation object.