Dynamic write-in data confidentiality detection method in distributed storage system

Abstract

The invention particularly relates to a dynamic write-in data confidentiality detection method in a distributed storage system. According to the dynamic write-in data confidentiality detection method in the distributed storage system, a detection agent is embedded in a data write-in path of a storage node, and an unpersistent data stream is intercepted in real time; the data sensitivity is dynamically evaluated by combining streaming analysis and a multi-modal analysis engine, and a security file is directly written into a storage node; after the screened sensitive files are output through a dynamic analysis cluster DAC, streaming multi-modal analysis is carried out; and in combination with context information and a central strategy library, carrying out hierarchical interception processing on detected problem files. According to the dynamic write-in data confidentiality detection method in the distributed storage system, sensitive content is intercepted before data persistence, normal write-in performance is not affected, hysteresis of traditional post scanning is broken through, high performance and high precision are achieved at the same time, and different safety compliance requirements can be met.

Description

Technical Field

[0001] This invention relates to the field of data security technology, and in particular to a method for detecting the confidentiality of dynamically written data in a distributed storage system. Background Technology

[0002] With the rapid development of cloud computing and big data technologies, distributed storage systems (such as Ceph, HDFS, and S3) are widely used in fields with strict data security requirements, such as finance, healthcare, and government, due to their advantages of high scalability, high reliability, and low cost. However, how to detect and prevent the unauthorized storage of sensitive information (such as personal privacy, trade secrets, and financial data) in real time during the data writing process has become a critical issue that urgently needs to be addressed.

[0003] Currently, mainstream sensitive data detection solutions can be mainly divided into the following three categories:

[0004] 1) Post-storage scanning solution: Relies on scheduled tasks to scan data already written to disk (such as ClamAV, data classification tools), but there is a detection lag, and sensitive data may have been stored in the storage system for a long time, resulting in compliance risks.

[0005] 2) Gateway-based detection solution: Deploy a data loss prevention (DLP) gateway between the client and the storage cluster. However, this type of solution often becomes a performance bottleneck and cannot detect metadata inside the storage system (such as user permissions, storage location, etc.), resulting in a high false alarm rate.

[0006] 3) Static rule matching scheme: Based on regular expressions or keyword matching to detect sensitive content, but it is difficult to deal with encrypted, encoded or unstructured data (such as text in images or PDFs), and lacks the ability to adapt to business scenarios.

[0007] In addition, existing technologies generally face the following challenges:

[0008] 1) Insufficient real-time performance: Traditional detection is usually performed after the data is written, and cannot intercept illegal content before it is persisted.

[0009] 2) Limited scalability: The centralized detection architecture struggles to handle the high-concurrency write demands of distributed storage, leading to performance degradation.

[0010] 3) High false alarm rate: Mechanical rule matching is prone to misclassifying legitimate data as sensitive information (such as ID card test samples in scientific research data).

[0011] Based on the above, this invention proposes a dynamic data write confidentiality detection method in a distributed storage system. The aim is to provide a low-latency, high-precision, and scalable dynamic data write confidentiality detection method that can analyze the content in real time during the data write process and dynamically adjust the detection strategy in combination with the business context (such as user role and storage path), thereby meeting strict data security compliance requirements while ensuring performance. Summary of the Invention

[0012] To overcome the shortcomings of existing technologies, this invention provides a simple and efficient method for detecting the confidentiality of dynamically written data in a distributed storage system.

[0013] This invention is achieved through the following technical solution:

[0014] A method for detecting the confidentiality of dynamically written data in a distributed storage system includes the following steps:

[0015] Step S1: Embed a detection agent in the data write path of the storage node to intercept non-persistent data streams in real time;

[0016] Step S2: Combining streaming parsing and multimodal analysis engines, dynamically assess data sensitivity and write secure files directly to storage nodes; after the filtered sensitive files are output by the dynamic analysis cluster DAC, streaming multimodal analysis is performed; combined with context information and the central policy library, problematic files are subjected to graded interception processing.

[0017] The central policy repository is configured with predefined grading standards for problem files and corresponding processing methods.

[0018] The context information includes user identity, storage path, and file type, dynamically adjusting detection intensity and response actions to generate tiered response instructions.

[0019] In step S1, a lightweight detection agent DAgent is embedded in the I / O path of the storage node to achieve real-time data scanning with nanosecond-level latency, avoiding the performance bottleneck of traditional gateway-type detection, and supporting multi-protocol adaptation, including S3 API protocol, POSIX standard and NFS protocol, to ensure the execution of unified security policies under different storage interfaces.

[0020] In step S2, the streaming parsing and multimodal analysis engine uses progressive parsing technology to perform streaming detection on data that has not been fully written, prioritizing the analysis of file headers, metadata, and key content to reduce waiting time.

[0021] By combining rule engines, machine learning models, and dynamic sandbox detection, accurate identification of structured and unstructured data can be achieved.

[0022] In step S2, the detected problematic files are subjected to tiered interception processing.

[0023] If malware is detected, the network connection is dropped via kernel eBPF;

[0024] If the problematic file is a custom-defined ordinary sensitive data, it will be discarded and a permission error will be returned.

[0025] If the problematic file is a user-defined suspected data file that requires administrator review, it will be written to the encrypted temporary storage area.

[0026] In step S2, source watermarking technology is used to inject an invisible identifier into the released document and record the user and time of operation in order to achieve leakage tracking.

[0027] The dynamic data write confidentiality detection system in this distributed storage system adopts a distributed embedded detection framework and also includes a streaming parsing and multimodal analysis engine, a context-aware policy engine, a central policy library, and a real-time response module.

[0028] The distributed embedded detection architecture embeds a lightweight detection agent DAgent in the I / O path of the storage node to achieve real-time data scanning with nanosecond latency, avoiding the performance bottleneck of traditional gateway-type detection, and supports multi-protocol adaptation, including S3 API protocol, POSIX standard and NFS protocol, to ensure the unified execution of security policies under different storage interfaces.

[0029] The streaming parsing and multimodal analysis engine is responsible for using progressive parsing technology to perform streaming detection on data that has not been fully written.

[0030] The context-aware strategy engine is responsible for obtaining the context information of sensitive files, including user identity, storage path, and file type.

[0031] The central policy library is configured with predefined hierarchical standards for problem files and corresponding processing methods;

[0032] The real-time response module is responsible for combining contextual information and the central policy library to perform hierarchical interception of detected problematic files, and uses source watermarking technology to inject invisible identifiers into released files, and records the user and time of operation to enable leakage tracking.

[0033] A dynamic data write security detection device in a distributed storage system is characterized by comprising a memory and a processor; the memory is used to store a computer program, and the processor is used to execute the computer program to implement the above-described method steps.

[0034] A readable storage medium, characterized in that: a computer program is stored on the readable storage medium, and the computer program, when executed by a processor, implements the above-described method steps.

[0035] The beneficial effects of this invention are: the dynamic data write confidentiality detection method in the distributed storage system intercepts sensitive content before data persistence without affecting normal write performance, overcomes the lag of traditional post-scanning, achieves both high performance and high precision, and can adapt to different security and compliance requirements. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Appendix Figure 1 This is a schematic diagram of the dynamic data write confidentiality detection method in the distributed storage system of the present invention. Detailed Implementation

[0038] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions in the embodiments of this invention will be clearly and completely described below in conjunction with the embodiments of this invention. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0039] The method for detecting the confidentiality of dynamically written data in this distributed storage system includes the following steps:

[0040] Step S1: Embed a detection agent in the data write path of the storage node to intercept non-persistent data streams in real time;

[0041] Step S2: Combining streaming parsing and multimodal analysis engines, dynamically assess data sensitivity and write secure files directly to storage nodes; after the filtered sensitive files are output by the dynamic analysis cluster DAC, streaming multimodal analysis is performed; combined with context information and the central policy library, problematic files are subjected to graded interception processing.

[0042] The central policy repository is configured with predefined grading standards for problem files and corresponding processing methods.

[0043] The context information includes user identity (e.g., department, permissions), storage path (e.g., / confidential / ), and file type, dynamically adjusting detection intensity and response actions to generate tiered response instructions; for example, medical images are scanned only for DICOM metadata, while financial data is scanned in strict mode (full content detection), such as during non-working hours. Detection rules can be pulled from the central policy repository on demand via gRPC, and hot rule updates are supported.

[0044] In step S1, a lightweight detection agent DAgent is embedded in the I / O path of the storage node (such as Ceph OSD, HDFSDataNode) to achieve real-time data scanning with nanosecond latency, avoiding the performance bottleneck of traditional gateway-type detection, and supporting multi-protocol adaptation, including S3 API protocol, POSIX standard and NFS protocol, to ensure the execution of unified security policies under different storage interfaces.

[0045] In step S2, the streaming parsing and multimodal analysis engine uses progressive parsing technology to perform streaming detection on data that has not been fully written, prioritizing the analysis of file headers, metadata, and key content to reduce waiting time.

[0046] By combining rule engines, machine learning models, and dynamic sandbox detection, it achieves accurate identification of structured data (such as database tables) and unstructured data (such as PDFs / images).

[0047] In step S2, the detected problematic files are subjected to tiered interception processing.

[0048] If malware is detected, the network connection is dropped via kernel eBPF;

[0049] If the problematic file is a custom-defined ordinary sensitive data, it will be discarded and a permission error will be returned.

[0050] If the problematic file is a user-defined suspected data file that requires administrator review, it will be written to the encrypted temporary storage area.

[0051] In step S2, source watermarking technology is used to inject invisible identifiers (such as PDF steganography) into the released files and record the user and time of operation in order to achieve leakage tracking.

[0052] The dynamic data write confidentiality detection system in this distributed storage system adopts a distributed embedded detection framework and also includes a streaming parsing and multimodal analysis engine, a context-aware policy engine, a central policy library, and a real-time response module.

[0053] The distributed embedded detection architecture embeds a lightweight detection agent DAgent in the I / O path of storage nodes (such as Ceph OSD and HDFS DataNode) to achieve real-time data scanning with nanosecond latency, avoiding the performance bottleneck of traditional gateway-based detection, and supports multi-protocol adaptation, including S3 API protocol, POSIX standard and NFS protocol, to ensure the unified execution of security policies under different storage interfaces.

[0054] The streaming parsing and multimodal analysis engine is responsible for using progressive parsing technology to perform streaming detection on data that has not been fully written.

[0055] The context-aware policy engine is responsible for obtaining the context information of sensitive files, including user identity (such as department, permissions), storage path (such as / confidential / ) and file type;

[0056] The central policy library is configured with predefined hierarchical standards for problem files and corresponding processing methods;

[0057] The real-time response module is responsible for combining contextual information and the central policy library to perform hierarchical interception of detected problematic files, and uses source watermarking technology to inject invisible identifiers (such as PDF steganography) into released files, and records the user and time of operation to enable leakage tracking.

[0058] The dynamic data write security detection device in the distributed storage system includes a memory and a processor; the memory is used to store a computer program, and the processor is used to execute the computer program to implement the above-described method steps.

[0059] The readable storage medium stores a computer program that, when executed by a processor, implements the above-described method steps.

[0060] Compared with existing technologies, the dynamic data write confidentiality detection method in this distributed storage system has the following characteristics:

[0061] First, by deeply embedding security detection into the I / O stack of the storage system, it overcomes the lag of traditional post-scanning, intercepts sensitive content before data persistence, does not affect normal write performance, and effectively solves the problem of real-time confidentiality detection during the write phase in distributed storage.

[0062] Secondly, by balancing lightweight initial screening with dynamic depth detection, high performance and high precision are achieved.

[0063] Third, it is a detection framework that adapts to multiple protocols (S3 / POSIX / block devices), which can intelligently adjust the detection intensity based on the user, data location, and service tag. It supports dynamic policy loading and low-latency response, and has real-time and high-precision characteristics.

[0064] Fourth, for the first time, non-content features such as user roles and storage locations are incorporated into the detection decision, reducing the false alarm rate, supporting non-stop updates to detection rules, and adapting to new sensitive data patterns;

[0065] Fifth, the programmable response mechanism supports multi-level response strategies such as interception, isolation, and auditing, and can adapt to different security and compliance requirements.

[0066] The foregoing has provided a detailed description of a method for detecting the confidentiality of dynamically written data in a distributed storage system, as exemplified by this invention. This section uses specific examples to illustrate the principles and implementation methods of the invention. These examples are merely for the purpose of helping to understand the core ideas of this invention. All other embodiments obtained by those skilled in the art without creative effort, without departing from the principles of this invention, should fall within the scope of protection of this invention.

Claims

1. A method for detecting the confidentiality of dynamically written data in a distributed storage system, characterized in that: Includes the following steps: Step S1: Embed a detection agent in the data write path of the storage node to intercept non-persistent data streams in real time; Step S2: Combining streaming parsing and multimodal analysis engines, dynamically assess data sensitivity and write secure files directly to storage nodes; after the filtered sensitive files are output by the dynamic analysis cluster DAC, streaming multimodal analysis is performed; combined with context information and the central policy library, problematic files are subjected to graded interception processing. The central policy repository is configured with predefined grading standards for problem files and corresponding processing methods.

2. The method for detecting the confidentiality of dynamically written data in a distributed storage system according to claim 1, characterized in that: The context information includes user identity, storage path, and file type, dynamically adjusting detection intensity and response actions to generate tiered response instructions.

3. The method for detecting the confidentiality of dynamically written data in a distributed storage system according to claim 1, characterized in that: In step S1, a lightweight detection agent DAgent is embedded in the I / O path of the storage node to achieve real-time data scanning with nanosecond-level latency, avoiding the performance bottleneck of traditional gateway-type detection, and supporting multi-protocol adaptation, including S3 API protocol, POSIX standard and NFS protocol, to ensure the execution of unified security policies under different storage interfaces.

4. The method for detecting the confidentiality of dynamically written data in a distributed storage system according to claim 1, characterized in that: In step S2, the streaming parsing and multimodal analysis engine uses progressive parsing technology to perform streaming detection on data that has not been fully written, prioritizing the analysis of file headers, metadata, and key content to reduce waiting time. By combining a rule engine, machine learning models, and dynamic sandbox detection, the system can identify structured and unstructured data.

5. The method for detecting the confidentiality of dynamically written data in a distributed storage system according to claim 1, characterized in that: In step S2, the detected problematic files are subjected to tiered interception processing. If malware is detected, the network connection is dropped via kernel eBPF; If the problematic file is a custom-defined ordinary sensitive data, it will be discarded and a permission error will be returned. If the problematic file is a user-defined suspected data file that requires administrator review, it will be written to the encrypted temporary storage area.

6. The method for detecting the confidentiality of dynamically written data in a distributed storage system according to claim 5, characterized in that: In step S2, source watermarking technology is used to inject an invisible identifier into the released document and record the user and time of operation in order to achieve leakage tracking.

7. A dynamic data write security detection system in a distributed storage system, characterized in that: The method for implementing any one of claims 1 to 6 employs a distributed embedded detection framework and further includes a streaming parsing and multimodal analysis engine, a context-aware strategy engine, a central strategy library, and a real-time response module. The distributed embedded detection architecture embeds a lightweight detection agent DAgent in the I / O path of the storage node to achieve real-time data scanning with nanosecond latency, avoiding the performance bottleneck of traditional gateway-type detection, and supports multi-protocol adaptation, including S3 API protocol, POSIX standard and NFS protocol, to ensure the unified execution of security policies under different storage interfaces. The streaming parsing and multimodal analysis engine is responsible for using progressive parsing technology to perform streaming detection on data that has not been fully written. The context-aware strategy engine is responsible for obtaining the context information of sensitive files, including user identity, storage path, and file type. The central policy library is configured with predefined hierarchical standards for problem files and corresponding processing methods; The real-time response module is responsible for combining contextual information and the central policy library to perform hierarchical interception of detected problematic files, and uses source watermarking technology to inject invisible identifiers into released files, and records the user and time of operation to enable leakage tracking.

8. A device for detecting the confidentiality of dynamically written data in a distributed storage system, characterized in that: It includes a memory and a processor; the memory is used to store a computer program, and the processor is used to execute the computer program to implement the steps of the method as described in any one of claims 1 to 6. A readable storage medium, characterized in that: The readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 6.

Images