A java-based local sensitive data scanning method

Abstract

This invention discloses a Java-based method for scanning local sensitive data, belonging to the field of sensitive data detection technology. This method achieves efficient, accurate, and comprehensive detection of sensitive information in various file types, including local text, Office documents, and compressed files, through five steps: initializing scan configuration, file traversal filtering, adaptive parsing of multiple file formats, asynchronous multi-threaded sensitive data matching, and generating a scan report. It effectively solves problems such as poor cross-platform compatibility, rigid rules, incomplete format parsing, and low scanning performance in existing technologies. It can be widely applied in scenarios such as office automation, government processing, and software engineering, and has advantages such as easy deployment, strong scalability, and high security.

Description

Technical Field

[0001] This invention relates to the field of sensitive data detection and processing technology, specifically to a Java-based local sensitive data scanning method. It is applicable to scanning and detection tasks in various scenarios such as document detection, data standardization, and data anomaly marking, and is particularly suitable for detecting sensitive information in multi-source heterogeneous text, forms, compressed files, and other files. It can be widely applied in practical industrial scenarios such as office automation, software engineering, and government processing. This invention belongs to the category of sensitive data detection invention patents and meets the technical requirements stipulated in Article 2, Paragraph 2 of the Patent Law of the People's Republic of China. This invention does not involve abstract intellectual activity rules and methods, and has strong practicality and outstanding innovation. Background Technology

[0002] With the formal implementation of laws and regulations such as the Data Security Law and the Personal Information Protection Law, enterprises and individuals have an increasingly urgent need for compliant management of locally stored sensitive data (such as ID card numbers, bank card numbers, mobile phone numbers, keys, passwords, etc.). Locally stored sensitive data is usually scattered and stored in various formats such as text files, configuration files, Office documents, and compressed files. It is characterized by its scattered distribution, diverse formats, and strong concealment, making manual screening extremely inefficient and prone to omissions.

[0003] Most existing sensitive data scanning tools suffer from the following shortcomings: 1) Poor cross-platform compatibility, relying on specific operating system environments and unable to adapt to local environments on multiple platforms such as Windows, Linux, and macOS; 2) Fixed scanning rules, not supporting user-defined sensitive data types, resulting in insufficient adaptability; 3) Weak ability to parse complex file formats such as compressed files and nested documents, leading to scanning blind spots; 4) Low scanning performance, with excessive memory consumption and a tendency to crash when scanning large files in batches; 5) Lack of a standardized scanning report generation mechanism, failing to intuitively display key information such as the distribution of sensitive data and risk levels.

[0004] The Java language boasts cross-platform compatibility, object-oriented features, and a rich ecosystem, making it an effective solution for local sensitive data scanning. However, current Java-based local sensitive data scanning solutions lack a comprehensive system for rule matching, file parsing, performance optimization, and report generation, making it difficult to meet the enterprise-level needs for comprehensive, efficient, and accurate scanning of local sensitive data. Therefore, developing a high-performance Java local sensitive data scanning method that is adaptable to multiple platforms, supports custom rules, provides full-format parsing, and addresses a pressing technical challenge in this field. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a Java-based method, system, and storage medium for scanning local sensitive data. Through dynamic rule loading, adaptive parsing of multi-format files, asynchronous batch scanning, multi-level caching optimization, and visualization report generation, it achieves comprehensive, efficient, and accurate scanning of local sensitive data, solving the technical problems of poor cross-platform compatibility, rigid rules, incomplete format parsing, and low performance in existing technologies.

[0006] A Java-based method for scanning local sensitive data according to an embodiment of the present invention includes the following steps: S1. Initialize the scan configuration. Based on Java scan, load the configuration file to obtain the scan root directory, exclusion directory rules, sensitive data rule library, and file format whitelist. The sensitive data rule library has built-in preset rules and also supports users to customize sensitive data matching rules in YAML / JSON format. The rule types include regular expressions, keyword matching, and validation algorithms.

[0007] S2. File Traversal and Filtering: Recursively scan all files in the root directory using the Java File Traversal API, filter paths that do not need to be scanned based on exclusion directory rules, filter files to be scanned based on file format whitelists, and generate a list of files to be scanned.

[0008] S3. Adaptive parsing of multiple file formats is based on a Java file parsing toolset. It adaptively calls the corresponding parsing module according to the file extension of the file to be scanned to extract the plain text content in the file. The parsing modules include a text file parsing module, an Office document parsing module, and a compressed file parsing module.

[0009] S4. Asynchronous multi-threaded sensitive data matching allocates the files to be scanned to a Java thread pool and uses a multi-level matching algorithm to scan the text content in parallel.

[0010] S5. Scan Report Generation: After the scan is completed, an HTML / Excel format scan report is generated based on the Java reporting engine. The report includes sensitive data statistics, risk distribution, document list, data details, and optimization suggestions. Attached Figure Description

[0011] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 illustrates a Java-based method for scanning local sensitive data proposed in this invention. Detailed Implementation

[0012] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0013] Referring to Figure 1, a Java-based method for scanning local sensitive data includes the following steps: S1. Initialize the scan configuration. Based on Java scan, load the configuration file to obtain the scan root directory, exclusion directory rules, sensitive data rule library, and file format whitelist. The sensitive data rule library has built-in preset rules and also supports users to customize sensitive data matching rules in YAML / JSON format. The rule types include regular expressions, keyword matching, and validation algorithms.

[0014] S2. File Traversal and Filtering: Recursively scan all files in the root directory using the Java File Traversal API, filter paths that do not need to be scanned based on exclusion directory rules, filter files to be scanned based on file format whitelists, and generate a list of files to be scanned.

[0015] S3. Adaptive parsing of multiple file formats is based on a Java file parsing toolset. It adaptively calls the corresponding parsing module according to the file extension of the file to be scanned to extract the plain text content in the file. The parsing modules include a text file parsing module, an Office document parsing module, and a compressed file parsing module.

[0016] S4. Asynchronous multi-threaded sensitive data matching allocates the files to be scanned to a Java thread pool and uses a multi-level matching algorithm to scan the text content in parallel.

[0017] S5. Scan Report Generation: After the scan is completed, an HTML / Excel format scan report is generated based on the Java reporting engine. The report includes sensitive data statistics, risk distribution, document list, data details, and optimization suggestions.

[0018] Optionally, step S1 includes the following steps: S11. Configure the scan directory; S12. Configure sensitive data matching rules.

[0019] Optionally, step S3 includes the following steps: S31. Single file parsing: supports txt, csv, xls, xlsx, doc, and docx formats; S32. Compressed package parsing: supports ZIP, RAR, and 7Z formats, and adopts streaming decompression technology to scan while decompressing, avoiding large compressed packages from occupying a lot of disk space.

[0020] Optionally, step S4 includes the following steps: S41. Asynchronous multithreading; S42. Data sensitivity algorithm verification. Example 1

[0021] S1. Initialize the configuration scan by loading the application.yml configuration file, setting the scan root directory to D:\data, the excluded directory to D:\data\system, and the file whitelist to include txt, csv, xlsx, docx, and zip; load the sensitive data rule library, with built-in rules for ID cards, mobile phone numbers, and bank cards, and user-defined rules: regular expressions for enterprise tax numbers and secretKey.

[0022] S2. File traversal: The directory is recursively traversed using Java Files, filtering out excluded paths, selecting 120 files to be scanned, and generating a file list.

[0023] S3. File parsing directly reads text from txt and csv files; uses POI to parse docx files; uses easyexcel to parse xlsx files; and uses streaming decompression on zip archives, parsing nested files layer by layer to extract all text content.

[0024] S4. Multi-threaded matching starts a thread pool that adapts to the number of CPU cores, and distributes the file to 10 worker threads; first, it quickly filters by keywords, then locates the data by regular expression matching, and finally verifies the phone number by regular expression to confirm whether there is any illegal data in the file.

[0025] S5. Report Generation: Generates an HTML scan report, which calculates the total number of sensitive data, the number of high-risk files, the percentage of each data type, and marks the risk level.

Claims

1. A Java-based method for scanning local sensitive data, characterized in that, Includes the following steps: S1. Initialize scan configuration: Based on Java scan, load the configuration file to obtain the scan root directory, exclusion directory rules, sensitive data rule library, and file format whitelist; among them, the sensitive data rule library has built-in preset rules, and also supports users to customize sensitive data matching rules through YAML / JSON format. The rule types include regular expressions, keyword matching, and validation algorithms. S2. File Traversal and Filtering: Recursively scan all files in the root directory using the Java file traversal API, filter paths that do not need to be scanned according to exclusion directory rules, filter files to be scanned according to file format whitelist, and generate a list of files to be scanned; S3. Adaptive parsing of multiple file formats: Based on the Java file parsing toolset, it adaptively calls the corresponding parsing module according to the file extension of the file to be scanned to extract the plain text content in the file; the parsing modules include text file parsing module, Office document parsing module, and compressed file parsing module; S4. Asynchronous Multithreaded Sensitive Data Matching: The file to be scanned is allocated to a Java thread pool, and a multi-level matching algorithm is used to scan the text content in parallel; S5. Scan Report Generation: After the scan is completed, an HTML / Excel format scan report is generated based on the Java reporting engine. The report includes sensitive data statistics, risk distribution, file list, data details, and optimization suggestions.

2. The Java-based method for scanning sensitive local data according to claim 1, characterized in that, Step S1 includes the following steps: S11. Configure the scan directory; S12. Configure sensitive data matching rules.

3. The Java-based method for scanning sensitive local data according to claim 1, characterized in that, Step S3 includes the following steps: S31. Single file parsing: Supports txt, csv, xls, xlsx, doc, and docx formats; S32. Compressed Package Parsing: Supports ZIP, RAR, and 7Z formats, and uses streaming decompression technology to scan while decompressing, avoiding the large disk space occupied by decompressing large compressed packages.

4. The Java-based method for scanning sensitive local data according to claim 1, characterized in that, The verification algorithm in step S4 includes: S41. Asynchronous multithreading; S42. Data sensitivity algorithm verification.

5. A Java-based local sensitive data scanning system, characterized in that, The method is used to implement the method of claim 1. The method can be applied to multiple industries such as office automation, software engineering, and government affairs processing. It is compatible with the detection of various types of files such as documents, forms, and compressed packages. It has strong cross-platform compatibility, flexible and scalable rules, low deployment cost, low operation threshold, and strong reusability.

Images