Sensitive file retrieval method and system based on white list database

Abstract

The invention belongs to the technical field of file retrieval. According to the sensitive file retrieval method and system based on the white list database, before file retrieval and detection are started, the white list database is queried, and whether a file path is recorded in a record which passes detection of the white list database or not is judged; if the file path exists in the detected record of the white list database, directly skipping a file corresponding to the file path, and entering a detection process of a next file; otherwise, executing strategy detection; after strategy detection is passed, file paths and corresponding strategy identifiers are recorded in a white list database, and files in the white list database are directly skipped during retrieval; according to the method, after the file passes through the detection strategy, the path of the file is recorded in the white list in the database, and when the same detection strategy is executed subsequently, the system can directly skip the recorded file, so that the retrieval efficiency is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of file retrieval technology, and in particular to a sensitive file retrieval method based on a whitelist database, a sensitive file retrieval system based on a whitelist database, a computer device, a computer-readable storage medium and a computer program product. Background Art

[0002] The statements in this section merely provide background art related to the present invention and do not necessarily constitute prior art.

[0003] File retrieval refers to searching for specific pattern content in a given file. In the sensitive information leakage prevention system, file content sensitivity retrieval is one of the core links. As the amount of data grows, the efficiency of file content retrieval directly affects the system performance and user experience. In the traditional file retrieval process, in order to ensure the security of the file, the system usually performs strict policy detection on each file. However, when the number of files is large and they are updated frequently, the performance bottleneck problem caused by repeated detection becomes increasingly prominent, and traditional solutions are unable to cope with this challenge.

[0004] In recent years, office data and Internet transmission have exploded, and the security situation of sensitive data has become increasingly severe. There is an urgent need to identify and protect files containing sensitive data. However, the current file types are becoming increasingly complex, and it takes a lot of time to search for specific types of file content, especially image files, which require a lot of time and computer resources. These complex file types increase the difficulty of file retrieval and seriously affect the speed of file retrieval.

[0005] The commonly used solutions at present are file traversal + file content extraction + feature matching. File traversal refers to the operation of listing all the drive letters, folders and files under the drive letters of the computer according to the current logical structure to obtain the list of all files calculated locally; file content extraction refers to the operation of performing in-depth analysis of the format of a specific file type to obtain the entire content of the file. Because there are many types of calculated files, the extraction of file content is time-consuming and expensive. For example, to extract the content of a picture type, OCR technology is required, which consumes a lot of GPU resources for image processing, with huge resource overhead and a long time consumption; feature matching refers to matching file content based on keywords and regular expressions to query whether the file content has relevant illegal features.

[0006] Existing technologies usually use a direct traversal and detection method to verify the policy of all files. Even if the file content has not changed, it will be repeatedly detected. This method not only wastes computing resources, but also seriously affects the response speed of the system, especially in large enterprises and cloud storage environments, where the retrieval efficiency problem is particularly obvious; the traditional file detection (i.e. reading the file content and analyzing the sensitivity of the file content) process lacks a whitelist mechanism (i.e. skipping non-sensitive files that have been detected), resulting in the need to detect all files for each retrieval, even if the file content has not changed, it will be repeatedly detected; the existing whitelist mechanism is generally static and does not have the ability to be dynamically updated and maintained, making it difficult to adapt to frequent changes in the file system; existing solutions mostly rely on simple caching or file system-level optimization, and fail to effectively use database technology to improve retrieval speed. Summary of the invention

[0007] In order to address the shortcomings of the prior art, the present invention provides a sensitive file retrieval method and system based on a whitelist database. After a file passes the detection strategy, its path is recorded in the whitelist in the database. When the same detection strategy is executed subsequently, the system can directly skip the recorded files, thereby significantly improving the retrieval efficiency.

[0008] In order to achieve the above object, the present invention adopts the following technical solution:

[0009] In a first aspect, the present invention provides a sensitive file retrieval method based on a whitelist database.

[0010] A sensitive file retrieval method based on a whitelist database includes the following processes:

[0011] Before the file retrieval and detection begins, query the whitelist database to determine whether the file path is recorded in the records of the whitelist database that have passed the detection;

[0012] If the file path exists in the record of the whitelist database that has passed the test, the file corresponding to the file path is directly skipped and the next file detection process is entered; otherwise, the policy detection is performed;

[0013] After the policy detection is passed, the file path and the corresponding policy identifier are recorded in the whitelist database, and the files in the whitelist database are directly skipped during retrieval.

[0014] As a further limitation of the first aspect of the present invention, the whitelist database is automatically updated in response to the addition, deletion or content change of files.

[0015] As a further limitation of the first aspect of the present invention, if the detection strategy changes, the whitelist data is automatically adjusted to adapt to the changed detection strategy.

[0016] As a further limitation of the first aspect of the present invention, a whitelist database is queried using a database index and an optimized query statement.

[0017] As a further limitation of the first aspect of the present invention, the processes of file retrieval and file detection are processed in parallel.

[0018] As a further limitation of the first aspect of the present invention, asynchronous IO operations are used during file retrieval and file detection.

[0019] In a second aspect, the present invention provides a sensitive file retrieval system based on a whitelist database.

[0020] A sensitive file retrieval system based on a whitelist database, comprising:

[0021] The whitelist query unit is configured to: query the whitelist database before the file retrieval and detection start, and determine whether the file path is recorded in the records of the whitelist database that have passed the detection;

[0022] The file detection unit is configured to: if the file path exists in the record of the whitelist database that has passed the detection, directly skip the file corresponding to the file path and enter the detection process of the next file; otherwise, perform policy detection;

[0023] The whitelist updating unit is configured to: after the policy detection is passed, record the file path and the corresponding policy identifier into the whitelist database, and directly skip the files in the whitelist database during retrieval.

[0024] In a third aspect, the present invention provides a computer device, comprising: a processor and a computer-readable storage medium;

[0025] a processor adapted to execute a computer program;

[0026] A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the sensitive file retrieval method based on the whitelist database as described in the first aspect of the present invention is implemented.

[0027] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program is suitable for being loaded by a processor and executing the sensitive file retrieval method based on a whitelist database as described in the first aspect of the present invention.

[0028] In a fifth aspect, the present invention provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the sensitive file retrieval method based on a whitelist database as described in the first aspect of the present invention.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] 1. The present invention significantly improves the retrieval speed. By pre-recording the file paths that have passed the detection, the present invention can directly skip these files in subsequent retrieval, avoiding repeated detection processes and reducing the computational burden of the system. Compared with traditional methods, the file retrieval speed of the present invention has been significantly improved, especially when the number of files is large, this improvement is more obvious.

[0031] 2. The present invention reduces system resource consumption. Traditional file retrieval and detection methods require comprehensive detection of each file. Even if the file content has not changed, this repeated detection leads to a large amount of resource waste. The present invention reduces unnecessary computing operations through a whitelist mechanism, thereby reducing the system's CPU, memory and IO resource consumption and improving the overall operating efficiency of the system.

[0032] 3. The present invention improves the system response speed. The present invention reduces repeated operations in the file retrieval process, allowing the system to complete file security detection and management tasks more quickly, thereby improving the system response speed; for users, this optimization significantly reduces waiting time and improves the user experience of the system.

[0033] 4. The present invention enhances the dynamic adaptability of the system. Through the dynamic whitelist management mechanism, the present invention can respond to changes in file content and policy adjustments in a timely manner to ensure that the system is always in the best state. This dynamic adaptability ensures the accuracy and effectiveness of the whitelist, thereby improving the security and reliability of the system.

[0034] 5. The present invention effectively solves many problems in the prior art by combining database technology and whitelist mechanism, significantly improves the efficiency of file retrieval and detection, reduces system resource consumption, improves user experience, and enhances the security and dynamic adaptability of the system.

[0035] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0037] Figure 1 A schematic diagram of a process flow of a sensitive file retrieval method based on a whitelist database provided in Embodiment 1 of the present invention;

[0038] Figure 2 A schematic diagram of a sensitive file retrieval system based on a whitelist database provided in Embodiment 2 of the present invention;

[0039] Figure 3 A schematic diagram of a computer device provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0040] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0041] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0042] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0043] Embodiment 1:

[0044] This implementation proposes a sensitive file retrieval method based on a whitelist database, including:

[0045] (1) Optimization of file detection process.

[0046] Pre-retrieval query: Before the file retrieval and detection begins, the system first queries the database whitelist to determine whether the file path exists in the records that have passed the specific policy;

[0047] Conditional skip detection: If the query result shows that the file has passed the detection of the corresponding policy, the system directly skips the file and enters the detection process of the next file; otherwise, the system performs regular detection and updates the whitelist after passing;

[0048] Database update: When a file passes the policy detection, the system will record the file path and the corresponding policy identifier in the whitelist database for subsequent retrieval and skipping;

[0049] (2) Dynamic whitelist management.

[0050] Automatic update mechanism: In response to the addition, deletion or content change of files, the system will automatically update the whitelist database to ensure the accuracy and validity of the whitelist information; (for example, the whitelist query will use the file's characteristics to calculate the MD5 query. When the file content changes, the MD5 will change, ensuring that the file will not be queried from the whitelist, and ensuring that sensitive information added to the file content will not be skipped because it has been added to the whitelist).

[0051] Policy adjustment adaptability: If the system policy changes (such as adding or modifying a policy), the whitelist will automatically adjust to ensure the correct application and execution of the new policy. (For example: this method associates a file with a whitelisted policy. The original policy is "secret". A file is added to the whitelist because the "secret" keyword is not retrieved. Later, the policy is changed to "confidential". If the file is in the whitelist that meets the "confidential" keyword, it will not be skipped).

[0052] (3) Optimize retrieval speed.

[0053] Efficient database query: Use database indexes and optimized query statements to speed up whitelist matching, reduce waiting time during file detection, and store whitelist association queries in the process memory to increase retrieval speed.

[0054] Parallel processing: Use multi-threaded or multi-process processing technology to parallelize the file retrieval and detection process, further improving the system's processing speed (use independent threads for the storage and operation of the whitelist and use transactions when storing the whitelist to avoid affecting the original business due to the database operation of the whitelist).

[0055] Asynchronous IO operation: In the process of file retrieval and detection, asynchronous IO operation is used to avoid the performance bottleneck caused by synchronous operation and improve the resource utilization of the system (the storage and operation of the whitelist use independent threads and transactions when storing, so as to avoid the database operation of the whitelist affecting the original business).

[0056] like Figure 1 As shown, determine whether it is in the whitelist. If so, the search ends. If not, read the text content to determine whether the file content is sensitive. If so, analyze and quote. If not, update the whitelist based on file characteristics and review strategies, and the search ends.

[0057] Embodiment 2:

[0058] like Figure 2 As shown, this implementation provides a sensitive file retrieval system based on a whitelist database, including:

[0059] The whitelist query unit is configured to: query the whitelist database before the file retrieval and detection start, and determine whether the file path is recorded in the records of the whitelist database that have passed the detection;

[0060] The file detection unit is configured to: if the file path exists in the record of the whitelist database that has passed the detection, directly skip the file corresponding to the file path and enter the detection process of the next file; otherwise, perform policy detection;

[0061] The whitelist updating unit is configured to: after the policy detection is passed, record the file path and the corresponding policy identifier into the whitelist database, and directly skip the files in the whitelist database during retrieval.

[0062] It is understandable that the above-mentioned various units can be separately or all combined into one or several other units to constitute, or some (some) units therein can also be split into multiple smaller units in function to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the function of a unit can also be realized by multiple units, or the function of multiple units can be realized by one unit. In other embodiments of the present application, the system can also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0063] According to another embodiment of the present application, the system described in this embodiment can be constructed, and the method of Example 1 of the present application can be implemented by running a computer program (including program code) capable of executing the steps involved in the corresponding method described in Example 1 on a general computing device such as a computer including processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM). The computer program can be recorded on, for example, a computer-readable recording medium, and loaded into the above-mentioned computing device through the computer-readable recording medium and run therein.

[0064] Embodiment 3:

[0065] like Figure 3 As shown, this implementation provides an electronic device, which includes a processor 1001, a communication interface 1002, and a computer-readable storage medium 1003. The processor 1001, the communication interface 1002, and the computer-readable storage medium 1003 may be connected via a bus or other means.

[0066] Among them, the communication interface 1002 is used to receive and send data, the computer-readable storage medium 1003 can be stored in the memory of the electronic device, the computer-readable storage medium 1003 is used to store a computer program, the computer program includes program instructions, and the processor 1001 is used to execute the program instructions stored in the computer-readable storage medium 1003.

[0067] Processor 1001 (or CPU (Central Processing Unit)) is the computing core and control core of the electronic device, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to implement corresponding method processes or corresponding functions.

[0068] The processor 1001 is configured to execute the following process:

[0069] Before the file retrieval and detection begins, query the whitelist database to determine whether the file path is recorded in the records of the whitelist database that have passed the detection;

[0070] If the file path exists in the record of the whitelist database that has passed the test, the file corresponding to the file path is directly skipped and the next file detection process is entered; otherwise, the policy detection is performed;

[0071] After the policy detection is passed, the file path and the corresponding policy identifier are recorded in the whitelist database, and the files in the whitelist database are directly skipped during retrieval.

[0072] The detailed process is described in Example 1 and will not be repeated here.

[0073] Embodiment 4:

[0074] This implementation provides a computer-readable storage medium (Memory), which is a memory device in an electronic device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the electronic device and, of course, extended storage media supported by the electronic device. The computer-readable storage medium provides a storage space that stores the processing system of the electronic device.

[0075] In addition, the storage space also stores one or more instructions suitable for being loaded and executed by the processor, and these instructions may be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here may be a high-speed RAM memory, or a non-volatile memory, such as at least one disk memory; optionally, it may also be at least one computer-readable storage medium located away from the aforementioned processor.

[0076] In one embodiment, the computer-readable storage medium stores one or more instructions; the processor loads and executes the one or more instructions stored in the computer-readable storage medium to implement the following process:

[0077] Before the file retrieval and detection begins, query the whitelist database to determine whether the file path is recorded in the records of the whitelist database that have passed the detection;

[0078] If the file path exists in the record of the whitelist database that has passed the test, the file corresponding to the file path is directly skipped and the next file detection process is entered; otherwise, the policy detection is performed;

[0079] After the policy detection is passed, the file path and the corresponding policy identifier are recorded in the whitelist database, and the files in the whitelist database are directly skipped during retrieval.

[0080] The detailed process is described in Example 1 and will not be repeated here.

[0081] Embodiment 5:

[0082] The present implementation provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs the following process:

[0083] Before the file retrieval and detection begins, query the whitelist database to determine whether the file path is recorded in the records of the whitelist database that have passed the detection;

[0084] If the file path exists in the record of the whitelist database that has passed the test, the file corresponding to the file path is directly skipped and the next file detection process is entered; otherwise, the policy detection is performed;

[0085] After the policy detection is passed, the file path and the corresponding policy identifier are recorded in the whitelist database, and the files in the whitelist database are directly skipped during retrieval.

[0086] The detailed process is described in Example 1 and will not be repeated here.

[0087] A person skilled in the art can appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0088] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing computer program instructions on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instruction can be stored in a computer-readable storage medium or transmitted by a computer-readable storage medium. The computer instruction can be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (for example, coaxial cable, optical fiber, digital line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data processing device such as a server, a data center, etc. that contains one or more available media integration. Available media can be magnetic media (for example, floppy disk, hard disk, tape), optical media (for example, DVD), or semiconductor media (for example, solid-state drive (Solid State Disk, SSD)) and the like.

[0089] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A sensitive file retrieval method based on a whitelist database, characterized in that: The process includes: Before the file retrieval and detection begins, query the whitelist database to determine whether the file path is recorded in the records of the whitelist database that have passed the detection; If the file path exists in the record of the whitelist database that has passed the test, the file corresponding to the file path is directly skipped and the test process of the next file is entered; Otherwise, perform strategy detection; After the policy detection is passed, the file path and the corresponding policy identifier are recorded in the whitelist database, and the files in the whitelist database are directly skipped during retrieval.

2. The sensitive file retrieval method based on the whitelist database as claimed in claim 1, characterized in that: Automatically update the whitelist database in response to the addition, deletion or content change of files.

3. The sensitive file retrieval method based on a whitelist database as claimed in claim 1 or 2, characterized in that: If the detection strategy changes, the whitelist data is automatically adjusted to adapt to the changed detection strategy.

4. The sensitive file retrieval method based on a whitelist database as claimed in claim 1 or 2, characterized in that: Use database indexes and optimized query statements to query the whitelist database.

5. The sensitive file retrieval method based on a whitelist database as claimed in claim 1 or 2, characterized in that: The processes of file retrieval and file detection are processed in parallel.

6. The sensitive file retrieval method based on the whitelist database as claimed in claim 5, characterized in that: During file retrieval and file detection, asynchronous IO operations are used.

7. A sensitive file retrieval system based on a whitelist database, characterized in that it includes: The whitelist query unit is configured to: query the whitelist database before the file retrieval and detection start, and determine whether the file path is recorded in the records of the whitelist database that have passed the detection; The file detection unit is configured to: if the file path exists in the record of the whitelist database that has passed the detection, directly skip the file corresponding to the file path and enter the detection process of the next file; Otherwise, perform strategy detection; The whitelist updating unit is configured to: after the policy detection is passed, record the file path and the corresponding policy identifier into the whitelist database, and directly skip the files in the whitelist database during retrieval.

8. A computer device, characterized in that: include: a processor and a computer readable storage medium; a processor adapted to execute a computer program; A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the sensitive file retrieval method based on the whitelist database as described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which is suitable for being loaded by a processor and executing the sensitive file retrieval method based on a whitelist database as described in any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product includes a computer program, and when the computer program is executed by a processor, it implements the sensitive file retrieval method based on the whitelist database as described in any one of claims 1 to 6.

Images