Sensitive information identification method and device, electronic equipment and storage medium
By performing file and text sampling and detection in cloud storage services, massive unstructured sensitive personal information can be efficiently identified, which solves the problem of low recognition efficiency of sensitive information in cloud storage, and achieves comprehensive and accurate identification of PB-level data, while avoiding the problem of excessive resource occupation.
Patent Information
- Application Number
- CN202510357053.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-25
- Publication Date
- 2025-06-27
AI Technical Summary
How to efficiently identify massive amounts of unstructured sensitive personal information in cloud storage services and reduce the risk of data leakage.
By performing discrete sampling detection on both file and text, the amount of data identified by a single sensitive information is reduced. The specific steps include obtaining the target bucket and identity credentials, traversing the storage directory, sampling files and texts, and finally identifying sensitive information on the target detection text.
It realizes efficient and sensitive information identification of PB-level cloud storage data, ensures the comprehensiveness and accuracy of identification, and avoids excessive use of processing resources, ensuring the stability of online business performance.
Smart Images
Figure CN120217441A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data security technology, and in particular to a sensitive information identification method, device, electronic device and storage medium. Background Art
[0002] With the development of cloud computing and big data technology, Object Storage Service (OSS) has gradually become the mainstream storage medium for unstructured data. In the Internet era, a large amount of sensitive personal information exists in the form of unstructured data, such as ID card photos, credit reports, etc. How to identify these unstructured sensitive personal information in order to protect them and reduce the risk of data leakage has become a security issue that needs to be solved urgently. Summary of the invention
[0003] The present application provides a sensitive information identification method, device, electronic device and storage medium, which can improve the accuracy and recognition performance of sensitive information identification. The technical solution is as follows:
[0004] According to one aspect of the present application, a sensitive information identification method is provided, the method comprising:
[0005] Obtain the target storage bucket where the file to be detected is located in the object storage service and the target identity credential required to access the target storage bucket;
[0006] Accessing the target storage bucket based on the target identity credential, and traversing multiple storage directories in the target storage bucket to determine a file directory storing files from the multiple storage directories;
[0007] Perform file sampling processing on the stored files in the file directory to obtain the target detection file to be detected;
[0008] Performing text sampling processing on the target detection file to obtain the target detection text to be detected;
[0009] Sensitive information is identified on the target detection text to obtain a sensitive information identification result.
[0010] According to another aspect of the present application, a sensitive information identification device is provided, the device comprising:
[0011] The first acquisition module is used to obtain the target storage bucket where the file to be detected is located in the object storage service and the target identity credential required to access the target storage bucket;
[0012] A first determination module, configured to access the target storage bucket based on the target identity credential, and traverse multiple storage directories in the target storage bucket to determine a file directory storing files from the multiple storage directories;
[0013] A first sampling module, configured to perform file sampling processing on the stored files in the file directory to obtain a target detection file to be detected;
[0014] A second sampling module, configured to perform text sampling processing on the target detection file to obtain a target detection text to be detected;
[0015] A first recognition module, configured to perform sensitive information recognition on the target detection text to obtain a sensitive information recognition result.
[0016] According to one aspect of the present application, there is provided an electronic device, including: a processor and a memory storing a program, the program including instructions, and the instructions, when executed by the processor, cause the processor to execute the sensitive information recognition method as described above.
[0017] According to another aspect of the present application, there is provided a non-transitory computer-readable storage medium storing computer instructions, and the computer instructions are used to cause the computer to execute the sensitive information recognition method as described above.
[0018] According to another aspect of the present application, there is provided a computer program product, the computer program product including computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned sensitive information recognition method.
[0019] The beneficial effects brought by the technical solutions provided in the embodiments of the present application at least include:
[0020] For PB-level cloud storage data (or cloud storage files) in OSS, discrete sampling detection is performed in two dimensions: file sampling and text sampling, to reduce the amount of data for sensitive information identification in a single pass, enabling the detection of PB-level data without consuming excessive processing resources; and the sampling strategies in the two dimensions can also ensure the discreteness of the sampled data, thereby ensuring the comprehensiveness and accuracy of sensitive information identification. Moreover, by reasonably selecting the target detection text to be finally detected through an extraction strategy, the detection of PB-level data volume can be achieved while avoiding affecting the performance of other online services; and by comprehensively using methods such as regular expressions, sensitive information features, and sensitive information recognition models for sensitive information identification, the accuracy of sensitive information identification can be improved; in addition, for files that do not contain the three elements, a sensitive file feature library is also established to match them, which can ensure the comprehensiveness of sensitive information identification. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In the following description of exemplary embodiments with reference to the accompanying drawings, more details, features, and advantages of the present application are disclosed. In the drawings:
[0022] Figure 1 A flowchart of a method for identifying sensitive information according to an exemplary embodiment of the present application is shown;
[0023] Figure 2 A flowchart of another method for identifying sensitive information according to an exemplary embodiment of the present application is shown;
[0024] Figure 3 A schematic structural diagram of a device for identifying sensitive information provided by an embodiment of the present application;
[0025] Figure 4 A block diagram of the structure of an exemplary electronic device capable of implementing the embodiments of the present application is shown. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0026] Embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Instead, these embodiments are provided to more thoroughly and completely understand the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not used to limit the protection scope of the present application.
[0027] It should be understood that the various steps recited in the method embodiments of the present application can be executed in a different order and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this regard.
[0028] As used herein, the term "including" and its variations are open-ended, that is, "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the following description. It should be noted that the concepts such as "first", "second", etc. mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units. It should be noted that the modifications of "one" and "multiple" mentioned in this application are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise clearly indicated in the context, it should be understood as "one or more". The names of the messages or information exchanged between multiple devices in the embodiments of this application are only for illustrative purposes and are not used to limit the scope of these messages or information.
[0029] The solution of the present invention will be described below with reference to the accompanying drawings. The technical solutions provided by the embodiments of the present invention will be described in detail through specific embodiments and their application scenarios.
[0030] Please refer to Figure 1 , which shows a flowchart of a sensitive information recognition method according to an exemplary embodiment of the present application. This method will be described by taking its application to an electronic device as an example. As Figure 1 shown, this method includes:
[0031] Step 101, obtain the target storage bucket where the file to be detected is located in the object storage service and the target identity credential required to access the target storage bucket.
[0032] Among them, the object storage service is a distributed object storage service, which can store and manage unstructured data in the form of objects. Common unstructured data includes documents, pictures, audio, video, etc. (collectively referred to as "files" in this application). This application mainly relates to how to identify sensitive information in files (unstructured data) stored in OSS. In the object storage service, a bucket is the smallest storage unit of OSS, and different files can be stored in different directories under this unit. In one possible implementation, if it is necessary to access a file stored in OSS and identify sensitive information in it, an identity credential for accessing the file is required, similar to an account password. That is, first obtain the target storage bucket where the file to be detected is located in the object storage service, and the target identity credential required to access the target storage bucket, so as to access the target storage bucket based on the target identity credential.
[0033] Step 102: Access the target storage bucket based on the target identity credential, and traverse multiple storage directories in the target storage bucket to determine the file directory storing files from the multiple storage directories.
[0034] Since different data under the target storage bucket are stored in different directories respectively, the storage directories are irregular, the directory levels are not clear, there are deep directory structures, and some directories may store files while some do not. In order to identify the sensitive information of the stored files in the target storage bucket, after accessing the target storage bucket based on the target identity credential, it is also necessary to recursively (or traverse) multiple storage directories under the target storage bucket to be detected, so as to determine and mark the file directories storing files from the multiple storage directories, and then identify the sensitive information of the stored files under the file directories.
[0035] Step 103: Perform file sampling on the stored files under the file directory to obtain the target detection files to be detected.
[0036] Considering that there are usually a large number of stored files under the file directory to be detected, which may be PB-level storage data (PB is a unit of data storage capacity, which is equal to 2 to the 50th power of bytes). If sensitive information detection is directly performed on all files, it may occupy a large amount of processing resources, thus affecting other online business functions. This application uses a reasonable sampling strategy to select the objects for final sensitive information identification. Specifically, it includes file sampling and text sampling. First, perform file sampling on the stored files under the file directory to select the target detection files to be detected from several stored files to be detected. Among them, the file sampling process adopts random sampling to ensure the discrete sampling effect of the files.
[0037] Step 104: Perform text sampling on the target detection files to obtain the target detection texts to be detected.
[0038] After determining the target detection files, perform text sampling on the target detection files again to select the target detection texts to be detected from each target detection file, so as to further reduce the amount of data for single sensitive information identification. Among them, the text sampling process also adopts random sampling to ensure the discrete sampling effect of the texts.
[0039] Step 105: Identify the sensitive information of the target detection texts to obtain the sensitive information identification result.
[0040] After determining the final detection object - the target detection texts, a preset sensitive information identification algorithm can be used to identify the sensitive information of the target detection texts to obtain the sensitive information identification result. Among them, the sensitive information identification result indicates the sensitive texts and non-sensitive texts in the target detection texts.
[0041] In summary, the embodiments of the present application provide a sensitive information recognition method: for PB-level cloud storage data (or cloud storage files) in OSS, discrete sampling detection is performed in two dimensions: file sampling and text sampling, so as to reduce the amount of data for sensitive information recognition in a single time, enabling the detection of PB-level data without occupying too much processing resources; and the sampling strategies in the two dimensions can also ensure the discreteness of the sampled data, thereby ensuring the comprehensiveness and accuracy of sensitive information recognition.
[0042] When performing file sampling, in order to further avoid the situation where the size of the sampled files is too large, which affects other online services when performing sensitive information recognition on the files. When performing sampling, the file size is also restricted.
[0043] Please refer to Figure 2 , which shows a flowchart of another sensitive information recognition method according to an exemplary embodiment of the present application. This method is described by taking its application to an electronic device as an example. As Figure 2 shown, this method includes:
[0044] Step 201, obtain the target storage bucket where the file to be detected is located in the object storage service and the target identity credential required to access the target storage bucket.
[0045] Step 202, access the target storage bucket based on the target identity credential, and traverse multiple storage directories in the target storage bucket to determine the file directory storing the files from the multiple storage directories.
[0046] The implementation manners of Step 201 and Step 202 can refer to the above embodiments, and will not be elaborated in this embodiment.
[0047] Step 203, if the number of directories in the file directory is greater than the first quantity threshold, perform sampling processing on the file directory to obtain a target directory.
[0048] When performing file sampling, in order to avoid only sampling the stored files in a small number of file directories, resulting in poor discreteness of the file directories. In a possible implementation manner, when it is determined that the number of directories in the file directory is greater than the first quantity threshold, sampling processing can be performed on the file directory. First, randomly select target directories that meet the preset quantity requirements from the file directory.
[0049] Exemplarily, the first quantity threshold can be 100. If the number of directories in the file directory is 120, which is greater than the first quantity threshold, then randomly select 100 target directories from the 120 file directories, and then perform sensitive information recognition on the stored files in the selected 100 target directories.
[0050] Optionally, if the number of directories in the file directory is less than the first quantity threshold, it is not necessary to sample the file directory, and all file directories can be directly determined as the target directories.
[0051] Step 204: Based on the stored files in the target directory, determine the target detection files to be detected.
[0052] After determining the target directory to be detected, further based on the stored files in the target directory, determine the target detection files to be detected. Considering that after restricting the directory, the number of files in the stored files in the directory may also be large, in order to avoid identifying sensitive information for all the stored files in the target directory, it is also necessary to limit the number of files of the target detection files to be detected.
[0053] In an exemplary example, step 204 may further include step 204A and step 204B.
[0054] Step 204A: For each target directory, obtain the number of files of the stored files in the target directory.
[0055] Step 204B: If the number of files is greater than the second quantity threshold, sample the stored files in the target directory based on the second quantity threshold to determine the target detection files in the target directory.
[0056] To ensure that files in each target directory can be randomly selected, when sampling the files in the target directory, for each target directory, obtain the number of files of the stored files in the target directory, and after the number of files is greater than the second quantity threshold, sample the stored files in the target directory based on the second quantity threshold to determine the target detection files in this directory; repeat this step until all target directories are traversed, and determine the set of target detection files sampled from all target directories as the target detection files to be detected subsequently.
[0057] Optionally, if the number of files of the stored files in the target directory is less than the second quantity threshold, the stored files in this target directory can be directly determined as the target detection files in this target directory.
[0058] Exemplarily, the second quantity threshold may be 6. If there are 8 files stored in directory 1, then randomly select 6 of them as the target detection files; if there are 4 files stored in target 2, then directly use these 4 files as the target detection files.
[0059] When selecting target detection files from the target directory, in addition to considering the number of files, the file size also needs to be considered. If the file size is too large, it will also affect the device performance. Therefore, when sampling the stored files in the target directory based on the second quantity threshold, the file sizes of each stored file in the target directory can also be obtained to filter out the stored files with file sizes lower than the preset threshold; then, based on the second quantity threshold, sample the stored files with file sizes lower than the preset threshold to determine the target detection files in the target directory.
[0060] Exemplarily, the preset threshold can be 20M. There are 8 files in the target directory, and the file sizes of each file are: File 1 (1M), File 2 (2M), File 3 (1M), File 4 (1M), File 5 (30M), File 6 (1M), File 7 (2M), File 8 (4M); first, it is judged that the number of stored files in the target directory is greater than the second quantity threshold of 6, indicating that file sampling needs to be performed on the stored files in the target directory. Then, the file sizes of each stored file in the target directory are obtained respectively. It is found that the file size of File 5 is greater than the preset threshold, so File 5 does not participate in file sampling. Only 6 files are selected from the remaining 7 files (File 1, File 2, File 3, File 4, File 6, File 7, and File 8) as the target detection files in this target directory.
[0061] Step 205, perform text sampling on the target detection files to obtain the target detection text to be detected.
[0062] After determining the target detection files to be detected, in order to ensure the security of the detection and avoid data leakage, a dedicated detection area for sensitive information can be established in advance. This detection area needs to be logged in through a jump server for re-authentication to prevent external personnel from logging in to the detection area and causing the risk of data leakage; then, download the target detection files to the detection area so that sensitive information in the target detection files can be identified in the detection area. Among them, a jump server is a network security device or computer. Only authorized users or administrators can connect to the jump server, and appropriate credentials must be provided to further access the internal system (the internal system is equivalent to the detection area in this application).
[0063] Before identifying sensitive information in the target detection file, it is also necessary to perform text sampling on the target detection file to further reduce the amount of data to be identified and obtain the target detection text to be detected. Considering that the types of target detection files may vary, different text sampling methods are set for different types of files. Specifically, if the target detection file is a word type file, paragraphs are extracted from the target detection file to obtain the target detection text to be detected; if the target detection file is an excel type file or a csv type file, columns or rows are extracted from the target detection file to obtain the target detection text to be detected; if the target detection file is a picture type file, text is extracted from the target detection file to obtain the target detection text to be detected.
[0064] Optionally, if the target detection file is a compressed file, it is also necessary to decompress the target detection file and then select different text sampling methods according to the above file types.
[0065] Step 206, obtain the regular expressions and sensitive information features corresponding to the three elements respectively.
[0066] When identifying sensitive information in the target detection text, there are mainly two identification directions. One is to identify sensitive information that matches the three elements, and the other is to identify sensitive information that does not match the three elements but matches the sensitive file features. Among them, the three elements refer to mobile phone numbers, bank card numbers, and ID card numbers; non-three-element sensitive information refers to bank statements, consumption statements, etc.
[0067] For the method of identifying sensitive information that matches the three elements, first obtain the regular expressions and sensitive information features corresponding to the three elements respectively, and then preliminarily screen the target detection text based on the regular expressions and sensitive information features to screen out the detection text that is likely to belong to the three elements; then use the large model to determine the sensitive information again for the screened detection text to obtain the final sensitive information identification result.
[0068] Exemplarily, the regular expression corresponding to the mobile phone number among the three elements can be: ^1[3-9]\d{9}$; the regular expression corresponding to the ID card number can be: ^\d{6}(18|19|20)?\d{2}(0[1-9]|1[0-2])(0[1-9]|
[12] \d|3
[01] )\d{3}[\dXx]$; the regular expression corresponding to the bank card number can be: ^\d{16,19}$. It should be noted that the card number formats of different banks may vary, and business personnel can also write more complex and personalized regular expressions according to needs to achieve accurate matching.
[0069] Exemplarily, for sensitive information features, they can be: the first 3-digit number segment of a mobile phone number, the first 6-digit card BIN of a bank card, the provincial and municipal code of an ID card number, the 8-digit date of birth, etc.
[0070] Step 207, perform sensitive information recognition on the target detection text based on regular expressions, and determine the first detection text that matches the regular expressions.
[0071] After obtaining the regular expressions and sensitive information features corresponding to the three elements respectively, first use the regular expressions to perform sensitive information recognition on the target detection text, and preliminarily screen out the first detection text that matches the regular expressions, that is, obtain the first sub-detection text that matches the regular expression corresponding to the mobile phone number, the second sub-detection text that matches the regular expression corresponding to the ID card number, and the third sub-detection text that matches the regular expression corresponding to the bank card number respectively.
[0072] Step 208, perform recognition on the first detection text based on the sensitive information features, and determine the second detection text that matches the sensitive information features.
[0073] After obtaining the first detection text (the first sub-detection text, the second sub-detection text, and the third sub-detection text), the first detection text can be further screened and recognized according to the sensitive information features of each of the three elements to filter out the first detection text that does not match the sensitive information features, and obtain the second detection text that matches the sensitive information features.
[0074] Specifically, the first sub-detection text is recognized using the sensitive information features corresponding to the mobile phone number to screen out the matching fourth sub-detection text, the second sub-detection text is recognized using the sensitive information features corresponding to the ID card number to screen out the matching fifth sub-detection text, and the third sub-detection text is recognized using the sensitive information features corresponding to the bank card number to screen out the matching sixth sub-detection text. That is, the second detection text includes the fourth sub-detection text, the fifth sub-detection text, and the sixth sub-detection text.
[0075] Step 209, input the second detection text into the sensitive information recognition model, and determine the first sensitive text that conforms to the three elements.
[0076] In order to further improve the accuracy of sensitive information recognition, a sensitive information recognition model is also pre-trained to use the sensitive information recognition model to perform sensitive information recognition on the second detection text again. Among them, the training samples of the sensitive information recognition model are the sample text and the corresponding annotation category of the sample text. If the sample text is sensitive information, the corresponding annotation category is yes, and if the sample text is non-sensitive information, the corresponding annotation category is no.
[0077] Optionally, considering the differences in the characteristics of the three elements, in order to improve the accuracy of identifying sensitive information (i.e., the three elements), different sensitive information recognition models are pre-trained. For example, a first recognition model for identifying mobile phone numbers, a second recognition model for identifying ID card numbers, and a third recognition model for identifying bank card numbers are trained. That is, the sensitive information recognition model includes the first recognition model, the second recognition model, and the third recognition model.
[0078] In a possible implementation, the second detection text is input into the sensitive information recognition model respectively to obtain the recognition result output by the sensitive information recognition model, and then the first sensitive text that conforms to the three elements is determined according to the recognition result. Specifically, the fourth sub-detection text is input into the first recognition model, and the first recognition model determines whether it is a mobile phone number and outputs the first recognition result "yes or no"; the fifth sub-detection text is input into the second recognition model, and the second recognition model determines whether it is an ID card number and outputs the second recognition result "yes or no"; the sixth sub-detection text is input into the third recognition model, and the third recognition model determines whether it is a bank card number and outputs the third recognition result "yes or no". Then, the fourth sub-detection text with the first recognition result of "yes", the fifth sub-detection text with the second recognition result of "yes", and the sixth sub-detection text with the third recognition result of "yes" are determined as the finally recognized first sensitive text that conforms to the three elements.
[0079] For sensitive information that does not conform to the three elements, a special sensitive file feature library is also set up to match it based on the sensitive file feature library to further screen sensitive information. In a corresponding possible implementation, the second detection text that does not match the regular expression and the pre-constructed sensitive file feature library are obtained, and the second detection text is matched based on the sensitive file feature library to determine the second sensitive text that matches the sensitive file feature library.
[0080] In summary, the sensitive information recognition result of the final target detection text is the first sensitive text and the second sensitive text. After the first sensitive text and the second sensitive text are recognized, the first sensitive text and the second sensitive text can be desensitized and fed back to the associated business party.
[0081] Optionally, after the sensitive information recognition result is determined, the corresponding risk level can be set according to the number of sensitive texts in the target detection file, and the more sensitive texts there are, the higher the risk level.
[0082] In this embodiment, by selecting the final target detection text to be detected through a reasonable extraction strategy, it is possible to detect data in the PB level while avoiding affecting the performance of other online services. Moreover, by comprehensively using methods such as regular expressions, sensitive information features, and sensitive information recognition models for sensitive information recognition, the recognition accuracy of sensitive information can be improved. In addition, for files that do not contain the three elements, a sensitive file feature library is established to match them, which can ensure the comprehensiveness of sensitive information recognition.
[0083] Please refer to Figure 3 , which is a schematic structural diagram of a sensitive information recognition device provided by an embodiment of the present application. Exemplarily, as Figure 3 shown, the device 300 includes:
[0084] A first acquisition module 301, configured to acquire a target storage bucket in which a file to be detected is located in an object storage service and a target identity credential required to access the target storage bucket;
[0085] A first determination module 302, configured to access the target storage bucket based on the target identity credential and traverse multiple storage directories in the target storage bucket to determine a file directory in which a file is stored from the multiple storage directories;
[0086] A first sampling module 303, configured to perform file sampling processing on the stored files in the file directory to obtain a target detection file to be detected;
[0087] A second sampling module 304, configured to perform text sampling processing on the target detection file to obtain a target detection text to be detected;
[0088] A first recognition module 305, configured to perform sensitive information recognition on the target detection text to obtain a sensitive information recognition result.
[0089] Optionally, the first sampling module 303 is further configured to:
[0090] If the number of directories in the file directory is greater than a first quantity threshold, perform sampling processing on the file directory to obtain a target directory;
[0091] Based on the stored files in the target directory, determine the target detection file to be detected.
[0092] Optionally, the first sampling module 303 is further configured to:
[0093] For each target directory, acquire the number of files of the stored files in the target directory;
[0094] If the number of the files is greater than a second quantity threshold, sample the stored files in the target directory based on the second quantity threshold to determine the target detection files in the target directory.
[0095] Optionally, the first sampling module 303 is further configured to:
[0096] Obtain the file sizes of the respective stored files in the target directory;
[0097] Based on the second quantity threshold, sample the stored files with file sizes lower than a preset threshold to determine the target detection files in the target directory.
[0098] Optionally, the second sampling module 304 is further configured to:
[0099] If the target detection file is a word type file, perform paragraph extraction on the target detection file to obtain the target detection text to be detected;
[0100] If the target detection file is an excel type file or a csv type file, perform column extraction or row extraction on the target detection file to obtain the target detection text to be detected;
[0101] If the target detection file is a picture type file, perform text extraction on the target detection file to obtain the target detection text to be detected.
[0102] Optionally, the first recognition module 305 is further configured to:
[0103] Obtain the regular expressions and sensitive information features corresponding to the three elements respectively;
[0104] Based on the regular expressions, perform sensitive information recognition on the target detection text to determine the first detection text that matches the regular expressions;
[0105] Based on the sensitive information features, perform recognition on the first detection text to determine the second detection text that matches the sensitive information features;
[0106] Input the second detection text into a sensitive information recognition model to determine the first sensitive text that conforms to the three elements.
[0107] Optionally, the apparatus further includes:
[0108] A second acquisition module, configured to acquire the second detection text that does not match the regular expressions;
[0109] A third acquisition module, configured to acquire a pre-constructed sensitive file feature library;
[0110] A second recognition module, configured to match the second detected text based on the sensitive file feature library, and determine a second sensitive text that matches the sensitive file feature library.
[0111] In summary, the embodiment of the present application provides a sensitive information recognition device: for PB-level cloud storage data (or cloud storage files) in OSS, discrete sampling detection is performed in two dimensions: file sampling and text sampling, so as to reduce the amount of data for sensitive information recognition at one time, so that the detection of PB-level data can be realized without occupying too much processing resources; and the sampling strategies in the two dimensions can also ensure the discreteness of the sampled data, thereby ensuring the comprehensiveness and accuracy of sensitive information recognition. Moreover, by means of a reasonable extraction strategy to select the final target detection text to be detected, the detection of PB-level data volume can be realized while avoiding affecting the performance of other online services; and by comprehensively using methods such as regular expressions, sensitive information features, and sensitive information recognition models for sensitive information recognition, the recognition accuracy of sensitive information can be improved; in addition, for files that do not contain the three elements, a sensitive file feature library is also established to match them, which can ensure the comprehensiveness of sensitive information recognition.
[0112] An exemplary embodiment of the present application further provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program that can be executed by the at least one processor, and when the computer program is executed by the at least one processor, it is used to cause the electronic device to execute the sensitive information recognition method according to the embodiment of the present application.
[0113] An exemplary embodiment of the present application further provides a non-transitory computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor of a computer, it is used to cause the computer to execute the sensitive information recognition method according to the embodiment of the present application.
[0114] An exemplary embodiment of the present application further provides a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, it is used to cause the computer to execute the sensitive information recognition method according to the embodiment of the present application.
[0115] Reference Figure 4, a block diagram of an electronic device 400 that can be a server or a client of the present application will now be described. It is an example of a hardware device that can be applied to various aspects of the present application. The electronic device is intended to represent various forms of digital electronic computer devices, such as, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, personal digital processors, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.
[0116] As Figure 4 shown, the electronic device 40 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 can also be stored. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.
[0117] Multiple components in the electronic device 400 are connected to the I / O interface 405, including: an input unit 406, an output unit 407, a storage unit 408, and a communication unit 409. The input unit 406 can be any type of device that can input information into the electronic device 400. The input unit 406 can receive input digital or character information, and generate key signal inputs related to the user settings and / or function controls of the electronic device. The output unit 407 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 408 can include but is not limited to a magnetic disk, an optical disk. The communication unit 409 allows the electronic device 400 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.
[0118] The computing unit 401 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 401 executes the various methods and processes described above. For example, in some embodiments, Figure 1 , Figure 2 The method shown can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 408. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 400 via the ROM 402 and / or the communication unit 409. In some embodiments, the computing unit 401 can be configured to execute Figure 1 , Figure 2 The method shown in any other suitable manner (e.g., by means of firmware).
[0119] The program code for implementing the methods of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the program codes are executed by the processor or controller, the functions / operations specified in the flowchart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, executed partially on the machine and partially on a remote machine as an independent software package, or executed entirely on a remote machine or server.
[0120] In the context of the present application, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0121] As used in this application, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., a magnetic disk, an optical disk, a memory, a programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.
[0122] In order to provide for interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can also be used to provide for interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, speech input, or tactile input).
[0123] The systems and techniques described herein can be implemented in a computing system that includes a back-end component (e.g., as a data server), or a computing system that includes a middleware component (e.g., an application server), or a computing system that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system that includes any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0124] A computer system can include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship to each other.
Claims
1. A sensitive information identification method, characterized in that: The method comprises: Obtain the target storage bucket where the file to be detected is located in the object storage service and the target identity credential required to access the target storage bucket; Accessing the target storage bucket based on the target identity credential, and traversing multiple storage directories in the target storage bucket to determine a file directory storing files from the multiple storage directories; Perform file sampling processing on the stored files in the file directory to obtain the target detection file to be detected; Performing text sampling processing on the target detection file to obtain the target detection text to be detected; Sensitive information is identified on the target detection text to obtain a sensitive information identification result.
2. The method according to claim 1, characterized in that The file sampling process is performed on the stored files in the file directory to obtain the target detection file to be detected, including: If the number of directories in the file directory is greater than a first number threshold, sampling the file directory to obtain a target directory; Based on the stored files in the target directory, the target detection file to be detected is determined.
3. The method according to claim 2, characterized in that The step of determining the target detection file to be detected based on the storage file in the target directory includes: For each of the target directories, obtaining the number of the stored files in the target directory; If the number of files is greater than a second number threshold, a sampling process is performed on the stored files under the target directory based on the second number threshold to determine the target detection files under the target directory.
4. The method according to claim 3, characterized in that The sampling process of the storage files under the target directory based on the second quantity threshold to determine the target detection files under the target directory includes: Obtain the file size of each of the stored files in the target directory; Based on the second quantity threshold, a sampling process is performed on the storage files whose file sizes are lower than a preset threshold to determine the target detection files under the target directory.
5. The method according to any one of claims 1 to 4, characterized in that: The performing text sampling processing on the target detection file to obtain the target detection text to be detected includes: If the target detection file is a word type file, extract paragraphs from the target detection file to obtain the target detection text to be detected; If the target detection file is an excel type file or a csv type file, extract the target detection file by column or by row to obtain the target detection text to be detected; If the target detection file is a picture file, text extraction is performed on the target detection file to obtain the target detection text to be detected.
6. The method according to any one of claims 1 to 4, characterized in that: The step of performing sensitive information identification on the target detection text to obtain a sensitive information identification result includes: Get the regular expressions and sensitive information features corresponding to the three elements respectively; Based on the regular expression, sensitive information is identified on the target detection text, and a first detection text matching the regular expression is determined; Identify the first detection text based on the sensitive information feature, and determine a second detection text that matches the sensitive information feature; The second detection text is input into the sensitive information recognition model to determine the first sensitive text that meets the three elements.
7. The method according to claim 6, characterized in that The method further comprises: Acquire a second detection text that does not match the regular expression; Obtain a pre-built sensitive file signature library; The second detection text is matched based on the sensitive file feature library to determine a second sensitive text that matches the sensitive file feature library.
8. A sensitive information identification device, characterized in that: The device comprises: The first acquisition module is used to obtain the target storage bucket where the file to be detected is located in the object storage service and the target identity credential required to access the target storage bucket; A first determination module, configured to access the target storage bucket based on the target identity credential, and traverse multiple storage directories in the target storage bucket to determine a file directory storing files from the multiple storage directories; A first sampling module is used to perform file sampling processing on the stored files in the file directory to obtain a target detection file to be detected; A second sampling module is used to perform text sampling processing on the target detection file to obtain the target detection text to be detected; The first recognition module is used to perform sensitive information recognition on the target detection text to obtain a sensitive information recognition result.
9. An electronic device, comprising: processor; as well as Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to execute the sensitive information identification method according to any one of claims 1-7.
10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the sensitive information identification method according to any one of claims 1-7.