File sensitive information detection method and device, electronic equipment and vehicle

Abstract

This application provides a method, apparatus, electronic device, and vehicle for detecting sensitive information in documents. The method involves acquiring a document to be detected, which has corresponding first file metadata. The first file metadata is matched against second file metadata stored in a preset file library. If a match is found in the file library corresponding to the first file metadata, the sensitive information in the document to be detected is determined based on the second file metadata. If no match is found in the file library corresponding to the first file metadata, a target data block is extracted from the document to be detected, and the sensitive information in the target data block is detected to obtain the sensitive information in the document to be detected. This application provides document-level and data-level filtering of the document to be detected, performing sensitive information detection only on target data blocks in documents that have not been previously detected, thus improving the scanning efficiency of sensitive information detection.

Description

Technical Field

[0001] This application relates to the field of information detection technology, and in particular to a method, apparatus, electronic device, and vehicle for detecting sensitive information in a document. Background Technology

[0002] SO files (Shared Objects) are dynamic link library files used in Linux and Android systems. Currently, there are relatively mature solutions for detecting sensitive information in visible text files such as TXT (Text Files), JSON (JavaScript Object Notation), and XML (Extensible Markup Language) in systems or software. However, methods for detecting sensitive information in SO files are relatively few, and most methods focus only on reverse engineering and malicious code analysis of SO files. Existing SO file sensitive information detection solutions typically employ a method of indiscriminately extracting strings from the entire file before detection. This method contains a large number of invalid interference strings, resulting in low detection efficiency.

[0003] Therefore, how to obtain a more efficient and reliable method for detecting sensitive information in SO files is an urgent problem to be solved in this field. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, and vehicle for detecting sensitive information in SO files, aiming to improve the problem of how to obtain a more efficient and reliable method for detecting sensitive information in SO files.

[0005] To address the aforementioned problems, this application discloses a method for detecting sensitive information in a document, comprising: Obtain the file to be detected, wherein the file to be detected has corresponding first file metadata, and the first file metadata is information reflecting the content features and structural features of the file to be detected; The first file metadata is matched with the second file metadata stored in the preset file library. The second file metadata is obtained based on the content and structural features of historically detected files. The second file metadata contains a record of whether there is sensitive information in the file. If a second file metadata corresponding to the first file metadata is matched in the file library, then the sensitive information in the file to be detected is determined based on the second file metadata. If no second file metadata corresponding to the first file metadata is matched in the file library, then the target data block in the file to be detected is extracted, and the sensitive information in the target data block is detected to obtain the sensitive information in the file to be detected.

[0006] This application embodiment first determines whether the file to be detected is a previously detected file based on the first file metadata and the second file metadata stored in the file library. If the second file metadata corresponding to the first file metadata is matched in the file library, the file to be detected can be determined to be a previously detected file, and the sensitive information can be directly determined based on the previously stored second file metadata without re-detecting the data in the previously detected files. If the second file metadata corresponding to the first file metadata is not matched in the file library, the file to be detected can be determined to be a previously undetected file, and the data in it needs to be detected. However, only the target data blocks in the file to be detected are extracted for sensitive information detection, without detecting all the data. By performing file-level and data-level filtering on the file to be detected in this application embodiment, and only performing sensitive information detection on the target data blocks in files that have not been detected in the past, the low scanning efficiency caused by indiscriminate string extraction from the entire file before detection in the prior art is solved, the scanning efficiency of sensitive information detection is improved, and sensitive information in the file to be detected can be effectively detected.

[0007] Optionally, the file to be detected has multiple partitions, each partition containing corresponding partition data. Extracting target data blocks from the file to be detected and detecting sensitive information within those target data blocks to obtain the sensitive information in the file to be detected includes: Determine the target partition from the plurality of partitions; Extract partition data from the target partition of the file to be detected; The partitioned data is divided into blocks to obtain the target data block, and the target data block has a corresponding first hash value; The first hash value is matched with a second hash value stored in a preset data block library. The second hash value is obtained based on the hash values ​​of historically detected data blocks. The second hash value has a record of whether there is a corresponding sensitive information about the data block. If a second hash value corresponding to the first hash value is matched in the data block library, the sensitive information in the target data block is determined based on the second hash value; If no second hash value corresponding to the first hash value is found in the data block library, then sensitive information matching is performed on the target data block to obtain the sensitive information in the target data block; The sensitive information in the target data block is identified as the sensitive information in the file to be detected.

[0008] This application embodiment extracts partition data from the target partition of a file requiring data-level screening. After dividing the partition data into blocks, it confirms whether the target data block is a previously detected target data block based on a first hash value and a second hash value stored in the data block library. If a second hash value corresponding to the first hash value is matched in the data block library, the target data block can be determined to be a previously detected data block, and the sensitive information within it can be directly determined based on the historically stored second hash value, without needing to re-detect the previously detected data block. If no second hash value corresponding to the first hash value is matched in the data block library, the target data block can be determined to be a previously undetected data block, requiring sensitive information matching within it. Thus, sensitive information matching only needs to be performed on the previously undetected data blocks identified in the target partition of the file to be detected, improving scanning efficiency.

[0009] Optionally, dividing the partitioned data into blocks includes: If the partition data has a preset identifier and / or a preset separator, then the partition data is divided into blocks according to the preset identifier and / or the preset separator; If the partition data does not contain the preset identifier and / or the preset separator, then the partition data is divided into blocks according to the preset data block size.

[0010] In this embodiment, partition data is divided into blocks using preset identifiers and / or preset delimiters. When there are no preset identifiers and / or preset delimiters in the partition data, the partition data is directly divided into blocks according to the preset data block size to obtain target data blocks, which are used to subsequently confirm whether the target data block is a target data block that has been detected in the past.

[0011] Optionally, the step of performing sensitive information matching on the target data block to obtain sensitive information in the target data block includes: Determine the entropy value of the target data block; If the entropy value of the target data block is higher than a preset entropy threshold, then the encoding method and / or compression method of the target data block are identified; According to the encoding method and / or compression method of the target data block, the target data block is decoded and / or decompressed to obtain the processed target data block; Sensitive information matching is performed on the processed target data block to obtain the sensitive information in the target data block.

[0012] The embodiments of this application can decode and / or decompress the encoded and / or compressed data in the target data block before performing sensitive information matching, so as to avoid missing the encoded and compressed data in the target data block and solve the problem in the prior art that the hard-coded information in SO files that has been processed by encoding, compression, etc. cannot be identified, resulting in a detection blind spot.

[0013] Optionally, after determining the entropy value of the target data block, the method includes: If the entropy value of the target data block is lower than the preset entropy threshold, then sensitive information matching is performed on the target data block to obtain the sensitive information in the target data block.

[0014] The embodiments of this application can directly perform sensitive information matching on plaintext data in the target data block, thereby improving the efficiency of sensitive information matching.

[0015] Optionally, the file library stores the file's sensitive information, and determining the sensitive information in the file to be detected based on the second file metadata includes: Based on the file sensitive information corresponding to the second file metadata, the sensitive information in the file to be detected is determined; After obtaining the sensitive information in the file to be detected, the method includes: The first file metadata and sensitive information of the file to be detected are recorded in the file database to obtain the updated second file metadata and updated file sensitive information.

[0016] In this embodiment, the first file metadata of newly detected files is recorded in the file library. If there is corresponding sensitive information, it is also recorded in the file library. In subsequent file-level screening, it can be determined whether the file to be detected is a file that has been detected in the past and whether it contains sensitive information. Based on the sensitive information recorded in the file library, the sensitive information in the files that have been detected in the past can be directly determined, thereby avoiding sensitive information detection on all files and improving scanning efficiency.

[0017] Optionally, the data block library stores the sensitive information of the data blocks, and the step of determining the sensitive information in the target data block based on the second hash value includes: Based on the sensitive information of the data block corresponding to the second hash value, the sensitive information in the target data block is determined; After obtaining the sensitive information in the target data block, the method includes: The first hash value and sensitive information of the target data block are recorded in the data block library to obtain the updated second hash value and the updated sensitive information of the data block.

[0018] In this embodiment, the first hash value of the newly detected data block is recorded in the data block library. If there is corresponding sensitive information, it is also recorded in the data block library. In subsequent data-level filtering, it can be determined whether the target data block is a previously detected data block and whether it contains sensitive information. Based on the sensitive information recorded in the data block library, the sensitive information in the previously detected data blocks can be directly determined, thereby avoiding sensitive information detection on all files and improving scanning efficiency.

[0019] Optionally, the second file metadata and the second hash value each have corresponding information weights, and the method includes: If a second file metadata corresponding to the first file metadata is matched in the file library, the information weight of the second file metadata corresponding to the first file metadata is updated; and if a second hash value corresponding to the first hash value is matched in the data block library, the information weight of the second hash value corresponding to the first hash value is updated. The second file metadata whose information weight is lower than a preset weight threshold and which does not correspond to the file sensitive information is removed from the file library, and / or the second hash value whose information weight is lower than the preset weight threshold and which does not correspond to the data block sensitive information is removed from the data block library.

[0020] In this application embodiment, second file metadata and second hash values ​​that have not been matched for a long time and for which there is no corresponding sensitive information are removed from the database, thereby optimizing the file database and data block database and improving the efficiency of each match.

[0021] This application also discloses a sensitive information detection device for documents, including: The file acquisition module is used to acquire the file to be detected. The file to be detected has corresponding first file metadata, which is information reflecting the content features and structural features of the file to be detected. The first matching module is used to match the first file metadata with the second file metadata stored in a preset file library. The second file metadata is obtained based on the content and structural features of historically detected files, and the second file metadata has a record of whether there is sensitive information in the file. The information determination module is used to determine sensitive information in the file to be detected based on the second file metadata if a second file metadata corresponding to the first file metadata is matched in the file library. The information detection module is used to extract the target data block in the file to be detected and detect the sensitive information in the target data block if no second file meta information corresponding to the first file meta information is matched in the file library, thereby obtaining the sensitive information in the file to be detected.

[0022] This application also discloses an electronic device, including a processor and a memory, wherein the memory is used to store computer programs; the processor is used to execute the programs stored in the memory to implement the sensitive information detection method for files as described in one or more of the embodiments of this application.

[0023] This application also discloses a vehicle that includes electronic devices as described in the embodiments of this application. Attached Figure Description

[0024] Figure 1 This is a flowchart of a method for detecting sensitive information in a document according to an embodiment of this application; Figure 2 This is a flowchart illustrating the implementation of a sensitive information detection method for documents provided in an embodiment of this application; Figure 3 This is a structural diagram of the sensitive information detection device for documents provided in the embodiments of this application; Figure 4 This is a structural diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0025] To make the technical problems, technical solutions, and beneficial effects solved by this application clearer, the following detailed description is provided in conjunction with embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0026] SO files (Shared Objects) are dynamic link library files used in Linux and Android systems. Currently, there are relatively mature solutions for detecting sensitive information in visible text files such as TXT (Text Files), JSON (JavaScript Object Notation), and XML (Extensible Markup Language) in systems or software. However, methods for detecting sensitive information in SO files are relatively limited, often focusing only on reverse engineering and malware analysis. Existing SO file sensitive information detection solutions typically involve indiscriminately extracting strings from the entire file before detection. This method contains a large number of invalid interference strings, resulting in low detection efficiency and numerous duplicate false positives. Furthermore, existing solutions cannot identify hard-coded information in SO files that has undergone encoding or compression, creating detection blind spots.

[0027] Therefore, how to obtain a more efficient and reliable method for detecting sensitive information in SO files is an urgent problem to be solved in this field.

[0028] This application provides a method for detecting sensitive information in a document, including: Obtain the file to be detected, wherein the file to be detected has corresponding first file metadata, and the first file metadata is information reflecting the content features and structural features of the file to be detected; The first file metadata is matched with the second file metadata stored in the preset file library. The second file metadata is obtained based on the content and structural features of historically detected files. The second file metadata contains a record of whether there is sensitive information in the file. If a second file metadata corresponding to the first file metadata is matched in the file library, then the sensitive information in the file to be detected is determined based on the second file metadata. If no second file metadata corresponding to the first file metadata is matched in the file library, then the target data block in the file to be detected is extracted, and the sensitive information in the target data block is detected to obtain the sensitive information in the file to be detected.

[0029] This application embodiment first determines whether the file to be detected is a previously detected file based on the first file metadata and the second file metadata stored in the file library. If the second file metadata corresponding to the first file metadata is matched in the file library, the file to be detected can be determined to be a previously detected file, and the sensitive information can be directly determined based on the previously stored second file metadata without re-detecting the data in the previously detected files. If the second file metadata corresponding to the first file metadata is not matched in the file library, the file to be detected can be determined to be a previously undetected file, and the data in it needs to be detected. However, only the target data blocks in the file to be detected are extracted for sensitive information detection, without detecting all the data. By performing file-level and data-level filtering on the file to be detected in this application embodiment, and only performing sensitive information detection on the target data blocks in files that have not been detected in the past, the low scanning efficiency caused by indiscriminate string extraction from the entire file before detection in the prior art is solved, the scanning efficiency of sensitive information detection is improved, and sensitive information in the file to be detected can be effectively detected.

[0030] Example 1 This application provides a method for detecting sensitive information in a document. Please refer to the following embodiments. Figure 1 This includes the following steps: S110: Obtain the file to be detected, wherein the file to be detected has corresponding first file metadata, and the first file metadata is information reflecting the content features and structural features of the file to be detected.

[0031] The file to be detected in this embodiment can be an SO file. In step S110, the file to be detected is obtained by finding all SO files on the system and outputting the paths of the SO files.

[0032] The file to be detected has corresponding first file metadata, which reflects the content and structural characteristics of the file. In one embodiment, the first file metadata may include a list of exported function names, a list of dynamic link libraries, the size of the .rodata section, and a hash value. The list of exported function names refers to the functions provided by the dynamic link libraries of the SO file that can be called by other programs. The list of dynamic link libraries refers to other shared libraries that the SO file depends on at runtime. The size of the .rodata section is the size of the section of the SO file containing read-only data during program execution. Hash is a process of transforming an input (message) of arbitrary length into a fixed-length output (hash value) using a hash algorithm; the hash value of the first file metadata is the hash value corresponding to the SO file.

[0033] S120: Match the first file metadata with the second file metadata stored in the preset file library. The second file metadata is obtained based on the content and structural features of historically detected files. The second file metadata has a record of whether there is sensitive information in the file.

[0034] The file library in this application stores file metadata corresponding to historically detected SO files, namely, second file metadata obtained based on the content and structural features of historically detected files. The file library also stores whether the SO file contains corresponding file sensitive information. When an unfamiliar SO file is detected for the first time, the file library records the file metadata of the SO file, whether the SO file contains file sensitive information, and the corresponding file sensitive information if it does.

[0035] In one embodiment, the second file metadata may include a list of exported function names, a list of dynamic link libraries, the size of the .rodata section, hash, etc., of historically detected SO files.

[0036] In step S120, the first file metadata of the file to be detected is matched with the second file metadata stored in the file library to check whether there is corresponding second file metadata in the file library.

[0037] S130: If a second file metadata corresponding to the first file metadata is matched in the file library, then the sensitive information in the file to be detected is determined based on the second file metadata.

[0038] In step S130, if a second file metadata corresponding to the first file metadata is matched in the file library, i.e., a previously detected SO file is matched, the presence of sensitive file information in the file to be detected can be directly determined based on whether the SO file recorded in the file library contains sensitive file information. If the file to be detected contains sensitive file information, the sensitive file information corresponding to the matched historical SO file recorded in the file library is identified as the sensitive information of the file to be detected, and the relevant information of the file to be detected is recorded in the file library.

[0039] After determining the sensitive information of the file to be tested based on the file library records, there is no need to further test the data in the file; the data testing process can be skipped.

[0040] S140: If no second file metadata corresponding to the first file metadata is matched in the file library, then the target data block in the file to be detected is extracted, and the sensitive information in the target data block is detected to obtain the sensitive information in the file to be detected.

[0041] In step S140, if no second file metadata corresponding to the first file metadata is matched in the file library, that is, the file to be detected has not been detected in the historical detection, then the target data block in the file to be detected is extracted, and the sensitive information in the target data block is detected to obtain the sensitive information in the file to be detected.

[0042] In one embodiment, before extracting the target data block from the file to be tested, it is first determined whether the file to be tested is a packed or encrypted file. The content of a packed SO file is obfuscated, and an unpacked SO file cannot be used for sensitive information detection. Therefore, it is necessary to attempt to unpack or unencrypt the packed or encrypted SO file. Specific open-source tools can be used to identify whether an SO file is packed. If it is a packed file, an attempt is made to unpack it using the open-source tool. The SO file that can be unpacked is the one that can be processed for subsequent file data detection and processing. Packed or encrypted SO files that can be unpacked and unpacked or unencrypted SO files are recorded as processable files to be tested, and the target data block is extracted from the file to be tested. Packed or encrypted SO files that cannot be unpacked are recorded as unprocessable files to be tested, and subsequent manual processing is required to detect sensitive information.

[0043] This application embodiment first determines whether the file to be detected is a previously detected file based on the first file metadata and the second file metadata stored in the file library. If the second file metadata corresponding to the first file metadata is matched in the file library, the file to be detected can be determined to be a previously detected file, and the sensitive information can be directly determined based on the previously stored second file metadata without re-detecting the data in the previously detected files. If the second file metadata corresponding to the first file metadata is not matched in the file library, the file to be detected can be determined to be a previously undetected file, and the data in it needs to be detected. However, only the target data blocks in the file to be detected are extracted for sensitive information detection, without detecting all the data. By performing file-level and data-level filtering on the file to be detected in this application embodiment, and only performing sensitive information detection on the target data blocks in files that have not been detected in the past, the low scanning efficiency caused by indiscriminate string extraction from the entire file before detection in the prior art is solved, the scanning efficiency of sensitive information detection is improved, and sensitive information in the file to be detected can be effectively detected.

[0044] Optionally, the file to be detected has multiple partitions, each partition containing corresponding partition data. Extracting target data blocks from the file to be detected and detecting sensitive information within those target data blocks to obtain the sensitive information in the file to be detected includes: Determine the target partition from the plurality of partitions; Extract partition data from the target partition of the file to be detected; The partitioned data is divided into blocks to obtain the target data block, and the target data block has a corresponding first hash value; The first hash value is matched with a second hash value stored in a preset data block library. The second hash value is obtained based on the hash values ​​of historically detected data blocks. The second hash value has a record of whether there is a corresponding sensitive information about the data block. If a second hash value corresponding to the first hash value is matched in the data block library, the sensitive information in the target data block is determined based on the second hash value; If no second hash value corresponding to the first hash value is found in the data block library, then sensitive information matching is performed on the target data block to obtain the sensitive information in the target data block; The sensitive information in the target data block is identified as the sensitive information in the file to be detected.

[0045] In this embodiment, the file to be tested contains multiple partitions, each with corresponding partition data. If the file to be tested is an SO file, then the partitions in the SO file contain multiple sections, each with corresponding data.

[0046] This application's embodiments determine the target partition from multiple partitions, and extract only the partition data from the target partition of the file to be detected for sensitive information detection, thereby improving scanning efficiency. In one embodiment, the target partition can be the .rodata section and .data section of an SO file, and data from the .rodata section and .data section of the SO file is extracted.

[0047] The partitioned data is divided into blocks to obtain target data blocks, and the hash value of the target data block is calculated as the first hash value. The data block library in this embodiment stores the hash values ​​corresponding to historically detected data blocks, i.e., the second hash value obtained based on historically detected data blocks. The data block library also stores whether the data block has corresponding sensitive information. When the data block is scanned for the first time, the data block library records the data block hash value of the data block, whether the data block has sensitive information, and the sensitive information corresponding to the presence of sensitive information. The file sensitive information of an SO file consists of the sensitive information of the data blocks corresponding to the data blocks of the SO file.

[0048] The first hash value of the target data block is matched with the second hash value stored in the data block library to check if there is a corresponding second hash value in the data block library.

[0049] If a second hash value corresponding to the first hash value is matched in the data block database, meaning a historically detected data block has been hit, the presence of sensitive information in the target data block can be directly determined based on whether the target data block contains such sensitive information, as recorded in the data block database. If the target data block contains sensitive information, the sensitive information corresponding to the historically detected data block recorded in the data block database is identified as the sensitive information of the target data block, and the relevant information of the target data block is recorded in the data block database.

[0050] Once the sensitive information of the target data block is determined based on the data block library records, there is no need to perform sensitive data detection on the target data block again, and the sensitive data detection process can be skipped.

[0051] If no second hash value corresponding to the first hash value is found in the data block database, meaning that the target data block has not been detected in historical detection, then sensitive information matching is performed on the target data block to obtain the sensitive information in the target data block.

[0052] Sensitive information in the target data block is identified as sensitive information in the file to be detected.

[0053] This application embodiment extracts partition data from the target partition of a file requiring data-level screening. After dividing the partition data into blocks, it confirms whether the target data block is a previously detected target data block based on a first hash value and a second hash value stored in the data block library. If a second hash value corresponding to the first hash value is matched in the data block library, the target data block can be determined to be a previously detected data block, and the sensitive information within it can be directly determined based on the historically stored second hash value, without needing to re-detect the previously detected data block. If no second hash value corresponding to the first hash value is matched in the data block library, the target data block can be determined to be a previously undetected data block, requiring sensitive information matching within it. Thus, sensitive information matching only needs to be performed on the previously undetected data blocks identified in the target partition of the file to be detected, improving scanning efficiency.

[0054] Optionally, dividing the partitioned data into blocks includes: If the partition data has a preset identifier and / or a preset separator, then the partition data is divided into blocks according to the preset identifier and / or the preset separator; If the partition data does not contain the preset identifier and / or the preset separator, then the partition data is divided into blocks according to the preset data block size.

[0055] In this embodiment of the application, it is determined whether there is a preset identifier and / or a preset separator in the partition data. If there is a preset identifier and / or a preset separator, the partition data can be divided into blocks according to the preset identifier and / or the preset separator.

[0056] In one embodiment, the preset identifier may include .text, .debug_info, etc. For typical SO files, a Python module can be used to extract relevant data from the partitioned data based on preset identifiers such as .text and .debug_info to achieve block segmentation and obtain the target data block. The Python module is a module containing Python code.

[0057] In one embodiment, the preset delimiter may include the \0 delimiter, etc. In SO files generated from C code, the code is generally separated by \0, so this type of SO file can use the \0 delimiter to divide the partition data into blocks.

[0058] If the partition data of the file to be tested does not contain .text, .debug_info, or \0 delimiters, the partition data is directly divided into overlapping blocks with a fixed preset data block size, such as 4k or 8k, to ensure that each target data block obtained is the same size.

[0059] In this embodiment, partition data is divided into blocks using preset identifiers and / or preset delimiters. When there are no preset identifiers and / or preset delimiters in the partition data, the partition data is directly divided into blocks according to the preset data block size to obtain target data blocks, which are used to subsequently confirm whether the target data block is a target data block that has been detected in the past.

[0060] Optionally, the step of performing sensitive information matching on the target data block to obtain sensitive information in the target data block includes: Determine the entropy value of the target data block; If the entropy value of the target data block is higher than a preset entropy threshold, then the encoding method and / or compression method of the target data block are identified; According to the encoding method and / or compression method of the target data block, the target data block is decoded and / or decompressed to obtain the processed target data block; Sensitive information matching is performed on the processed target data block to obtain the sensitive information in the target data block.

[0061] Before performing sensitive information matching on the target data block, this embodiment first determines whether the target data block has been encoded or compressed by checking its entropy value. Specifically, if the data in the target data block has been encoded or compressed, the entropy value of the target data block will be relatively high, making direct sensitive information identification impossible. The entropy value of the target data block can be viewed using the binwalk tool. For example, data compressed using methods such as zip and gz generally has an entropy value of 7.5 or higher. The preset entropy threshold can be obtained statistically from the entropy values ​​of historical encoded and / or compressed data blocks, or set empirically.

[0062] If the entropy value of the target data block is higher than a preset entropy threshold, then the data in the target data block has been encoded and / or compressed. The encoding and / or compression methods of the target data block are identified. For example, common data characteristics of encoding and / or compression methods include: ZLIB: Begins with 0x78 0x9C or 0x78 0xDA; GZIP: Begins with 0x1F 0x8B 0x08; BASE64: Contains only AZ, az, 0-9, +, / (standard Base64) or -, _ (URL-safe Base64), may be padded with = at the end, and the encoded data length is a multiple of 4; Hex: Contains only 0-9, AF / af, no other special characters, and has an even length.

[0063] For a target data block that has been encoded and / or compressed, it is necessary to first decode and / or decompress the target data block according to the encoding method and / or compression method to obtain the processed target data block. The data in the processed target data block is visible characters, which can be used for sensitive information matching to obtain the sensitive information in the target data block.

[0064] In one embodiment, sensitive information can be scanned and matched using methods such as regular expression matching to obtain the sensitive information in the target data block.

[0065] In one embodiment, if the entropy value of the target data block is higher than a preset entropy threshold, but the target data block cannot identify the encoding method and / or compression method, and / or cannot be decoded and / or decompressed, then the target data block is recorded and manually processed.

[0066] The embodiments of this application can decode and / or decompress the encoded and / or compressed data in the target data block before performing sensitive information matching, so as to avoid missing the encoded and compressed data in the target data block and solve the problem in the prior art that the hard-coded information in SO files that has been processed by encoding, compression, etc. cannot be identified, resulting in a detection blind spot.

[0067] Optionally, after determining the entropy value of the target data block, the method includes: If the entropy value of the target data block is lower than the preset entropy threshold, then sensitive information matching is performed on the target data block to obtain the sensitive information in the target data block.

[0068] In this embodiment of the application, if the entropy value of the target data block is lower than the preset entropy value threshold, it is determined that the target data block is likely to be visible text data, and sensitive information matching can be performed directly on the target data block to obtain the sensitive information in the target data block.

[0069] In one embodiment, sensitive information can be scanned and matched using methods such as regular expression matching to obtain the sensitive information in the target data block.

[0070] The embodiments of this application can directly perform sensitive information matching on plaintext data in the target data block, thereby improving the efficiency of sensitive information matching.

[0071] Optionally, the file library stores the file's sensitive information, and determining the sensitive information in the file to be detected based on the second file metadata includes: Based on the file sensitive information corresponding to the second file metadata, the sensitive information in the file to be detected is determined; After obtaining the sensitive information in the file to be detected, the method includes: The first file metadata and sensitive information of the file to be detected are recorded in the file database to obtain the updated second file metadata and updated file sensitive information.

[0072] In this embodiment, since the file library records the file metadata of historically detected SO files, whether the SO file contains sensitive file information, and the corresponding sensitive file information when it does, if the file library matches the second file metadata corresponding to the first file metadata, i.e., a historically detected SO file is matched, the presence of sensitive file information in the file library can be used to directly determine whether the file to be detected contains sensitive file information. If the file to be detected contains sensitive file information, the sensitive file information corresponding to the matched historical SO file recorded in the file library is determined as the sensitive information of the file to be detected.

[0073] If the file to be detected is an SO file that has not been detected in the past, then regardless of whether the file contains sensitive information, the first file metadata and whether there is sensitive information must be recorded in the file database to update the second file metadata in the file database. The next time the file is detected, the database records can be used to directly determine whether the file contains sensitive information, thus improving scanning efficiency.

[0074] For files containing sensitive information to be detected, the first file metadata and the corresponding sensitive information of the file to be detected need to be recorded in the file library so as to update the second file metadata and the corresponding file sensitive information in the file library.

[0075] In one embodiment, the file path of the file to be detected, the corresponding sensitive information, and other information can be recorded in a file library using Excel, SQLite, or other similar methods.

[0076] In this embodiment, the first file metadata of newly detected files is recorded in the file library. If there is corresponding sensitive information, it is also recorded in the file library. In subsequent file-level screening, it can be determined whether the file to be detected is a file that has been detected in the past and whether it contains sensitive information. Based on the sensitive information recorded in the file library, the sensitive information in the files that have been detected in the past can be directly determined, thereby avoiding sensitive information detection on all files and improving scanning efficiency.

[0077] Optionally, the data block library stores the sensitive information of the data blocks, and the step of determining the sensitive information in the target data block based on the second hash value includes: Based on the sensitive information of the data block corresponding to the second hash value, the sensitive information in the target data block is determined; After obtaining the sensitive information in the target data block, the method includes: The first hash value and sensitive information of the target data block are recorded in the data block library to obtain the updated second hash value and the updated sensitive information of the data block.

[0078] In this embodiment, since the data block library records the hash values ​​of historically detected data blocks, whether the data block contains sensitive information, and the sensitive information corresponding to the data block when it does, if a second hash value corresponding to the first hash value is matched in the data block library, i.e., a historically detected data block is hit, the presence of sensitive information in the target data block can be directly determined based on whether the target data block contains sensitive information, according to the data block library's record of whether the target data block contains sensitive information. If the target data block contains sensitive information, the sensitive information corresponding to the hit historical data block recorded in the data block library is determined as the sensitive information of the target data block.

[0079] If the target data block is a data block that has not been detected in the past, then regardless of whether the target data block contains sensitive information, the hash value of the target data block and whether it contains sensitive information must be recorded in the data block database to update the second hash value of the data block database. The next time the data block is detected, the database records can be used to directly determine whether the data block contains sensitive information, thus improving scanning efficiency.

[0080] For target data blocks containing sensitive information, the first hash value and the corresponding sensitive information of the target data block need to be recorded in the data block database in order to update the second hash value and the corresponding sensitive information of the data block in the data block database.

[0081] In this embodiment, the first hash value of the newly detected data block is recorded in the data block library. If there is corresponding sensitive information, it is also recorded in the data block library. In subsequent data-level filtering, it can be determined whether the target data block is a previously detected data block and whether it contains sensitive information. Based on the sensitive information recorded in the data block library, the sensitive information in the previously detected data blocks can be directly determined, thereby avoiding sensitive information detection on all files and improving scanning efficiency.

[0082] Optionally, the second file metadata and the second hash value each have corresponding information weights, and the method includes: If a second file metadata corresponding to the first file metadata is matched in the file library, the information weight of the second file metadata corresponding to the first file metadata is updated; and if a second hash value corresponding to the first hash value is matched in the data block library, the information weight of the second hash value corresponding to the first hash value is updated. The second file metadata whose information weight is lower than a preset weight threshold and which does not correspond to the file sensitive information is removed from the file library, and / or the second hash value whose information weight is lower than the preset weight threshold and which does not correspond to the data block sensitive information is removed from the data block library.

[0083] In this embodiment of the application, both the second file metadata in the file library and the second hash value in the data block library are assigned initial information weights upon being added to the library. For example, the initial weight of the newly added information can be set to 1000.

[0084] Each time information in the database is matched, its information weight increases regardless of whether it corresponds to sensitive information. Specifically, if a second file metadata corresponding to the first file metadata is matched in the file database, the information weight of the second file metadata corresponding to the first file metadata is updated; and if a second hash value corresponding to the first hash value is matched in the data block database, the information weight of the second hash value corresponding to the first hash value is updated. In one embodiment, the information weight of information that is not matched is reduced each time it is scanned. For example, the information weight increases by 10 for each match and decreases by 5 for each miss.

[0085] In one embodiment, since the more times information is hit, the greater its information weight will be, information with high information weight can be prioritized for matching in subsequent scans.

[0086] Information records that have not been matched for a long time and do not contain corresponding sensitive information will be cleaned up. Specifically, a preset weight threshold can be set to remove second file metadata with a weight lower than the preset weight threshold and no corresponding file sensitive information from the file database, and / or, second hash values ​​with a weight lower than the preset weight threshold and no corresponding data block sensitive information from the data block database. In one example, the preset weight threshold can be set to 5, and only information not manually entered into the database can be removed during the cleanup.

[0087] In this application embodiment, second file metadata and second hash values ​​that have not been matched for a long time and for which there is no corresponding sensitive information are removed from the database, thereby optimizing the file database and data block database and improving the efficiency of each match.

[0088] Example 2 refer to Figure 2 This is a flowchart illustrating the implementation of a sensitive information detection method for documents according to an embodiment of this application. The sensitive information detection method for documents according to an embodiment of this application may include the following steps: Preparation phase: Locate all SO files on the system, output the paths of the SO files, obtain the SO files to be detected, and input the files to be detected into the sensitive information detection system for detection.

[0089] Step 1: File-level filtering. The first file metadata of the file to be tested is matched against the second file metadata stored in the file database to check if a corresponding second file metadata exists in the database. If a second file metadata matching the first file metadata is found in the database, meaning a previously detected SO file has been matched, further data detection of the file is unnecessary and the data detection process can be skipped. If no second file metadata matching the first file metadata is found in the database, it is first determined whether the file to be tested is a packed or encrypted file. For packed or encrypted SO files, unpacking or other operations are attempted.

[0090] Step 2: Data-level Filtering. Extract partition data from the target partition of the file to be detected, divide the partition data into blocks to obtain target data blocks, and calculate the hash value of each target data block as the first hash value. Match the first hash value of the target data block with the second hash value stored in the data block database to check if a corresponding second hash value exists in the database. If a second hash value corresponding to the first hash value is found in the data block database, meaning a previously detected data block has been matched, then further sensitive data detection of this target data block is unnecessary, and the sensitive data detection process can be skipped. If no second hash value corresponding to the first hash value is found in the data block database, meaning the target data block has not been detected in historical detections, then the target data block needs to undergo the subsequent sensitive data identification process.

[0091] Step 3: Content Analysis. The entropy value of the target data block is used to determine whether it has been encoded or compressed. If the entropy value is higher than a preset threshold, the data in the target data block has been encoded and / or compressed. The encoding and / or compression methods of the target data block are then identified. Based on the encoding and / or compression methods, the target data block is decoded and / or decompressed to obtain the processed target data block. Sensitive information matching is then performed to obtain the sensitive information within the target data block.

[0092] Step 4: Self-learning. The file metadata of the SO files obtained in this detection, whether the SO file contains sensitive information and, if so, the corresponding sensitive information for the SO file, as well as the hash value of the data blocks, whether the data block contains sensitive information and, if so, the corresponding sensitive information for the data block, are recorded in the file database and data block database respectively. The information weights in the databases are updated based on the detection results, and the information in the databases is optimized according to these information weights.

[0093] Step 5: Result Processing. Output the detection report for this sensitive information detection.

[0094] This application embodiment first determines whether the file to be detected is a previously detected file based on the first file metadata and the second file metadata stored in the file library. If the second file metadata corresponding to the first file metadata is matched in the file library, the file to be detected can be determined to be a previously detected file, and the sensitive information can be directly determined based on the previously stored second file metadata without re-detecting the data in the previously detected files. If the second file metadata corresponding to the first file metadata is not matched in the file library, the file to be detected can be determined to be a previously undetected file, and the data in it needs to be detected. However, only the target data blocks in the file to be detected are extracted for sensitive information detection, without detecting all the data. By performing file-level and data-level filtering on the file to be detected in this application embodiment, and only performing sensitive information detection on the target data blocks in files that have not been detected in the past, the low scanning efficiency caused by indiscriminate string extraction from the entire file before detection in the prior art is solved, the scanning efficiency of sensitive information detection is improved, and sensitive information in the file to be detected can be effectively detected.

[0095] This application also provides a sensitive information detection device 30 for documents. Please refer to [link / reference]. Figure 3 ,include: The file acquisition module 310 is used to acquire a file to be detected, wherein the file to be detected has corresponding first file metadata, and the first file metadata is information reflecting the content features and structural features of the file to be detected. The first matching module 320 is used to match the first file metadata with the second file metadata stored in a preset file library. The second file metadata is obtained based on the content features and structural features of historically detected files, and the second file metadata has a record of whether there is sensitive information in the file. The information determination module 330 is used to determine the sensitive information in the file to be detected based on the second file meta-information if a second file meta-information corresponding to the first file meta-information is matched in the file library. The information detection module 340 is used to extract the target data block in the file to be detected and detect the sensitive information in the target data block if no second file meta information corresponding to the first file meta information is matched in the file library, thereby obtaining the sensitive information in the file to be detected.

[0096] Optionally, the file to be detected has multiple partitions, each partition containing corresponding partition data, and the information detection module 340 includes: The target partition determination submodule is used to determine the target partition from the plurality of partitions; The partition data extraction submodule is used to extract partition data from the target partition of the file to be detected; The data segmentation submodule is used to segment the partitioned data into blocks to obtain the target data block, and the target data block has a corresponding first hash value; The second matching submodule is used to match the first hash value with the second hash value stored in the preset data block library. The second hash value is obtained based on the hash value of historically detected data blocks and has a record of whether there is a corresponding sensitive information about the data block. The data information determination submodule is used to determine the sensitive information in the target data block based on the second hash value if a second hash value corresponding to the first hash value is matched in the data block library. The data block information matching submodule is used to perform sensitive information matching on the target data block if no second hash value corresponding to the first hash value is matched in the data block library, so as to obtain the sensitive information in the target data block. The sensitive information determination submodule is used to determine the sensitive information in the target data block as the sensitive information in the file to be detected.

[0097] Optionally, the data segmentation submodule is specifically used for: If the partition data has a preset identifier and / or a preset separator, then the partition data is divided into blocks according to the preset identifier and / or the preset separator; If the partition data does not contain the preset identifier and / or the preset separator, then the partition data is divided into blocks according to the preset data block size.

[0098] Optionally, the data block information matching submodule is specifically used for: Determine the entropy value of the target data block; If the entropy value of the target data block is higher than a preset entropy threshold, then the encoding method and / or compression method of the target data block are identified; According to the encoding method and / or compression method of the target data block, the target data block is decoded and / or decompressed to obtain the processed target data block; Sensitive information matching is performed on the processed target data block to obtain the sensitive information in the target data block.

[0099] Optionally, after determining the entropy value of the target data block, the process includes: If the entropy value of the target data block is lower than the preset entropy threshold, then sensitive information matching is performed on the target data block to obtain the sensitive information in the target data block.

[0100] Optionally, the file library stores the file's sensitive information, and the information determination module 330 includes: The file sensitive information determination submodule is used to determine the sensitive information in the file to be detected based on the file sensitive information corresponding to the second file metadata. The device includes: The file library update module is used to record the first file metadata and sensitive information of the file to be detected into the file library, so as to obtain the updated second file metadata and updated file sensitive information.

[0101] Optionally, the data block library stores sensitive information about the data blocks, and the data information determination submodule includes: Based on the sensitive information of the data block corresponding to the second hash value, the sensitive information in the target data block is determined; The device includes: The data block library update module is used to record the first hash value and sensitive information of the target data block into the data block library to obtain the updated second hash value and the updated sensitive information of the data block.

[0102] Optionally, the second file metadata and the second hash value each have corresponding information weights, and the device includes: The information weight update module is used to update the information weight of the second file meta-information corresponding to the first file meta-information if a second file meta-information corresponding to the first file meta-information is matched in the file library; and to update the information weight of the second hash value corresponding to the first hash value if a second hash value corresponding to the first hash value is matched in the data block library. The information optimization module is used to remove second file meta-information whose information weight is lower than a preset weight threshold and which does not have corresponding file sensitive information from the file library, and / or to remove the second hash value whose information weight is lower than the preset weight threshold and which does not have corresponding data block sensitive information from the data block library.

[0103] This application also provides an electronic device 40, please refer to... Figure 4 It includes a processor 410 and a memory 420, wherein the memory 410 is used to store computer programs; the processor 420 is used to execute the programs stored in the memory 410 to implement the sensitive information detection method of the document described in any embodiment of this application.

[0104] This application also provides a vehicle that includes electronic devices as described in the embodiments of this application.

[0105] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the sensitive information detection method for files described in any embodiment of this application.

[0106] In this application, "multiple" refers to two or more.

[0107] In this application, unless otherwise expressly defined, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0108] The terms “first,” “second,” “third,” “fourth,” etc., in this application (if present) are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0109] In this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, in this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0110] Unless otherwise specified, all steps in this application may be performed sequentially or randomly. For example, if the method includes steps A and B, it means that the method may include steps A and B performed sequentially, or it may include steps B and A performed sequentially. For example, if the method may also include step C, it means that step C may be added to the method in any order. For example, the method may include steps A, B, and C, or it may include steps A, C, and B, or it may include steps C, A, and B, etc.

[0111] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for detecting sensitive information in a document, characterized in that, include: Obtain the file to be detected, wherein the file to be detected has corresponding first file metadata, and the first file metadata is information reflecting the content features and structural features of the file to be detected; The first file metadata is matched with the second file metadata stored in the preset file library. The second file metadata is obtained based on the content and structural features of historically detected files. The second file metadata contains a record of whether there is sensitive information in the file. If a second file metadata corresponding to the first file metadata is matched in the file library, then the sensitive information in the file to be detected is determined based on the second file metadata. If no second file metadata corresponding to the first file metadata is matched in the file library, then the target data block in the file to be detected is extracted, and the sensitive information in the target data block is detected to obtain the sensitive information in the file to be detected.

2. The method for detecting sensitive information in a document according to claim 1, characterized in that, The file to be detected has multiple partitions, each containing corresponding partition data. The process of extracting target data blocks from the file to be detected and detecting sensitive information within those target data blocks to obtain sensitive information from the file to be detected includes: Determine the target partition from the plurality of partitions; Extract partition data from the target partition of the file to be detected; The partitioned data is divided into blocks to obtain the target data block, and the target data block has a corresponding first hash value; The first hash value is matched with a second hash value stored in a preset data block library. The second hash value is obtained based on the hash values ​​of historically detected data blocks. The second hash value has a record of whether there is a corresponding sensitive information about the data block. If a second hash value corresponding to the first hash value is matched in the data block library, the sensitive information in the target data block is determined based on the second hash value; If no second hash value corresponding to the first hash value is found in the data block library, then sensitive information matching is performed on the target data block to obtain the sensitive information in the target data block; The sensitive information in the target data block is identified as the sensitive information in the file to be detected.

3. The method for detecting sensitive information in a document according to claim 2, characterized in that, The step of dividing the partitioned data into blocks includes: If the partition data has a preset identifier and / or a preset separator, then the partition data is divided into blocks according to the preset identifier and / or the preset separator; If the partition data does not contain the preset identifier and / or the preset separator, then the partition data is divided into blocks according to the preset data block size.

4. The method for detecting sensitive information in a document according to claim 2, characterized in that, The step of performing sensitive information matching on the target data block to obtain the sensitive information in the target data block includes: Determine the entropy value of the target data block; If the entropy value of the target data block is higher than a preset entropy threshold, then the encoding method and / or compression method of the target data block are identified; According to the encoding method and / or compression method of the target data block, the target data block is decoded and / or decompressed to obtain the processed target data block; Sensitive information matching is performed on the processed target data block to obtain the sensitive information in the target data block.

5. The method for detecting sensitive information in a document according to claim 4, characterized in that, After determining the entropy value of the target data block, the method includes: If the entropy value of the target data block is lower than the preset entropy threshold, then sensitive information matching is performed on the target data block to obtain the sensitive information in the target data block.

6. The method for detecting sensitive information in a document according to claim 2, characterized in that, The file library stores the file's sensitive information, and the step of determining the sensitive information in the file to be detected based on the second file metadata includes: Based on the file sensitive information corresponding to the second file metadata, the sensitive information in the file to be detected is determined; After obtaining the sensitive information in the file to be detected, the method includes: The first file metadata and sensitive information of the file to be detected are recorded in the file database to obtain the updated second file metadata and updated file sensitive information.

7. The method for detecting sensitive information in a document according to claim 6, characterized in that, The data block library stores the sensitive information of the data blocks, and the step of determining the sensitive information in the target data block based on the second hash value includes: Based on the sensitive information of the data block corresponding to the second hash value, the sensitive information in the target data block is determined; After obtaining the sensitive information in the target data block, the method includes: The first hash value and sensitive information of the target data block are recorded in the data block library to obtain the updated second hash value and the updated sensitive information of the data block.

8. The method for detecting sensitive information in a document according to claim 7, characterized in that, The second file metadata and the second hash value each have corresponding information weights, and the method includes: If a second file metadata corresponding to the first file metadata is matched in the file library, the information weight of the second file metadata corresponding to the first file metadata is updated; and if a second hash value corresponding to the first hash value is matched in the data block library, the information weight of the second hash value corresponding to the first hash value is updated. The second file metadata whose information weight is lower than a preset weight threshold and which does not correspond to the file sensitive information is removed from the file library, and / or the second hash value whose information weight is lower than the preset weight threshold and which does not correspond to the data block sensitive information is removed from the data block library.

9. An electronic device, characterized in that, Including processor and memory, among which Memory, used to store computer programs; A processor is used to execute a program stored in memory to implement the sensitive information detection method for a file as described in any one of claims 1-8.

10. A vehicle, characterized in that, It includes the electronic device as described in claim 9.

Images