Text feature-based file sensitive content detection method and device, equipment and medium
By preprocessing files and generating feature vectors, and combining the file feature library to calculate similarity, the security and accuracy issues of sensitive content detection in public network environments are solved, achieving efficient and accurate detection from cloud servers to mobile terminals.
Patent Information
- Application Number
- CN202511622387.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-07
- Publication Date
- 2026-01-30
Smart Images
Figure CN121434804A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of information security technology, and in particular to a method, apparatus, device, and medium for detecting sensitive content in documents based on text features. Background Technology
[0002] Currently, the methods for detecting sensitive content in files and their limitations are as follows: (1) Precise hashing techniques, such as MD5 (Message Digest Algorithm MD5) and SHA (Secure Hash Algorithm) series, can only identify completely identical file copies and are ineffective against any content modifications (such as format conversion, local addition or deletion); (2) Fuzzy hashing techniques, such as ssdeep (an open-source fuzzy hashing tool) and simhash (Similarity Hashing), can identify local continuous modifications, but have weak resistance to non-continuous tampering and have a high collision probability, resulting in a high false alarm rate, which cannot meet the requirements for detecting data fragment content in public network deployment scenarios; (3) Deep learning embedding model technology can capture semantic similarity, but the model parameters are large and the computational overhead is high, making it difficult to run efficiently and in real time in resource-constrained scenarios such as servers, PC (Personal Computer) clients, mobile terminals and edge devices. At the same time, the feature vectors generated by it have a major security risk of being reverse-engineered by attack techniques such as vec2text to restore the original text fragments.
[0003] As can be seen from the above, how to meet the highest data confidentiality requirements for secure deployment in untrusted environments such as the public network, adapt to the full-scenario, low-resource deployment needs from cloud servers to mobile terminals, solve the security, performance and accuracy challenges in cross-network and cross-device file sensitive content detection, and improve the accuracy of file sensitive content detection and reduce the probability of collisions are problems that need to be solved in this field. Summary of the Invention
[0004] In view of this, the purpose of this invention is to provide a method, apparatus, device, and medium for detecting sensitive content in files based on text features. This method can meet the highest data confidentiality requirements for secure deployment in untrusted environments such as public networks, adapt to all scenarios and low-resource deployment needs from cloud servers to mobile terminals, solve the challenges of security, performance, and accuracy in detecting sensitive content in files across networks and devices, and improve the accuracy of sensitive content detection. The specific solution is as follows:
[0005] Firstly, this application discloses a method for detecting sensitive content in documents based on text features, including:
[0006] The file to be tested is preprocessed to obtain text blocks; the preprocessing includes content extraction, character cleaning, and text segmentation.
[0007] The text block is segmented to obtain a vocabulary set. The hash value of each word in the vocabulary set is calculated. The hash value is mapped to different index positions of the initial feature vector to obtain the feature vector. The feature vector is normalized and quantized to obtain the target feature vector.
[0008] The target feature vectors are concatenated to obtain a text feature set;
[0009] A set of file feature samples is selected from a preset file feature library, and the text similarity between the text feature set and the set of file feature samples is calculated.
[0010] The text similarity is used to detect sensitive content in the file to be detected.
[0011] Optionally, the preprocessing of the file to be detected to obtain text blocks includes:
[0012] Obtain the file to be detected, and use a parser to extract the content of the file to obtain the text content;
[0013] The control characters and special symbols in the text content are cleaned to obtain the text content including valid characters;
[0014] The text content, including valid characters, is divided into text blocks according to a preset text block length to obtain each text block.
[0015] Optionally, the preprocessing of the file to be detected to obtain text blocks includes:
[0016] Obtain the file to be detected, and use a parser to extract the content of the file to obtain the text content;
[0017] The control characters and special symbols in the text content are cleaned to obtain the text content including valid characters;
[0018] The text content, including valid characters, is divided into text blocks according to a preset text block length to obtain each text block.
[0019] Optionally, the step of normalizing and quantizing the feature vector to obtain the target feature vector includes:
[0020] The feature vector is normalized using the L2 norm and then quantized using 8 bits to obtain the target feature vector.
[0021] Optionally, the step of normalizing and quantizing the feature vector to obtain the target feature vector includes:
[0022] The feature vector is normalized using the L2 norm and then quantized using 8 bits to obtain the target feature vector.
[0023] Optionally, calculating the text similarity between the text feature set and the document feature sample set includes:
[0024] The spatial distance algorithm is used to calculate the similarity between the text feature set and each file feature sample in the file feature sample set, and the highest similarity is taken as the initial text similarity.
[0025] The text similarity score is obtained by calculating a weighted average of all the initial text similarities.
[0026] Optionally, the step of using the text similarity to detect sensitive content in the file to be detected includes:
[0027] Set detection thresholds according to business needs;
[0028] Determine whether the text similarity is greater than the detection threshold;
[0029] If the text similarity is greater than the detection threshold, it is determined that the sensitive content of the file to be detected has been leaked.
[0030] If the text similarity is not greater than the detection threshold, it is determined that the sensitive content of the file to be detected has not been leaked.
[0031] Secondly, this application discloses a document sensitive content detection device based on text features, comprising:
[0032] The preprocessing module is used to preprocess the file to be detected to obtain text blocks; preprocessing includes content extraction, character cleaning, and text segmentation.
[0033] The normalization and feature quantization processing module is used to segment the text block to obtain a vocabulary set, calculate the hash value of each word in the vocabulary set, map the hash value to different index positions of the initial feature vector to obtain the feature vector, and perform normalization and feature quantization processing on the feature vector to obtain the target feature vector.
[0034] The concatenation module is used to concatenate the target feature vectors to obtain a text feature set;
[0035] The text similarity calculation module is used to select a set of file feature samples from a preset file feature library and calculate the text similarity between the text feature set and the set of file feature samples.
[0036] The file sensitive content detection module is used to detect sensitive content in the file to be detected by using the text similarity.
[0037] Thirdly, this application discloses an electronic device, including:
[0038] Memory, used to store computer programs;
[0039] A processor is used to execute the computer program to implement the aforementioned method for detecting sensitive content in documents based on text features.
[0040] Fourthly, this application discloses a computer storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the aforementioned disclosed method for detecting sensitive file content based on text features.
[0041] As can be seen, this application provides a method for detecting sensitive content in files based on text features, including preprocessing the file to be detected to obtain text blocks; the preprocessing includes content extraction, character cleaning, and text segmentation; segmenting the text blocks to obtain a vocabulary set, calculating the hash value of each word in the vocabulary set, mapping the hash value to different index positions of an initial feature vector to obtain a feature vector, normalizing and quantizing the feature vector to obtain a target feature vector; concatenating the target feature vector to obtain a text feature set; selecting a set of file feature samples from a preset file feature library, calculating the text similarity between the text feature set and the set of file feature samples; and using the text similarity to detect sensitive content in the file to be detected. This application preprocesses the document to be detected to obtain text blocks, segments these blocks to obtain a vocabulary set, calculates the hash value of each word in the vocabulary set, and maps the hash values to different index positions of the initial feature vector to obtain the feature vector. This method randomly and disperses word frequency information to different positions of the high-dimensional vector, completely destroying the original semantic and sequence information while preserving statistical information, ensuring the irreversibility, low collision rate, and security of the features. The feature vector is then normalized and quantized to obtain the target feature vector, which eliminates the influence of text length, improves the accuracy of similarity calculation, greatly compresses feature storage space, and facilitates large-scale deployment. Finally, the target feature vector is concatenated to obtain the text features. The system selects a set of file feature samples from a pre-defined file feature library, calculates the text similarity between the text feature set and the file feature sample set, and achieves accurate capture of local content similarity by dividing the document into blocks and generating features independently, combined with cross-block similarity calculation. This improves the ability to identify partial plagiarism and content tampering, and uses text similarity to detect sensitive content in the file to be tested. It meets the highest data confidentiality requirements for secure deployment in untrusted environments such as public networks, and is adaptable to all scenarios and low-resource deployment needs from cloud servers to mobile terminals. It solves the challenges of security, performance and accuracy in cross-network and cross-device file sensitive content detection, and improves the accuracy of file sensitive content detection. Attached Figure Description
[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0043] Figure 1 This is a flowchart of a document sensitive content detection method based on text features disclosed in this application;
[0044] Figure 2 This is a flowchart illustrating a specific process for generating a text feature set as disclosed in this application;
[0045] Figure 3 This is a flowchart illustrating a specific method for calculating text similarity as disclosed in this application;
[0046] Figure 4 This is a schematic diagram of a document sensitive content detection device based on text features disclosed in this application;
[0047] Figure 5 This application provides a structural diagram of an electronic device. Detailed Implementation
[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0049] Currently, the methods for detecting sensitive content in files and their limitations are as follows: (1) Precise hashing techniques, such as MD5 and SHA series, can only identify identical file copies and are ineffective against any content modifications (such as format conversion, local addition or deletion); (2) Fuzzy hashing techniques, such as ssdeep and simhash, can identify local continuous modifications, but have weak resistance to non-continuous tampering and have a high collision probability, resulting in a high false alarm rate, which cannot meet the requirements for detecting data fragment content in public network deployment scenarios; (3) Deep learning embedding model technology can capture semantic similarity, but the model parameters are large and the computational overhead is high, making it difficult to run efficiently and in real time in resource-constrained scenarios such as servers, PC clients, mobile terminals and edge devices. At the same time, the feature vectors generated by it have a major security risk of being reverse-engineered by attack techniques such as vec2text to restore the original text fragments. As can be seen from the above, how to meet the highest data confidentiality requirements for secure deployment in untrusted environments such as the public network, adapt to the full-scenario, low-resource deployment needs from cloud servers to mobile terminals, solve the security, performance and accuracy challenges in cross-network and cross-device file sensitive content detection, and improve the accuracy of file sensitive content detection and reduce the probability of collisions are problems that need to be solved in this field.
[0050] See Figure 1 As shown in the figure, this invention discloses a method for detecting sensitive content in documents based on text features, which may specifically include:
[0051] Step S11: Preprocess the file to be detected to obtain text blocks; preprocessing includes content extraction, character cleaning, and text segmentation.
[0052] In this embodiment, a file to be detected is obtained, and a parser is used to extract the content of the file to obtain text content; the control characters and special symbols in the text content are cleaned to obtain text content including valid characters; the text content including valid characters is divided into text blocks according to a preset text block length to obtain each text block.
[0053] Specifically, the process involves acquiring the file to be detected, recursively extracting it using a parser such as Apache Tika, and obtaining the text content. This text content includes nested documents, OCR (Optical Character Recognition) results, and compressed files. The text content is then cleaned, retaining only valid characters such as Chinese, English, and numbers, while removing control characters and special symbols, resulting in a continuous text stream containing valid characters. This continuous text stream is then segmented according to a preset text block length (300-500 characters). If the last block is insufficient, it is padded with the previous block, forming independent blocks and ensuring the independence of local features. This lays the foundation for capturing leaked fragment content.
[0054] Step S12: Segment the text block to obtain a vocabulary set, calculate the hash value of each word in the vocabulary set, map the hash value to different index positions of the initial feature vector to obtain the feature vector, and perform normalization and feature quantization processing on the feature vector to obtain the target feature vector.
[0055] In this embodiment, a preset fixed word segmenter is used to segment the text block to obtain a vocabulary set; the frequency of each word in the vocabulary set in the text block is calculated; a hash function is used to calculate the hash value of each word in the vocabulary set; based on the frequency, the hash value is mapped to different index positions of the initial feature vector, and then the word frequency value of the word corresponding to the frequency is accumulated to the corresponding index position to obtain the feature vector; the feature vector is subjected to L2 norm normalization processing and 8-bit quantization processing to obtain the target feature vector.
[0056] This application utilizes a preset fixed word segmenter to segment text blocks, such as the Jieba Chinese word segmenter or the HanLP word segmenter, to obtain a vocabulary set. Simultaneously, it calculates the TF (frequency) of each word within the text block. Then, it uses hash functions with different seeds to calculate the hash value of each word in the vocabulary set multiple times, obtaining a hash value set for each word. The hash value sets of all words are mapped to different index positions of the initial feature vector to obtain the feature vector Vd of the text block. The feature vector Vd is then subjected to L2 norm normalization to eliminate the influence of text length, and 8-bit quantization is performed to compress the storage space of Vd to 25% of its original size.
[0057] The hash value mapping process is as follows: Initialize the D-dimensional feature Vd (D=128-1024), with an initial value of 0 for each index position. For each word w, perform the following:
[0058] For j=1 to K:
[0059] index=MurmurHash3(word,seedj) modD
[0060] vector[index] += TF(word)
[0061] The specific steps are as follows:
[0062] (1) Select different seeds from the preset seed bank;
[0063] (2) Calculate the hash value of each word w by using a fixed, efficient hash function, such as MurmurHash3 or XXHash. Each word w will be calculated K times: hash(w)_1, hash(w)_..., hash(w)_K;
[0064] (3) Map each hash value result of the word w to different positions of the index of the initial feature vector: index_w1, index_..., index_wK (by modulo operation), and increment the value of the mapped position by 1;
[0065] (4) Accumulate the word frequency value fq of word w (or the product of word frequency and word weight) to the corresponding index positions index_w1, index_..., index_wK to obtain the feature vector.
[0066] Hash mapping can disrupt the block semantic structure and significantly reduce the probability of word collisions, achieving irreversibility, low collision rate, and no preservation of word features.
[0067] Step S13: Concatenate the target feature vectors to obtain the text feature set.
[0068] In this embodiment, all block feature vectors are concatenated in block order to form the final text feature set of the file. .
[0069] In this embodiment, the specific process for generating the text feature set is as follows: Figure 2As shown, the normalization in this application eliminates the influence of text length, improves the accuracy of similarity calculation, and the quantization process greatly compresses the feature storage space (e.g., reduces it by more than 75%), which is convenient for large-scale deployment. Furthermore, this invention uses multiple independent seeds to perform multiple hash mappings on the same word, randomly and dispersedly projecting word frequency information to different positions of the high-dimensional vector. While retaining statistical information, it completely destroys the original semantic and sequence information, ensuring the irreversibility, low collision rate and security of features.
[0070] This application is applicable to cross-network environments and resource-constrained devices, and can prevent the leakage of original information. The algorithm of this invention only relies on basic operations, such as hashing, addition, and normalization, without loading or running any pre-trained models. Its memory usage is only the feature vector itself, such as a 300-dimensional floating-point vector ≈ 1.2KB, and the computational complexity is linear O(n). Feature extraction can be completed in milliseconds, perfectly adapting to the low-resource deployment requirements of all scenarios from cloud servers to mobile terminals. Deep learning models, on the other hand, require GB-level memory and second-level latency, making edge deployment difficult.
[0071] Furthermore, the features generated by this invention are mathematical representations of word frequency statistics after irreversible hash mapping, completely lacking semantic information or sequence structure that can be used to reconstruct the original text. This fundamentally eliminates reverse engineering attacks such as vec2text and meets the highest data confidentiality requirements for secure deployment in untrusted environments such as the public internet. In contrast, features from deep learning models have the risk of reversibility.
[0072] Step S14: Select a set of file feature samples from the preset file feature library, and calculate the text similarity between the text feature set and the set of file feature samples.
[0073] In this embodiment, file feature samples that meet the preset monitoring conditions are stored in a preset file feature library deployed in a retrieval domain under the public network; or, file feature samples that meet the preset monitoring conditions are stored in a preset file feature library in the detection device, and then a set of file feature samples is selected from the preset file feature library. The similarity between the text feature set and each file feature sample in the file feature sample set is calculated using a spatial distance algorithm, and the similarity with the largest value is taken as the initial text similarity. The weighted average of all the initial text similarities is calculated to obtain the text similarity.
[0074] In this embodiment, the specific process for calculating text similarity is as follows: Figure 3 As shown, the set of file feature samples of sensitive files that need to be monitored is stored in a preset file feature library in the retrieval domain (which can be deployed on the public network) or the detection device. The set of file feature samples is then selected from the preset file feature library to be detected. For each feature block, a spatial distance algorithm (such as cosine or inner product) is used to calculate its relationship with each sample file. The initial text similarity of all block features is calculated, and the maximum similarity result is taken. Finally, a weighted average is used to calculate the similarity between all blocks of the file G_query to be detected and all blocks of the sample file G_sample, which is the text similarity sim of the file. Features of the first block Features of the second block For the features of the m-th block, The initial text similarity for the first block. The initial text similarity for the second block. The initial text similarity is for the nth block.
[0075] This application achieves accurate capture of local content similarity by dividing documents into blocks and generating features independently, and combining the cross-similarity calculation between blocks, thus significantly improving the ability to identify partial plagiarism and content tampering.
[0076] Step S15: Detect sensitive content in the file to be detected using the text similarity.
[0077] In this embodiment, a detection threshold is set according to business requirements; it is determined whether the text similarity is greater than the detection threshold; if the text similarity is greater than the detection threshold, it is determined that the sensitive content of the file to be detected has been leaked; if the text similarity is not greater than the detection threshold, it is determined that the sensitive content of the file to be detected has not been leaked.
[0078] Specifically, a detection threshold is set according to business needs. For example, if the detection threshold r is 0.85, it is determined whether the text similarity sim is greater than r. If it is greater, it is determined that the sensitive content of the file to be detected has been leaked. If it is not greater, it is determined that the sensitive content of the file to be detected has not been leaked.
[0079] This application can not only identify overall similar files, but also accurately locate and quantify local content plagiarism, such as a paragraph of text or a table. Its robustness to non-continuous tampering (such as inserting irrelevant paragraphs) and semantic transformation (such as partial synonym replacement) is superior to or equal to that of deep learning models and far superior to traditional hashing techniques, while maintaining extremely high processing efficiency. It can solve the security, performance and accuracy problems in detecting sensitive content in files across networks and devices, and improve the robustness and sensitivity of features to complex content tampering and local plagiarism. It can effectively identify various leakage modes, from full-text copying and paragraph plagiarism to the theft of key data fragments. Under the premise of ensuring that the original file content is absolutely irreversible, it can accurately characterize the statistical distribution characteristics of file content to support similarity calculation, and can be securely applied to untrusted environments such as the public network for storage and comparison.
[0080] In this embodiment, the file to be detected is preprocessed to obtain text blocks. The preprocessing includes content extraction, character cleaning, and text segmentation. The text blocks are segmented to obtain a vocabulary set. The hash value of each word in the vocabulary set is calculated, and the hash value is mapped to different index positions of the initial feature vector to obtain a feature vector. The feature vector is normalized and quantized to obtain a target feature vector. The target feature vector is concatenated to obtain a text feature set. A set of file feature samples is selected from a preset file feature library, and the text similarity between the text feature set and the set of file feature samples is calculated. The text similarity is used to detect sensitive content in the file to be detected. This application preprocesses the document to be detected to obtain text blocks, segments these blocks to obtain a vocabulary set, calculates the hash value of each word in the vocabulary set, and maps the hash values to different index positions of the initial feature vector to obtain the feature vector. This method randomly and disperses word frequency information to different positions of the high-dimensional vector, completely destroying the original semantic and sequence information while preserving statistical information, ensuring the irreversibility, low collision rate, and security of the features. The feature vector is then normalized and quantized to obtain the target feature vector, which eliminates the influence of text length, improves the accuracy of similarity calculation, greatly compresses feature storage space, and facilitates large-scale deployment. Finally, the target feature vector is concatenated to obtain the text features. The system selects a set of file feature samples from a pre-defined file feature library, calculates the text similarity between the text feature set and the file feature sample set, and achieves accurate capture of local content similarity by dividing the document into blocks and generating features independently, combined with cross-block similarity calculation. This improves the ability to identify partial plagiarism and content tampering, and uses text similarity to detect sensitive content in the file to be tested. It meets the highest data confidentiality requirements for secure deployment in untrusted environments such as public networks, and is adaptable to all scenarios and low-resource deployment needs from cloud servers to mobile terminals. It solves the challenges of security, performance and accuracy in cross-network and cross-device file sensitive content detection, and improves the accuracy of file sensitive content detection.
[0081] See Figure 4 As shown, this embodiment of the invention discloses a document sensitive content detection device based on text features, which may specifically include:
[0082] Preprocessing module 11 is used to preprocess the file to be detected to obtain text blocks; preprocessing includes content extraction, character cleaning, and text segmentation;
[0083] The normalization and feature quantization processing module 12 is used to segment the text block to obtain a vocabulary set, calculate the hash value of each word in the vocabulary set, map the hash value to different index positions of the initial feature vector to obtain the feature vector, and perform normalization and feature quantization processing on the feature vector to obtain the target feature vector.
[0084] The splicing module 13 is used to splice the target feature vector to obtain a text feature set;
[0085] The text similarity calculation module 14 is used to select a set of file feature samples from a preset file feature library and calculate the text similarity between the text feature set and the set of file feature samples.
[0086] The file sensitive content detection module 15 is used to detect sensitive content in the file to be detected by using the text similarity.
[0087] In this embodiment, the file to be detected is preprocessed to obtain text blocks. The preprocessing includes content extraction, character cleaning, and text segmentation. The text blocks are segmented to obtain a vocabulary set. The hash value of each word in the vocabulary set is calculated, and the hash value is mapped to different index positions of the initial feature vector to obtain a feature vector. The feature vector is normalized and quantized to obtain a target feature vector. The target feature vector is concatenated to obtain a text feature set. A set of file feature samples is selected from a preset file feature library, and the text similarity between the text feature set and the set of file feature samples is calculated. The text similarity is used to detect sensitive content in the file to be detected. This application preprocesses the document to be detected to obtain text blocks, segments these blocks to obtain a vocabulary set, calculates the hash value of each word in the vocabulary set, and maps the hash values to different index positions of the initial feature vector to obtain the feature vector. This method randomly and disperses word frequency information to different positions of the high-dimensional vector, completely destroying the original semantic and sequence information while preserving statistical information, ensuring the irreversibility, low collision rate, and security of the features. The feature vector is then normalized and quantized to obtain the target feature vector, which eliminates the influence of text length, improves the accuracy of similarity calculation, greatly compresses feature storage space, and facilitates large-scale deployment. Finally, the target feature vector is concatenated to obtain the text features. The system selects a set of file feature samples from a pre-defined file feature library, calculates the text similarity between the text feature set and the file feature sample set, and achieves accurate capture of local content similarity by dividing the document into blocks and generating features independently, combined with cross-block similarity calculation. This improves the ability to identify partial plagiarism and content tampering, and uses text similarity to detect sensitive content in the file to be tested. It meets the highest data confidentiality requirements for secure deployment in untrusted environments such as public networks, and is adaptable to all scenarios and low-resource deployment needs from cloud servers to mobile terminals. It solves the challenges of security, performance and accuracy in cross-network and cross-device file sensitive content detection, and improves the accuracy of file sensitive content detection.
[0088] In some specific embodiments, the preprocessing module 11 may specifically include:
[0089] The content extraction module is used to acquire the file to be detected, and to extract the content of the file to be detected using a parser to obtain the text content;
[0090] The cleaning module is used to clean the control characters and special symbols in the text content to obtain the text content including valid characters;
[0091] The text segmentation module is used to segment the text content, which includes valid characters, into text blocks according to a preset text block length, thereby obtaining each text block.
[0092] In some specific embodiments, the normalization and feature quantization processing module 12 may specifically include:
[0093] The segmentation module is used to segment text blocks using a preset fixed word segmenter to obtain a vocabulary set;
[0094] The frequency calculation module is used to calculate the frequency of each word in the vocabulary set within the text block.
[0095] The hash value calculation module is used to calculate the hash value of each word in the vocabulary set using a hash function;
[0096] The hash value mapping module is used to map the hash value to different index positions of the initial feature vector based on the frequency, and then accumulate the word frequency value of the word corresponding to the frequency to the corresponding index position to obtain the feature vector.
[0097] In some specific embodiments, the normalization and feature quantization processing module 12 may specifically include:
[0098] The L2 norm normalization and 8-bit quantization modules are used to perform L2 norm normalization on the feature vector and 8-bit quantization on the feature vector to obtain the target feature vector.
[0099] In some specific embodiments, the text similarity calculation module 14 may specifically include:
[0100] The first file feature sample storage module is used to store file feature samples that meet the preset monitoring conditions into a preset file feature library deployed in the retrieval domain under the public network;
[0101] The second document feature sample storage module is used to store document feature samples that meet preset monitoring conditions into the preset document feature library in the detection device.
[0102] In some specific embodiments, the text similarity calculation module 14 may specifically include:
[0103] The initial text similarity calculation module is used to calculate the similarity between the text feature set and each file feature sample in the file feature sample set using a spatial distance algorithm, and the highest similarity value is taken as the initial text similarity.
[0104] The weighted average calculation module is used to calculate the weighted average of all initial text similarities to obtain the text similarity score.
[0105] In some specific embodiments, the file sensitive content detection module 15 may specifically include:
[0106] The detection threshold setting module is used to set the detection threshold according to business requirements;
[0107] The judgment module is used to determine whether the text similarity is greater than the detection threshold;
[0108] The sensitive content leakage determination module is used to determine that the sensitive content of the file to be detected has been leaked if the text similarity is greater than the detection threshold.
[0109] The document sensitive content non-disclosure determination module is used to determine that the document sensitive content of the document to be detected has not been disclosed if the text similarity is not greater than the detection threshold.
[0110] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the text feature-based file sensitive content detection method disclosed in any of the foregoing embodiments.
[0111] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.
[0112] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored on it include operating system 221, computer program 222 and data 223, etc., and the storage method can be temporary storage or permanent storage.
[0113] The operating system 221 manages and controls the various hardware devices on the electronic device 20 and the computer program 222 to enable the processor 21 to perform operations and processing on the data 223 in the memory 22. The operating system 221 can be Windows, Unix, Linux, etc. The computer program 222, in addition to including a computer program capable of performing the text-feature-based file sensitive content detection method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, may further include computer programs capable of performing other specific tasks. The data 223 may include data received by the text-feature-based file sensitive content detection device from external devices, as well as data collected by its own input / output interface 25.
[0114] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.
[0115] Furthermore, embodiments of this application also disclose a computer-readable storage medium storing a computer program. When the computer program is loaded and executed by a processor, it implements the steps of the document sensitive content detection method based on text features disclosed in any of the foregoing embodiments.
[0116] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0117] The foregoing has provided a detailed description of a method, apparatus, device, and storage medium for detecting sensitive content in documents based on text features, as provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. A method for detecting sensitive content of a file based on text features, characterized in that, The method comprises the following steps: preprocessing a to-be-detected file to obtain text blocks; the preprocessing comprises content extraction, character cleaning, and text blocking; segmenting the text blocks to obtain a vocabulary set, calculating hash values of each vocabulary in the vocabulary set, mapping the hash values to different index positions of an initial feature vector to obtain a feature vector, normalizing and quantizing the feature vector to obtain a target feature vector; splicing the target feature vectors to obtain a text feature set; filtering a file feature sample set from a preset file feature library, and calculating a text similarity between the text feature set and the file feature sample set; performing file sensitive content detection on the to-be-detected file by using the text similarity.
2. The method of claim 1, wherein, The preprocessing of the to-be-detected file to obtain the text blocks comprises the following steps: obtaining a to-be-detected file, and extracting content of the to-be-detected file by using a parser to obtain text content; cleaning control symbols and special symbols in the text content to obtain the text content comprising valid characters; blocking the text content comprising valid characters according to a preset text block length to obtain text blocks. 3.The method of claim 1, wherein, The segmentation of the text blocks to obtain a vocabulary set, the calculation of hash values of each vocabulary in the vocabulary set, and the mapping of the hash values to different index positions of an initial feature vector to obtain a feature vector comprise the following steps: segmenting a text block by using a preset fixed segmenter to obtain a vocabulary set; calculating the frequency of each vocabulary in the vocabulary set in the text block; calculating hash values of each vocabulary in the vocabulary set by using a hash function; mapping the hash values to different index positions of an initial feature vector based on the frequency, and then adding the term frequency value of the vocabulary corresponding to the frequency to the corresponding index position to obtain a feature vector.
4. The method of claim 1, wherein, The normalization and quantization of the feature vector to obtain a target feature vector comprise the following steps: performing L2 norm normalization on the feature vector, and performing 8-bit quantization on the feature vector to obtain a target feature vector.
5. The method of claim 1, wherein, Before the filtering of the file feature sample set from the preset file feature library, the following steps are further included: storing file feature samples satisfying a preset monitoring condition in a preset file feature library deployed in a public network; or, storing file feature samples satisfying a preset monitoring condition in a preset file feature library in a detection device.
6. The method of claim 1, wherein, The calculation of the text similarity between the text feature set and the file feature sample set comprises the following steps: calculating the similarity between the text feature set and each file feature sample in the file feature sample set by using a spatial distance algorithm, taking the maximum numerical value as an initial text similarity; performing weighted average calculation on all initial text similarities to obtain a text similarity.
7. The method of claim 1 to 6, wherein, The file sensitive content detection on the to-be-detected file by using the text similarity comprises the following steps: setting a detection threshold according to business requirements; determining whether the text similarity is greater than the detection threshold; if the text similarity is greater than the detection threshold, determining that the file sensitive content of the to-be-detected file is leaked. If the text similarity is not greater than the detection threshold, it is determined that the file sensitive content of the to-be-detected file is not leaked.
8. An apparatus for detecting sensitive content of a file based on text features, the apparatus comprising: The method comprises the steps of: A preprocessing module is configured to preprocess the to-be-detected file to obtain text blocks. Preprocessing includes content extraction, character cleaning, and text blocking. A normalization and feature quantization processing module is configured to split the text blocks to obtain a vocabulary set, calculate hash values of each vocabulary in the vocabulary set, map the hash values to different index positions of an initial feature vector to obtain a feature vector, and perform normalization and feature quantization processing on the feature vector to obtain a target feature vector. A splicing module is configured to splice the target feature vector to obtain a text feature set. A text similarity calculation module is configured to filter a file feature sample set from a preset file feature library and calculate a text similarity between the text feature set and the file feature sample set. A file sensitive content detection module is configured to perform file sensitive content detection on the to-be-detected file by using the text similarity.
9. An electronic device, comprising: The method comprises the steps of: A memory is configured to save a computer program. A processor is configured to execute the computer program to implement the file sensitive content detection method based on text features according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, A memory is configured to save a computer program. When the computer program is executed by a processor, the file sensitive content detection method based on text features according to any one of claims 1 to 7 is implemented.