A text sensitive information detection method, device and equipment and storage medium
By preprocessing and vectorizing long texts, and combining them with a modified Faster R-CNN model, the problem of the inability to accurately identify sensitive information in long texts in existing technologies has been solved, and flexible sensitive information detection has been achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANDONG INSPUR SCI RES INST CO LTD
- Filing Date
- 2023-09-28
- Publication Date
- 2026-05-29
AI Technical Summary
Existing methods for identifying sensitive information cannot accurately identify sensitive information appearing in long texts, and their identification range is small and inflexible.
The text is preprocessed using a preset text processing method, and then transformed into text vectors using a preset training corpus length and word embedding model. A modified Faster R-CNN model is then constructed to detect sensitive information, and finally the recognition result is determined.
It achieves accurate identification of sensitive information in long texts, improves the flexibility of detection, and eliminates the need to pre-set detection categories.
Smart Images
Figure CN117235209B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of deep learning technology, and in particular to a method, apparatus, device, and storage medium for detecting sensitive text information. Background Technology
[0002] To address the issue of privacy breaches, data anonymization technology has emerged. The process involves first identifying sensitive information, and then performing anonymization operations such as replacement and encryption on the identified sensitive information.
[0003] Currently, traditional methods for sensitive information identification involve manually specifying data or using regular expressions to match and identify text. These methods can only identify single data items as a whole, resulting in a limited scope, lack of flexibility, and an inability to accurately identify sensitive information appearing in long texts. Therefore, a solution needs to be found for accurately identifying sensitive information appearing in long texts. Summary of the Invention
[0004] In view of this, the purpose of this invention is to provide a method, apparatus, device, and storage medium for detecting sensitive information in text, capable of accurately identifying sensitive information appearing in long texts. The specific solution is as follows:
[0005] Firstly, this application discloses a method for detecting sensitive text information, including:
[0006] The obtained text to be detected is preprocessed based on a preset text processing method to obtain the processed text.
[0007] The processed file is transformed and concatenated based on a preset training corpus length and a preset word embedding model to obtain the target text vector.
[0008] A target sensitive information detection model is constructed based on a preset target recognition model, and the target text vector is input into the target sensitive information detection model to obtain the target recognition result;
[0009] Based on the target recognition results, the sensitive information recognition results of the text to be detected are determined.
[0010] Optionally, the step of preprocessing the acquired text to be detected based on a preset text processing method to obtain processed text includes:
[0011] The obtained text to be detected is segmented into words based on the bidirectional maximum matching algorithm to obtain the segmented text.
[0012] The characters in the segmented text are adjusted to obtain the processed text.
[0013] Optionally, the step of transforming and concatenating the processed file based on a preset training corpus length and a preset word embedding model to obtain the target text vector includes:
[0014] The character length in the processed file is adjusted to the target length based on the preset corpus length to obtain the adjusted text;
[0015] The characters in the adjusted text are converted into word vectors based on the continuous bag-of-words model, and all the word vectors are concatenated to obtain the target text vector.
[0016] Optionally, adjusting the character length in the processed file to a target length based on a preset corpus length to obtain the adjusted text includes:
[0017] The character length in the processed file is adjusted to the target length based on preset padding code instructions and preset corpus length to obtain the adjusted text.
[0018] Optionally, the step of constructing a target sensitive information detection model based on a preset target recognition model includes:
[0019] Modify the Anchor and RPN in the Faster R-CNN model to obtain a target-sensitive information detection model.
[0020] Optionally, determining the sensitive information identification result of the text to be detected based on the target identification result includes:
[0021] The target recognition result is input to the text to be detected through a preset mapping method to obtain the sensitive information recognition result of the text to be detected.
[0022] Secondly, this application discloses a text-sensitive information detection device, comprising:
[0023] The text preprocessing module is used to preprocess the acquired text to be detected based on a preset text processing method to obtain the processed text.
[0024] The text conversion module is used to convert and concatenate the processed file based on a preset training corpus length and a preset word embedding model to obtain a target text vector.
[0025] The model recognition module is used to construct a target sensitive information detection model based on a preset target recognition model, and input the target text vector into the target sensitive information detection model to obtain the target recognition result;
[0026] The result determination module is used to determine the sensitive information recognition result of the text to be detected based on the target recognition result.
[0027] Optionally, the text preprocessing module includes:
[0028] The text segmentation unit is used to segment the obtained text to be detected based on the bidirectional maximum matching algorithm to obtain the segmented text.
[0029] A character adjustment unit is used to adjust the characters in the segmented text to obtain the processed text.
[0030] Thirdly, this application discloses an electronic device, comprising:
[0031] Memory, used to store computer programs;
[0032] A processor is used to execute the computer program to implement the aforementioned text-sensitive information detection method.
[0033] Fourthly, this application discloses a computer-readable storage medium for storing a computer program, which, when executed by a processor, implements the aforementioned text-sensitive information detection method.
[0034] As can be seen, in this application, the acquired text to be detected is first preprocessed using a preset text processing method to obtain processed text; the processed text is then transformed and concatenated based on a preset training corpus length and a preset word embedding model to obtain a target text vector; a target sensitive information detection model is constructed based on a preset target recognition model, and the target text vector is input into the target sensitive information detection model to obtain a target recognition result; the sensitive information recognition result of the text to be detected is determined based on the target recognition result. That is, the acquired text to be detected is preprocessed, transformed, and concatenated to obtain a target text vector with uniform length and dimensions; then, the target text vector is recognized by the target sensitive information detection model constructed based on the preset target recognition model to obtain a target recognition result, thereby determining the sensitive information recognition result. This solves the drawback of traditional sensitive information detection algorithms that can only detect the entire input segment, and also eliminates the need to pre-define the detection categories, thus improving the flexibility of detection. Attached Figure Description
[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0036] Figure 1 This is a flowchart of a text-sensitive information detection method disclosed in this application;
[0037] Figure 2 This is a schematic diagram of a specific text-sensitive information detection method disclosed in this application;
[0038] Figure 3 This is a flowchart of a specific text-sensitive information detection method disclosed in this application;
[0039] Figure 4 This is a schematic diagram of the structure of a text-sensitive information detection device disclosed in this application;
[0040] Figure 5 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation
[0041] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0042] Currently, traditional methods for sensitive information identification involve manually specifying data or using regular expressions to match and identify text. These methods can only identify single data items as a whole, resulting in a limited scope, lack of flexibility, and inability to accurately identify sensitive information appearing in long texts. This application will specifically introduce a method that can accurately identify sensitive information in the text to be detected.
[0043] See Figure 1 As shown in the figure, this application discloses a method for detecting sensitive text information, including:
[0044] Step S11: Perform text preprocessing on the obtained text to be detected based on the preset text processing method to obtain the processed text.
[0045] In this embodiment, the text preprocessing of the acquired text to be detected based on a preset text processing method to obtain processed text includes: performing word segmentation on the acquired text to be detected based on a bidirectional maximum matching algorithm to obtain segmented text; and adjusting the characters in the segmented text to obtain processed text. That is, the bidirectional maximum matching algorithm is used to segment the numbers, symbols, etc., of the text to be detected one by one, and then stop words are removed. This removes some meaningless words from the text, improving the accuracy of subsequent text recognition.
[0046] Step S12: Based on the preset training corpus length and preset word embedding model, the processed file is transformed and concatenated to obtain the target text vector.
[0047] In this embodiment, adjusting the character length in the processed file to a target length based on a preset corpus length to obtain the adjusted text includes: adjusting the character length in the processed file to a target length based on a preset padding code instruction and a preset corpus length to obtain the adjusted text. That is, the input text length is standardized, such as using... <padding>The input is transformed into a predetermined number of M words by padding or splitting redundant characters. Using a word embedding model, each word is converted into a predetermined N-dimensional vector, and all word vectors are concatenated, thus transforming the input text into an M*N dimensional vector.
[0048] Step S13: Construct a target sensitive information detection model based on a preset target recognition model, and input the target text vector into the target sensitive information detection model to obtain the target recognition result.
[0049] In this embodiment, a target sensitive information detection model is constructed based on a preset target recognition model, and the target text vector is input into the target sensitive information detection model to obtain the target recognition result. Specifically, constructing the target sensitive information detection model based on the preset target recognition model includes modifying the Anchor and RPN (Region Proposal Network) in the Faster R-CNN (Region-CNN) model to obtain the target sensitive information detection model. The target sensitive information detection model, built upon the preset target recognition model, can vertically scan the input vector and classify the candidate boxes according to their sensitivity type. It should be noted that the sensitive information detection model uses a modified Faster R-CNN model, where the Anchor and RPN are modified to ensure that the width of all candidate boxes is N. Then, the target text vector is input into the target sensitive information detection model to obtain the target recognition result corresponding to the target text vector.
[0050] Step S14: Determine the sensitive information recognition result of the text to be detected based on the target recognition result.
[0051] In this embodiment, determining the sensitive information identification result of the text to be detected based on the target identification result includes: inputting the target identification result onto the text to be detected using a preset mapping method to obtain the sensitive information identification result of the text to be detected. That is, the sensitive information identified in the target text vector is mapped to the positions corresponding to the sensitive information in the text to be detected, thereby obtaining the corresponding text recognition result. The sensitive information includes, but is not limited to, mobile phone numbers, addresses, and ID card numbers.
[0052] It can be seen that, as Figure 2 As shown, firstly, the acquired text to be detected is preprocessed using a preset text processing method to obtain processed text; secondly, the processed text is transformed and concatenated based on a preset training corpus length and a preset word embedding model to obtain a target text vector; thirdly, a target sensitive information detection model is constructed based on a preset target recognition model, and the target text vector is input into the target sensitive information detection model to obtain a target recognition result; finally, the sensitive information recognition result of the text to be detected is determined based on the target recognition result. That is, by preprocessing and transforming the acquired text to be detected to obtain a target text vector with uniform length and dimensions, the target text vector is then recognized by the target sensitive information detection model constructed based on the preset target recognition model to obtain a target recognition result, thereby determining the sensitive information recognition result. This solves the drawback of traditional sensitive information detection algorithms that can only detect the entire input segment, and also eliminates the need to pre-define the detection categories, thus improving the flexibility of detection.
[0053] See Figure 3 As shown in the illustration, this application discloses a specific method for detecting sensitive text information. First, the input text is processed, including word segmentation and stop word removal, specifically using a bidirectional maximum matching algorithm to segment numbers, symbols, etc., one by one. Then, the length is standardized, such as by using... <padding>The input is converted into M words by padding or segmenting redundant characters. Then, a word embedding model is used to convert each word into an N-dimensional vector, and all word vectors are concatenated to convert the input text into an M*N dimensional vector; where M is the maximum length in the training corpus. The word embedding model uses a continuous bag-of-words (CBOW) model. Next, a sensitive information detection model is built based on the object recognition model. This model can vertically scan the input vector and classify candidate boxes by sensitivity type. The sensitive information detection model uses a modified Faster R-CNN model, with the Anchor and RPN modified to ensure that the width of all candidate boxes is N. Finally, the recognition results of the object recognition model are mapped onto the long input text to obtain the sensitive information detection results. This approach overcomes the limitation of traditional sensitive information detection algorithms that can only detect the entire input segment, and it eliminates the need to pre-define the detection categories, thus improving the flexibility of detection.
[0054] refer to Figure 4 The present application also discloses a text-sensitive information detection device, comprising:
[0055] The text preprocessing module 11 is used to preprocess the acquired text to be detected based on a preset text processing method to obtain the processed text.
[0056] Text conversion module 12 is used to convert and concatenate the processed file based on a preset training corpus length and a preset word embedding model to obtain a target text vector;
[0057] Model recognition module 13 is used to construct a target sensitive information detection model based on a preset target recognition model, and input the target text vector into the target sensitive information detection model to obtain the target recognition result;
[0058] The result determination module 14 is used to determine the sensitive information recognition result of the text to be detected based on the target recognition result.
[0059] As can be seen, in this embodiment, the acquired text to be detected is preprocessed, transformed, and concatenated to obtain a target text vector with uniform length and dimensions. Then, a target sensitive information detection model constructed based on a preset target recognition model is used to identify the target text vector to obtain the target recognition result and thus determine the sensitive information recognition result. In this way, the drawback of traditional sensitive information detection algorithms that can only detect the entire input segment is solved, and at the same time, there is no need to pre-set the detection type, thereby improving the flexibility of detection.
[0060] In some specific embodiments, the text preprocessing module 11 may specifically include:
[0061] The text segmentation unit is used to segment the obtained text to be detected based on the bidirectional maximum matching algorithm to obtain the segmented text.
[0062] A character adjustment unit is used to adjust the characters in the segmented text to obtain the processed text.
[0063] In some specific embodiments, the text conversion module 12 may specifically include:
[0064] The character length adjustment submodule is used to adjust the character length in the processed file to a target length based on a preset corpus length to obtain the adjusted text.
[0065] The vector transformation unit is used to convert the characters in the adjusted text into word vectors based on the continuous bag-of-words model, and to concatenate all the word vectors to obtain the target text vector.
[0066] In some specific embodiments, the character length adjustment submodule may specifically include:
[0067] The instruction execution unit is used to adjust the character length in the processed file to a target length based on a preset padding code instruction and a preset corpus length to obtain the adjusted text.
[0068] In some specific embodiments, the model recognition module 13 may specifically include:
[0069] The model modification unit is used to modify the Anchor and RPN in the Faster R-CNN model to obtain a target-sensitive information detection model.
[0070] In some specific embodiments, the result determination module 14 may specifically include:
[0071] The result mapping unit is used to input the target recognition result to the text to be detected through a preset mapping method to obtain the sensitive information recognition result of the text to be detected.
[0072] Furthermore, embodiments of this application also disclose an electronic device, Figure 5 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.
[0073] Figure 5 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the text-sensitive information detection method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.
[0074] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.
[0075] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.
[0076] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the text-sensitive information detection method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.
[0077] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned method for detecting sensitive text information. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.
[0078] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.
[0079] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0080] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.
[0081] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0082] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.< / padding> < / padding>
Claims
1. A method for detecting sensitive information in text, characterized in that, include: The obtained text to be detected is preprocessed based on a preset text processing method to obtain the processed text. The processed file is transformed and concatenated based on a preset training corpus length and a preset word embedding model to obtain a target text vector; wherein, the transformation and concatenation of the processed file based on the preset training corpus length and the preset word embedding model to obtain a target text vector includes: adjusting the character length in the processed file to a target length based on the preset corpus length to obtain adjusted text; converting the characters in the adjusted text into word vectors based on a continuous bag-of-words (CBOW) model, and concatenating all the word vectors to obtain the target text vector; wherein, the preset word embedding model adopts the continuous bag-of-words (CBOW) model, converting each word into an N-dimensional word vector, and concatenating all the word vectors to generate an M×N two-dimensional text vector matrix; wherein, adjusting the character length in the processed file to a target length based on the preset corpus length to obtain the adjusted text includes: adjusting the character length in the processed file to the target length based on preset padding code instructions and a preset corpus length to obtain the adjusted text; A target-sensitive information detection model is constructed based on a preset target recognition model, and the target text vector is input into the target-sensitive information detection model to obtain the target recognition result; wherein, the construction of the target-sensitive information detection model based on the preset target recognition model includes: modifying the Anchor and RPN in the Faster R-CNN model so that the width of all candidate boxes is fixed to the dimension N of the word vector, so as to obtain the target-sensitive information detection model; the target-sensitive information detection model performs vertical scanning of the target text vector and classifies the candidate boxes according to their sensitivity type to obtain the target recognition result; Based on the target recognition results, the sensitive information recognition results of the text to be detected are determined.
2. The text-sensitive information detection method according to claim 1, characterized in that, The text preprocessing of the acquired text to be detected based on a preset text processing method to obtain processed text includes: The obtained text to be detected is segmented into words based on the bidirectional maximum matching algorithm to obtain the segmented text. The characters in the segmented text are adjusted to obtain the processed text.
3. The text-sensitive information detection method according to any one of claims 1 to 2, characterized in that, The step of determining the sensitive information identification result of the text to be detected based on the target identification result includes: The target recognition result is input to the text to be detected through a preset mapping method to obtain the sensitive information recognition result of the text to be detected.
4. A text-sensitive information detection device, characterized in that, include: The text preprocessing module is used to preprocess the acquired text to be detected based on a preset text processing method to obtain the processed text. A text conversion module is used to convert and concatenate the processed file based on a preset training corpus length and a preset word embedding model to obtain a target text vector. The conversion and concatenation of the processed file based on the preset training corpus length and the preset word embedding model to obtain the target text vector includes: adjusting the character length in the processed file to a target length based on the preset corpus length to obtain adjusted text; converting the characters in the adjusted text into word vectors based on a continuous bag-of-words (CBOW) model, and concatenating all the word vectors to obtain the target text vector; wherein the preset word embedding model uses a continuous bag-of-words (CBOW) model, converting each word into an N-dimensional word vector, and concatenating all the word vectors to generate an M×N two-dimensional text vector matrix; wherein adjusting the character length in the processed file to a target length based on the preset corpus length to obtain the adjusted text includes: adjusting the character length in the processed file to the target length based on preset padding code instructions and a preset corpus length to obtain the adjusted text. The model recognition module is used to construct a target sensitive information detection model based on a preset target recognition model, and input the target text vector into the target sensitive information detection model to obtain the target recognition result; wherein, constructing the target sensitive information detection model based on the preset target recognition model includes: modifying the Anchor and RPN in the Faster R-CNN model so that the width of all candidate boxes is fixed to the dimension N of the word vector, so as to obtain the target sensitive information detection model; the target sensitive information detection model performs vertical scanning of the target text vector and classifies the candidate boxes according to their sensitivity type to obtain the target recognition result; The result determination module is used to determine the sensitive information recognition result of the text to be detected based on the target recognition result.
5. The text-sensitive information detection device according to claim 4, characterized in that, The text preprocessing module includes: The text segmentation unit is used to segment the obtained text to be detected based on the bidirectional maximum matching algorithm to obtain the segmented text. A character adjustment unit is used to adjust the characters in the segmented text to obtain the processed text.
6. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the text sensitive information detection method as described in any one of claims 1 to 3.
7. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the text sensitive information detection method as described in any one of claims 1 to 3.