Method and device for desensitizing sensitive data and storage medium

Abstract

The invention discloses a method and device for desensitizing sensitive data and a storage medium. The method for desensitizing the sensitive data comprises the steps of obtaining target data, and determining a non-long text field and a long text field according to the length of a text field in the target data; performing desensitization processing on non-long text field data in the target data; performing preliminary desensitization processing on the long text field data in the target data to obtain a preliminary desensitization processing result; performing clustering embedding desensitization processing on the long text field data in the target data according to the preliminary desensitization processing result to obtain a clustering embedding desensitization processing result; and performing normal enhanced desensitization processing on the long text field data in the target data according to a clustering embedding desensitization processing result to obtain a desensitization data result of the long text field data. According to the scheme provided by the invention, relatively high data security and privacy protection level can be obtained.

Description

Technical Field

[0001] The present invention relates to the application field of natural language processing technology, and in particular to a method, device and storage medium for desensitizing sensitive data. Background Art

[0002] The application of natural language processing technology involves the parsing, analysis and processing of sensitive data, but how to accurately and effectively desensitize sensitive data has become a problem that needs to be solved urgently.

[0003] In related technologies, due to the constraints of existing algorithms, only preliminary desensitization of data can be performed, and the desensitization effect is not good, and there is room for further improvement. Summary of the invention

[0004] In order to solve the above technical problems, the embodiments of the present invention hope to provide a method, device and storage medium for desensitizing sensitive data, so as to at least solve the problem of poor data desensitization effect in related technologies.

[0005] The technical solution of the present invention is achieved in this way:

[0006] An embodiment of the present invention provides a method for desensitizing sensitive data, including:

[0007] Acquire target data, and determine a non-long text field and a long text field according to the length of a text field in the target data;

[0008] Performing desensitization processing on the non-long text field data;

[0009] Performing preliminary desensitization processing on the long text field data to obtain a preliminary desensitization processing result;

[0010] Performing clustering embedding desensitization processing on the long text field data according to the preliminary desensitization processing result to obtain a clustering embedding desensitization processing result;

[0011] According to the clustering embedding desensitization processing result, the long text field data is subjected to normal enhanced desensitization processing to obtain a desensitized data result of the long text field data.

[0012] Optionally, the acquiring target data includes:

[0013] Acquire collected data from the target data source through pre-authorization to form a target data set;

[0014] storing the collected data in an initial database;

[0015] The target data is acquired from the initial database.

[0016] Optionally, desensitizing the non-long text field data in the target data includes: desensitizing consecutive numbers in the non-long text field data in the target data to obtain a first desensitizing result; desensitizing non-numeric characters in the non-long text field data in the target data to obtain a second desensitizing result; and obtaining a desensitizing result for the non-long text field data based on the first desensitizing result and the second desensitizing result.

[0017] Further, optionally, desensitizing continuous numbers in non-long text field data in the target data to obtain a first desensitizing result includes: screening a non-long text field set from the target data; obtaining a first vocabulary corresponding to the pre-trained language model; wherein each row in the first vocabulary contains only one token character content, and the token character content includes: Chinese characters, English characters and special characters; wherein the meaning of the token character content is: when the token character in the first vocabulary is a Chinese character, the token character content is a single word; when the token character content in the first vocabulary is an English character, the token character content is a single complete English word; when the token character content in the first vocabulary is a special character, the token character content is a single complete special character; traversing each row of the first vocabulary, storing each row of the content in a key-value pair database in the form of a key-value pair to obtain a first table; wherein the key-value pair form includes: the row number of each row of the content is the key, and the token character content is the value; traversing each field under the non-long text field set, and desensitizing continuous numbers in the non-long text field data in combination with the first table.

[0018] Optionally, traversing each field under the non-long text field set, and desensitizing the continuous numbers in the non-long text field data in combination with the first table includes: adding the values ​​at each position in the continuous numbers to obtain a first value; dividing the first value by the total number of rows in the first vocabulary and taking the remainder to obtain a second value; obtaining the token character content with the key as the second value in the first table; passing the token character content through a word vector conversion model to obtain a first target vector; dividing the overall value of the continuous number by the total number of rows in the first vocabulary and taking the remainder to obtain a third value; obtaining the token character content with the key as the third value in the first table; passing the token character content through a word vector conversion model to obtain a second target vector; by traversing the first target vector and the second target vector, adding the element values ​​at the same index position and then dividing them by the position index and taking the remainder, to obtain the remainder value of each index position in the first target vector and the second target vector; replacing the continuous numbers with the remainder value at each index position by concatenation to obtain the desensitization result of the continuous numbers.

[0019] Optionally, desensitizing non-numeric characters in non-long text field data in the target data to obtain a second desensitizing result includes: screening a set of non-long text fields from the target data; obtaining a special character table and a second vocabulary corresponding to a pre-trained language model; wherein each row in the special character table and the second vocabulary contains only one token character content, and the special character table is used to supplement the special characters in the second vocabulary; the token character content is a single complete special character in the special character table; traversing each row of the special character table and the second vocabulary in a preset order, extracting special characters and English characters in each row of content, and storing them in a key-value pair database in the form of key-value pairs to obtain a second table; and extracting Chinese characters in each row of content, and storing them in the key-value pair database in the form of key-value pairs to obtain a third table; wherein the character content of each row of content is the key, and the row number of each row of content is the value; traversing each field under the non-long text field set, and desensitizing non-numeric characters in the non-long text field data in combination with the second table and the third table.

[0020] Further, optionally, traversing each field under the non-long text field set, and desensitizing the non-numeric characters in the non-long text field data in combination with the second table and the third table include: extracting the non-numeric characters in each field to obtain a candidate set of non-numeric characters to be processed; traversing each element in the candidate set of non-numeric characters to be processed, recording it as a target element; constructing an initialization probability set based on the target element; wherein, constructing the initialization probability set based on the target element includes: if the target element is a key in the second table, then finding the value corresponding to the key in the second table through the second table mapping, putting the value into the initialization probability set, and taking the value as the numerator and the length of the third table as the denominator, dividing and taking the remainder, obtaining the remainder value, and putting the remainder value into the initialization probability set. rate set; in the third table, with the remainder value as the value, the key corresponding to the value in the third table is obtained through the third table mapping, and the value is put into the initialization probability set; if the target element is the key in the third table, the value corresponding to the key in the third table is found through the third table mapping, and the value is put into the initialization probability set, and the value is used as the numerator and the length of the second table as the denominator, and the remainder is obtained by division to obtain the remainder value, and the remainder value is put into the initialization probability set. In the second table, with the remainder value as the value, the key corresponding to the value in the second table is obtained through the second table mapping, and the value is put into the initialization probability set; in the initialization probability set, an element is randomly selected with equal probability to replace the target element, so that non-numeric characters are desensitized.

[0021] Optionally, a preliminary desensitization process is performed on the long text field data in the target data to obtain a preliminary desensitization process result, including: screening a long text field set from the target data; using a preset sensitive word filtering model to filter the target sensitive words contained in the long text in the long text field set, and processing the target sensitive words through a preset strategy to obtain a filtered text; wherein the preset strategy includes: a one-third probability of keeping unchanged, a one-third probability of replacing the target sensitive words with preset symbols, and / or a one-third probability of replacing with any English letters; translating the filtered text into a preset type of foreign language content, extracting summary information from the filtered text and each type of foreign language content obtained by translation through a summary information extraction model to obtain summary information content; vectorizing each summary information content according to the sentence vector model to obtain a vector of a preset length, wherein all vectors have the same length and each element value in the vector is a non-zero integer; adding the element values ​​at the same position in all vectors, averaging them in integer form to obtain a comprehensive average vector, and concatenating the values ​​of the elements at each position in the comprehensive average vector with a first preset symbol to obtain a preliminary desensitization process result.

[0022] Further, optionally, clustering and embedding desensitizing processing is performed on the long text field data in the target data based on the preliminary desensitizing processing result, and the clustering and embedding desensitizing processing result includes: for all fields in the target data belonging to the long text field set, clustering prediction is performed using a clustering model to obtain clustering categories; the clustering categories are connected in series to the end of the preliminary desensitizing processing result through a second preset symbol to obtain a clustering and desensitizing processing result; through the constructed number and letter mapping dictionary, the element value of each position in the comprehensive average vector in the preliminary desensitizing processing result is traversed, the element value is the numerator, the total number of letters in the mapping dictionary is the denominator, the remainder value is taken after division, and the corresponding letter value is obtained through the mapping dictionary with the remainder value as the key, and the symbol in the clustering and desensitizing processing result is replaced by the letter value in turn to obtain the clustering and embedding desensitizing processing result.

[0023] Optionally, performing normal enhanced desensitization processing on the long text field data in the target data based on the clustered embedded desensitization processing result includes: obtaining the number of tokens of the long text field data in the target data; inputting the number of tokens into the normal distribution probability function to obtain the corresponding probability value; inputting the probability value into the sigmoid activation function to obtain the sigmoid value; retaining all data after the decimal point in the sigmoid value, and stringifying all data after the decimal point in the sigmoid value to obtain a stringified result; concatenating the stringified result to the end of the clustered embedded desensitization processing result to obtain the desensitized data result of the long text field data.

[0024] An embodiment of the present invention provides a device for desensitizing sensitive data, including: an acquisition module, used to acquire target data, and determine a non-long text field and a long text field according to the length of the text field in the target data; a first desensitization module, used to perform desensitization processing on the non-long text field data in the target data; a second desensitization module, used to perform preliminary desensitization processing on the long text field data in the target data, and obtain a preliminary desensitization processing result; a third desensitization module, used to perform clustering and embedding desensitization processing on the long text field data in the target data according to the preliminary desensitization processing result, and obtain a clustering and embedding desensitization processing result; a fourth desensitization module, used to perform normal enhanced desensitization processing on the long text field data in the target data according to the clustering and embedding desensitization processing result, and obtain a desensitized data result of the long text field data.

[0025] An embodiment of the present invention provides a storage medium, which includes a stored program, and when the program is running, controls a device where the storage medium is located to execute: the above-mentioned method for desensitizing sensitive data.

[0026] The embodiments of the present invention provide a method, device and storage medium for desensitizing sensitive data, which comprises acquiring target data; desensitizing non-long text field data in the target data; performing preliminary desensitization on long text field data in the target data to obtain preliminary desensitization results; performing clustering embedded desensitization on long text field data in the target data based on the preliminary desensitization results to obtain clustering embedded desensitization results; performing positive enhanced desensitization on long text field data in the target data based on the clustering embedded desensitization results to obtain desensitized data results of the long text field data, thereby being able to obtain a technical effect of having a higher level of data security and privacy protection, and providing a more economical, reliable and secure data processing solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0028] Figure 1 A schematic diagram of a process for desensitizing sensitive data provided by an embodiment of the present invention;

[0029] Figure 2 A schematic diagram of a flow chart of another method for desensitizing sensitive data provided by an embodiment of the present invention;

[0030] Figure 3 A schematic diagram of a device for desensitizing sensitive data provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first", "second" and the like in the specification, claims and drawings of the present invention are used to distinguish different objects rather than to limit a specific order.

[0033] It should also be noted that the following embodiments of the present invention may be executed separately, or in combination with each other, and the embodiments of the present invention do not impose any specific limitations on this.

[0034] The embodiment of the present invention provides a method for desensitizing sensitive data. Figure 1 A flowchart of a method for desensitizing sensitive data provided by an embodiment of the present invention; Figure 1 As shown, the method for desensitizing sensitive data provided in the embodiment of the present application includes:

[0035] Step S102, acquiring target data, and determining a non-long text field and a long text field according to the length of a text field in the target data;

[0036] Optionally, obtaining the target data in step S102 includes: obtaining collected data from a target data source in a pre-authorized manner to form a target data set; storing the target data set in an initial database; and obtaining the target data from the initial database.

[0037] Specifically, Figure 2 A flowchart of another method for desensitizing sensitive data provided by an embodiment of the present invention; Figure 2 As shown, in the embodiment of the present application, the pre-authorization method includes: at least one of pre-registration, pre-authorization and / or through legal channels; downloading social event data from a professional database (i.e., the target data source in the embodiment of the present application) based on the pre-authorization method, and obtaining the social event data disclosed by the website (i.e., the target data source in the embodiment of the present application) through legal channels based on web crawler technology, and forming a target data set by structurally integrating and splicing various types of social event data;

[0038] In the embodiment of the present application, each piece of data spliced ​​to form the target data set includes the following fields, specifically: "id" field (unique identifier of the social event), "type_id" field (unique identifier of the category to which the social event belongs), "occurrence time" field (indicating the time when the social event occurs), "social event processing time", "trial time of social event-related cases", "related subject 1_name" field (records that there are two related subjects in the occurrence of the social event, namely related subject 1 and related subject 2, and related subject 1_name represents the corresponding name of the subject. For example, when the content of the social event is "Zhang XX went to the First People's Hospital of a certain place for treatment", the event subject 1_name is Zhang XX), "related subject 1_id" field (records the social event There are two related subjects when an event occurs, namely related subject 1 and related subject 2. Related subject 1_id indicates the id corresponding to the subject, that is, the unique identifier. When the subject is an individual, the id is the ID number. When the subject is an institution, the id is the unified social credit code. For example, when the content of a social event is "Zhang XX and Shi XX reached a business cooperation", the event subject 1_id is the id corresponding to Zhang XX, that is, the unique identifier, which is the ID number in this case), "Related subject 1 mobile phone number" field, "Related subject 1 email" field, "Related subject 1 address" field, "Related subject 2_name" field (record that there are two related subjects when a social event occurs, namely related subject 1 and related subject 2, and related subject 2_name indicates the name corresponding to the subject.

[0039] For example, when the content of the event is "Zhang XX went to the First People's Hospital of a certain place for treatment", the event subject 2_name is the First People's Hospital of a certain place), "related subject 2_id" field (records that there are two related subjects in the event, namely related subject 1 and related subject 2. Related subject 2_id represents the id corresponding to the subject, that is, the unique identifier. When the subject is an individual, the id is the ID number. When the subject is an institution, the id is the unified social credit code.

[0040] Construct an initial database and store the target data set in the initial database.

[0041] Step S104, performing desensitization processing on the non-long text field data in the target data.

[0042] Among them, in the embodiment of the present application, the non-long text field data is content data with a length less than or equal to 30;

[0043] Optionally, desensitizing the non-long text field data in the target data in step S104 includes: desensitizing continuous numbers in the non-long text field data in the target data to obtain a first desensitizing result; desensitizing non-numeric characters in the non-long text field data in the target data to obtain a second desensitizing result; and obtaining a desensitizing result of the non-long text field data based on the first desensitizing result and the second desensitizing result.

[0044] Specifically, Figure 2 As shown, in the embodiment of the present application, the desensitizing processing of the non-long text field data in the target data includes two parts: desensitizing the continuous numbers in the non-long text field data (that is, the first part), and desensitizing the non-numeric characters in the non-long text field data (that is, the second part).

[0045] The first part is to desensitize the continuous numbers in the non-long text field data as follows:

[0046] A set of non-long text fields is screened from target data; a first vocabulary corresponding to a pre-trained language model is obtained; wherein each row in the first vocabulary contains only one token character content, and the token character content includes: Chinese characters, English characters and special characters; wherein the meaning of the token character content is: when the token character content in the first vocabulary is Chinese characters, the token character content is a single word; when the token character content in the first vocabulary is English characters, the token character content is a single complete English word; when the token character content in the first vocabulary is a special character, the token character content is a single complete special character; traverse each row of the first vocabulary, and store each row of the content in a key-value pair database in the form of a key-value pair to obtain a first table; wherein the key-value pair form includes: the row number of each row of the content is the key, and the token character content is the value; traverse each field under the non-long text field set, and perform desensitization processing on continuous numbers in the non-long text field data in combination with the first table.

[0047] Specifically, Figure 2 As shown, the non-long text field set filtered from the target data includes: assuming that it is judged to include the "id" field, the "type_id" field, the "event occurrence time" field, the "related subject 1_name" field, the "related subject 1_id" field, the "related subject 1 mobile phone number" field, the "related subject 1 email" field, the "related subject 1 address" field, the "related subject 2_name" field, the "related subject 2_id" field, the "related subject 2 mobile phone number" field, the "related subject 2 email" field, and the "related subject 2 address" field;

[0048] Obtaining the first vocabulary corresponding to the pre-trained language model includes: downloading the corresponding vocabulary file vocab.txt (i.e., the first vocabulary in the embodiments of the present application) under the pre-trained language models bert, roberta, or easybert (i.e., the pre-trained language model in the embodiments of the present application). Each line in vocab.txt contains only one token character content, including Chinese characters, English characters, and special characters. Among them, the meaning of the token character content is that when the token character content in vocab.txt is a Chinese character, the character is a single Chinese character with a length of 1; when the token character content is an English character, the character is a single complete English word; when the token character content is a special character, the character is a single complete special character.

[0049] In the embodiments of the present application, traverse each line content of the first vocabulary, and store each line content in the redis database in the form of key-value pairs to obtain the first table, including: traverse each line content of vocab.txt, and store each content in the redis database in the form of key-value pairs. The formed table is denoted as rb1 (i.e., the first table in the embodiments of the present application). Among them, the line number where each line content is located is the key, and the token character content is the value. For example, the line number of the first line in vocab.txt is 1, and the token character content is "睨", then the content of this line is stored in redis in the form of {1: "睨"}, and the formed table is denoted as rb1. Using the table rb1 formed by the redis database can improve the efficiency of the conversion mapping operation in the digital desensitization process.

[0050] It should be noted that the redis database in the embodiments of the present application is a database essentially in the form of "(key-value) pairs". Because the "(key-value) pair" database has a mapping attribute, it can greatly improve the query efficiency. Therefore, constructing the "redis database" based on the relationship of some data in the "initial database" is to utilize the characteristics of the "redis database" to improve the query efficiency. Other key-value pair databases of the same type such as "DynamoDB database", "Memcached database", and "etcd database" can also be used.

[0051] Further, optionally, traversing each field under the non-long text field set and desensitizing the continuous numbers in the non-long text field data in combination with the first table include: adding the values ​​at each position in the continuous numbers to obtain a first value; dividing the first value by the total number of rows in the first vocabulary and taking the remainder to obtain a second value; obtaining the token character content with the key being the second value in the first table; passing the token character content through a word vector conversion model to obtain a first target vector; dividing the overall value of the continuous number by the total number of rows in the first vocabulary and taking the remainder to obtain a third value; obtaining the token character content with the key being the third value in the first table; passing the token character content through a word vector conversion model to obtain a second target vector; traversing the first target vector and the second target vector, adding the element values ​​at the same index position and then dividing them by the position index and taking the remainder to obtain the remainder value for each index position in the first target vector and the second target vector; replacing the continuous numbers with the remainder value at each index position by concatenation to obtain the desensitization result of the continuous numbers.

[0052] Specifically, Figure 2 As shown, traversing each field under the non-long text field set and desensitizing the continuous numbers in the non-long text field data in combination with the first table includes: traversing each field under the non-long text field set, wherein each field may be composed of one or more combinations of numbers, Chinese characters, English characters, and special characters. For the continuous numbers, desensitization conversion is performed in three stages, specifically:

[0053] Assume that the content of the "Related Subject 1 Email" field of an event in the target database is "xdl417810@xxx.com". In the first stage, that is, add the values at each position in this continuous number to obtain a value, divide this value (i.e., the first numerical value in the embodiments of the present application) by the total number of lines in vocab.txt and take the remainder to obtain a value denoted as tp1 (i.e., the second numerical value in the embodiments of the present application). In table rb1, map to obtain the value corresponding to the key tp1, that is, the token character content, and perform token character-level vector conversion on this token character content through word vector conversion models such as bert, roberta, easybert (i.e., the word vector conversion model in the embodiments of the present application), etc., to obtain a vector (i.e., the first target vector in the embodiments of the present application), where the element values at each position in this vector are all positive integers, and the vector length ranges from 8 to 128. For example: when the field content is "xdl417810@xxx.com", the continuous number obtained is 417810. Add the values at each position of this number, that is, obtain 20 by 4 + 1 + 7 + 8 + 1 + 0. Divide 20 by the total number of lines in vocab.txt and take the remainder, and the obtained value is denoted as tp1. Assume that tp1 is also 20. Map in table rb1 to obtain the value corresponding to the key tp1, that is, the token character content. Assume it is "乐". Pass the character "乐" through the word vector conversion model roberta to obtain a vector. Assume this vector is [5, 7, 11, 3, 98, 22, 35, 7], denoted as e_a_1 (i.e., the first target vector in the embodiments of the present application);

[0054] Among them, the vector directly obtained through conversion by word vector conversion models such as bert, roberta, easybert, etc. (i.e., the vector to be processed described below), where the numerical values of the elements at each position are not necessarily all positive integers. Therefore, the following method can be optionally adopted to process the vector directly obtained through conversion to ensure that the elements at each position in the finally obtained vector (i.e., the first target vector) are all positive integers.

[0055] First, construct a list ls, ls = [a, b, c, d], where a, b, c, d are all arbitrary positive integers and a, b, c, d are all not equal. The elements in the vector to be processed may include five types: positive integers, negative integers, positive decimals, negative decimals, and 0. For further desensitization, establish rules for mapping and index fusion to process these five types of elements:

[0056] Case 1: When an element is a positive integer, add the value of the element to the index value corresponding to the element in the vector, and replace the element with the calculated value (case 1 adds the element itself and the index value, which is different from the following case, retains part of the information of the vector to be processed itself, and enhances the distinction between different vectors);

[0057] Case 2: When the element is a negative integer, take a from the list ls, add a to the index value corresponding to the element in the vector to be processed, and replace the element with the calculated value;

[0058] Case 3: When the element is a positive decimal, take b from the list ls, add b to the index value corresponding to the element in the vector to be processed, and replace the element with the calculated value;

[0059] Case 4: When the element is a negative decimal, take c from the list ls, add c to the index value corresponding to the element in the vector to be processed, and replace the element with the calculated value;

[0060] Case 5: When the element is 0, take d from the list ls, add d to the corresponding index value of the element in the vector to be processed, and replace the element with the calculated value. After the traversal is completed, all elements in the vector are positive integers.

[0061] Here is a detailed example: Let's assume that a=1, b=2, c=3, d=4, so the list ls=[1,2,3,4]. Assume that the current vector cv=[0,5,-3,0.8,-0.2], since the element at index 0 is 0, according to the above case 5, it is easy to know that d is taken from the list ls, d=4, d is added to index 0, 4+0=4, and 4 is used to replace the element at cv index 0; since the element at index 1 is 5, according to the above case 1, it is easy to know that the element 5 is added to index 1, 5+1=6, and 6 is used to replace the element at cv index 1; since the element at index 2 is -3, according to the above case 2, it is easy to know that from the list l Take a from s, a=1, add a to index 2, 1+2=3, use 3 to replace the element at index 2 of cv; since the element at index 3 is 0.8, according to the above situation 3, it is easy to know that b is taken from list ls, b=2, b is added to index 3, 2+3=5, and 5 is used to replace the element at index 3 of cv; since the element at index 4 is -0.2, according to the above situation 4, it is easy to know that c is taken from list ls, c=3, c is added to index 4, 3+4=7, and 7 is used to replace the element at index 4 of cv. After traversing all elements, all elements in vector cv are positive integers.

[0062] Of course, other processing methods may be used to ensure that each bit of data in the final vector is a positive integer.

[0063] In the second stage, the continuous digits are regarded as a whole, and the whole value is divided by the total number of lines in vocab.txt and the remainder is taken to obtain a value denoted as tp2 (i.e., the third numerical value in the embodiments of the present application). The value corresponding to the key tp2 is obtained through mapping in the table rb1, that is, the token character content. Then, the token character content is subjected to character-level vector conversion through a word vector conversion model, such as bert, roberta, easybert, etc., to obtain a vector (i.e., the second target vector in the embodiments of the present application). Each element value at each position in the vector is a positive integer (where the specific method for positive integer conversion processing is not limited. For example, the method provided in the above example can be used to process the vector directly obtained after word vector conversion to ensure that each element value at each position in the finally obtained vector is a positive integer). The vector length ranges from 8 to 128. For example: Suppose the continuous digits are 417810, then tp2 is 417810. The value corresponding to the key tp2 is obtained through mapping in the table rb1, that is, the token character content. Suppose it is "景". The character "景" is passed through the word vector conversion model roberta to obtain a vector. Suppose the vector is [24, 4, 11, 7, 96, 21, 35, 12], denoted as e_a_2 (i.e., the second target vector in the embodiments of the present application); among them, the vector lengths of the vector e_a_1 and the vector e_a_2 obtained through the word vector conversion model roberta are the same.

[0064] In the third stage, the vectors e_a_1 and e_a_2 are traversed, and the element values at the same index position are added and then divided by the position index and the remainder is taken to obtain the remainder value at each index position respectively. The remainder values are concatenated to replace the continuous digits, and thus the desensitization of the continuous digits is completed. For example: Suppose the vector e_a_1 is [5, 7, 11, 3, 98, 22, 35, 7], and the vector e_a_2 is [24, 4, 11, 7, 96, 21, 35, 12]. The values at the index position 1 in the vectors e_a_1 and e_a_2 are 5 and 24 respectively. Adding them gives 5 + 24 = 29, and dividing by the index position 1 gives a remainder of 29; the values at the index position 2 in the vectors e_a_1 and e_a_2 are 7 and 4 respectively. Adding them gives 7 + 4 = 11, and dividing by the index position 2 gives a remainder of 1, and so on, until the values at the index position 8 in the vectors e_a_1 and e_a_2 are 7 and 12 respectively. Adding them gives 7 + 12 = 19, and dividing by the index position 8 gives a remainder of 3. At this time, the desensitization of the continuous digits is completed.

[0065] The second part: The desensitization processing of non-numeric characters in non-long text field data is specifically as follows:

[0066] Optionally, desensitizing non-numeric characters in non-long text field data in the target data to obtain a second desensitizing result includes: screening a set of non-long text fields from the target data; obtaining a special character table and a second vocabulary corresponding to a pre-trained language model; wherein each row in the special character table and the second vocabulary contains only one token character content, and the special character table is used to supplement the special characters in the second vocabulary; the token character content is a single complete special character in the special character table; traversing each row of the special character table and the second vocabulary in a preset order, extracting special characters and English characters in each row of content, and storing them in a key-value pair database in the form of key-value pairs to obtain a second table; and extracting Chinese characters in each row of content, and storing them in the key-value pair database in the form of key-value pairs to obtain a third table; wherein the character content of each row of content is the key, and the row number of each row of content is the value; traversing each field under the non-long text field set, and desensitizing non-numeric characters in the non-long text field data in combination with the second table and the third table.

[0067] Specifically, in the embodiment of the present application, non-numeric characters include: Chinese characters, English characters and special characters;

[0068] Among them, Figure 2 As shown, it is assumed that the non-long text field set to be filtered includes the "id" field, the "type_id" field, the "event occurrence time" field, the "related subject 1_name" field, the "related subject 1_id" field, the "related subject 1 mobile phone number" field, the "related subject 1 email" field, the "related subject 1 address" field, the "related subject 2_name" field, the "related subject 2_id" field, the "related subject 2 mobile phone number" field, the "related subject 2 email" field, and the "related subject 2 address" field;

[0069] Download the special character table special_token.txt (i.e., the special character table in the embodiment of the present application) and the corresponding vocabulary vocab.txt (i.e., the second vocabulary in the embodiment of the present application) under the pre-trained language model BERT, Roberta or EasyBERT (i.e., the pre-trained language model in the embodiment of the present application). Each line in special_token.txt and vocab.txt contains only one token character content, where special_token.txt includes a large number of special characters, which can supplement the number of special characters in vocab.txt. vocab.txt includes Chinese characters, English characters and special characters.

[0070] Among them, the meaning of the token character content is as follows: when the token is a single complete special character in special_token.txt, and when the token character content is a Chinese character in vocab.txt, the character is a single Chinese character with a length of 1; when the token character content is an English character, the character is a single complete English word; when the token character content is a special character, the character is a single complete special character;

[0071] Traverse each line of the content of special_token.txt and vocab.txt in sequence, extract the special characters and English characters in each line, and store them in the redis database (i.e., the key-value pair database in the embodiments of the present application) in the form of key-value pairs, and converge to form a table, denoted as rp1 (i.e., the second table in the embodiments of the present application); then extract the Chinese characters in each line, store them in the redis database in the form of key-value pairs, and converge to form a table, denoted as rp2 (i.e., the third table in the embodiments of the present application). Among them, the character content of each line is the key, and the line number where the line is located is the value. For example: the line number of the first line in special_token.txt is 1, and the token character is "@", then the content of this line is stored in redis in the form of {"@": 1}, and used as a piece of data in the table denoted as rp1. The line number of the 24th line in vocab.txt is 24, and the token character content is "f", then the content of this line is stored in redis in the form of {"f": 24}, and used as a piece of data in the table denoted as rp1. The line number of the first line in vocab.txt is 1, and the token character content is "睨", then the content of this line is stored in redis in the form of {"睨": 1}, and used as a piece of data in the table rp2. The tables rp1 and rp2 formed using the redis database can improve the efficiency of the conversion mapping operation during the token character content desensitization process.

[0072] It should be noted that in the embodiments of the present application, the first vocabulary and the second vocabulary can be the same vocabulary, and the vocabulary vocab.txt can be obtained only once during the process of obtaining it, subject to the method for desensitizing sensitive data provided by the embodiments of the present application, and no specific limitation is made.

[0073] Further, optionally, traversing each field under the non-long text field set, and desensitizing the non-numeric characters in the non-long text field data in combination with the second table and the third table include: extracting the non-numeric characters in each field to obtain a candidate set of non-numeric characters to be processed; traversing each element in the candidate set of non-numeric characters to be processed, recording it as a target element; constructing an initialization probability set based on the target element; wherein, constructing the initialization probability set based on the target element includes: if the target element is a key in the second table, then finding the value corresponding to the key in the second table through the second table mapping, putting the value into the initialization probability set, and taking the value as the numerator and the length of the third table as the denominator, dividing and taking the remainder, obtaining the remainder value, and putting the remainder value into the initialization probability set. rate set; in the third table, with the remainder value as the value, the key corresponding to the value in the third table is obtained through the third table mapping, and the value is put into the initialization probability set; if the target element is the key in the third table, the value corresponding to the key in the third table is found through the third table mapping, and the value is put into the initialization probability set, and the value is used as the numerator and the length of the second table as the denominator, and the remainder is obtained by division to obtain the remainder value, and the remainder value is put into the initialization probability set. In the second table, with the remainder value as the value, the key corresponding to the value in the second table is obtained through the second table mapping, and the value is put into the initialization probability set; in the initialization probability set, an element is randomly selected with equal probability to replace the target element, so that non-numeric characters are desensitized.

[0074] Among them, Figure 2 As shown, each field in the non-long text field set is traversed, where the field may be a combination of numbers, Chinese characters, English characters, and special characters. For the Chinese characters, English characters, and special characters, desensitization conversion is performed in two stages, specifically:

[0075] Assume that the content of the "Related Subject 1 Email" field of an event in the target database is "xdl417810@qq.com". In stage one, extract the non-numeric characters in the field content to obtain a candidate set of non-numeric characters to be processed, traverse each element in the candidate set of non-numeric characters to be processed, record it as the target element, construct an initialization probability set for the target element, record it as pro_set={}, and make the following judgment: If the target element is a key in rp1 (i.e., the second table in the embodiment of the present application), find the corresponding value in rp1 through mapping, put the corresponding value into pro_set, and use the value as the numerator, rp2 (i.e., the third table in the embodiment of the present application) The length is used as the denominator, and a remainder value is obtained by division. The remainder value is put into pro_set (i.e., the initialization probability set in the embodiment of the present application). In rp2, the remainder value is used as the value to map the key corresponding to the value in rp2, and the value is put into pro_set; if the target element is a key in rp2, the corresponding value in rp2 of the key is found through mapping, and the corresponding value is put into pro_set, and the value is used as the numerator and the length of rp1 is used as the denominator. A remainder value is obtained by division, and the remainder value is put into pro_set. In rp1, the remainder value is used as the value to map the key corresponding to the value in rp1, and the value is put into pro_set.

[0076] In the second stage, an element is randomly selected from pro_set with equal probability to replace the target element. Similarly, after traversing the candidate set of non-numeric characters to be processed, the desensitization of non-numeric characters can be completed.

[0077] Example 1: Assuming the field content is "xdl417810@qq.com", the candidate set of non-numeric characters to be processed is "xdl@qq.com", denoted as hou_ls. Assuming that when traversing to the target element "x", "x" is obtained through mapping as the key in table rp1, and the corresponding value in rp1 is 630124, the length of rp2 is len(rp2), assuming that 630123 is the numerator and len(rp2) is the denominator to take the remainder, we get The remainder value is 32, and the key is "match" in table rp2 through mapping with 32 as the value. 630124, "32" and "match" are stored in the initialization probability set pro_set, and pro_set = {630124, 32,"match"} is obtained; each element in pro_set = {630124, 32,"match"} has a 1 / 3 probability of being hit to replace the original "x". Assuming that "match" is hit, "match" replaces "x". By analogy, after traversing "xdl417810@qq.com", the desensitization of non-numeric characters can be completed.

[0078] Example 2: Assuming that the field content is "No. 46, Ha Mou Road, Wu Mou City", the candidate set of non-numeric characters to be processed is "No. Ha Mou Road, Wu Mou City", denoted as hou_ls. Assuming that when traversing to the target element "Wu", "Wu" is obtained as the key in table rp2 through mapping, and the corresponding value in rp2 is 3201. The length of rp1 is len(rp1). Assuming that 3201 is used as the numerator and len(rp1) is used as the denominator to take the remainder, the remainder value is 6, and in table rp1, 6 is used as the value through the mapping relationship to obtain the key "&", 3201, "6" and "&" are stored in the initialization probability set pro_set, and pro_set = {3201, 6,"&"} is obtained; in pro_set = {3201, 6,"&"}, each element has a 1 / 3 probability of being hit to replace the original "Wu", and assuming that "&" is hit, "&" replaces "Wu". By analogy, after traversing "Wu Mou City Ha Mou Road No.", the desensitization of non-numeric characters can be completed.

[0079] Step S106, performing preliminary desensitization processing on the long text field data in the target data to obtain a preliminary desensitization processing result;

[0080] Among them, the long text field data in the embodiment of the present application is content data with a length greater than 30.

[0081] Optionally, in step S106, the long text field data in the target data is subjected to preliminary desensitization processing, and the preliminary desensitization processing result is obtained, including: screening a long text field set from the target data; using a preset sensitive word filtering model to filter the target sensitive words contained in the long text in the long text field set, and processing the target sensitive words through a preset strategy to obtain a filtered text; wherein the preset strategy includes: a one-third probability of keeping unchanged, a one-third probability of replacing the target sensitive words with preset symbols, and / or a one-third probability of replacing with any English letters; translating the filtered text into a preset type of foreign language content, extracting summary information from the filtered text and each type of foreign language content obtained by translation through a summary information extraction model to obtain summary information content; vectorizing each summary information content according to the sentence vector model to obtain a vector of a preset length, wherein all vectors have the same length and each element value in the vector is a positive integer; adding the element values ​​in the same position in all vectors, averaging them in integer form, obtaining a comprehensive average vector, and concatenating the values ​​of the elements at each position in the comprehensive average vector with a first preset symbol to obtain a preliminary desensitization processing result.

[0082] Specifically, Figure 2As shown, first, filter the long text field set, assuming that it includes the "content of the appeal of the relevant subject 1" field (record that there are two relevant subjects in the event, namely, the relevant subject 1 and the relevant subject 2, and record that the relevant subject 1 is the party that made the appeal) and the "event processing result" field;

[0083] Second, use a sensitive word filtering model such as DFA, NaiveFilter or BS (i.e., the preset sensitive word filtering model in the embodiment of the present application) to filter the target sensitive words contained in the long text, and process the target sensitive words with three strategies (i.e., the preset strategies in the embodiment of the present application), namely: one-third probability remains unchanged, one-third probability replaces the target sensitive words with "*" (i.e., the preset symbol in the embodiment of the present application), and one-third probability replaces them with any English letters.

[0084] For example, if the "Main content of the event" field contains a content such as "On October 18, 2022, they had inappropriate behavior", the DFA model is used to identify the target sensitive word "inappropriate", and "inappropriate" can be filtered. The target sensitive word "inappropriate" is filtered using three strategies, namely: one-third probability of keeping the original sensitive word, that is, the result after filtering is "On October 18, 2022, they had inappropriate behavior"; one-third probability of replacing it with "*", that is, the result after filtering is "On October 18, 2022, they had * behavior"; one-third probability of replacing it with "any English word", assuming that the hit "any English word" is "k", then the result after filtering is "On October 18, 2022, they had k behavior". The text content after filtering is recorded as mg_a;

[0085] Third, the text content after sensitive word filtering is translated into several foreign language contents (i.e., the preset types of foreign language contents in the embodiment of the present application), and then the summary information extraction model CNN+Attention, Jiagu, Hanlp, T5 or Bart (i.e., the summary information extraction model in the embodiment of the present application) is used to extract summary information from the filtered text content mg_a (i.e., the filtered text in the embodiment of the present application) and each translated foreign language content to obtain the corresponding summary information content.

[0086] For example: the above text content mg_a is translated into English, German, and Greek, and the foreign language contents obtained are recorded as mg_y, mg_d, and mg_x respectively. The probabilistic information extraction model CNN_Attention is used to extract summary information of mg_a, mg_y, mg_d, and mg_x respectively, and the summary information contents obtained are mg_a_2, mg_y_2, mg_d_2, and mg_x_2 respectively.

[0087] Fourth, use a sentence vector model such as fasttext, gensim or roberta (i.e., the sentence vector model in the embodiment of the present application) to vectorize each summary information content, and obtain a vector with a length of 16-512 (i.e., the preset length in the embodiment of the present application), wherein the lengths of all the vectors obtained are consistent, and the value of each element in the vector is a positive integer, and the values ​​of the elements in the same position in all the vectors obtained are added, rounded to integers and averaged to obtain a comprehensive average vector, and the values ​​of the elements at each position in the comprehensive average vector are concatenated with the symbol "-" (i.e., the first preset symbol in the embodiment of the present application) to obtain the value after preliminary desensitization.

[0088] For example, using the fasttext model, the summary information mg_a_2, mg_y_2, mg_d_2, and mg_x_2 are vectorized to obtain vectors of length 16, which are [109,2,93,1,5,11,6,8,19,82,65,10,28,76,3,14], [200,12,43,2,55,11,0,8,9,98,65,100,88,76,33,81], [10,12,41,21,65,1,0,7,9,98,35,108,88,76,803,71], [0,12,3,10 ,51,110,66,14,9,8,65,1,8,96,3,35], add the values ​​of the elements in the same position of the four vectors, round them to integers and take the average, and get a comprehensive average vector of [80,10,45,8,44,33,18,9,12,72,58,55,53,81,210,50], recorded as mg_p, concatenate the values ​​of each element in mg_p with the symbol "-", and get the initial desensitized value of "80-10-45-8-44-33-18-9-12-72-58-55-53-81-210-50", recorded as mg_p_2. Similarly, after traversing the long text field set, the desensitization of non-numeric characters can be completed.

[0089] Step S108, performing clustering embedding desensitization processing on the long text field data in the target data according to the preliminary desensitization processing result, to obtain a clustering embedding desensitization processing result;

[0090] Optionally, in step S108, clustering and embedding desensitization processing is performed on the long text field data in the target data according to the preliminary desensitization processing result, and the clustering and embedding desensitization processing result includes: for all fields in the target data belonging to the long text field set, clustering prediction is performed using a clustering model to obtain clustering categories; the clustering categories are connected to the end of the preliminary desensitization processing result through a second preset symbol to obtain a clustering desensitization processing result; through the constructed number and letter mapping dictionary, the element value of each position in the comprehensive average vector in the preliminary desensitization processing result is traversed, the element value is the numerator, the total number of letters in the mapping dictionary is the denominator, the remainder value is taken after division, and the corresponding letter value is obtained through the mapping dictionary with the remainder value as the key, and the symbol in the clustering desensitization processing result is replaced by the letter value in turn to obtain a clustering and embedding desensitization processing result.

[0091] Specifically, Figure 2 As shown, for all fields in the initial database belonging to the long text field set, clustering models dbscan, kmeans, kmeans++ or single-pass (i.e., the clustering model in the embodiment of the present application) are used to perform cluster prediction on the data under the field, wherein the number of clustering categories under each field is 26, and the clustering category names are integer values ​​of 1-26, i.e., 1, 2, 3, 4, ..., 24, 25, 26, respectively, and the predicted categories are connected in series to the tail of the preliminary desensitization processing result with the symbol "&*&" (i.e., the second preset symbol in the embodiment of the present application). Among them, the "target data" in the embodiment of the present application refers to a certain piece of data, and the target data is generated when the initial database is traversed.

[0092] For example: for the target data, the content of the "Main content of the event" field is "On October 18, 2022, they had inappropriate behavior...", using the single-pass clustering model, it is predicted that the cluster category it belongs to is 7. Assuming that the tail value mg_p_2 of the preliminary desensitization processing result is "80-10-45-8-44-33-18-9-12-72-58-55-53-81-210-50", the predicted category is concatenated to the tail of the preliminary desensitization processing result with the symbol "&*&" to obtain "80-10-45-8-44-33-18-9-12-72-58-55-53-81-210-50&*&7", recorded as mg_p_3.

[0093] Construct a mapping dictionary ys_dic = {1: "a", 2: "b", 3: "c", 4: "d", 5: "e", 6: "f", 7: "g", 8: "h", 9: "i", 10: "j", 11: "k", 12: "l", 13: "m", 14: "n", 15: "o", 16: "p", 17: "q", 18: "r", 19: "s", 20: "t", 21: "u", 22: "v", 23: "w", 24:"x", 25:"y", 26:"z"} (i.e., the number and letter mapping dictionary in the embodiment of the present application), traverse the element values ​​at each position in the comprehensive average vector, with the element values ​​as the numerator and the number 26 (i.e., the total number of letters in the mapping dictionary in the embodiment of the present application) as the denominator, take the remainder value after division, and use the remainder value as the key to obtain the corresponding letter value through the mapping dictionary ys_dic, and replace the corresponding values ​​with the connecting symbols in mg_p_3 in turn, that is, "-" and "&*&".

[0094] For example: Assume that mg_p is [80,10,45,8,44,33,18,9,12,72,58,55,53,81,210,50], according to the above (3.1) operation, we can get mg_p_2 as "80-10-45-8-44-33-18-9-12-72-58-55-53-81-210-50", and mg_p_3 as "80-10-45-8-44-33-18-9-12-72-58-55-53-81-210-50". &*&7", with the element value in mg_p as the numerator and the number 26 as the denominator, the remainder values ​​after division are 2, 10, 19, 8, 18, 7, 18, 9, 12, 20, 6, 3, 1, 3, 2, 24, and the remainder values ​​are used as the keys in the mapping dictionary ys_dic, and the corresponding letter values ​​are "b", "j", "s", "h", "r", "g", "r", "i", "l", "t", "f", "c", "a", "c", "b", "x". Replace the special symbols "-" and "&*&" in mg_p_3 with all the corresponding letter values ​​in order, and get "80b10j45s8h44r33g18r9i12l72t58f55c53a81c210b50x7", recorded as mg_p_4. By analogy, after traversing the long text field set, the desensitization of non-numeric characters can be completed.

[0095] Step S110, performing normal enhanced desensitization processing on the long text field data in the target data according to the clustering embedding desensitization processing result, and obtaining a desensitized data result of the long text field data.

[0096] Optionally, in step S110, performing normal enhanced desensitization processing on the long text field data in the target data according to the clustering embedded desensitization processing result includes: obtaining the number of tokens of the long text field data in the target data; inputting the number of tokens into the normal distribution probability function to obtain the corresponding probability value; inputting the probability value into the sigmoid activation function to obtain the sigmoid value; retaining all data after the decimal point in the sigmoid value, and stringifying all data after the decimal point in the sigmoid value to obtain a stringified result; concatenating the stringified result to the end of the clustering embedded desensitization processing result to obtain a desensitized data result of the long text field data.

[0097] Specifically, Figure 2 As shown, the long text field data in the target data (remember that the long text field data is the content data with a length greater than 30) is subjected to a normal sigmoid enhanced desensitization process (i.e., the normal enhanced desensitization process in the embodiment of the present application). Specifically: traverse all the data in the initial database, and for each piece of data, calculate the number of tokens of each long text field data (remember that the long text field data is the content data with a length greater than 30, otherwise the content data is non-long text field data). In order to obtain the maximum random perturbation, the number of tokens under the field data containing the maximum number of tokens is used as the input variable of the standard normal distribution probability function with a mean of 0 and a variance of 1, and a probability value in the range of 0-1 is obtained; then the probability value is used as the input variable of the sigmoid activation function to calculate the sigmoid value, and the sigmoid value is also in the range of 0-1. In order to enhance the random perturbation, it is necessary to remove the fixed part "0." in the sigmoid value, retain all data after the decimal point of the sigmoid value, and stringify it;

[0098] The stringified result is concatenated to the end of the clustering embedding desensitization processing result to obtain the final stage desensitized data result of the long text field.

[0099] Among them, the meaning of the token character content is a single numeric character, a single complete special character, a single Chinese character, or a single complete English word (assuming the text content is "Li's email address is: xdh417810@qq.com", the numeric characters are 4, 1, 7, 8, 1, 0, so the number of numeric characters is 6; the special characters are "@", ":", ".", so the number of special characters is 3; the Chinese characters are: Li, Mou, De, Di, Zhi, Wei, so the number of Chinese characters is 6; a single complete English word is email, so the number of English characters is 1. The total number of tokens accumulated is 6+3+6+1=16). For example: Assume that the field data containing the maximum number of tokens is the "Main Content of Event" field, and the maximum number of tokens is 618. The standard normal distribution probability formula is as follows:

[0100]

[0101] Let x be any positive integer between 1 and 10000, and obtain the probability value according to the standard normal distribution probability formula P(x), denoted as a.

[0102] Then use a as the input of the sigmoid function. The formula of the sigmoid function is as follows:

[0103]

[0104] Let y = a, and use the basic sigmoid function formula sigmoid(y) to obtain the sigmoid value, denoted as b. Obviously, the range of b is between 0 and 1. Assuming b is 0.648912, remove the fixed part "0." in the b value, retain all data after the decimal point of the b value, and stringify to obtain "648912". Concatenate "648912" into all non-long text field data and embed the tail of the clustering and desensitizing processing results. For example, the clustering and embedding desensitizing processing result is 80b10j45s8h44r33g18r9i12l72t58f55c53a81c210b50x7", recorded as mg_p_4, and the desensitized data obtained after concatenation is "80b10j45s8h44r33g18r9i12l72t58f55c53a81c210b50x7648912", recorded as mg_p_5. Similarly, after traversing the long text field set, the clustering and embedding desensitization of non-numeric characters can be completed.

[0105] For each piece of data in the initial database, the above steps are combined to perform desensitization processing, and each piece of data after desensitization processing is stored in the new database, and the desensitized storage of all data can be completed.

[0106] The method for desensitizing sensitive data provided in the embodiment of the present application introduces a sentence vector fusion desensitization strategy based on special_tokend.txt and vocab.txt to desensitize non-long text field data; introduces a sensitive word filtering model to desensitize long text field data, further uses a clustering embedding strategy to perform deeper data desensitization, and constructs a positive sigmoid enhancement strategy to perform enhanced desensitization and finally obtain the result, replacing the traditional data desensitization method. The invention can obtain data with higher data security and privacy protection levels, and provide a more economical, reliable and secure data processing solution.

[0107] The method for desensitizing sensitive data provided in the embodiment of the present application is to improve the data security and privacy protection level. This patent aims to fill the gap in current technology and provide a more reliable and secure data processing solution.

[0108] The embodiment of the present invention provides a method for desensitizing sensitive data, which comprises the following steps: acquiring target data; desensitizing non-long text field data in the target data; performing preliminary desensitizing on long text field data in the target data to obtain preliminary desensitizing results; performing clustering embedded desensitizing on long text field data in the target data based on the preliminary desensitizing results to obtain clustering embedded desensitizing results; performing positive enhanced desensitizing on long text field data in the target data based on the clustering embedded desensitizing results to obtain desensitized data results for the long text field data, thereby achieving a technical effect of having a higher level of data security and privacy protection, and providing a more economical, reliable and secure data processing solution.

[0109] The embodiment of the present invention provides a device for desensitizing sensitive data. Figure 3 A schematic diagram of a device for desensitizing sensitive data provided by an embodiment of the present invention; Figure 3 As shown, the device for desensitizing sensitive data provided by the embodiment of the present application includes: an acquisition module 30, used to acquire target data, and determine the non-long text field and the long text field according to the length of the text field in the target data; a first desensitization module 32, used to desensitize the non-long text field data in the target data; a second desensitization module 34, used to perform preliminary desensitization on the long text field data in the target data to obtain a preliminary desensitization result; a third desensitization module 36, used to perform clustering embedded desensitization on the long text field data in the target data according to the preliminary desensitization result, and obtain a clustering embedded desensitization result; a fourth desensitization module 38, used to perform normal enhanced desensitization on the long text field data in the target data according to the clustering embedded desensitization result, and obtain a desensitized data result for the long text field data.

[0110] An embodiment of the present invention provides a storage medium, which includes a stored program, and when the program is running, controls a device where the storage medium is located to execute: the above-mentioned method for desensitizing sensitive data.

[0111] The above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention.

Claims

1. A method for desensitizing sensitive data, characterized in that: include: Acquire target data, and determine a non-long text field and a long text field according to the length of a text field in the target data; Performing desensitization processing on the non-long text field data; Performing preliminary desensitization processing on the long text field data to obtain a preliminary desensitization processing result; Performing clustering embedding desensitization processing on the long text field data according to the preliminary desensitization processing result to obtain a clustering embedding desensitization processing result; According to the clustering embedding desensitization processing result, the long text field data is subjected to normal enhanced desensitization processing to obtain a desensitized data result of the long text field data.

2. The method for desensitizing sensitive data according to claim 1, characterized in that: The acquiring of target data comprises: Acquire collected data from the target data source through pre-authorization to form a target data set; storing the collected data in an initial database; The target data is acquired from the initial database.

3. The method for desensitizing sensitive data according to claim 1 or 2, characterized in that: The desensitizing process of the non-long text field data includes: Performing desensitization processing on the continuous numbers in the non-long text field data to obtain a first desensitization result; Performing desensitization processing on non-numeric characters in the non-long text field data to obtain a second desensitization result; The desensitization processing result of the non-long text field data is obtained according to the first desensitization result and the second desensitization result.

4. The method for desensitizing sensitive data according to claim 3, characterized in that: The desensitizing the continuous numbers in the non-long text field data to obtain a first desensitization result includes: Filtering a non-long text field set from the target data; Obtain a first vocabulary corresponding to a pre-trained language model; wherein each row in the first vocabulary contains only one token character content, and the token character content includes: Chinese characters, English characters and special characters; wherein the meaning of the token character content is: when the token character content in the first vocabulary is a Chinese character, the token character content is a single word; when the token character content in the first vocabulary is an English character, the token character content is a single complete English word; when the token character content in the first vocabulary is a special character, the token character content is a single complete special character; Traversing each line of the first vocabulary, storing each line of the content in a key-value pair database in the form of a key-value pair, and obtaining a first table; wherein the key-value pair form includes: the line number of each line of the content as a key, and the token character content as a value; Each field in the non-long text field set is traversed, and continuous numbers in the non-long text field data are desensitized in combination with the first table.

5. The method for desensitizing sensitive data according to claim 4, characterized in that: The traversing each field in the non-long text field set and performing desensitization processing on the continuous numbers in the non-long text field data in combination with the first table comprises: Adding the values ​​at each position in the consecutive numbers to obtain a first value; Divide the first value by the total number of rows in the first vocabulary and take the remainder to obtain a second value; Obtaining the token character content whose key is the second value in the first table; Pass the token character content through the word vector conversion model to obtain a first target vector; Divide the entire value of the consecutive numbers by the total number of rows in the first vocabulary and take the remainder to obtain a third value; Obtaining the token character content whose key is the third value in the first table; Pass the token character content through the word vector conversion model to obtain a second target vector; By traversing the first target vector and the second target vector, adding the element values ​​at the same index position and dividing by the position index, and taking the remainder, a remainder value of each index position in the first target vector and the second target vector is obtained; The remainder value of each index position is replaced by the continuous numbers through series connection to obtain a desensitization result of the continuous numbers.

6. The method for desensitizing sensitive data according to claim 3, characterized in that: The desensitizing the non-numeric characters in the non-long text field data to obtain a second desensitizing result includes: Filtering a non-long text field set from the target data; Obtain a special character table and a second vocabulary corresponding to the pre-trained language model; wherein each row in the special character table and the second vocabulary contains only one token character content, and the special character table is used to supplement the special characters in the second vocabulary; and the token character content is a single complete special character in the special character table; Traversing each line of the special character table and the second vocabulary table in a preset order, extracting the special characters and English characters in each line of the content, and storing them in a key-value pair database in the form of key-value pairs to obtain a second table; and extracting the Chinese characters in each line of the content, and storing them in the key-value pair database in the form of key-value pairs to obtain a third table; wherein the character content of each line of the content is the key, and the line number of each line of the content is the value; Each field in the non-long text field set is traversed, and non-numeric characters in the non-long text field data are desensitized in combination with the second table and the third table.

7. The method for desensitizing sensitive data according to claim 6, characterized in that: The traversing each field in the non-long text field set and desensitizing the non-numeric characters in the non-long text field data in combination with the second table and the third table comprises: Extracting non-numeric characters from each field to obtain a candidate set of non-numeric characters to be processed; Traverse each element in the candidate set of non-numeric characters to be processed and record it as a target element; Constructing an initialization probability set according to the target element; wherein, constructing an initialization probability set according to the target element comprises: if the target element is a key in the second table, finding the value corresponding to the key in the second table through the second table mapping, putting the value into the initialization probability set, and taking the value as the numerator, the length of the third table as the denominator, dividing and taking the remainder, obtaining the remainder value, and putting the remainder value into the initialization probability set; in the third table, taking the remainder value as the value, mapping the third table to obtain the key corresponding to the value in the third table, and putting the value into the initialization probability set; if the target element is a key in the third table, finding the value corresponding to the key in the third table through the third table mapping, putting the value into the initialization probability set, and taking the value as the numerator, the length of the second table as the denominator, dividing and taking the remainder, obtaining the remainder value, and putting the remainder value into the initialization probability set; in the second table, taking the remainder value as the value, mapping the second table to obtain the key corresponding to the value in the second table, and putting the value into the initialization probability set; In the initialization probability set, an element is randomly selected with equal probability to replace the target element, so that the non-numeric characters are desensitized.

8. The method for desensitizing sensitive data according to claim 1 or 2, characterized in that: The performing preliminary desensitization processing on the long text field data in the target data to obtain the preliminary desensitization processing result includes: Filtering a set of long text fields from the target data; Filtering target sensitive words contained in the long text in the long text field set using a preset sensitive word filtering model, and processing the target sensitive words using a preset strategy to obtain filtered text; wherein the preset strategy includes: one-third probability of keeping the target sensitive words unchanged, one-third probability of replacing the target sensitive words with preset symbols, and / or one-third probability of replacing them with any English letters; Translate the filtered text into a preset type of foreign language content, and extract summary information from the filtered text and each type of translated foreign language content using a summary information extraction model to obtain summary information content; Each summary information content of the sentence vector model is vectorized to obtain a vector of a preset length, wherein all vectors have the same length and each element value in the vector is a non-zero integer; the element values ​​at the same position in all the vectors are added and averaged in the form of integers to obtain a comprehensive average vector, and the values ​​of the elements at each position in the comprehensive average vector are concatenated with a first preset symbol to obtain the preliminary desensitization processing result.

9. The method for desensitizing sensitive data according to claim 8, characterized in that: The clustering embedding desensitization processing is performed on the long text field data in the target data according to the preliminary desensitization processing result, and the clustering embedding desensitization processing result is obtained, including: For all fields in the target data that belong to the long text field set, cluster prediction is performed using a clustering model to obtain cluster categories; The clustering category is serially connected to the end of the preliminary desensitization processing result through a second preset symbol to obtain a clustering desensitization processing result; Through the constructed number and letter mapping dictionary, the element value of each position in the comprehensive average vector in the preliminary desensitization processing result is traversed, the element value is the numerator, the total number of letters in the mapping dictionary is the denominator, the remainder value is taken after division, and the corresponding letter value is obtained through the mapping dictionary with the remainder value as the key, and the letter values ​​are replaced with the symbols in the cluster desensitization processing result in turn to obtain the clustered embedded desensitization processing result.

10. The method for desensitizing sensitive data according to claim 9, characterized in that: The performing normal enhanced desensitization processing on the long text field data in the target data according to the clustering embedding desensitization processing result comprises: Obtain the number of tokens of the long text field data in the target data; Input the number of tokens into the normal distribution probability function to obtain the corresponding probability value; Input the probability value into a sigmoid activation function to obtain a sigmoid value; retaining all data after the decimal point in the sigmoid value, and performing stringization on all data after the decimal point in the sigmoid value to obtain a stringized result; The stringization result is concatenated to the end of the clustering embedding desensitization processing result to obtain the desensitized data result of the long text field data.

11. A device for desensitizing sensitive data, characterized in that: include: An acquisition module, used for acquiring target data, and determining a non-long text field and a long text field according to the length of a text field in the target data; A first desensitization module, used for desensitizing the non-long text field data in the target data; A second desensitization module is used to perform preliminary desensitization processing on the long text field data in the target data to obtain a preliminary desensitization processing result; A third desensitization module is used to perform clustering embedding desensitization processing on the long text field data in the target data according to the preliminary desensitization processing result to obtain a clustering embedding desensitization processing result; The fourth desensitization module is used to perform a normal enhanced desensitization process on the long text field data in the target data according to the clustering embedding desensitization process result to obtain a desensitized data result of the long text field data.

12. A storage medium comprising a stored program, characterized in that: When the program is running, the device where the storage medium is located is controlled to execute: the method for desensitizing sensitive data as described in any one of claims 1 to 10.

Images