Method and apparatus for data de-identification preserving data characteristics

Abstract

The application provides a data desensitization method and device for preserving data characteristics, and belongs to the field of data security, wherein the method comprises the following steps: obtaining original plaintext data of a data source; identifying sensitive data in the original plaintext data; analyzing the data type and data length of the sensitive data, wherein the data type is used to represent whether the sensitive data needs to preserve text characteristics; and performing desensitization processing on the sensitive data according to the data type and data length. Through the embodiment of the application, the technical problem that data characteristics are lost due to desensitization of sensitive data in the related art is solved, and the business value of the sensitive data is preserved while ensuring data security.

Description

Technical Field

[0001] This invention relates to the field of data security, and more specifically, to a data desensitization method and apparatus that preserves data characteristics. Background Technology

[0002] In related technologies, to protect sensitive data, it is necessary to de-identify the data to remove its sensitivity before using it in development and testing environments, thus preventing the leakage of sensitive data. Data de-identification refers to the transformation of data for certain sensitive information using de-identification rules. However, due to the massive amount of data in commercial banks, de-identifying every piece of data according to specific rules would be computationally intensive and inefficient. Therefore, it is essential to identify the data and improve the de-identification rules before performing data de-identification.

[0003] According to national standards, sensitive information needs to be anonymized. Sensitive information includes: basic personal information (family relationships / address / phone number); personal identification information (ID card / military officer's certificate / passport / driver's license / work permit / social security card / residence permit); personal biometric information (personal genes / fingerprint / voiceprint / palm print / earprint / iris / facial recognition features); online identity information (password / password protection answer / user's personal digital certificate); and personal health and physiological information (relevant records of personal illness and treatment, such as symptoms / hospitalization records / medical orders / lab reports / surgical and anesthesia records / nursing records / medication records / drug and food allergy information / reproductive information / past medical history / treatment history / family medical history / current medical history). Information such as: history of infectious diseases, etc.; personal property information (bank account / identification information / deposit information / real estate information / loan records / credit information / transaction and consumption records / bank statements, etc.); personal communication information (communication records and content / SMS / MMS / emails / and data describing personal communications, etc.); contact information (address book / friend list / group list / email address list, etc.); personal internet browsing history; personal frequently used device information; unique device identifiers (such as IMEI / AndroidID / IDFA / OpenUDID / GUID / SIM card IMSI information, etc.); personal location information (whereabouts / precise location information / accommodation information); other personal information (political views / religious beliefs / marital history / sexual orientation); conventional data anonymization methods often anonymize users' real names, mobile phone numbers, ID cards, and bank card numbers through replacement, invalidation, scrambling, symmetric encryption, etc., and most of these methods are for small amounts of data. This paper involves anonymizing sensitive personal information data in TB-level big data scenarios. In distributed federated modeling, the relationships between multiple data sources are jointly modeled, and distributed model training, such as transfer learning, is conducted. Data security and the protection of user privacy are paramount during this process. The parties involved in data federation often do not want their data directly exposed to others, so they may directly delete or replace sensitive data. This process also reduces useful data within the sensitive data, such as patterns useful for training the model.

[0004] No efficient and accurate solution has yet been found to address the aforementioned issues in the relevant technologies. Summary of the Invention

[0005] This invention provides a data desensitization method and apparatus that preserves data characteristics, in order to solve the technical problems in related technologies.

[0006] According to an embodiment of the present invention, a data anonymization method for preserving data characteristics is provided, comprising: obtaining original plaintext data from a data source; identifying sensitive data in the original plaintext data; parsing the data type and data length of the sensitive data, wherein the data type is used to characterize whether the sensitive data needs to retain text characteristics; and performing anonymization processing on the sensitive data according to the data type and data length.

[0007] Furthermore, identifying sensitive data in the original plaintext data includes: obtaining table structure information and sampled data of the original plaintext data, wherein the table structure information is a description of the data structure of the original plaintext data; using a double-array trie tree to search for sensitive words in the table structure information and the sampled data, wherein the double-array trie tree includes a first trie tree and a second trie tree, the first trie tree is used to perform a depth-first search on the table structure information and the sampled data, and the second trie tree is used to perform a breadth-first search on the table structure information and the sampled data; if the table structure information and the sampled data contain sensitive words, extracting the sensitive data at the location of the sensitive words.

[0008] Furthermore, after searching for sensitive words in the table structure information and the sampled data using a double-array trie tree, the method further includes: if the table structure information and the sampled data do not contain sensitive words, calling a fixed rule template, wherein the fixed rule template integrates multiple user-defined functions (UDFs), each UDF being used to identify a sensitive data type; and using the multiple UDFs to match each column of the original plaintext data respectively. If a target column containing sensitive data is matched, the target column data is extracted from the original plaintext data and identified as sensitive data.

[0009] Further, the desensitization processing of the sensitive data according to the data type and data length includes: calculating the data length of the sensitive data and / or calculating the text length of the annotation text of the sensitive data, counting the number of sensitive words in the sensitive data, and calculating the sensitive word length of all sensitive words; if the data length is less than a first length threshold and the number of sensitive words is greater than a first quantity threshold, the sensitive data is completely desensitized; if the data length is greater than the first length threshold and less than a second length threshold, and the number of sensitive words is greater than the second quantity threshold, the sensitive data is completely desensitized; if the text length is less than the first length threshold and the number of sensitive words is greater than the second quantity threshold, the sensitive data is completely desensitized; if the data length is greater than the second length threshold, the length ratio of the sensitive word length to the data length is calculated; if the length ratio is lower than a proportion threshold, the sensitive data is partially desensitized; if the length ratio is higher than a proportion threshold, the sensitive data is completely desensitized.

[0010] Furthermore, the complete desensitization of the sensitive data includes: if the data type of the sensitive data is a specified type, extracting non-numerical data and first numerical data from the sensitive data, wherein the specified type indicates that the sensitive data needs to retain text characteristics; performing MD5 encryption on the non-numerical data to obtain encrypted data; mapping the encrypted data according to a preset encoding table to obtain corresponding second numerical data; performing desensitization processing on the first numerical data to obtain first desensitized data; and performing desensitization processing on the second numerical data to retain data characteristics to obtain second desensitized data.

[0011] Furthermore, the second numerical data undergoes desensitization processing to preserve its data characteristics, resulting in the second desensitized data. This process includes: scaling the plaintext data corresponding to the second numerical data according to an overall expectation to obtain scaled data; extracting the frequency domain features, time domain features, and convolutional transformation features of the scaled data to obtain a feature combination; and mapping the second numerical data based on the feature combination to obtain the second desensitized data of the second numerical data.

[0012] Furthermore, the local desensitization of the sensitive data includes: locating the character position of the sensitive word in the sensitive data; using the character position as a reference position, extracting M characters before the reference position and N characters after the reference position in the sensitive data, where M and N are both positive integers greater than 0; replacing the M characters and the N characters with preset characters to obtain the desensitized data of the sensitive data.

[0013] According to another embodiment of the present invention, a data anonymization device for preserving data characteristics is provided, comprising: an acquisition module for acquiring original plaintext data from a data source; an identification module for identifying sensitive data in the original plaintext data; a parsing module for parsing the data type and data length of the sensitive data, wherein the data type is used to characterize whether the sensitive data needs to retain text characteristics; and an anonymization module for performing anonymization processing on the sensitive data according to the data type and data length.

[0014] Furthermore, the identification module includes: an acquisition unit, configured to acquire table structure information and sampled data of the original plaintext data, wherein the table structure information is a description of the data structure of the original plaintext data; a search unit, configured to search for sensitive words in the table structure information and the sampled data using a double-array trie tree, wherein the double-array trie tree includes a first trie tree and a second trie tree, the first trie tree is used to perform a depth-first search on the table structure information and the sampled data, and the second trie tree is used to perform a breadth-first search on the table structure information and the sampled data; and an extraction unit, configured to extract sensitive data at the location of the sensitive words if the table structure information and the sampled data contain sensitive words.

[0015] Furthermore, the identification module also includes: a calling unit, configured to, after the search unit uses a double-array trie tree to search for sensitive words in the table structure information and the sampled data, if the table structure information and the sampled data do not contain sensitive words, call a fixed rule template, wherein the fixed rule template integrates multiple user-defined functions (UDFs), each UDF being used to identify a sensitive data type; a matching unit uses the multiple UDFs to match each column of data in the original plaintext data respectively; if the extraction unit matches target column data containing sensitive data, it extracts the target column data from the original plaintext data and identifies the target column data as sensitive data.

[0016] Furthermore, the desensitization module includes: a calculation unit, used to calculate the data length of the sensitive data and / or the text length of the annotation text of the sensitive data, count the number of sensitive words in the sensitive data, and calculate the sensitive word length of all sensitive words; and a desensitization unit, used to perform full desensitization of the sensitive data if the data length is less than a first length threshold and the number of sensitive words is greater than a first quantity threshold; to perform full desensitization of the sensitive data if the data length is greater than the first length threshold and less than a second length threshold and the number of sensitive words is greater than the second quantity threshold; to perform full desensitization of the sensitive data if the text length is less than the first length threshold and the number of sensitive words is greater than the second quantity threshold; and to calculate the length ratio of the sensitive word length to the data length if the data length is greater than the second length threshold, and to perform partial desensitization of the sensitive data if the length ratio is lower than a proportion threshold, and to perform full desensitization of the sensitive data if the length ratio is higher than a proportion threshold.

[0017] Furthermore, the desensitization unit includes: a first extraction subunit, used to extract non-numerical data and first numerical data from the sensitive data if the data type of the sensitive data is a specified type, wherein the specified type indicates that the sensitive data needs to retain text characteristics; an encryption subunit, used to perform MD5 encryption on the non-numerical data to obtain encrypted data; a mapping subunit, used to map the encrypted data according to a preset encoding table to obtain corresponding second numerical data; and a first desensitization subunit, used to perform desensitization processing on the first numerical data to obtain first desensitized data, and perform desensitization processing on the second numerical data to retain data characteristics to obtain second desensitized data.

[0018] Furthermore, the first desensitization subunit is also used to: scale the plaintext data corresponding to the second numerical data according to the overall expectation to obtain scaled data; extract the frequency domain features, time domain features, and convolutional transformation features of the scaled data to obtain a feature combination; and map the second numerical data based on the feature combination to obtain the second desensitized data of the second numerical data.

[0019] Furthermore, the desensitization unit includes: a positioning subunit, used to locate the character position of the sensitive word in the sensitive data; a second extraction subunit, used to extract M characters before the reference position and N characters after the reference position in the sensitive data, where M and N are both positive integers greater than 0; and a second desensitization subunit, used to replace the M characters and the N characters with preset characters to obtain desensitized data of the sensitive data.

[0020] According to another aspect of the embodiments of this application, a storage medium is also provided, the storage medium including a stored program that executes the above steps when the program is run.

[0021] According to another aspect of the embodiments of this application, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; wherein: the memory is used to store computer programs; and the processor is used to execute the steps in the above method by running the programs stored in the memory.

[0022] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the steps in the above-described method.

[0023] Through the embodiments of this invention, the original plaintext data of the data source is obtained, sensitive data in the original plaintext data is identified, and the data type and data length of the sensitive data are parsed. The data type is used to characterize whether the sensitive data needs to retain textual characteristics. Sensitive data is de-identified according to the data type and data length. By parsing the data length of the sensitive data, partial or complete de-identification of the sensitive data can be performed, improving the de-identification speed. By parsing the data type of the sensitive data, sensitive data that needs to include textual characteristics can be de-identified without losing the spatial distribution and temporal relationship characteristics of the sensitive data itself, preserving the regular information of the sensitive data. This solves the technical problem of loss of data characteristics caused by de-identification of sensitive data in related technologies, ensuring data security while preserving the business value of sensitive data. Attached Figure Description

[0024] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings: Figure 1 This is a hardware structure block diagram of a server according to an embodiment of the present invention; Figure 2 This is a flowchart of a data desensitization method that preserves data characteristics according to an embodiment of the present invention; Figure 3 This is a flowchart of a sensitive information identification method according to an embodiment of the present invention; Figure 4 This is a flowchart of a distributed big data de-identification method according to an embodiment of the present invention; Figure 5 This is a structural block diagram of a data desensitization device that preserves data characteristics according to an embodiment of the present invention. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, and not all of them. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present application. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present application can be combined with each other.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] Example 1 The method embodiment provided in Embodiment 1 of this application can be executed in a computer, processor, server, or similar processing device. Taking running on a server as an example, Figure 1 This is a hardware structure block diagram of a server according to an embodiment of the present invention. Figure 1 As shown, a server may include one or more ( Figure 1 Only one is shown in the image. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. Optionally, the server may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the server described above. For example, the server may also include components that are more complex than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0028] The memory 104 can be used to store server programs, such as application software programs and modules, like the server program corresponding to a data desensitization method for preserving data characteristics in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the server program stored in the memory 104, thereby implementing the aforementioned method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the server via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0029] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the server's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0030] This embodiment provides a data anonymization method that preserves data characteristics. Figure 2 This is a flowchart of a data anonymization method that preserves data characteristics according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps: Step S202: Obtain the original plaintext data from the data source; The solution in this embodiment can be applied to distributed big data anonymization in federated learning modeling processes, and may include one or more data sources.

[0031] Step S204: Identify sensitive data in the original plaintext data; Step S206: Parse the data type and data length of the sensitive data, where the data type is used to characterize whether the sensitive data needs to retain text characteristics; The text characteristics in this embodiment can be spatial distribution characteristics, temporal relationship characteristics, etc., depending on business requirements.

[0032] Step S208: De-identify sensitive data according to data type and data length.

[0033] Through the above steps, the original plaintext data of the data source is obtained, sensitive data in the original plaintext data is identified, and the data type and length of the sensitive data are parsed. The data type is used to characterize whether the sensitive data needs to retain its textual characteristics. Sensitive data is de-identified based on its data type and length. By parsing the data length of the sensitive data, partial or complete de-identification can be performed, improving the de-identification speed. By parsing the data type of the sensitive data, de-identification can be performed on sensitive data that needs to retain textual characteristics. While de-identifying the sensitive data, the spatial distribution and temporal relationship characteristics of the sensitive data itself are not lost, preserving the regular information of the sensitive data. This solves the technical problem of data characteristic loss caused by de-identification of sensitive data in related technologies, ensuring data security while preserving the business value of sensitive data.

[0034] In one embodiment of this example, identifying sensitive data in the original plaintext data includes: S11, Obtain the table structure information and sampled data of the original plaintext data, wherein the table structure information is the description information of the data structure of the original plaintext data; In one example, a Spark distributed computing cluster is used to extract 1,000 data points and table structure descriptions from a multi-dimensional table in parallel. This data DataFrame and table structure description dictionary serve as input to a personal sensitive information identification model.

[0035] S12, a double-array trie tree is used to search for sensitive words in the table structure information and sampled data. The double-array trie tree includes a first trie tree and a second trie tree. The first trie tree is used to perform a depth-first search on the table structure information and sampled data, and the second trie tree is used to perform a breadth-first search on the table structure information and sampled data. The first trie includes a set of sensitive words, and the second trie includes a set of non-sensitive words.

[0036] Optionally, the sensitive word set is based on sensitive information extraction. According to relevant sensitive information definition standards, sensitive information includes: basic personal information (family relationship / address / phone number); personal identity information (ID card / military officer's certificate / passport / driver's license / work permit / social security card / residence permit); personal biometric information (personal genes / fingerprint / voiceprint / palmprint / ear / iris / facial recognition features); online identity information (password / password protection answer / user's personal digital certificate); personal health and physiological information (relevant records generated by personal illness and treatment, such as symptoms / hospitalization records / doctor's orders / lab reports / surgical and anesthesia records / nursing records / medication records / drug and food allergy information / fertility information / past medical history / treatment status / family medical history / current medical history). Personal information includes: history of infectious diseases, etc.; personal property information (bank account / identification information / deposit information / real estate information / loan records / credit information / transaction and consumption records / bank statements, etc.); personal communication information (communication records and content / SMS / MMS / emails / and data describing personal communications, etc.); contact information (address book / friend list / group list / email address list, etc.); personal internet browsing history; personal frequently used device information; unique device identifier (such as IMEI / AndroidID / IDFA / OpenUDID / GUID / SIM card IMSI information, etc.); personal location information (whereabouts / precise location information / accommodation information); and other personal information (political views / religious beliefs / marital history / sexual orientation). Collecting keyword corpora related to the above-defined sensitive personal information may include the following: "Basic Personal Information": ["Name", "Birthday", "Gender", "Ethnicity", "Nationality", "Family Relationship", "Address", "Telephone Number", "Email Address"]; "Personal Identity Information": ["ID Card", "Military Officer's Certificate", "Passport", "Driver's License", "Work Permit", "Access Permit", "Social Security Card", "Residence Permit"]; "Personal Biometric Information": ["Personal Genes", "Fingerprint", "Voiceprint", "Pattern Print", "Auricle", "Iris", "Facial Recognition Features"]; "Personal Health and Physiological Information": ["Relevant Records of Personal Illness Treatment", "Symptoms", "Hospitalization Records", "Doctor's Orders", "Laboratory Reports", "Surgery and Anesthesia Records", "Nursing Records", "Medication Records", "Allergy Information", "Reproductive Health Information", "Medical History", "Treatment Status", "Family Medical History", "Present Illness History", "History of Infectious Diseases", "Health Status", "Weight", "Height", "Lung Capacity"]; "Personal Education and Work Information": ["Occupation", "Position", "Work Unit", "Education Level"] "Degree", "Education", "Work Experience", "Training Records", "Transcripts"]; "Personal Asset Information": ["Bank Account", "Authentication Information", "Password", "Deposit Information", "Amount of Funds", "Payment", "Received Payment", "Real Estate Information", "Loan Records", "Credit Information", "Transactions", "Consumption Records", "Transaction History", "Virtual Currency", "Virtual Transactions", "Game Redemption Codes", "Virtual Property Information"]; "Personal Communication Information": ["Communication Records", "Communication Content", "SMS", "MMS", "Email", "Personal Communications"]; "Contact Information": ["Address Book", "Friends List", "Group List", "Email Address List"]; "Personal Internet Access Records": ["Logs", "Personal Information", "Operation Records", "Browsing Records", "Software Usage Records", "Click Records", "Favorites List"]; "Personal Frequently Used Device Information": ["Hardware Serial Number", "MAC Address", "Software List", "Unique Device Identifier Address", "IMEI", "AndroidID", "IDFA", "OpenUDID", "GUID"]; "SIM card", "IMSI information"]; "Personal location information": ["Ways", "Location information", "Accommodation information", "Latitude and longitude"]; "Personal information": ["Political views", "Religious beliefs", "Marital history", "Sexual orientation", "Not disclosed"];'Interest', 'Responsible for', 'Settlement', 'Profit', 'Brokerage', 'Name', 'Contact Person', 'Password', 'Account', 'Plan', 'Brokerage', 'Code', 'Company', 'Group', 'Address', 'Username', 'Gross Profit', 'Revenue', 'Number of Orders', 'Phone', 'Order Number', 'Number', 'Order', 'Party', 'Supplier', 'Distributor', 'Document', 'ID Card', 'Residence Permit', 'Passport', 'Role', 'Name', 'Abbreviation'.

[0037] The key to constructing the sensitive information identification model in this embodiment is building a double-array search tree based on the aforementioned personal sensitive information keywords. Constructing a double-array search tree allows for a trade-off between space and the time required to match the original plaintext of the sensitive information, thereby accelerating identification. This is mainly achieved by utilizing the common prefixes of the strings to save matching time; its complexity is O(L), which depends only on the length of the original plaintext information to be queried.

[0038] S13, If the table structure information and sampled data contain sensitive words, extract the sensitive data at the location of the sensitive words.

[0039] This embodiment constructs a double-array trie using key information of sensitive data, and employs a combination of depth-first and breadth-first search to achieve sensitive information matching and identification. It also performs association identification on closely related columns such as age and date of birth. Figure 3 This is a flowchart of a sensitive information identification method according to an embodiment of the present invention.

[0040] In another implementation scenario of this embodiment, after searching for sensitive words in the table structure information and sampled data using a double-array trie tree, the method further includes: if the table structure information and sampled data do not contain sensitive words, calling a fixed rule template, wherein the fixed rule template integrates multiple user-defined functions (UDFs), each UDF being used to identify a sensitive data type; using multiple UDFs to match each column of the original plaintext data; if a target column containing sensitive data is matched, extracting the target column data from the original plaintext data and identifying the target column data as sensitive data.

[0041] In some instances of this embodiment, for data columns without column name comments, a fixed template rule matching method is used for identification. For example, a fixed rule template is predefined for sensitive data with a fixed format, such as "ID card number", "mobile phone number", "IP address", "bank card number" and "device unique identifier". The rule template is written into a UDF function supported by Spark, and then the template function is used to match each column in the dataset to identify whether the current data is personal sensitive information data.

[0042] In this embodiment, the desensitization process for sensitive data based on data type and data length includes: S21, calculate the data length of sensitive data and / or calculate the text length of the annotation text of sensitive data, count the number of sensitive words in sensitive data, and calculate the length of all sensitive words; S22, if the data length is less than the first length threshold and the number of sensitive words is greater than the first quantity threshold, all sensitive data is anonymized; if the data length is greater than the first length threshold and less than the second length threshold, and the number of sensitive words is greater than the second quantity threshold, all sensitive data is anonymized; if the text length is less than the first length threshold and the number of sensitive words is greater than the second quantity threshold, all sensitive data is anonymized; if the data length is greater than the second length threshold, the length ratio of sensitive word length to data length is calculated. If the length ratio is lower than the proportion threshold, the sensitive data is partially anonymized; if the length ratio is higher than the proportion threshold, the sensitive data is completely anonymized.

[0043] In one implementation scenario, if the data length is less than 30 characters and a single sensitive keyword appears, it is considered fully anonymized. If the comment text length is greater than 30 characters or the data length is greater than 30 characters but less than 100 characters, and more than two sensitive words appear, it is considered fully anonymized. If the data length is greater than 100 characters and the total length of the sensitive words is less than 20% of the data length, it is partially anonymized. If the data length is greater than 100 characters and the total length of the sensitive words is greater than 20% of the data length, it is fully anonymized.

[0044] In one example, desensitizing sensitive data includes: if the data type of the sensitive data is a specified type, extracting non-numerical data and first numerical data from the sensitive data, wherein the specified type indicates that the sensitive data needs to retain textual characteristics; performing MD5 encryption on the non-numerical data to obtain encrypted data; mapping the encrypted data according to a preset encoding table to obtain corresponding second numerical data; desensitizing the first numerical data to obtain first desensitized data; and desensitizing the second numerical data while retaining its data characteristics to obtain second desensitized data.

[0045] When performing desensitization on the first numerical data to obtain the first desensitized data, desensitization processing can be performed with or without retaining data characteristics, depending on the data type of the first numerical data. For sensitive information that does not need to retain data characteristics, the following desensitization operations are performed on different types of data that need to be desensitized: Replacement and desensitization: Replacing sensitive plaintext data with data that is drastically different from and completely unrelated to the plaintext, such as generating a UUID for replacement. Replacing data with other methods will cause the anonymized data to completely lose its business meaning. This applies to personal information such as accounts, passwords, and unique device identifiers for frequently used personal devices—data that is completely random and meaningless for intelligent data modeling.

[0046] Random Transformation Desensitization: This method uses a random function to randomly replace the original plaintext. Numerical data types are randomly increased or decreased by percentage; date types are randomly increased or decreased by the number of days; character data is randomly replaced with letters or numbers. Random transformation desensitization is suitable for personal information such as mobile phone numbers and ID card numbers. Data shuffling: This method de-identifies data by randomly swapping positions, thus disrupting the original meaning. Its characteristics include preserving the data range, features, and business meaning of the fields, but sacrificing security as there is a possibility of data reversion. It is generally used for de-identifying less important personal information. Example: 20211093 -> 93011220.

[0047] MD5 encryption: Encrypts data to be anonymized, and the user decrypts it using a different key to obtain the original data. It is used less frequently. Characteristics: It does not guarantee data characteristics and business meaning, has security risks such as key leakage and insufficient encryption strength, and consumes a lot of cluster computing power. Suitable for anonymizing business data that requires reverse engineering. Example: abc -> TH3Wwi2wif51ga.

[0048] On the other hand, the second desensitized data is processed to preserve its data characteristics, which includes: scaling the plaintext data corresponding to the second numerical data according to the overall expectation to obtain scaled data; extracting the frequency domain features, time domain features, and convolutional transformation features of the scaled data to obtain feature combinations; and mapping the second numerical data based on the feature combinations to obtain the second desensitized data of the second numerical data.

[0049] Preserving data characteristics for data anonymization allows for the removal of sensitive data that retains its distribution characteristics, making it difficult to reverse-parse plaintext. This lays the foundation for federated learning. It can be used for anonymizing sensitive information useful in the intelligent data modeling and reasoning process.

[0050] To convert non-numerical data to numerical data, following natural language processing methods, the non-numerical data undergoes the following transformation and mapping: The non-numerical data is desensitized by irreversible MD5 encryption. English letters and numbers are shuffled and encoded. The MD5 encrypted results are then mapped using an encoding table. The mapped numerical data undergoes the following desensitization process.

[0051] Numerical data (which can be transformed numerical data or numerical data extracted from sensitive data) is anonymized. The method for anonymizing numerical data while preserving its characteristics is as follows: To preserve the original plaintext distribution of sensitive data, a one-to-one mapping can be performed according to certain rules. However, mapping according to rules is easy to reverse according to the rules, which can only be used in some scenarios.

[0052] To preserve the distribution of the original plaintext, it is necessary to retain its dispersion. Based on the processing principle of image mean filtering, the original numerical plaintext is first scaled according to the overall expectation. This scaling method can maintain the original distribution of the data and provides higher security.

[0053] To increase the difficulty of reverse engineering the de-identified data, we selected a method to extract features from the filtered data using frequency domain features, time domain features, and convolutional transform features with a Gaussian kernel. The combined results of these feature extractions were then used to map the original plaintext. This resulted in de-identified data that preserved the characteristics of the original plaintext.

[0054] In one example, the algorithm logic for desensitizing numerical data includes: Let the dataset X = (x1, x2, ..., xn), where P(xi) (i = 1, ..., n) is the frequency of xi, E(X) is the expectation of dataset X, and D(X) is the variance of dataset X.

[0055] When considering the distribution characteristics expressed by statistical features, we mainly consider sensitivity and stability. Sensitivity refers to the sensitivity of a statistical indicator to changes in the corresponding indicator when the data changes, while stability refers to the degree of stability of a statistical indicator in response to changes in the data.

[0056] Time-domain features: These describe the waveform of a signal according to a specific time sequence. Statistical features of time-domain signals affected by dimensions include variance, standard deviation, extrema, peak value, root square amplitude, and RMS value. Statistical features of time-domain signals largely unaffected by dimensions include kurtosis factor, peak value, skewness factor, waveform factor, margin factor, impulse factor, and gap factor. The kurtosis index describes the distribution of a variable and has high sensitivity, making it more suitable for the desensitization method of this invention compared to other indices. Therefore, we choose kurtosis as the time-domain feature. Kurtosis is the fourth central moment of a random variable divided by the fourth power of the standard deviation, mainly used to test the degree to which a signal deviates from a normal distribution. The formula is as follows: Mean: Standard deviation: ; Obtain the kurtosis factor: ; Frequency Domain Characteristics: Frequency domain analysis allows observation of signal characteristics at different frequencies. Generally, time domain analysis is more intuitive, while frequency domain representation is more concise. Observing signals in the frequency domain makes the analysis of problems more in-depth and convenient. Methods for calculating frequency domain characteristics include: frequency, centroid frequency, average frequency, root mean square frequency, and frequency standard deviation. Here, the root mean square frequency is used to measure frequency domain characteristics. The formula is as follows: Root mean square frequency: ; Feature extraction using one-dimensional convolution with Gaussian kernels: Convolution is essentially filtering the signal; convolution uses an impulse function to represent the excitation function, and then solves for the zero-state response of the system based on the impulse response.

[0057] Convolution is a transformation (integration) of multiple functions. For a linear time-invariant system, the input can be decomposed into the sum of many impulses of different intensities (integration in the time domain), and the output is the sum (or integral) of the responses produced by these impulses acting on the system. Therefore, the physical meaning of convolution is to express the relationship between the input, the system impulse response, and the output in the time domain.

[0058] One-dimensional convolution: F_c(x) = g(x) f(x); First, reflect the function x(k) relative to the origin, then shift it to the right by a distance t. Multiply the two functions and integrate the result to obtain the output at point t. Repeat this process for each value of t to obtain the output curve.

[0059] One-dimensional Gaussian kernel function: ;u is the mean. The Gaussian convolution kernel is the only transformation kernel that achieves scaling, and it is also the only linear kernel. A one-dimensional Gaussian kernel convolution operation is performed on datasets f(x) and g(x), where N is the dimension of the dataset f(x) = x. ; After applying mean filtering to the above calculation results and performing addition, subtraction, multiplication, and concatenation operations, the desensitized result Y is: .

[0060] In another example, local desensitization of sensitive data includes: locating the character position of the sensitive word in the sensitive data; using the character position as the reference position, extracting M characters before the reference position and N characters after the reference position in the sensitive data, where M and N are both positive integers greater than 0; replacing the M characters and N characters with preset characters to obtain the desensitized data of the sensitive data.

[0061] In this embodiment, when de-identifying sensitive data, a distributed de-identification task can be used. For distributed de-identification tasks, Spark MapPartion is used to process multi-party data in parallel. An interruption strategy is implemented if the de-identification task is abnormally interrupted during execution, and incremental execution of the de-identification task ensures that already de-identified data is not re-processed. In a production example of this invention, Figure 4 This is a flowchart of a distributed big data anonymization method according to an embodiment of the present invention, such as... Figure 4 As shown, the processing flow includes: Obtain data access permissions for all parties involved in federated modeling; The sensitive data identification method is encapsulated into a UDF function that can run on a Spark cluster; Record sensitive data and information; The desensitization methods for different data types will be written as UDF functions; By connecting to Hive source data using Spark + HiveJDBC, a distributed de-identification computation task is implemented by calling the sensitive data identification UDF function on the source data and the corresponding de-identification method UDF function on the sensitive data. For incremental data, use Spark Streaming + Kafka to call the sensitive data identification and desensitization methods. The anonymized data is stored in the corresponding data storage medium according to business needs, using either HDFS or Greenplum.

[0062] Record the desensitization process and encapsulate the desensitization identification and desensitization method into an API for use by all parties involved in the federated modeling process.

[0063] In a production example, this invention uses the HDFS distributed file system and Greenplum distributed database as the storage medium for the anonymized data. This invention constructs a personal sensitive information identification model to categorize the original plaintext data into three types: no anonymization required, partially anonymized, and fully anonymized. Data requiring no anonymization retains its original plaintext, partially anonymized data is anonymized, and different types of data requiring full anonymization are anonymized. This preserves the relative spatial distribution and temporal sequence of the original plaintext, minimizing the loss in data distribution. It satisfies both the requirements of federated learning modeling and data security, making it suitable for distributed anonymization operations on massive datasets.

[0064] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0065] Example 2 This embodiment also provides a data desensitization device that preserves data characteristics. This device is used to implement the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0066] Figure 5 This is a structural block diagram of a data desensitization device that preserves data characteristics according to an embodiment of the present invention, such as... Figure 5 As shown, the device includes: The acquisition module 50 is used to acquire the original plaintext data from the data source; The identification module 52 is used to identify sensitive data in the original plaintext data; The parsing module 54 is used to parse the data type and data length of the sensitive data, wherein the data type is used to characterize whether the sensitive data needs to retain text characteristics; The desensitization module 55 is used to desensitize the sensitive data according to the data type and data length.

[0067] Optionally, the identification module includes: an acquisition unit, configured to acquire table structure information and sampled data of the original plaintext data, wherein the table structure information is a description of the data structure of the original plaintext data; a search unit, configured to search for sensitive words in the table structure information and the sampled data using a double-array trie tree, wherein the double-array trie tree includes a first trie tree and a second trie tree, the first trie tree is used to perform a depth-first search on the table structure information and the sampled data, and the second trie tree is used to perform a breadth-first search on the table structure information and the sampled data; and an extraction unit, configured to extract sensitive data at the location of the sensitive words if the table structure information and the sampled data contain sensitive words.

[0068] Optionally, the identification module further includes: a calling unit, configured to, after the search unit searches for sensitive words in the table structure information and the sampled data using a double-array trie tree, if the table structure information and the sampled data do not contain sensitive words, call a fixed rule template, wherein the fixed rule template integrates multiple user-defined functions (UDFs), each UDF being used to identify a sensitive data type; a matching unit uses the multiple UDFs to match each column of data in the original plaintext data respectively; and if the extraction unit matches target column data containing sensitive data, extracts the target column data from the original plaintext data and identifies the target column data as sensitive data.

[0069] Optionally, the desensitization module includes: a calculation unit, configured to calculate the data length of the sensitive data and / or the text length of the annotation text of the sensitive data, count the number of sensitive words in the sensitive data, and calculate the length of all sensitive words; and a desensitization unit, configured to: desensitize all sensitive data if the data length is less than a first length threshold and the number of sensitive words is greater than a first quantity threshold; desensitize all sensitive data if the data length is greater than the first length threshold and less than a second length threshold and the number of sensitive words is greater than the second quantity threshold; desensitize all sensitive data if the text length is less than the first length threshold and the number of sensitive words is greater than the second quantity threshold; and calculate the length ratio of the sensitive word length to the data length if the data length is greater than the second length threshold, and if the length ratio is lower than a proportion threshold, desensitize the sensitive data partially, and if the length ratio is higher than a proportion threshold, desensitize the sensitive data completely.

[0070] Optionally, the desensitization unit includes: a first extraction subunit, configured to extract non-numerical data and first numerical data from the sensitive data if the data type of the sensitive data is a specified type, wherein the specified type indicates that the sensitive data needs to retain text characteristics; an encryption subunit, configured to perform MD5 encryption on the non-numerical data to obtain encrypted data; a mapping subunit, configured to map the encrypted data according to a preset encoding table to obtain corresponding second numerical data; and a first desensitization subunit, configured to perform desensitization processing on the first numerical data to obtain first desensitized data, and perform desensitization processing on the second numerical data to retain data characteristics to obtain second desensitized data.

[0071] Optionally, the first desensitization subunit is further configured to: scale the plaintext data corresponding to the second numerical data according to the overall expectation to obtain scaled data; extract the frequency domain features, time domain features, and convolutional transformation features of the scaled data to obtain a feature combination; and map the second numerical data based on the feature combination to obtain the second desensitized data of the second numerical data.

[0072] Optionally, the desensitization unit includes: a positioning subunit for locating the character position of the sensitive word in the sensitive data; a second extraction subunit for extracting M characters before the reference position and N characters after the reference position in the sensitive data, where M and N are both positive integers greater than 0; and a second desensitization subunit for replacing the M characters and the N characters with preset characters to obtain desensitized data of the sensitive data.

[0073] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0074] Example 3 Embodiments of the present invention also provide a storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when running.

[0075] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps: S1, obtain the original plaintext data from the data source; S2, Identify sensitive data in the original plaintext data; S3, parse the data type and data length of the sensitive data, wherein the data type is used to characterize whether the sensitive data needs to retain text characteristics; S4, perform desensitization processing on the sensitive data according to the data type and data length.

[0076] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0077] Embodiments of the present invention also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0078] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0079] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program: S1, obtain the original plaintext data from the data source; S2, Identify sensitive data in the original plaintext data; S3, parse the data type and data length of the sensitive data, wherein the data type is used to characterize whether the sensitive data needs to retain text characteristics; S4, perform desensitization processing on the sensitive data according to the data type and data length.

[0080] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0081] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0082] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0083] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0084] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0085] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0086] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0087] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data anonymization method that preserves data characteristics, characterized in that, include: Obtain the raw plaintext data from the data source; Identify sensitive data in the original plaintext data; The data type and length of the sensitive data are parsed, wherein the data type is used to characterize whether the sensitive data needs to retain text characteristics; The sensitive data is desensitized according to the data type and data length; The process of desensitizing the sensitive data based on the data type and data length includes: calculating the data length of the sensitive data and / or calculating the text length of the annotation text of the sensitive data; counting the number of sensitive words in the sensitive data; and calculating the length of all sensitive words; performing full or partial desensitization on the sensitive data; wherein, performing full desensitization on the sensitive data includes: if the data type of the sensitive data is a specified type, extracting non-numerical data and a first numerical data from the sensitive data, wherein the specified type indicates that the sensitive data needs to retain textual characteristics; performing MD5 encryption on the non-numerical data to obtain encrypted data; and then... The encrypted data is mapped according to a preset encoding table to obtain the corresponding second numerical data; the first numerical data is de-identified to obtain first de-identified data, and the second numerical data is de-identified while retaining its data characteristics to obtain second de-identified data; wherein, the de-identification process of the second numerical data while retaining its data characteristics to obtain the second de-identified data includes: scaling the plaintext data corresponding to the second numerical data according to the overall expectation to obtain scaled data; extracting the frequency domain features, time domain features, and convolutional transformation features of the scaled data to obtain a feature combination; and mapping the second numerical data based on the feature combination to obtain the second de-identified data of the second numerical data.

2. The method according to claim 1, characterized in that, Identifying sensitive data in the original plaintext data includes: Obtain the table structure information and sampled data of the original plaintext data, wherein the table structure information is the descriptive information of the data structure of the original plaintext data; A double-array trie tree is used to search for sensitive words in the table structure information and the sampled data. The double-array trie tree includes a first trie tree and a second trie tree. The first trie tree is used to perform a depth-first search on the table structure information and the sampled data, and the second trie tree is used to perform a breadth-first search on the table structure information and the sampled data. If the table structure information and the sampled data contain sensitive words, extract the sensitive data at the location of the sensitive words.

3. The method according to claim 2, characterized in that, After searching the table structure information and the sampled data using a double-array trie tree, the method further includes: If the table structure information and the sampled data do not contain sensitive words, a fixed rule template is invoked. The fixed rule template integrates multiple user-defined functions (UDFs), each of which is used to identify a sensitive data type. The various UDFs are used to match each column of the original plaintext data respectively; If a target column containing sensitive data is matched, the target column data is extracted from the original plaintext data and identified as sensitive data.

4. The method according to claim 1, characterized in that, Desensitizing the sensitive data, either completely or partially, includes: If the data length is less than a first length threshold and the number of sensitive words is greater than a first quantity threshold, all sensitive data is anonymized; if the data length is greater than the first length threshold and less than a second length threshold, and the number of sensitive words is greater than the second quantity threshold, all sensitive data is anonymized; if the text length is less than the first length threshold and the number of sensitive words is greater than the second quantity threshold, all sensitive data is anonymized; if the data length is greater than the second length threshold, the length ratio of the sensitive word length to the data length is calculated; if the length ratio is lower than a proportion threshold, the sensitive data is partially anonymized; if the length ratio is higher than a proportion threshold, all sensitive data is anonymized.

5. The method according to claim 1, characterized in that, Local desensitization of the sensitive data includes: Locate the character position of sensitive words in the sensitive data; Using the character position as the reference position, extract M characters before the reference position and N characters after the reference position from the sensitive data, where M and N are both positive integers greater than 0; The M characters and N characters are replaced with preset characters to obtain the desensitized data of the sensitive data.

6. A data desensitization device that preserves data characteristics, characterized in that, include: The acquisition module is used to acquire the original plaintext data from the data source; The identification module is used to identify sensitive data in the original plaintext data; The parsing module is used to parse the data type and data length of the sensitive data, wherein the data type is used to characterize whether the sensitive data needs to retain text characteristics; The data anonymization module is used to anonymize the sensitive data according to the data type and data length. The desensitization module is further configured to: calculate the data length of the sensitive data and / or calculate the text length of the annotation text of the sensitive data; count the number of sensitive words in the sensitive data; and calculate the length of all sensitive words; perform full or partial desensitization on the sensitive data; wherein, performing full desensitization on the sensitive data includes: if the data type of the sensitive data is a specified type, extracting non-numerical data and a first numerical data from the sensitive data, wherein the specified type indicates that the sensitive data needs to retain textual characteristics; performing MD5 encryption on the non-numerical data to obtain encrypted data; and encoding the encrypted data according to a preset encoding table. Mapping is performed to obtain the corresponding second numerical data; the first numerical data is de-identified to obtain first de-identified data; the second numerical data is de-identified while retaining its data characteristics to obtain second de-identified data; wherein, the de-identification process of the second numerical data while retaining its data characteristics to obtain the second de-identified data includes: scaling the plaintext data corresponding to the second numerical data according to the overall expectation to obtain scaled data; extracting the frequency domain features, time domain features, and convolutional transformation features of the scaled data to obtain a feature combination; and mapping the second numerical data based on the feature combination to obtain the second de-identified data of the second numerical data.

7. A storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the method described in any one of claims 1 to 5 when it is run.

8. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the method as described in any one of claims 1 to 5.

Images