Character type data generalization desensitization method and device based on dynamic programming, medium and product

Abstract

The invention discloses a character type data generalization desensitization method and device based on dynamic programming, a medium and a product, and relates to the technical field of privacy data protection. The method comprises the steps that parameters are initialized, and a relation matrix is created; establishing a data reverse index; constructing a relation matrix by retrieving an inverted index; calculating a passing closure, and constructing an equivalence class set; reconstructing and filtering an equivalence class under the constraint of a suppression number; generalizing the equivalence class based on dynamic programming; and constructing a generalization result set based on the equivalence generalization result. According to the method, the longest common substring between the data is calculated on the basis of word segmentation, so that the accuracy of the data can be kept to the greatest extent; based on the character type data similarity and the longest common substring, the data does not need to contain a hierarchical relationship; reconstructing an equivalence class by gradually reducing a similarity threshold value until a suppression number constraint is met, and ensuring that the suppression number does not exceed an allowable maximum suppression number; data priori knowledge is not needed, the generalization process is controlled by parameters, and human intervention is not needed.

Description

Technical Field

[0001] This application relates to the field of privacy data protection technology, and in particular to a method, device, medium and product for generalizing and de-identifying character data based on dynamic programming. Background Technology

[0002] With the development and application of technologies such as computers and mobile internet, the issue of privacy leaks involved in the collection and publication of personal information has increasingly attracted people's attention. To achieve a balance between the usability and privacy of published data, data anonymization is necessary to protect personal privacy information. For ease of research, personal attribute information is divided into identifying attributes, quasi-identifying attributes, sensitive attributes, and non-sensitive attributes. Identifying attributes are those that can uniquely identify the object corresponding to a record; quasi-identifying attributes are those that can re-identify the object corresponding to a record by connecting to an external data source; sensitive attributes contain sensitive information about the object; and non-sensitive attributes are those that do not re-identify the object or only contain non-sensitive information.

[0003] For character-based quasi-identifier attributes, common k-anonymity enhancement methods often employ a method of manually specifying the generalization level (called the domain generalization level). This involves dividing the character data into multiple regions according to manually determined rules, and gradually omitting different regions to form the domain generalization level. However, manually specifying the domain generalization level is not conducive to maintaining the accuracy of the generalization results, and it has high requirements for data quality. If the data is not standardized enough and cannot accurately match the domain generalization level, it will lead to an increase in the suppression number, further affecting the generalization effect. Furthermore, the data needs to contain a clear hierarchical relationship, such as dividing addresses into multiple regions according to administrative divisions to constitute the domain generalization level. Summary of the Invention

[0004] The purpose of this application is to provide a character-based data generalization de-identification method, device, medium, and product based on dynamic programming, which can maintain the accuracy of the data to the greatest extent and does not require the data itself to contain hierarchical relationships.

[0005] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a character-based data generalization de-identification method based on dynamic programming, including: Obtain the target dataset; Each data item in the target dataset is assigned a unique number to obtain a numbered dataset; the numbered dataset includes data and its corresponding number. An initial relation matrix is ​​created based on the numbered dataset; the row and column numbers of the initial relation matrix correspond one-to-one with the numbering. An inverted index is created for the data in the numbered dataset; the inverted index contains the numbers of the data. Set all values ​​in the initial relation matrix to 0; for each row of the initial relation matrix, using the data in the corresponding numbered dataset as the retrieval basis, retrieve the first k data entries with text similarity greater than a preset similarity threshold from the inverted index, and set the corresponding values ​​in the initial relation matrix to 1 to obtain the updated relation matrix; The transitive closure is calculated based on the updated relation matrix, and the interrelated data are aggregated to construct a set of equivalence classes. The sum of the sizes of equivalence classes less than k in the set of equivalence classes is used to obtain the current suppression number; Determine if the current suppression count is less than or equal to the preset suppression budget; if not, subtract the similarity threshold adjustment factor from the similarity threshold, update the similarity threshold, and return "set all position values ​​of the initial relation matrix to 0; for each row of the initial relation matrix, use the data in the corresponding numbered dataset of that row as the retrieval basis, retrieve the first k data with text similarity greater than the preset similarity threshold from the inverted index, and set the corresponding position values ​​in the initial relation matrix to 1 to obtain the updated relation matrix"; if yes, remove equivalence classes less than k from the equivalence class set to obtain the filtered equivalence class set; For each equivalence class in the set of filtered equivalence classes, the longest common substring calculation method based on dynamic programming is used to merge all data in the equivalence class one by one, generating a generalized string representing the equivalence class, and obtaining the equivalence generalization result set. Using each generalized string in the equivalent generalization result set, replace the original values ​​of all data corresponding to the equivalence class to obtain the generalized desensitized dataset.

[0006] Optionally, a longest common substring calculation method based on dynamic programming is used to merge all data within an equivalence class one by one, generating a generalized string representing the equivalence class, thus obtaining an equivalence generalization result set, specifically including: Use the first data in the equivalence class as the generalized string; Select the next data in the equivalence class as the current data to be processed; The generalized string and the current data to be processed are segmented into words respectively to obtain two word lists; Based on the two word lists, a dynamic programming table is constructed; the dynamic programming table is used to calculate the longest common subsequence between the two word lists. Backtrack the dynamic programming table and update the generalized string based on the word matching results; Determine whether the current data to be processed is the last data in the equivalence class; if yes, determine the updated generalized string as the final generalized string of the equivalence class; if no, return to the step of "selecting the next data in the equivalence class as the current data to be processed".

[0007] Optionally, a dynamic programming table is constructed based on the two lexical lists, specifically including: Create a dynamic programming table to number the lexical units in the two lexical lists starting from 1; the number of rows and columns in the dynamic programming table corresponds to the length of the two lexical lists plus 1, and the rows and columns in the dynamic programming table are numbered starting from 0, with the row number and column number starting from 1 corresponding to the lexical number respectively; Set all positions in the dynamic programming table to 0; Starting from the first row and first column of the dynamic programming table, traverse and calculate the value of each position in row priority order; if the two words corresponding to the current position are the same, then set the value of the current position to the value of the position above and to the left of it plus 1, otherwise, set the value of the current position to the largest value between the values ​​of the positions above and to the left of it.

[0008] Optionally, the dynamic programming table is backtracked, and the generalized string is updated based on the matching of lexical terms, specifically including: Initialize a temporary string to be empty, and set the current position to the bottom right corner of the dynamic programming table; Determine whether the two tokens corresponding to the current position are the same; if they are the same, insert the token at the beginning of the temporary string and take the position above and to the left of the current position as the current position; if they are not the same, if the beginning of the temporary string does not start with a wildcard, insert a wildcard at the beginning of the temporary string and take the position with the larger value between the position above and to the left of the current position as the current position. Until the current position moves to the 0th row or 0th column of the dynamic programming table, the temporary string obtained after backtracking is used as the updated generalized string.

[0009] Optionally, the wildcard is an asterisk, used to represent any word.

[0010] Optionally, the transitive closure is calculated based on the updated relation matrix, and the interrelated data are aggregated to construct a set of equivalence classes, specifically including: The transitive closure of the updated relation matrix is ​​calculated using depth-first search, breadth-first search, or the Warshall algorithm, and the interrelated data are aggregated to construct a set of equivalence classes.

[0011] Optionally, an inverted index is created for the data in the numbered dataset, specifically including: Use the ElasticSearch search engine to create an inverted index for the data in the numbered dataset.

[0012] In a second aspect, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the character data generalization desensitization method based on dynamic programming as described above.

[0013] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the character data generalization desensitization method based on dynamic programming described above.

[0014] Fourthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the character data generalization desensitization method based on dynamic programming described above.

[0015] According to the specific embodiments provided in this application, the following technical effects are disclosed: This application provides a character-based data generalization de-identification method, device, medium, and product based on dynamic programming. The method includes: acquiring a target dataset; assigning a unique number to each piece of data in the target dataset to obtain a numbered dataset; the numbered dataset includes data and corresponding numbers; creating an initial relation matrix based on the numbered dataset; the row and column numbers of the initial relation matrix correspond one-to-one with the numbers; establishing an inverted index for the data in the numbered dataset; the inverted index contains the numbers of the data; setting all values ​​in the initial relation matrix to 0; for each row of the initial relation matrix, using the data in the corresponding numbered dataset as the retrieval basis, retrieving the first k pieces of data with a text similarity greater than a preset similarity threshold from the inverted index, and setting the corresponding values ​​in the initial relation matrix to 1 to obtain an updated relation matrix; calculating the transitive closure based on the updated relation matrix, and aggregating the interrelated data to construct an equivalence class set; and counting the size of equivalence classes less than k in the equivalence class set. The sum of the smaller values ​​yields the current suppression count; it is then determined whether the current suppression count is less than or equal to the preset suppression budget; if not, the similarity threshold is updated by subtracting the similarity threshold adjustment factor from the similarity threshold, and the result is returned: "Set all values ​​in the initial relation matrix to 0; for each row of the initial relation matrix, using the data in the corresponding numbered dataset as the retrieval basis, retrieve the first k data entries with text similarity greater than the preset similarity threshold from the inverted index, and set the corresponding values ​​in the initial relation matrix to 1 to obtain the updated relation matrix"; if yes, remove equivalence classes less than k from the equivalence class set to obtain a filtered equivalence class set; for each equivalence class in the filtered equivalence class set, the longest common substring calculation method based on dynamic programming is used to merge all data within the equivalence class one by one, generating a generalized string representing the equivalence class, resulting in an equivalence generalization result set; each generalized string in the equivalence generalization result set is used to replace the original values ​​corresponding to all data in the corresponding equivalence class to obtain a generalized desensitized dataset. This application maximizes data accuracy by calculating the longest common substring between data based on word segmentation; it is based on character-type data similarity and the longest common substring, without requiring the data itself to contain hierarchical relationships; it reconstructs equivalence classes by gradually reducing the similarity threshold until the inhibition number constraint is met, ensuring that the inhibition number does not exceed the maximum allowed inhibition number; it does not require prior data knowledge, the generalization process is controlled by parameters, and no human intervention is required. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is an application environment diagram of a character data generalization desensitization method based on dynamic programming in one embodiment of this application.

[0018] Figure 2 This is a flowchart illustrating a character data generalization desensitization method based on dynamic programming, provided in Embodiment 1 of this application.

[0019] Figure 3 This is a flowchart illustrating a character data generalization desensitization method based on dynamic programming, provided in Embodiment 2 of this application.

[0020] Figure 4 This is a schematic diagram of the relationship matrix provided for Embodiment 2 of this application.

[0021] Figure 5 This is a schematic diagram of the equivalence class provided in Embodiment 2 of this application.

[0022] Figure 6 This is a schematic representation of dynamic programming provided in Embodiment 2 of this application.

[0023] Figure 7 This is a schematic diagram of the generalization result of the equivalence class provided in Embodiment 2 of this application.

[0024] Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] The metric for measuring data anonymization is the k-anonymity constraint, which states that for any given record in a dataset, there exist at least k-1 other records that have identical values ​​for all quasi-identification attributes. k-anonymity augmentation involves generalizing and suppressing a dataset to satisfy the k-anonymity constraint, ensuring that each equivalence class is at least k in size. This prevents attackers from identifying sensitive individual information through quasi-identification attributes, thus protecting privacy. An equivalence class is a subset of records with the same quasi-identification attribute value. Generalization involves replacing existing values ​​with more general values ​​to expand equivalence classes and ensure they satisfy the k-anonymity constraint. Suppression involves removing isolated data that cannot be generalized into an equivalence class satisfying the k-anonymity constraint. K-anonymity augmentation typically performs generalization and suppression operations under the constraints of the k-anonymity constraint (k is called the "anonymity factor") and the maximum suppression number (called the "suppression budget").

[0027] For character-type quasi-identifier attributes, common k-anonymity enhancement methods often employ a method of manually specifying the generalization level (called the domain generalization level). This involves dividing the character data into multiple regions according to manually determined rules, and gradually omitting different regions to form the domain generalization level. However, this method of manually specifying the domain generalization level has the following drawbacks: (1) Generalizing to a higher level by hiding specific regions is not conducive to maintaining the accuracy of the generalization results; (2) The data needs to contain obvious hierarchical relationships, such as dividing the address into multiple areas according to administrative divisions, thus forming a domain generalization hierarchy; (3) High requirements for data quality. If the data is not standardized enough and cannot accurately match the domain generalization level, it will lead to an increase in the number of suppressions, which will further affect the generalization effect. (4) Sufficient prior knowledge of the data is required. When the hierarchical structure is complex and the data scale is large, it is difficult to determine the domain generalization level manually.

[0028] This application proposes a character-based data generalization desensitization method based on dynamic programming, comprising the following steps: (1) initializing parameters and creating a relation matrix; (2) establishing an inverted index for the data; (3) constructing a relation matrix by retrieving the inverted index; (4) calculating the transitive closure and constructing a set of equivalence classes; (5) reconstructing and filtering equivalence classes under the constraint of the inhibition number; (6) generalizing the equivalence classes based on dynamic programming; and (7) constructing a generalization result set based on the generalization result of the equivalence classes. This application can maximize the accuracy of the data by calculating the longest common substring between data based on word segmentation; based on the similarity of character-based data and the longest common substring, it does not require the data itself to contain hierarchical relationships; by gradually reducing the similarity threshold to reconstruct equivalence classes until the inhibition number constraint is met, it ensures that the inhibition number does not exceed the maximum allowed inhibition number; no prior knowledge of the data is required, the generalization process is controlled by parameters, and no human intervention is required.

[0029] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0030] The character data generalization desensitization method based on dynamic programming provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be set up independently, integrated into server 104, or placed in the cloud or on another server.

[0031] The terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster composed of multiple servers, or it can be a cloud server.

[0032] Example 1: In one exemplary embodiment, such as Figure 2 As shown, a character-type data generalization de-identification method based on dynamic programming is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps S1 to S10. Wherein: S1. Obtain the target dataset.

[0033] In this embodiment, the parameters are first initialized. The initialization parameters include the similarity threshold (or a preset similarity threshold), the similarity threshold adjustment factor, the anonymity factor, and the suppression budget (or a preset suppression budget).

[0034] S2. Assign a unique number to each data item in the target dataset to obtain a numbered dataset; the numbered dataset includes the data and the corresponding number.

[0035] In this embodiment, the data in the dataset are numbered consecutively starting from 1.

[0036] S3. Create an initial relation matrix based on the numbered dataset; the row numbers and column numbers of the initial relation matrix correspond one-to-one with the numbering.

[0037] S4. Create an inverted index for the data in the numbered dataset; the inverted index contains the data's number.

[0038] In this embodiment, the ElasticSearch search engine is used to create an inverted index for the data in the numbered dataset.

[0039] S5. Set all position values ​​of the initial relation matrix to 0; for each row of the initial relation matrix, based on the data in the corresponding numbered dataset of that row, retrieve the first k data (k is the anonymity factor) with a text similarity greater than a preset similarity threshold from the inverted index, and set the corresponding position value in the initial relation matrix to 1 to obtain the updated relation matrix.

[0040] S6. Calculate the transitive closure based on the updated relation matrix, and aggregate the interrelated data to construct a set of equivalence classes.

[0041] In this embodiment, the transitive closure of the updated relation matrix is ​​calculated using depth-first search, breadth-first search, or the Warshall algorithm, and the interrelated data are aggregated to construct a set of equivalence classes.

[0042] S7. Calculate the sum of the sizes of equivalence classes less than k in the set of equivalence classes to obtain the current suppression number.

[0043] S8. Determine whether the current suppression number is less than or equal to the preset suppression budget; if not, subtract the similarity threshold adjustment factor from the similarity threshold, update the similarity threshold, and return "set all position values ​​of the initial relation matrix to 0; for each row of the initial relation matrix, use the data in the corresponding number dataset of that row as the retrieval basis, retrieve the first k data with text similarity greater than the preset similarity threshold from the inverted index, and set the corresponding position values ​​in the initial relation matrix to 1 to obtain the updated relation matrix"; if yes, remove equivalence classes less than k from the equivalence class set to obtain the filtered equivalence class set.

[0044] S9. For each equivalence class in the set of filtered equivalence classes, the longest common substring calculation method based on dynamic programming is used to merge all data in the equivalence class one by one to generate a generalized string representing the equivalence class, and obtain the equivalence generalization result set.

[0045] In this embodiment, for each equivalence class in the set of filtered equivalence classes, a dynamic programming table is constructed, and then the dynamic programming table is backtracked to obtain the set of equivalence generalization results.

[0046] Specifically: S91. Take the first data in the equivalence class as the generalized string.

[0047] S92. Select the next data in the equivalence class as the current data to be processed.

[0048] S93. Perform word segmentation on the generalized string and the current data to be processed respectively to obtain two word lists.

[0049] S94. Based on the two word lists, construct a dynamic programming table; the dynamic programming table is used to calculate the longest common subsequence between the two word lists.

[0050] In this embodiment, a dynamic programming table is constructed using two word lists, specifically including: S941. Number the lexical units in the two lexical lists starting from 1, and create a dynamic programming table; the number of rows and columns of the dynamic programming table corresponds to the length of the two lexical lists plus 1, and the rows and columns of the dynamic programming table are numbered starting from 0, and the row number and column number starting from 1 correspond to the lexical number respectively.

[0051] S942. Set all positions in the dynamic programming table to 0.

[0052] S943. Starting from the first row and first column of the dynamic programming table, traverse and calculate the value of each position in row priority order; if the two words corresponding to the current position are the same, then set the value of the current position to the value of the position above and to the left of it plus 1, otherwise, set the value of the current position to the largest value among the values ​​of the positions above or to the left of it.

[0053] In this embodiment, starting from the first row and first column of the dynamic programming table, the values ​​of each position are calculated sequentially from left to right and from top to bottom: if the two words corresponding to the current position are the same, the value of the current position is set to the value of the position above and to the left plus 1; otherwise, the value of the current position is set to the largest value among the values ​​of the positions above or to the left.

[0054] S95. Backtrack the dynamic programming table and update the generalized string based on the matching of lexical units.

[0055] In this embodiment, backtracking the dynamic programming table and updating the generalized string specifically includes: S951. Initialize a temporary string to be empty, and set the current position to the lower right corner of the dynamic programming table.

[0056] S952. Determine whether the two tokens corresponding to the current position are the same; if they are the same, insert the token at the beginning of the temporary string and take the position above and to the left of the current position as the current position; if they are not the same, if the beginning of the temporary string does not start with a wildcard, insert a wildcard at the beginning of the temporary string and take the position with the larger value between the position above and to the left of the current position as the current position.

[0057] S953. Until the current position moves to the 0th row or 0th column of the dynamic programming table, the temporary string obtained after backtracking is used as the updated generalized string.

[0058] As a specific implementation method, in this embodiment, backtracking the dynamic programming table and updating the generalized string can be performed according to the following steps: The generic string is initialized to an empty string, and the current position is set to the bottom right corner of the dynamic programming table.

[0059] If the two terms corresponding to the current position are the same, the term is added to the beginning of the generalized string, and the position to the left of the current position is taken as the current position. Otherwise, if the generalized string does not start with a wildcard, the wildcard (e.g., ...) is added to the beginning of the generalized string. Add it to the beginning of the generic string, and take the position with the larger value between the position above and to the left of the current position as the current position; If the current position is already in row 0 or column 0, the step ends; otherwise, proceed to the previous step.

[0060] S96. Determine whether the current data to be processed is the last data in the equivalence class; if yes, determine the updated generalized string as the final generalized string of the equivalence class; if no, return to the step of "selecting the next data in the equivalence class as the current data to be processed".

[0061] S10. Using each generalized string in the equivalent generalization result set, replace the original value corresponding to all data in the corresponding equivalence class to obtain the generalized desensitized dataset.

[0062] This embodiment maximizes data accuracy by calculating the longest common substring between data based on word segmentation; it is based on character-type data similarity and the longest common substring, without requiring the data itself to contain hierarchical relationships; it reconstructs equivalence classes by gradually reducing the similarity threshold until the inhibition number constraint is met, ensuring that the inhibition number does not exceed the maximum allowed inhibition number; it does not require prior data knowledge, the generalization process is controlled by parameters, and no human intervention is required.

[0063] This embodiment preserves data semantics and structural information to the greatest extent while protecting privacy by automatically constructing equivalence classes, dynamically programming to calculate the longest common substring, and replacing the original data with generalized strings. The resulting generalized de-identified dataset has the following characteristics: Privacy protection: satisfies k-anonymity constraints to prevent individual data re-identification; Semantic preservation: retains data commonalities through the longest common substring; Structure preservation: the generalized data can still be used for classification, clustering, and other analyses; Automated processing: no manual definition of the generalization level is required, making it suitable for large-scale data processing.

[0064] In practical applications, the generalized desensitized dataset obtained in this embodiment can be widely used in the following technical fields: I. Medical and Health Data Sharing and Scientific Research Analysis: Application scenarios: Hospitals, disease control centers, and medical research institutions need to share case data for disease research, epidemiological analysis, etc., while protecting patient privacy.

[0065] Application method: Generalize and desensitize character fields such as patient address, hospital visited, and disease name; For example: generalizing "Peking Union Medical College Hospital" to " The hospital generalized "type II diabetes" to " sick ”; Researchers can use the desensitized dataset to perform statistical analyses such as disease distribution and treatment patterns, without being able to trace back to specific individuals.

[0066] II. Financial Risk Control and Anti-Fraud Analysis: Application scenarios: Banks, insurance companies, and internet finance companies need to analyze transaction behavior, address information, etc., to build risk control models without revealing customer identities.

[0067] Application method: Generalize text information such as customer address, merchant name, and transaction remarks; For example: generalizing "Lujiazui, Pudong New Area, Shanghai" to " New District ", generalizing "XX Technology Company" to " company"; The risk control system trains models for abnormal transaction detection and gang fraud identification based on generalized data.

[0068] III. Government Data Openness and Public Service Optimization: Application scenario: Government departments want to open up population, economic, and education data to the public for use in public policy research, urban planning, etc., but need to protect citizens' privacy.

[0069] Application method: De-identify fields such as address, workplace, and educational background in population data; For example: generalizing "Tsinghua University Computer Science Department" to " University Tie"; Open datasets are available to universities, research institutions, and enterprises for social analysis, policy evaluation, and other purposes.

[0070] IV. Intelligent Customer Service and Voice / Text Analysis: Application scenario: Businesses need to analyze text data such as customer service conversations and user feedback to optimize services, but these conversations may contain users' private information.

[0071] Application method: Generalize entities such as addresses, names, and organization names in the dialogue text; For example, generalizing "I want to complain about the Zhongguancun branch of XX Bank" to "I want to complain about..." bank "branch"; The anonymized text can be used for intent recognition, sentiment analysis, and service optimization modeling.

[0072] V. Educational Data Analysis and Student Privacy Protection: Application scenario: Schools or educational institutions may want to analyze students' learning behaviors, grade distribution, etc., but need to protect students' identity information.

[0073] Application method: Generalize based on student's class, home address, school name, etc.; For example: generalizing "Class 1, Grade 12 of Beijing No. 4 High School" to " Senior 3 of high school class"; Educational researchers can use anonymized data to conduct research such as learning behavior analysis and teaching effectiveness evaluation.

[0074] VI. Logistics and Supply Chain Data Collaboration: Application scenario: Logistics companies need to share data such as delivery addresses and warehouse locations with partners to optimize route planning, but they also need to protect customer address privacy.

[0075] Application method: Generalize the delivery address while retaining regional characteristics; For example: generalizing "No. 123, Tiyu West Road, Tianhe District, Guangzhou" to " city Tiyu West Road Number"; Partners can perform regional delivery volume analysis and route optimization based on generalized addresses.

[0076] This anonymization method is applicable to any scenario requiring privacy protection of character-based data while retaining its analytical value. In practical applications, the anonymity factor (k) and suppression budget are adjusted according to business needs to balance privacy protection and data usability; combining it with domain-specific dictionaries or entity recognition tools improves the accuracy of word segmentation and generalization; and when outputting the generalized dataset, an explanation of the anonymization rules can be attached to facilitate subsequent data users' understanding of the data semantics.

[0077] Example 2: In one exemplary embodiment, such as Figure 3 As shown, a character data generalization desensitization method based on dynamic programming is provided, including the following steps: Without loss of generality, let the dataset be: A = [{"key":1,"value":"Nanjing University of Science and Technology"}, {"key":2,"value":"Wuhan University of Technology"}, {"key":3,"value":"Army Engineering University"}, {"key":4,"value":"Naval University of Engineering"}, {"key":5,"value":"Air Force Engineering University"}).

[0078] Step 1: Initialize parameters: Anonymity factor k=2, similarity threshold τ=0.4, similarity threshold adjustment factor δ=0.1, suppression budget α=0.

[0079] Step 2: Create an inverted index: Use ElasticSearch or other search engines to create inverted indexes for your data.

[0080] Step 3: Construct the relationship matrix: For each data entry, retrieve the two most similar data entries from the inverted index, provided their text similarity is not lower than a similarity threshold. Construct a relationship matrix, such as... Figure 4 As shown.

[0081] Step 4: Construct equivalence classes: Use algorithms such as the Warshall algorithm or DFS / BFS traversal method to solve the transitive closure, and obtain the equivalence class set E = {A1, A2}, where the equivalence class A1 = {1, 2} and the equivalence class A2 = {3, 4, 5}, as Figure 5 shown.

[0082] Step 5, reconstruct and filter equivalence classes: The sizes of the two equivalence classes are 2 and 3 respectively, both of which are greater than or equal to the anonymity factor (k = 2). Therefore, the suppression number is 0, meeting the suppression budget constraint.

[0083] Step 6, generalize equivalence classes: Taking the equivalence class A1 as an example, first initialize the generalization string with the data numbered 1. Segment the data numbered 2 and the generalization string respectively to obtain two token lists e1 = <"Nanjing", "University of Science and Technology">, e2 = <"Wuhan", "University of Science and Technology">. Construct a dynamic programming table with these two token lists, as Figure 6 shown.

[0084] Initialize the generalization string as an empty string; Starting from the lower right corner position of the dynamic programming table, that is, the (3, 3) position. Since the corresponding two tokens are both "University of Science and Technology", add "University of Science and Technology" to the head of the generalization string to get a new generalization string "University of Science and Technology", and move to the upper left position, that is, move to the (2, 2) position; (2, 2) The corresponding tokens at the position are both "University of Science and Technology", so add "University of Science and Technology" to the head of the generalization string to get a new generalization string "University of Science and Technology", and move to the upper left position, that is, move to the (1, 1) position; The corresponding tokens at the (1, 1) position are different, so add a wildcard to the head of the generalization string to get a new generalization string "University of Science and Technology", and move to the position with the larger value above and to the left. Since the values of its upper and left positions are both 0, move to any position, such as (0, 1); Since the current position has reached the boundary, the generalization process ends, and the generalization result of the equivalence class is "University of Science and Technology", as Figure 7 shown.

[0085] Perform the same processing on the equivalence class to obtain the generalization result as "Engineering University".

[0086] Step 7, construct the result set: Replace the data within the equivalence class with the equivalence class generalization result to obtain the final generalized data set as: "University of Science and Technology"}, Polytechnic University"}, Engineering University Engineering University Engineering University”}).

[0087] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 8 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media to run. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a character-based data generalization and desensitization method based on dynamic programming.

[0088] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0089] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0090] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0091] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0092] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0093] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0094] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0095] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0096] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A character-based data generalization desensitization method based on dynamic programming, characterized in that, include: Obtain the target dataset; Each data item in the target dataset is assigned a unique number to obtain a numbered dataset. The numbered dataset includes data and corresponding numbers; An initial relation matrix is ​​created based on the numbered dataset; The row and column numbers of the initial relation matrix correspond one-to-one with the numbering; An inverted index is created for the data in the numbered dataset; the inverted index contains the numbers of the data. Set all values ​​in the initial relation matrix to 0; for each row of the initial relation matrix, using the data in the corresponding numbered dataset as the retrieval basis, retrieve the first k data entries with text similarity greater than a preset similarity threshold from the inverted index, and set the corresponding values ​​in the initial relation matrix to 1 to obtain the updated relation matrix; The transitive closure is calculated based on the updated relation matrix, and the interrelated data are aggregated to construct a set of equivalence classes. The sum of the sizes of equivalence classes less than k in the set of equivalence classes is used to obtain the current suppression number; Determine if the current suppression count is less than or equal to the preset suppression budget; if not, subtract the similarity threshold adjustment factor from the similarity threshold, update the similarity threshold, and return "set all position values ​​of the initial relation matrix to 0; for each row of the initial relation matrix, use the data in the corresponding numbered dataset of that row as the retrieval basis, retrieve the first k data with text similarity greater than the preset similarity threshold from the inverted index, and set the corresponding position values ​​in the initial relation matrix to 1 to obtain the updated relation matrix"; if yes, remove equivalence classes less than k from the equivalence class set to obtain the filtered equivalence class set; For each equivalence class in the set of filtered equivalence classes, the longest common substring calculation method based on dynamic programming is used to merge all data in the equivalence class one by one, generating a generalized string representing the equivalence class, and obtaining the equivalence generalization result set. Using each generalized string in the equivalent generalization result set, replace the original values ​​of all data corresponding to the equivalence class to obtain the generalized desensitized dataset.

2. The character data generalization desensitization method based on dynamic programming according to claim 1, characterized in that, A longest common substring calculation method based on dynamic programming is used to merge all data within an equivalence class one by one, generating a generalized string representing the equivalence class, resulting in an equivalence generalization result set, which specifically includes: Use the first data in the equivalence class as the generalized string; Select the next data in the equivalence class as the current data to be processed; The generalized string and the current data to be processed are segmented into words respectively to obtain two word lists; Based on the two word lists, a dynamic programming table is constructed; the dynamic programming table is used to calculate the longest common subsequence between the two word lists. Backtrack the dynamic programming table and update the generalized string based on the word matching results; Determine whether the current data to be processed is the last data in the equivalence class; if yes, determine the updated generalized string as the final generalized string of the equivalence class; if no, return to the step of "select the next data in the equivalence class as the current data to be processed".

3. The character data generalization desensitization method based on dynamic programming according to claim 2, characterized in that, Based on the two lexical lists, a dynamic programming table is constructed, specifically including: Create a dynamic programming table to number the lexical units in the two lexical lists starting from 1; the number of rows and columns in the dynamic programming table corresponds to the length of the two lexical lists plus 1, and the rows and columns in the dynamic programming table are numbered starting from 0, with the row number and column number starting from 1 corresponding to the lexical number respectively; Set all positions in the dynamic programming table to 0; Starting from the first row and first column of the dynamic programming table, traverse and calculate the value of each position in row priority order; if the two words corresponding to the current position are the same, then set the value of the current position to the value of the position above and to the left of it plus 1, otherwise, set the value of the current position to the largest value of the values ​​of the positions above or to the left of it.

4. The character data generalization desensitization method based on dynamic programming according to claim 2, characterized in that, Backtracking the dynamic programming table, and updating the generalized string based on the word matching results, specifically including: Initialize a temporary string to be empty, and set the current position to the bottom right corner of the dynamic programming table; Determine whether the two tokens corresponding to the current position are the same; if they are the same, insert the token at the beginning of the temporary string and take the position above and to the left of the current position as the current position; if they are not the same, if the beginning of the temporary string does not start with a wildcard, insert a wildcard at the beginning of the temporary string and take the position with the larger value between the position above and to the left of the current position as the current position. Until the current position moves to the 0th row or 0th column of the dynamic programming table, the temporary string obtained after backtracking is used as the updated generalized string.

5. The character data generalization desensitization method based on dynamic programming according to claim 4, characterized in that, The wildcard is an asterisk, used to represent any word.

6. The character data generalization desensitization method based on dynamic programming according to claim 1, characterized in that, The transitive closure is calculated based on the updated relation matrix, and the interrelated data are aggregated to construct a set of equivalence classes, specifically including: The transitive closure of the updated relation matrix is ​​calculated using depth-first search, breadth-first search, or the Warshall algorithm, and the interrelated data are aggregated to construct a set of equivalence classes.

7. The character data generalization desensitization method based on dynamic programming according to claim 1, characterized in that, To create an inverted index for the data in the numbered dataset, specifically including: Use the ElasticSearch search engine to create an inverted index for the data in the numbered dataset.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that the processor executes the computer program to implement the character-type data generalization desensitization method based on dynamic programming as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the character data generalization desensitization method based on dynamic programming as described in any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the character data generalization desensitization method based on dynamic programming as described in any one of claims 1-7.

Images