A high-throughput real-time data desensitization method

Abstract

The application provides a high-throughput real-time data desensitization method, relates to the technical field of data processing, and comprises the following steps: acquiring a service data stream and a rule configuration stream; distributing rule objects to the local memories of each computing node, constructing a field path prefix tree, and storing an accurate matching rule list and a regular matching rule list in the nodes of the prefix tree; recursively traversing data records, identifying all fields and field paths that need to be matched; splitting the field path into a to-be-matched path segment sequence, sequentially matching each path segment in the to-be-matched path segment sequence from the root node of the prefix tree, obtaining a target node of a matching path, collecting the regular matching rule lists of each node on the traversal path, and obtaining a candidate regular rule set; acquiring a matching rule; and executing a corresponding desensitization algorithm on a field value according to the desensitization algorithm type in the matching rule to obtain a desensitized data record. The application can guarantee data security compliance and improve the throughput performance of real-time stream processing.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a high-throughput real-time data desensitization method. Background Technology

[0002] With the development of big data and cloud computing technologies, industries such as finance, the Internet of Things, and healthcare are generating massive amounts of real-time data streams. This data often contains a large amount of sensitive personal identification information, such as ID card numbers, bank card numbers, and mobile phone numbers. Under increasingly stringent global privacy regulations, enterprises must anonymize sensitive data during transmission. Traditional offline batch anonymization methods can no longer meet the timeliness requirements of real-time stream computing. Existing technical solutions typically employ two methods: application-layer hard-coding anonymization or database lookup matching anonymization. The hard-coding method directly writes the anonymization rules into the stream computing job logic. If the anonymization rules need to be modified, the entire data stream process must be stopped and restarted, leading to business interruption. The database lookup matching method synchronously queries an external database to obtain the anonymization rules when the real-time computing node processes each piece of data, but the high network I / O latency severely impacts the processing speed of the entire data stream.

[0003] Chinese patent CN117290882A discloses a log desensitization method. This method specifies the types and keys of sensitive fields that may appear in a transaction or payment system through a configuration file. When printing logs, the request parameters are processed and converted to JSON format. The converted data is then iterated through, matching key values ​​during the iteration. If the current key matches a configured sensitive field, desensitization is performed. After identifying the sensitive field type, the corresponding digest rule is called, and the digest result replaces the sensitive field in the log. This patent uses a traversal method to match sensitive fields one by one, which limits matching efficiency when the data structure is complex or the number of sensitive fields is large. Summary of the Invention

[0004] In view of this, the present invention provides a high-throughput real-time data de-identification method, which realizes real-time hot updating of de-identification rules through distributed state mirroring and dual-stream collaborative mechanism, and at the same time uses field path prefix tree indexing technology to reduce the computational overhead of rule matching, thereby improving the throughput performance of real-time stream processing while ensuring data security and compliance.

[0005] The technical solution of this invention is implemented as follows: This invention provides a high-throughput real-time data anonymization method, comprising: S1. Obtain the business data stream and rule configuration stream. The rule object in the rule configuration stream contains field paths, matching type identifiers, regular expression strings, de-identification algorithm types, and priorities. Among them, the matching type identifier of the rule object includes exact matching and regular expression matching. S2. Distribute the rule object to the local memory of each computing node through a broadcast mechanism, build a field path prefix tree on each computing node, split the field path of the rule object into a path segment sequence according to the delimiter, build the nodes of the prefix tree layer by layer according to the path segment sequence, and store the exact matching rule list and the regular expression matching rule list in the nodes of the prefix tree. S3. Recursively traverse the data records in the business data flow to identify all fields that need to be matched and their field paths; S4. Split the field path into a sequence of path segments to be matched according to the delimiter. Starting from the root node of the prefix tree, match each path segment in the sequence of path segments to be matched in turn to obtain the target node of the matching path. During the matching process, collect the list of regular expression matching rules for each node on the traversed path to obtain the candidate regular expression rule set. S5. Based on the target node and the candidate regular expression rule set, and following the principle that exact matching takes precedence over regular expression matching, obtain the matching rule from the exact matching rule list or the candidate regular expression rule set of the target node. S6. Based on the desensitization algorithm type in the matching rules, execute the corresponding desensitization algorithm on the field values ​​to obtain the desensitized data records.

[0006] Preferably, in step S1, the priority of the rule object is an integer value, ranging from 0 to 1000, with a larger value indicating a higher priority.

[0007] Preferably, in step S2, for rule objects with a matching type of exact match, the method for constructing a field path prefix tree includes: Split the field paths of the rule object into an array of path segments according to the delimiter; Starting from the root node of the prefix tree, locate each path segment in the path segment array in turn; Search for the child node corresponding to the path segment in the child node mapping table of the currently processed node; If the child node exists, then the child node will be the next node to be processed. If the child node does not exist, create a new node and insert it into the child node mapping table of the currently processed node; After traversing all path segments, the last node processed is the leaf node of the path for that field. The rule object is added to the exact match rule list of that leaf node, and the exact match rule list is sorted in descending order of priority.

[0008] Preferably, in step S2, for rule objects with a matching type of regular expression matching, the method for constructing a field path prefix tree includes: Extract a fixed prefix from the regular expression string of the rule object. The extraction method is to scan from the beginning of the regular expression string and extract all literal characters up to the first regular expression metacharacter as the fixed prefix. If the fixed prefix is ​​empty, the rule object is added to the regular expression matching rule list of the root node of the prefix tree; If the fixed prefix is ​​not empty, then split the fixed prefix according to the separator to obtain the path segment array; The corresponding node is located layer by layer in the prefix tree by matching the rule object exactly. The rule object is added to the regular expression matching rule list of that node, and the regular expression matching rule list is sorted in descending order of priority.

[0009] Preferably, in step S4, the method for collecting the set of candidate regular rules includes: Initialize the candidate regular expression rule set to an empty list; Add the list of regular expression matching rules for the root node of the prefix tree to the candidate regular expression rule set; Starting from the root node of the prefix tree, for each path segment in the sequence of path segments to be matched, the corresponding child node is searched in the child node mapping table of the current node in turn; If a child node exists, add the list of regular expression matching rules for that child node to the candidate regular expression rule set and move to that child node to continue processing the next path segment; If the child node does not exist, the traversal is terminated. The final candidate regular expression rule set contains regular expression matching rules for all nodes along the traversal path from the root node to the matching termination node.

[0010] Preferably, in step S5, the method for obtaining the matching rules includes: If the sequence of path segments to be matched completely matches the target node in the prefix tree, then check whether the exact matching rule list of the target node is not empty; If the exact match rule list is not empty, then the rule with the highest priority in the list is used as the matching rule. If the list of exact matching rules is empty, then the regular expression matching rules in the candidate regular expression rule set are matched in turn, and the first matching regular expression rule is taken as the matching rule.

[0011] Preferably, when a rule object with a matching type of regular expression is received in step S2, the regular expression compilation method is called to compile the regular expression string of the rule object into a pre-compiled object, a process-level pre-compiled object cache is maintained, and the key-value pair of the rule's unique identifier and the pre-compiled object is stored in the cache; during the regular expression matching process in step S5, the pre-compiled object is directly retrieved from the cache for matching operation.

[0012] Preferably, in step S6, the de-identification algorithm includes masking algorithm, truncation algorithm, hashing algorithm, substitution algorithm, encryption algorithm and generalization algorithm.

[0013] Preferably, in step S3, the method for recursively traversing the data records includes: Define a recursive traversal function, with input parameters including the current data node, the current field path, and a reference to the root node of the prefix tree; Recursively traverse the data records starting from the root node, and determine the type of the current data node: If the current data node is an object type, iterate through all fields of the object, and for each field, concatenate the field name to the current field path to construct a new field path. Then, use the field value as the new current data node to recursively call the traversal function. If the current data node is an array, then traverse all elements of the array, keeping the current field path unchanged, and recursively call the traversal function with the element value as the new current data node. If the current data node is a basic data type or a string type, mark the node as a field to be matched and record its current field path; After the recursive traversal is completed, all fields that need to be matched and their field paths are obtained.

[0014] Preferably, the method further includes: Receive rule change events from the rule configuration stream and update the field path prefix tree according to the change type: For newly added rule events, insert the new rule node into the prefix tree according to step S2; For rule modification events, delete the old rule node and insert the modified new rule node in the prefix tree; For rule deletion events, remove the corresponding rule node from the prefix tree.

[0015] The present invention has the following advantages over the prior art: (1) The present invention distributes the de-identification rules to the local memory of each computing node through a broadcast mechanism. During the data processing, the rule information is read directly from the local memory, avoiding the network communication overhead caused by external database queries or remote service calls, eliminating the network latency bottleneck in the rule acquisition process, and improving the throughput performance of data processing. (2) The present invention adopts a dual-stream collaborative architecture, which separates the business data stream from the rule configuration stream. The rule change event is captured in real time through the broadcast stream mechanism and synchronized to each computing node. When a rule is added, modified or deleted, the rule update can be completed without stopping the operation, avoiding the service interruption caused by rule adjustment in the traditional hard-coding method and ensuring the continuity of business processing. (3) The present invention constructs a field path prefix tree index structure and organizes the de-identification rules hierarchically according to the field path. When matching rules, it is only necessary to locate the target node and collect candidate rules layer by layer along the path. The computational complexity of rule matching is reduced from being proportional to the total number of rules to being proportional to the path depth. It can still maintain stable matching performance when the number of rules increases, thus improving the scalability of the system. (4) The present invention pre-compiles and caches the compiled objects of the regular expression matching type rules, and directly reuses the compiled objects to perform the matching operation in the subsequent matching process, thereby avoiding the computational overhead caused by repeatedly compiling the regular expression and reducing the consumption of processor resources; (5) The present invention sets a priority attribute for the rule object and arranges the rule list in descending order of priority during the rule matching process. When a field matches a rule, the corresponding desensitization algorithm is executed immediately and subsequent matching is terminated, ensuring that each field is desensitized only once, avoiding the problem of data format damage or complete information loss caused by multiple desensitization operations. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a schematic diagram of the dual-stream collaborative architecture of the present invention; Figure 3 This is a schematic diagram of the field path prefix tree index structure of the present invention; Figure 4 This is a schematic diagram of the recursive traversal of data in this invention; Figure 5 This is a schematic diagram of the rule matching process of the present invention. Detailed Implementation

[0018] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0019] like Figure 1As shown, this invention provides a high-throughput real-time data anonymization method, comprising: S1, acquiring business data streams and rule configuration streams, wherein the rule objects in the rule configuration streams include field paths, matching type identifiers, regular expression strings, anonymization algorithm types, and priorities; wherein the matching type identifiers of the rule objects include exact matching and regular expression matching; S2, distributing the rule objects to the local memory of each computing node through a broadcast mechanism, constructing a field path prefix tree on each computing node, splitting the field paths of the rule objects into path segment sequences according to delimiters, building nodes of the prefix tree layer by layer according to the path segment sequences, and storing an exact matching rule list and a regular expression matching rule list in the nodes of the prefix tree; S3, processing the data in the business data streams... S4. Recursively traverse the records to identify all fields that need to be matched and their field paths; S5. Split the field paths into a sequence of path segments to be matched according to the delimiter, and match each path segment in the sequence of path segments to be matched sequentially starting from the root node of the prefix tree to obtain the target node of the matching path. During the matching process, collect the list of regular expression matching rules for each node on the traversed path to obtain a set of candidate regular expression rules; S6. Based on the target node and the set of candidate regular expression rules, and following the principle that exact matching takes precedence over regular expression matching, obtain the matching rule from the list of exact matching rules or the set of candidate regular expression rules for the target node; S7. According to the desensitization algorithm type in the matching rule, execute the corresponding desensitization algorithm on the field value to obtain the desensitized data record.

[0020] In one embodiment of the present invention, step S1 includes: like Figure 2 As shown, a dual-stream collaborative architecture is established, with the system containing two data streams: a business data stream and a rule configuration stream. The business data stream carries real-time business data that needs to be de-identified, and originates from the message queue or data bus of the business system; The rule configuration stream carries the change instructions for the de-identification rules, which originate from the database change log or message queue of the configuration center; When a de-identification rule is added, modified, or deleted, the change event is captured in real time and sent to the rule configuration stream; The data structure of the rule configuration object includes the following fields: a) Unique rule identifier: ruleId, a string type, used to globally and uniquely identify a de-identification rule; b) Target field path: fieldPath, a string type, using dot separators to indicate nesting levels, such as "data.userInfo.idCard"; c) Match type: matchType, an enumeration type, including EXACT (exact match) and REGEX (regular expression match); d) Regular expression string: patternString, a string type, effective when matchType is REGEX, storing the regular expression pattern, such as ".password." or " e) Algorithm type: algorithmType, an enumeration type that specifies the desensitization strategy to be used, including MASK, TRUNCATE, HASH, REPLACE, ENCRYPT, GENERALIZE, etc.; f) Algorithm parameters: algorithmParams, a mapping type that stores the parameters required for algorithm execution, such as the number of prefix and suffix bits to retain in the masking algorithm; g) Rule priority: priority, an integer value ranging from 0 to 1000. The larger the value, the higher the priority, used to resolve conflicts between multiple rules; h) Version number: version, a long integer value that monotonically increases. The version number is incremented by one each time a rule is changed, used to track the version evolution of rules and prevent state rollback.

[0021] In one embodiment of the present invention, step S2 includes: By leveraging the broadcast state mechanism of a distributed computing framework, the rule configuration stream is converted into a broadcast stream; When a new de-identification rule arrives, the distributed computing framework will distribute the rule to all computing nodes; In the `processBroadcastElement` function of each compute node, the following operations are performed: Parse the rule configuration object and extract information from fields such as ruleId, fieldPath, matchType, patternString, algorithmType, algorithmParams, priority, and version; Version control checks read the version number of the stored rule from the broadcast status of the current node, compare the version field of the new rule with the current version number, and only execute the subsequent update operation if the new version number is greater than the current version number; otherwise, the rule change event is discarded to avoid the state rollback caused by rule change events arriving out of order. The parsed rule object is updated in the local BroadcastState. The broadcast state is stored in the local memory of the compute node to ensure that each compute node has a complete copy of the rule. The update of the broadcast state adopts a copy-on-write mechanism to ensure the atomicity of the state update and avoid concurrent read and write conflicts. like Figure 3 As shown, after the rule object is distributed to each computing node, the field path prefix tree index structure is constructed synchronously. The data structure of the prefix tree node is defined as follows: The node path segment identifier (segment), a string type, corresponds to the name of a path segment in the path field; the child node mapping table (children), using a HashMap structure, where the keys are strings representing the child path segment names and the values ​​are references to child node objects; the exact matching rule list (exactRules), using an ArrayList structure, stores all rule objects that exactly match the complete path to this node; the regular expression matching rule list (regexRules), also using an ArrayList structure, stores all regular expression rule objects that match the path prefix to this node; the prefix tree index structure is stored in local memory as part of the broadcast state, with the root node... For empty paths, each node maintains a mapping table pointing to its child nodes, as well as a list of precise rules and regular expression rules for the node's position; For rule objects with an exact match type, the field paths of the rule object are split into an array of path segments according to the delimiter. For example, data.userInfo.idCard is split into data, userInfo, and idCard. Starting from the root node of the prefix tree... Initially, locate each path segment in the path segment array sequentially; for each path segment in the path segment array... ,in L represents the path depth, and the key to look up the children mapping table of the current node is... The child node is selected; if the child node exists, it is set as the new current node; otherwise, a new node object is created, and its segment is set to [value]. Insert the new node into the children mapping table of the current node and set the new node as the new current node; after traversing all path segments, the last node processed is the leaf node of the path in that field. Add the rule object to the exact matching rule list of the leaf node, and sort the exact matching rule list in descending order of priority, maintaining the insertion order if the priorities are the same; For rule objects with a matching type of regular expression matching, a fixed prefix is ​​extracted from the regular expression string patternString of the rule object. The extraction method is to scan from the beginning of the regular expression string and extract all literal characters up to the first regular expression metacharacter as the fixed prefix. Regular expression metacharacters include the period (.) and the asterisk (.). The string contains escape sequences such as question mark (?), plus sign (+), square brackets [ ], curly braces {}, parentheses ( ), vertical bar (|), caret (^), dollar sign ($), backslash (\), etc.; It replaces the escape sequence . in the extracted string with a regular period (.) as a path prefix; for example, the regular expression user: The prefix extracted from email is "user", and the prefix extracted from regular expression ".password" is an empty string; if the prefix is ​​always an empty string, then the rule object is added to the root node of the prefix tree. The `regexRules` list of regular expression matching rules indicates that the rule should attempt to match all fields. If the fixed prefix is ​​not empty, the fixed prefix is ​​split by the delimiter to obtain an array of path segments, and the corresponding node is located layer by layer in the prefix tree according to the method of the exact matching rule object. Add the rule object to the regular expression matching rule list regexRules of this node, and sort the regular expression matching rule list in descending order of priority, and maintain the insertion order if the priorities are the same; Upon receiving a rule object with a matching type of regular expression, a pre-compilation operation of the regular expression is performed. The `Pattern.compile` method is called to compile the `patternString` string in the rule into a `Pattern` object. A process-level local cache `compiledPatterns` is maintained, using a thread-safe `ConcurrentHashMap` structure, with the rule's `ruleId` string as the key and the compiled `Pattern` object as the value. The key-value pairs of `ruleId` and `Pattern` objects are stored in the `compiledPatterns` cache. In subsequent data processing, the compiled `Pattern` object is directly retrieved from the `compiledPatterns` cache by `ruleId`, and the `pattern.matcher` method is called to perform the matching operation, avoiding the computational overhead of repeatedly compiling the regular expression. When a rule update causes a change in the regular expression, the old `Pattern` object is first removed from the `compiledPatterns` cache by `ruleId`, and then the new regular expression is compiled and inserted into the cache, ensuring that the compiled objects in the cache are always synchronized with the current rule state. The regular expression cache is stored in local memory as an auxiliary structure for the broadcast state.

[0022] In one embodiment of the present invention, after step S2, the business data stream and the rule configuration broadcast stream are associated through the connect operator of the distributed computing framework, so that the business data can access the rule information in the broadcast state during processing; in the data element processing function processElement, for each arriving business data record, the prefix tree index structure and regular expression cache are read from the BroadcastState in local memory, without initiating any network requests or external query operations, realizing rule acquisition with zero network latency; the de-identified rules are distributed to the local memory of each computing node through the broadcast mechanism, eliminating the network communication overhead of the traditional database matching method and improving the data processing throughput.

[0023] like Figure 4 As shown, in one embodiment of the present invention, step S3 includes: For structured business data records, a recursive traversal function traverseAndMask is defined to perform a depth-first traversal of the nested structure of the data; The input parameters of the recursive traversal function include the current data node currentNode, the current field path currentPath, and the trieRoot reference to the prefix tree root node; Recursively traverse the data records starting from the root node, and determine the type of the current data node currentNode; If the current data node is an object type, such as JSONObject or Map, check if the object contains a specific marker field, which indicates that the object has been processed. If the marker field exists and its value is true, skip the object and return to the parent recursive call to avoid infinite recursion caused by circular references. If the marker field does not exist or its value is false, iterate through all fields of the object. For each field, append the field name to the current field path to construct a new field path, newPath. If currentPath is an empty string at the root node, newPath is directly fieldName; otherwise, newPath is the result of connecting currentPath and fieldName with a dot (.). Using the field value fieldValue as the new current data node, newPath as the new current field path, and trieRoot as the reference to the root node of the prefix tree, the traversal function is recursively called; after traversing all fields, the call returns to the parent recursive call. If the current data node is an array type, such as JSONArray or List, then all elements of the array are traversed. For each element, the current field path `currentPath` remains unchanged, and the element value `element` is used as the new current data node, with `trieRoot` as the reference to the root node of the prefix tree. The traversal function is called recursively. In the array scenario, the path does not change, that is, all array elements share the parent path. In the rule configuration, the special symbol `[]` is supported to represent any element in the array. For example, the field path `orders[].productName` means that the `productName` field of all order objects in the `orders` array is de-identified. When the prefix tree is constructed, `[]` is stored as a special path segment key. When the prefix tree is traversed, when an array type node is encountered and the next level path segment is `[]`, the subsequent path matching and de-identification operation is performed on all elements of the array. After traversing all elements, the recursive call to the upper level is returned. If the current data node is a basic data type or a string type, such as integer, floating-point, boolean, or string type, then the node is marked as a field to be matched and its current field path, currentPath, is recorded. After the recursive traversal is completed, all fields that need to be matched and their field paths are obtained. When the traverseAndMask function is initially called, currentNode is the root node object of the business data, currentPath is an empty string, and trieRoot is the root node of the prefix tree index. The recursive traversal ensures that all fields at all levels in the data object are processed correctly.

[0024] In one embodiment of the present invention, step S4 includes: For each field in the business data to be de-identified, obtain the complete path `currentPath` of that field. The field path is formed by connecting all levels of names from the data root node to the field using periods (.), for example, `data.userInfo.mobile`. Then, divide the field path `currentPath` into a sequence of path segments to be matched, denoted as... Where L represents the path depth, Indicates the name of the path segment at level i; Initialize the candidate regular expression rule set `candidateRegexRules` as an empty list; start from the root node of the prefix tree. To begin, set the current node to `currentNode`. Add the root node's list of regular expression matching rules (regexRules) to the candidate regular expression rule set (candidateRegexRules), because global regular expression rules need to attempt to match all fields; for each path segment in the sequence of path segments to be matched... ,in Search sequentially in the children's mapping table of the current node (currentNode) for the key... The child node; if the child node exists, the child node is assigned to currentNode, and the list of regular expression matching rules for the child node, regexRules, is added to the candidate regular expression rule set, candidateRegexRules. Then, the process moves to the child node to continue processing the next path segment; if the child node does not exist, the traversal is terminated. After the traversal is complete, currentNode points to the target node of the path to be matched. If the complete path exists in the prefix tree; the final candidate regular expression rule set `candidateRegexRules` contains regular expression matching rules for all nodes on the traversal path from the root node to the matching termination node; through prefix tree indexing, the computational cost of rule matching is optimized from being proportional to the total number of rules R to being proportional to the path depth L and the number of candidate regular expression rules k, where This significantly reduces matching overhead.

[0025] like Figure 5 As shown, in one embodiment of the present invention, step S5 includes: If the path segment sequence to be matched completely matches the target node in the prefix tree, then check whether the exact matching rule list (exactRules) of the target node (currentNode) is not empty; if the exact matching rule list (exactRules) is not empty, then directly return the first rule in the list as the matching result, because the priority of the exact matching rule is higher than that of the regular expression matching rule, and the exactRules list has been sorted in descending order of priority, so the first rule is the exact matching rule with the highest priority. If the exact matching rule list `exactRules` is empty, then the regular expression matching rules in the candidate regular expression rule set `candidateRegexRules` are matched sequentially. For each regular expression rule, the compiled `Pattern` object is retrieved from the `compiledPatterns` cache using `rule.getRuleId()`. The `pattern.matcher(currentPath).matches()` method is called to determine whether the complete field path `currentPath` matches the regular expression. If a match is successful, the rule is immediately returned as the matching result, and the first successfully matched regular expression rule is taken as the matching rule. Since the `candidateRegexRules` set inherits the priority order of the `regexRules` list of each node during the collection process, the first successfully matched rule is the regular expression matching rule with the highest priority. If all regular expression rules do not match, `null` is returned, indicating that the field does not need to be desensitized.

[0026] In one embodiment of the present invention, step S6 includes: Based on the matched rules, the corresponding desensitization strategy execution function is called to desensitize the field values. The desensitization strategy is implemented using the strategy pattern, defining a unified desensitization strategy interface DesensitizationStrategy, which includes the desensitize method. This method takes the original field value originalValue and the algorithm parameters params as input and returns the desensitized field value. The masking algorithm retains the previous value of the field. Position and after The character is replaced with an asterisk or other mask character in the middle; the formula for calculating the masked field value is: Where v represents the original field value, Indicates the length of the field value. and The integer values ​​specified in the algorithm parameters params represent the prefix and suffix lengths to be retained, respectively, and represent a single asterisk character; The TRUNCATE algorithm retains only the first n characters of a field value and deletes the rest; the formula for calculating the truncated field value is... , where n is an integer value specified in the algorithm parameter params, representing the number of characters to retain; The hash algorithm uses a message digest algorithm to perform a one-way hash transformation on the field value, generating a fixed-length hash value; the formula for calculating the hashed field value is as follows: ,in This represents the hash function. The specific hash algorithm type, such as MD5 or SHA256, is specified through the hashAlgorithm field in the algorithm parameter params. The replacement algorithm REPLACE uniformly replaces field values ​​with predefined placeholder strings; the replaced field values ​​are... ,in The placeholder string specified in the algorithm parameter params, such as [SENSITIVE] or six asterisks; The ENCRYPT encryption algorithm uses either a symmetric encryption algorithm or a format-preserving encryption algorithm to encrypt field values; the formula for calculating the encrypted field value is as follows: ,in This indicates the encryption function using key k, and IV represents the initialization vector. The key and initialization vector are passed through the key and iv fields in the algorithm parameters params. The GENERALIZE generalization algorithm generalizes precise numeric values ​​or dates to intervals or ranges; for numeric types, the formula for calculating the generalized field value is... Where v is the original value and g is the generalization granularity, specified by the granularity field in the algorithm parameters params. This indicates rounding down; for example, someone who is 25 years old has... The generalized algorithm yielded an age range of 20 to 30 years old. Since step S5 returns the highest priority matching rule, only the de-identification algorithm corresponding to that rule is executed. Once execution is complete, the process stops immediately, and no further matching or execution of other rules is attempted. This short-circuit mechanism avoids data unavailability issues caused by multiple de-identifications of the same field. For example, executing a masking algorithm to replace numbers with asterisks and then executing a hash algorithm to hash the asterisks would produce a meaningless hash value. The priority mechanism ensures that when multiple rules match the same field path, only the highest priority rule takes effect.

[0027] In one embodiment of the present invention, after the desensitization algorithm is executed in step S6, a metadata object auditLog of the desensitization operation is constructed for end-to-end traceability and compliance auditing. The data structure of the metadata object includes the following fields: The trace identifier, traceId, is a string type that represents a unique identifier for data flow. This identifier comes from a globally unique identifier field pre-carried in the business data record, or a UUID identifier dynamically generated during data processing. The timestamp is a long integer value that records the precise millisecond time of the de-identification operation, expressed in milliseconds since January 1, 1970, 00:00:00 UTC. fieldPath, a string type, is the complete path to the de-identified field of the record, separated by periods (.), for example, data.userInfo.mobile; The rule identifier (ruleId), a string type, is a unique identifier that records the matched de-identification rule; AlgorithmType, a string, records the name of the de-identification algorithm used, such as MASK, HASH, ENCRYPT, etc. The operation success status isSuccess, a boolean type, which records whether the de-identification operation was successfully executed. When an exception occurs during the execution of the de-identification algorithm, the status flag is false, and the status flag is true when the execution is completed normally. Metadata objects are serialized in a structured data format; a dedicated side output tag, auditTag, is defined through the side output stream mechanism of the distributed computing framework; after the de-identification operation is completed in the data element processing function processElement, the ctx.output method is called to send the metadata object to the side output stream; the side output stream is independent of the main output stream of business data, and metadata records can be written separately to the audit log system or a dedicated message queue topic to achieve full-link traceability and compliance auditing of the de-identification operation, while not affecting the processing performance of the business data stream.

[0028] In one embodiment of the present invention, when a rule change event is received in the rule configuration stream, the corresponding status update operation is performed in the broadcast element processing function processBroadcastElement according to the change type. For rule addition events, parse the new rule configuration object, extract all field information, and insert the new rule node into the prefix tree according to step S2; if matchType is EXACT exact match, then locate the prefix tree node layer by layer after splitting by fieldPath, insert the new rule into the exactRules list of the leaf node, and sort it in descending order of priority; if matchType is REGEX regular expression match, then extract the fixed prefix from patternString, locate the prefix tree node by the prefix, insert the new rule into the regexRules list of the node, and sort it in descending order of priority; if the rule contains a regular expression, then call the Pattern.compile method to compile the regular expression, and add the key-value pair of ruleId and Pattern object to the compiledPatterns cache; For rule modification events, parse the modified rule configuration object and extract the ruleId field; locate and delete the old rule in the prefix tree, and search for the target node layer by layer in the prefix tree according to the fieldPath of the old rule. In the exactRules or regexRules list of the target node, find and delete the corresponding rule object according to the ruleId; if the old rule contains a regular expression, delete the old Pattern object from the compiledPatterns cache by ruleId; process the modified new rule as a rule addition event and insert it into the prefix tree and cache. For rule deletion events, parse the rule deletion event and extract the ruleId and fieldPath fields; locate and remove the corresponding rule node in the prefix tree, and the removal method is the same as the deletion steps for rule modification events; delete the corresponding Pattern object from the compiledPatterns cache by ruleId; Insertion, deletion, and modification operations in the prefix tree are all incremental updates, operating only on the affected nodes and rule objects without rebuilding the entire index structure, ensuring efficient state updates. Dynamic hot updates of rules adopt an eventual consistency model, where rule change events are distributed to all computing nodes via broadcast streams, and each node independently updates its local broadcast state. There may be slight time differences in the state updates of different nodes, but eventually the states of all nodes will converge to consistency. This mechanism enables real-time hot updates of de-identification rules. Rule changes are triggered by configuration streams, and jobs can switch de-identification algorithms and adjust de-identification scopes without restarting, avoiding service interruptions caused by rule adjustments in traditional hard-coding methods.

[0029] In one embodiment of the present invention, a state snapshot mechanism based on a distributed computing framework is adopted, in which the broadcast state is periodically persisted to reliable storage along with job checkpoints; the persisted content includes: complete serialized data of the prefix tree index structure, including the segment, children, exactRules, and regexRules fields of all nodes; serialized data of the regular expression cache compiledPatterns, storing the mapping relationship between ruleId and regular expression strings, the Pattern object itself is not serializable, stored in string form, and recompiled during recovery; and the version field of all rule objects, used for version verification after fault recovery; When a job fails and needs recovery, the distributed computing framework restores the broadcast state from the most recent successful checkpoint. It reads the serialized data of the prefix tree index structure from persistent storage, deserializes it to construct the prefix tree object, and restores all nodes and the rule list. It also reads the serialized data from the regular expression cache from persistent storage, iterates through all the mappings between ruleId and regular expression strings, and calls the Pattern.compile method to recompile the regular expressions one by one, rebuilding the compiledPatterns cache. Finally, it restores the version field of the rules to ensure that subsequent rule change events can be correctly verified. This state recovery mechanism ensures that the rule logic remains consistent, without loss or rollback, after failure recovery, improving system reliability.

[0030] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A high-throughput real-time data anonymization method, characterized in that, include: S1. Obtain the business data stream and rule configuration stream. The rule object in the rule configuration stream contains field paths, matching type identifiers, regular expression strings, de-identification algorithm types, and priorities. Among them, the matching type identifier of the rule object includes exact matching and regular expression matching. S2. Distribute the rule object to the local memory of each computing node through a broadcast mechanism, build a field path prefix tree on each computing node, split the field path of the rule object into a path segment sequence according to the delimiter, build the nodes of the prefix tree layer by layer according to the path segment sequence, and store the exact matching rule list and the regular expression matching rule list in the nodes of the prefix tree. For rule objects with a matching type of exact match, the methods for constructing the field path prefix tree include: Split the field paths of the rule object into an array of path segments according to the delimiter; Starting from the root node of the prefix tree, locate each path segment in the path segment array in turn; Search for the child node corresponding to the path segment in the child node mapping table of the currently processed node; If the child node exists, then the child node will be the next node to be processed. If the child node does not exist, create a new node and insert it into the child node mapping table of the currently processed node; After traversing all path segments, the last node to be processed is the leaf node of the path in this field. The rule object is added to the exact match rule list of this leaf node, and the exact match rule list is sorted in descending order of priority. For rule objects with a matching type of regular expression matching, the methods for constructing a field path prefix tree include: Extract a fixed prefix from the regular expression string of the rule object. The extraction method is to scan from the beginning of the regular expression string and extract all literal characters up to the first regular expression metacharacter as the fixed prefix. If the fixed prefix is ​​empty, the rule object is added to the regular expression matching rule list of the root node of the prefix tree; If the fixed prefix is ​​not empty, then split the fixed prefix according to the separator to obtain the path segment array; The corresponding node is located layer by layer in the prefix tree according to the method of exact matching rule objects. The rule object is added to the regular expression matching rule list of that node, and the regular expression matching rule list is sorted in descending order of priority. S3. Recursively traverse the data records in the business data flow to identify all fields that need to be matched and their field paths; S4. Split the field path into a sequence of path segments to be matched according to the delimiter. Starting from the root node of the prefix tree, match each path segment in the sequence of path segments to be matched in turn to obtain the target node of the matching path. During the matching process, collect the list of regular expression matching rules for each node on the traversed path to obtain the candidate regular expression rule set. S5. Based on the target node and the candidate regular expression rule set, and following the principle that exact matching takes precedence over regular expression matching, obtain the matching rule from the exact matching rule list or the candidate regular expression rule set of the target node. S6. Based on the desensitization algorithm type in the matching rules, execute the corresponding desensitization algorithm on the field values ​​to obtain the desensitized data records.

2. The high-throughput real-time data desensitization method according to claim 1, characterized in that, In step S1, the priority of the rule object is an integer value, ranging from 0 to 1000. The larger the value, the higher the priority.

3. The high-throughput real-time data anonymization method according to claim 1, characterized in that, In step S4, the methods for collecting the candidate regularization rule set include: Initialize the candidate regular expression rule set to an empty list; Add the list of regular expression matching rules for the root node of the prefix tree to the candidate regular expression rule set; Starting from the root node of the prefix tree, for each path segment in the sequence of path segments to be matched, the corresponding child node is searched in the child node mapping table of the current node in turn; If a child node exists, add the list of regular expression matching rules for that child node to the candidate regular expression rule set and move to that child node to continue processing the next path segment; If the child node does not exist, the traversal is terminated. The final candidate regular expression rule set contains regular expression matching rules for all nodes along the traversal path from the root node to the matching termination node.

4. The high-throughput real-time data desensitization method according to claim 3, characterized in that, In step S5, the methods for obtaining the matching rules include: If the sequence of path segments to be matched completely matches the target node in the prefix tree, then check whether the exact matching rule list of the target node is not empty; If the exact match rule list is not empty, then the rule with the highest priority in the list is used as the matching rule. If the list of exact matching rules is empty, then the regular expression matching rules in the candidate regular expression rule set are matched in turn, and the first matching regular expression rule is taken as the matching rule.

5. A high-throughput real-time data desensitization method according to claim 4, characterized in that, When a rule object with a matching type of regular expression is received in step S2, the regular expression compilation method is called to compile the regular expression string of the rule object into a pre-compiled object, a process-level pre-compiled object cache is maintained, and the key-value pair of the rule's unique identifier and the pre-compiled object is stored in the cache; In the regular expression matching process of step S5, the pre-compiled object is directly retrieved from the cache for matching.

6. The high-throughput real-time data desensitization method according to claim 1, characterized in that, In step S6, the de-identification algorithms include masking algorithms, truncation algorithms, hashing algorithms, substitution algorithms, encryption algorithms, and generalization algorithms.

7. A high-throughput real-time data desensitization method according to claim 1, characterized in that, In step S3, the methods for recursively traversing the data records include: Define a recursive traversal function, with input parameters including the current data node, the current field path, and a reference to the root node of the prefix tree; Recursively traverse the data records starting from the root node, and determine the type of the current data node: If the current data node is an object type, iterate through all fields of the object, and for each field, concatenate the field name to the current field path to construct a new field path. Then, use the field value as the new current data node to recursively call the traversal function. If the current data node is an array, then traverse all elements of the array, keeping the current field path unchanged, and recursively call the traversal function with the element value as the new current data node. If the current data node is a basic data type or a string type, mark the node as a field to be matched and record its current field path; After the recursive traversal is completed, all fields that need to be matched and their field paths are obtained.

8. A high-throughput real-time data desensitization method according to claim 1, characterized in that, The method further includes: Receive rule change events from the rule configuration stream and update the field path prefix tree according to the change type: For newly added rule events, insert the new rule node into the prefix tree according to step S2; For rule modification events, delete the old rule node and insert the modified new rule node in the prefix tree; For rule deletion events, remove the corresponding rule node from the prefix tree.

Images