Sensitive data full-link desensitization method and system
By using fingerprint information to build policy mapping relationships in a microservice architecture and propagating de-identification policies across services, the problem of existing sensitive data de-identification methods being unable to balance accuracy and efficiency is solved, achieving accurate and rapid de-identification of sensitive data across the entire chain.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHEJIANG FINGARD TECH CO LTD
- Filing Date
- 2026-05-28
- Publication Date
- 2026-06-23
AI Technical Summary
Existing technologies for sensitive data desensitization in microservice architectures cannot balance accuracy and efficiency. Reliance on rule-based identification leads to high maintenance costs, cannot identify dynamically generated sensitive data, and cannot provide mandatory protection from the system's underlying layer.
By determining the de-identification strategy at the initial stage of sensitive data entering the system, using fingerprint information to build a strategy mapping relationship, and propagating the de-identification strategy across services, consistency across the entire link is ensured. A hash algorithm is used to calculate fingerprint information and perform matching and replacement before log output to reduce computational overhead.
It achieves accurate desensitization of sensitive data across the entire data chain, reduces the risk of plaintext leakage of sensitive data, improves desensitization efficiency, avoids inconsistencies in desensitization results caused by independent judgment of service nodes, and reduces computational overhead.
Smart Images
Figure CN122263176A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of sensitive data desensitization technology, and in particular to a method and system for end-to-end sensitive data desensitization. Background Technology
[0002] With the widespread adoption of microservice architecture and distributed systems, system logs have become a core data carrier for system operation and maintenance, troubleshooting, and security auditing. However, in actual development, developers often unintentionally output data containing sensitive user information during the logging process, such as mobile phone numbers, ID card numbers, bank card numbers, email addresses, and other personal privacy data.
[0003] Existing technologies typically achieve log anonymization in the following ways: log scanning based on regular expressions, which performs pattern matching and replacement on log content; log filtering based on rule engines, which completes identification and anonymization by configuring rules for sensitive fields; and centralized anonymization processing in log collection systems such as ELK and Fluentd.
[0004] However, the above methods have the following technical drawbacks: they are highly dependent on rule recognition, the rule maintenance cost is high, and sensitive data is easily missed; they require scanning and matching the entire log content at runtime, resulting in significant performance overhead; they cannot effectively identify sensitive data dynamically generated during business operations, and the coverage is limited; they rely on developers to actively follow development specifications, and cannot achieve mandatory protection of de-identification capabilities from the system's underlying layer. Summary of the Invention
[0005] This application addresses the technical problem of existing sensitive data de-identification methods failing to balance accuracy and efficiency. It provides a method and system for end-to-end sensitive data de-identification. By determining the de-identification strategy and associating it with fingerprint information at the initial stage when sensitive data enters the system, the de-identification strategy is transformed from a full scan after log output to a matching and replacement before output, improving de-identification efficiency. Simultaneously, the de-identification strategy is propagated across services along with the fingerprint information throughout the entire process, ensuring that the same sensitive data is replaced using a consistent de-identification strategy throughout the entire process, thus guaranteeing the accuracy of de-identification.
[0006] To achieve the aforementioned technical objectives, this application provides a technical solution: a method for end-to-end sensitive data desensitization, comprising the following steps: extracting data fields with preset sensitive markers from business data as sensitive data; calculating fingerprint information of the sensitive data according to a preset fingerprint construction strategy; determining a desensitization strategy based on the preset sensitive markers of the sensitive data; establishing a mapping relationship between fingerprint information and the desensitization strategy; generating a sensitive context using the sensitive data, fingerprint information, and mapping relationship; in the business chain, responding to downstream service call requests, calling the sensitive context of the upstream service, and synchronously transmitting the fingerprint information and mapping relationship in the sensitive context to the downstream service; generating a sensitive context for the downstream service by combining the sensitive data, fingerprint information, and mapping relationship corresponding to the downstream service's business data with the fingerprint information and mapping relationship of the upstream service; and responding to log output function calls, desensitizing the generated original log string according to the current service's sensitive context, and outputting desensitized log information.
[0007] Furthermore, the process of calculating fingerprint information of sensitive data according to a preset fingerprint construction strategy, determining desensitization strategy according to preset sensitive tags of sensitive data, and establishing a mapping relationship between fingerprint information and desensitization strategy includes: calculating fingerprint information of sensitive data using a hash algorithm, determining desensitization strategy of sensitive data according to the strategy attributes of preset sensitive tags, and mapping fingerprint information corresponding to the same sensitive data to desensitization strategy to establish a mapping relationship between fingerprint information and desensitization strategy.
[0008] Furthermore, the step of responding to a downstream service call request by invoking the sensitive context of the upstream service and synchronously transmitting the fingerprint information and mapping relationship in the sensitive context to the downstream service includes: responding to a downstream service call request by the upstream service by invoking the sensitive context of the upstream service, encapsulating the fingerprint information, mapping relationship and corresponding data signature information in the sensitive context into a sensitive context token, and transmitting the sensitive context token to the downstream service.
[0009] Furthermore, the process of generating a sensitive context for a downstream service by combining the sensitive data, fingerprint information, and mapping relationship corresponding to the downstream service business data with the fingerprint information and mapping relationship of the upstream service includes: in the downstream service, in response to the receipt of a sensitive context token, recalculating the signature of the fingerprint information and mapping relationship in the sensitive context token to obtain a calculated signature, comparing the calculated signature with the data signature information in the sensitive context token, and determining whether to combine the fingerprint information and mapping relationship of the upstream service to generate a sensitive context for the downstream service based on the comparison result.
[0010] Furthermore, the step of determining whether to combine the fingerprint information and mapping relationship of the upstream service to generate the sensitive context of the downstream service according to the comparison result includes: if the comparison result is consistent, then the fingerprint information of the downstream service is calculated based on the sensitive data of the downstream service business data, the fingerprint information of the downstream service is matched with the fingerprint information of the upstream service, the mapping relationship of the downstream service is selected based on the matching result, and the fingerprint information and mapping relationship of the downstream service are combined with the fingerprint information and mapping relationship of the upstream service to generate the sensitive context of the downstream service; if the comparison result is inconsistent, then the sensitive context of the downstream service is generated based on the sensitive data, fingerprint information and mapping relationship of the downstream service.
[0011] Furthermore, in response to the log output function call, the generated original log string is de-identified based on the sensitive context of the current service, and the de-identified log information is output. This includes: in response to the log output function call, determining the current service based on the unique identifier of the request link corresponding to the current thread, and retrieving the sensitive context of the current service; determining the string to be de-identified in the original log string based on the multi-level candidate matching index structure and string matching logic, and replacing the string to be de-identified according to the de-identification strategy in the sensitive context of the current service.
[0012] Furthermore, the step of determining the string to be de-identified in the original log string based on the multi-level candidate matching index structure and string matching logic, and replacing the string to be de-identified according to the de-identification strategy in the current service's sensitive context, includes: constructing a multi-level candidate matching index structure based on the first character of the sensitive data in the sensitive context and the length of the original data; performing a single linear scan of the original log string using the multi-level candidate matching index structure; determining the string to be de-identified based on the longest string matching logic and the result of the single linear scan; and retrieving the de-identification strategy in the sensitive context to replace the string to be de-identified.
[0013] Furthermore, the step of determining the string to be desensitized based on the longest string matching logic and the result of a single linear scan includes: extracting the string to be determined sequentially from the result of a single linear scan in descending order of the original data length according to the longest string matching logic; calculating the character hash cumulative verification value of the string to be determined at each extraction, and comparing the character hash cumulative verification value with the sensitive hash cumulative verification value corresponding to the first character of the same sensitive data and the original data length; If the comparison matches, the string to be determined is then identified as the string to be de-identified.
[0014] Furthermore, it also includes storing the mapping relationship between the fingerprint information of sensitive data and the de-identification strategy in a global cache; wherein, the global cache is a two-layer mapping structure, using the combination of service identifier, node identifier and request link identifier as the first-level index key, and fingerprint information as the second-level index key.
[0015] Another technical solution provided in this application is a sensitive data end-to-end desensitization system for implementing the above-mentioned method, comprising: a sensitive data capture module for extracting data fields with preset sensitive tags from business data as sensitive data; a sensitive context management module for calculating fingerprint information of sensitive data according to a preset fingerprint construction strategy, determining a desensitization strategy according to the preset sensitive tags of sensitive data, establishing a mapping relationship between fingerprint information and desensitization strategy, and generating a sensitive context with sensitive data, fingerprint information, and mapping relationship; a sensitive context propagation module for responding to downstream service call requests, calling the sensitive context of upstream services, and synchronously transmitting the fingerprint information and mapping relationship in the sensitive context to downstream services; and a log desensitization engine for responding to log output function calls, replacing the generated original log string according to the sensitive context of the current service, and outputting desensitized log information.
[0016] The beneficial effects of this application are as follows: 1. Each time business data enters the business system, data fields with preset sensitive markers are extracted to obtain sensitive data. Fingerprint calculation and desensitization strategies are then performed on this sensitive data. A mapping relationship is established between the fingerprint information of the sensitive data and the desensitization strategy, eliminating the need for plaintext transmission of sensitive data across services in the microservice business chain, thus reducing the risk of plaintext leakage. Simultaneously, in the business chain, downstream services only need to extract sensitive data, generate fingerprint information, and establish mapping relationships for their own business data, without needing to traverse all business data for repeated identification, reducing computational overhead. Consequently, log output does not require a full regular expression scan for identification; the sensitive context of the current service can be directly extracted for accurate desensitization of the original logs, balancing data security and desensitization efficiency throughout the entire business system chain.
[0017] 2. First, match the fingerprint information of downstream services with the fingerprint information of upstream services. If consistent fingerprint information exists, the mapping relationship is not reconstructed. If inconsistent fingerprint information does not exist, the mapping relationship of downstream services is constructed based on the fingerprint information of downstream services. This avoids the repeated construction of mapping relationships, achieves consistency of the end-to-end de-identification strategy, and ensures that the same sensitive data is processed using a consistent de-identification strategy throughout the entire microservice call chain, regardless of how many service nodes it flows through. This avoids inconsistencies in de-identification results caused by independent judgments and differences in strategy configurations among service nodes.
[0018] 3. By constructing a multi-level candidate matching index structure based on the first character of sensitive data and the length of the original data, the scanning process is transformed into a single linear scan. When the scanned character matches the first character, the string to be de-identified is directly extracted based on the longest length, without the need for complex pattern recognition or regular expression backtracking of the entire log text, thus improving the de-identification efficiency. Attached Figure Description
[0019] Figure 1 This is a flowchart illustrating a method for end-to-end desensitization of sensitive data in one embodiment of this application.
[0020] Figure 2 This is a schematic diagram illustrating the generation process of sensitive contexts for downstream services in another embodiment of this application. Detailed Implementation
[0021] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description of this application is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely one preferred embodiment of this application and are only used to explain this application. They do not limit the scope of protection of this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0022] like Figure 1 As shown in the first embodiment of this application, the sensitive data end-to-end desensitization method includes the following steps: Extract data fields with preset sensitive tags from business data as sensitive data, calculate fingerprint information of sensitive data according to preset fingerprint construction strategy, determine de-identification strategy according to preset sensitive tags of sensitive data, establish mapping relationship between fingerprint information and de-identification strategy, and generate sensitive context with sensitive data, fingerprint information and mapping relationship; In the business chain, in response to the downstream service call request, the sensitive context of the upstream service is invoked, and the fingerprint information and mapping relationship in the sensitive context are synchronously transmitted to the downstream service. The sensitive context of downstream services is generated by combining the sensitive data, fingerprint information, and mapping relationship corresponding to the downstream service business data with the fingerprint information and mapping relationship of the upstream service. In response to the log output function call, the generated raw log string is de-identified according to the current service's sensitivity context, and the de-identified log information is output.
[0023] In this embodiment, the "end-to-end" refers to the context propagation chain of the same business request within the microservice call chain, including the request entry point, in-service processing, cross-service calls, downstream service reception, and log output. Each time business data enters the business system, data fields with preset sensitive markers are extracted to obtain sensitive data. Fingerprint calculation and de-identification strategies are then performed on the sensitive data. A mapping relationship is established between the fingerprint information of the sensitive data and the de-identification strategy, ensuring that plaintext sensitive data does not need to be transmitted across services in the microservice business chain, reducing the risk of plaintext sensitive data leakage. Simultaneously, in the business chain, downstream services only need to extract sensitive data, generate fingerprint information, and establish mapping relationships for their own business data, without needing to traverse all business data for repeated identification, reducing computational overhead. Consequently, log output does not require a full regular expression scan for identification; the sensitive context of the current service can be directly extracted to accurately de-identify the original logs, balancing data security and de-identification efficiency throughout the entire business system chain.
[0024] Specifically, developers define sensitive data fields in the object model using preset sensitive tag annotations, eliminating the need to write additional desensitization logic or configure desensitization rules. Preset sensitive tags are annotations or configuration file tags pre-configured on business data fields to identify them as sensitive data fields; different types of sensitive data correspond to different preset sensitive tags.
[0025] Therefore, developers only need to declare field attributes by pre-setting sensitive tags, without having to write any desensitization logic or configure desensitization rules in the business code. The capture, fingerprint calculation, context management and replacement of sensitive data are all automatically completed by the underlying framework, avoiding the risk of human-caused sensitive data leakage due to developer negligence or omission of rule configuration, while reducing R&D and maintenance costs.
[0026] In this embodiment, the preset sensitive tag is @Sensitive.
[0027] The fingerprint information of sensitive data is calculated according to a preset fingerprint construction strategy. A de-identification strategy is determined based on preset sensitive markers for the sensitive data. The mapping relationship between fingerprint information and the de-identification strategy is established, including: The fingerprint information of sensitive data is calculated using a hash algorithm, and the desensitization strategy of sensitive data is determined based on the policy attributes of the preset sensitive tags. The fingerprint information corresponding to the same sensitive data is mapped to the de-identification strategy to establish a mapping relationship between fingerprint information and de-identification strategy.
[0028] During the deserialization, database decryption, or RPC (Remote Procedure Call) request entry stages of business data entering the current service, detect data fields with preset sensitive markers in the business data and extract the corresponding field values as sensitive data.
[0029] In this embodiment, the developer pre-declares the @Sensitive annotation on sensitive fields of the object model. For example, @Sensitive private String phoneNumber indicates that the phoneNumber field is marked as sensitive data and needs to be automatically de-identified. At this time, the extracted sensitive data is registered in the sensitive context manager, which is SensitiveContext.register(value, maskStrategy). This means that the original text (value) of a sensitive data and its corresponding de-identification strategy (maskStrategy) are bound together and registered in the sensitive context of the current request.
[0030] The hash algorithm is an irreversible hash algorithm, such as SHA256. The hash value is used to calculate the hash of the plaintext sensitive data, serving as the fingerprint information. The de-identification strategy is determined by the strategy attribute in the @Sensitive annotation. For example, if the strategy attribute is a mobile phone number, the de-identification strategy is "keep the first 3 and last 4 characters, replace the middle characters with asterisks". At this point, the fingerprint information and de-identification strategy corresponding to the same sensitive data are associated and mapped, and stored in the sensitive context. The sensitive context is implemented based on the thread context (ThreadLocal) and is bound to the lifecycle of the current request, ensuring that it can be accessed within the same request thread.
[0031] Sensitive data is identified at the initial stage of business system processing, and a sensitive context is constructed for cross-business flow. This allows log output to be replaced only based on the established sensitive context, reducing CPU and memory overhead and the time complexity of de-identification operations compared to performing regular expression scanning and pattern matching on the entire log text, thus improving business flow efficiency. Simultaneously, through the cross-service synchronous transmission of fingerprint information and mapping relationships, the same sensitive data is processed using a consistent de-identification strategy throughout the entire business chain, regardless of the number of service nodes it passes through. This avoids discrepancies in de-identification results caused by independent judgments and inconsistent rules among different service nodes.
[0032] In this embodiment, when business data initially enters the service node, the initial construction of the aforementioned sensitive context needs to be performed. The difference is that when the upstream service needs to call the downstream service, the sensitive context of the upstream service is called, and the fingerprint information and mapping relationship in the sensitive context are synchronously transmitted to the downstream service. At this time, the transmission does not contain the plaintext of sensitive data.
[0033] Specifically, when an upstream service initiates an RPC (Remote Procedure Call) / HTTP (Hypertext Transfer Protocol) call, the system automatically encapsulates the fingerprint information and mapping relationships in the current sensitive context into a Sensitive Context Token (SCT), and passes it to the downstream service via HTTP Header or RPC Metadata to ensure data transmission during calls from thread pools, message queues (MQ), Dubbo (a distributed service framework), etc. The structure of the Sensitive Context Token (SCT) includes: service identifier (service ID), server node identifier (server node ID), unique identifier of the request chain (TraceID), an array of sensitive data objects (maskData), and data signature information (signature). The fingerprint information and mapping relationships are stored in the array of sensitive data objects.
[0034] At this point, in response to downstream service call requests, the sensitive context of the upstream service is invoked, and the fingerprint information and mapping relationship in the sensitive context are synchronously transmitted to the downstream service, including: In response to the upstream service's call request to the downstream service, the sensitive context of the upstream service is invoked, and the fingerprint information, mapping relationship and its corresponding data signature information in the sensitive context are encapsulated into a sensitive context token, which is then passed to the downstream service.
[0035] Correspondingly, the sensitive data, fingerprint information, and mapping relationships corresponding to the downstream service business data are combined with the fingerprint information and mapping relationships of the upstream services to generate the sensitive context of the downstream services, including: In the downstream service, in response to the receipt of the sensitive context token, the fingerprint information and mapping relationship in the sensitive context token are re-signed to obtain the calculated signature. The calculated signature is compared with the data signature information in the sensitive context token. Based on the comparison result, it is determined whether to combine the fingerprint information and mapping relationship of the upstream service to generate the sensitive context of the downstream service.
[0036] In this embodiment, while transmitting the fingerprint information and mapping relationship of the upstream service to the downstream service, the corresponding data signature information is also transmitted to the downstream service simultaneously. The downstream service performs signature verification and comparison to ensure the accuracy of the data transmitted subsequently.
[0037] Understandably, data signature information can be calculated using digital signature algorithms based on fingerprint information and mapping relationships. By pre-storing keys in the node's secure environment, downstream services can determine whether the fingerprint information and mapping relationships transmitted by the upstream service are forged or corrupted based on the digital signature information, ensuring the secure and trustworthy transmission of context.
[0038] like Figure 2As shown, determining whether to combine the fingerprint information and mapping relationship of the upstream service to generate the sensitive context of the downstream service based on the comparison results includes: If the comparison results are consistent, the fingerprint information of the downstream service is calculated based on the sensitive data of the downstream service business data. The fingerprint information of the downstream service is matched with the fingerprint information of the upstream service. The mapping relationship of the downstream service is selected based on the matching result. The sensitive context of the downstream service is generated by combining the fingerprint information and mapping relationship of the downstream service with the fingerprint information and mapping relationship of the upstream service. If the comparison results are inconsistent, a sensitive context for the downstream service is generated based on the sensitive data, fingerprint information, and mapping relationship of the downstream service.
[0039] If the signature verification does not match, it indicates that the content of the sensitive context token transmitted by the upstream service may have been tampered with or the source may be untrustworthy. In this case, the downstream service will no longer combine the fingerprint information and mapping relationship of the upstream service to prevent untrustworthy information from entering the system and ensure the security of the information transmitted in the context.
[0040] In this embodiment, if the signature verification matches, fingerprint information is calculated based on the sensitive data in the downstream service business data. At this point, instead of immediately constructing a mapping relationship for the downstream services, the fingerprint information of the downstream services is first matched with the fingerprint information of the upstream services. If matching fingerprint information exists, the mapping relationship is not reconstructed for that fingerprint information. If matching fingerprint information does not exist, the mapping relationship for the downstream services is constructed based on the fingerprint information of the downstream services. This avoids the repeated construction of mapping relationships, ensuring consistency of the end-to-end desensitization strategy. It guarantees that the same sensitive data is processed using a consistent desensitization strategy throughout the entire microservice call chain, regardless of how many service nodes it passes through, avoiding inconsistencies in desensitization results caused by independent judgments and policy configuration differences among service nodes.
[0041] Understandably, if the signature verification is inconsistent, the sensitive context of the downstream service will be generated directly based on the business data of the downstream service according to the sensitive context generation steps, without merging it with the sensitive context passed from the upstream service.
[0042] In response to the log output function call, the generated raw log string is de-identified based on the current service's sensitivity context, and the de-identified log information output includes: In response to the log output function call, determine the current service based on the unique identifier of the request chain corresponding to the current thread, and retrieve the sensitive context of the current service; Based on the multi-level candidate matching index structure and string matching logic, the string to be de-identified in the original log string is determined, and the string to be de-identified is replaced according to the de-identification strategy in the current service's sensitive context.
[0043] In this embodiment, during log output, matching is performed only on the sensitive context of the current service, without conducting full log pattern recognition or regular expression scanning. Combined with the rapid location capabilities of the multi-level candidate matching index structure, the computational complexity of log replacement is reduced. Simultaneously, since the de-identification is based on the complete sensitive context bound to the unique identifier of the request chain, the fingerprint information of any sensitive data captured and registered throughout the entire lifecycle of the current service request is stored in the sensitive context, along with its corresponding de-identification strategy mapping relationship. Therefore, regardless of the form or location of the sensitive data in the log text, it can be locked by the fingerprint matching logic and de-identified, avoiding the risk of missed detection due to incomplete regular expression rules or the inability to identify dynamic data. Furthermore, by using the unique identifier of the request chain corresponding to the current thread to obtain the current sensitive context, in high-concurrency, multi-request intertwined scenarios, the de-identification logic of each service request is completely and naturally isolated, without interference, avoiding cross-request de-identification strategy pollution and improving the accuracy of log de-identification for each service request.
[0044] Specifically, the sensitive context also stores the first character of the sensitive data and the original data length. At this point, based on the multi-level candidate matching index structure and string matching logic, the string to be de-identified in the original log string is determined. The string to be de-identified is then replaced according to the de-identification strategy in the current service's sensitive context, including: A multi-level candidate matching index structure is constructed based on the first character of the sensitive data in the sensitive context and the length of the original data. A single linear scan of the original log string is performed using a multi-level candidate matching index structure; The string to be de-identified is determined based on the longest string matching logic and the result of a single linear scan, and the de-identification strategy in the sensitive context is invoked to replace the string to be de-identified.
[0045] In this embodiment, the sensitive data context corresponding to the current thread is first obtained. A sensitive data first-character index table is constructed based on the first character of the sensitive data in the context. Then, the original data is sorted in descending order by length to construct a length index, forming a multi-level candidate matching index structure. During a single linear scan, when a scanned character matches the first-character index, that character is included in the single linear scan result. The string to be de-identified is determined based on the longest original data length corresponding to the first-character index. A de-identification strategy is then determined based on this first-character index and the longest original data length, and the string to be de-identified is replaced.
[0046] In this embodiment, a multi-level candidate matching index structure is constructed based on the first character of the sensitive data and the length of the original data. This transforms the scanning process into a single linear scan. When the scanned character matches the first character, the string to be de-identified is directly extracted based on the longest length, eliminating the need for complex pattern recognition or regular expression backtracking of the entire log text and improving de-identification efficiency. Furthermore, the construction of the multi-level candidate matching index structure is based on the first character of the sensitive data and the length of the original data in the sensitive context. The de-identification strategy in the sensitive context is determined by the preset sensitive markers of the sensitive data and remains consistent throughout the entire propagation chain. Therefore, regardless of which node the current service is at in the chain, the same de-identification strategy is used to replace sensitive data with the same first character and the same length, ensuring the consistency of the de-identification results throughout the entire chain at the execution level.
[0047] In other cases, the sensitive context stores fingerprint information, mapping relationships, the first character of sensitive data, the original data length, the accumulated checksum of the sensitive hash, and the last usage time. In this case, the strings to be de-identified are determined based on the longest string matching logic and the results of a single linear scan, including: Based on the longest string matching logic, the strings to be determined are extracted sequentially from the results of a single linear scan, sorted in descending order of the original data length. During each extraction, the character hash cumulative check value of the string to be determined is calculated, and the character hash cumulative check value is compared with the sensitive hash cumulative check value corresponding to the first character of the same sensitive data and the original data length. If the comparison matches, the string to be determined is then identified as the string to be de-identified.
[0048] In this scenario, a quick coarse screening is achieved through the first character index, medium-granularity filtering is achieved through the original data length, and the check value achieves the final precise matching. This avoids sensitive data with a shorter original data length being incorrectly subjected to the desensitization strategy of longer data with the same first character, thus preventing missed desensitization or incorrect desensitization morphology.
[0049] Understandably, when the cumulative checksum of the character hash matches the cumulative checksum of the sensitive hash, the matching of the next string length will not proceed.
[0050] In this embodiment, the character hash cumulative check value is the sum of the hash codes of the first and last characters of the string to be determined, and the sensitive hash cumulative check value is the sum of the hash codes of the first and last characters of the sensitive data.
[0051] In other embodiments, determining the string to be de-identified based on the longest string matching logic and the result of a single linear scan includes: Based on the longest string matching logic, the strings to be determined are extracted sequentially from the results of a single linear scan, sorted in descending order of the original data length. During each extraction, the character hash cumulative check value and the complete string hash value of the string to be determined are calculated. The character hash cumulative check value and the complete string hash value are then compared with the sensitive hash cumulative check value and fingerprint information corresponding to the first character of the same sensitive data, the original data length, and the same sensitive data. If the comparison matches, the string to be determined is then identified as the string to be de-identified.
[0052] In this embodiment, the string to be de-identified is determined by simultaneously determining whether the hash codes of the first and last characters of the string to be identified, the hash value of the complete string, and the fingerprint information are the same as the cumulative verification value of the sensitive hash of the corresponding sensitive data, thus ensuring the accuracy of the matching of the string to be de-identified.
[0053] As a second embodiment of this application, the sensitive data end-to-end desensitization method further includes: The mapping relationship between the fingerprint information of sensitive data and the de-identification strategy is stored in a global cache; The global cache has a two-layer mapping structure, using a combination of service identifier, node identifier and request link identifier as the first-level index key, and fingerprint information as the second-level index key.
[0054] In this embodiment, the query namespace is limited by a first-level index key, preventing the same fingerprint information from being cross-hit across different services, nodes, or request chains, thus avoiding pollution from desensitization strategies across services, nodes, or requests. Furthermore, this embodiment supports periodically and passively evicting old data according to a preset expiration time, and also supports checking the current capacity each time data is stored, and actively cleaning up data when a certain threshold is exceeded.
[0055] Specifically, in response to a data storage request, the global cache performs signature verification on the data storage request. After successful verification, the mapping relationship to be stored is stored in the global cache according to the corresponding first-level index key and second-level index key, and the storage time is recorded. In response to a data retrieval request, the global cache performs signature verification on the data retrieval request. After successful verification, the corresponding mapping relationship is queried according to the first-level index key and second-level index key in the request, and the access time is updated. The global cache monitors the amount of data. When the amount of data exceeds a preset threshold, some data is evicted according to the storage time order in a first-in-first-out (FIFO) strategy until the amount of data drops to half of the preset threshold. The global cache records operation logs for data storage, data retrieval, and eviction operations. The operation logs include the requester's network address, operation time, and operation type.
[0056] It is understandable that the mapping relationship includes at least the de-identification strategy corresponding to the fingerprint information.
[0057] By employing a two-layer mapping structure combined with a time-based eviction mechanism, the amount of data in the global cache is consistently kept within a preset threshold. When the data volume exceeds the threshold, automatic eviction is triggered, clearing the cache to half the threshold level using a first-in, first-out (FIFO) strategy. This prevents memory overflow caused by the continuous accumulation of sensitive data fingerprints and mapping relationships, ensuring stable system operation. Furthermore, using sensitive data fingerprints as secondary index keys, the same sensitive data only needs to establish a mapping relationship and be cached upon its first appearance within the same request chain. Subsequent occurrences directly hit the cache, avoiding redundant operations such as repeatedly calculating fingerprints, determining strategies, and establishing mappings for the same sensitive data within the chain, thus reducing computational overhead.
[0058] In this embodiment, a combination of service identifier, node identifier, and request chain identifier is used as the primary index key, achieving natural isolation of cached data between different services, nodes, and requests. Simultaneously, the unified storage of mapping relationships in the global cache, in conjunction with the sensitive context token mechanism, ensures consistency in end-to-end data anonymization.
[0059] As a third embodiment of this application, a sensitive data end-to-end de-identification system includes: The sensitive data capture module is used to extract data fields with preset sensitive tags from business data as sensitive data; The sensitive context management module is used to calculate the fingerprint information of sensitive data according to the preset fingerprint construction strategy, determine the de-identification strategy according to the preset sensitive tags of sensitive data, establish the mapping relationship between fingerprint information and de-identification strategy, and generate a sensitive context with sensitive data, fingerprint information and mapping relationship. The sensitive context propagation module is used to respond to downstream service call requests, invoke the sensitive context of the upstream service, and synchronously transmit the fingerprint information and mapping relationship in the sensitive context to the downstream service. The log desensitization engine is used to respond to log output function calls, desensitize the generated raw log strings according to the current service's sensitivity context, and output desensitized log information.
[0060] In this embodiment, the sensitive data capture layer is deployed at the initial stage when sensitive data enters the business system, such as data deserialization, database decryption, and RPC request entry points. It connects to the sensitive context manager and is used to register the captured sensitive data to the sensitive context of the current request. The Sensitive Context Manager, implemented based on thread context, connects to the sensitive data capture layer to receive sensitive data registration, connects to the global sensitive data cache to complete the cache registration of the mapping relationship between sensitive data fingerprints and de-identification policies, and connects to the log de-identification engine to provide a set of sensitive data for log replacement.
[0061] Sensitive data is cached globally, connected to the sensitive context manager to receive storage requests for mapping relationships, and connected to the log de-identification engine to respond to pull requests for mapping relationships. At the same time, it provides data support for cross-service context propagation in a microservice architecture.
[0062] The log desensitization engine is deployed in the log output stage. It is implemented by rewriting the output function of the log framework, connecting with the sensitive context manager to obtain the sensitive data context of the current request, and connecting with the global sensitive data cache to query the mapping relationship. It completes batch and accurate desensitization and replacement of log content before log printing.
[0063] In cross-service call scenarios, the upstream service's sensitive context manager transmits fingerprint information and mapping relationships to the downstream service through a sensitive context token (SCT). After receiving the information, the downstream service's sensitive context manager restores the sensitive context and merges it with the newly added sensitive data obtained by the downstream service's own sensitive data capture layer. This data is then updated in the global sensitive data cache and the SCT and continues to be passed on.
[0064] The specific embodiments described above are preferred embodiments of the sensitive data full-link desensitization method and system of this application, and are not intended to limit the specific implementation scope of this application. The scope of this application includes but is not limited to the specific embodiments described above. All equivalent changes made in accordance with the shape and structure of this application are within the protection scope of this application.
Claims
1. A method for desensitizing sensitive data across the entire data chain, characterized by: Includes the following steps: Extract data fields with preset sensitive tags from business data as sensitive data, calculate fingerprint information of sensitive data according to preset fingerprint construction strategy, determine de-identification strategy according to preset sensitive tags of sensitive data, establish mapping relationship between fingerprint information and de-identification strategy, and generate sensitive context with sensitive data, fingerprint information and mapping relationship; In the business chain, in response to the downstream service call request, the sensitive context of the upstream service is invoked, and the fingerprint information and mapping relationship in the sensitive context are synchronously transmitted to the downstream service. The sensitive context of downstream services is generated by combining the sensitive data, fingerprint information, and mapping relationship corresponding to the downstream service business data with the fingerprint information and mapping relationship of the upstream service. In response to the log output function call, the generated raw log string is de-identified according to the current service's sensitivity context, and the de-identified log information is output.
2. The sensitive data end-to-end desensitization method as described in claim 1, characterized in that: The fingerprint information of sensitive data is calculated according to a preset fingerprint construction strategy. A de-identification strategy is determined based on preset sensitive markers for the sensitive data. The mapping relationship between fingerprint information and the de-identification strategy is established, including: The fingerprint information of sensitive data is calculated using a hash algorithm, and the desensitization strategy of sensitive data is determined based on the policy attributes of the preset sensitive tags. The fingerprint information corresponding to the same sensitive data is mapped to the de-identification strategy to establish a mapping relationship between fingerprint information and de-identification strategy.
3. The sensitive data end-to-end desensitization method as described in claim 1, characterized in that: The step of responding to a downstream service call request by invoking the sensitive context of the upstream service and synchronously transmitting the fingerprint information and mapping relationship of the sensitive context to the downstream service includes: In response to the upstream service's call request to the downstream service, the sensitive context of the upstream service is invoked, and the fingerprint information, mapping relationship and its corresponding data signature information in the sensitive context are encapsulated into a sensitive context token, which is then passed to the downstream service.
4. The sensitive data end-to-end desensitization method as described in claim 3, characterized in that: The sensitive data, fingerprint information, and mapping relationships corresponding to the downstream service business data, combined with the fingerprint information and mapping relationships of the upstream service, generate the sensitive context of the downstream service, including: In the downstream service, in response to the receipt of the sensitive context token, the fingerprint information and mapping relationship in the sensitive context token are re-signed to obtain the calculated signature. The calculated signature is compared with the data signature information in the sensitive context token. Based on the comparison result, it is determined whether to combine the fingerprint information and mapping relationship of the upstream service to generate the sensitive context of the downstream service.
5. The sensitive data end-to-end desensitization method as described in claim 4, characterized in that: The step of determining whether to combine the fingerprint information and mapping relationship of the upstream service to generate the sensitive context of the downstream service based on the comparison results includes: If the comparison results are consistent, the fingerprint information of the downstream service is calculated based on the sensitive data of the downstream service business data. The fingerprint information of the downstream service is matched with the fingerprint information of the upstream service. The mapping relationship of the downstream service is selected based on the matching result. The sensitive context of the downstream service is generated by combining the fingerprint information and mapping relationship of the downstream service with the fingerprint information and mapping relationship of the upstream service. If the comparison results are inconsistent, a sensitive context for the downstream service is generated based on the sensitive data, fingerprint information, and mapping relationship of the downstream service.
6. The sensitive data end-to-end desensitization method as described in claim 1, characterized in that: In response to the log output function call, the generated original log string is de-identified according to the current service's sensitivity context, and the de-identified log information is output, including: In response to the log output function call, determine the current service based on the unique identifier of the request chain corresponding to the current thread, and retrieve the sensitive context of the current service; Based on the multi-level candidate matching index structure and string matching logic, the string to be de-identified in the original log string is determined, and the string to be de-identified is replaced according to the de-identification strategy in the current service's sensitive context.
7. The sensitive data end-to-end desensitization method as described in claim 6, characterized in that: The process of determining the string to be de-identified in the original log string based on the multi-level candidate matching index structure and string matching logic, and replacing the string to be de-identified according to the de-identification strategy in the current service's sensitive context, includes: A multi-level candidate matching index structure is constructed based on the first character of the sensitive data in the sensitive context and the length of the original data. A single linear scan of the original log string is performed using a multi-level candidate matching index structure; The string to be de-identified is determined based on the longest string matching logic and the result of a single linear scan, and the de-identification strategy in the sensitive context is invoked to replace the string to be de-identified.
8. The sensitive data end-to-end desensitization method as described in claim 7, characterized in that: The process of determining the string to be de-identified based on the longest string matching logic and the result of a single linear scan includes: Based on the longest string matching logic, the strings to be determined are extracted sequentially from the results of a single linear scan, sorted in descending order of the original data length. During each extraction, the character hash cumulative check value of the string to be determined is calculated, and the character hash cumulative check value is compared with the sensitive hash cumulative check value corresponding to the first character of the same sensitive data and the original data length. If the comparison matches, the string to be determined is then identified as the string to be de-identified.
9. The sensitive data end-to-end desensitization method as described in claim 1, characterized in that: Also includes: The mapping relationship between the fingerprint information of sensitive data and the de-identification strategy is stored in a global cache; The global cache has a two-layer mapping structure, using a combination of service identifier, node identifier and request link identifier as the first-level index key, and fingerprint information as the second-level index key.
10. A sensitive data end-to-end desensitization system, used to implement the method as described in any one of claims 1 to 9, characterized in that: include: The sensitive data capture module is used to extract data fields with preset sensitive tags from business data as sensitive data; The sensitive context management module is used to calculate the fingerprint information of sensitive data according to the preset fingerprint construction strategy, determine the de-identification strategy according to the preset sensitive tags of sensitive data, establish the mapping relationship between fingerprint information and de-identification strategy, and generate a sensitive context with sensitive data, fingerprint information and mapping relationship. The sensitive context propagation module is used to respond to downstream service call requests, invoke the sensitive context of the upstream service, and synchronously transmit the fingerprint information and mapping relationship in the sensitive context to the downstream service. The log desensitization engine is used to respond to log output function calls, desensitize the generated raw log strings according to the current service's sensitivity context, and output desensitized log information.