A data protection method and system based on field splitting and distributed desensitization

Abstract

The application discloses a data protection method and system based on field splitting and distributed desensitization, and the method comprises the following steps: obtaining an authorized external application, splitting sensitive information into several independent fields, and sending the sensitive fields to corresponding desensitization service nodes respectively, each desensitization service node is independently set and does not interact with any data, the desensitization service node generates a local unique and irreversible mask identifier MID based on the complete randomness of the received field plaintext data by using an incremental algorithm, stores the mapping relationship between the MID and the plaintext data, returns the MID to the external application, the application database of the external system does not store any plaintext data of the sensitive field, and all administrators have no access right to the complete data set of the desensitization service node. The application can realize that even if the application database or all desensitization service nodes are attacked, the complete sensitive information cannot be reconstructed, and the data security is maximally protected on the basis of not increasing the computing power cost.

Description

Technical Field

[0001] This invention belongs to the field of data security technology, and in particular, relates to a data protection method and system based on field splitting and distributed de-identification. Background Technology

[0002] With the rapid digital development of modern society, a large amount of sensitive user data is widely collected and used in fields such as healthcare, finance, e-commerce, and social media. Leaks of this data can cause enormous economic losses and even serious social risks, posing a severe challenge to privacy protection. Traditional data protection schemes have significant risks of data leakage, which can originate from external attacks or internal threats (including malicious or negligent database administrators). This makes it difficult to ensure data security while simultaneously guaranteeing data availability. Furthermore, the emergence of quantum computing also threatens traditional cryptographic-based data anonymization mechanisms, highlighting the need for a secure data protection architecture for the post-quantum era.

[0003] Existing technologies such as database-native data masking (embedding masking algorithms in the database engine), proxy-based data masking (deploying masking at the middleware or API layer), homomorphic encryption (HE), and trusted execution environments (TEE) all face their own limitations. Database-native and proxy-based methods are susceptible to single points of failure; if the database or middleware is compromised, all sensitive data will be exposed. HE allows direct computation on encrypted data without exposing plaintext, providing strong privacy protection, but this also incurs extremely high computational overhead; compared to plaintext processing in real-time analytics tasks, the performance loss is typically over 100 times, making it unsuitable for large-scale production. TEE protects the confidentiality of masking algorithms and data execution by using hardware secure zones, shielding them from host operating system-level attacks, but its high hardware requirements lead to high deployment costs and performance overhead, limiting its scalability in production environments.

[0004] In addition, some recent studies have mentioned encrypting and de-identifying sensitive data in distributed systems, but they mainly focus on performance optimizations such as computational parallelism or multi-tenant isolation, without substantially improving data security. Summary of the Invention

[0005] To address the above problems, the present invention aims to provide a data protection scheme based on field splitting and distributed desensitization. By severing the connection paths of related fields within sensitive information, the complexity of reconstructing sensitive information after a data breach is increased, thereby mitigating internal and external security threats without increasing computing power costs. To achieve the above objective, the present invention discloses the following technical solution: A data protection method based on field splitting and distributed data masking includes: Authorized external applications split the input sensitive information into several independent fields and send the sensitive fields to the corresponding de-identification service nodes for data de-identification. Each de-identification service node is set up independently and there is no data interaction between them. The de-identification service node returns the de-identification result to the external application. Neither the external application nor the administrators of each de-identification service node have the right to access the complete dataset of the de-identification service node.

[0006] Preferably, the data desensitization includes: The de-identification service node receives plaintext data of fields to be de-identified sent completely randomly by external applications, generates a locally unique and irreversible mask identifier MID for the corresponding plaintext data using an incrementing algorithm, calculates the hash address using the hash function XXHash3_64, and stores the mapping relationship between MID and plaintext data.

[0007] Preferably, returning the desensitization results to an external application includes: The MID is returned to the external application without returning the mapping relationship between the MID and the plaintext data. The application database of the external application does not store any plaintext data of sensitive fields.

[0008] Preferably, the administrator's lack of access to the complete dataset of the de-identification service node includes: Administrators do not have the right to access or modify any AppId, MID, plaintext data of sensitive fields, or the mapping relationship between MID and plaintext data.

[0009] Preferably, the mapping relationship between the stored MID and plaintext data includes: Construct a DataHash table that supports mapping plaintext data of sensitive fields to their corresponding MIDs. Calculate the hash value of the original field using XXHash3_64 and use it as the main index of the DataHash table. When a collision occurs, use an auxiliary hash function to calculate the probe step size and store the plaintext data and MID mapping in the DataHash table accordingly. Construct a MidIndexHash table that supports reverse mapping from MID to its corresponding field in plaintext data. Store the MID and its primary index in the DataHash table in the MidIndexHash table and use secondary probing to handle collisions.

[0010] Preferably, each de-identification service node is equipped with an independent remote backup receiver to enable remote asynchronous backup of the data stored on that node.

[0011] Preferably, the data protection method further includes: The external application sends the MID of the sensitive fields contained in the information to be called to the corresponding de-identification service node. The de-identification service node searches for the corresponding plaintext data in the hash table based on the received MID and returns the plaintext data to the external application. The returned plaintext data is neither displayed on the external application's interactive interface nor stored in the external application's application database. It is only sent to other applications according to the instructions of the external application administrator.

[0012] This invention also discloses a data protection system based on field splitting and distributed de-identification, used to implement the above-mentioned data protection method, including: The identity authentication module is used to assign and manage application identifiers (AppId) for external applications; The API service module is used to enable data interaction between external applications and various de-identification service nodes, including AppId, verification request, de-identification request, query request, plaintext data, and MID; The request scheduling module is used to intelligently allocate each request task to the corresponding service module; The masking service module is used to generate a locally unique and irreversible mask identifier (MID) for the plaintext data of the field to be masked. It uses the hash function XXHash3_64 to calculate the hash address and store the mapping relationship between the MID and the plaintext data. The query service module supports bidirectional lookup between MID and plaintext data; The persistence module is used to serialize the mapping between MID and plaintext data, the AppId dictionary, and administrator information to disk; The backup module is used to maintain the incremental backup queue for each hash table, transmit updates to the remote backup receiver via TCP, and receive initialization, incremental, and full backup requests.

[0013] Preferably, the request scheduling module uses Tokio's multi-threaded asynchronous runtime to allocate request tasks.

[0014] Preferably, the backup module adopts a TCP incremental backup strategy, which establishes a TCP connection with the remote backup host for each backup operation, and only serializes and transmits records that have been added or updated since the last backup, and then closes the connection.

[0015] Preferably, the backup module uses a dedicated queue and an asynchronous thread to implement data backup and retransmission. Each key table defines an independent retransmission queue. After successful writing, the backup receiver sends acknowledgment data containing the table name and offset. The task will only be dequeued after receiving valid acknowledgment data. If no valid acknowledgment data is received within the configured timeout period, the dedicated queue or asynchronous thread will trigger retransmission.

[0016] Compared with the prior art, the present invention has at least the following beneficial effects: 1. This invention proposes a privacy protection mechanism for sensitive information. By combining field splitting with distributed de-identified storage, the data structure of sensitive information is decoupled, thereby fundamentally severing the association paths between fields within the information. Since the application database does not store any plaintext data of sensitive fields, even if the database is compromised, no usable sensitive information will be leaked. Each de-identified service node stores only the original data of one sensitive field and its corresponding MID, and there is no data mapping relationship between the sensitive fields corresponding to different de-identified service nodes. Therefore, even if the data of all de-identified service nodes is leaked, attackers cannot reconstruct the complete sensitive information. This invention overcomes the security bottleneck of traditional single-point de-identification protection schemes, significantly enhances the overall system's anti-attack capability and privacy protection strength, while retaining the performance advantages of a distributed architecture.

[0017] 2. This invention minimizes the privileges of all administrators. For example, administrators of external applications can only maintain the external application system and have no right to access the complete dataset of all de-identified service nodes. Administrators of de-identified service nodes are also strictly prohibited from accessing the complete dataset of all de-identified service nodes and can only perform related monitoring and maintenance work on system resources. This invention objectively ensures that even privileged internal personnel cannot view or reconstruct sensitive data across decoupled fields, thereby improving the system's ability to resist internal threats.

[0018] 3. This invention utilizes the complete randomness of the original data sent by external applications and the auto-incrementing algorithm for generating MIDs in the desensitized service nodes to ensure that it is impossible to obtain the mapping relationship between MIDs and original data in each desensitized service node through any algorithm or reasoning method. Even using enumeration, it is impossible to combine complete sensitive information. In particular, when the amount of desensitized data is very large, the combination space grows exponentially. Even with quantum acceleration algorithms, detailed enumeration is still computationally infeasible for actual system scales. Furthermore, verification of the combined information is also infeasible if the application database data is not leaked. Therefore, post-quantum security can be achieved.

[0019] 4. The present invention is equipped with a high-performance distributed data protection system. This system is based on the Rust language, the Tokio asynchronous runtime for performing lightweight concurrent tasks, and the high-performance Rust web framework Actix-Web for building low-latency services to support high-concurrency processing. Its high throughput, low latency and stability have been verified in a real server environment. Compared with HE and TEE, the system of the present invention does not rely on special hardware or complex encryption calculations, has low computational overhead, and is more suitable for practical deployment. Attached Figure Description

[0020] Figure 1 This is an architecture diagram of a data protection method based on field splitting and distributed de-identification in Example 1; Figure 2 This is a flowchart of the data anonymization process for sensitive information in Example 2; Figure 3 This is a flowchart of how de-identified data is accessed in Example 2. Detailed Implementation

[0021] The present invention will be further described below with reference to the accompanying drawings, but this is not intended to limit the present invention in any way. Any modifications or substitutions made based on the teachings of the present invention shall fall within the protection scope of the present invention.

[0022] Example 1 Through analysis of sensitive information, researchers have found that a single field does not constitute meaningful sensitive information. In fact, sensitive or valuable information is usually generated by the combination of two or more fields. For example, a single name or ID number has no value on its own, but the combination of name and ID number can reveal the identity of a specific individual and be used for software registration, financial consumption, etc., leading to the theft of identity information.

[0023] Based on the above facts, this embodiment proposes a data protection method based on field splitting and distributed de-identification, see [link to relevant documentation]. Figure 1 The specific process is as follows:

[0024] 1. Field splitting Inputting complete sensitive information into an authorized external application : , in, Indicates the first A single field, Indicates sensitive information Total number of fields included; Based on sensitive information Its own structural composition, containing sensitive information Split into Independent fields All of the above fields are sensitive fields, or are composed of sensitive and non-sensitive fields; The sensitive information includes, but is not limited to, personal data, medical records, financial transaction information, and e-commerce transaction information; the fields include, but are not limited to, name, age, telephone number, address, ID number, job title, savings, transaction history, financial records, medical history, and physical examination records.

[0025] 2. Distributed desensitization Each sensitive field is assigned an independent and dedicated de-identification service node, and no data interaction occurs between the de-identification service nodes; An external application calls an authorized Web API to send the fields to be de-identified to the corresponding de-identification service node for data de-identification. The de-identification service node receives the fields... For each input plaintext data, an auto-incrementing algorithm is used to generate a locally unique and irreversible mask identifier MID for the corresponding plaintext data. The hash address is calculated using the fast unencrypted hash function XXHash3_64, which is optimized for high-throughput applications. The mapping relationship between MID and plaintext data is stored locally on the de-identification service node.

[0026] In storing the mapping relationship between MID and plaintext data, this embodiment employs two complementary open addressing strategies applied to different core hash tables to effectively resolve potential conflicts: ①DataHash table - double hash The DataHash table is a custom-designed hash table used to support hashes from sensitive fields. Plaintext data is mapped to its corresponding MID using a double hashing method. Each piece of plaintext data first calculates a hash value using XXHash3_64 and uses it as the primary index of the DataHash table. When a collision occurs, an auxiliary hash function is used to calculate the probe step size, and the plaintext data is mapped to the MID and stored in the DataHash table accordingly. This minimizes clustering effects and ensures fast lookup performance, achieving efficient and uniformly distributed storage. ②MidIndexHash table - quadratic probing The MidIndexHash table is used to support reverse mapping from MID to its corresponding field. Since each entry in the DataHash table contains MID information, the MID information is stored together with its main index in the DataHash table in the MidIndexHash table for further indexing, which enables efficient reverse lookup. The MidIndexHash table uses quadratic probing to handle collisions, and the probing step size increases quadratically when a collision occurs, thereby effectively reducing the main clustering and maintaining a low average search latency.

[0027] It is important to note that these two conflict resolution strategies are not combined in the same hash table, but are applied to different tables for different lookup purposes.

[0028] 3. Decoupled storage and backup Each de-identification service node returns the de-identification results to the external application and stores them in its application database. The de-identification results include MIDs but do not include the mapping relationship between MIDs and plaintext data of fields. The application database of the external application stores a set of MIDs, but does not store any plaintext data of sensitive fields or any mapping between plaintext data and MIDs. Restrict all administrators' access to the complete dataset of each de-identified service node. That is, neither the administrator of the external application nor the administrator of the de-identified service node has the right to access or modify any plaintext data of AppId, MID, sensitive fields, or the mapping relationship between MID and plaintext data. Each data masking service node is configured with an independent remote backup receiver to enable remote asynchronous backup of the data stored on that node.

[0029] 4. Accessing sensitive information When an external application needs to access sensitive information, it sends the MID containing the sensitive fields to the corresponding de-identification service node through an authorized Web API. After receiving the MID, the de-identification service node searches for the plaintext data corresponding to the MID in a hash table and returns it to the external application. The external application obtains the complete sensitive information by querying each de-identification service node and reconstructing the returned results. The returned plaintext data is neither displayed on the external application's interface nor stored in the external application's application database; it is only sent to other applications according to the instructions of the external application administrator.

[0030] The data protection method provided in this embodiment has strong privacy protection and anti-information reconstruction capabilities. Even if an external attacker breaches the system... When dealing with a single node, only a portion of the fields stored on those nodes can be recovered. Since each node only stores the data for its corresponding field and there is no cross-node mapping (MID is unique only within its own node), attackers cannot reconstruct the complete sensitive information. Even if an attacker attempts to brute-force all possible combinations of these fields, the exponential combination space makes enumeration computationally infeasible. Therefore, the probability of an attacker correctly reconstructing the complete sensitive information is almost zero. More importantly, attackers cannot actually verify which of the enumerated combinations represent the real sensitive information, because incorrect combinations will produce semantically invalid or meaningless results with no operational value. This fundamentally enhances the system's security. At the same time, this method also minimizes administrator privileges to prevent internal threats, ensuring the overall security integrity of the system.

[0031] Furthermore, the field splitting + distributed desensitization architecture adopted in this method does not rely on cryptographic difficulty, that is, unlike traditional encryption schemes that rely on computational hardness assumptions. Therefore, it is not easily attacked by quantum algorithms. Its combination complexity and the unverifiability of reconstructed records endow it with strong post-quantum security. At the same time, the architecture also maintains real-time performance and system scalability. It can be expanded to desensitization servers for other fields based on changes in requirements. The expansion development is relatively easy and the engineering cycle is very short, and the deployment difficulty is low.

[0032] Example 2 This embodiment provides a specific application of the method in Embodiment 1. In this embodiment, ID card number de-identification server, telephone number de-identification server and address de-identification server are deployed on three independent servers in different physical nodes. A human resources system establishes data connection channels with each of the three de-identification servers through the standardized interfaces provided by each of the three de-identification servers.

[0033] When the human resources system enters employee information (including "name, ID number, phone number, and address"), it first breaks down the employee's sensitive personal information into independent fields: "name," "ID number," "phone number," and "address." "Name" is identified as a non-sensitive field and requires no data masking, while "ID number," "phone number," and "address" are considered sensitive fields and require masking. Then, the human resources system submits a data masking request to the corresponding data masking server based on the type of field requiring masking. Upon receiving the request, the data masking server verifies the validity of the AppId and field type. If the verification is successful, it receives the plaintext data for the corresponding fields such as "ID number," "phone number," and "address," and performs the data masking operation based on preset masking rules, generating a masked result (MID). Finally, the human resources system receives the masking results returned by each data masking server and stores them in the corresponding fields of its application database form.

[0034] The above data anonymization process can be found in [link / reference]. Figure 2 Examples of data stored on the server side for de-identifying ID card numbers are shown in Table 1, examples of data stored on the server side for de-identifying phone numbers are shown in Table 2, examples of data stored on the server side for de-identifying addresses are shown in Table 3, and examples of data stored on the database forms of the human resources system are shown in Table 4.

[0035] Table 1. Example of stored data on the ID card number anonymization server.

[0036] Table 2. Example of stored data on the phone number anonymization server.

[0037] Table 3. Example of stored data on the address-de-identified server.

[0038] Table 4. Example of stored data in the database forms of the human resources system application.

[0039] The human resources system and the insurance company's insurance purchase system are connected through an interface. When purchasing insurance for an employee, the human resources system needs to provide the insurance company with information such as the employee's name, ID number, phone number, and address. On the employee insurance purchase page of the human resources system, business operators can only see the employee's name and the submit button; other fields are hidden. When a business operator enters or selects an employee's name, the human resources system backend first retrieves the corresponding ID number (MID), phone number (MID), and address (MID) from the local application database. Then, based on the field type, it submits a request to the corresponding anonymized server to retrieve the original plaintext. If the AppId and field verification are successful, the anonymized server further obtains the MID to be queried and finds the corresponding original plaintext in a hash table based on preset query rules. The anonymized server then returns the original plaintext to the human resources system. The human resources system reassembles the received original plaintext fields to obtain complete employee information (including "name, ID number, phone number, and address"). Finally, after the business operator clicks the submit button, the human resources system backend directly submits the employee information to the insurance company via an interface. The insurance company, upon obtaining the employee information, can then process the relevant insurance business. Throughout this application scenario, the employee insurance purchase page does not display employee information; that is, the human resources system administrator is not aware of the employee's real and sensitive information.

[0040] The above process for retrieving de-identified data can be found in [link to documentation]. Figure 3 .

[0041] In this embodiment, from the perspective of internal threats, since the database of the external application only stores non-sensitive fields and MID, even if the database administrator maliciously or negligently leaks information, since MID itself has no potential value in the absence of a mapping relationship, illegal actors will not cause sensitive information to be leaked when they obtain this data.

[0042] From the perspective of external threats, since ID card numbers, phone numbers, and addresses are de-identified on different and independent servers, each de-identification server only stores the original plaintext of one field and its MID mapping relationship, and does not store any connection relationship between fields. Therefore, even if an attacker successfully obtains de-identified data by attacking a certain de-identification server, they can only obtain a certain set of fields (such as ID card numbers) and their MID mapping relationship. However, a single field cannot be considered sensitive information. Only when two or more original fields are combined together can they be considered sensitive information, thus preventing the leakage of sensitive information.

[0043] Furthermore, if an attacker breaches multiple de-identification servers and obtains the original plaintext of multiple fields, the attacker still faces the challenge of finding the correct combination of the original plaintext. For example, if each de-identification server contains 10 data entries... 6 If we consider only two original fields (such as ID number + address), then the enumeration count is as high as 10. 12 Furthermore, this number expands exponentially with the increase in the number of fields, making it less likely for an attacker to enumerate sensitive information. Even if an attacker exhaustively enumerates all possible field combinations using a quantum computer, the attacker cannot verify which specific combination is the correct sensitive information because the de-identification server does not store any mapping relationship between fields, and incorrect field combinations are naturally not sensitive information (e.g., Wang Xiaoming + No. 12, Maoer Hutong, Chaoyang District, Beijing). In this case, no sensitive information will be leaked.

[0044] In summary, sensitive data leakage will only occur if multiple de-identified servers and application databases are compromised simultaneously. This would require collusion between database administrators and external attackers, which is an extreme case in practical applications and almost impossible to achieve. This method has achieved the most favorable protection for privacy data under objective conditions.

[0045] Example 3 This embodiment provides a data protection system based on field splitting and distributed data masking to implement the data protection method in Embodiment 1. The data protection system includes: The identity authentication module is used to assign and manage application identifiers (AppId) for external applications; The API service module is used to enable data interaction between external applications and various de-identification service nodes, including AppId, verification request, de-identification request, query request, plaintext data, and MID; The request scheduling module is used to intelligently allocate each request task to the corresponding service module; The masking service module is used to generate a locally unique and irreversible mask identifier (MID) for the plaintext data of the field to be masked. It uses the hash function XXHash3_64 to calculate the hash address and store the mapping relationship between the MID and the plaintext data. The query service module supports bidirectional lookup between MID and plaintext data; The persistence module is used to serialize the mapping between MID and plaintext data, the AppId dictionary, and administrator information to disk; The backup module is used to maintain the incremental backup queue for each hash table, transmit updates to the remote backup receiver via TCP, and receive initialization, incremental, and full backup requests.

[0046] The data protection system uses the Rust programming language, which provides zero-cost abstraction, ownership-based memory safety, and fearless concurrency. It can build security-critical and performance-sensitive systems. Compared with C / C++, Rust eliminates common memory errors while maintaining relatively low efficiency.

[0047] The API service module provides a RESTful API through Actix-Web, supporting JSON-based request parsing, parameter validation, and automatically generated Swagger documentation. Actix-Web is a lightweight asynchronous Rust web framework that uses the Actor model and achieves industry-leading throughput performance.

[0048] The request scheduling module uses Tokio's multi-threaded asynchronous runtime to allocate request tasks. Tokio supports generating lightweight tasks across CPU cores and minimizes context switching overhead, enabling the service to efficiently handle thousands of concurrent requests. To improve submission and consumption efficiency, this system implements a multi-producer single-consumer (MPSC) queue as an internal task pipeline. This design ensures load-balanced parallel scheduling of de-identification and query tasks without introducing lock bottlenecks.

[0049] The de-identification service module and query service module use XXHash3_64 as the core hash function. The DataHash table and MidIndexHash table are the core structures of the de-identification mapping. The two hash tables use double hashing and quadratic probing to resolve collisions, thereby minimizing clustering and ensuring stable search performance even under high load factors.

[0050] The persistence module and backup module work together to ensure data persistence, high availability, and disaster recovery; specifically: 1. Local persistence The mapping between MID and the original plaintext, the AppId dictionary, and administrator information are serialized to disk using serde_json, a Rust library for fast and efficient JSON serialization and deserialization. This ensures the immediate persistence of data and allows for rapid recovery after a server restart, thereby enhancing the robustness of the system in the production environment without relying on an external relational database.

[0051] 2. TCP Incremental Backup Strategy Each backup operation establishes a TCP connection with the remote backup host, and closes the connection after the backup transfer is complete. This design incurs connection establishment overhead for each transfer, but maintains the simplicity and statelessness of network interaction. Serializing and transmitting only records that have been added or updated since the last backup can greatly reduce the amount of data transmitted. Backup data structures include table names, offsets, and serialized bytes to ensure accurate reconstruction during recovery; Based on the TCP incremental backup strategy, this system achieves the goal of maintaining the latest off-site backup with minimal overhead. In actual deployment verification, the incremental TCP backup operation achieved an average round-trip latency of approximately 2.14 milliseconds, which is about twice as fast as the traditional distributed system backup protocol (reported latency of approximately 4.3 milliseconds). This shows that the strategy effectively minimizes synchronization latency while ensuring data consistency.

[0052] 3. Efficient backup and retransmission mechanism Each key table defines an independent retransmission queue, storing metadata such as the offset of the backup task, the receiver's IP, and the timeout timestamp; Acknowledgment (ACK) protocol: After a successful write, the backup receiver sends an acknowledgment containing the table name and offset. The task will only be dequeued after a valid ACK is received. Retransmission based on timeout: When no ACK is received within the configured timeout period, a dedicated monitoring thread or asynchronous task will trigger a retransmission. Network anomaly tolerance: If a TCP connection is broken, the task will be retransmitted when the connection is re-established, thus ensuring eventual consistency.

[0053] The backup module uses a dedicated queue and asynchronous threads to achieve reliable data transmission in the event of network failure.

[0054] Example 4 This embodiment tests the system from Embodiment 3 in a real server environment.

[0055] I. Experimental Environment The test environment configuration is shown in Table 5. Each performance test was repeated five times, and the results were reported in the form of mean ± standard deviation to ensure statistical reliability.

[0056] Table 5. Real Server Environment Test Configuration Parameters

[0057] II. Functional Testing 1. Test Objective Functional testing aims to verify the correctness of the following functions: Identity authentication module: The assigned AppId is unique, persistent, and non-repeating; The data masking service module returns a unique and irreversible MID for each input field. Query service module: Enables bidirectional lookup between MID and original fields; Administrator module: Tests role-based access control for super administrators and regular administrators; Backup module: Data integrity, ACK confirmation, and retransmission functions under fault conditions.

[0058] 2. Testing Methods Identity authentication module test: By calling the allocation API through Swagger UI, a unique AppId was successfully obtained, correctly inserted into the memory hash table, and persisted to disk. After persistence, the backup system transmitted incremental data to the receiver, which correctly stored and returned a valid ACK. The sender successfully parsed the ACK and removed the corresponding task from the retransmission queue. De-identification service module testing: Send valid field plaintext and request a unique MID. If the request contains an invalid AppId or a data type mismatch, the API returns an HTTP 400 error with a corresponding message to confirm that input validation has been performed. The newly generated MID and field plaintext are correctly inserted into the hash table, persisted, and backed up through full ACK processing and task queue management. Query service module test: Both bidirectional lookups from MID to plaintext field and from plaintext field to MID returned the correct results. Invalid AppId or mismatched field types resulted in an HTTP 400 response with an accurate error message. Administrator module test: The super administrator successfully logged in, viewed the hash table load factor, viewed and modified existing administrator records, and registered a new regular administrator. A regular administrator can successfully log in, view the hash table load factor, and view existing administrator records, but has no right to modify them; Security verification: All administrators are prohibited from accessing or modifying anonymized data and AppId mappings to protect data confidentiality; New administrator registration: Registration information containing a valid username and password will be inserted into the memory administrator hash table without duplication and persisted immediately. It will be backed up with a successful ACK confirmation. Invalid registration information due to duplicate usernames or format violations will generate corresponding error messages. Backup module test: Intentionally blocking ACKs to simulate a failure due to loss will trigger a retransmission task after a timeout, which will be correctly overwritten by the receiver.

[0059] 3. Test Results All API interfaces of the above modules returned expected results, with a functional error rate of 0%; the backup module achieved end-to-end correctness, a complete ACK cycle, and reliable retransmission under simulated failure conditions; the administrator function enforced strict role-based access control and data confidentiality.

[0060] The functional test results show that the system of the present invention meets the design requirements of functional integrity, data consistency, access control and backup robustness, laying a solid foundation for subsequent performance and security evaluation.

[0061] III. API Performance Testing 1. Test objectives and conditions We used wrk (an open-source HTTP benchmarking tool) to perform performance tests on the AppId assignment, query, and de-identification API interfaces. The test conditions were as follows: Duration: 30 seconds for anonymization and query interface test, 5 seconds for AppId assignment interface test. Number of threads: 4 Number of connections: 50, 100, 200 2. Test Results

[0062] Table 6 API Interface Test Results

[0063] The CPU percentage parameter in Table 6 reflects the multi-core utilization in a 4-core CPU environment. When the utilization exceeds 100%, it means that more than one core is being used at the same time (for example, 200% means that two cores are being fully utilized). This is the expected behavior in multi-threaded asynchronous operation, indicating that the system effectively utilizes multi-core parallelism to achieve high throughput without creating a bottleneck for single-core performance.

[0064] For the anonymized API: with 200 concurrent connections, a throughput of 3840 requests / second was achieved, with an average latency of approximately 52 milliseconds and no errors.

[0065] For the query API: bidirectional queries for MID and plaintext fields achieved a throughput of 6000-7000 requests / second, with latency increasing linearly but still below an acceptable threshold (<30 milliseconds).

[0066] For the AppId allocation API: it achieves a throughput of 350 requests / second with 100-200 concurrent connections, but the latency is relatively high, mainly due to the AppId generation logic and persistence operations.

[0067] Throughout the above tests, memory usage remained low (<0.15% of the 16GB system memory).

[0068] In summary, the system proposed in this invention leverages Rust's memory safety and Tokio's asynchronous concurrency to achieve high throughput and stable latency for de-identification and query APIs under real-world workloads. Despite the overhead of AppId generation and persistence, AppId allocation maintains acceptable performance. Test experiments have evaluated and verified the system's correctness, concurrency performance, and reliability, demonstrating its feasibility for large-scale deployment in privacy-critical sectors such as finance, healthcare, and e-commerce.

[0069] It should be noted that the sequence numbers of the above embodiments of the present invention are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, apparatus, article, or method. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0070] The above are only some embodiments of the present invention, and are used only to illustrate the technical solutions of the present invention and not to limit the patent scope of the present invention. Equivalent structural or procedural transformations made by those skilled in the art based on the content of the present invention specification and drawings, without departing from the spirit and scope of the technical solutions of the present invention, should all be covered within the protection scope of the claims of the present invention.

Claims

1. A data protection method based on field splitting and distributed data masking, characterized in that, include: Authorized external applications split the input sensitive information into several independent fields and send the sensitive fields to the corresponding de-identification service nodes for data de-identification. Each de-identification service node is set up independently and there is no data interaction between them. The de-identification service node returns the de-identification result to the external application. Neither the external application nor the administrator of each de-identification service node has the right to access the complete dataset of the de-identification service node. The data desensitization includes: The de-identification service node receives plaintext data of the fields to be de-identified sent completely randomly by external applications, generates a locally unique and irreversible mask identifier MID for the corresponding plaintext data using an incrementing algorithm, calculates the hash address using the hash function XXHash3_64 and stores the mapping relationship between MID and plaintext data. The step of returning the desensitization results to external applications includes: The MID is returned to the external application without returning the mapping relationship between the MID and the plaintext data. The application database of the external application does not store any plaintext data of sensitive fields. Administrators do not have access to the complete dataset of the de-identified service node, including: Administrators do not have the right to access or modify any application identifier AppId, MID, plaintext data of sensitive fields, or the mapping relationship between MID and plaintext data.

2. The data protection method based on field splitting and distributed de-identification according to claim 1, characterized in that, The mapping relationship between the stored MID and plaintext data includes: Construct a DataHash table that supports mapping plaintext data of sensitive fields to their corresponding MIDs. Calculate the hash value of the original field using XXHash3_64 and use it as the main index of the DataHash table. When a collision occurs, use an auxiliary hash function to calculate the probe step size and store the plaintext data and MID mapping in the DataHash table accordingly. Construct a MidIndexHash table that supports reverse mapping from MID to its corresponding field in plaintext data. Store the MID and its primary index in the DataHash table in the MidIndexHash table and use secondary probing to handle collisions.

3. The data protection method based on field splitting and distributed desensitization according to claim 1, characterized in that, Each data masking service node is configured with an independent remote backup receiver to enable remote asynchronous backup of the data stored on that node.

4. The data protection method based on field splitting and distributed desensitization according to claim 1, characterized in that, Also includes: The external application sends the MID of the sensitive fields contained in the information to be called to the corresponding de-identification service node. The de-identification service node searches for the corresponding plaintext data in the hash table based on the received MID and returns the plaintext data to the external application. The returned plaintext data is neither displayed on the external application's interactive interface nor stored in the external application's application database. It is only sent to other applications according to the instructions of the external application administrator.

5. A data protection system based on field splitting and distributed data masking, used to implement the data protection method as described in any one of claims 1 to 4, characterized in that, include: The identity authentication module is used to assign and manage application identifiers (AppId) for external applications; The API service module is used to enable data interaction between external applications and various de-identification service nodes, including AppId, verification request, de-identification request, query request, plaintext data, and MID; The request scheduling module is used to intelligently allocate each request task to the corresponding service module; The masking service module is used to generate a locally unique and irreversible mask identifier (MID) for the plaintext data of the field to be masked. It uses the hash function XXHash3_64 to calculate the hash address and store the mapping relationship between the MID and the plaintext data. The query service module supports bidirectional lookup between MID and plaintext data; The persistence module is used to serialize the mapping between MID and plaintext data, the AppId dictionary, and administrator information to disk; The backup module is used to maintain the incremental backup queue for each hash table, transmit updates to the remote backup receiver via TCP, and receive initialization, incremental, and full backup requests.

6. The data protection system based on field splitting and distributed de-identification according to claim 5, characterized in that, The request scheduling module uses Tokio's multi-threaded asynchronous runtime to allocate request tasks.

7. The data protection system based on field splitting and distributed de-identification according to claim 5, characterized in that, The backup module uses a dedicated queue and asynchronous threads to implement data backup and retransmission. Each hash table defines an independent retransmission queue. After successful writing, the backup receiver sends acknowledgment data containing the table name and offset. The task will only be dequeued after receiving valid acknowledgment data. If no valid acknowledgment data is received within the configured timeout period, the dedicated queue or asynchronous thread will trigger retransmission.

Images