A method for desensitizing personal information data

Abstract

The present application relates to a method for desensitizing personal information data, and relates to the technical field of data processing. The method includes confirming the file structure and configuring the data parsing model, setting the source file template and the output file adaptation template; reading the file through multiple threads, including obtaining basic file information, the total number of bytes in the file, the total number of lines in the file, and the maximum number of threads; calculating the number of bytes processed by a single thread, and determining the starting and ending byte positions of each thread; and performing data parsing and desensitization processing, wherein multiple threads execute the data parsing and desensitization processing. The present application has the effect of being able to achieve efficient information desensitization processing.

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and in particular to a method for desensitizing personal information data. Background Art

[0002] With the development of retail banking, data centers have accumulated a large amount of personal customer information collected during account opening and business processing, including names, ID numbers, ID types, mobile phone numbers, bank card numbers, and other sensitive data. To develop their business, banks often need to collaborate with third-party institutions. In this process, to facilitate innovative data applications and meet compliance requirements, sufficient data desensitization must be performed to reduce the data's sensitivity level.

[0003] In terms of customer information processing, a mainstream method currently used by banks is to assign a unique identification code to a single customer in the ECIF system, and use this code to identify the customer when interacting with third-party non-bank institutions. In this way, only the information identification of the customer's "person" dimension can be met, and it cannot be refined to the card level, which cannot be met in many scenarios that require bank card interaction. The fundamental reason is that the ECIF system is based on customer lifecycle management of corporate customer information and cannot go deep into specific businesses to explain the relationship between "people" and "cards." In addition, in terms of data desensitization, common algorithms such as replacement, encryption, hashing, and obfuscation can complete data desensitization and interaction in specific scenarios, but there is a lack of integrity in the data desensitization link.

[0004] Currently, there are various differences in the scenarios in which banks cooperate with third-party institutions. Relying on single-user identification and traditional desensitization methods cannot meet or require customized development. However, customized services not only increase the operating costs of enterprises, but also cause great difficulties for the development and operation and maintenance of the service provider. Summary of the Invention

[0005] In order to achieve efficient information desensitization processing, this application provides a method for desensitizing personal information data.

[0006] This application provides a method for desensitizing personal information data using the following technical solutions:

[0007] A method for desensitizing personal information data, confirming the file structure and configuring the data parsing model, setting the source file template and the output file adaptation template;

[0008] Multi-threaded file reading, including obtaining basic file information, total file bytes, total file lines, and the maximum number of threads; and calculating the number of bytes processed by a single thread, determining the starting and ending byte positions of each thread;

[0009] Data parsing and desensitizing processing: multiple threads execute data parsing and desensitizing processing.

[0010] By adopting the above technical solution, the processing scope of each thread is dynamically allocated based on the total number of bytes, total number of lines, and maximum number of threads in the file, optimizing resource utilization. By dynamically allocating the processing scope of each thread based on the total number of bytes, total number of lines, and maximum number of threads in the file, optimizing resource utilization. Through multi-threaded parallel processing, the processing efficiency of large-scale data files is significantly improved; multiple file formats (such as fixed-length formats or delimited formats) are supported to adapt to the needs of different data sources; and through data parsing and desensitization, the security and privacy of sensitive information are ensured.

[0011] Preferably, determining the byte starting position and ending position of each thread comprises the following steps:

[0012] Roughly calculate the end position of each thread; the rough end position is equal to the total number of bytes in the file divided by the maximum number of threads; accurately calculate the start and end positions of each group of threads, assuming the thread coordinates t(x,y), x is the start coordinate, y is the end position, then the x and y coordinate values ​​corresponding to f(t) are ((t)b, (t+1)b), t starts at 0 by default, max(t) = C, and t+1 is less than C; where C is the number of threads, and C is set to twice the number of hardware cores; b is the rough processing length of each thread, and b = Z / C, where Z is the total number of bytes in the file.

[0013] By adopting the above technical solution, through the combination of rough calculation and precise calculation, the data block range processed by each thread is ensured to be accurate, thereby achieving the effect of optimizing thread task allocation, reducing computing overhead and improving overall processing efficiency.

[0014] Preferably, the number of bytes processed by a single thread is equal to the average number of bytes processed by each thread plus the number of bytes remaining in the row of data where the avg position of the file is located, and the average number of bytes processed by each thread is equal to the total number of bytes in the file divided by the maximum number of threads.

[0015] By adopting the above technical solution, by calculating the average number of bytes processed by each thread and considering the integrity of the file line data, the workload of each thread is balanced, avoiding data breakage or loss caused by improper data block segmentation.

[0016] Preferably, the multi-threaded file reading further includes verifying the integrity of each thread, including: F(Cxs)=(xs / (NIO( / n)))?Cxs:(xs+(xsNIO( / n))); in F(Cxs), F is the function name, Cxs is the independent variable, C is the total number of threads, xs is the thread number of the current position to be calculated, that is, Cxs is the corrected value after the integrity check of the thread end position xs under the constraint of the total number of threads C; xs / (NIO( / n)) is the file processing position calculated by each assigned thread; NIO( / n) is a delimiter for data rows or data blocks.

[0017] By adopting the above technical solution, by verifying whether the end position of each thread is a complete row or data block, data fragmentation is prevented and the accuracy and consistency of data processing in a multi-threaded environment are ensured.

[0018] As an advantage, it also includes the fuzzy calculation thread processing unit group length t(x), and

[0019]

[0020] Where t(buf) is the size of the buffer, x is the number of threads, and (Floor) is the floor function.

[0021] By adopting the above technical solution, the size of the data block processed by each thread is dynamically adjusted according to the buffer size and the number of threads, thereby optimizing resource utilization.

[0022] Preferably, it also includes accurately locating the end mark, and the process of accurately locating the end mark includes reading byte blocks of fixed size after completing file stream positioning and verifying the end marks in the byte blocks one by one until a valid end mark is found.

[0023] By adopting the above technical solution, by reading byte blocks in fixed size and verifying the end mark, it is ensured that the data block processed by each thread is complete, reducing redundant operations of data reading and improving processing efficiency.

[0024] Preferably, the starting position of the Nth subsequent thread is set to the ending position of its previous thread, ie, the N-1th thread, plus one position.

[0025] By adopting the above technical solution, by setting the starting position of the subsequent thread to the ending position of the previous thread plus one, the continuity of data processing is ensured and data omission or repeated processing is avoided.

[0026] Preferably, the data analysis and desensitization process includes the following steps:

[0027] When the application starts, it loads the template configuration in the data parsing model and parses the specified desensitized fields in each row of data according to the template configuration; it generates ciphertext data using a hybrid obfuscation algorithm;

[0028] Using key-value pairs to store plaintext and the ciphertext data;

[0029] The plaintext and ciphertext data are distributedly cached.

[0030] By adopting the above technical solutions and through template configuration, data parsing can be automated; a hybrid obfuscation algorithm is used to desensitize sensitive fields to ensure data security; key-value pairs are used to store plaintext and ciphertext data to support fast query and verification; and distributed caching supports high concurrency processing in a multi-threaded environment.

[0031] As a preference, it also includes thread safety and exception handling: using efficient lock-free AtomicLong to implement thread-safe integer operations, and judging whether the number of lines read from the file AtomicLong readCount is equal to the number of lines written AtomicLong writeCount. If they are equal, the file processing is completed normally; if they are not equal, the file processing is abnormal.

[0032] By adopting the above technical solution, safe operation in a multi-threaded environment is achieved through a lock-free mechanism to avoid data competition; by comparing the number of read rows with the number of written rows, the completeness of data processing can be quickly detected, so that when an anomaly is found, the exception handling process can be entered in time to ensure the accuracy and completeness of data processing.

[0033] In summary, this application includes at least one of the following beneficial technical effects:

[0034] 1. Significantly improve data processing efficiency through multi-threaded parallel processing and dynamic task allocation;

[0035] 2. Use a hybrid desensitizing algorithm and distributed caching mechanism to ensure the security and consistency of sensitive information;

[0036] 3. Ensure the accuracy and integrity of data processing through thread safety mechanisms and exception handling;

[0037] 4. Support multiple file formats and dynamic resource allocation to adapt to data processing needs of different scales. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is the flow chart for the implementation of this application;

[0039] Figure 2 is a flow chart of step S1;

[0040] Figure 3is a flow chart of step S2;

[0041] Figure 4 This is a flowchart of step S3. DETAILED DESCRIPTION

[0042] The following is combined with Figure 1 This application is described in further detail.

[0043] The embodiments of the present application disclose a method for desensitizing personal information data.

[0044] Reference Figure 1 A method for desensitizing personal information data includes the following steps:

[0045] S1: Confirm the file structure and configure the data parsing model.

[0046] S2: Multi-threaded reading of files.

[0047] S3: Data analysis and desensitization processing.

[0048] S4: Thread safety and exception handling.

[0049] Reference Figure 2 ,S1: Confirm the file structure and configure the data parsing model, including the following steps:

[0050] S11: Confirm the file structure. Confirm the structural characteristics of each line of data in the file. The file can be in a fixed-length format or a delimiter format (such as |). In order to parse each field in each line of data, a data parsing model must be configured.

[0051] S12: Source file template configuration: Configure the source file template, specify the delimiter for each row of data and the attributes of each field. For example:

[0052] FILE::SPLIT[|]::[a:str:4][b:str:4][c:str:6][papers:str:19][card:str:19][phone:str:12][sum:str:5]

[0053] Among them, SPLIT[|]" indicates that the separator of each line of data in the source file is "|", and the [] indicates the corresponding field attributes in each line of data. Taking [a:str:4] as an example, a is the field name, str is right-padded with spaces (long is left-padded with 0, decimal is added with decimal places), and 4 is the field length.

[0054] S13: Output file adaptation template configuration:

[0055] FILE::SPLIT[|]::[a:str:4][b:str:4][c:str:6][papers:str:19][card:str:19][phone:str:12][sum:str:5]

[0056] Among them, "SPLIT[|+|]" indicates that the delimiter for each line of data in the output file is "|+|", and [] indicates the corresponding domain attributes and desensitization configuration in each line of data. Taking [papers:str:19:Mapping(PAPERS)] as an example, papers represents the document number keyword, and Mapping(PAPERS) indicates desensitizing the sensitive document number in this line of data.

[0057] Reference Figure 3 , S2: Multi-threaded reading of files, including the following steps:

[0058] S21: Obtain basic file information, including the total number of bytes in the file (sum), the total number of lines in the file (count), and the maximum number of threads (maxThreadNum). Assume that the total number of bytes in the file (sum) = 100,000, the total number of lines in the file (count) = 1,000, and the maximum number of threads (maxThreadNum) = 10.

[0059] S22: Calculate the number of bytes processed by a single thread. The average number of bytes processed by each thread is calculated using the formula: avg = sum / maxThreadNum. The number of bytes processed by a single thread is equal to the average number of bytes processed by each thread plus the number of bytes remaining in the row of data corresponding to the avg position in the file. This is the starting and ending positions of the bytes processed by each thread.

[0060] S23: Determine the byte starting position and ending position of each thread.

[0061] The rough end position is calculated by dividing the total number of bytes in the file by the maximum number of threads, i.e., int i = sum / maxThreadNum. The first thread group in the fuzzy calculation processes the unit group length. The t(x) function defaults to t(core), which determines the data block size processed by each thread based on the number of CPU cores. For example, in x0(0, t(x)), x0 is a thread group, indicating that it starts at position 0 and processes a data block of length t(x).

[0062] Accurately calculate the starting and ending positions of each group of threads. Suppose the thread coordinates are t(x,y), where x is the starting coordinate and y is the ending position. Then the x and y coordinate values ​​corresponding to f(t) are ((t)b, (t+1)b). t starts at 0 by default, max(t) = C, and t+1 is less than C.

[0063] Where C is the number of threads, and C is set to twice the number of hardware cores; b is the rough processing length of each thread, and b = Z / C, where Z is the total number of bytes in the file.

[0064] In each thread, the data block range from x(0) to x is x(0)(x0a,xb); the data block range from x to xs is x(xb,xsb); the data block range from xs to F(Cxs) is xs(x:xsb,xsb); the data block range from F(Cxs) to max(CoreF(NIO)) is Cxs(xs:xsb,Cxsb).

[0065] Round down to calculate t(x) and calculate the byte block size processed by each group of threads. The relevant formula is:

[0066]

[0067] Where t(buf) is the size of the buffer, and x is the number of threads. By dividing the buffer size by x*2 and rounding up, we get t(x), the size of the data block processed by each thread.

[0068] S24: Verify the integrity of each thread. Each thread calculates whether the current end position is a complete row or complete data to prevent data fragmentation. The relevant formula is:

[0069] F(Cxs)=(xs / (NIO( / n)))? Cxs:(xs+(xsNIO( / n)))

[0070] F(Cxs) ensures that each thread ends processing a complete data row or block, preventing data fragmentation. xs represents the end position of the current thread. NIO( / n) represents the delimiter for a data row or block (such as a newline character, \n). If xs is an integer multiple of NIO( / n), Cxs is used; otherwise, xs is adjusted to the end position of the next complete data row or block.

[0071] Then, read 1024 bits and verify 1024 bytes. When the verification reaches the end mark, mark it (for example, mark it as e);

[0072] Next, each group of threads obtains the corresponding start identifier based on the calculation result of the previous group of threads, that is, the start position of the n+1 thread is the end position of the n thread+1.

[0073] The final result is the byte range for each thread as shown below:

[0074] thread1->0-9000(byte)

[0075] thread2->9001-18012(byte)

[0076] thread3->18013-27453(byte) ......

[0077] Reference Figure 4 ,S3: Data parsing and desensitization processing, including the following steps:

[0078] S31: When the application starts, it loads the template configuration in the data parsing model and parses the specified desensitized fields in each row of data according to the template configuration. Through templated configuration, data parsing is automated, reducing manual intervention and improving processing efficiency. It supports multiple file formats (such as fixed-length or delimited formats) to adapt to different data source requirements. It ensures that the parsing of each field is strictly executed according to the template definition to avoid data parsing errors.

[0079] S32: Generate ciphertext data through a desensitization algorithm.

[0080] Hybrid obfuscation algorithms, such as Triple DES combined with the date format $Y$m$d, are used to desensitize sensitive fields. For example, the ID card number field "papers" is desensitized to generate ciphertext data. Sensitive information is desensitized using complex encryption algorithms (such as Triple DES) to prevent data leakage. Ciphertext generation can be enhanced by incorporating dynamic factors such as date format, enhancing the randomness and irreversibility of desensitization. While ensuring security, the algorithm design balances performance and is suitable for large-scale data processing.

[0081] S33: Plaintext and ciphertext data storage.

[0082] Use key-value pairs map to store plaintext and ciphertext data in the format of<bcp:p:plain_txt cipher_txt> and<bcp:c:cipher_txt plain_txt> .

[0083] Among them, plain_txt is plaintext data, and cipher_txt is ciphertext data.

[0084] By storing plaintext and ciphertext data in key-value pairs, the mapping relationship between plaintext and ciphertext can be quickly queried and verified, and a one-to-one correspondence between plaintext and ciphertext can be ensured, thus avoiding data loss or misalignment and supporting subsequent processing and analysis of plaintext and ciphertext data.

[0085] S34: Distributed caching of plaintext and ciphertext data

[0086] Redis implements distributed locking to determine whether the plaintext-ciphertext mapping exists. If so, the data is not inserted; otherwise, the data is batched and inserted into the cache. This distributed cache and locking mechanism supports high concurrency in multi-threaded environments, preventing data conflicts. This ensures the uniqueness of the plaintext-ciphertext mapping, preventing duplicate insertions and data redundancy. Batch inserts reduce the number of cache operations and improve system performance.

[0087] S4: Thread safety and exception handling.

[0088] Thread-safe integer operations are implemented using the efficient lock-free AtomicLong implementation. This lock-free mechanism ensures secure operation in multi-threaded environments, avoiding data contention and lock overhead. After file processing is complete, the AtomicLong readCount count of lines read is checked to ensure the integrity of data processing and ensure accuracy. If they are equal, the file processing is completed normally; if not, the file processing is abnormal and exception handling is initiated to prevent data loss or error propagation.

[0089] The implementation principle of a method for desensitizing personal information data in an embodiment of the present application is as follows: large files are read in parallel through multiple threads, and large files are divided into multiple data blocks, with each thread responsible for processing one data block. The total number of bytes and lines of the file is precalculated, and the byte range processed by each thread is dynamically allocated based on the maximum number of threads. By accurately calculating the starting and ending positions of each thread, it is ensured that the data block processed by each thread is a complete line or data block, avoiding data fragmentation. Specifically, the end position of the thread is adjusted by verifying whether the end position is a complete line to ensure data integrity.

[0090] The above are all preferred embodiments of the present application, and are not intended to limit the scope of protection of the present application. Therefore, any equivalent changes made based on the structure, shape, and principle of the present application should be included in the scope of protection of the present application.

Claims

1. A method for desensitizing personal information data, characterized by: include: Confirm the file structure and configure the data parsing model, set the source file template and output file adaptation template; Multi-threaded reading of files, including obtaining basic file information, obtaining the total number of bytes in the file, the total number of lines in the file, and the maximum number of threads; and calculating the number of bytes processed by a single thread, and determining the starting and ending byte positions of each thread; Data parsing and desensitization processing, multiple threads perform data parsing and desensitization processing; Determining the starting and ending byte positions for each thread involves the following steps: Roughly calculate the end position of each thread; the rough end position is equal to the total number of bytes in the file divided by the maximum number of threads; Accurately calculate the starting and ending positions of each group of threads. Suppose the thread coordinates are t(x,y), where x is the starting coordinate and y is the ending position. Then the x and y coordinate values ​​corresponding to f(t) are ((t)b, (t+1)b). t starts at 0 by default, max(t) = C, and t+1 is less than C. Where C is the number of threads, and C is set to twice the number of hardware cores; b is the rough processing length of each thread, and b=Z / C, where Z is the total number of bytes in the file.

2. The method for desensitizing personal information data according to claim 1, characterized in that: The number of bytes processed by a single thread is equal to the average number of bytes processed by each thread plus the number of bytes remaining in the row of data where the avg position of the file is located. The average number of bytes processed by each thread is equal to the total number of bytes in the file divided by the maximum number of threads.

3. The method for desensitizing personal information data according to claim 1, characterized in that: The multi-threaded file reading also includes verifying the integrity of each thread, including: In F(Cxs), F is the function name, Cxs is the independent variable, C is the total number of threads, and xs is the thread number of the current position to be calculated. That is, Cxs is the corrected value after the integrity check of the thread end position xs under the constraint of the total number of threads C; xs / (NIO( / n)) calculates the file processing position for each assigned thread; NIO( / n) is the delimiter of data rows or data blocks.

4. The method for desensitizing personal information data according to claim 1, characterized in that: It also includes the fuzzy calculation thread processing unit group length t(x), and , where t(buf) is the size of the buffer, x is the number of threads, and (Floor) is the floor function.

5. The method for desensitizing personal information data according to claim 1, characterized in that: It also includes accurately locating the end mark. The accurate positioning of the end mark process includes reading byte blocks according to a fixed size after completing the file stream positioning and verifying the end marks in the byte blocks one by one until a valid end mark is found.

6. A method for desensitizing personal information data according to claim 5, characterized in that: The starting position of the Nth subsequent thread is set to the end position of its previous thread, that is, the N-1th thread, plus one position.

7. The method for desensitizing personal information data according to claim 1, characterized in that: The data analysis and desensitization process includes the following steps: When the application starts, the template configuration in the data parsing model is loaded, and the masked fields specified in each row of data are parsed according to the template configuration; Generate ciphertext data through a hybrid obfuscation algorithm; Using key-value pairs to store plaintext and the ciphertext data; The plaintext and ciphertext data are distributedly cached.

8. The method for desensitizing personal information data according to claim 1, characterized in that: It also includes thread safety and exception handling: using efficient lock-free AtomicLong to implement thread-safe integer operations, and judging whether the number of lines read from the file (AtomicLong readCount) is equal to the number of lines written (AtomicLong writeCount). If they are equal, the file processing is completed normally; if not, the file processing is abnormal.

Images