Log desensitization method and device, equipment and medium

Abstract

The invention belongs to the field of big data, and relates to a log desensitization method and device, equipment and a medium, and the method comprises the steps: obtaining a concurrent thread count, a desensitization rule, a queue size parameter and an output directory path from a configuration file; and acquiring a plurality of to-be-desensitized log lines based on the output directory path, generating a line serial number of each to-be-desensitized log line through an atomic counter, assembling the line serial number and each to-be-desensitized log line into a plurality of messages with serial numbers, and putting the messages into a blocking queue determined by a queue size parameter. And creating a thread pool based on the number of concurrent threads, reading messages from the blocking queue by threads, replacing sensitive data according to a desensitization rule to obtain messages containing line serial numbers and desensitization log lines, and finally writing the desensitization logs into a desensitization log file in order according to the line serial numbers. The method can be applied to the business fields of financial science and technology, insurance, medical treatment and the like, and can solve the problem that the desensitized data is difficult to position and trace.

Description

Technical Field

[0001] This application relates to the field of big data and is applied to online processing business scenarios such as fintech, insurance, and healthcare. In particular, it relates to a log desensitization method, device, equipment, and medium. Background Technology

[0002] With the improvement of data security regulations, enterprises are facing increasingly stringent compliance requirements for data processing. Logs, as the core carrier of system operation and user actions, contain sensitive data such as ID card numbers, bank card numbers, and mobile phone numbers. Security protection requires de-identification processing. Log de-identification technology, by transforming, replacing, or encrypting sensitive information, prevents information leakage while retaining analytical value for troubleshooting and behavior auditing, and has become an essential part of system design.

[0003] In practical applications, the requirements for de-identification are particularly prominent in audit logs. For example, account operation logs in financial transaction systems, medical record access logs in medical systems, and insurance claims process logs contain sensitive information and must also meet compliance traceability requirements. However, existing de-identification technologies have significant drawbacks: First, there is no association identifier between de-identified logs and original logs, making it impossible to locate de-identified logs through the original information, affecting traceability; second, they often use full processing, resulting in redundant calculations and wasted resources, and multi-threaded processing can easily lead to log order disorder, disrupting the time sequence of audit logs; third, there is no configurable resource management, a lack of data buffering mechanisms, and poor processing stability.

[0004] In summary, existing technologies suffer from problems such as difficulty in tracing data after anonymization, low efficiency in incremental processing, disordered log order, and insufficient stability. Summary of the Invention

[0005] The purpose of this application is to provide a log de-identification method, apparatus, computer equipment, and storage medium to solve the problems of existing log de-identification technologies, such as difficulty in tracing de-identified data, low incremental processing efficiency, disordered log order, and insufficient stability.

[0006] Firstly, a log desensitization method is provided, which adopts the following technical solution: The system retrieves the concurrent thread count, desensitization rules, queue size parameter, and output directory path from the log desensitization program's configuration file. Based on the output directory path, it obtains multiple log lines to be desensitized and generates a line number for each line using an atomic counter. Each log line and its line number are assembled to generate multiple numbered messages, which are then placed into a blocking queue. The capacity of the blocking queue is determined by the queue size parameter. A thread pool is created based on the concurrent thread count, and each thread in the pool reads multiple numbered messages from the blocking queue. Based on the desensitization rules, sensitive data is replaced in the read numbered messages to obtain multiple desensitized messages, each including a line number and a desensitized log line. Based on the line number, the desensitized log lines are written sequentially to a desensitized log file.

[0007] Secondly, a log desensitization device is provided, which adopts the following technical solution: The first acquisition module is used to obtain the number of concurrent threads, desensitization rules, queue size parameters and output directory path from the configuration file of the log desensitization program; The second acquisition module is used to acquire multiple log lines to be de-identified based on the output directory path, and generate a line number for each log line to be de-identified using an atomic counter; The assembly module is used to assemble each log line to be de-identified and its line number, generate multiple messages with sequence numbers, and put the multiple messages with sequence numbers into a blocking queue. The capacity of the blocking queue is determined by the queue size parameter. The read module is used to create a thread pool based on the number of concurrent threads, and read multiple messages with sequence numbers from the blocking queue through each thread in the thread pool; The replacement module is used to replace sensitive data in multiple read messages with sequence numbers based on the de-identification rules, so as to obtain multiple de-identified messages. Each de-identified message includes a line sequence number and a de-identification log line. The first writing module is used to write the de-identified log lines into the de-identified log file in an orderly manner based on the line number.

[0008] Thirdly, a computer device is provided, which adopts the following technical solution: The system retrieves the concurrent thread count, desensitization rules, queue size parameter, and output directory path from the log desensitization program's configuration file. Based on the output directory path, it obtains multiple log lines to be desensitized and generates a line number for each line using an atomic counter. Each log line and its line number are assembled to generate multiple numbered messages, which are then placed into a blocking queue. The capacity of the blocking queue is determined by the queue size parameter. A thread pool is created based on the concurrent thread count, and each thread in the pool reads multiple numbered messages from the blocking queue. Based on the desensitization rules, sensitive data is replaced in the read numbered messages to obtain multiple desensitized messages, each including a line number and a desensitized log line. Based on the line number, the desensitized log lines are written sequentially to a desensitized log file.

[0009] Fourthly, a computer-readable storage medium is provided, which adopts the following technical solution: The system retrieves the concurrent thread count, desensitization rules, queue size parameter, and output directory path from the log desensitization program's configuration file. Based on the output directory path, it obtains multiple log lines to be desensitized and generates a line number for each line using an atomic counter. Each log line and its line number are assembled to generate multiple numbered messages, which are then placed into a blocking queue. The capacity of the blocking queue is determined by the queue size parameter. A thread pool is created based on the concurrent thread count, and each thread in the pool reads multiple numbered messages from the blocking queue. Based on the desensitization rules, sensitive data is replaced in the read numbered messages to obtain multiple desensitized messages, each including a line number and a desensitized log line. Based on the line number, the desensitized log lines are written sequentially to a desensitized log file.

[0010] Compared with existing technologies, the embodiments of this application have the following main advantages: Through multi-dimensional technical design, the core defects of existing log de-identification technologies are specifically addressed. Specifically, firstly, an atomic counter generates a unique line number for each log line to be de-identified. This line number accompanies the entire log processing flow, becoming the association identifier between the original log and the de-identified log. The de-identified log record can be quickly located using the line number, completely solving the problem of difficult data traceability after de-identification. Secondly, the de-identified log file is detected based on the output directory path, and the latest timestamp is extracted. Only incremental log lines with modification times greater than this timestamp are processed, avoiding redundant calculations caused by full processing. Simultaneously, the line number ensures orderly writing of de-identified log lines, resolving the log order disorder problem caused by multi-threaded processing and ensuring the temporal correlation of audit logs. Thirdly, the number of concurrent threads is read from the configuration file to flexibly adjust the processing concurrency capability. Combined with the queue size parameter to limit the blocking queue capacity, data buffering and flow control are implemented to avoid program crashes caused by data accumulation, improving the resource adaptability and operational stability of the de-identification processing. Attached Figure Description

[0011] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is an exemplary system architecture diagram to which this application can be applied; Figure 2 A flowchart of an embodiment of the log desensitization method according to this application; Figure 3 This is a schematic diagram of the structure of one embodiment of the log desensitization device according to this application; Figure 4 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0013] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0014] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0015] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0016] like Figure 1As shown, system architecture 100 may include terminal device 101, network 102, and server 103. Terminal device 101 may be a laptop 1011, tablet 1012, or mobile phone 1013. Network 102 is used as a medium to provide a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired, wireless communication links, or fiber optic cables.

[0017] Users can use terminal device 101 to interact with server 103 via network 102 to receive or send messages, etc. Various communication client applications can be installed on terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0018] Terminal device 101 can be various electronic devices with a display screen and support web browsing. In addition to laptops 1011, tablets 1012, or mobile phones 1013, terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer, and a desktop computer, etc.

[0019] Server 103 can be a server that provides various services, such as a backend server that provides support for the pages displayed on terminal device 101.

[0020] It should be noted that the log desensitization method provided in this application embodiment is generally executed by a server / terminal device, and correspondingly, the log desensitization device is generally set in the server / terminal device.

[0021] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0022] Continue to refer to Figure 2 The diagram illustrates a flowchart of an embodiment of the log desensitization method according to this application. The log desensitization method includes the following steps: Step S201: Obtain the number of concurrent threads, desensitization rules, queue size parameters, and output directory path from the configuration file of the log desensitization program.

[0023] The log desensitization program is a computer-executable program that performs desensitization processing on sensitive log data. Its core logic covers the entire process of log reading, desensitization processing, and ordered writing. Deployed on the server side and initialized with parameters based on configuration files, it is used to desensitize various logs containing sensitive information, such as financial transaction audit logs and medical record access logs. For example, for account operation logs generated by financial systems, this program can automatically replace sensitive fields such as bank card numbers in the logs.

[0024] The configuration file is a text file that stores the parameters required for the log anonymization program to run. It can be in XML, YAML, or Properties format and is stored in the program deployment directory. It represents the core configuration information for program operation and provides the basis for program execution. For example, the configuration file can record key parameters such as the number of concurrent threads being 8 and the queue size parameter being 10000.

[0025] Among these parameters, the number of concurrent threads is one of the core parameters read from the configuration file. It represents the upper limit of the number of threads used in the log de-identification program to process messages to be de-identified in parallel, and is used to adjust processing efficiency to adapt to different server hardware performance (such as the number of CPU cores). For example, when the server has a 16-core CPU, the number of concurrent threads can be configured to 12 to make full use of hardware resources and improve the de-identification speed.

[0026] Among them, the desensitization rules are pre-defined and stored sensitive data processing rules in the configuration file. They represent the replacement logic for different types of sensitive information and are used to guide the program to transform, replace, or encrypt sensitive data. For example, the desensitization rule for a mobile phone number can be set to retain the first 3 and last 4 digits, and replace the middle 4 digits with asterisks, i.e., 138****5678.

[0027] The queue size parameter is a numerical parameter defined in the configuration file. It represents the maximum number of messages to be de-identified that the blocking queue can hold. It is used to limit the buffer capacity of the blocking queue to prevent data accumulation from causing program crashes. For example, when the queue size parameter is configured to 5000, the blocking queue can store a maximum of 5000 messages with sequence numbers.

[0028] The output directory path is the file system path specified in the configuration file, which represents the storage location of the de-identified log files. It also serves as the basis for detecting historical de-identified logs, locating the target log directory, and storing the final de-identified logs.

[0029] Step S202: Based on the output directory path, obtain multiple log lines to be de-identified, and generate a line number for each log line to be de-identified using an atomic counter.

[0030] Among them, multiple log lines to be de-identified are log data lines that are selected from the source log directory and need to be de-identified. Each line is an independent text record, representing the original log unit that has not undergone de-identification processing, and is used as the processing object of the de-identification program.

[0031] The atomic counter is a computer program component that ensures conflict-free counting operations in a multi-threaded environment. Deployed in the log reading module of the log de-identification program, it represents the unique counting capability of log lines and is used to generate a unique line number for each log line to be de-identified. For example, the atomic counter can start from 1 and generate consecutive numbers 1, 2, 3... for the read log lines.

[0032] The line number is a unique numerical identifier generated by an atomic counter and is bound one-to-one to a single log line to be anonymized. It represents the original time sequence of the log line and is used to solve the problem of log order disorder caused by multi-threaded processing. For example, line number 5 corresponds to the 5th generated account login operation log line in the original log.

[0033] Step S203: Assemble each log line to be de-identified and its line number to generate multiple messages with sequence numbers. Place the multiple messages with sequence numbers into a blocking queue. The capacity of the blocking queue is determined by the queue size parameter.

[0034] Assembly is a data processing action in the log de-identification process. It integrates the log line to be de-identified and its corresponding line number into a structured data unit, representing the binding process of the two data units, and is used to generate a transmittable message with a sequence number. For example, the line number "8" is integrated with "2025-08-09 10:00:00 Account login mobile number: 13812345678" into a message with a sequence number.

[0035] Among them, multiple numbered messages are structured data sets generated through an assembly process. Each message contains a row number and a log row to be anonymized, representing a structured data unit to be anonymized, and is stored in a blocking queue for the thread pool to read and process. For example, the set contains multiple messages such as "Sequence 1 - Log Row A" and "Sequence 2 - Log Row B".

[0036] The blocking queue is a thread-safe data buffer component initialized based on the queue size parameter. Following the first-in, first-out (FIFO) principle, it serves as a temporary storage container for messages to be anonymized, balancing the speed of log reading and anonymization processing. For example, when log generation is too fast, the blocking queue can temporarily store excess messages, preventing program memory overflow.

[0037] Step S204: Create a thread pool based on the number of concurrent threads, and read multiple messages with sequence numbers from the blocking queue through each thread in the thread pool.

[0038] The thread pool is a thread management component created based on the number of concurrent threads. It contains multiple reusable processing threads and serves as a carrier of parallel processing capabilities. It is used to efficiently read sequenced messages from the blocking queue and perform de-identification operations. For example, when the number of concurrent threads is 8, the thread pool initializes 8 threads to process messages simultaneously.

[0039] Each thread is an independent execution unit within the thread pool, capable of reading messages from the blocking queue and executing de-identification rules. It represents a single parallel processing chain used to replace sensitive data in a single message with a sequence number. For example, thread 1 reads message number 3, and thread 2 reads message number 5, completing the de-identification in parallel.

[0040] Step S205: Based on the de-identification rules, sensitive data is replaced in the multiple read messages with sequence numbers to obtain multiple de-identified messages. Each de-identified message includes a line sequence number and a de-identification log line.

[0041] Sensitive data replacement refers to the modification operation performed on sensitive fields in serial numbered messages according to the de-identification rules. It represents the processing of sensitive information and is used to convert the original sensitive data into de-identified data. For example, the ID number "11010111234" in the log line is replaced with "110****1234".

[0042] Among them, multiple de-identified messages are a set of structured data generated after the sensitive data replacement is completed. Each message contains a row number and a de-identified log line, representing a processed log data unit, which is used to write the de-identified log file in an orderly manner.

[0043] Among them, the de-identified log line is a text log unit after stripping the line number from the de-identified message. It represents a single log record that has completed the de-identification and is used to finally write it into the de-identified log file.

[0044] Step S206: Write the de-identified log lines into the de-identified log file in an orderly manner based on the line number.

[0045] Among them, the de-identified log file is a text file that stores the de-identified log lines. It is located in the output directory path and represents the final product of the de-identification process. It is used to retain compliant log data.

[0046] This application's embodiments address the core shortcomings of existing log anonymization technologies through multi-dimensional technical design. Specifically, firstly, an atomic counter generates a unique line number for each log line to be anonymized. This line number accompanies the entire log processing flow, serving as an identifier linking the original log and the anonymized log. The line number allows for rapid location of anonymized log records, completely resolving the difficulty of data traceability after anonymization. Secondly, the anonymized log file is detected based on the output directory path, and the latest timestamp is extracted. Only incremental log lines modified after a time greater than this timestamp are processed, avoiding redundant calculations caused by full processing. Simultaneously, the line number ensures ordered writing of anonymized log lines, resolving log order disorder issues caused by multi-threaded processing and guaranteeing the temporal correlation of audit logs. Thirdly, the number of concurrent threads is read from the configuration file to flexibly adjust processing concurrency capabilities. Combined with queue size parameters to limit the capacity of the blocking queue, data buffering and flow control are implemented to prevent program crashes caused by data accumulation, improving the resource adaptability and operational stability of the anonymization process.

[0047] In some optional implementations of this embodiment, step 202, obtaining multiple log lines to be de-identified based on the output directory path, specifically includes the following steps: When the log de-identification program starts, it checks whether a de-identified log file exists in the target log directory based on the output directory path. If a de-identified log file exists in the target log directory, it obtains the latest timestamp from the de-identified log file. It then obtains the log file from the source log directory, which contains multiple log lines. For each log line in the log file, it extracts the modification time corresponding to each log line. Log lines whose modification time is greater than the latest timestamp are identified as log lines to be de-identified.

[0048] The target log directory is a file system directory pointed to by the output directory path, representing the storage location of historical de-identified logs, and is used to detect whether de-identified log files exist when the program starts.

[0049] The latest timestamp is a time identifier extracted from the de-identified log files in the target log directory, representing the deadline of the last de-identification process, and is used to filter incremental logs.

[0050] The source log directory is a file system path that stores the original, un-anonymized log files, representing the source location of the original logs, and is used to read the log files to be processed.

[0051] The log file is a text file stored in the source log directory. It contains multiple log lines generated by time and represents the storage medium of the original logs, providing the original data for de-identification processing.

[0052] The modification time is the generation or last modification time of each log line in the log file, representing the time attribute of the log line. It is used to compare with the latest timestamp to filter incremental logs. For example, if the modification time of a log line is "2025-08-09 09:30:00", and it is greater than the latest timestamp, it is determined to be a log line to be de-identified.

[0053] In one example, taking the scenario of de-identifying audit logs from a financial transaction system as an example, the specific execution is as follows: After the log de-identification program starts, it reads the output directory path in the configuration file and checks whether de-identified log files exist in that path (target log directory). If they exist, it extracts the latest timestamp "2025-08-09 09:00:00". Then, it accesses the source log directory and reads the log file, which contains 1000 account operation log lines. It extracts the modification time of each log line line by line. For example, the modification time of log line 501 is "2025-08-09 09:10:00", and log line 800 is "2025-08-09 09:45:00", both of which are greater than the latest timestamp. The modification time of log line 300 is "2025-08-09 08:50:00", which is less than the latest timestamp. Finally, 500 log lines with a modification time greater than "2025-08-09 09:00:00" were selected and identified as log lines to be anonymized. This example uses timestamp comparison to achieve incremental filtering, avoiding redundant calculations on the first 300 processed log lines and improving processing efficiency.

[0054] This application's embodiments locate the target log directory by outputting the directory path and extracting the latest timestamp. Combined with the modification time of the source log lines, incremental log lines awaiting de-identification are filtered. Only incremental data that has not yet been de-identified is processed, avoiding redundant calculations caused by full scans and significantly saving server computing power and processing time. Simultaneously, the timestamp-based filtering logic precisely adapts to the continuously generated nature of audit logs, ensuring that only newly added log lines are processed. This satisfies real-time de-identification requirements while avoiding invalid processing of historical logs, effectively solving the problem of low incremental processing efficiency in existing technologies and improving the overall processing efficiency of log de-identification.

[0055] In some optional implementations, step 203, if a de-identified log file exists in the target log directory, retrieves the latest timestamp from the de-identified log file, specifically including the following steps: If a de-identified log file exists in the target log directory, read the metadata information of the de-identified log file; extract timestamp data from the metadata information, the timestamp data contains multiple time records; sort the multiple time records according to time order to obtain a timestamp sequence; and determine the latest timestamp of the de-identified log file based on the timestamp sequence.

[0056] Metadata information refers to data describing the attributes of the de-identified log file itself, not the business log content within the log file. It is usually automatically recorded by the file system and associated with the file, including core attributes such as file creation time, last modification time, last access time, file size, storage path, and permission information. It is a key data set that characterizes the basic state of the file.

[0057] The timestamp data is a subset of time-related data filtered from the metadata information of the de-identified log file. It represents the record of key time nodes in the file's life cycle, including file creation timestamp, last modification timestamp, last access timestamp, etc.

[0058] Among them, multiple time records refer to specific time entries that constitute timestamp data. Each record corresponds to a specific time node and exists in the form of "time type + time value". It can be the file-level time of the default record in the metadata (such as creation, modification, and access time) or the business-level time that the program writes to the metadata (such as the generation time of the last log in the file).

[0059] A timestamp sequence is an ordered set of times formed by sorting multiple time records in chronological order, such as from morning to night. It is a structured organization of scattered time records, and each time node has a unique position in the sequence.

[0060] In one example, taking the processing of de-identified log files in a financial trading system as an example, when the log de-identification program detects the existence of the de-identified log file in the target log directory, it first calls the file system interface to read its metadata information. This information includes the file creation time "2025-08-09 08:00:00", the last modification time "2025-08-09 18:30:00", the last access time "2025-08-09 18:40:00", and the "last log write time: 2025-08-09 18:29:55" pre-stored by the program. Next, the four time-related entries mentioned above are extracted from the metadata as timestamp data, which include the four time records: "2025-08-09 08:00:00", "2025-08-09 18:30:00", "2025-08-09 18:40:00", and "2025-08-09 18:29:55". These timestamps are then sorted from earliest to latest to obtain the timestamp sequence [2025-08-09 08:00:00, 2025-08-09 18:29:55, 2025-08-09 18:30:00, 2025-08-09 18:40:00]. Finally, the last character in the sequence, "2025-08-09 18:40:00", is taken as the latest timestamp of the de-identified log file for subsequent filtering of incremental logs.

[0061] This application's embodiments extract timestamp data by reading the metadata information of the anonymized log file. This eliminates the need to open and scan massive amounts of business logs within the file, significantly reducing data read volume, lowering server I / O consumption, and improving timestamp acquisition efficiency. Simultaneously, by collecting and sorting multiple time records to form a timestamp sequence, it comprehensively covers multi-dimensional time characteristics such as file creation, modification, and business writes, avoiding potential biases from single time records. Finally, the latest timestamp determined based on the sequence accurately reflects the deadline of the last log processing, providing a reliable basis for subsequently filtering incremental logs with "modification times greater than the latest timestamp," effectively avoiding duplicate processing of historical logs and further improving overall anonymization efficiency.

[0062] In some optional implementations, step S202, generating the line number for each log line to be de-identified using an atomic counter, specifically includes the following steps: Initialize the atomic counter, setting its initial value to a preset value; for each log line to be de-identified, trigger the atomic counter's auto-increment operation based on the preset value to obtain the corresponding line number.

[0063] The preset value refers to the initial value set in advance according to the log processing scenario requirements before initializing the atomic counter. It can be stored in the configuration module of the log desensitization program or in the code preset logic and can be flexibly adjusted according to the actual business scenario. Its core function is to provide the starting counting benchmark for the atomic counter and ensure that the generated line sequence number meets the log traceability requirements.

[0064] The auto-increment operation refers to the thread-safe incrementing of the atomic counter for each log line to be anonymized in a multi-threaded environment. This increment is guaranteed by the atomic counter's internal synchronization mechanism, preventing counting conflicts (such as duplicate or missing numbers) caused by concurrent multi-threaded operations. Starting with a preset value, each trigger increments the value by 1, directly generating a line number that corresponds uniquely to each log line to be anonymized. This is the core action for achieving unique identification of the line number, ensuring that even when multiple threads process logs simultaneously, consecutive and unique line numbers are generated.

[0065] In one example, taking the anonymization of medical record access logs in a medical system as an example, when the program initializes the atomic counter, the default value is set to 1 because this is the first time processing this type of log. When the first log line to be anonymized ("2025-08-10 09:00 Doctor A accessed Patient B's medical record") is read, the increment operation is triggered, and the counter changes from 1 to 2, with the line number being 1. When the second log line is read, the counter increments again to 3, with the line number being 2. Even if multiple threads read the logs simultaneously, the increment operation of the atomic counter can ensure that the line numbers are 1, 2, 3... without repetition, accurately binding each log line.

[0066] This application's embodiment initializes the atomic counter with a preset value, flexibly adapting to scenarios of initial processing or continuation of historical logs, ensuring the continuity of line numbers. The thread safety of the auto-increment operation avoids sequence number conflicts caused by multi-threaded concurrency, ensuring that each log line has a unique identifier. The combination of these two aspects makes line number generation efficient and accurate, providing a reliable basis for subsequent de-identified log tracing and orderly writing, and solving the sequence number chaos problem of traditional counting methods.

[0067] In some optional implementations, step 205, based on the de-identification rules, involves replacing sensitive data in the read sequence-numbered messages to obtain the de-identified messages. This specifically includes the following steps: Based on preset identification rules, the read serial numbered messages are analyzed to identify sensitive data; based on desensitization rules, the sensitive data is classified and processed to determine multiple sensitive data categories and replacement methods for each sensitive category; the sensitive data is replaced using the replacement method to obtain the desensitized messages.

[0068] The identification rules refer to the predefined and stored sensitive data judgment logic in the log desensitization program. They are the core basis for distinguishing between ordinary and sensitive data and can be designed in conjunction with dimensions such as data format, characteristic keywords, and field attributes. For example, the identification rules for ID card numbers can be set as "18 digits, the first 6 digits are the administrative division code, the 7th-14th digits are the date of birth, and the 17th digit is the gender identifier"; the identification rules for mobile phone numbers can be set as "11 digits, starting with 13, 14, 15, 17, 18, or 19". Their function is to allow the program to automatically scan messages with serial numbers, accurately locate sensitive data that needs to be desensitized, and avoid omissions or misjudgments.

[0069] Among them, analysis refers to the process by which the log desensitization program scans, matches, and verifies the text content in numbered messages according to preset recognition rules, covering actions such as data format parsing, feature extraction, and rule comparison.

[0070] Sensitive data refers to data contained in serial numbered messages that may endanger personal privacy or corporate security if leaked. It is the core object of desensitization processing and can cover personal identity information (ID number, name, address), financial information (bank card number, payment password, balance), contact information (mobile phone number, email, social media account), business sensitive information (medical record number, insurance policy number, transaction serial number), etc.

[0071] Among them, classification processing refers to the process of dividing the identified sensitive data into different categories according to data type based on desensitization rules, and matching a specific replacement method for each category of data. This is a key step in achieving "differentiated desensitization". For example, after the program identifies three categories of sensitive data, such as "Name: Zhang San", "Mobile Number: 13812345678" and "Bank Card Number: 6228480402561234567", it will classify them into "Name", "Mobile Number", and "Bank Card Number" categories, and then correspond to the replacement methods of "surname visible, first name hidden", "first 3 and last 4 visible, middle hidden", and "first 6 and last 4 visible, middle hidden" respectively, to ensure that the desensitization of different types of sensitive data is both compliant and retains the necessary distinguishability.

[0072] Among these, multiple sensitive categories refer to sensitive data groups with different data characteristics and desensitization requirements obtained after classification and processing. Each data group corresponds to a specific type of information, reflecting the diversity and differentiated processing needs of sensitive data. For example, in the desensitization scenario of financial transaction logs, multiple sensitive categories include "mobile phone number," "bank card number," "ID card number," and "transaction password"; in the medical log scenario, they include "patient name," "medical record number," "diagnosis result," and "medication record." Different categories require different replacement methods to balance security and log usability.

[0073] Replacement methods refer to predefined methods for hiding sensitive information for different sensitive categories of data. They are specific means of achieving data anonymization and must balance data security (avoiding leakage) with log value (preserving key features needed for analysis and traceability). Replacement methods can include partial replacement, format replacement, encryption replacement, symbol replacement, etc.

[0074] In one example, let's take the anonymized message from a financial transaction system, "Serial Number 101: 2025-08-10 10:30, User Name: Li Si, Mobile Number: 13987654321, Bank Card Number: 6226091234567890123, Transfer Amount: 8000 yuan," as an example. The first step is that the program analyzes the message according to preset recognition rules: based on "11 digits starting with 13, 14, etc.", "13987654321" is identified as sensitive data; based on "16-19 digits including the issuing bank's identifier," "6226091234567890123" is identified as sensitive data; and based on "2-4 Chinese characters often paired with the keyword 'name'," "Li Si" is identified as sensitive data. The second step is to classify and process according to the desensitization rules: classify "Li Si" as "Name" (replacement method: surname visible, given name hidden), classify "13987654321" as "Mobile Number" (replacement method: first 3 and last 4 visible, middle hidden), and classify "6226091234567890123" as "Bank Card Number" (replacement method: first 6 and last 4 visible, middle hidden). The third step is to perform the replacement operation: replace "Li Si" with "Li*", "13987654321" with "1394321", and "6226091234567890123" with "622609*0123". The final result is the de-identified message "Serial number 101: 2025-08-10 10:30, User name: Li*, Mobile number: 139*4321, Bank card number: 622609**0123, Transfer amount: 8000 yuan".

[0075] This application's embodiments analyze numbered messages using identification rules, enabling automated and accurate identification of sensitive data. This avoids omissions and misjudgments inherent in manual analysis, significantly improving desensitization efficiency, especially suitable for batch processing scenarios involving massive amounts of logs. Categorization and the division of multiple sensitive categories allow for differentiated replacement methods for different types of sensitive data. This avoids the loss of log value caused by a single, crude desensitization approach (e.g., only hiding the middle part of a phone number while retaining the beginning and end features for easy traceability), while ensuring that the desensitization of various types of sensitive data complies with compliance requirements. Clear replacement methods standardize and reproducible the desensitization process, resulting in high consistency in desensitization results across different batches of logs. Simultaneously, the replaced data retains necessary non-sensitive features, preserving the logs' usability in scenarios such as troubleshooting and business auditing, effectively balancing data security and log availability.

[0076] In some optional implementations, step 206, based on the line number, writes the de-identified log lines into the de-identified log file in an orderly manner, specifically including the following steps: Based on the row number, the de-identified log rows are placed into a priority queue, which sorts the de-identified log rows according to their row number. The write thread reads the de-identified log rows in the priority queue in order of their row numbers to obtain ordered de-identified log row data, which is then written to the de-identified log file.

[0077] The priority queue is a thread-safe data storage component with automatic sorting capabilities. Deployed in the ordered processing module of a log de-identification program, its core feature is its ability to sort stored de-identified log lines in real time according to preset priority rules, such as line number, ensuring that elements within the queue remain in an ordered state. Unlike the "first-in, first-out" rule of ordinary queues, the priority queue prioritizes outputting the de-identified log line with the highest priority, i.e., the smallest line number. Even if multiple threads concurrently store unordered de-identified log lines, the queue can quickly adjust the order through an internal sorting mechanism, such as binary heap sort, to avoid element chaos.

[0078] The write thread is an independent execution unit within the log masking program, specifically responsible for writing ordered masked log lines to a file. It operates independently of the thread pool handling masking tasks, exhibiting single-threaded serial execution characteristics, and is deployed within the program's log output module. Its core function is to read masked log lines sequentially from a priority queue, avoiding issues such as overlapping content and disordered order caused by multi-threaded file writing. Simultaneously, it can perform file creation and content appending operations according to file writing rules, such as log rolling strategies.

[0079] The ordered de-identified log lines are a structured log collection formed by the writing thread reading them sequentially from a priority queue according to their line numbers. Each log line contains the de-identified business content and its corresponding line number, representing the final ordered form of the de-identified log. Its order is completely consistent with the generation order of the original log lines to be de-identified. This data collection retains the log's business information (such as operation time and content) while ensuring temporal correlation through line numbers.

[0080] In one example, taking the de-identified log processing of an insurance claims system as an example, after multi-threaded de-identification, three unordered de-identified log lines are generated: "Line No. 2 - 2025-08-10 10:05 Claim application: User A, Policy No. 1234", "Line No. 1 - 2025-08-10 10:00 Login operation: User B", and "Line No. 3 - 2025-08-10 10:10 Underwriting approved: Policy 1234". First, the program stores the three log lines into a priority queue based on their line numbers. The queue is automatically sorted by line number from smallest to largest, with the internal order adjusted to "Line No. 1 log - Line No. 2 log - Line No. 3 log". Second, an independent writing thread starts, reading the sorted log lines sequentially from the priority queue. First, it reads the login operation log of line No. 1 and writes it to the de-identified log file in the format of "time + content". Then, it reads the claim application log of line No. 2 and appends it. Finally, the underwriting for line number 3 was read and appended via the log. The final log order in the file was completely consistent with the original operation sequence, meeting the audit requirements of the subsequent claims process.

[0081] The priority queue in this embodiment can automatically sort the de-identified log lines by line number, effectively solving the log disorder problem caused by multi-threaded de-identification. Even if multiple threads concurrently output logs with different line numbers, the queue can adjust the order in real time to ensure the temporal correlation of the logs and avoid affecting the usability of auditing and traceability scenarios due to disordered order. The independent writing thread adopts a single-threaded serial writing method to avoid problems such as content overlap and format disorder caused by multiple threads operating on the file at the same time, ensuring the security and stability of file writing. The combination of priority queue and writing thread makes the finally written de-identified log line data in the file orderly, which not only retains the temporal characteristics of the original log, but also ensures the integrity of the file content, perfectly solving the contradiction of "multi-threaded efficiency and log order cannot be achieved at the same time" in the prior art.

[0082] In some optional implementations, after step 206, where the de-identified log lines are written sequentially to the de-identified log file based on the line number, the following steps are also included: The system monitors the log size and write interval of the de-identified log file in real time. When the log size of the de-identified log file is greater than or equal to a preset size threshold, or the write interval is greater than or equal to a preset time interval, the system stops writing data to the de-identified log file. When a new de-identified log line is detected, a new de-identified log file is generated. Based on the line number corresponding to the new de-identified log line, the system writes the new de-identified log line to the new de-identified log file in an orderly manner through the write thread.

[0083] Among them, the new de-identified log line refers to the de-identified log record generated by the log de-identification program after the de-identified log file stops being written, which has not yet been stored in any de-identified log file. In essence, it is a log unit with a serial number that has completed the replacement of sensitive data.

[0084] Among them, the new de-identified log file refers to the file automatically created by the log de-identification program to store new de-identified log lines after the de-identified log file stops writing due to reaching the size threshold or the writing interval threshold.

[0085] In one example, taking the de-identification scenario of medical record access logs in a medical system as an example, the current de-identified log file size threshold in the configuration file is 500MB, the write interval threshold is 2 hours, and the output directory path is "xxxxx". First, the program monitors the current de-identified log file in real time. When the file size reaches 500MB (stores de-identified log lines with sequence numbers 1-50000), it immediately stops writing data to it. Second, 10 minutes later, the program processes 100 newly added original medical record access logs in the source log directory, generating new de-identified log lines with sequence numbers 50001-50100. After detecting these new log lines, it automatically creates new de-identified log files in the output directory. Third, the write thread appends the new de-identified log lines sequentially to the new de-identified log file according to the line sequence numbers 50001-50100, ensuring that the log order in the new file is consistent with the original access sequence. If the new de-identified log file does not reach 500MB within the next 2 hours, but the write interval is 2 hours, the program will stop writing and create a new de-identified log file to store subsequent new log lines.

[0086] This application's embodiments, on the one hand, avoid storage risks caused by excessively large single de-identified log files by real-time monitoring of log size and write intervals and triggering file rolling, such as file corruption leading to significant data loss and management difficulties, including slow opening and time-consuming transmission. Simultaneously, splitting files by thresholds facilitates time-segmented log archiving, such as daily or file sequence number backups and precise queries; for example, locating logs within a specific hour only requires opening the corresponding file. On the other hand, upon detecting a new de-identified log line, a new file is automatically generated and written sequentially, ensuring continuous and uninterrupted log data storage. Furthermore, the sequence numbers of the new file and the original file are consistent, guaranteeing the integrity of the log timeline and preventing log omissions or sequence breaks due to file switching. Moreover, this mechanism requires no manual intervention, achieving automated management of de-identified log storage and significantly reducing operational costs.

[0087] It should be emphasized that, in order to further ensure the privacy and security of the above-mentioned concurrent thread count, de-identification rules, queue size parameters, output directory path, line number, and de-identified log line, the above-mentioned concurrent thread count, de-identification rules, queue size parameters, output directory path, line number, and de-identified log line can also be stored in a blockchain node.

[0088] The blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0089] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0090] Further reference Figure 3 As a response to the above Figure 2 To implement the method shown, this application provides an embodiment of a log desensitization device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0091] like Figure 3 As shown, the log desensitization device 400 of this embodiment includes: a first acquisition module 401, a second acquisition module 402, an assembly module 403, a reading module 404, a replacement module 405, and a first writing module 406. Wherein: The first acquisition module 401 is used to obtain the number of concurrent threads, the desensitization rules, the queue size parameter and the output directory path from the configuration file of the log desensitization program; The second acquisition module 402 is used to acquire multiple log lines to be de-identified based on the output directory path, and generate a line number for each log line to be de-identified through an atomic counter; Assembly module 403 is used to assemble each log line to be de-identified and its line number, generate multiple messages with sequence numbers, and put the multiple messages with sequence numbers into a blocking queue. The capacity of the blocking queue is determined by the queue size parameter. The read module 404 is used to create a thread pool based on the number of concurrent threads, and read multiple messages with sequence numbers from the blocking queue through each thread in the thread pool; The replacement module 405 is used to replace sensitive data in multiple read messages with sequence numbers based on the de-identification rules, so as to obtain multiple de-identified messages. Each de-identified message includes a line sequence number and a de-identification log line. The first writing module 406 is used to write the de-identified log lines into the de-identified log file in an orderly manner based on the line sequence number.

[0092] This application's embodiments address the core shortcomings of existing log anonymization technologies through multi-dimensional technical design. Specifically, firstly, an atomic counter generates a unique line number for each log line to be anonymized. This line number accompanies the entire log processing flow, serving as an identifier linking the original log and the anonymized log. The line number allows for rapid location of anonymized log records, completely resolving the difficulty of data traceability after anonymization. Secondly, the anonymized log file is detected based on the output directory path, and the latest timestamp is extracted. Only incremental log lines modified after a time greater than this timestamp are processed, avoiding redundant calculations caused by full processing. Simultaneously, the line number ensures ordered writing of anonymized log lines, resolving log order disorder issues caused by multi-threaded processing and guaranteeing the temporal correlation of audit logs. Thirdly, the number of concurrent threads is read from the configuration file to flexibly adjust processing concurrency capabilities. Combined with queue size parameters to limit the capacity of the blocking queue, data buffering and flow control are implemented to prevent program crashes caused by data accumulation, improving the resource adaptability and operational stability of the anonymization process.

[0093] In one embodiment, the second acquisition module 402 includes: The detection submodule is used to detect whether de-identified log files exist in the target log directory based on the output directory path when the log de-identification program starts. The first acquisition submodule is used to obtain the latest timestamp from the de-identified log file if a de-identified log file exists in the target log directory; The second acquisition submodule is used to retrieve log files from the source log directory. The log files contain multiple log lines. The extraction submodule is used to extract the modification time corresponding to each log line in the log file; The "Determine" submodule is used to identify log lines whose modification time is greater than the latest timestamp as log lines to be de-identified.

[0094] This application's embodiments locate the target log directory by outputting the directory path and extracting the latest timestamp. Combined with the modification time of the source log lines, incremental log lines awaiting de-identification are filtered. Only incremental data that has not yet been de-identified is processed, avoiding redundant calculations caused by full scans and significantly saving server computing power and processing time. Simultaneously, the timestamp-based filtering logic precisely adapts to the continuously generated nature of audit logs, ensuring that only newly added log lines are processed. This satisfies real-time de-identification requirements while avoiding invalid processing of historical logs, effectively solving the problem of low incremental processing efficiency in existing technologies and improving the overall processing efficiency of log de-identification.

[0095] In one embodiment, the first acquisition submodule is further configured to: if a de-identified log file exists in the target log directory, read the metadata information of the de-identified log file; extract timestamp data from the metadata information, the timestamp data containing multiple time records; sort the multiple time records based on time order to obtain a timestamp sequence; and determine the latest timestamp of the de-identified log file based on the timestamp sequence.

[0096] This application's embodiments extract timestamp data by reading the metadata information of the anonymized log file. This eliminates the need to open and scan massive amounts of business logs within the file, significantly reducing data read volume, lowering server I / O consumption, and improving timestamp acquisition efficiency. Simultaneously, by collecting and sorting multiple time records to form a timestamp sequence, it comprehensively covers multi-dimensional time characteristics such as file creation, modification, and business writes, avoiding potential biases from single time records. Finally, the latest timestamp determined based on the sequence accurately reflects the deadline of the last log processing, providing a reliable basis for subsequently filtering incremental logs with "modification times greater than the latest timestamp," effectively avoiding duplicate processing of historical logs and further improving overall anonymization efficiency.

[0097] In one embodiment, the second acquisition module 402 includes: The initialization submodule is used to initialize the atomic counters, and the initial value of the atomic counters is set to a preset value; The trigger submodule is used to trigger the auto-increment operation of the atomic counter for each log line to be de-identified, based on a preset value, to obtain the corresponding line number.

[0098] This application's embodiment initializes the atomic counter with a preset value, flexibly adapting to scenarios of initial processing or continuation of historical logs, ensuring the continuity of line numbers. The thread safety of the auto-increment operation avoids sequence number conflicts caused by multi-threaded concurrency, ensuring that each log line has a unique identifier. The combination of these two aspects makes line number generation efficient and accurate, providing a reliable basis for subsequent de-identified log tracing and orderly writing, and solving the sequence number chaos problem of traditional counting methods.

[0099] In one embodiment, the replacement module 405 includes: The analysis submodule is used to analyze the read serial numbered messages according to preset identification rules to determine sensitive data; The classification submodule is used to classify sensitive data according to the de-identification rules, determine the sensitive data in multiple sensitive categories and the replacement method for each sensitive category; The Replacement submodule is used to replace sensitive data using a replacement method to obtain de-identified messages.

[0100] This application's embodiments analyze numbered messages using identification rules, enabling automated and accurate identification of sensitive data. This avoids omissions and misjudgments inherent in manual analysis, significantly improving desensitization efficiency, especially suitable for batch processing scenarios involving massive amounts of logs. Categorization and the division of multiple sensitive categories allow for differentiated replacement methods for different types of sensitive data. This avoids the loss of log value caused by a single, crude desensitization approach (e.g., only hiding the middle part of a phone number while retaining the beginning and end features for traceability), while ensuring that the desensitization of various sensitive data complies with compliance requirements. Clear replacement methods standardize and reproducible the desensitization process, resulting in high consistency across different batches of logs. Simultaneously, the replaced data retains necessary non-sensitive features, preserving the logs' usability in scenarios such as troubleshooting and business auditing, effectively balancing data security and log availability.

[0101] In one embodiment, the first writing module 406 includes: This is placed into a submodule, which is used to put the de-identified log lines into a priority queue based on the line number. The priority queue sorts the de-identified log lines according to the line number. The read submodule is used to read the de-identified log lines in the priority queue in the order of their line numbers using the write thread, and then write the ordered de-identified log line data to the de-identified log file.

[0102] The priority queue in this embodiment can automatically sort the de-identified log lines by line number, effectively solving the log disorder problem caused by multi-threaded de-identification. Even if multiple threads concurrently output logs with different line numbers, the queue can adjust the order in real time to ensure the temporal correlation of the logs and avoid affecting the usability of auditing and traceability scenarios due to disordered order. The independent writing thread adopts a single-threaded serial writing method to avoid problems such as content overlap and format disorder caused by multiple threads operating on the file at the same time, ensuring the security and stability of file writing. The combination of priority queue and writing thread makes the finally written de-identified log line data in the file orderly, which not only retains the temporal characteristics of the original log, but also ensures the integrity of the file content, perfectly solving the contradiction of "multi-threaded efficiency and log order cannot be achieved at the same time" in the prior art.

[0103] In one embodiment, the log desensitization device 400 further includes: The monitoring module is used to monitor the log size and write interval of the de-identified log files in real time. The first write module is stopped when the log size of the de-identified log file is greater than or equal to a preset size threshold, or the write interval is greater than or equal to a preset time interval. The generation module is used to generate a new de-identified log file when a new de-identified log line is detected. The second writing module is used to write the new de-identified log lines into the new de-identified log file in an orderly manner through the writing thread, based on the line number corresponding to the new de-identified log line.

[0104] This application's embodiments, on the one hand, avoid storage risks caused by excessively large single de-identified log files by real-time monitoring of log size and write intervals and triggering file rolling, such as file corruption leading to significant data loss and management difficulties, including slow opening and time-consuming transmission. Simultaneously, splitting files by thresholds facilitates time-segmented log archiving, such as daily or file sequence number backups and precise queries; for example, locating logs within a specific hour only requires opening the corresponding file. On the other hand, upon detecting a new de-identified log line, a new file is automatically generated and written sequentially, ensuring continuous and uninterrupted log data storage. Furthermore, the sequence numbers of the new file and the original file are consistent, guaranteeing the integrity of the log timeline and preventing log omissions or sequence breaks due to file switching. Moreover, this mechanism requires no manual intervention, achieving automated management of de-identified log storage and significantly reducing operational costs.

[0105] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0106] Computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected via a system bus. It should be noted that only computer device 6 with memory 61, processor 62, and network interface 63 is shown in the figure; however, it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described herein is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0107] Computer devices can include desktop computers, laptops, handheld computers, and cloud servers. These devices allow for human-computer interaction with users through keyboards, mice, remote controls, touchpads, or voice-activated devices.

[0108] The memory 61 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 61 may be an internal storage unit of the computer device 6, such as the hard disk or memory of the computer device 6. In other embodiments, the memory 61 may also be an external storage device of the computer device 6, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 6. Of course, the memory 61 may include both the internal storage unit and the external storage device of the computer device 6. In this embodiment, the memory 61 is typically used to store the operating system and various application software installed on the computer device 6, such as computer-readable instructions for log desensitization methods. In addition, the memory 61 may also be used to temporarily store various types of data that have been output or will be output.

[0109] In some embodiments, processor 62 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. Processor 62 is typically used to control the overall operation of computer device 6. In this embodiment, processor 62 is used to execute computer-readable instructions stored in memory 61 or to process data, such as computer-readable instructions for executing a log desensitization method.

[0110] The network interface 63 may include a wireless network interface or a wired network interface, which is typically used to establish a communication connection between the computer device 6 and other electronic devices.

[0111] This application's embodiments address the core shortcomings of existing log anonymization technologies through multi-dimensional technical design. Specifically, firstly, an atomic counter generates a unique line number for each log line to be anonymized. This line number accompanies the entire log processing flow, serving as an identifier linking the original log and the anonymized log. The line number allows for rapid location of anonymized log records, completely resolving the difficulty of data traceability after anonymization. Secondly, the anonymized log file is detected based on the output directory path, and the latest timestamp is extracted. Only incremental log lines modified after a time greater than this timestamp are processed, avoiding redundant calculations caused by full processing. Simultaneously, the line number ensures ordered writing of anonymized log lines, resolving log order disorder issues caused by multi-threaded processing and guaranteeing the temporal correlation of audit logs. Thirdly, the number of concurrent threads is read from the configuration file to flexibly adjust processing concurrency capabilities. Combined with queue size parameters to limit the capacity of the blocking queue, data buffering and flow control are implemented to prevent program crashes caused by data accumulation, improving the resource adaptability and operational stability of the anonymization process.

[0112] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the log desensitization method described above.

[0113] This application's embodiments address the core shortcomings of existing log anonymization technologies through multi-dimensional technical design. Specifically, firstly, an atomic counter generates a unique line number for each log line to be anonymized. This line number accompanies the entire log processing flow, serving as an identifier linking the original log and the anonymized log. The line number allows for rapid location of anonymized log records, completely resolving the difficulty of data traceability after anonymization. Secondly, the anonymized log file is detected based on the output directory path, and the latest timestamp is extracted. Only incremental log lines modified after a time greater than this timestamp are processed, avoiding redundant calculations caused by full processing. Simultaneously, the line number ensures ordered writing of anonymized log lines, resolving log order disorder issues caused by multi-threaded processing and guaranteeing the temporal correlation of audit logs. Thirdly, the number of concurrent threads is read from the configuration file to flexibly adjust processing concurrency capabilities. Combined with queue size parameters to limit the capacity of the blocking queue, data buffering and flow control are implemented to prevent program crashes caused by data accumulation, improving the resource adaptability and operational stability of the anonymization process.

[0114] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of the various embodiments of this application.

[0115] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

[0116] The software tools or components not belonging to our company that appear in the embodiments of this application are merely examples and do not represent actual use.

Claims

1. A log anonymization method, characterized in that, Includes the following steps: Obtain the number of concurrent threads, desensitization rules, queue size parameters, and output directory path from the configuration file of the log desensitization program; Based on the output directory path, multiple log lines to be de-identified are obtained, and a line number for each log line to be de-identified is generated using an atomic counter; Each log line to be de-identified and its line number are assembled to generate multiple messages with sequence numbers. The multiple messages with sequence numbers are then placed into a blocking queue, the capacity of which is determined by the queue size parameter. A thread pool is created based on the number of concurrent threads, and each thread in the thread pool reads the multiple messages with sequence numbers from the blocking queue. Based on the desensitization rules, sensitive data is replaced in the multiple numbered messages to obtain multiple desensitized messages. Each desensitized message includes the row number and the desensitized log row. Based on the row number, the de-identified log rows are written sequentially into the de-identified log file.

2. The method according to claim 1, characterized in that, The step of obtaining multiple log lines to be de-identified based on the output directory path specifically includes: When the log de-identification program starts, it checks whether de-identified log files exist in the target log directory based on the output directory path; If the de-identified log file exists in the target log directory, then obtain the latest timestamp from the de-identified log file; Obtain a log file from the source log directory, the log file containing multiple log lines; For each log line in the log file, extract the modification time corresponding to each log line; Log lines whose modification time is greater than the latest timestamp are identified as log lines to be de-identified.

3. The method according to claim 2, characterized in that, The step of obtaining the latest timestamp from the de-identified log file if it exists in the target log directory specifically includes: If the de-identified log file exists in the target log directory, then read the metadata information of the de-identified log file; Extract timestamp data from the metadata information; the timestamp data contains multiple time records. Based on the time sequence, the multiple time records are sorted to obtain a timestamp sequence, and the latest timestamp of the de-identified log file is determined according to the timestamp sequence.

4. The method according to claim 1, characterized in that, The step of generating the line number for each log line to be de-identified using an atomic counter specifically includes: Initialize the atomic counter, and set the initial value of the atomic counter to a preset value; For each log line to be de-identified, the atomic counter is incremented based on a preset value to obtain the corresponding line number.

5. The method according to claim 1, characterized in that, The step of replacing sensitive data in the read sequence-numbered message based on the desensitization rules to obtain the desensitized message specifically includes: According to preset identification rules, the read serial numbered messages are analyzed to determine sensitive data; According to the desensitization rules, the sensitive data is classified and processed to determine the sensitive data of multiple sensitive categories and the replacement method for each sensitive category; The sensitive data is replaced using the aforementioned replacement method to obtain the de-identified message.

6. The method according to claim 1, characterized in that, The step of writing the de-identified log lines into the de-identified log file in an orderly manner based on the line number specifically includes: Based on the row number, the de-identified log row is placed into a priority queue, and the priority queue sorts the de-identified log rows according to the row number. The write thread reads the de-identified log lines in the priority queue in the order of the line numbers to obtain ordered de-identified log line data, which is then written to the de-identified log file.

7. The method according to claim 1, characterized in that, After the step of writing the de-identified log lines into the de-identified log file in an orderly manner based on the line number, the method further includes: Real-time monitoring of the log size and write interval of the de-identified log file; When the log size of the de-identified log file is greater than or equal to a preset size threshold, or when the writing interval is greater than or equal to a preset time interval, stop writing data to the de-identified log file; When a new de-identified log line is detected, a new de-identified log file is generated; Based on the row number corresponding to the new de-identified log row, the new de-identified log row is written to the new de-identified log file in an orderly manner by the write thread.

8. A log desensitization device, characterized in that, include: The first acquisition module is used to obtain the number of concurrent threads, desensitization rules, queue size parameters and output directory path from the configuration file of the log desensitization program; The second acquisition module is used to acquire multiple log lines to be de-identified based on the output directory path, and generate a line number for each log line to be de-identified using an atomic counter; An assembly module is used to assemble each log line to be de-identified and the line number to generate multiple messages with sequence numbers, and put the multiple messages with sequence numbers into a blocking queue. The capacity of the blocking queue is determined by the queue size parameter. The reading module is used to create a thread pool based on the number of concurrent threads, and read the multiple messages with sequence numbers from the blocking queue through each thread in the thread pool; The replacement module is used to replace sensitive data in the multiple numbered messages read based on the desensitization rules, so as to obtain multiple desensitized messages. Each desensitized message includes the row number and the desensitized log row. The first writing module is used to write the de-identified log lines into the de-identified log file in an orderly manner based on the line number.

9. A computer device, characterized in that, It includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the log de-identification method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the log desensitization method as described in any one of claims 1 to 7.

Images