Log desensitization method and device, storage medium and electronic device

Abstract

The invention discloses a log desensitization method and device, a storage medium, an electronic device and a computer program product. The method comprises the steps that log data to be desensitized are intercepted; screening sensitive texts in the log data by adopting a preset first filtering rule according to the application scene of the log data, and identifying first sensitive data in the log data; screening sensitive fields in the log data by adopting a preset second filtering rule according to the data structure of the log data, and identifying second sensitive data in the log data; the first sensitive data and the second sensitive data in the log data are replaced with corresponding hash values through a preset hash function, desensitized target log data are obtained, and the hash function is used for indicating the conversion relation between the sensitive data and the hash values. Therefore, the security of the log data is improved.

Description

Technical Field

[0001] This application relates to the field of data security, and more specifically, to a log de-identification method and apparatus, storage medium, electronic device, and computer program product. Background Technology

[0002] With the increasing complexity of the network environment and growing awareness of personal information protection, people have higher and higher requirements for the security of sensitive information. In today's information age, enterprises and organizations are vulnerable to cyberattacks and data breaches when processing log files containing sensitive information.

[0003] In related technologies, sensitive user information is directly stored through log recording.

[0004] However, the protection methods in related technologies are insufficient in protecting sensitive information. Summary of the Invention

[0005] This application provides a log desensitization method and apparatus, storage medium, electronic device, and computer program product.

[0006] According to one aspect of the embodiments of this application, a log de-identification method is provided. The method includes: intercepting log data to be de-identified; filtering sensitive text in the log data according to the application scenario of the log data using a preset first filtering rule to identify first sensitive data in the log data; filtering sensitive fields in the log data according to the data structure of the log data using a preset second filtering rule to identify second sensitive data in the log data; replacing the first sensitive data and the second sensitive data in the log data with corresponding hash values ​​using a preset hash function to obtain de-identified target log data, wherein the hash function is used to indicate the conversion relationship between sensitive data and hash values.

[0007] In an exemplary embodiment, the step of filtering sensitive text in the log data according to the application scenario of the log data using a preset first filtering rule, and identifying the first sensitive data in the log data, includes: determining the application scenario of the log data based on the source and type of the log data, and matching a set of regular expressions corresponding to the application scenario; reading the log data, traversing the log data to find text in the log data that matches the regular expressions in the set of regular expressions; and taking the text that matches the regular expressions in the set of regular expressions as the first sensitive data.

[0008] In an exemplary embodiment, the step of filtering sensitive fields in the log data according to the data structure of the log data using a preset second filtering rule to identify the second sensitive data in the log data includes: according to the data structure of the log data, calling a parsing library corresponding to the data structure to parse the log data and determine the field content of the log data; matching the field content of the log data with a preset field set to extract the sensitive fields in the field content of the log data as the second sensitive data.

[0009] In an exemplary embodiment, replacing the first sensitive data and the second sensitive data in the log data with corresponding hash values ​​using a preset hash function to obtain de-identified target log data includes: determining the data types corresponding to the first sensitive data and the second sensitive data respectively; selecting hash functions corresponding to the first sensitive data and the second sensitive data respectively, based on their respective data types, to ensure the uniqueness of the generated hash values; converting the first sensitive data and the second sensitive data using the hash functions corresponding to the first sensitive data and the second sensitive data respectively, to obtain hash values ​​corresponding to the first sensitive data and the second sensitive data, wherein each of the first sensitive data and each of the second sensitive data corresponds to a unique hash value; locating the target positions of the first sensitive data and the second sensitive data in the log data; and replacing the first sensitive data and the second sensitive data at the corresponding target positions with the hash values ​​corresponding to the first sensitive data and the second sensitive data respectively, to obtain de-identified target log data.

[0010] In an exemplary embodiment, the method further includes: obtaining sensitive information to be queried input by the target object; processing the sensitive information to be queried using the preset hash function to generate a target hash value corresponding to the sensitive information to be queried; traversing the target log data according to the target hash value to query whether there is a hash value matching the target hash value in the target log data; and if target log data containing a hash value matching the target hash value is found, feeding back the found target log data to the target object.

[0011] In an exemplary embodiment, the method further includes: extracting data features of the intercepted log data to be de-identified; determining the business type corresponding to the intercepted log data to be de-identified based on the data features; determining the risk level corresponding to the log data to be de-identified based on the business type; and adjusting the first filtering rule and the second filtering rule corresponding to the log data to be de-identified based on the risk level.

[0012] According to another aspect of the embodiments of this application, a log desensitization device is also provided, the device comprising:

[0013] The interception module is used to intercept log data that needs to be de-identified;

[0014] The first identification module is used to filter sensitive text in the log data according to the application scenario of the log data using a preset first filtering rule, and to identify the first sensitive data in the log data.

[0015] The second identification module is used to filter sensitive fields in the log data according to the data structure of the log data using a preset second filtering rule, and to identify the second sensitive data in the log data;

[0016] The desensitization module is used to replace the first sensitive data and the second sensitive data in the log data with corresponding hash values ​​using a preset hash function to obtain the desensitized target log data. The hash function is used to indicate the conversion relationship between sensitive data and hash values.

[0017] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer-readable storage medium, and the computer program is configured to execute the above-described log desensitization method at runtime.

[0018] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the above-described log desensitization method through the computer program.

[0019] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps of the methods described in various embodiments of this application.

[0020] The aforementioned log anonymization method, which includes intercepting log data, identifying sensitive information, and using hash functions for anonymization, constructs a comprehensive and efficient data protection mechanism. This mechanism not only prevents the direct exposure of sensitive information at the source but also ensures data security through accurate identification and hash replacement, while preserving data availability and queryability. The system can intelligently process data according to specific application scenarios and data structures, thus ensuring user privacy while meeting business needs and compliance requirements, achieving a balance between data security and usability, and improving the security of log data. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a hardware structure block diagram of the log desensitization method according to an embodiment of this application;

[0024] Figure 2 This is a flowchart of a log desensitization method according to an embodiment of this application;

[0025] Figure 3 This is a second flowchart of a log desensitization method according to an embodiment of this application;

[0026] Figure 4 This is a flowchart of a log desensitization method according to an embodiment of this application;

[0027] Figure 5 This is the fourth flowchart of a log desensitization method according to an embodiment of this application;

[0028] Figure 6 This is the fifth flowchart of a log desensitization method according to an embodiment of this application;

[0029] Figure 7 This is a flowchart of a log desensitization method according to an embodiment of this application;

[0030] Figure 8 This is a structural block diagram of a log desensitization device according to an embodiment of this application. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data used in this way can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] The methods and embodiments provided in this application can be executed on a computer terminal or similar computing device. Taking running on a computer terminal as an example, Figure 1 This is a hardware structure block diagram of a computer terminal for the log desensitization method according to an embodiment of this application. For example... Figure 1 As shown, a computer terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor unit (MPU) or a programmable logic device (PLD)) and a memory 104 for storing data are also shown. In one exemplary embodiment, the computer terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that… Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the computer terminal described above. For example, the computer terminal may also include components that are more complex than those described above. Figure 1 The more or fewer components shown, or having the same Figure 1 Equivalent functions or ratios shown Figure 1 The functions shown have more different configurations.

[0034] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the log desensitization method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0035] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the computer terminal. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0036] The collection, storage, use, processing, transmission, provision, and disclosure of financial data or user data involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0037] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0038] This embodiment provides a log desensitization method. Figure 2 This is a flowchart of an optional log desensitization method according to an embodiment of this application, the process including the following steps S200-S230:

[0039] Step S200: Intercept the log data to be de-identified.

[0040] Specifically, before log data is recorded, the system first intercepts it to identify and de-identify sensitive information. This step occurs during the log data transmission path, ensuring that log information is securely protected before storage.

[0041] For example, a log interception plugin can be designed and implemented using the custom plugin feature of the Log4j2 framework. When the application attempts to log, this plugin reads and controls the log data stream before the log output.

[0042] Log data refers to text data generated during application runtime to record status, events, and information. The Log4j2 framework is a widely used logging tool that allows developers to customize logging behavior and formatting, including adding custom plugins.

[0043] Step S210: Based on the application scenario of the log data, a preset first filtering rule is used to filter sensitive text in the log data and identify the first sensitive data in the log data.

[0044] Specifically, the system uses preset first filtering rules to identify and filter sensitive information in text form, such as usernames and phone numbers, based on the specific application scenario of the log data. This filtering process aims to accurately locate sensitive data, laying the foundation for subsequent de-identification processing.

[0045] For example, a regular expression rule library can be built, containing formats and patterns for common sensitive information, such as ID card numbers and bank account numbers. Whenever log data is intercepted, the system uses these regular expressions to match the text content in the logs and identify sensitive information.

[0046] The first filtering rule is a set of preset rules used to identify and filter sensitive text information in log data, implemented using regular expressions. The application scenario refers to the specific environment or business scenario in which the log data is generated, such as user transactions or system logins. Different scenarios may involve different types of sensitive information.

[0047] Step S220: Based on the data structure of the log data, use the preset second filtering rules to filter sensitive fields in the log data and identify the second sensitive data in the log data.

[0048] Specifically, for the structured portions of log data, such as JSON or XML format data, the system uses preset second filtering rules to identify and filter out sensitive fields. This process ensures comprehensive coverage of sensitive information in structured data, improving the comprehensiveness and accuracy of de-identification.

[0049] For example, design a structured data parser capable of reading and parsing log data in JSON or XML format. The parser applies second filtering rules that specify the field names or paths to be examined to identify sensitive information.

[0050] The second filtering rule is specifically designed for sensitive fields in structured log data, and typically includes information such as field name, path, and data type. Sensitive fields are specific fields in structured log data that contain sensitive information.

[0051] Step S230: Replace the first sensitive data and the second sensitive data in the log data with the corresponding hash values ​​using a preset hash function to obtain the desensitized target log data.

[0052] The hash function is used to indicate the conversion relationship between sensitive data and hash values.

[0053] Specifically, after identifying sensitive information, the system uses a preset hash function to replace all sensitive information (including information in text form and structured fields) with the corresponding hash value, thereby generating de-identified log data. This replacement process protects privacy while preserving the unique identifier of the data.

[0054] For example, the system is configured with a hash algorithm, such as SHA-256, to process all identified sensitive information. When the system identifies the first or second sensitive data, it calls the hash function to convert the data into a fixed-length hash value, and then uses these hash values ​​to replace the sensitive information in the original log.

[0055] A hash function is an algorithm that converts an input of arbitrary length into a fixed-length output. It is irreversible and commonly used for data security and privacy protection. A hash value, generated by a hash function, is a fixed-length output value used to replace sensitive information. It uniquely identifies the input information but cannot be reverse-engineered.

[0056] In this embodiment, the method of this application, including intercepting log data, identifying sensitive information (sensitive fields in text and structured data), and using a hash function for de-identification, constructs a comprehensive and efficient data protection mechanism. This mechanism not only prevents the direct exposure of sensitive information at the source but also ensures data security through accurate identification and hash replacement, while preserving data availability and queryability. The system can intelligently process data according to specific application scenarios and data structures, thus ensuring user privacy while also meeting business needs and compliance requirements, achieving a balance between data security and usability, and improving the security of log data.

[0057] In one embodiment, such as Figure 3 As shown, step S210 involves filtering sensitive text in the log data according to the application scenario of the log data using a preset first filtering rule, thereby identifying the first sensitive data in the log data. This includes steps S300-S320:

[0058] Step S300: Determine the application scenario of the log data based on its source and type, and then match it with a set of regular expressions corresponding to the application scenario.

[0059] Specifically, based on the source and type of log data, its application scenario is determined, and then a set of regular expressions matching that scenario is selected for subsequent sensitive information filtering. This mechanism makes log anonymization more accurate and scenario-based, improving the security and targeting of data processing.

[0060] For example, the system pre-defines corresponding sets of regular expressions for different application scenarios. When log data enters the system, its source (such as which system module or service it belongs to) and type (such as transaction logs, security logs, etc.) are analyzed first. Then, based on this information, the corresponding set of regular expressions is searched and applied. For example, for transaction logs, regular expressions containing information such as bank account numbers and credit card numbers may be applied; for security logs, rules containing information such as IP addresses and login attempts may be applied. Through scenario-based matching, the system can more accurately locate and process sensitive information, reducing the possibility of false positives or false negatives and improving the effectiveness of data protection. At the same time, using different rule sets for different application scenarios also enhances the system's flexibility and scalability.

[0061] The application scenario refers to the context in which the log data is generated, such as user transactions, system logins, and security audits. Different scenarios may involve sensitive information of different types and formats. A regular expression set is a predefined set of pattern matching rules used to identify specific formatted information in text, such as ID card numbers and mobile phone numbers.

[0062] Step S310: Read the log data and traverse the log data to find text in the log data that matches the regular expressions in the regular expression set.

[0063] Specifically, after determining the application scenario and rule set, the system begins to read and traverse log data, searching for text that matches a preset regular expression. This text may be the carrier of sensitive information.

[0064] For example, the system designs a log reading and analysis component that can traverse log data line by line or segment by segment. During the traversal, the component applies a selected set of regular expressions to match the log content to identify potentially sensitive information. Traversing the log data and applying regular expressions for matching ensures that the system comprehensively examines the log content, without overlooking any parts that may contain sensitive information.

[0065] Log data reading refers to the process of reading log files from the log generation source or log storage location. Traversal refers to the process of accessing and processing every element in the data structure, ensuring that every part is checked.

[0066] Step S320: Text that matches the regular expressions in the set of regular expressions is taken as the first sensitive data.

[0067] Specifically, once text matching a preset regular expression is found in the log data, the system identifies this text as the first sensitive data, preparing for the next step of desensitization processing.

[0068] For example, in the log analysis component, when text is found to match a rule in the set of regular expressions, the system marks this text as sensitive information and records its position and type in the log, providing clear guidance for subsequent replacement operations.

[0069] The first sensitive data is text information identified in log data that matches a preset regular expression. This information may contain personal privacy or corporate secrets.

[0070] In this embodiment, the system constructs an efficient and accurate sensitive information identification process by determining the application scenario based on the source and type of log data, selecting and applying the corresponding set of regular expressions, and traversing the log data to identify sensitive text. This process not only automatically adjusts the identification strategy according to the scenario, improving the targeting and accuracy of identification, but also ensures that all potentially sensitive information is discovered and processed in a timely manner through comprehensive traversal and marking. Finally, information marked as the first sensitive data will be further processed and replaced with hash values, thereby effectively protecting user privacy and enterprise data security.

[0071] In one embodiment, such as Figure 4 As shown, step S220 involves filtering sensitive fields in the log data according to the data structure of the log data using a preset second filtering rule, thereby identifying the second sensitive data in the log data. This includes steps S400-S410:

[0072] Step S400: Based on the data structure of the log data, call the parsing library corresponding to the data structure to parse the log data and determine the field content of the log data.

[0073] Specifically, based on the data structure characteristics of the received log data, the system calls the corresponding parsing library to perform in-depth analysis of the log data in order to identify and determine the field content in the logs. This process is the foundation of structured data processing, providing structured information for subsequent identification of sensitive fields.

[0074] For example, the system pre-installs various parsing libraries, such as the Jackson library for JSON data and DOM or SAX parsers for XML data. When log data is received, the system first determines its data structure type and then calls the corresponding parsing library. The parsing library decomposes the log data into a set of structured fields, each containing a specific name and value, facilitating subsequent analysis and processing of the field content.

[0075] Data structure refers to the relationships and organization of data elements. Common data structures include JSON and XML. Different data structures are suitable for storing and exchanging different types of data. Field content refers to specific data items in structured data, existing in the form of key-value pairs, such as name, account number, and date.

[0076] Step S410: Match the field content of the log data with a preset field set to extract the sensitive fields in the field content of the log data as the second sensitive data.

[0077] Specifically, after obtaining the field content of the log data, the system compares and matches it with a preset set of sensitive fields to identify fields that may contain sensitive information. These fields will be regarded as secondary sensitive data and used for subsequent de-identification processing.

[0078] For example, the system maintains a set of fields containing various known sensitive fields, such as "credit_card_number", "phone_number", and "id_number". Each parsed field is compared with this set of fields. If a field name matches a sensitive field in a preset set, that field is considered to contain sensitive information and is recorded as secondary sensitive data for subsequent processing. This field matching process accurately identifies sensitive information in structured log data, preventing non-sensitive fields from being mishandled and ensuring the targeted nature of the de-identification process.

[0079] The field set is a predefined collection containing the names of all sensitive fields that the system needs to monitor, used for precise matching in the parsed log data. Sensitive fields: Fields in structured log data that may contain personal privacy or trade secrets require special protection. Secondary sensitive data, identified through field matching, specifically contains data items with sensitive information; this data will undergo anonymization in the next step.

[0080] In this embodiment, firstly, the system, based on the data structure characteristics of the log data, calls the corresponding parsing library to convert it into structured field content. This process ensures the understanding and parsing of complex data formats, providing an accurate information source for subsequent field matching. Next, by precisely matching the field content with a preset set of sensitive fields, the system can identify and mark fields containing sensitive information, i.e., the second set of sensitive data. This series of operations not only improves the accuracy and efficiency of sensitive information identification and avoids misprocessing of non-sensitive data, but also maintains the integrity and readability of the log data, providing necessary clues for subsequent security audits and troubleshooting. Through structured parsing and precise field matching, the system protects user privacy and business secrets while ensuring the effective use and efficient processing of data, significantly improving the security and compliance level of large enterprise information systems when processing log data containing sensitive information.

[0081] In one embodiment, such as Figure 5 As shown, in step S230, the first and second sensitive data in the log data are replaced with their corresponding hash values ​​using a preset hash function to obtain the de-sensitized target log data. This includes steps S500-S540:

[0082] Step S500: Determine the data types corresponding to the first sensitive data and the second sensitive data respectively.

[0083] Specifically, after identifying and separating the first sensitive data in text form and the second sensitive data in structured fields, the system determines the data type of these two types of sensitive data, providing a basis for further processing and the selection of hash functions.

[0084] For example, the system has a built-in type recognition module containing a data type mapping table that records the standard format and corresponding data type for each type of sensitive information. After the first and second sensitive data are separated, the type recognition module analyzes this data, matching it against the standard format in the mapping table to determine its specific data type. For instance, an ID card number might be identified as a numeric string; a phone number might be identified as a string in phone number format. Establishing the data type helps in the subsequent selection of a suitable hash function, as different data types may require hash algorithms with different characteristics to ensure the uniqueness and security of the generated hash value.

[0085] Among them, data type refers to the kind or format of data, such as numbers, strings, dates and times, etc. Determining the data type is crucial for the correct processing of data.

[0086] Step S510: Select a hash function corresponding to the first sensitive data and the second sensitive data respectively, based on the data types corresponding to the first sensitive data and the second sensitive data respectively, so that the generated hash value is unique.

[0087] Specifically, after determining the data type of the sensitive data, the system selects the most suitable hash function based on these types to ensure that the converted hash value is unique for each piece of sensitive data. Uniqueness means that the hash value is unique for each piece of sensitive data; even if two pieces of data are very similar, the hash function should be able to produce different hash values ​​to avoid confusion or leakage of sensitive information.

[0088] For example, the system pre-configures a series of hash functions, including but not limited to SHA-256, MD5, and bcrypt. Each function has its own characteristics, such as hash length, collision rate, and computation speed. Once the data type is determined, the system selects the most suitable function from these options, typically matching the data type characteristics (such as length and numerical range) with the characteristics of the hash function to achieve the optimal balance. For instance, for longer, sensitive data, the SHA-256 function, which generates longer hash values, might be chosen to reduce the chance of collisions.

[0089] Step S520: Using the hash functions corresponding to the first sensitive data and the second sensitive data respectively, the first sensitive data and the second sensitive data are transformed to obtain the hash values ​​corresponding to the first sensitive data and the second sensitive data.

[0090] Each first sensitive data and each second sensitive data corresponds to a unique hash value.

[0091] Specifically, the system uses the hash function selected in the previous step to transform the first sensitive data and the second sensitive data, generating hash values ​​that correspond one-to-one with the original sensitive data, so as to achieve information desensitization.

[0092] For example, the system includes a hash conversion component that reads data type information from the type identification module, calls a matching hash function, and performs a hash operation on the sensitive data. After processing by the hash function, the sensitive data generates a fixed-length, irreversible hash value. This process is the core of the entire de-identification process. Hash conversion ensures that sensitive information is effectively transformed, protecting data security while preserving the data's unique identifier, thus providing the possibility for subsequent encrypted queries.

[0093] The hash value, generated by a hash function, is a fixed-length encoding used to represent the original sensitive data. Ideally, even slight changes to the original data will result in a completely different hash value, ensuring the uniqueness of the data.

[0094] Step S530: Locate the target positions of the first and second sensitive data in the log data.

[0095] Specifically, after completing the hash transformation of sensitive data, the system needs to locate the exact position of this data in the original log in order to perform the replacement operation and ensure the integrity of the log structure. Locating the target location of sensitive data is crucial to ensuring the integrity of the log structure and the accuracy of the de-identification effect. Only with accurate location information can the replacement operation be guaranteed to be correct and error-free, without disrupting the original layout and information chain of the log.

[0096] For example, during the reading and analysis of log data, the system records metadata for each piece of sensitive data, including the data's location information within the log. This location information can be a character index, field name, or even a more complex structured path (such as a key-value pair path in JSON). Once the hash value is generated, the system uses this location information to accurately pinpoint the hash value to its corresponding location within the original log data.

[0097] The target location refers to the specific location of the first and second sensitive data in the original log. This location information is used to guide the replacement of the hash value.

[0098] Step S540: Replace the first sensitive data and the second sensitive data at the corresponding target locations with the hash values ​​corresponding to the first sensitive data and the second sensitive data respectively, to obtain the de-identified target log data.

[0099] Specifically, after locating the exact location of the sensitive data, the system uses hash values ​​to replace the sensitive information in the original logs, ultimately generating target log data that is completely desensitized while maintaining structural integrity.

[0100] For example, the log data is modified based on the location information obtained from the previous step. The replacement engine inserts the hash value into the correct position in the log, replacing the original sensitive data. This process requires extremely high precision to avoid affecting the rest of the log. The log data after sensitive information replacement maintains the original log's structure and functionality while effectively masking the sensitive information.

[0101] In this embodiment, firstly, through type identification, the system can select the optimal hash function for different types of sensitive information, ensuring the uniqueness and security of hash values ​​and reducing the risk of data collisions. Secondly, sensitive information is transformed through hash function processing, protecting data privacy while retaining the unique identifier of the information, providing a foundation for subsequent encrypted queries. Thirdly, accurate target location positioning ensures the accuracy of the replacement operation, maintains the integrity of the log structure, and avoids accidental damage to log data. Finally, through hash value replacement, de-identified target log data is generated. This data is not only secure but also maintains readability and analyzability, effectively supporting the system's operation and maintenance and auditing needs. In summary, this process greatly improves the security, efficiency, and flexibility of log data processing. The method in this embodiment constitutes a comprehensive and refined sensitive information de-identification process. From data type determination to intelligent hash function selection, and then to hash value generation and accurate replacement, each link is closely connected and interacts with each other, jointly ensuring a balance between security protection and availability of log data.

[0102] In one embodiment, such as Figure 6 As shown, the method further includes steps S600-S620:

[0103] Step S600: Obtain the sensitive information to be queried input from the target object, process the sensitive information to be queried using a preset hash function, and generate a target hash value corresponding to the sensitive information to be queried.

[0104] Specifically, the system features a user interface that allows target users (such as security analysts and operations personnel) to input sensitive information to be queried. The system then processes this input using a pre-defined hash function to generate a target hash value, which serves as the keyword for subsequent searches.

[0105] For example, the system includes an input component capable of receiving target sensitive information from the user. Once the sensitive information is input, a pre-defined hash function module is invoked to perform a hash operation on the input information, generating a fixed-length hash value. This value is used for subsequent query operations. The process of generating the target hash value ensures that the sensitive information input by the user is not directly exposed during the search process, while preserving the unique identifier of the information, facilitating subsequent precise matching and querying, and improving the security and efficiency of the query.

[0106] The target hash value is generated by processing the sensitive information input by the user using a preset hash function, and is used for searching.

[0107] Step S610: Traverse the target log data according to the target hash value to query whether there is a hash value that matches the target hash value in the target log data.

[0108] Specifically, the system will traverse the target log data that has been anonymized, and look for a hash value that matches the target hash value in order to locate the relevant log entry.

[0109] For example, the query engine is responsible for traversing the stored target log data, using the generated target hash value as the search key, and comparing the hash values ​​in the logs. If a match is found, it means that the original sensitive information was recorded in the current log entry (although it has been anonymized), thus locating the relevant log record. By traversing the target log data and using hash values ​​for matching, the system can efficiently and securely locate log entries containing sensitive information, avoiding the privacy leakage risks that may arise from directly searching for sensitive information.

[0110] The target log data is de-identified log data, where sensitive information has been replaced with hash values. The query engine is a system component responsible for searching and locating hash values ​​in the target log data that match the input hash value.

[0111] Step S620: If target log data containing a hash value that matches the target hash value is found, the retrieved target log data is returned to the target object.

[0112] Specifically, once the query engine locates an entry in the target log data that matches the target hash value, the system will return the log data entry containing the matching hash value to the querying user or target object for further analysis or troubleshooting.

[0113] For example, the system is designed with a feedback mechanism. When the query engine finds a matching hash value, it compiles the log entries containing these hash values ​​into a list or report and returns it to the target object through a user interface or API. The feedback information may include the timestamp of the log entry, its context, and the location where the original sensitive information was replaced by the hash value, but it does not include the original sensitive information. The feedback mechanism not only provides query results but also protects user privacy because the original sensitive information is not directly displayed. Users can only see de-identified log data containing the location information of the sensitive information, thus enhancing the system's compliance and security.

[0114] The feedback mechanism is a mechanism in the system used to return query results to the target object that initiated the request, ensuring that the user can receive the query results in a timely manner.

[0115] In this embodiment, firstly, by having the user input sensitive information and generating a target hash value, data security during the query process is ensured, preventing the direct exposure of sensitive information. Secondly, the target hash value is used to traverse the target log data for precise matching. This process not only greatly improves search efficiency but also ensures the accuracy of the results, providing strong support for system maintenance and troubleshooting. Finally, a feedback mechanism securely returns the query results to the target object. This design satisfies the user's need to locate sensitive information while protecting privacy, enhancing system compliance and user experience. A secure and efficient hash value query mechanism has been constructed for searching and locating log entries containing sensitive information.

[0116] In one embodiment, such as Figure 7 As shown, the method further includes steps S700-S730:

[0117] Step S700: Extract the data features of the intercepted log data to be de-identified.

[0118] Specifically, upon receiving log data, the system first extracts data features to analyze and understand the nature and content of the log data. This process is the cornerstone of subsequent steps, helping the system identify the source, structure, and potentially sensitive information types of the logs. Through data feature extraction, the system can perform preliminary understanding and classification of the log data, providing a basis for subsequent risk assessment and filtering rule adjustments.

[0119] For example, the system includes a data feature extraction module that can read log data, parse its format, and extract key information such as the system module that generated the log, the timestamp, the operation type, and specific fields or data patterns. For instance, if the log is about user transactions, the module will look for transaction-related fields such as transaction amount and user ID.

[0120] Data features are specific attributes or patterns in data that can be identified and analyzed, used to describe the source, content, and structure of the data.

[0121] Step S710: Determine the business type corresponding to the intercepted log data to be de-identified based on the data characteristics.

[0122] Specifically, the system determines the business type of the log data based on the extracted data characteristics. Different business types may involve different types and levels of sensitive information, and this determination helps in subsequent more accurate risk level assessments.

[0123] For example, the system has a built-in business type recognition engine that includes a business type mapping table, which can match data features with specific business types. For instance, if the data features include transaction amount and bank card number information, the system will identify it as a financial transaction log and classify it as a "financial business" type.

[0124] Among them, the business type is based on the content and nature of the log data, classifying it into the specific business areas of the enterprise or organization, such as financial transactions, user login, system auditing, etc.

[0125] Step S720: Determine the risk level of the log data to be anonymized based on the business type.

[0126] Specifically, the system assesses the potential risk level of log data based on its business type. Determining the risk level is the basis for adjusting filtering rules, ensuring that the depth and breadth of data anonymization processing matches the sensitivity of the data.

[0127] For example, the system has pre-defined risk level assessment rules that associate business types with risk levels. For instance, financial transaction logs might be marked as "high risk" because they may contain sensitive information such as bank accounts and credit cards.

[0128] The risk level is a security rating given by the system based on the business type of the log data and the potential impact of sensitive information. It can be divided into low risk, medium risk, and high risk.

[0129] Step S730: Adjust the first and second filtering rules corresponding to the log data to be de-identified according to the risk level.

[0130] Specifically, based on the determined risk level, the system dynamically adjusts the first filtering rule (for sensitive information in text form) and the second filtering rule (for sensitive fields in structured data) to ensure that the de-identification strategy is adapted to the sensitivity of the data.

[0131] For example, the filtering rule adjustment module can select or generate corresponding filtering rules from a preset rule base based on the risk level. For instance, for high-risk financial transaction logs, the system may enable more comprehensive and stringent filtering rules, covering more types of sensitive information, such as bank account numbers, ID numbers, and transaction amounts. By dynamically adjusting the filtering rules, the system can keep pace with the ever-changing data environment and business needs, ensuring the comprehensiveness and effectiveness of sensitive information desensitization, while avoiding data loss or performance overhead caused by over-processing.

[0132] In this embodiment, firstly, by extracting data features, the system can accurately understand the source and nature of log data, providing crucial information for subsequent processing. Next, based on the business type determined by the data features, the system can further assess the risk level of the log data. This step ensures that the system can adopt differentiated processing strategies for data of different business types. Finally, based on the risk level, the system dynamically adjusts the sensitive information filtering rules, achieving refined management of the de-identification strategy. This protects user privacy and enterprise data security while avoiding over-processing of non-sensitive information, improving overall processing efficiency and system performance. This ensures both data security and processing efficiency.

[0133] In one embodiment, the method further includes:

[0134] Acquire log data and perform real-time data analysis and learning based on the log data.

[0135] Specifically, a machine learning module was designed and implemented to continuously monitor and analyze anonymized log data to identify new data patterns and anomalous behaviors. Using methods such as time series analysis and cluster analysis, the module automatically learns the distribution patterns and trends of sensitive information in the log data, which are then used to dynamically adjust subsequent filtering rules.

[0136] Analyze the context of the log data to determine its meaning.

[0137] Specifically, natural language processing (NLP) techniques are introduced to analyze the context in which sensitive information appears in log data. Context analysis parameters are set, such as the frequency of sensitive information occurrence and its correlation with other keywords, to intelligently determine whether information is sensitive in the current context. NLP models are used to predict the "true sensitivity" of sensitive information, and the prediction results determine whether to perform de-identification or what level of de-identification to choose.

[0138] It intelligently and dynamically adjusts the filtering rules.

[0139] Specifically, based on the results of real-time learning, the complexity and coverage of the first and second filtering rules are dynamically adjusted to adapt to the evolution of data patterns. Using reinforcement learning algorithms, the algorithm self-optimizes its rule adjustment strategy based on feedback from log analysis results after the filtering rules are adjusted, minimizing resource consumption and loss of data analyzability.

[0140] Specifically, context-aware hash value management is implemented based on the context of log data.

[0141] Specifically, a hash value context management mechanism is established to record the specific values ​​of sensitive information after hashing within a specific context. When used for hash-based queries, this mechanism can more accurately locate log records based on context information, improving query efficiency.

[0142] Specifically, filtering rules are adaptively adjusted based on the context of log data learned in real time. A rule integration framework is designed to seamlessly integrate the filtering rules learned in real time with a pre-set rule set. Based on learned data patterns and business changes, the applicable parts of the rule set are intelligently selected, avoiding conflicts and redundant processing between rules.

[0143] Among them, context awareness refers to the system's ability to understand the environment and context of information in a text, thereby more accurately judging the attributes and value of the information.

[0144] In this embodiment, the intelligent dynamic filtering and context-aware de-identification method can automatically optimize filtering rules based on real-time data streams, reducing the frequency and complexity of manual adjustments and improving the targeting and efficiency of the strategy. Through context awareness, the system avoids over-processing of non-sensitive information, retaining more valuable contextual information, which aids in troubleshooting and behavior auditing. Intelligent judgment and context analysis can more accurately identify and process sensitive information, reducing the risk of privacy leaks, complying with regulatory requirements, and enhancing the enterprise's data governance capabilities. The integration of context-adaptive filtering rules enables the system to quickly respond to changes in data patterns and adjustments to business needs, ensuring the continuous effectiveness of de-identification processing. By introducing innovative technologies such as real-time data analysis and learning, context-aware de-identification decision-making, and intelligent dynamic adjustment of filtering rules, the intelligence and adaptability of log data de-identification processing are significantly enhanced, effectively protecting the security of sensitive information and improving the analytical and utilization value of data and the system's resource management efficiency.

[0145] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0146] This embodiment also provides a log desensitization device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0147] Figure 8 This is a structural block diagram of an optional log desensitization device according to an embodiment of this application. For example... Figure 8 As shown, it includes:

[0148] Interception module 801 is used to intercept log data that needs to be de-identified.

[0149] The first identification module 802 is used to filter sensitive text in the log data according to the application scenario of the log data using a preset first filtering rule, and to identify the first sensitive data in the log data.

[0150] The second identification module 803 is used to filter sensitive fields in the log data according to the data structure of the log data using a preset second filtering rule, and to identify the second sensitive data in the log data.

[0151] The desensitization module 804 is used to replace the first sensitive data and the second sensitive data in the log data with the corresponding hash values ​​using a preset hash function to obtain the desensitized target log data. The hash function is used to indicate the conversion relationship between sensitive data and hash values.

[0152] In an exemplary embodiment, the above-described apparatus is further configured to: determine the application scenario of the log data based on the source and type of the log data, and match it with a set of regular expressions corresponding to the application scenario; read the log data and traverse the log data to find text in the log data that matches the regular expressions in the set of regular expressions; and use the text that matches the regular expressions in the set of regular expressions as first sensitive data.

[0153] In an exemplary embodiment, the above-described apparatus is further configured to: parse the log data according to the data structure of the log data by calling a parsing library corresponding to the data structure, and determine the field content of the log data; match the field content of the log data with a preset field set to extract sensitive fields from the field content of the log data, which are then used as second sensitive data.

[0154] In an exemplary embodiment, the apparatus is further configured to: determine the data types corresponding to the first sensitive data and the second sensitive data respectively; select hash functions corresponding to the first sensitive data and the second sensitive data respectively, based on their respective data types, to ensure the uniqueness of the generated hash values; convert the first sensitive data and the second sensitive data using the hash functions corresponding to them respectively, to obtain hash values ​​corresponding to the first sensitive data and the second sensitive data, wherein each piece of first sensitive data and each piece of second sensitive data corresponds to a unique hash value; locate the target positions of the first sensitive data and the second sensitive data in the log data; and replace the first sensitive data and the second sensitive data at the corresponding target positions with the hash values ​​corresponding to them respectively, to obtain the de-identified target log data.

[0155] In an exemplary embodiment, the above-described apparatus is further configured to: acquire sensitive information to be queried input by the target object; process the sensitive information to be queried using a preset hash function to generate a target hash value corresponding to the sensitive information to be queried; traverse target log data according to the target hash value to query whether a hash value matching the target hash value exists in the target log data; and, if target log data containing a hash value matching the target hash value is found, return the found target log data to the target object.

[0156] In an exemplary embodiment, the apparatus is further configured to: extract data features of the intercepted log data to be anonymized; determine the service type corresponding to the intercepted log data to be anonymized based on the data features; determine the risk level corresponding to the log data to be anonymized based on the service type; and adjust the first filtering rule and the second filtering rule corresponding to the log data to be anonymized based on the risk level.

[0157] Embodiments of this application also provide a storage medium including a stored program, wherein the program executes any of the methods described above when it is run.

[0158] Optionally, in this embodiment, the storage medium may be configured to store program code for performing the following steps:

[0159] S1 intercepts log data that needs to be de-identified.

[0160] S2, based on the application scenario of the log data, uses a preset first filtering rule to filter sensitive text in the log data and identify the first sensitive data in the log data.

[0161] S3 uses a preset second filtering rule to filter sensitive fields in the log data based on the data structure of the log data, and identifies the second sensitive data in the log data.

[0162] S4, replace the first and second sensitive data in the log data with the corresponding hash values ​​using a preset hash function to obtain the de-identified target log data. The hash function is used to indicate the conversion relationship between sensitive data and hash values.

[0163] Embodiments of this application also provide an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0164] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0165] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0166] S1 intercepts log data that needs to be de-identified.

[0167] S2, based on the application scenario of the log data, uses a preset first filtering rule to filter sensitive text in the log data and identify the first sensitive data in the log data.

[0168] S3 uses a preset second filtering rule to filter sensitive fields in the log data based on the data structure of the log data, and identifies the second sensitive data in the log data.

[0169] S4, replace the first and second sensitive data in the log data with the corresponding hash values ​​using a preset hash function to obtain the de-identified target log data. The hash function is used to indicate the conversion relationship between sensitive data and hash values.

[0170] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0171] Embodiments of this application also provide a computer program product, including a non-volatile computer-readable storage medium storing the computer program product, wherein the computer program, when executed by a processor, implements the steps of the methods in various embodiments of this application.

[0172] Optionally, in this embodiment, the computer program described above can be configured to perform the following steps when executed by a processor:

[0173] S1 intercepts log data that needs to be de-identified.

[0174] S2, based on the application scenario of the log data, uses a preset first filtering rule to filter sensitive text in the log data and identify the first sensitive data in the log data.

[0175] S3 uses a preset second filtering rule to filter sensitive fields in the log data based on the data structure of the log data, and identifies the second sensitive data in the log data.

[0176] S4, replace the first and second sensitive data in the log data with the corresponding hash values ​​using a preset hash function to obtain the de-identified target log data. The hash function is used to indicate the conversion relationship between sensitive data and hash values.

[0177] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0178] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0179] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A log anonymization method, characterized in that, The method includes: Intercept log data that needs to be de-identified; Based on the application scenario of the log data, a preset first filtering rule is used to filter sensitive text in the log data, and the first sensitive data in the log data is identified. Based on the data structure of the log data, a preset second filtering rule is used to filter sensitive fields in the log data, and second sensitive data in the log data is identified; The first sensitive data and the second sensitive data in the log data are replaced with corresponding hash values ​​using a preset hash function to obtain the de-identified target log data. The hash function is used to indicate the conversion relationship between sensitive data and hash values.

2. The log anonymization method according to claim 1, characterized in that, The step of filtering sensitive text in the log data according to the application scenario of the log data using a preset first filtering rule, and identifying the first sensitive data in the log data, includes: The application scenario of the log data is determined based on its source and type, and then a set of regular expressions corresponding to the application scenario is matched. The log data is read and traversed to find text in the log data that matches the regular expressions in the set of regular expressions; Text that matches the regular expressions in the set of regular expressions is used as the first sensitive data.

3. The log anonymization method according to claim 1, characterized in that, The step of filtering sensitive fields in the log data according to the data structure of the log data using a preset second filtering rule, and identifying the second sensitive data in the log data, includes: Based on the data structure of the log data, the parsing library corresponding to the data structure is called to parse the log data and determine the field content of the log data; The field content of the log data is matched with a preset set of fields to extract sensitive fields from the field content of the log data, which are then used as the second sensitive data.

4. The log anonymization method according to any one of claims 1-3, characterized in that, The step of replacing the first sensitive data and the second sensitive data in the log data with their corresponding hash values ​​using a preset hash function to obtain the de-identified target log data includes: Determine the data types corresponding to the first sensitive data and the second sensitive data respectively; Based on the data types corresponding to the first sensitive data and the second sensitive data respectively, select the hash function corresponding to the first sensitive data and the second sensitive data respectively, so that the generated hash value is unique; The first sensitive data and the second sensitive data are transformed using hash functions corresponding to the first sensitive data and the second sensitive data respectively, to obtain hash values ​​corresponding to the first sensitive data and the second sensitive data, wherein each first sensitive data and each second sensitive data corresponds to a unique hash value; Locate the target positions of the first sensitive data and the second sensitive data in the log data; The first sensitive data and the second sensitive data at the corresponding target locations are replaced with hash values ​​corresponding to the first sensitive data and the second sensitive data, respectively, to obtain the de-identified target log data.

5. The log anonymization method according to any one of claims 1-3, characterized in that, The method further includes: Obtain the sensitive information to be queried input from the target object, process the sensitive information to be queried using the preset hash function, and generate a target hash value corresponding to the sensitive information to be queried; The target log data is traversed according to the target hash value to query whether there is a hash value that matches the target hash value in the target log data; If target log data containing a hash value that matches the target hash value is found, the retrieved target log data is returned to the target object.

6. The log anonymization method according to any one of claims 1-3, characterized in that, The method further includes: Extract the data features of the intercepted log data to be de-identified; The business type corresponding to the intercepted log data to be de-identified is determined based on the data characteristics. The risk level corresponding to the log data to be de-identified is determined based on the business type. Adjust the first and second filtering rules corresponding to the log data to be de-identified according to the risk level.

7. A log desensitization device, characterized in that, The device includes: The interception module is used to intercept log data that needs to be de-identified; The first identification module is used to filter sensitive text in the log data according to the application scenario of the log data using a preset first filtering rule, and to identify the first sensitive data in the log data. The second identification module is used to filter sensitive fields in the log data according to the data structure of the log data using a preset second filtering rule, and to identify the second sensitive data in the log data; The desensitization module is used to replace the first sensitive data and the second sensitive data in the log data with corresponding hash values ​​using a preset hash function to obtain the desensitized target log data. The hash function is used to indicate the conversion relationship between sensitive data and hash values.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method of any one of claims 1 to 6.

9. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method of any one of claims 1 to 6 through the computer program.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images