Log desensitization methods, devices, equipment, media, and program products

Abstract

This disclosure provides a log anonymization method, apparatus, device, medium, and program product, which can be applied to the fields of data analysis technology or fintech. The method includes: hot-loading an anonymization configuration file during the operation of a log management platform to obtain anonymization configuration data, the anonymization configuration data including a query identifier and preset operation data, the preset operation data being at least one of a predefined named field and anonymization rule data, and the query identifier corresponding to the preset operation data; parsing log data in the log management platform to obtain multiple sub-log data; querying the multiple sub-log data based on the query identifier to obtain target sub-log data; and, if the preset operation data corresponding to the query identifier includes anonymization rule data, using the anonymization rule data to anonymize the target sub-log data to obtain the anonymized sub-log data.

Description

Technical Field

[0001] This disclosure relates to the fields of data analytics technology or fintech, and more specifically to a log desensitization method, apparatus, device, medium, and program product. Background Technology

[0002] A log management platform is a service used to centrally collect, store, analyze, and manage log files. It is crucial for monitoring system status, analyzing application performance, security auditing, and troubleshooting. Currently, log management platforms can only parse and extract fields from log data, but cannot anonymize log information. Existing anonymization methods require a restart of the log management platform after each configuration of new anonymized information, resulting in poor efficiency. Summary of the Invention

[0003] In view of the above problems, this disclosure provides log desensitization methods, apparatus, devices, media and program products.

[0004] According to the first aspect of this disclosure, a log de-identification method is provided, comprising: hot-loading a de-identification configuration file during the operation of the aforementioned log management platform to obtain de-identification configuration data, wherein the de-identification configuration data includes a query identifier and preset operation data, wherein the preset operation data is at least one of a predefined naming field and de-identification rule data, and the query identifier corresponds to the preset operation data; parsing log data in the aforementioned log management platform to obtain multiple sub-log data; querying the multiple sub-log data based on the query identifier to obtain target sub-log data; and, if the preset operation data corresponding to the query identifier includes de-identification rule data, de-identifying the target sub-log data using the de-identification rule data to obtain de-identified sub-log data.

[0005] According to embodiments of this disclosure, the aforementioned de-identification configuration file further includes query rules corresponding to the aforementioned query identifier. The aforementioned querying of the aforementioned multiple sub-log data to obtain target sub-log data based on the aforementioned query identifier includes: determining a target query rule from the aforementioned query rules based on the target query identifier; and using the aforementioned target query rule to query the target sub-log data from the aforementioned multiple sub-log data.

[0006] According to embodiments of this disclosure, the aforementioned de-identification configuration file further includes query rules corresponding to the aforementioned query identifier. The aforementioned querying of the aforementioned multiple sub-log data to obtain target sub-log data based on the aforementioned query identifier includes: determining a target query rule from the aforementioned query rules based on the target query identifier; and using the aforementioned target query rule to query the target sub-log data from the aforementioned multiple sub-log data.

[0007] According to embodiments of this disclosure, the method further includes: resetting the current state information to the initial state information when the input character is not present in the state machine.

[0008] According to embodiments of this disclosure, the method further includes: generating encrypted sub-log data based on the target sub-log data using an encryption key; generating structured log data based on the encrypted sub-log data and the de-identified sub-log data; and saving the structured log data to a database.

[0009] According to embodiments of this disclosure, the method further includes: in response to receiving a log query request from a terminal, obtaining a user identifier from the log query request; and if the user identifier is not a management user identifier, sending the de-identified sub-log data to the terminal.

[0010] According to an embodiment of this disclosure, the method further includes: when the user identifier is a management user identifier, using the encryption key to decrypt the encrypted sub-log data to obtain the target sub-log data; and sending the target sub-log data to the terminal.

[0011] According to embodiments of this disclosure, the log management platform includes multiple log collection nodes, and the method further includes: in response to receiving raw log data sent by the multiple log collection nodes, performing word segmentation processing on the raw log data to obtain word-segmented log data, wherein the raw log data is natural language text data; performing part-of-speech tagging on the word-segmented log data to obtain tagged log data; inputting the tagged log data into a preset natural language processing model, and outputting the log data, wherein the log data is structured text data.

[0012] The second aspect of this disclosure provides a log de-identification apparatus, comprising: a hot-loading module, configured to hot-load a de-identification configuration file to obtain de-identification configuration data during the operation of the log management platform, wherein the de-identification configuration data includes a query identifier and preset operation data, wherein the preset operation data is at least one of a predefined naming field and de-identification rule data, and the query identifier corresponds to the preset operation data; a parsing module, configured to parse log data in the log management platform to obtain multiple sub-log data; a query module, configured to query the multiple sub-log data based on the query identifier to obtain target sub-log data; and a de-identification module, configured to de-identify the target sub-log data using the de-identification rule data when the preset operation data corresponding to the query identifier includes de-identification rule data to obtain de-identified sub-log data.

[0013] According to embodiments of this disclosure, the above-mentioned de-identification configuration file further includes query rules corresponding to the above-mentioned query identifier. The above-mentioned query module includes: a target query rule determination submodule, used to determine a target query rule from the above-mentioned query rules based on the target query identifier; and a target sub-log query submodule, used to query target sub-log data from the above-mentioned multiple sub-log data using the above-mentioned target query rule.

[0014] According to an embodiment of this disclosure, the target sub-log query submodule includes: a construction unit, configured to construct a state machine based on the target query rules; a state transition unit, configured to input the characters in the sub-log data into the state machine in an orderly manner for each sub-log data, so that the state machine performs state transition based on the input characters and the current state information to obtain transition state information; and a target sub-log determination unit, configured to determine the sub-log data as target sub-log data when the transition state information is end state information.

[0015] According to an embodiment of this disclosure, the target sub-log query submodule further includes a reset unit, used to reset the current state information to the initial state information when the input character is not present in the state machine.

[0016] According to embodiments of this disclosure, the apparatus further includes: an encryption module for generating encrypted sub-log data based on the target sub-log data using an encryption key; a structured log data generation module for generating structured log data based on the encrypted sub-log data and the de-identified sub-log data; and a storage module for storing the structured log data in a database.

[0017] According to an embodiment of this disclosure, the apparatus further includes: an acquisition module, configured to acquire a user identifier in the log query request in response to receiving a log query request from a terminal; and a first sending module, configured to send the de-identified sub-log data to the terminal if the user identifier is not a management user identifier.

[0018] According to an embodiment of this disclosure, the apparatus further includes: a decryption module, configured to decrypt the encrypted sub-log data using the encryption key to obtain the target sub-log data when the user identifier is an administrator user identifier; and a second sending module, configured to send the target sub-log data to the terminal.

[0019] According to embodiments of this disclosure, the log management platform includes multiple log collection nodes, and the device further includes: a word segmentation module, used to perform word segmentation processing on the raw log data sent by the multiple log collection nodes in response to receiving the raw log data to obtain word-segmented log data, wherein the raw log data is natural language text data; a tagging module, used to perform part-of-speech tagging on the word-segmented log data to obtain tagged log data; and an output module, used to input the tagged log data into a preset natural language processing model and output the log data, wherein the log data is structured text data.

[0020] A third aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0021] A fourth aspect of this disclosure also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0022] The fifth aspect of this disclosure also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0023] According to the embodiments of this disclosure, the log management platform loads the de-identification configuration file at runtime to obtain de-identification configuration data. Based on the query identifier in the de-identification configuration data, it finds the corresponding target sub-log data. When the preset operation data includes de-identification rule data, the de-identification rule data is used to de-identify the target sub-log to obtain the de-identified sub-log data. This method is highly targeted and flexible. Furthermore, the log management platform performs hot loading of the de-identification configuration file, which eliminates the need to reload after modifying the de-identification configuration data. The modified de-identification rule data can be used directly for de-identification, thus improving efficiency. Attached Figure Description

[0024] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0025] Figure 1 This diagram illustrates an application scenario of the log desensitization method and apparatus according to embodiments of the present disclosure.

[0026] Figure 2 A flowchart illustrating a log desensitization method according to an embodiment of the present disclosure is shown schematically.

[0027] Figure 3 A schematic diagram illustrating the structure of a log desensitization apparatus according to an embodiment of the present disclosure is shown.

[0028] Figure 4 A block diagram schematically illustrates an electronic device suitable for implementing a log desensitization method according to an embodiment of the present disclosure. Detailed Implementation

[0029] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0030] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0031] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0032] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0033] It should be noted that in the embodiments disclosed herein, certain software, components, models, and other existing solutions in the industry may be mentioned. These should be considered as exemplary and are intended only to illustrate the feasibility of implementing the technical solutions disclosed herein. However, they do not mean that the applicant has used or necessarily used such solutions.

[0034] In the technical solution disclosed herein, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding operation entry points are provided for users to choose to authorize or refuse.

[0035] In scenarios involving automated decision-making using personal information, the methods, devices, and systems provided in this disclosure all offer users corresponding entry points for choosing to agree to or reject the automated decision-making results. If the user chooses to reject, the process proceeds to the expert decision-making stage. Here, "automated decision-making" refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests, or economic, health, and credit status through computer programs, and then making a decision. Here, "expert decision-making" refers to the activity of making decisions by personnel who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.

[0036] Centralized log management platforms store logs collected from multiple systems. These log data include sensitive information such as names, ID numbers, landline numbers, mobile phone numbers, addresses, email addresses, passwords, and account numbers. It is necessary to anonymize this sensitive information to protect personal privacy without affecting data usability. Currently, the tools integrated into log management platforms only support decoding and extracting fields from rule-based logs, but cannot anonymize the logs. Furthermore, existing log anonymization methods require restarting the log management platform after configuring the anonymization information, resulting in poor efficiency.

[0037] In view of the above, embodiments of this disclosure provide a log de-identification method, characterized in that the method is applied to a log management platform, and the method includes: hot-loading a de-identification configuration file during the operation of the log management platform to obtain de-identification configuration data, the de-identification configuration data including a query identifier and preset operation data, the preset operation data being at least one of a predefined naming field and de-identification rule data, and the query identifier having a corresponding relationship with the preset operation data; parsing the log data in the log management platform to obtain multiple sub-log data; querying the multiple sub-log data based on the query identifier to obtain target sub-log data; and when the preset operation data corresponding to the query identifier includes de-identification rule data, de-identifying the target sub-log data using the de-identification rule data to obtain de-identified sub-log data.

[0038] Figure 1The illustration shows an application scenario of the log desensitization method and apparatus according to embodiments of the present disclosure.

[0039] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0040] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0041] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0042] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0043] It should be noted that the log desensitization method provided in this embodiment can generally be executed by server 105. Correspondingly, the log desensitization device provided in this embodiment can generally be located in server 105. The log desensitization method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the log desensitization device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0044] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0045] The following will be based on Figure 1 The described scene, through Figure 2 The log desensitization method of the disclosed embodiments is described in detail.

[0046] Figure 2 A flowchart illustrating a log desensitization method according to an embodiment of the present disclosure is shown schematically.

[0047] like Figure 2 As shown, the log desensitization method in this embodiment includes operations S210 to S240.

[0048] When operating S210, the de-identified configuration file is hot-loaded to obtain the de-identified configuration data while the log management platform is running.

[0049] The desensitization configuration data includes query identifiers and preset operation data. The preset operation data is at least one of the predefined named fields and desensitization rule data. The query identifier and the preset operation data have a corresponding relationship.

[0050] For ease of understanding, the following terms are explained in this disclosure.

[0051] Hot reloading is a technique that allows the system to automatically reload changes made to the code while the application is running, without stopping and restarting the entire application.

[0052] According to the embodiments of this disclosure, the above-mentioned de-identification configuration file can be written using grok syntax. The de-identification configuration data can be, for example, %{SYNTAX:SEMANTIC}, where SYNTAX is the query identifier and SEMANTIC is the preset operation data. It can include at least one of predefined named fields and de-identification rule data. The complete de-identification configuration data can be %{pattern:field:operation}. In order to be compatible with the original grok syntax, the de-identification configuration data can also be configured as: %{pattern::operation}, %{pattern:field}, %{pattern:field:}, where pattern represents the query identifier, field represents the predefined named field, and operation represents the de-identification rule data.

[0053] When operating S220, the log data in the log management platform is parsed to obtain multiple sub-log data.

[0054] According to embodiments of this disclosure, the log data includes multiple log data from different systems. The sub-log data can be separated by a newline separator or by a custom line break, and the log data can be parsed to obtain multiple sub-log data.

[0055] In operation S230, based on the query identifier, multiple sub-log data are queried to obtain the target sub-log data.

[0056] In operation S240, if the preset operation data corresponding to the query identifier includes de-identification rule data, the target sub-log data is de-identified using the de-identification rule data to obtain the de-identified sub-log data.

[0057] According to embodiments of this disclosure, the corresponding target sub-log data can be found based on the query identifier, and then the target sub-log data can be de-identified according to the de-identification rules.

[0058] For example, for a log message like “2024-01-01T12:34:56 INFO 192.168.1.10 [root] This is a sample log message.”, the de-identification configuration data can be configured as follows:

[0059] "SENSITIVE_IP replace(*.*.*.*)

[0060] SENSITIVE_USER replace("(\\d{0,1}\\d*)", "$1**"))

[0061] %{TIMESTAMP_ISO8601:timestamp}

[0062] %{LOGLEVEL:log_level} %{IP:client_ip: SENSITIVE_IP}

[0063] \[%{USERNAME:username: SENSITIVE_USER}\]

[0064] %{GREEDYDATA:message}”

[0065] Taking log time extraction as an example, the query identifier "TIMESTAMP_ISO8601" retrieves sub-log data matching the ISO 8601 standard date and time format. The target sub-log data is "2024-01-01T12:34:56". Since "timestamp" is a predefined named field, the final formatted sub-log data is "timestamp": "2024-01-01T12:34:56". Taking username extraction as another example, the query identifier is "USERNAME". "username" is a predefined named field, and "SENSITIVE_USER" is the de-identification rule data. This means the corresponding username is replaced based on the regular expression ("(\\d{0,1}\\d*)","$1**"). The final de-identified sub-log data is "username": "r**". The log data extracted using the above de-identification configuration is shown below: {"timestamp": "2024-01-01T12:34:56","log_level": "INFO","client_ip": "*.*.*.*", "username": "r**", "message": "This is a sample log message."}.

[0066] According to the embodiments of this disclosure, the log management platform loads the de-identification configuration file at runtime to obtain de-identification configuration data. Based on the query identifier in the de-identification configuration data, it finds the corresponding target sub-log data. When the preset operation data includes de-identification rule data, the de-identification rule data is used to de-identify the target sub-log to obtain the de-identified sub-log data. This method is highly targeted and flexible. Furthermore, the log management platform performs hot loading of the de-identification configuration file, which eliminates the need to reload after modifying the de-identification configuration data. The modified de-identification rule data can be used directly for de-identification, thus improving efficiency.

[0067] According to an embodiment of this disclosure, the de-identification configuration file also includes query rules corresponding to query identifiers. Based on the query identifier, multiple sub-log data are queried to obtain target sub-log data, including: determining a target query rule from the query rules based on the target query identifier; and using the target query rule to query the target sub-log data from multiple sub-log data.

[0068] According to embodiments of this disclosure, the target query rules described above may be, for example, using regular expressions to query log data, or using string comparison or graph matching methods to query log data; this disclosure does not impose any limitations on this.

[0069] According to an embodiment of this disclosure, target sub-log data is obtained from multiple sub-log data using target query rules, including: constructing a state machine based on the target query rules; for each sub-log data, inputting the characters in the sub-log data into the state machine in an orderly manner, so that the state machine performs state transitions based on the input characters and the current state information to obtain transition state information; when the transition state information is the end state information, the sub-log data is determined as the target sub-log data.

[0070] According to embodiments of this disclosure, a state machine can be constructed based on target query rules, and the aforementioned state machine may be, for example, a finite state machine.

[0071] For example, for a regular expression with the target query rule "\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b", which is used to match an email address, a finite state machine will be constructed. This finite state machine includes an initial state, intermediate states (including multiple sub-intermediate states for matching characters), and a final state. "b[A-Za-z0-9._%+-]" is used to match the name part before the "@" identifier. When the finite state machine finds a character corresponding to the above regular expression, it remains in an intermediate state and waits for an "@" symbol. Then, it matches the domain name data corresponding to "[A-Za-z0-9.-]+" and performs state transitions in the sub-intermediate state according to the input character. After matching the domain name data, the state machine matches the point data and the subsequent top-level domain information of "[A-Za-z]{2,}". After all target query rules are successfully matched, the transition state information reaches the end state, which means that the corresponding sub-log data can be identified as the target sub-log data.

[0072] According to embodiments of this disclosure, the method further includes: resetting the current state information to the initial state information when there is no input character in the state machine.

[0073] According to the embodiments of this disclosure, by configuring query rules corresponding to query identifiers in the de-identification configuration file, the corresponding query rules can be directly found through the query identifiers, and the corresponding sub-log data can be queried through the query rules, which increases the scalability and flexibility of the program. Furthermore, by constructing a state machine through the target query rules, the log management platform becomes more flexible, and by constructing a state machine, a full scan of all data is avoided, reducing the occupation of computing resources and alleviating the computing burden.

[0074] According to embodiments of this disclosure, the method further includes: generating encrypted sub-log data based on target sub-log data using an encryption key; generating structured log data based on the encrypted sub-log data and the de-identified sub-log data; and saving the structured log data to a database.

[0075] According to embodiments of this disclosure, target sub-log data can be encrypted using any one of symmetric keys, asymmetric keys, and hybrid keys to generate encrypted sub-log data. Furthermore, index data corresponding to the target sub-log data can be generated using hash algorithms such as MD5, SHA-256, and SHA-512 and saved to the database. This allows users to quickly find the corresponding encrypted and de-identified sub-log data based on the index data during searches.

[0076] According to embodiments of this disclosure, the method further includes: in response to receiving a log query request from a terminal, obtaining a user identifier from the log query request; and if the user identifier is not a management user identifier, sending de-identified sub-log data to the terminal.

[0077] According to an embodiment of this disclosure, the method further includes: when the user identifier is a management user identifier, using an encryption key to decrypt the encrypted sub-log data to obtain the target sub-log data; and sending the target sub-log data to the terminal.

[0078] According to an embodiment of this disclosure, after receiving a log query request from a terminal, it first determines whether the user identifier in the request is a management user identifier. If it is not a management user, the corresponding de-identified sub-log data is read from the database and sent to the terminal. If it is a management user, the encrypted sub-log data is read from the database and decrypted, and the decrypted sub-log data (i.e., the target sub-log data) is sent to the terminal.

[0079] According to embodiments of this disclosure, the encryption key and encrypted sub-log data can also be transmitted to the terminal multiple times and decrypted at the terminal to reduce the risk of data leakage.

[0080] According to embodiments of this disclosure, by differentiating different user roles and providing corresponding log data, the risk of sensitive information leakage can be reduced. For non-administrative users, providing de-identified data can hide sensitive data, reduce the risk of data leakage, and improve the availability and response speed of the system. For administrative users, providing decrypted logs can facilitate fault finding in the event of system failure.

[0081] According to an embodiment of this disclosure, the log management platform includes multiple log collection nodes, and the method further includes: in response to receiving raw log data sent by multiple log collection nodes, performing word segmentation processing on the raw log data to obtain word segmented log data, wherein the raw log data is natural language text data; performing part-of-speech tagging on the word segmented log data to obtain tagged log data; inputting the tagged log data into a preset natural language processing model, and outputting log data, wherein the log data is structured text data.

[0082] According to the embodiments of this disclosure, the above-mentioned raw log data is unprocessed log information collected from multiple log collection nodes belonging to different systems. This log information is often in different formats, usually natural language text information, and may contain data such as timestamps, event descriptions, and error codes.

[0083] According to embodiments of this disclosure, the aforementioned preset natural language processing model may be, for example, a BERT (Bidirectional Encoder Representations from Transformers) model, a GPT (Generative Pre-trained Transformer) model, etc., and this disclosure does not impose any limitations on it.

[0084] According to embodiments of this disclosure, by performing word segmentation on the original log data, the continuous text string is divided into meaningful individual words or phrases, and the parts of speech (such as nouns, verbs, adjectives, etc.) are labeled and input into a natural language processing model, thereby obtaining a structured text in a unified format, which improves the readability and usability of the original log data. Furthermore, by generating structured text data in a unified format, it is convenient for subsequent queries.

[0085] Based on the above log de-identification method, this disclosure also provides a log de-identification device. The following will combine... Figure 3 The device is described in detail.

[0086] Figure 3 A schematic block diagram of a log desensitization apparatus according to an embodiment of the present disclosure is shown.

[0087] like Figure 3 As shown, the log desensitization device 300 in this embodiment includes a hot-loading module 310, a parsing module 320, a query module 330, and a desensitization module 340.

[0088] The hot-loading module 310 is used to hot-load the de-identified configuration file to obtain de-identified configuration data during the operation of the log management platform. The de-identified configuration data includes a query identifier and preset operation data. The preset operation data is at least one of a predefined named field and de-identification rule data. The query identifier and the preset operation data have a corresponding relationship. In one embodiment, the hot-loading module 310 can be used to execute the operation S210 described above, which will not be repeated here.

[0089] The parsing module 320 is used to parse the log data in the log management platform to obtain multiple sub-log data. In one embodiment, the parsing module 320 can be used to perform the operation S220 described above, which will not be repeated here.

[0090] The query module 330 is used to query multiple sub-log data based on a query identifier to obtain the target sub-log data. In one embodiment, the query module 330 can be used to perform the operation S230 described above, which will not be repeated here.

[0091] The de-identification module 340 is used to de-identify the target sub-log data using the de-identification rule data when the preset operation data corresponding to the query identifier includes de-identification rule data, thereby obtaining de-identified sub-log data. In one embodiment, the de-identification module 340 can be used to perform the operation S240 described above, which will not be repeated here.

[0092] According to embodiments of this disclosure, the above-mentioned de-identification configuration file further includes query rules corresponding to the above-mentioned query identifier. The above-mentioned query module includes: a target query rule determination submodule, used to determine a target query rule from the above-mentioned query rules based on the target query identifier; and a target sub-log query submodule, used to query target sub-log data from the above-mentioned multiple sub-log data using the above-mentioned target query rule.

[0093] According to an embodiment of this disclosure, the target sub-log query submodule includes: a construction unit, configured to construct a state machine based on the target query rules; a state transition unit, configured to input the characters in the sub-log data into the state machine in an orderly manner for each sub-log data, so that the state machine performs state transition based on the input characters and the current state information to obtain transition state information; and a target sub-log determination unit, configured to determine the sub-log data as target sub-log data when the transition state information is end state information.

[0094] According to an embodiment of this disclosure, the target sub-log query submodule further includes a reset unit, used to reset the current state information to the initial state information when the input character is not present in the state machine.

[0095] According to embodiments of this disclosure, the apparatus further includes: an encryption module for generating encrypted sub-log data based on the target sub-log data using an encryption key; a structured log data generation module for generating structured log data based on the encrypted sub-log data and the de-identified sub-log data; and a storage module for storing the structured log data in a database.

[0096] According to an embodiment of this disclosure, the apparatus further includes: an acquisition module, configured to acquire a user identifier in the log query request in response to receiving a log query request from a terminal; and a first sending module, configured to send the de-identified sub-log data to the terminal if the user identifier is not a management user identifier.

[0097] According to an embodiment of this disclosure, the apparatus further includes: a decryption module, configured to decrypt the encrypted sub-log data using the encryption key to obtain the target sub-log data when the user identifier is an administrator user identifier; and a second sending module, configured to send the target sub-log data to the terminal.

[0098] According to embodiments of this disclosure, the log management platform includes multiple log collection nodes, and the device further includes: a word segmentation module, used to perform word segmentation processing on the raw log data sent by the multiple log collection nodes in response to receiving the raw log data to obtain word-segmented log data, wherein the raw log data is natural language text data; a tagging module, used to perform part-of-speech tagging on the word-segmented log data to obtain tagged log data; and an output module, used to input the tagged log data into a preset natural language processing model and output the log data, wherein the log data is structured text data.

[0099] According to embodiments of this disclosure, any multiple modules among the hot-loading module 310, parsing module 320, query module 330, and desensitization module 340 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the hot-loading module 310, parsing module 320, query module 330, and desensitization module 340 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the hot-loading module 310, parsing module 320, query module 330, and desensitization module 340 may be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0100] Figure 4 A block diagram schematically illustrates an electronic device suitable for implementing a log desensitization method according to an embodiment of the present disclosure.

[0101] like Figure 4As shown, an electronic device 400 according to an embodiment of the present disclosure includes a processor 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage portion 408 into a random access memory (RAM) 403. The processor 401 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 401 may also include onboard memory for caching purposes. The processor 401 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0102] RAM 403 stores various programs and data required for the operation of electronic device 400. Processor 401, ROM 402, and RAM 403 are interconnected via bus 404. Processor 401 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 402 and / or RAM 403. It should be noted that the programs may also be stored in one or more memories other than ROM 402 and RAM 403. Processor 401 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0103] According to embodiments of this disclosure, the electronic device 400 may further include an input / output (I / O) interface 405, which is also connected to a bus 404. The electronic device 400 may also include one or more of the following components connected to the input / output (I / O) interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the input / output (I / O) interface 405 as needed. A removable medium 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 410 as needed so that computer programs read from it can be installed into the storage section 408 as needed.

[0104] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0105] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 402 and / or RAM 403 and / or one or more memories other than ROM 402 and RAM 403 described above.

[0106] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the log de-identification method provided in the embodiments of this disclosure.

[0107] When the computer program is executed by the processor 401, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0108] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via communication section 409, and / or installed from removable medium 411. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0109] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by processor 401, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0110] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0111] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0112] Those skilled in the art will understand that the features described in the various embodiments of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments of this disclosure can be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0113] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A log anonymization method, characterized in that, The method is applied to a log management platform, and the method includes: During the operation of the log management platform, the de-identification configuration file is hot-loaded to obtain de-identification configuration data. The de-identification configuration data includes a query identifier and preset operation data. The preset operation data is at least one of a predefined naming field and de-identification rule data. The query identifier has a corresponding relationship with the preset operation data. The log data in the log management platform is parsed to obtain multiple sub-log data; Based on the query identifier, the target sub-log data is obtained by querying the multiple sub-log data; When the preset operation data corresponding to the query identifier includes de-identification rule data, the target sub-log data is de-identified using the de-identification rule data to obtain de-identified sub-log data; the de-identification configuration file also includes query rules corresponding to the query identifier, and the step of querying the multiple sub-log data based on the query identifier to obtain the target sub-log data includes: The target query rule is determined from the query rules based on the target query identifier; Using the target query rule, target sub-log data is retrieved from the multiple sub-log data; the process of retrieving target sub-log data from the multiple sub-log data using the target query rule includes: Construct a state machine based on the target query rules; For each sub-log data, the characters in the sub-log data are input into the state machine in order, so that the state machine performs state transition based on the input characters and the current state information to obtain the transition state information. If the transfer status information is an end status information, the sub-log data is determined as the target sub-log data.

2. The method according to claim 1, characterized in that, The method further includes: If the input character is not present in the state machine, the current state information is reset to the initial state information.

3. The method according to claim 1, characterized in that, The method further includes: Using the encryption key, encrypted sub-log data is generated based on the target sub-log data; Structured log data is generated based on the encrypted sub-log data and the de-identified sub-log data; The structured log data is saved to the database.

4. The method according to claim 3, characterized in that, The method further includes: In response to receiving a log query request from the terminal, the user identifier in the log query request is obtained; If the user identifier is not the management user identifier, the de-identified sub-log data is sent to the terminal.

5. The method according to claim 4, characterized in that, The method further includes: If the user identifier is an administrator user identifier, the encrypted sub-log data is decrypted using the encryption key to obtain the target sub-log data; The target sub-log data is sent to the terminal.

6. The method according to claim 1, characterized in that, The log management platform includes multiple log collection nodes, and the method further includes: In response to receiving raw log data sent by the plurality of log collection nodes, the raw log data is processed by word segmentation to obtain word segmented log data, wherein the raw log data is natural language text data; The segmented log data is tagged with part-of-speech tags to obtain tagged log data; The labeled log data is input into a preset natural language processing model, and the log data is output as structured text data.

7. A log desensitization device, characterized in that, The device is installed on a log management platform, and the device includes: The hot-loading module is used to hot-load the de-identified configuration file to obtain de-identified configuration data when the log management platform is running. The de-identified configuration data includes a query identifier and preset operation data. The preset operation data is at least one of a predefined named field and de-identification rule data. The query identifier has a corresponding relationship with the preset operation data. The parsing module is used to parse the log data in the log management platform to obtain multiple sub-log data; The query module is used to query the multiple sub-log data based on the query identifier to obtain the target sub-log data; The de-identification module is used to de-identify the target sub-log data using the de-identification rule data when the preset operation data corresponding to the query identifier includes de-identification rule data, so as to obtain de-identified sub-log data. The desensitization configuration file also includes query rules corresponding to the query identifier, and the query module includes: The target query rule determination submodule is used to determine the target query rule from the query rules based on the target query identifier; the target sub-log query submodule is used to query the target sub-log data from the multiple sub-log data using the target query rule; The target sub-log query sub-module includes: The system includes a construction unit for constructing a state machine based on the target query rules; a state transition unit for sequentially inputting characters from each sub-log data into the state machine, enabling the state machine to perform state transitions based on the input characters and current state information to obtain transition state information; and a target sub-log determination unit for determining the sub-log data as target sub-log data when the transition state information is an end state information.

8. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 6.

Images