Log desensitization method and device, equipment, medium and program product

Abstract

The invention provides a log desensitization method and device, equipment, a medium and a program product, and can be applied to the technical field of data analysis or the field of financial science and technology. The method comprises the steps that when a log management platform runs, a desensitization configuration file is subjected to hot loading to obtain desensitization configuration data, the desensitization configuration data comprises a query identifier and preset operation data, the preset operation data is at least one of a predefined named field and desensitization rule data, and the query identifier and the preset operation data have a corresponding relation; analyzing the log data in the log management platform to obtain multiple pieces of sub-log data; on the basis of the query identifier, querying the multiple pieces of sub-log data to obtain target sub-log data; and under the condition that the preset operation data corresponding to the query identifier comprises desensitization rule data, desensitizing the target sub-log data by utilizing the desensitization rule data to obtain desensitized sub-log data.

Description

Technical Field

[0001] The present disclosure relates to the field of data analysis technology or financial technology, and more specifically to a log desensitization method, device, equipment, medium and program product. Background Art

[0002] The log management platform is a service used to centrally collect, store, analyze and manage log files. It is essential for monitoring system status, analyzing application performance, security auditing and troubleshooting. In related technologies, the log management platform can only parse and extract fields from log data, but cannot desensitize log information. Existing desensitization methods require the log management platform to be restarted each time new desensitized information is configured, which is inefficient. Summary of the invention

[0003] In view of the above problems, the present disclosure provides a log desensitization method, apparatus, device, medium and program product.

[0004] According to the first aspect of the present disclosure, a log desensitizing method is provided, including: hot loading a desensitizing configuration file when the above-mentioned log management platform is running to obtain desensitizing configuration data, the above-mentioned desensitizing configuration data includes a query identifier and preset operation data, the above-mentioned preset operation data is at least one of a predefined named field and desensitizing rule data, and the above-mentioned query identifier has a corresponding relationship with the above-mentioned preset operation data; parsing the log data in the above-mentioned log management platform to obtain multiple sub-log data; based on the above-mentioned query identifier, querying the above-mentioned multiple sub-log data to obtain target sub-log data; in a case where the preset operation data corresponding to the above-mentioned query identifier includes desensitizing rule data, using the above-mentioned desensitizing rule data to desensitize the above-mentioned target sub-log data to obtain desensitized sub-log data.

[0005] According to an embodiment of the present disclosure, the above-mentioned desensitizing configuration file also includes a query rule corresponding to the above-mentioned query identifier, and the above-mentioned querying the above-mentioned multiple sub-log data based on the above-mentioned query identifier to obtain the target sub-log data includes: determining the target query rule from the above-mentioned query rule based on the target query identifier; using the above-mentioned target query rule to query and obtain the target sub-log data from the above-mentioned multiple sub-log data.

[0006] According to an embodiment of the present disclosure, the above-mentioned desensitizing configuration file also includes a query rule corresponding to the above-mentioned query identifier, and the above-mentioned querying the above-mentioned multiple sub-log data based on the above-mentioned query identifier to obtain the target sub-log data includes: determining the target query rule from the above-mentioned query rule based on the target query identifier; using the above-mentioned target query rule to query and obtain the target sub-log data from the above-mentioned multiple sub-log data.

[0007] According to an embodiment of the present disclosure, the method further includes: in a case where the input character does not exist in the state machine, resetting the current state information to initial state information.

[0008] According to an embodiment of the present disclosure, the above method also includes: using an encryption key to generate encrypted sub-log data based on the above-mentioned target sub-log data; generating structured log data based on the above-mentioned encrypted sub-log data and the above-mentioned desensitized sub-log data; and saving the above-mentioned structured log data into a database.

[0009] According to an embodiment of the present disclosure, the above method also includes: in response to receiving a log query request from a terminal, obtaining a user identifier in the above log query request; when the above user identifier is not an administrative user identifier, sending the above desensitized sub-log data to the above terminal.

[0010] According to an embodiment of the present disclosure, the method further includes: when the user identifier is a management user identifier, using the encryption key to decrypt the encrypted sub-log data to obtain the target sub-log data; and sending the target sub-log data to the terminal.

[0011] According to an embodiment of the present disclosure, the above-mentioned log management platform includes multiple log collection nodes, and the above-mentioned method also includes: in response to receiving the original log data sent by the above-mentioned multiple log collection nodes, performing word segmentation processing on the above-mentioned original log data to obtain word segmentation log data, and the above-mentioned original log data is natural language text data; performing part-of-speech tagging on the above-mentioned word segmentation log data to obtain tagged log data; inputting the above-mentioned tagged log data into a preset natural language processing model, and outputting the above-mentioned log data, and the above-mentioned log data is structured text data.

[0012] The second aspect of the present disclosure provides a log desensitization device, including: a hot loading module, used to hot load the desensitization configuration file when the above-mentioned log management platform is running to obtain desensitization configuration data, the above-mentioned desensitization configuration data includes a query identifier and preset operation data, the above-mentioned preset operation data is at least one of a predefined named field and desensitization rule data, and the above-mentioned query identifier has a corresponding relationship with the above-mentioned preset operation data; a parsing module, used to parse the log data in the above-mentioned log management platform to obtain multiple sub-log data; a query module, used to query the above-mentioned multiple sub-log data based on the above-mentioned query identifier to obtain target sub-log data; a desensitization module, used to desensitize the above-mentioned target sub-log data using the above-mentioned desensitization rule data to obtain desensitized sub-log data when the preset operation data corresponding to the above-mentioned query identifier includes desensitization rule data.

[0013] According to an embodiment of the present disclosure, the above-mentioned desensitizing configuration file also includes query rules corresponding to the above-mentioned query identifier, and the above-mentioned query module includes: a target query rule determination submodule, which is used to determine the target query rule from the above-mentioned query rules based on the target query identifier; a target sub-log query submodule, which is used to use the above-mentioned target query rule to query and obtain the target sub-log data from the above-mentioned multiple sub-log data.

[0014] According to an embodiment of the present disclosure, the above-mentioned target sub-log query sub-module includes: a construction unit, which is used to construct a state machine based on the above-mentioned target query rules; a state transfer unit, which is used to input the characters in the sub-log data into the above-mentioned state machine in order for each sub-log data, so that the above-mentioned state machine performs state transfer based on the input characters and current state information to obtain transfer state information; a target sub-log determination unit, which is used to determine the above-mentioned sub-log data as the target sub-log data when the above-mentioned transfer state information is the end state information.

[0015] According to an embodiment of the present disclosure, the target sub-log query sub-module further includes: a reset unit, configured to reset the current state information to initial state information when the input character does not exist in the state machine.

[0016] According to an embodiment of the present disclosure, the above-mentioned device also includes: an encryption module, which is used to generate encrypted sub-log data based on the above-mentioned target sub-log data using an encryption key; a structured log data generation module, which is used to generate structured log data based on the above-mentioned encrypted sub-log data and the above-mentioned desensitized sub-log data; and a saving module, which is used to save the above-mentioned structured log data into a database.

[0017] According to an embodiment of the present disclosure, the above-mentioned device also includes: an acquisition module, used to obtain the user identifier in the above-mentioned log query request in response to receiving the log query request from the terminal; a first sending module, used to send the above-mentioned desensitized sub-log data to the above-mentioned terminal when the above-mentioned user identifier is not an administrative user identifier.

[0018] According to an embodiment of the present disclosure, the above-mentioned device also includes: a decryption module, which is used to use the above-mentioned encryption key to decrypt the above-mentioned encrypted sub-log data to obtain the above-mentioned target sub-log data when the above-mentioned user identifier is an administrative user identifier; a second sending module, which is used to send the above-mentioned target sub-log data to the above-mentioned terminal.

[0019] According to an embodiment of the present disclosure, the above-mentioned log management platform includes multiple log collection nodes, and the above-mentioned device also includes: a word segmentation module, which is used to respond to the original log data sent by the above-mentioned multiple log collection nodes, and perform word segmentation processing on the above-mentioned original log data to obtain word segmentation log data, and the above-mentioned original log data is natural language text data; a tagging module, which is used to perform part-of-speech tagging on the above-mentioned word segmentation log data to obtain tagged log data; an output module, which is used to input the above-mentioned tagged log data into a preset natural language processing model, and output the above-mentioned log data, and the above-mentioned log data is structured text data.

[0020] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0021] The fourth aspect of the present disclosure further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the above computer program or instructions are executed by a processor.

[0022] The fifth aspect of the present disclosure further provides a computer program product, including a computer program or instructions, which implement the steps of the above method when the above computer program or instructions are executed by a processor.

[0023] According to the embodiments of the present disclosure, the log management platform loads the desensitizing configuration file at runtime to obtain the desensitizing configuration data, and finds the corresponding target sub-log data according to the query identifier in the desensitizing configuration data. When the preset operation data includes desensitizing rule data, the desensitizing rule data is used to desensitize the target sub-log to obtain the desensitized sub-log data. The desensitizing configuration file has high pertinence and flexibility, and the log management platform can hot-load the desensitizing configuration file without having to load it again after modifying the desensitizing configuration data. The modified desensitizing rule data can be directly used for desensitization, thereby improving efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0025] Figure 1 The following schematically shows an application scenario diagram of a log desensitization method and device according to an embodiment of the present disclosure;

[0026] Figure 2 The flowchart of the log desensitization method according to the embodiment of the present disclosure is schematically shown;

[0027] Figure 3 The structure block diagram of the log desensitization device according to the embodiment of the present disclosure is schematically shown;

[0028] Figure 4 A block diagram of an electronic device suitable for implementing a log desensitization method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0029] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0030] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0031] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0032] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0033] It should be noted that in the embodiments of the present disclosure, certain software, components, models and other existing solutions in the industry may be mentioned, which should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present disclosure, but it does not mean that the applicant has or will necessarily use the solution.

[0034] In the technical solution of the present disclosure, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0035] In the scenario of using personal information for automated decision-making, the methods, devices, and systems provided by the embodiments of the present disclosure provide users with corresponding operation portals for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating a person's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through computer programs, and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge and skills, and have reached a certain level of professionalism.

[0036] In the centralized log management platform, log data collected from multiple systems are stored. These log data include sensitive information such as name, ID number, landline, mobile phone number, address, email address, password, account number, etc. It is necessary to transform the sensitive information to protect personal privacy without affecting the use of data. In related technologies, the tools integrated in the log management platform only support decoding and extracting fields from regular logs, but cannot desensitize the logs. The related log desensitization method requires restarting the log management platform after configuring the desensitized information, which is inefficient.

[0037] In view of this, an embodiment of the present disclosure provides a log desensitization method, characterized in that the method is applied to a log management platform, and the method includes: hot loading a desensitization configuration file when the log management platform is running to obtain desensitization configuration data, the desensitization configuration data including a query identifier and preset operation data, the preset operation data is at least one of a predefined named field and desensitization rule data, and the query identifier and the preset operation data have a corresponding relationship; parsing the log data in the log management platform to obtain multiple sub-log data; based on the query identifier, querying the multiple sub-log data to obtain target sub-log data; when the preset operation data corresponding to the query identifier includes desensitization rule data, using the desensitization rule data to desensitize the target sub-log data to obtain desensitized sub-log data.

[0038] Figure 1The application scenario diagram of the log desensitization method and device according to the embodiments of the present disclosure is schematically shown.

[0039] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0040] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0041] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0042] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0043] It should be noted that the log desensitization method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the log desensitization device provided in the embodiment of the present disclosure can generally be set in the server 105. The log desensitization method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the log desensitization device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0044] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0045] The following will be based on Figure 1 The scene described by Figure 2 The log desensitization method of the disclosed embodiment is described in detail.

[0046] Figure 2 The flowchart of the log desensitization method according to an embodiment of the present disclosure is schematically shown.

[0047] like Figure 2 As shown, the log desensitization method of this embodiment includes operations S210 to S240.

[0048] In operation S210, the desensitizing configuration file is hot-loaded when the log management platform is running to obtain desensitizing configuration data.

[0049] Among them, the desensitizing configuration data includes a query identifier and preset operation data, the preset operation data is at least one of the predefined named fields and the desensitizing rule data, and the query identifier and the preset operation data have a corresponding relationship.

[0050] For ease of understanding, the present disclosure explains the following terms.

[0051] Hot reloading: It is a technology that allows the system to automatically reload code changes while the application is running, without having to stop and restart the entire application.

[0052] According to an embodiment of the present disclosure, the above-mentioned desensitizing configuration file can be written in grok syntax. The desensitizing configuration data can be, for example, %{SYNTAX:SEMANTIC}, where SYNTAX is a query identifier, SEMANTIC is preset operation data, and can include at least one of a predefined named field and desensitizing rule data. The complete desensitizing configuration data can be %{pattern:field:operation}. In order to be compatible with the original grok syntax, the desensitizing configuration data can also be configured as: %{pattern::operation}, %{pattern:field}, %{pattern:field:}, where pattern represents the query identifier, field represents the predefined named field, and operation represents the desensitizing rule data.

[0053] In operation S220, the log data in the log management platform is parsed to obtain a plurality of sub-log data.

[0054] According to an embodiment of the present disclosure, the above-mentioned log data includes multiple log data from different systems. The sub-log data may be separated by line breaks or may be custom-defined across lines. The log data is parsed to obtain multiple sub-log data.

[0055] In operation S230 , based on the query identifier, multiple sub-log data are queried to obtain target sub-log data.

[0056] In operation S240, when the preset operation data corresponding to the query identifier includes desensitization rule data, the target sub-log data is desensitized by using the desensitization rule data to obtain desensitized sub-log data.

[0057] According to the embodiment of the present disclosure, the corresponding target sub-log data can be found according to the query identifier, and the target sub-log data can be desensitized according to the desensitization rule data.

[0058] For example, for a log message such as "2024-01-01T12:34:56 INFO 192.168.1.10 [root] This is a sample log message.", the following configuration can be performed in the desensitization configuration data:

[0059] "SENSITIVE_IP replace(*.*.*.*)

[0060] SENSITIVE_USER replace("(\\d{0,1}\\d*)", "$1**"))

[0061] %{TIMESTAMP_ISO8601:timestamp}

[0062] %{LOGLEVEL:log_level} %{IP:client_ip: SENSITIVE_IP}

[0063] \[%{USERNAME:username: SENSITIVE_USER}\]

[0064] %{GREEDYDATA:message}”

[0065] Taking the time extraction of logs as an example, the query identifier is "TIMESTAMP_ISO8601", that is, the sub-log data matching the date and time format of the ISO 8601 standard is queried to obtain the target sub-log data "2024-01-01T12:34:56", and timestamp is a predefined named field. The final formatted sub-log data is "timestamp": "2024-01-01T12:34:56". Taking the user name extraction as an example, the query identifier is "USERNAME", "username" is a predefined named field, and "SENSITIVE_USER" is the desensitization rule data, that is, the corresponding user name is replaced based on the regular expression ("(\\d{0,1}\\d*)","$1**"), and the final desensitized sub-log data is "username": "r**". The log data extracted through the above desensitization configuration data is as follows: {"timestamp": "2024-01-01T12:34:56","log_level": "INFO","client_ip": "*.*.*.*", "username": "r**", "message": "This is a sample log message."}.

[0066] According to the embodiments of the present disclosure, the log management platform loads the desensitizing configuration file at runtime to obtain the desensitizing configuration data, and finds the corresponding target sub-log data according to the query identifier in the desensitizing configuration data. When the preset operation data includes desensitizing rule data, the desensitizing rule data is used to desensitize the target sub-log to obtain the desensitized sub-log data. The desensitizing configuration file has high pertinence and flexibility, and the log management platform can hot-load the desensitizing configuration file without having to load it again after modifying the desensitizing configuration data. The modified desensitizing rule data can be directly used for desensitization, thereby improving efficiency.

[0067] According to an embodiment of the present disclosure, the desensitizing configuration file also includes a query rule corresponding to the query identifier. Based on the query identifier, multiple sub-log data are queried to obtain target sub-log data, including: determining a target query rule from the query rules based on the target query identifier; and using the target query rule to query from multiple sub-log data to obtain the target sub-log data.

[0068] According to an embodiment of the present disclosure, the target query rule may be, for example, to query the log data using a regular expression, or to query the log data by string comparison or graph matching, etc., and the present disclosure does not limit this.

[0069] According to an embodiment of the present disclosure, target sub-log data is queried from multiple sub-log data using target query rules, including: building a state machine based on the target query rules; for each sub-log data, inputting the characters in the sub-log data into the state machine in order, so that the state machine performs state transition based on the input characters and current state information to obtain transition state information; when the transition state information is end state information, determining the sub-log data as the target sub-log data.

[0070] According to an embodiment of the present disclosure, a state machine may be constructed according to a target query rule, and the state machine may be, for example, a finite state machine.

[0071] For example, for a target query rule of "\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b", which is used to match an email address, a finite state machine will be constructed, which includes an initial state, an intermediate state (including multiple sub-intermediate states of matching characters), and an end state. "b[A-Za-z0-9._%+-]" is used to match the name part before the "@" identifier. When the finite state machine finds the character corresponding to the above regular expression, it remains in the intermediate state and waits for an "@" symbol, then matches the domain name data corresponding to "[A-Za-z0-9.-]+", and performs state transfer in the sub-intermediate state according to the input character. After matching the domain name data, the state machine matches the point data and the subsequent top-level domain name information of "[A-Za-z]{2,}". After all target query rules are matched successfully, the transfer state information reaches the end state, that is, the corresponding sub-log data can be determined as the target sub-log data.

[0072] According to an embodiment of the present disclosure, the method further includes: when there is no input character in the state machine, resetting the current state information to the initial state information.

[0073] According to the embodiments of the present disclosure, by configuring query rules corresponding to the query identifier in the desensitizing configuration file, the corresponding query rules can be directly found through the query identifier, and the corresponding sub-log data can be queried through the query rules, thereby increasing the scalability and flexibility of the program. Furthermore, by constructing a state machine through target query rules, the log management platform has greater flexibility, and by constructing a state machine, a comprehensive scan of all data can be avoided, thereby reducing the occupation of computing resources and alleviating the computing burden.

[0074] According to an embodiment of the present disclosure, the above method also includes: using an encryption key to generate encrypted sub-log data based on the target sub-log data; generating structured log data based on the encrypted sub-log data and the desensitized sub-log data; and saving the structured log data into a database.

[0075] According to the embodiments of the present disclosure, the target sub-log data can be encrypted by any one of a symmetric key, an asymmetric key and a hybrid key to generate encrypted sub-log data. The index data corresponding to the target sub-log data can also be generated by a hash algorithm such as the MD5 algorithm, the SHA-256 algorithm, the SHA-512 algorithm, etc. and saved in the database together, so that the corresponding encrypted sub-log data and the desensitized sub-log data can be quickly found based on the index data when the user searches.

[0076] According to an embodiment of the present disclosure, the method further includes: in response to receiving a log query request from a terminal, obtaining a user identifier in the log query request; and sending anonymized sub-log data to the terminal when the user identifier is not an administrative user identifier.

[0077] According to an embodiment of the present disclosure, the method further includes: when the user identifier is a management user identifier, using an encryption key to decrypt the encrypted sub-log data to obtain target sub-log data; and sending the target sub-log data to the terminal.

[0078] According to the embodiment of the present disclosure, after receiving a log query request sent by a terminal, it is first determined whether the user identifier in the request is an administrative user identifier. If it is not an administrative user, the corresponding anonymized sub-log data is read from the database and sent to the terminal. If it is an administrative user, the encrypted sub-log data is read from the database and decrypted, and the decrypted sub-log data (i.e., the target sub-log data) is sent to the terminal.

[0079] According to an embodiment of the present disclosure, the encryption key and the encrypted sub-log data may also be transmitted to the terminal in multiple times and decrypted at the terminal to reduce the risk of data leakage.

[0080] According to the embodiments of the present disclosure, by distinguishing different user roles and providing corresponding log data, the risk of sensitive information leakage can be reduced. For non-management users, providing desensitized data can hide sensitive data, reduce the risk of data leakage, and improve the availability and response speed of the system. For management users, decrypted logs are provided to facilitate fault finding in the event of a system failure.

[0081] According to an embodiment of the present disclosure, the above-mentioned log management platform includes multiple log collection nodes, and the method also includes: in response to receiving the original log data sent by the multiple log collection nodes, performing word segmentation processing on the original log data to obtain word segmentation log data, and the original log data is natural language text data; performing part-of-speech tagging on the word segmentation log data to obtain tagged log data; inputting the tagged log data into a preset natural language processing model, and outputting log data, and the log data is structured text data.

[0082] According to the embodiments of the present disclosure, the above-mentioned raw log data is unprocessed log information collected from multiple log collection nodes belonging to different systems. These log information are often in different formats, usually natural language text information, and may contain timestamps, event descriptions, error codes and other data.

[0083] According to an embodiment of the present disclosure, the above-mentioned preset natural language processing model can be, for example, a BERT (Bidirectional Encoder Representations from Transformers) model, a GPT (Generative Pre-trained Transformer) model, etc., and the present disclosure does not limit this.

[0084] According to an embodiment of the present disclosure, the original log data is segmented to divide continuous text strings into meaningful individual words or phrases, and the parts of speech (such as nouns, verbs, adjectives, etc.) are marked and input into a natural language processing model, thereby obtaining a structured text in a unified format, thereby improving the readability and usability of the original log data. Furthermore, by generating structured text data in a unified format, subsequent queries are facilitated.

[0085] Based on the above log desensitization method, the present disclosure also provides a log desensitization device. Figure 3 The device is described in detail.

[0086] Figure 3 The structural block diagram of the log desensitization device according to an embodiment of the present disclosure is schematically shown.

[0087] like Figure 3 As shown, the log desensitization device 300 of this embodiment includes a hot loading module 310 , a parsing module 320 , a query module 330 and a desensitization module 340 .

[0088] The hot loading module 310 is used to hot load the desensitization configuration file when the log management platform is running to obtain the desensitization configuration data, the desensitization configuration data includes a query identifier and preset operation data, the preset operation data is at least one of the predefined named fields and the desensitization rule data, and the query identifier has a corresponding relationship with the preset operation data. In one embodiment, the hot loading module 310 can be used to perform the operation S210 described above, which will not be repeated here.

[0089] The parsing module 320 is used to parse the log data in the log management platform to obtain multiple sub-log data. In one embodiment, the parsing module 320 can be used to perform the operation S220 described above, which will not be repeated here.

[0090] The query module 330 is used to query multiple sub-log data based on the query identifier to obtain target sub-log data. In one embodiment, the query module 330 can be used to perform the operation S230 described above, which will not be described in detail here.

[0091] The desensitizing module 340 is used to desensitize the target sub-log data using the desensitizing rule data to obtain desensitized sub-log data when the preset operation data corresponding to the query identifier includes desensitizing rule data. In one embodiment, the desensitizing module 340 can be used to perform the operation S240 described above, which will not be repeated here.

[0092] According to an embodiment of the present disclosure, the above-mentioned desensitizing configuration file also includes query rules corresponding to the above-mentioned query identifier, and the above-mentioned query module includes: a target query rule determination submodule, which is used to determine the target query rule from the above-mentioned query rules based on the target query identifier; a target sub-log query submodule, which is used to use the above-mentioned target query rule to query and obtain the target sub-log data from the above-mentioned multiple sub-log data.

[0093] According to an embodiment of the present disclosure, the above-mentioned target sub-log query sub-module includes: a construction unit, which is used to construct a state machine based on the above-mentioned target query rules; a state transfer unit, which is used to input the characters in the sub-log data into the above-mentioned state machine in order for each sub-log data, so that the above-mentioned state machine performs state transfer based on the input characters and current state information to obtain transfer state information; a target sub-log determination unit, which is used to determine the above-mentioned sub-log data as the target sub-log data when the above-mentioned transfer state information is the end state information.

[0094] According to an embodiment of the present disclosure, the target sub-log query sub-module further includes: a reset unit, configured to reset the current state information to initial state information when the input character does not exist in the state machine.

[0095] According to an embodiment of the present disclosure, the above-mentioned device also includes: an encryption module, which is used to generate encrypted sub-log data based on the above-mentioned target sub-log data using an encryption key; a structured log data generation module, which is used to generate structured log data based on the above-mentioned encrypted sub-log data and the above-mentioned desensitized sub-log data; and a saving module, which is used to save the above-mentioned structured log data into a database.

[0096] According to an embodiment of the present disclosure, the above-mentioned device also includes: an acquisition module, used to obtain the user identifier in the above-mentioned log query request in response to receiving the log query request from the terminal; a first sending module, used to send the above-mentioned desensitized sub-log data to the above-mentioned terminal when the above-mentioned user identifier is not an administrative user identifier.

[0097] According to an embodiment of the present disclosure, the above-mentioned device also includes: a decryption module, which is used to use the above-mentioned encryption key to decrypt the above-mentioned encrypted sub-log data to obtain the above-mentioned target sub-log data when the above-mentioned user identifier is an administrative user identifier; a second sending module, which is used to send the above-mentioned target sub-log data to the above-mentioned terminal.

[0098] According to an embodiment of the present disclosure, the above-mentioned log management platform includes multiple log collection nodes, and the above-mentioned device also includes: a word segmentation module, which is used to respond to the original log data sent by the above-mentioned multiple log collection nodes, and perform word segmentation processing on the above-mentioned original log data to obtain word segmentation log data, and the above-mentioned original log data is natural language text data; a tagging module, which is used to perform part-of-speech tagging on the above-mentioned word segmentation log data to obtain tagged log data; an output module, which is used to input the above-mentioned tagged log data into a preset natural language processing model, and output the above-mentioned log data, and the above-mentioned log data is structured text data.

[0099] According to an embodiment of the present disclosure, any multiple modules of the hot loading module 310, the parsing module 320, the query module 330 and the desensitization module 340 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the hot loading module 310, the parsing module 320, the query module 330 and the desensitization module 340 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in an appropriate combination of any of them. Alternatively, at least one of the hot loading module 310, the parsing module 320, the query module 330 and the desensitization module 340 may be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function may be executed.

[0100] Figure 4 A block diagram of an electronic device suitable for implementing a log desensitization method according to an embodiment of the present disclosure is schematically shown.

[0101] like Figure 4As shown, the electronic device 400 according to an embodiment of the present disclosure includes a processor 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage part 408 to a random access memory (RAM) 403. The processor 401 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 401 may also include an onboard memory for caching purposes. The processor 401 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0102] In RAM 403, various programs and data required for the operation of electronic device 400 are stored. Processor 401, ROM 402 and RAM 403 are connected to each other via bus 404. Processor 401 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 402 and / or RAM 403. It should be noted that the program can also be stored in one or more memories other than ROM 402 and RAM 403. Processor 401 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0103] According to an embodiment of the present disclosure, the electronic device 400 may further include an input / output (I / O) interface 405, which is also connected to the bus 404. The electronic device 400 may further include one or more of the following components connected to the input / output (I / O) interface 405: an input portion 406 including a keyboard, a mouse, etc.; an output portion 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 408 including a hard disk, etc.; and a communication portion 409 including a network interface card such as a LAN card, a modem, etc. The communication portion 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the input / output (I / O) interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 410 as needed, so that a computer program read therefrom is installed into the storage portion 408 as needed.

[0104] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0105] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 402 and / or RAM 403 described above and / or one or more memories other than ROM 402 and RAM 403.

[0106] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program includes a program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the log desensitization method provided by the embodiment of the present disclosure.

[0107] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 401. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0108] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 409, and / or installed from the removable medium 411. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0109] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 409, and / or installed from the removable medium 411. When the computer program is executed by the processor 401, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, apparatus, module, unit, etc. described above can be implemented by a computer program module.

[0110] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0111] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0112] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0113] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A log desensitization method, characterized in that: The method is applied to a log management platform, and the method includes: When the log management platform is running, hot loading the desensitization configuration file to obtain desensitization configuration data, wherein the desensitization configuration data includes a query identifier and preset operation data, wherein the preset operation data is at least one of a predefined named field and desensitization rule data, and the query identifier has a corresponding relationship with the preset operation data; Parsing the log data in the log management platform to obtain multiple sub-log data; Based on the query identifier, query the plurality of sub-log data to obtain target sub-log data; In the case where the preset operation data corresponding to the query identifier includes desensitizing rule data, the target sub-log data is desensitized using the desensitizing rule data to obtain desensitized sub-log data.

2. The method according to claim 1, characterized in that The desensitization configuration file also includes a query rule corresponding to the query identifier, and querying the plurality of sub-log data based on the query identifier to obtain target sub-log data includes: Determining a target query rule from the query rules based on the target query identifier; The target query rule is used to query and obtain target sub-log data from the multiple sub-log data.

3. The method according to claim 2, characterized in that The using the target query rule to query and obtain target sub-log data from the plurality of sub-log data includes: Building a state machine based on the target query rule; For each piece of sub-log data, the characters in the sub-log data are sequentially input into the state machine, so that the state machine performs state transition based on the input characters and current state information to obtain transition state information; When the transfer status information is end status information, the sub-log data is determined as target sub-log data.

4. The method according to claim 3, characterized in that The method further comprises: In the case that the input character does not exist in the state machine, the current state information is reset to initial state information.

5. The method according to claim 1, characterized in that The method further comprises: generating encrypted sub-log data based on the target sub-log data using an encryption key; Generating structured log data based on the encrypted sub-log data and the desensitized sub-log data; The structured log data is saved in a database.

6. The method according to claim 5, characterized in that The method further comprises: In response to receiving a log query request from a terminal, obtaining a user identifier in the log query request; When the user identifier is not an administrative user identifier, the desensitized sub-log data is sent to the terminal.

7. The method according to claim 6, characterized in that The method further comprises: In the case where the user identifier is a management user identifier, using the encryption key to decrypt the encrypted sub-log data to obtain the target sub-log data; The target sub-log data is sent to the terminal.

8. The method according to claim 1, characterized in that The log management platform includes a plurality of log collection nodes, and the method further includes: In response to receiving the original log data sent by the multiple log collection nodes, performing word segmentation processing on the original log data to obtain word segmentation log data, wherein the original log data is natural language text data; Performing part-of-speech tagging on the word segmentation log data to obtain tagged log data; The annotated log data is input into a preset natural language processing model, and the log data is output, where the log data is structured text data.

9. A log desensitization device, characterized in that: The device is installed on a log management platform, and includes: A hot loading module, used for hot loading the desensitization configuration file to obtain desensitization configuration data when the log management platform is running, wherein the desensitization configuration data includes a query identifier and preset operation data, wherein the preset operation data is at least one of a predefined named field and desensitization rule data, and the query identifier has a corresponding relationship with the preset operation data; A parsing module, used to parse the log data in the log management platform to obtain multiple sub-log data; A query module, configured to query the plurality of sub-log data to obtain target sub-log data based on the query identifier; The desensitizing module is used to desensitize the target sub-log data using the desensitizing rule data to obtain the desensitized sub-log data when the preset operation data corresponding to the query identifier includes desensitizing rule data.

10. An electronic device, comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

12. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images