Log information processing method and device, computer equipment and storage medium

Abstract

The invention relates to a log information processing method and device, computer equipment and a storage medium, and the method comprises the following steps: carrying out mark scanning on a to-be-processed log set based on a preset desensitization layer to obtain log sensitive information and log non-sensitive information; performing field positioning on the to-be-processed log set according to the log sensitive information to obtain a to-be-desensitized field set; performing desensitization processing based on a preset desensitization rule to obtain a desensitization field set; integrating the desensitization field set and the log non-sensitive information into a log entry set, and storing the log entry set into a preset annular queue to obtain an effective annular queue; performing asynchronous processing on the effective annular queue based on a preset thread pool to obtain a log processing result; and outputting the log processing result to the target log framework, and generating log record information. The method and the device can be applied to application scenes of a financial service system and a digital medical system, and effective and efficient desensitization and processing of the financial log data and the medical log data to be processed can be effectively realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, which can be applied to the fields of finance and medicine, and in particular relates to a log information processing method and device, a computer device and a storage medium. BACKGROUND

[0002] In today's financial technology field and digital medical field, with the rapid development of business and the explosive growth of data volume, log recording has become a key data resource for ensuring system security, stable operation, and problem tracking and auditing. However, logs often contain a large amount of sensitive information, such as user ID numbers, bank card numbers, mobile phone numbers in the financial technology field, and patient names, ID numbers, medical records, and test results in the digital medical field. These sensitive information involves important aspects such as user privacy, financial security, and medical health.

[0003] Currently, for log desensitization, the industry often uses regular matching for long text desensitization processing. Regular expressions are powerful in text matching, but when processing long text, backtracking problems often occur. Backtracking is a mechanism used by regular expression engines to try different paths to find matching text when a match fails. When the regular pattern is complex or the text is too long, the backtracking operation will increase significantly, causing negative effects in many aspects. In terms of performance, the system processing speed is greatly reduced, and the log recording efficiency is decreased, making it difficult to meet real-time and high-throughput requirements. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a log information processing method, device, computer device and storage medium to solve the problems of low desensitization and processing efficiency of log data.

[0005] In a first aspect, the embodiments of the present application provide a log information processing method, which adopts the technical solution as follows:

[0006] A set of logs to be processed is obtained, and the set of logs to be processed is marked and scanned based on a preset desensitization layer to obtain log sensitive information and log non-sensitive information;

[0007] The set of logs to be processed is field-positioned according to the log sensitive information to obtain a set of fields to be desensitized;

[0008] The set of fields to be desensitized is desensitized based on a preset desensitization rule to obtain a set of desensitized fields;

[0009] The set of desensitized fields and the log non-sensitive information are integrated into a set of log entries, and the set of log entries is stored in a preset ring queue to obtain an effective ring queue;

[0010] The effective ring queue is asynchronously processed based on a preset thread pool, and a log processing result is obtained.

[0011] The log processing result is output to a corresponding target log framework, and log record information is generated.

[0012] In a second aspect, the embodiments of the present application further provide a log information processing apparatus, which adopts the technical scheme as follows:

[0013] A log scanning module is configured to acquire a log set to be processed, and perform marked scanning on the log set to be processed based on a preset desensitization layer, so as to obtain log sensitive information and log non-sensitive information.

[0014] A field positioning module is configured to perform field positioning on the log set to be processed according to the log sensitive information, so as to obtain a field set to be desensitized.

[0015] A field desensitization module is configured to perform desensitization processing on the field set to be desensitized based on a preset desensitization rule, so as to obtain a desensitized field set.

[0016] A queue storage module is configured to integrate the desensitized field set and the log non-sensitive information into a log entry set, and store the log entry set into a preset ring queue, so as to obtain an effective ring queue.

[0017] A queue processing module is configured to perform asynchronous processing on the effective ring queue based on a preset thread pool, so as to obtain a log processing result.

[0018] A record generation module is configured to output the log processing result to a corresponding target log framework, and generate log record information.

[0019] In a third aspect, the embodiments of the present application further provide a computer device, which adopts the technical scheme as follows:

[0020] A computer device includes a memory and a processor, the memory stores computer readable instructions, and the processor executes the computer readable instructions to realize the steps of the log information processing method according to any one of the above.

[0021] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium, which adopts the technical scheme as follows:

[0022] A computer readable storage medium stores computer readable instructions, and the computer readable instructions are executed by a processor to realize the steps of the log information processing method according to any one of the above.

[0023] Compared with the prior art, the embodiment of the application has the following beneficial effects: the embodiment obtains a to-be-processed log set, performs marking scanning on the to-be-processed log set based on a preset desensitization layer, obtains log sensitive information and log non-sensitive information, performs field positioning on the to-be-processed log set according to the log sensitive information, obtains a to-be-desensitized field set, performs desensitization processing on the to-be-desensitized field set based on a preset desensitization rule, obtains a desensitized field set, integrates the desensitized field set and the log non-sensitive information into a log entry set, stores the log entry set into a preset ring queue, obtains an effective ring queue, performs asynchronous processing on the effective ring queue based on a preset thread pool, obtains a log processing result, and outputs the log processing result to a corresponding target log framework to generate log record information. Thus, effective and efficient desensitization and processing of to-be-processed log data are effectively realized. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the scheme in the application, the drawings needed in the description of the embodiments of the application will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0025] Figure 1 is an exemplary system architecture diagram to which the application can be applied;

[0026] Figure 2 a flowchart of one embodiment of the log information processing method according to the application;

[0027] Figure 3 is a structural schematic diagram of one embodiment of the log information processing device according to the application;

[0028] Figure 4 is a structural schematic diagram of one embodiment of the computer device according to the application. DETAILED DESCRIPTION

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the application belongs; the terms used in the specification of the application are only for the purpose of describing specific embodiments and are not intended to limit the application; the terms "include" and "have" and any variations thereof in the specification and claims of the application and the above description of drawings are intended to cover non-exclusive inclusion. The terms "first", "second" and the like in the specification and claims of the application or the above description of drawings are used to distinguish different objects, not to describe a specific order.

[0030] Reference to an "embodiment" herein means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily mutually exclusive or alternative embodiments. It is expressly understood that the embodiments described herein are merely example and that a person skilled in the art would readily recognize that claims may encompass a variety of embodiments.

[0031] In order to better understand the technical scheme of the present application, the technical scheme in the embodiments of the present application will be clearly and completely described below with reference to the drawings.

[0032] As shown in Figure 1 The system architecture 100 can include a terminal device 101, a network 102, and a server 103. The terminal device 101 can be a notebook computer 1011, a tablet computer 1012, or a mobile phone 1013. The network 102 is a medium for providing a communication link between the terminal device 101 and the server 103. The network 102 can include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0033] A user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0034] The terminal device 101 can be various electronic devices with a display screen and supporting web browsing. In addition to the notebook computer 1011, the tablet computer 1012, or the mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 (Moving Picture Experts Group Audio Layer III) player, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer, and a desktop computer, etc.

[0035] The server 103 can be a server providing various services, such as a background server supporting the pages displayed on the terminal device 101.

[0036] It should be noted that the log information processing method provided by the embodiments of the present application is generally executed by a server / terminal device, and accordingly, the log information processing apparatus is generally provided in a server / terminal device.

[0037] It should be understood that Figure 1The number of terminal devices, networks and servers in the system is only illustrative. Any number of terminal devices, networks and servers can be provided according to the implementation needs.

[0038] With reference to Figure 2 , a flow chart of one embodiment of a log information processing method according to the present application is shown. The log information processing method includes the following steps:

[0039] In step S10, a set of logs to be processed is obtained, and the set of logs to be processed is marked and scanned based on a preset desensitization layer to obtain log sensitive information and log non-sensitive information.

[0040] In this embodiment, the set of logs to be processed is a set of log data that needs to be processed. In the field of financial technology, a financial business system generates a large amount of logs, such as transaction logs, user login logs, etc. These logs contain various information, some of which may be sensitive, such as user bank card numbers, account balances, etc. The financial business system includes the background servers of online banking systems and mobile payment systems, etc. In the field of digital medicine, a digital medical system generates various logs, such as patient visit logs, medical device operation logs, etc. Taking a hospital information system as an example, when a patient makes an appointment, visits a doctor, and undergoes an examination, the data records generated by these operations constitute a set of logs to be processed. These logs may contain sensitive information such as the patient's name, ID number, and disease diagnosis. The preset desensitization layer will identify the sensitive information and non-sensitive information in the logs according to the preset rules. For example, for transaction logs, it will identify sensitive fields such as user identity information and bank card numbers and mark them. At the same time, it will distinguish other non-sensitive information such as transaction types (whether it is a transfer or a consumption). In the patient visit log, the patient's name, contact information, and disease diagnosis results will be marked as sensitive information. And fields such as the department of visit, the time of registration, etc. may be marked as non-sensitive information.

[0041] In step S20, the set of logs to be processed is field-positioned according to the log sensitive information to obtain a set of fields to be desensitized.

[0042] In this embodiment, the set of fields to be desensitized includes the position information of the fields to be desensitized. According to the marked log sensitive information, the system will accurately position the corresponding field position in the log. For example, the field where the bank card number is located in the log may be at a fixed position in the log, or it may be at a specific node in a structured data record. The sensitive field position in the log is accurately positioned. For example, in the log generated by an electronic medical record system, the field storing the patient's disease diagnosis result is found. This field may be at a specific position in the medical record data structure.

[0043] Step S30, desensitizing the set of fields to be desensitized based on a preset desensitization rule to obtain a set of desensitized fields.

[0044] In this embodiment, the set of desensitized fields is a set of log data after desensitizing the set of fields to be desensitized. The preset desensitization rule can include multiple ways. For example, for a bank card number, a partial masking method can be used, such as replacing the middle several digits of the bank card number with “****”. For a user ID card number, only part of the key information such as the birth date can be kept, and the rest is desensitized. For patient disease diagnosis information, fuzzy processing can be used, such as replacing the specific disease name with the first few digits of the disease code. For the contact information of the patient, part of the number can be hidden.

[0045] Step S40, integrating the set of desensitized fields and the non-sensitive information of the log into a set of log entries, and storing the set of log entries in a preset ring queue to obtain an effective ring queue;

[0046] In this embodiment, the desensitized fields and non-sensitive information are recombined into a complete set of log entries. For example, a transaction log entry can include desensitized bank card number, transaction type, transaction amount, and other information, and a patient visit log entry can include desensitized disease diagnosis information, department, doctor, and other information.

[0047] Step S50, based on a preset thread pool, asynchronously processing the effective ring queue to obtain a log processing result;

[0048] In this embodiment, the ring queue is a high-efficiency data storage structure. Storing the integrated log entries in the ring queue can ensure the ordered storage and fast reading of data. And the preset thread pool can asynchronously process these log entries. For example, multiple threads are used to simultaneously analyze the set of log entries in the effective ring queue, such as calculating the transaction success rate and other indicators in a certain time period, without affecting the normal writing and storage of logs.

[0049] Step S60, outputting the log processing result to the corresponding target log framework to generate log record information.

[0050] In the embodiment, the target log framework can be a system for log storage and management in a financial enterprise. The processed log results (such as transaction risk prompts analyzed) are output to the framework to generate complete log record information for auditing by a financial regulatory agency or for decision analysis by the enterprise itself, such as adjusting risk control strategies. The target log framework can also be a medical information management system of a hospital. The processed log results (such as medical resource use efficiency analyzed) are output to the system to generate log record information for decision making by hospital management, such as adjusting department staffing, and can also provide data support for medical regulatory departments.

[0051] The embodiment obtains a set of logs to be processed, performs marking scanning on the set of logs to be processed based on a preset desensitization layer to obtain log sensitive information and log non-sensitive information, performs field positioning on the set of logs to be processed according to the log sensitive information to obtain a set of fields to be desensitized, performs desensitization processing on the set of fields to be desensitized based on a preset desensitization rule to obtain a set of desensitized fields, integrates the set of desensitized fields and the log non-sensitive information into a set of log entries, stores the set of log entries in a preset ring queue to obtain an effective ring queue, performs asynchronous processing on the effective ring queue based on a preset thread pool to obtain a log processing result, and outputs the log processing result to a corresponding target log framework to generate log record information. Thus, effective and efficient desensitization and processing of the logs to be processed are effectively realized.

[0052] The method of the embodiment can be applied to log information processing in a digital medical system. The information system of a hospital collects pending logs generated in a series of medical processes such as registration, treatment, examination, and medicine taking. The logs contain information such as the name, ID number, contact information, disease diagnosis result, examination report, and medication record of a patient. A preset desensitization layer scans the logs, identifies and marks sensitive information such as the name, ID number, and disease diagnosis result of a patient, and distinguishes non-sensitive information such as the department of treatment, registration time, and name of a doctor. The specific positions of the sensitive fields such as the name, ID number, and disease diagnosis result of a patient in the logs are determined. The name of a patient is partially hidden, for example, only the surname is displayed and the given name is replaced with “*”. Part of the digits of the ID number is hidden. The disease diagnosis result is processed by fuzzification, for example, the specific disease name is replaced with the first few digits of the disease code. The desensitized name, ID number, and disease diagnosis result of a patient are integrated with the non-sensitive information such as the department of treatment, registration time, and name of a doctor to form complete patient treatment log entries. The integrated patient treatment log entries are stored in the preset ring queue of the hospital information system. The patient treatment log entries in the ring queue are processed asynchronously by using a preset thread pool, for example, the patient flow of each department of the hospital is counted and the treatment trend of a certain disease is analyzed. The processed patient treatment log results are output to the medical information management system of the hospital to generate complete patient treatment log records, which provide data support for the medical quality monitoring, resource allocation, and medical decision-making of the hospital.

[0053] The method of the embodiment can be applied to log information processing in a financial service system. The back-end system of an online bank collects all transaction logs within a period of time. The logs contain information such as the login information, transaction time, transaction amount, payee account, and payer bank card number of a user. A preset desensitization layer identifies sensitive information such as the bank card number and user ID number according to rules and marks them, and distinguishes non-sensitive information such as the transaction time and transaction type. The system locates the field positions of the bank card number and user ID number in the logs. The bank card number is partially masked, for example, several digits in the middle of the card number are replaced with “****”. Part of the digits of the ID number is hidden, and only the birth date and other information is retained. The desensitized bank card number and ID number are recombined with the non-sensitive information such as the transaction time, transaction type, and transaction amount to form complete transaction log entries. The integrated transaction log entries are stored in the preset ring queue of the system for subsequent processing. The transaction log entries in the ring queue are processed asynchronously by using a preset thread pool, for example, the success rate of transactions is counted and abnormal transaction situations are analyzed. The processed transaction log results are output to the log management framework of the online bank system to generate complete transaction log records, which facilitate auditing, data analysis, and risk monitoring of the bank.

[0054] In some optional implementations of the embodiment, the obtaining the to-be-processed log set, and performing label scanning on the to-be-processed log set based on the preset desensitization layer to obtain log sensitive information and log non-sensitive information includes the following steps.

[0055] Obtaining log identification information, and extracting an initial log set from a log database according to the log identification information;

[0056] In the embodiment, the log identification information is unique identification information corresponding to the initial log set, which can be an identifier composed of a log generation time range, a log source (such as a specific server, an application program module, etc.), a log level (such as an error log, a debugging log, etc.), or the like. The log identification information is used for matching and searching in the log database, so that the initial log set is quickly and accurately extracted.

[0057] Performing preprocessing on the initial log set to obtain the to-be-processed log set;

[0058] In the embodiment, the preprocessing of the initial log set includes various operations, such as removing duplicate log records, uniformly converting log formats (for example, converting different time formats into a standard format), filtering out obviously useless or irrelevant log contents to the target processing task, and correcting some obvious errors (such as correcting an error log level identifier, repairing a damaged log field, etc.). The to-be-processed log set is obtained by performing the above preprocessing on the initial log set.

[0059] Performing annotation label scanning on the to-be-processed log set based on the preset desensitization layer to obtain the log sensitive information and the log non-sensitive information.

[0060] In the embodiment, the system scans the to-be-processed log set after preprocessing according to the rules in the preset desensitization layer. In this process, the system checks each field or content in the log set according to the rules to determine whether it meets the characteristics of sensitive information. If it is sensitive information, it is marked to indicate that this part needs to be desensitized, and the log sensitive information is obtained. Otherwise, for non-sensitive information, it is marked as not needing desensitization, and the log non-sensitive information is obtained.

[0061] The embodiment obtains log identification information, extracts an initial log set from a log database according to the log identification information, performs preprocessing on the initial log set to obtain a to-be-processed log set, and performs annotation label scanning on the to-be-processed log set based on a preset desensitization layer to obtain log sensitive information and log non-sensitive information. Thus, preprocessing and marking of the to-be-processed log set are effectively realized to obtain corresponding log sensitive information and log non-sensitive information, which facilitates subsequent field positioning and log entry integration processing.

[0062] In some optional implementations of the present embodiment, the field positioning of the log sensitive information from the log set to be processed to obtain a set of fields to be desensitized includes the following steps:

[0063] Obtain a set of annotation markers corresponding to the log sensitive information;

[0064] In the present embodiment, the set of annotation markers is a set of specific markers or labels used to identify sensitive information in the log. These markers are generated according to the preset desensitization rules and strategies during the annotation marker scanning of the log, and are used to explicitly indicate which parts of the log belong to sensitive information. After completing the annotation marker scanning of the log sensitive information, the system will collect all the annotation markers corresponding to the log sensitive information to form the set of annotation markers.

[0065] According to the set of annotation markers, the corresponding sensitive field positioning of the log sensitive information is performed to obtain the desensitized field position information;

[0066] In the present embodiment, the purpose of sensitive field positioning is to determine the specific position of each sensitive information in the data structure in the log. This can be achieved by analyzing the marker content in the set of annotation markers, combined with the format and structure of the log. For example, according to the indication in the marker, the field name, field position index or the start and end position in the log text of the sensitive information can be determined. Among them, the desensitized field position information can be represented in various ways, such as in a database record, the column name and row number of the field can be used to identify, or in a text log, the start and end position offset of the character can be used to represent.

[0067] Integrate the desensitized field position information to obtain the set of fields to be desensitized.

[0068] In the present embodiment, the integration of the desensitized field position information is to summarize the scattered and individual field position information into a unified set. This set contains the position information of all fields that need to be desensitized. The integration process can include removing duplicate position information (if a sensitive field is repeatedly positioned in different markers), sorting and arranging the position information, so that the subsequent desensitization processing can operate in order and efficiently.

[0069] The embodiment obtains an annotation mark set corresponding to the log sensitive information, performs corresponding sensitive field positioning on the log sensitive information according to the annotation mark set, obtains to-be-desensitized field position information, and integrates the to-be-desensitized field position information to obtain the to-be-desensitized field set. Thus, the sensitive fields in the log sensitive information are effectively positioned and marked to obtain the to-be-desensitized field set after positioning, and subsequent desensitization processing is facilitated.

[0070] In some optional implementation of the embodiment, the desensitization processing of the to-be-desensitized field set based on the preset desensitization rule based on the multi-modal model includes the following steps:

[0071] The preset desensitization rule is obtained, and the preset desensitization rule is parsed to obtain a desensitization category rule.

[0072] In the embodiment, the preset desensitization rule is the basis and guideline of the entire desensitization process, which defines the desensitization manner to be adopted for different types of sensitive data. For example, personal identity information such as ID card number, bank card number, and mobile phone number, and confidential business data in enterprises should be provided with corresponding desensitization strategies. Parsing the preset desensitization rule can convert it into a desensitization category rule that can be understood and executed by the system. This process usually involves reading the rule file, syntax and semantic analysis of the rule content, and other operations, which converts the abstract and general desensitization rule into specific and executable instructions.

[0073] The field category corresponding to the to-be-desensitized field set is identified.

[0074] In the embodiment, the field category is a way of classifying sensitive data, which reflects the type and characteristics of sensitive data. The field category can include personal identity information (such as name, ID card number, mobile phone number, etc.), financial information (such as bank card number, account balance, etc.), medical and health information (such as medical records, diagnosis results, etc.), and enterprise confidential information (such as business plans, customer lists, etc.). The field category can be determined according to the field name, data format, data content, and business background information. For example, if the name of a field is “ID card number” and its data format conforms to the coding rules of ID card number (such as 18 digits and conforms to the corresponding check digit rules), it can be identified that the field belongs to the ID card number field in the personal identity information category.

[0075] The to-be-desensitized field set of the corresponding field category is desensitized according to the desensitization category rule to obtain the desensitized field set.

[0076] In this embodiment, different field categories have different sensitive characteristics and use scenarios, so different desensitization methods need to be adopted. For example, for the ID number in the personal identity information, a partial masking method (such as displaying only the first few and last few digits, and replacing the middle with asterisks) can be used, and for the bank card number, in addition to partial masking, hash encryption and other processing can also be performed. After desensitization processing is performed on all the desensitization fields in the desensitization field set, the obtained field set is the desensitization field set.

[0077] In this embodiment, the preset desensitization rule is obtained, and the preset desensitization rule is analyzed to obtain a desensitization category rule; the field category corresponding to the desensitization field set is identified; and the desensitization field set corresponding to the field category is desensitized according to the desensitization category rule, to obtain the desensitization field set. Thus, the desensitization field set after desensitization is obtained by effectively desensitizing the desensitization field set according to the preset desensitization rule, so as to facilitate subsequent integration processing of the log entry set.

[0078] In some optional implementation manners of this embodiment, the step of integrating the desensitization field set and the log non-sensitive information into a log entry set, and storing the log entry set in a preset ring queue to obtain an effective ring queue includes the following steps:

[0079] Obtaining field position information of the desensitization field set;

[0080] In this embodiment, the field position information refers to the specific position or position characteristics of the desensitization field in the original log record. It can be a character position offset in the log text, a field name or index in the log data structure, or a column name and row number in the log database.

[0081] According to the field position information, the desensitization field set is inserted into the log non-sensitive information to obtain the log entry set;

[0082] In this embodiment, according to the position of each desensitization field in the original log, the field position information is obtained, and each desensitization field in the desensitization field set is inserted into the corresponding position of the log non-sensitive information. The above process needs to ensure that the log entry set after insertion maintains consistency with the original log in data structure and content, except that the sensitive fields have been desensitized. For example, if the original log is a text string, the insertion operation is to replace the desensitized field string to the corresponding position in the original text according to the position offset; if the original log is a record in the database, the desensitized field is updated to the corresponding column and row position.

[0083] Identifying whether the preset ring queue is not full;

[0084] In the embodiment, the ring queue provides relevant state information or method, and whether the ring queue is full can be determined by querying the information. For example, whether the ring queue is full can be determined by comparing the number of currently stored log entries with the maximum capacity of the ring queue.

[0085] If the preset ring queue is not full, the log entry set is stored in the preset ring queue in timestamp order to obtain the effective ring queue.

[0086] In the embodiment, the timestamp is a time marker of log entry generation or processing, and storing the log entry set in timestamp order can ensure the time sequence of the log. When the log entry set is stored in the ring queue, the log entry set needs to be sorted first to ensure that they are stored in the queue in the order of timestamps. When the log entry is read from the queue for processing, the operation can be performed in the order of actual occurrence to ensure the integrity of the data and the accuracy of the processing.

[0087] If the preset ring queue is full, whether the preset ring queue is full is determined again after a predetermined time interval, until the preset ring queue is not full or the operation of storing the log entry set is stopped.

[0088] In the embodiment, whether the ring queue is full is determined again after a predetermined time interval, so that new log entry sets can be stored after the space in the queue is released by processing (such as being taken out by an asynchronous processing thread for subsequent processing) of the logs in the queue. If the queue remains full for a long time, the storage of the log entry set can be stopped after a certain number of attempts or a time limit is reached, so as to prevent the system resources from being excessively occupied and avoid affecting the stability and performance of the system.

[0089] In the embodiment, the field position information of the desensitization field set is obtained, the desensitization field set is inserted into the log non-sensitive information according to the field position information to obtain the log entry set, whether the preset ring queue is not full is identified, if the preset ring queue is not full, the log entry set is stored in the preset ring queue in timestamp order to obtain the effective ring queue, and if the preset ring queue is full, whether the preset ring queue is full is determined again after a predetermined time interval, until the preset ring queue is not full or the operation of storing the log entry set is stopped. Thus, the arrangement and storage of the log entry set are effectively realized to facilitate subsequent thread asynchronous processing.

[0090] In some optional implementation manners of the embodiment, the asynchronous processing of the effective ring queue based on the preset thread pool to obtain the log processing result includes the following steps:

[0091] extract the log item set from the effective ring queue according to a thread sequence of a preset thread pool;

[0092] In the embodiment, the preset thread pool is a mechanism for managing thread resources, which can create a certain number of threads in advance, and the threads can extract log item sets from the effective ring queue in sequence (such as first-in first-out).

[0093] perform log level judgment on the log item set to obtain log set level information;

[0094] In the embodiment, the log level is a way of classifying the importance and urgency of logs, and the log level includes DEBUG (debug), INFO (information), WARNING (warning), ERROR (error), CRITICAL (critical error), and the like. By performing log level judgment on the log item set, the level information of the log set can be determined, thereby providing a basis for subsequent log processing, such as deciding whether to process immediately, whether to send an alarm, and the like.

[0095] perform asynchronous processing on the log item set according to a corresponding log level strategy according to the log set level information, to obtain the log processing result.

[0096] In the embodiment, the log level strategy is a predefined processing manner for different log levels. For example, for logs of ERROR and CRITICAL levels, detailed error analysis may be performed immediately, the logs may be recorded to a specific error log file, an alarm may be triggered to notify an administrator, and the like; and for logs of INFO and DEBUG levels, the logs may be simply recorded to a normal log file, or discarded directly in some cases to save storage space. According to the log set level information, a corresponding strategy is selected for asynchronous processing, which can ensure that the logs are reasonably processed, while not affecting the running of the main thread, thereby improving the overall performance and response speed of the system. The result of asynchronous processing is the log processing result, which includes the storage location of the log, the processing state, the generated report, and the like.

[0097] The embodiment extracts the log item set from the effective ring queue according to a thread sequence of a preset thread pool, performs log level judgment on the log item set to obtain log set level information, and performs asynchronous processing on the log item set according to a corresponding log level strategy according to the log set level information, to obtain the log processing result. Thus, corresponding level strategies are adopted for targeted processing according to the log levels of different log item sets, to obtain accurate processed log processing results.

[0098] In some optional implementations of the embodiment, outputting the log processing result to the corresponding target log framework to generate log record information includes the following steps:

[0099] determining the target log framework according to the log processing result;

[0100] In this embodiment, factors such as the type of log processing result, the purpose, and the characteristics of the target system will affect the selection of the target log framework. For example, if the log needs to be deeply analyzed to mine business trends and problems, the target log framework can be a big data platform with powerful data analysis functions; if the log is mainly used for system monitoring and real-time alarm, the target framework can be a real-time monitoring system; if long-term storage and archiving of logs are required, the target framework can be a high-reliability cloud storage service.

[0101] format-adjusting the log processing result according to the output requirements of the target log framework to obtain an adjusted log processing result;

[0102] In this embodiment, different target log frameworks have different requirements for the format, structure, and content of logs. For example, one type of log framework may require logs to be stored in a specific JSON format, containing specific fields and metadata; another type of framework may require logs to be stored in plain text form and formatted according to specific requirements. In order for the log processing result to be correctly received, parsed, and used by the target log framework, the format of the log processing result needs to be adjusted according to its output requirements. Format adjustment operations include reorganization of log data, addition, deletion, or modification of fields, conversion of data types, addition or modification of metadata, etc. For example, converting the log processing result from the table structure format of one database to the JSON format required by the target log framework; or converting the timestamp format in the log from one standard to another according to the requirements of the target framework; additional information such as the source system identifier of the log, the unique number of the log, etc. can also be added to facilitate better management and differentiation of logs in the target log framework.

[0103] outputting the adjusted log processing result through the target log framework to generate the log record information.

[0104] In this embodiment, outputting the adjusted log processing result through the target log framework is to write the log data into the storage or display location of the target framework according to its specifications and processes. In this process, the target log framework may perform further processing on the log data, such as indexing, compression, encryption, backup, etc. to ensure the security, integrity, and availability of the log data. Ultimately, the generated log record information will exist in a way suitable for the target framework, which can be easily queried, analyzed, and used by users.

[0105] This embodiment determines the target log framework based on the log processing result; adjusts the format of the log processing result according to the output requirements of the target log framework to obtain an adjusted log processing result; and outputs the adjusted log processing result through the target log framework to generate the log record information. This effectively outputs the log processing result through the corresponding target log framework, thereby completing the effective processing of the log set to be processed.

[0106] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through computer-readable instructions. The computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes in the above-described method embodiments. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0107] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0108] Further references Figure 3 , as a response to the above Figure 1 The present application provides an embodiment of a log information processing device. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0109] like Figure 3 As shown, the log information processing device 700 of this embodiment includes: a log scanning module 701, a field positioning module 702, a field desensitization module 703, a queue storage module 704, a queue processing module 705, and a record generation module 706. Among them:

[0110] The log scanning module 701 is used to obtain a set of logs to be processed, and perform tag scanning on the set of logs to be processed based on a preset desensitization layer to obtain sensitive information and non-sensitive information of the logs;

[0111] The field positioning module 702 is configured to perform field positioning on the log set to be processed according to the log sensitive information, to obtain a field set to be desensitized.

[0112] The field desensitization module 703 is configured to perform desensitization processing on the field set to be desensitized based on a preset desensitization rule, to obtain a desensitized field set.

[0113] The queue storage module 704 is configured to integrate the desensitized field set and the log non-sensitive information into a log entry set, and store the log entry set into a preset ring queue, to obtain an effective ring queue.

[0114] The queue processing module 705 is configured to perform asynchronous processing on the effective ring queue based on a preset thread pool, to obtain a log processing result.

[0115] The record generation module 706 is configured to output the log processing result to a corresponding target log framework, to generate log record information.

[0116] The log information processing apparatus can obtain a log set to be processed, perform mark scanning on the log set to be processed based on a preset desensitization layer, to obtain log sensitive information and log non-sensitive information, perform field positioning on the log set to be processed according to the log sensitive information, to obtain a field set to be desensitized, perform desensitization processing on the field set to be desensitized based on a preset desensitization rule, to obtain a desensitized field set, integrate the desensitized field set and the log non-sensitive information into a log entry set, and store the log entry set into a preset ring queue, to obtain an effective ring queue, perform asynchronous processing on the effective ring queue based on a preset thread pool, to obtain a log processing result, and output the log processing result to a corresponding target log framework, to generate log record information. Thus, the log data to be processed can be effectively and efficiently desensitized and processed.

[0117] In some optional implementations of the embodiment, the log scanning module 701 includes a set extraction unit, a set preprocessing unit, and a mark scanning unit. Specifically,

[0118] The set extraction unit is configured to obtain log identification information, and extract an initial log set from a log database according to the log identification information.

[0119] The set preprocessing unit is configured to perform preprocessing on the initial log set, to obtain the log set to be processed.

[0120] The mark scanning unit is configured to perform annotation mark scanning on the log set to be processed based on a preset desensitization layer, to obtain the log sensitive information and the log non-sensitive information.

[0121] This embodiment provides a log scanning module 701 including a set extraction unit, a set preprocessing unit, and a tag scanning unit, thereby effectively preprocessing and tagging the log set to be processed to obtain corresponding log sensitive information and log non-sensitive information, thereby facilitating subsequent field positioning and log entry integration processing.

[0122] In some optional implementations of this embodiment, the field location module 702 includes: a set acquisition unit, a field location unit, and an information integration unit.

[0123] The set acquisition unit is used to acquire the annotation mark set corresponding to the sensitive information in the log;

[0124] The field locating unit is configured to locate the corresponding sensitive field of the log sensitive information according to the annotation mark set to obtain the location information of the field to be desensitized;

[0125] The information integration unit is used to integrate the position information of the fields to be desensitized to obtain the set of fields to be desensitized.

[0126] This embodiment effectively locates and marks the sensitive fields in the sensitive information of the log by setting up a field positioning module 702 including a set acquisition unit, a field positioning unit, and an information integration unit, so as to obtain a set of fields to be desensitized after positioning, which facilitates subsequent desensitization processing.

[0127] In some optional implementations of this embodiment, the field desensitization module 703 includes: a rule parsing unit, a category identification unit, and a desensitization processing unit.

[0128] The rule parsing unit is used to obtain a preset desensitization rule and parse the preset desensitization rule to obtain a desensitization category rule;

[0129] The category identification unit is used to identify the field category corresponding to the set of fields to be desensitized;

[0130] The desensitization processing unit is used to perform desensitization processing on the to-be-desensitized field set of the corresponding field category according to the desensitization category rule to obtain the desensitized field set.

[0131] This embodiment sets up a field desensitization module 703 including a rule parsing unit, a category identification unit, and a desensitization processing unit, thereby effectively desensitizing the desensitized field set according to the preset desensitization rules, and obtains the desensitized field set to facilitate the subsequent integration processing of the log entry set.

[0132] In some optional implementations of the embodiment, the queue storage module 704 includes a field information acquisition unit, an information integration unit, a queue judgment unit, a first processing unit, and a second processing unit. Wherein:

[0133] The field information acquisition unit is configured to acquire field position information of the desensitization field set;

[0134] The information integration unit is configured to insert the desensitization field set into the log non-sensitive information according to the field position information, to obtain the log entry set;

[0135] The queue judgment unit is configured to identify whether the preset ring queue is full;

[0136] The first processing unit is configured to, if the preset ring queue is not full, store the log entry set in the preset ring queue in a time stamp order, to obtain the effective ring queue;

[0137] The second processing unit is configured to, if the preset ring queue is full, judge whether the preset ring queue is full after a predetermined time interval, until the preset ring queue is not full or the operation of storing the log entry set is stopped.

[0138] The embodiment effectively realizes the arrangement and storage of the log entry set by setting the queue storage module 704 including the field information acquisition unit, the information integration unit, the queue judgment unit, the first processing unit, and the second processing unit, to facilitate subsequent thread asynchronous processing.

[0139] In some optional implementations of the embodiment, the queue processing module 705 includes a log set extraction unit, a log level judgment unit, and a log asynchronous processing unit. Wherein:

[0140] The log set extraction unit is configured to extract the log entry set from the effective ring queue according to a thread order of a preset thread pool;

[0141] The log level judgment unit is configured to perform log level judgment on the log entry set, to obtain log set level information;

[0142] The log asynchronous processing unit is configured to perform asynchronous processing on the log entry set according to a corresponding log level strategy according to the log set level information, to obtain the log processing result.

[0143] This embodiment sets up a queue processing module 705 including a log set extraction unit, a log level judgment unit, and a log asynchronous processing unit, thereby effectively implementing targeted processing based on the log levels of different log entry sets using corresponding level strategies to obtain accurately processed log processing results.

[0144] In some optional implementations of this embodiment, the record generation module 706 includes: a log frame determination unit, a result format adjustment unit, and a result output unit.

[0145] The log framework determining unit is configured to determine the target log framework according to the log processing result;

[0146] The result format adjustment unit is used to adjust the format of the log processing result according to the output requirements of the target log framework to obtain an adjusted log processing result;

[0147] The result output unit is used to output the adjustment log processing result through the target log framework to generate the log record information.

[0148] This embodiment provides a record generation module 706 including a log framework determination unit, a result format adjustment unit, and a result output unit, thereby effectively outputting the log processing results through the corresponding target log framework to complete the effective processing of the log set to be processed.

[0149] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0150] The computer device 8 includes a memory 81, a processor 82, and a network interface 83 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 8 with components 81-83, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0151] The computer device can be a desktop computer, a notebook computer, a palm computer, a cloud server, or the like. The computer device can interact with a user through a keyboard, a mouse, a remote controller, a touchpad, a voice control device, or the like.

[0152] The memory 81 includes at least one type of readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, or the like), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, or the like. In some embodiments, the memory 81 can be an internal storage unit of the computer device 8, such as a hard disk or a memory of the computer device 8. In other embodiments, the memory 81 can also be an external storage device of the computer device 8, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, or the like. Of course, the memory 81 can include both an internal storage unit and an external storage device of the computer device 8. In this embodiment, the memory 81 is generally used to store an operating system and various application software installed in the computer device 8, such as computer readable instructions of the log information processing method, or the like. In addition, the memory 81 can also be used to temporarily store various data that have been output or will be output.

[0153] The processor 82 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 82 is generally used to control the overall operation of the computer device 8. In this embodiment, the processor 82 is used to run computer readable instructions or process data stored in the memory 81, such as computer readable instructions of the log information processing method.

[0154] The network interface 83 can include a wireless network interface or a wired network interface, and is generally used to establish a communication connection between the computer device 8 and other electronic devices.

[0155] The embodiment can obtain a to-be-processed log set, perform mark scanning on the to-be-processed log set based on a preset desensitization layer, obtain log sensitive information and log non-sensitive information, perform field positioning on the to-be-processed log set according to the log sensitive information, obtain a to-be-desensitized field set, perform desensitization processing on the to-be-desensitized field set based on a preset desensitization rule, obtain a desensitized field set, integrate the desensitized field set and the log non-sensitive information into a log entry set, store the log entry set into a preset ring queue, obtain an effective ring queue, perform asynchronous processing on the effective ring queue based on a preset thread pool, obtain a log processing result, and output the log processing result to a corresponding target log framework to generate log record information. Thus, effective and efficient desensitization and processing of to-be-processed log data are effectively realized.

[0156] The application also provides another implementation, that is, a computer readable storage medium storing computer readable instructions executable by at least one processor to cause the at least one processor to perform the steps of the log information processing method as described above.

[0157] The embodiment can obtain a to-be-processed log set, perform mark scanning on the to-be-processed log set based on a preset desensitization layer, obtain log sensitive information and log non-sensitive information, perform field positioning on the to-be-processed log set according to the log sensitive information, obtain a to-be-desensitized field set, perform desensitization processing on the to-be-desensitized field set based on a preset desensitization rule, obtain a desensitized field set, integrate the desensitized field set and the log non-sensitive information into a log entry set, store the log entry set into a preset ring queue, obtain an effective ring queue, perform asynchronous processing on the effective ring queue based on a preset thread pool, obtain a log processing result, and output the log processing result to a corresponding target log framework to generate log record information. Thus, effective and efficient desensitization and processing of to-be-processed log data are effectively realized.

[0158] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be realized by means of software and a general hardware platform, and of course, can also be realized by hardware, but in many cases, the former is a better implementation. Based on such understanding, the technical solutions of the application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disk) and includes a plurality of instructions for causing a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the methods described in the various embodiments of the application.

[0159] Obviously, the above-described embodiments are only some embodiments but not all embodiments of the present application, the preferred embodiments of the present application are shown in the drawings, but do not limit the patent scope of the present application. The present application can be implemented in many different forms, and conversely, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing specific embodiments or equivalently replace some technical features thereof. Any equivalent structure made by using the content of the specification and drawings, directly or indirectly applied to other related technical fields, is also within the scope of the patent protection of the present application.

[0160] The non-company software tools or components appearing in the embodiments of the present application are only illustrative and do not represent actual use.

Claims

1. A log information processing method characterized by comprising: The method comprises the following steps: obtaining a to-be-processed log set, performing mark scanning on the to-be-processed log set based on a preset desensitization layer to obtain log sensitive information and log non-sensitive information; performing field positioning on the to-be-processed log set according to the log sensitive information to obtain a to-be-desensitized field set; performing desensitization processing on the to-be-desensitized field set based on a preset desensitization rule to obtain a desensitized field set; integrating the desensitized field set and the log non-sensitive information into a log entry set, and storing the log entry set in a preset ring queue to obtain an effective ring queue; performing asynchronous processing on the effective ring queue based on a preset thread pool to obtain a log processing result; outputting the log processing result to a corresponding target log framework to generate log record information.

2. The log information processing method according to claim 1, characterized by, The step of obtaining a to-be-processed log set and performing mark scanning on the to-be-processed log set based on a preset desensitization layer to obtain log sensitive information and log non-sensitive information comprises: obtaining log identification information, and extracting an initial log set from a log database according to the log identification information; preprocessing the initial log set to obtain the to-be-processed log set; performing annotation mark scanning on the to-be-processed log set based on a preset desensitization layer to obtain the log sensitive information and the log non-sensitive information.

3. The log information processing method according to claim 1, characterized by, The step of performing field positioning on the to-be-processed log set according to the log sensitive information to obtain a to-be-desensitized field set comprises: obtaining an annotation mark set corresponding to the log sensitive information; performing corresponding sensitive field positioning on the log sensitive information according to the annotation mark set to obtain to-be-desensitized field position information; integrating the to-be-desensitized field position information to obtain the to-be-desensitized field set.

4. The log information processing method according to claim 1, characterized by, The step of performing desensitization processing on the to-be-desensitized field set based on a preset desensitization rule to obtain a desensitized field set comprises: obtaining a preset desensitization rule and analyzing the preset desensitization rule to obtain a desensitization category rule; identifying a field category corresponding to the to-be-desensitized field set; performing desensitization processing on the to-be-desensitized field set of the corresponding field category according to the desensitization category rule to obtain the desensitized field set.

5. The log information processing method according to claim 1, characterized by, The step of integrating the desensitized field set and the log non-sensitive information into a log entry set, and storing the log entry set in a preset ring queue to obtain an effective ring queue comprises: obtaining field position information of the desensitized field set; inserting the desensitized field set into the log non-sensitive information according to the field position information to obtain the log entry set; identifying whether the preset ring queue is not full; if the preset ring queue is not full, storing the log entry set in the preset ring queue in a time stamp order to obtain the effective ring queue.

6. The log information processing method according to claim 1, characterized by, The step of performing asynchronous processing on the effective ring queue based on a preset thread pool to obtain a log processing result comprises: extracting the log entry set from the effective ring queue according to a thread order of a preset thread pool; performing log level judgment on the log entry set to obtain log set level information; According to the log set level information, the log entry set is asynchronously processed according to a corresponding log level policy, and the log processing result is obtained.

7. The log information processing method according to claim 1, characterized by, The step of outputting the log processing result to a corresponding target log framework to generate log record information specifically includes: determining the target log framework according to the log processing result; formatting the log processing result according to the output requirement of the target log framework to obtain an adjusted log processing result; outputting the adjusted log processing result through the target log framework to generate the log record information.

8. A log information processing apparatus characterized by comprising: Comprise: a log scanning module configured to obtain a log set to be processed, and perform label scanning on the log set to be processed based on a preset desensitization layer to obtain log sensitive information and log non-sensitive information; a field positioning module configured to perform field positioning on the log set to be processed based on the log sensitive information to obtain a field set to be desensitized; a field desensitization module configured to perform desensitization processing on the field set to be desensitized based on a preset desensitization rule to obtain a desensitized field set; a queue storage module configured to integrate the desensitized field set and the log non-sensitive information into a log entry set, and store the log entry set into a preset ring queue to obtain an effective ring queue; a queue processing module configured to perform asynchronous processing on the effective ring queue based on a preset thread pool to obtain a log processing result; a record generation module configured to output the log processing result to a corresponding target log framework to generate log record information.

9. A computer device, comprising: The computer readable storage medium stores computer readable instructions, and the computer readable instructions are executed by the processor to realize the steps of the log information processing method in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer readable instructions, and the computer readable instructions are executed by the processor to realize the steps of the log information processing method in any one of claims 1 to 7.

Images