Log processing system, configuration file generation components and storage media

Abstract

The embodiments of the present invention provide a configuration file generation component of a log processing system, a log processing system and a storage medium. The configuration file generation component includes a configuration interface, which is used to provide a number of selectable log templates to a user and display a number of display layer metadata; the display layer metadata can be input by a user and / or can be converted by the log template using a predetermined rule to read the log data of a specified type of log file; a first converter is used to derive a number of hidden information based on the display layer metadata and in combination with the selected log template and the log type associated with the log template, and send the display layer metadata and the derived hidden information as underlying metadata to a second converter; the second converter is used to generate a configuration file associated with the selected log template using the underlying metadata. The embodiments of the present invention can automatically generate a configuration file.

Description

Technical Field

[0001] Embodiments of the present invention relate to log management, and more particularly to a configuration file generation component of a log processing system, a log processing system, and a computer-readable storage medium. Background Art

[0002] The log processing system can help monitor software performance by collecting, analyzing, and visualizing log data. Usually, the log processing system needs to use multiple components such as log collection components, log cache components, log processing and usage components to complete different processing processes for log files. For each component, a dedicated configuration file needs to be configured. In each component of the system, the configuration file is used to define how to process and transmit log data. Usually, the configuration file is written by the user who uses each component. Each user needs to understand the configuration method of the component they use, as well as the regular expressions needed to write the configuration file, which has a high technical threshold. Summary of the invention

[0003] In view of the above technical problems, the present invention provides a configuration file generation component of a log processing system, a log processing system and a computer-readable storage medium, and the technical solution is as follows:

[0004] In a first aspect of an embodiment of the present invention, a configuration file generation component of a log processing system is provided, comprising:

[0005] A configuration interface, used to provide a number of selectable log templates to the user and display a number of presentation layer metadata; the presentation layer metadata can be input by the user and / or can be converted by the log template reading log data of a log file of a specified type using predetermined rules;

[0006] A first converter, configured to derive a plurality of hidden information based on the presentation layer metadata and in combination with the selected log template and the log type associated with the log template, and send the presentation layer metadata and the derived hidden information as underlying metadata to a second converter;

[0007] The second converter is used to generate a configuration file associated with the selected log template by using the underlying metadata.

[0008] According to a second aspect of the present invention, a log processing system is provided, comprising:

[0009] Several components for processing log files, used to process log files differently as different nodes on the log transmission link; and

[0010] The configuration file generation component described in the first aspect; each configuration file generated by the configuration file generation component corresponds to a component for processing log files.

[0011] According to a third aspect of the embodiments of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the function of any component in the first aspect and the second aspect of the embodiments of the present invention is realized.

[0012] The technical solution provided by the present invention provides users with a set of easy-to-understand log templates for users to select appropriate templates according to the type of log files. The log template uses predetermined rules to read log files of specified types. In order to design a log template, an embodiment of the present invention defines metadata to reflect the predetermined rules, and defines how to read data from log files and convert them into data in a specified format in the log template through the predetermined rules. The embodiment of the present invention divides metadata into display layer metadata and underlying metadata. Display layer metadata is metadata visible to users. Since these metadata are user-oriented, semantic expressions that are easy for users to understand are used. It can be seen that users can input the data to be configured through easy-to-understand semantic expressions, and automatically generate configuration files through the rules defined by the metadata, thereby simplifying the user's operation difficulty.

[0013] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0015] Figure 1 It is a schematic diagram of a log processing system of the related technology;

[0016] Figure 2 is a schematic diagram of a configuration file generation component in an embodiment of the present invention;

[0017] Figure 3 is an example of segmenting log file data in an embodiment of the present invention;

[0018] Figure 4 is a schematic diagram of a log processing system in an embodiment of the present invention;

[0019] Figure 5 is a flow chart of a configuration file generation component generating a configuration file in an embodiment of the present invention;

[0020] Figure 6a-6e It is an example of partial functional implementation of the configuration file generation component in the embodiment of the present invention;

[0021] Figure 7 is a flowchart of updating metadata in an embodiment of the present invention;

[0022] Figure 8 It is a schematic diagram that reflects the blood relationship of metadata. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be described in detail below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field should belong to the scope of protection of the present invention.

[0024] In the field of software development, log files refer to files automatically recorded by the software system, which contain detailed information about the system during operation. By uploading logs to the log processing system, users can use log data more effectively and improve the reliability, security and performance of the system. The core components of the log processing system usually include log collection components, log cache components, log processing and use components, log collection configuration components, etc. Figure 1 It shows the architecture of a log processing system based on the ELK (Elasticsearch, Logstash, Kibana) stack. ELK is a widely used open source platform for searching, analyzing, and visualizing log data.

[0025] The components in the figure constitute the log transmission link for each node in the log processing system to transmit log files. In order to complete its function, each component requires a special configuration file for configuration.

[0026] Configuration files are used to define how log data is processed and transmitted.

[0027] Flume Agent or Filebeat Agent components are log collection components responsible for collecting log data from the application system. The application system is the source of log data that generates log files and can be any software application or service. Flume and Filebeat are currently common open source log collection tools.

[0028] The Kafka component is a distributed stream processing platform, often used as a log cache component to build real-time data pipelines and stream applications. In the log processing system, Kafka can be used as a buffer for log file data to ensure data reliability and scalability.

[0029] The Logstash component is a server-side data processing pipeline that serves as a log processing and usage component for unified data collection and processing. It receives data from the Kafka component, can filter, transform, and enrich the data, and then sends the processed data to Elasticsearch.

[0030] The Elasticsearch component is a Lucene-based search engine. As a log collection configuration component, it is used to store, search, and analyze log data.

[0031] The Kibana component is an open source analysis and visualization platform designed to work with the Elasticsearch component. It allows users to visualize data in the Elasticsearch component in the form of charts, maps, and tables.

[0032] It is worth explaining. Figure 1 This is an example of a log management system built using open source tools. Each component can also be developed by developers themselves. Since the configuration files for each component need to be written separately by technicians, not only is the technical threshold high, but the operation is also complicated.

[0033] The embodiments of the present invention provide the following solutions to the problem of high technical threshold for generating configuration files.

[0034] The embodiment of the present invention provides a set of easy-to-understand log templates for users to input configuration data in the log templates, and each log file is associated with a set of log templates. The log template uses predetermined rules to read log files of specified types. In order to design a log template, the embodiment of the present invention defines log metadata to reflect the predetermined rules, and defines how to read data from the log file and convert it into formatted data in the log template through the predetermined rules.

[0035] The embodiment of the present invention divides log metadata into display layer data and underlying metadata. Display layer metadata is log metadata visible to users. Since these log metadata are user-oriented, semantic expressions that are easy for users to understand are used. Underlying metadata is log metadata that is invisible to users.

[0036] The embodiment of the present invention classifies log files according to the purpose of the logs, provides several optional templates for each type of log file, abstracts several log metadata into semantic expressions that are easy for users to understand and display them in the log template selected by the user, and allows the user to input other log metadata in the log template through easy-to-understand semantic expressions, and then uses these log metadata in the log template as the basis for generating a configuration file, automatically helping the user generate a configuration file, and simplifying the difficulty of operation.

[0037] Figure 2 The configuration file generation component 200 of the log processing system provided in the embodiment of the present invention provides a configuration interface 201, which provides a number of selectable log templates to the user and displays a number of presentation layer metadata 202; the presentation layer metadata can be input by the user, and / or can be converted by the log template using predetermined rules to read the log data of the specified type of log file.

[0038] The component also involves two converters. The first converter 203 obtains the presentation layer metadata 202 displayed on the configuration interface 201, and derives some hidden information based on the presentation layer metadata 202 and in combination with the selected log template and the log type associated with the log template. The hidden information is log metadata that does not appear as the presentation layer metadata 202 but is indispensable for generating the configuration file. The hidden information and the presentation layer metadata obtained from the configuration interface 201 are sent to the second converter 205 as the underlying metadata 204.

[0039] The second converter 205 generates a configuration file 206 associated with the selected log template using the underlying metadata.

[0040] The configuration interface 201 may have several configuration item controls that can be edited by the user, and the presentation layer metadata input by the user may be data input by the user using the configuration controls. The presentation layer metadata converted from the log data read by the log template is formatted data.

[0041] In the embodiment of the present invention, the types of log files can be divided into access logs, business logs and application logs, and a set of corresponding log templates is provided for each type of log.

[0042] Access logs record logs of application containers being accessed, such as tomcat accesslog, nginx accesslog, api gateway accesslog, etc. Tomcat AccessLog is a log generated by the Tomcat server, which records all HTTP requests sent to the Tomcat server. Nginx AccessLog is a log generated by the Nginx mail proxy server, which records all HTTP requests passing through Nginx. API Gateway AccessLog is a log generated by the API Gateway, which records all HTTP requests passing through the API Gateway.

[0043] It can be seen that access log records are generated when the application is accessed. Therefore, they are the basis for implementing full-link retrieval of the log transmission link and play an important role in troubleshooting through log files.

[0044] For access logs, the log metadata of the implementation mode of the present invention defines the rules for extracting log data. The predetermined rules can refer to the following examples:

[0045] Taking the log file format in Nginx's access_log configuration as an example, an example of extracting log data from the log file that Nginx records each HTTP request can be:

[0046] %h%l%u%t"%r"%s%b%D"%{X-Forwarded-For}i""%{User-Agent}i""%{Referer}i""%{yx_aui}c""%{yx_from}c""%{P_INFO}c""%{NTES-TRACEID}i""%{X-TRACEID}i""%{Host}i"

[0047] The meaning of each parameter is as follows:

[0048] %h: The host IP address of the request.

[0049] %l: Identifier of the request.

[0050] %u: The requested username.

[0051] %t: Timestamp of the request.

[0052] "%r": The requested line.

[0053] %s: The HTTP status code of the request.

[0054] %b: The size of the response body in bytes.

[0055] %D: Number of microseconds for request processing.

[0056] "%{X-Forwarded-For}i": The content of the X-Forwarded-For request header, which is used to obtain the client's real IP address.

[0057] : The content of the User-Agent request header, which indicates the information of the client browser.

[0058] "%{Referer}i": The content of the Referer request header, indicating the source page of the request.

[0059] "%{yx_aui}c": the content of the yx_aui request header, which is a custom request header and its specific meaning depends on the application.

[0060] "%{yx_from}c": the content of the yx_from request header, which is also a custom request header.

[0061] "%{P_INFO}c": The content of the P_INFO request header, which is also a custom request header.

[0062] "%{NTES-TRACEID}i": The content of the NTES-TRACEID request header, which is used to trace the request ID, usually used in distributed systems.

[0063] "%{X-TRACEID}i": The content of the X-TRACEID request header, which is also the ID used to trace the request.

[0064] "%{Host}i": The content of the Host request header, indicating the host name of the request.

[0065] Business logs can be defined in advance by users. Each business log records logs related to the business. Such logs can be part of the system design and are generally used in scenarios such as data statistical analysis, business monitoring, and log alarms.

[0066] The following is an example of a predetermined rule defined through log metadata to extract log data from a business log:

[0067] time=[time], traceId=[XTraceId], module=[module], topic=[topic], id=[id], message=[message]

[0068] Here is what each field means:

[0069] time=[time]: indicates the timestamp of the log record.

[0070] traceId = [XTraceId]: indicates the trace ID or request ID. This is a unique identifier.

[0071] module=[module]: indicates the name of the service module or component that generates the log.

[0072] topic = [topic]: indicates the function of the business module related to the log entry, such as "user management" and "order processing".

[0073] id=[id]: represents the unique identifier associated with the log entry.

[0074] message = [message]: indicates the specific message content of the log entry. This is a description of the event that occurred, which may include error information, status updates, or other related information. In the embodiment of the present invention, event refers specifically to the message body of the encapsulated log data stream transmitted on the log transmission link.

[0075] Application logs are logs other than the access logs and the business logs. Application logs allow multiple lines, and the format may not be agreed upon in advance. For the same tenant, several pre-agreed formats may be defined as predetermined rules.

[0076] The following is an example of a log template that extracts application logs through log metadata using the Jason structure definition:

[0077] {time=[time],

[0078] traceId = [XTraceId],

[0079] level=[level],

[0080] mdcKey1=[mdcValue1],...}[logMessage]

[0081] Here are the meanings of the parameters:

[0082] time=[time]: indicates the timestamp of the log record.

[0083] traceId=[XTraceId]: indicates the trace ID or request ID.

[0084] level=[level]: indicates the log level, such as INFO, DEBUG, WARN, ERROR, etc.

[0085] mdcKey1=[mdcValue1]: represents a key-value pair in a mapped diagnostic context (MDC).

[0086] [logMessage]: represents the actual log message.

[0087] To facilitate understanding, the following examples are given for display layer log metadata and underlying layer metadata:

[0088] The presentation layer log metadata may include one or more of the following data:

[0089] The log metadata number (logId) is used as the unique number of the log metadata corresponding to each log, which can be automatically generated.

[0090] Log ownership and file information, and the service code (serviceCode) to which the associated log belongs.

[0091] Service Code (serviceCode), which is the service code of CMDB (Configuration Management DataBase) service. Based on the service code, the tenant, product, service, host list and user list of each role of the service can be obtained, so that the user can only select the authorized service list. In the embodiment of the present invention, authorization means that the user belongs to a certain role of the service. CMDB: Configuration Management Database (Configuration ManagementDataBase), used to manage basic data such as personnel, hardware, software and relationships in IT organizations. The following are the definitions of the following related names of CMDB involved in the implementation of the present invention:

[0092] Tenant: An entity maintained by the CMDB administrator, containing multiple products. Each tenant can manage multiple products, but each product can only belong to one tenant.

[0093] Product: An entity applied for by the product owner and approved by the administrator. It can contain multiple services and must belong to a single tenant.

[0094] Service: An independent publishing unit subordinate to a product. Each service belongs to a product and can contain multiple service instances, which are uniquely identified by a service code.

[0095] Service instance: A specific service process running on a host, representing the actual running entity of the service.

[0096] Host: A physical or virtual device, such as a physical machine, virtual machine, or container, that hosts a service instance.

[0097] User: An individual with basic information such as email address and name, who may be assigned specific roles.

[0098] Role: A category of permissions granted to users associated with a service or product, including technical managers, product managers, data managers, R&D personnel, testers, monitors, etc.

[0099] The presentation layer log metadata also includes:

[0100] File information, including log file type, log corresponding file directory, file name, etc. File directory and file name configuration supports wildcard and variable replacement.

[0101] File directory example: / home / logs / ${serviceName}

[0102] Example file name: stdout.log.yyyy-MM-dd

[0103] The presentation layer metadata also defines the corresponding data model after the data in the log file is segmented. The fields in the data model are displayed in the log template in the form of structured data, which can include field name, field description, type, constraint conditions, whether to be used for indexing, time field and other information. Figure 3 For example, the key column is the field name of several fields after being split from a log file. The table shows the string type, description, constraint, field type, and whether Elasticsearch is used for data retrieval for these split fields.

[0104] The presentation layer metadata also defines the field types supported by the data in the log template, such as 0 for boolean (Boolean type), 1 for int (integer), 2 for long (long integer), 3 for double (double-precision floating point number), 4 for string (string), 5 for timeString (formatted time string, several common types are predefined), 6 for jsonObjectString (i.e. object), and 7 for jsonArrayString (i.e. array).

[0105] The presentation layer metadata also defines the constraints of the configuration fields supported by the log template, such as time field formatting checks, enumeration values, ranges, non-empty, etc.

[0106] In addition, the presentation layer metadata can also include some additional information, such as:

[0107] The multi-line beginning configuration can be used to configure the beginning judgment rules of multi-line logs for application logs. In the implementation of the present invention, it can be designed to be automatically generated according to the log template by default and allow users to modify it.

[0108] Whether it contains sensitive information;

[0109] Whether to trace the data and how long to trace the data;

[0110] Whether it is an audit log;

[0111] Test text;

[0112] Estimated peak value. In the example of the present invention, this additional information may be optional. When creating a new log template, the user may set the peak TPS (Transactions Per Second) of the log printing.

[0113] In addition to the same log metadata as the presentation layer log metadata, the underlying log metadata also includes hidden information. The hidden information can be derived based on the presentation layer metadata in combination with the associated log template and log type.

[0114] The hidden information may include: log format, log segmentation regular expression, resource grouping, traffic configuration, log header, resource pool and other information.

[0115] The log format is generated through a log template, which corresponds to the Pattern in the log printing configuration in the application code that generates the log file.

[0116] The log segmentation regular expression is automatically generated for the user by the log template. Each log processing component on the log transmission link can convert log lines into structured data through the regular expression of this field.

[0117] Resource grouping is also automatically generated through log templates and can be designed to allow operators (users of the system that operates and maintains log processing) to adjust it. Through resource grouping, the various components that make up the log transmission link can be determined, so that resource information related to log collection, processing and storage can be associated. For example, it can be server resources for accessing various log processing components. For example, resource pools can include: data routing resource pool, Kafka resource pool, ES resource pool, Flink resource pool, data lake configuration, etc.

[0118] The flow control configuration is automatically generated by the log template based on resources and historical traffic.

[0119] The log header is used to record the log header transparent transmission type, log header transparent transmission type, Kafka Topic name, Kafka table name, data warehouse table name, etc.

[0120] The log processing system can match the associated collection, processing, and storage resources based on resource grouping and attributes in the log metadata (log type, whether to enter the data lake, whether to audit), thereby determining which components on the log transmission link are needed to process the log files.

[0121] It is worth pointing out that in order to simplify the log configuration management interaction process, the implementation mode of the present invention designs the configuration file generation process as a one-click publishing operation mode, that is, through the user's configuration in the selected log template, the configuration file corresponding to each component in the entire log transmission link can be automatically generated. The user generates multiple configuration files through one operation, and each configuration file corresponds to a component for processing log files on the log transmission link.

[0122] In addition, the log processing system mentioned in the embodiment of the present invention supports a multi-tenant architecture. The customized log specifications in the log template are different for different tenants, and different tenants can be isolated so that each tenant has an independent storage space.

[0123] Figure 4 It is a schematic diagram of a log processing system 400 including the above-mentioned configuration file generation component. As shown in the figure, the log processing system 400 includes several components for processing log files, which are used to perform different processing on log files as different nodes on the log transmission link; and a configuration file generation component 200; the configuration file generation component 200 generates a corresponding configuration file for each component that processes log files. Components that process log files, such as log collection components, log cache components, log processing and use components, and log collection configuration components, can be referenced. Figure 1 Please understand and I will not elaborate on it here.

[0124] See also Figure 5 , which exemplarily shows the process of generating a configuration file using the configuration file generation component of the log processing system.

[0125] S501, when a user needs to collect a new log file, he will visit the log management list page to query the services and log file list that the currently logged-in user is allowed to access. In addition to this method, you can also obtain the specified service and log file list through search. Figure 6a As shown, the log files accessible to the user are A and B. The service corresponding to log A is "bounty service", and the service corresponding to log B is "platform basic service".

[0126] S502, click the "Add Log" entry to open the "Add Log" page, and the interface presents several display layer metadata, such as Figure 6b Example (only part of the display layer data is shown in the figure), the user selects the appropriate log category and log template according to their own needs, and completes the relevant configuration data in the configuration control. The configuration file generation component automatically completes the log metadata based on the predetermined rules of the log template.

[0127] The configuration file generation component also provides a function that allows users to quickly add fields. Since the metadata-related rules have been pre-defined, the field name, field type, description and constraints are inferred through the accessed log files A and B, and the structured data is quickly generated from the data segmented from the log files. Figure 6c As shown in the figure, users can quickly generate new fields in the original log metadata by entering the new fields "age" and "name" in the text box.

[0128] If there is a test text in the presentation layer metadata, the data in the log template is tested in S503 to automatically complete the verification of the log metadata. For an example of the test text, refer to Figure 6d . Users can determine whether the test text is consistent with the expected format based on the test text; users can also modify the test text for testing. After the user clicks the test, it verifies whether the regular expression, constraint conditions, etc. match. If it fails, the specific reason is prompted. If it passes, the log metadata of the test is displayed in a structured form.

[0129] In S504, the user submits the configured log metadata and generates an approval flow.

[0130] In the implementation mode of the present invention, the service technical person in charge approves the request by default, and notifies the data person in charge. If the request fails to pass the approval, the editing permission is reopened, and the user enters S502 to modify the presentation layer metadata.

[0131] S505 publishes the approved log metadata. After publishing, the configuration file generation component automatically generates a configuration file and sends it to each component to start the log file collection process.

[0132] Users can use data monitoring to confirm in real time whether log collection is normal. Figure 6e shown.

[0133] In the implementation mode of the present invention, the configuration file generation component provides a log replication function. Through this function, users can quickly copy existing log metadata and adjust the service to which it belongs. Through the log replication method, you can choose to share Kafka Topic, ES index, data warehouse table name and other information with the replication source (that is, the replicated log template).

[0134] In an embodiment of the present invention, the configuration file generation component provides the function of modifying log metadata. Without closing the collection entrance of the collected log file, the log metadata can be modified by adjusting the multi-line beginning configuration (i.e., multi-line judgment rules), resource grouping and other display layer metadata of the application log.

[0135] In the implementation manner of the present invention, if other attributes need to be modified, the entry to the log collection file needs to be closed first, and the entry to the log collection file can be reopened after verification after the modification.

[0136] In the embodiment of the present invention, the log templates provided by the configuration file generation component include general templates, which support pre-configuration by the administrators of the platform operator and the tenant. The general template can assist in quickly filling in the new log stage. The general template provides several pre-defined regular pools to write concise regular expressions. Users can select several regular expressions as needed and splice them into the final regular expression required. By reusing variables, it helps administrators to Figure 1 For example, IPORHOST can match IPv4 or IPv6 or host name. IPORHOST can directly reuse the regular expressions of IP and HOSTNAME. When configuring the log template, you can also reference these expressions in the regular pool by using %{paramKey}. The specific form of the regular expression will not be described in detail.

[0137] When the user is using the configuration file to generate the components, the components included in the log processing system may be updated. Therefore, it is necessary to update the configuration file in a timely manner. In the embodiment of the present invention, a control flow for the components that process log files is also designed. Automatically generate log component control information based on log metadata, log collection resource pool data and CMDB data, see Figure 4 (Log metadata management process and log component control process)

[0138] After the configuration file is generated, the log metadata change message is triggered by the successful addition or modification of logs, changes in the server list caused by expansion and contraction, changes in the log component resource pool, etc. The log metadata change message is sent to MQ (message queue, which refers to the message middleware used for message publishing and subscription, such as Kafka and RocketMQ), and the configuration management service started by the configuration file generation component monitors it. For the processing process, please refer to Figure 7 .

[0139] S701: Get and process messages from the MQ queue;

[0140] S702: Check whether the configuration needs to be updated. If so, execute S703. Otherwise, discard and continue to execute step S701.

[0141] S703: Generate configuration control information of the log component;

[0142] S704: Release configuration control information. For example, a transit control service may be used to send control information to various components of the log transmission link to complete the configuration release action.

[0143] After the configuration file is sent to each component, each component on the log transmission link uses the configuration file to complete the processing of the collected log files:

[0144] The log collection end (the flume / fliebeat agent component in the figure) monitors configuration changes in the form of a plug-in and pulls the changed configuration in real time to complete the control.

[0145] Components such as Kafka components, Flink components, ElasticSearch components, data lakes, and data warehouses provide APIs to receive configuration control information.

[0146] The configuration control information of the log components corresponding to each component on the log transmission link is as follows:

[0147] The configuration control information of the log collection component includes:

[0148] Information indicating the source of data, such as log metadata, data routing resource pool, CMDB, etc.

[0149] Information that represents the composition of data, such as locating files. Locating files is a list of servers for log collection that can be found through the service code (serviceCode) of the log metadata and the service instance (Instance) of the CMDB. The directory and file name of the corresponding server can be found through the file information of the log metadata. If wildcards are included, they need to be converted into regular expressions.

[0150] Control the messages encapsulated by events (i.e. log messages on the log transmission link), for example, how to identify multi-line logs, define event extension headers, such as adding information such as IP, HOST, resource grouping, etc. to the extension header, and desensitize sensitive information (that is, set sensitive information to non-sensitive information).

[0151] Messages that control resource allocation, for example, resource grouping and data routing through log metadata to control data flow, setting flow limits through flow control configuration, and controlling the resources occupied by collection.

[0152] The configuration control information of Flink components includes:

[0153] Information indicating the data source, such as log metadata, Flink resource pool, etc.

[0154] Information that represents the data structure, such as the Flink task name (the purpose is to ensure uniqueness based on the log meta-information numbering) and the corresponding resource grouping, such as CPU, memory, and totalSlots.

[0155] The configuration control information of Kafka components includes:

[0156] Information indicating the data source, such as log metadata + Kafka resource pool.

[0157] Information that represents the composition of the data, such as topic (used to ensure uniqueness based on the log metadata numbering), cluster, partition, etc.

[0158] The configuration control information of the data lake components includes:

[0159] Information indicating the source of data, such as log metadata, data lake cluster configuration, etc.

[0160] Information that represents the composition of data, such as Hive table name (based on the log metadata information numbering to ensure uniqueness), cluster, partition (based on the date extracted from the time column defined in the log metadata), field (based on the log metadata definition), etc.

[0161] The configuration control information of ES components includes:

[0162] Information indicating the data source, such as log metadata and ES resource pool.

[0163] Information indicating the composition of data, such as ES index name (based on the log metadata number to ensure uniqueness) and ES mapping field (based on the log metadata definition).

[0164] Information indicating a log rotation policy and information indicating a log cleanup policy, where the log cleanup policy is used to determine the log retention period based on whether the log is used for auditing.

[0165] In the implementation manner of the present invention, log metadata can be used for system operation, such as strategic log downgrading, partially recovering resources used for log collection and processing, and reducing server load. For example, business peaks can be discovered based on log metadata, and scheduled downgrading operations can be performed before the business peak occurs, or temporary downgrading can be performed to deal with sudden traffic. The downgrade rule can be to downgrade the collection of application logs, such as current limiting, until the collection is closed, and other operations. For business logs and access logs, the default implementation manner of the present invention is not to downgrade, but the platform operator is allowed to actively initiate downgrades. The log processing system automatically filters the current-limited logs from low to high according to the priority of the resource grouping until the blockage disappears.

[0166] In addition, the implementation manner of the present invention can also set a log list that is exempted from downgrading to ensure that the specified log files are not downgraded and collected.

[0167] The embodiments of the present invention also provide a solution for the quality management of the entire life cycle of logs by the log processing system, and perform data quality control on each link of the entire life cycle, such as log creation, collection, use, and deletion. The log processing system also includes a log data lineage analysis component, which defines the lineage relationship of log data in the log file based on the underlying metadata, the component that processes the log file, and the monitoring data reported by the component that processes the log file, and visualizes the health status of the log file in the log transmission link based on the defined lineage relationship.

[0168] Figure 8 Generate a visual log transmission link diagram based on the lineage of log data.

[0169] The log source includes the application information of the printed log (service code, service name, etc.), log directory and log file name.

[0170] The log collection task includes the health status of the server and the collection process (flume), the health status of the log collection task, the running time, the last heartbeat information, and the log update time.

[0171] The log temporary storage area includes the Kafka cluster, the corresponding topic and the update time.

[0172] Log processing includes the health status and running time of tasks such as log segmentation, log data synchronization, and log data monitoring.

[0173] Log applications include ES clusters, ES indexes, and update time.

[0174] Users can monitor the dashboard and monitor alarm items based on the visualization of log data lineage, such as abnormalities in Hive tables and data warehouse business indicators.

[0175] In an embodiment of the present invention, the log processing system also includes a quality control component, which monitors the log data in the log file in real time based on the health status of the log metadata displayed by the log data lineage analysis component, and performs exception processing when an exception is monitored.

[0176] When the quality control component finds dirty data, the exception handling that can be performed may include one or more of the following methods, such as closing the log, deleting the log, forcibly deleting the log, modifying the log, scoring the log quality, or identifying the log quality risks.

[0177] Log closing: The quality control component checks whether there are running data applications (such as dashboards, monitoring items, data warehouse business indicators, etc.) through the lineage relationship of log metadata. If so, the user confirms whether to close the log (that is, stop collecting). In addition, the relevant person in charge of the service can be notified through instant messaging tools or emails.

[0178] Log deletion: Check the lineage of log data to see if there are any running data applications (such as dashboards, monitoring items, data warehouse business indicators, etc.). If so, deletion is not allowed (log metadata is deleted) until the dependency of the related application has been removed. Similarly, the person in charge of the service can be notified through instant messaging tools or emails.

[0179] A forced deletion function is provided, which allows forced deletion of log metadata after approval by the service technical manager and data manager.

[0180] Log modification: When changing log metadata, the consistency between the new log metadata and the current log metadata is detected in real time, and the compatibility of the new log metadata with the data warehouse table model is inferred, such as field type compatibility and whether the fields actually used in the data warehouse are deleted.

[0181] The quality control component can also score the log data of the log file based on the above quality monitoring results, and improve the quality of the relevant log data according to the quality score. The scoring rule example is as follows:

[0182] The default total score of the quality score is 100 points, and a deduction system is adopted, with an upper limit of 100 points. The deduction items include:

[0183] If log data segmentation fails (for example, the reason for the failure is that the segmented log data does not match the predefined format), one point will be deducted for each line of log.

[0184] If the constraint check defined in the log metadata fails, one point will be deducted for each log line.

[0185] In the embodiment of the present invention, multiple lines of logs in the application log can be counted as one line. The upper limit of the deduction points for each log per day depends on the error rate of the statistical period, with each 10 points as one level, rounded up.

[0186] The embodiments of the present invention provide a calculation formula for the error rate, and an example is as follows:

[0187] Error rate = number of error lines in the statistical period / total number of lines in the statistical period. For example, if a total of 100 lines of logs are generated in the statistical period, and 8 lines are erroneous (resulting in deductions), the error rate is 8%.

[0188] If the error rate is 8%, the maximum deduction point is 10 points, and if the error rate is 90.1%, the maximum deduction point is 100 points.

[0189] In addition, the penalty can decay over time. Specific examples could be:

[0190] The sliding window algorithm is used to record only the deductions in the past 30 days. Deductions over 30 days are no longer counted. The deductions on the nth day are decayed to 1 / n of the original deduction value. The deduction value can be a decimal. If the original deduction value is 10 points, the deduction after 3 days is 10 / 3, that is, 3.33 points, and the deduction after 10 days is 10 / 10, that is, 1 point.

[0191] Regarding the rules for deducting points, the embodiments of the present invention also provide a reset entry, and the deducting points are calculated again after the reset.

[0192] The embodiment of the present invention provides a quality risk identification rule based on the log quality score, for example, if the quality score trend decreases or is lower than 80 points, the relevant person in charge of the service responsible for collecting the log is notified. And the notified person is allowed to perform special processing on specific logs, such as adding the log to the whitelist and no longer alarming within a specific period.

[0193] The implementation manner of the present invention can display the log quality score as a log quality red and black list as one of the measurement indicators of service stability.

[0194] The red and black list can display four dimensions: product dimension, service dimension, log metadata dimension, and responsible person dimension.

[0195] Improve the quality of log data based on log quality score. The improvement process can be:

[0196] Step 1: Find the corresponding log and click Data Monitoring to view the corresponding data anomaly details;

[0197] Step 2: Check the status curve of log processing (distribution of abnormal logs);

[0198] Step 3: Check the sampling of the exception log. If the number of errors in the last hour is 0, you can expand the time range;

[0199] Step 4: Exception handling

[0200] Common exceptions: including regular expression matching exceptions, constraint conditions not being met, JSON format errors, etc.

[0201] Corrections can be made based on the abnormal information, such as:

[0202] For regular expression matching exceptions, it is necessary to determine whether the log printing exception occurs or the log template needs to be upgraded.

[0203] For exceptions caused by constraint failure, it is necessary to determine whether the exception occurs in log printing or in modifying the constraint.

[0204] The present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the functions of the various components described in any of the above embodiments are implemented.

[0205] The above description is only a specific implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A configuration file generation component of a log processing system, characterized in that: include: A configuration interface, used to provide a number of selectable log templates to the user and display a number of presentation layer metadata; the presentation layer metadata can be input by the user and / or can be converted by the log template reading log data of a log file of a specified type using predetermined rules; A first converter, configured to derive a plurality of hidden information based on the presentation layer metadata and in combination with the selected log template and the log type associated with the log template, and send the presentation layer metadata and the derived hidden information as underlying metadata to a second converter; The second converter is used to generate a configuration file associated with the selected log template by using the underlying metadata.

2. The configuration file generation component according to claim 1, characterized in that: The types of log files include access logs, business logs and application logs, and each type of log file corresponds to a set of log templates; the access log is used to record the log of application container access, the business log is used to record business-related logs, and the application log is other logs other than the access log and the business log.

3. The configuration file generation component according to claim 1, characterized in that: The log processing system is a multi-tenant architecture, and the log metadata defines different rules for different tenants.

4. The configuration file generation component according to claim 1, characterized in that: There are multiple configuration files, and each configuration file corresponds to a component for processing log files on the log transmission link.

5. The configuration file generation component according to claim 2, characterized in that: The log template includes a general template, which is used to provide a plurality of predefined regular expression subsets for the user to select a target subset and then splice it into a target regular expression.

6. A log processing system, characterized in that: include: Several components for processing log files, used to process log files differently as different nodes on the log transmission link; as well as The configuration file generation component according to any one of claims 1 to 5; each configuration file generated by the configuration file generation component corresponds to a component for processing log files.

7. The log processing system according to claim 6, characterized in that: The components for processing log files include a log collection component, a log cache component, a log processing and use component, and a log collection configuration component.

8. The log processing system according to claim 6, characterized in that: Also includes: The log data lineage analysis component is used to define the lineage relationship of the underlying metadata based on the underlying metadata, the components that process the log files, and the monitoring data reported by the components that process the log files, and to visualize the health status of the log files in the log transmission link based on the defined lineage relationship.

9. The log processing system according to claim 7, characterized in that: Also includes: The quality control component is used to monitor the log data in the log file in real time based on the health status of the log file displayed by the log data lineage analysis component, and perform exception handling when an exception is monitored.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the function of any component in any one of claims 1 or 9 is realized.

Images