Log-based index data processing system and method

Abstract

The invention relates to the technical field of data processing, in particular to a log-based index data processing system and method.An index data specification module formulates a standard format of index data and indicates the index data output by a service system to be stored in a log file form; the configuration item module provides dynamic configuration information for the acquisition, aggregation and storage module in the form of a configuration file; the index data acquisition module reads specified log files in parallel and extracts index data details in a specified time range into a memory; the index data aggregation module is used for grouping and aggregating the collected index data details according to a preset rule to generate aggregated index data; the index data storage module stores the aggregated index data; and the index data visualization module is used for visually displaying the aggregated index data. According to the invention, the index data is recorded in a log form, decoupling of the index data and a service system is realized, and an automatic and reliable index data processing system is provided.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a log-based indicator data processing system and method. Background Art

[0002] Cloud computing covers a wide range of service types, including Infrastructure as a Service (IaaS), Platform as a Service (PaaS) and Software as a Service (SaaS). Under these service types, many different business systems have been further subdivided and developed to meet diverse business needs.

[0003] These business systems do not operate in isolation, but are interrelated and interdependent, forming a complex whole. For example, when a user runs an e-commerce application on a cloud platform, the application may involve the collaborative work of multiple business systems. Order processing may involve the order center system and the commodity system, the payment link involves the payment gateway system, and user data management involves the user management system. Each business system generates a large amount of performance indicator and traffic indicator data in daily operation. These indicator data are extremely valuable to developers, operation and maintenance personnel, and data analysts. Performance indicators such as CPU usage, memory usage, disk I / O, network traffic, etc. can intuitively reflect the occupation of system resources and the operating status of the system; while traffic indicators such as interface access volume, interface time consumption, transaction volume, user volume, etc. can reveal the business activities of the system and user behavior patterns.

[0004] Due to the complexity of business systems in the cloud computing field and the importance of indicator data, it is unrealistic to manually collect and analyze these indicator data. Therefore, an automated and reliable indicator data processing system is needed. Summary of the invention

[0005] In view of this, the present invention provides a log-based indicator data processing system and method to solve the problem that it is unrealistic to manually collect and analyze these indicator data due to the complexity of business systems in the cloud computing field and the importance of indicator data.

[0006] In a first aspect, the present invention provides a log-based indicator data processing system, the system comprising: an indicator data specification module, a configuration item module, an indicator data collection module, an indicator data aggregation module, an indicator data storage module and an indicator data visualization module;

[0007] The indicator data specification module is used to formulate a standard format for indicator data and instruct the indicator data output by the business system to be stored in the form of a log file;

[0008] The configuration item module is used to provide dynamic configuration information for the indicator data collection module, the indicator data aggregation module and the indicator data storage module in the form of a configuration file;

[0009] The indicator data collection module is used to read the specified log files in parallel and extract the indicator data details within the specified time range into the memory;

[0010] The indicator data aggregation module is used to group and aggregate the collected indicator data details according to preset rules to generate aggregated indicator data;

[0011] The indicator data storage module is used to store the aggregated indicator data in the target time series database;

[0012] The indicator data visualization module is used to visualize the aggregated indicator data through a target visualization tool.

[0013] In an optional embodiment, the indicator data specification module is also used to indicate that the format of a single-line log record is a json data format, and to store the indicator data output by the business system in a key-value pair in a key-value format; the storage field definition of the indicator data includes a field name, a data type, and a field meaning.

[0014] In an optional implementation, the configuration items in the configuration item module include a metric_log_path configuration item, a counter_metrics configuration item, an empty_strs configuration item, a remote_write_addr configuration item, and a read_interval configuration item;

[0015] The metric_log_path configuration item is used to specify the log file path corresponding to the metric data to be collected;

[0016] The counter_metrics configuration item is used to specify the metric name of the counter type;

[0017] The empty_strs configuration item is used to specify an invalid string set;

[0018] The remote_write_addr configuration item is used to specify the remotewrite interface address of the target time series database;

[0019] The read_interval configuration item is used to specify the timing scheduling frequency of the indicator collection module.

[0020] In an optional implementation, the indicator data collection module includes a main thread and coroutine collaboration unit, a timing scheduling unit, a data collection and filtering unit, and a data sending and recording unit;

[0021] The main thread and coroutine collaboration unit is used to read multiple log files in parallel based on the mode of collaboration between the main thread and multiple coroutines;

[0022] The timing scheduling unit is used to control the data collection frequency of each coroutine through the read_interval configuration item in the configuration item module to trigger the indicator data collection operation;

[0023] The data collection and filtering unit is used to instruct each indicator data collection operation to obtain the indicator data details within the target time range, and filter the collected indicator data based on the target key field;

[0024] The data sending and recording unit is used to record the current collection position and file inode number after each indicator data collection operation is completed, and send the collected indicator data details to the indicator data aggregation module.

[0025] In an optional implementation, the indicator data aggregation module includes a data grouping and aggregation unit and an indicator type differentiation unit;

[0026] The data grouping and aggregation unit is used to group the indicator data details according to preset rules when receiving the indicator data details sent by the indicator data collection module, and aggregate the indicator data details under the same group to generate aggregated indicator data;

[0027] The indicator type distinguishing unit is used to, before performing aggregation processing on the indicator data details, divide the indicator data details into a counter type and an average type based on the counter_metrics configuration item in the configuration item module, so as to perform aggregation operations respectively.

[0028] In an optional embodiment, the indicator data storage module is also used to convert the aggregate indicator data into a time series data format through a target standard protocol interface of the target time series database, and push the aggregate indicator data in the time series data format to the target time series database for storage.

[0029] In a second aspect, the present invention provides a log-based indicator data processing method, the method is applied to a log-based indicator data processing system as described above, the method comprising:

[0030] Start the main thread in the indicator data collection module and obtain the log file path from the configuration item module;

[0031] According to the number of log file paths, a corresponding number of coroutines are created through the main thread; each coroutine is used to process a log file; the log file is obtained by storing the indicator data output by the business system in the form of a log file by the indicator data specification module;

[0032] According to the data collection frequency configured in the configuration item module, the indicator data collection operation corresponding to each coroutine is executed through the indicator data collection module, and the indicator data details are obtained;

[0033] The indicator data aggregation module groups and aggregates the indicator data details according to preset rules to generate aggregated indicator data;

[0034] The aggregated indicator data is stored in a target time series database through the indicator data storage module;

[0035] The aggregated indicator data is visualized through the indicator data visualization module.

[0036] In a third aspect, the present invention provides a computer device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to thereby execute the above-mentioned first aspect or any corresponding embodiment thereof.

[0037] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to cause a computer to execute a computer device according to the first aspect or any corresponding embodiment thereof.

[0038] In a fifth aspect, the present invention provides a computer program product, including computer instructions, where the computer instructions are used to enable a computer to execute a computer device according to the first aspect or any corresponding embodiment thereof.

[0039] The present invention records the indicator data in the form of a log file, thereby decoupling the indicator data from the business system, so that the collection and storage of the indicator data no longer depends on the business system itself, but is independently performed through the log file, which can reduce the performance burden of the business system because there is no need to process the push logic of the indicator data within the business system. At the same time, through the log file, the indicator data can be viewed and analyzed in more detail, which is convenient for problem troubleshooting and performance monitoring. In addition, the present invention provides a reliable and universal log format specification, which helps to standardize the recording method of indicator data. The unified format makes the indicator data easier to parse and process, improves the readability and operability of the data, and also facilitates data exchange between different business systems and tools.

[0040] Secondly, the indicator data collection module of the present invention adopts the mode of main thread plus N coroutines to collect indicator data, which can realize the simultaneous parallel collection of multiple log files. This parallel processing method significantly improves the efficiency of data collection, especially when processing a large number of log files, which can reduce the total time required for collection and improve the overall processing speed.

[0041] Furthermore, the indicator data collection module of the present invention triggers the indicator data collection operation at a fixed frequency through a timing scheduling algorithm, and can adjust the collection frequency according to actual needs, balance the server load and the real-time requirements of data, so that the business system can adjust the collection strategy according to different business scenarios and performance requirements, and optimize resource use. In the indicator aggregation process of the indicator data aggregation module, the present invention also automatically verifies invalid strings based on the empty_strs configuration item. If an invalid string is found, the server machine name of the current collection indicator log is automatically obtained and backfilled. This automatic verification and backfilling mechanism ensures the accuracy and integrity of the indicator data, and avoids inaccurate data or false alarms of monitoring alarms caused by invalid attribute values. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 is a structural schematic diagram of a log-based indicator data processing system according to an embodiment of the present invention;

[0044] Figure 2 is a flow chart of a log-based indicator data processing method according to an embodiment of the present invention;

[0045] Figure 3 is a flowchart of another log-based indicator data processing method according to an embodiment of the present invention;

[0046] Figure 4 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0048] It should be noted that the various terms in the present invention are first explained as follows:

[0049] Json (JavaScript Object Notation): is a lightweight data exchange format. Json stores and transmits data objects in text form. It has the characteristics of clear structure, easy reading and writing, and is often used for data exchange between network applications.

[0050] Timestamp: A number that represents a specific time, used to represent the number of seconds or nanoseconds that have passed since a specific point in time. In Linux systems, the accuracy of timestamps can reach nanoseconds.

[0051] Inode: A characteristic description used to identify a file in the operating system and is the unique identifier of the file.

[0052] TimeSeries: A data sequence arranged in chronological order, used to represent the time evolution data of a dynamic scale or process.

[0053] Prometheus: An open source monitoring solution for collecting and aggregating metrics as time series data.

[0054] Grafana: An open source data visualization tool that can perform data monitoring and data statistics and has an alarm function.

[0055] In the field of cloud computing, it is usually necessary for a series of business systems to work together to complete specific business operations. For example, when a user opens a cloud host, from the creation of the user order to the completion of the cloud host opening, the entire order will be circulated in multiple business systems such as the portal system, order center, commodity system, underlying resource management system, monitoring center, etc., and finally the resource opening of the cloud host will be completed. In the process of receiving a large number of requests and processing a large number of businesses, each business system will generate a large amount of indicator data, such as: the received request URL, the number of requests, the request time, the business type, the business processing time, the business processing status, etc. Based on the premise of log indicator data, the present invention can perform real-time indicator data collection, indicator data aggregation, and indicator data storage on the indicator data of each business system, and finally realize the real-time visualization display of massive indicator data through visualization tools. At the same time, corresponding data charts can be configured on the visualization tool according to the collected indicator data, such as request success rate, average request time, order completion rate, average order time, etc. Developers, operation and maintenance personnel, and managers can use these real-time indicator data charts to accurately monitor the operation of each system, the response status of each service interface, and detect system failure points, allowing staff to quickly locate and solve problems, greatly reducing the time spent on locating problems.

[0056] According to an embodiment of the present invention, a log-based indicator data processing system embodiment is provided. Figure 1 is a structural diagram of a log-based indicator data processing system 10 according to an embodiment of the present invention. Figure 1 As shown, the system 10 includes: an indicator data specification module 110, a configuration item module 120, an indicator data collection module 130, an indicator data aggregation module 140, an indicator data storage module 150 and an indicator data visualization module 160;

[0057] The indicator data specification module 110 is used to formulate a standard format for indicator data and instruct the indicator data output by the business system to be stored in the form of a log file;

[0058] The configuration item module 120 is used to provide dynamic configuration information for the indicator data collection module 130, the indicator data aggregation module 140 and the indicator data storage module 150 in the form of a configuration file; the dynamic configuration information includes a log file path, an indicator type, a collection frequency, etc.;

[0059] The indicator data collection module 130 is used to read the specified log files in parallel and extract the indicator data details within the specified time range into the memory;

[0060] The indicator data aggregation module 140 is used to group and aggregate the collected indicator data details according to preset rules to generate aggregated indicator data;

[0061] The indicator data storage module 150 is used to store the aggregated indicator data in the target time series database;

[0062] The indicator data visualization module 160 is used to visualize the aggregated indicator data through a target visualization tool.

[0063] Further, the indicator data specification module 110 of the indicator data processing system of this embodiment is used to formulate the standard format of the indicator data. The business system needs to output the indicator data in strict accordance with the indicator data specification and store it in the log file. The configuration item module 120 is used to configure the dynamic configuration information on which the three modules of the indicator data collection module 130, the indicator data aggregation module 140 and the indicator data storage module 150 rely in the form of a configuration file. After the indicator data collection module 130 is used to specify one or more log files, the indicator data collection module 130 reads the specified log files in parallel and extracts the indicator data details within the specified time range (such as 1 minute) into the memory. The indicator data aggregation module 140 is used to group the indicator details cached in the memory by the indicator data collection module 130 according to the full amount fields defined by the indicator specification, and then aggregate the indicator data under each group, and generate the indicator data of the specified time range (1 minute) according to the indicator type, such as the sum of indicators for 1 minute, the average value of indicators for 1 minute, etc. The indicator data storage module 150 is used to use the target time series database (such as open source software prometheus) to store indicator data. After the indicator aggregation module aggregates and generates indicator data every minute, it pushes the indicator data to the time series database of the open source software prometheus for storage through the remote write method provided by the target time series database (such as open source software prometheus). The indicator data visualization module 160 is used to complete the visualization display of indicator data by installing the target visualization tool (such as grafana tool), configuring the prometheus data source, and writing the prometheus expression.

[0064] In an optional embodiment, the indicator data specification module 110 is also used to indicate that the format of a single-line log record is a json data format, and to store the indicator data output by the business system in a key-value pair format; the storage field definition of the indicator data in the log file (that is, the storage form of the indicator data in the log file) includes the field name, data type and field meaning.

[0065] Further, the indicator data specification module 110 of the present embodiment stipulates that the indicator data is stored in the form of a log file and adopts a standard JSON data format. This method is different from the method of directly processing or pushing indicator data in the business system, and realizes the decoupling of indicator data from the business system. The business system only needs to output the indicator data to the log file in the agreed format, and the subsequent data collection, processing and other links can be operated independently of the business system, reducing the coupling between the two. The indicator data specification module 110 of the present embodiment requires that the indicator data be stored in the standard JSON format, making the data more standardized. The key-value pair structure in JSON format is clear and easy to understand and parse. Whether it is a developer viewing a log file or a subsequent data processing program reading data, it can more conveniently identify and process the specific information of each indicator, improve the readability of the data, and facilitate data management and analysis. Indicator data field definition, the indicator log must contain the following fields in Table 1, and strictly abide by the constraints and data types of each field.

[0066] Table 1

[0067]

[0068] In an optional implementation, the configuration items in the configuration item module 120 include a metric_log_path configuration item, a counter_metrics configuration item, an empty_strs configuration item, a remote_write_addr configuration item, and a read_interval configuration item;

[0069] The metric_log_path configuration item is used to specify the log file path corresponding to the metric data to be collected;

[0070] The counter_metrics configuration item is used to specify the metric name of the counter type;

[0071] The empty_strs configuration item is used to specify an invalid string set;

[0072] The remote_write_addr configuration item is used to specify the remote write interface address of the target time series database;

[0073] The read_interval configuration item is used to specify the timing scheduling frequency of the indicator collection module.

[0074] Further, the configuration item module 120 in this embodiment introduces a variety of configuration items, which realizes accurate control of key parameters during the index data collection process, greatly improves the flexibility and controllability of the collection process, and realizes accurate control strategies of log file paths, index types, and collection frequencies through dynamic configuration information. Previous technologies may lack such detailed configuration capabilities in these aspects, resulting in the collection process being not flexible enough and difficult to adapt to the needs of different scenarios. However, this embodiment can be configured in a targeted manner according to actual conditions to optimize the collection process and resource utilization. The setting of the empty_strs configuration item in the configuration item module 120 is used to process invalid characters in the index data. The empty_strs configuration item can first verify whether there are characters in the specified invalid string set in the data. If so, it can be replaced with valid data as needed, which helps to improve the quality and accuracy of the data, avoid data processing errors or misunderstandings caused by invalid characters, and ensure that subsequent data aggregation, storage, and visualization operations are based on reliable data.

[0075] metric_log_path configuration item (metric log file path): The metric_log_path configuration item exists in an array format and can specify multiple file paths at a time. This enables the metric data collection module 130 to clearly know from which log files the metric data is collected, which is suitable for situations where there may be multiple log files in a complex system that record metric data of different types or sources. For example, in a cloud computing system, there may be log files for computing nodes, log files for storage nodes, etc. Through this configuration item, all relevant log file paths can be specified at once, which facilitates unified data collection.

[0076] counter_metrics configuration item (counter type indicator name): used to specifically specify which indicator data belongs to the counter type. In subsequent data aggregation operations, specific aggregation methods can be used for counter type indicators according to the counter_metrics configuration item, such as calculating the total number of times the indicator appears within a certain period of time. This helps to perform targeted statistical analysis on different types of indicators and better understand the frequency of occurrence of certain events or operations in the system.

[0077] empty_strs configuration item (invalid string set): You can pre-define an invalid string set. When collecting indicator data, check whether each indicator data contains these invalid characters according to this configuration. If found, you can replace them with valid data according to the predetermined rules.

[0078] The remote_write_addr configuration item (Prometheus remote write interface address) specifies the target address where the indicator data is pushed after aggregation, that is, the remote write interface of the Prometheus time series database. This configuration ensures that the aggregated indicator data can be accurately sent to the specified storage location, so that the data can be smoothly stored in the Prometheus time series database, providing a data basis for subsequent query, analysis and visualization.

[0079] Read_interval configuration item (indicator data collection module timing scheduling frequency): By setting the read_interval configuration item, the working rhythm of the indicator data collection module 130 can be accurately controlled. The collection frequency can be flexibly adjusted according to server performance and business requirements to optimize the overall system performance.

[0080] In an optional implementation, the indicator data collection module 130 includes a main thread and coroutine collaboration unit, a timing scheduling unit, a data collection and filtering unit, and a data sending and recording unit;

[0081] The main thread and coroutine collaboration unit is used to read multiple log files in parallel based on the mode of collaboration between the main thread and multiple coroutines;

[0082] The timing scheduling unit is used to control the data collection frequency of each coroutine through the read_interval configuration item in the configuration item module 120 to trigger the indicator data collection operation;

[0083] The data collection and filtering unit is used to instruct each indicator data collection operation to obtain the indicator data details within the target time range (the target time range can be 1 minute), and filter the collected indicator data based on the target key fields (i.e., the three fields of metric_name indicator name, metric_value indicator value, and metric_timestamp indicator data timestamp);

[0084] The data sending and recording unit is used to record the current collection position and file inode number after each indicator data collection operation is completed, and send the collected indicator data details to the indicator data aggregation module 140.

[0085] Furthermore, the main thread and the coroutine collaboration unit of this embodiment adopts the mode of main thread + N coroutines (main thread and multiple coroutines collaborate) to collect index data, realizes the parallel collection of multiple log files, and improves the efficiency of data collection; the data collection and filtering unit indicates that in a single collection process, complete index data within 1 minute is collected, and the aggregated index data per minute is generated under the aggregation operation of the index data aggregation module 140; after each data collection is completed, the data is sent and the recording unit records the current collection position (offset) and the file inode number, and stores it in the form of a file. The collection method in the related art only continuously collects data line by line. Compared with the related art, the function of collecting 1 minute of data at a time and controlling the collection frequency through the read_interval configuration item in this embodiment is innovative.

[0086] The main thread and coroutine collaboration unit of this embodiment has a main thread, and N coroutines are combined to complete the index data collection function, where the number of coroutines N depends on the number of log file paths configured by the log file path (metric_log_path) in the configuration item module 120. For example, if metric_log_path configures 5 log file paths and all file paths are valid, then N=5. When the main thread is running, it traverses each file path in the metric_log_path configuration item, verifies whether the file path is valid, removes invalid paths, and retains only valid paths. When the configured 5 log files are all valid, the main thread will create 5 coroutines and bind a log file path to each coroutine, and this coroutine is responsible for the index data collection work of the log file bound to it.

[0087] The timing scheduling unit enables each coroutine to trigger an indicator data collection operation at a fixed frequency under the control of the timing scheduling algorithm. The execution frequency is determined by the timing scheduling frequency (read_interval configuration item) in the configuration item module 120, in seconds, and the default is 1 second. If read_interval=2 is configured, each coroutine will collect the bound indicator log file once every 2 seconds.

[0088] The data collection and filtering unit also enables the coroutine to obtain the current inode number (file unique identifier) ​​of the log file bound to it, the previous inode number of the file, and the previous read position offset of the file before performing the collection work. By comparing the current inode number of the file and the previous inode number, it is determined whether the file has been replaced. When the current and subsequent inode numbers are inconsistent, it is determined that the log file has been replaced with a new file, and the previous read position offset of the file is reset to 0, and the collection starts from the starting position of the file, and the current inode number of the file is updated to the corresponding file. When the current and subsequent inode numbers are consistent, the file reading position is offset by the previous read position offset of the file, and reading starts from the offset position.

[0089] The data distribution and recording unit enables the coroutine to collect the detailed indicator data within a time range of 1 minute in a single collection work. The coroutine reads the bound log file line by line, extracts the text information of the changed line and converts it into jsonObject (i.e. JSON object). By judging whether the jsonObject contains the three fields of metric_name indicator name, metric_value indicator value, and metric_timestamp indicator data timestamp, it identifies whether the line data is indicator data. In this way, non-indicator logs can be filtered out and only indicator log data can be collected. When reading the first indicator data, the timestamp information (metric_timestamp) of the indicator data is parsed, and then it is formatted into a minute-level timestamp and recorded in the indicator start time metricStartTime of this collection work. Next, the coroutine will read each indicator data line by line and judge whether the difference between its timestamp information metric_timestamp and metricStartTime exceeds 60. If it does not exceed 60, the current indicator data will be cached in the indicator record set (metricDetailList). If it exceeds 60, it means that the current indicator data and the first indicator data do not belong to the same minute. At this time, the coroutine stops reading the file, updates the reading position offset of the current file and records it in the corresponding file, and sends the set metricDetailList of all indicator detail records read this time to the indicator data aggregation module 140. At this point, the coroutine completes a single indicator data collection task and waits for the next scheduling of the timing scheduling algorithm.

[0090] The data sending and recording unit is also used to record the inode number and file reading position of the indicator log file. Each indicator log file will have a file with a suffix of .inode and .offset corresponding to it. Both files are named after the absolute path of the file is converted to avoid data errors caused by duplicate log file names. If the path of the indicator log file is: / data / log / test_project.log, then during the operation of the indicator data collection module 130, two files data.log.test_project.inode (used to record the inode number) and data.log.test_project.offset (used to record the file reading position) will be generated accordingly.

[0091] In an optional implementation, the indicator data aggregation module 140 includes a data grouping and aggregation unit and an indicator type distinction unit;

[0092] The data grouping and aggregation unit is used to group the indicator data details according to preset rules when receiving the indicator data details sent by the indicator data collection module 130, and aggregate the indicator data details under the same group to generate aggregated indicator data;

[0093] The indicator type distinguishing unit is used to, before performing aggregation processing on the indicator data details, divide the indicator data details into a counter type and an average type based on the counter_metrics configuration item in the configuration item module 120 to perform aggregation operations respectively.

[0094] Furthermore, the indicator data aggregation module 140 receives the complete indicator data within 1 minute collected by the indicator data collection module 130, and groups and aggregates the indicator data details to generate aggregate indicator data per minute. At the same time, the indicator data is divided into two categories according to the counter_metrics configuration item: counter type and average type. For counter type indicators, count is used as the value of the aggregate indicator, indicating the total number of times the indicator appears within 1 minute; for average type indicators, total / count is used as the value of the aggregate indicator, indicating the average value of the indicator within 1 minute.

[0095] After receiving the indicator details set sent by the indicator data collection module 130, the indicator data aggregation module 140 traverses the indicator details for grouping statistics and aggregates the indicator data values ​​under the same group. The rule for the indicator details grouping is: initialize an array propertyList of indicator attribute names, add the additional attribute names in the indicator details metric_name (indicator name), metric_idc (resource pool), metric_job (indicator classification label), metric_alert_tag (server identifier), and metric_labels (indicator additional field) to propertyList, then sort the elements of propertyList in positive order, traverse the sorted propertyList, obtain the attribute value corresponding to the indicator and concatenate them in order to obtain the string valueStr, and finally convert the string valueStr into a 32-bit character key value using the MD5 algorithm, and use the key value as the basis for the indicator details grouping. Through this algorithm, it can be ensured that the same key value will be obtained only when the indicator attributes are completely consistent.

[0096] While grouping the indicators, perform aggregation operations based on the type of indicator data: determine whether the current indicator is a counter type indicator based on the indicator name. The counter type indicator is determined by the counter_metrics configuration item. If the indicator name exists in the counter_metrics configuration item, it is a counter type indicator. Otherwise, it is determined to be an average type indicator.

[0097] For counter type indicators, the counter count under the corresponding group of the indicator is used for cumulative counting, and the indicator value (metric_value) is migrated to metric_lables, which can be used for grouping and displaying based on metric_value based on prometheus expression in the visualization module; for average value type indicators, the counter count under the corresponding group of the indicator is used for cumulative counting, and the indicator value (metric_value) is accumulated to the total attribute under the group.

[0098] In the indicator data aggregation module 140, after the indicator details are grouped, each group initializes an aggregation indicator, and fills the information such as metric_name (indicator name), metric_idc (resource pool), metric_job (indicator classification label), metric_alert_tag (server identifier) ​​used for grouping into the aggregation indicator, and sets the metric_timestamp of the aggregation indicator to the metricStartTime when the batch of indicators is collected, and the aggregation data of each group can be obtained. Because the indicator detail data is the indicator data collected within 1 minute, the aggregation data represents the indicator aggregation information of 1 minute: for counter type indicators, count is used as the value of the aggregation indicator, indicating the total number of times the indicator appears within 1 minute; for average type indicators, use as the value of the aggregation indicator, indicating the average value of the indicator within 1 minute.

[0099] In addition, when backfilling the aggregated indicator information, the indicator data aggregation module 140 determines whether metric_alert_tag is an invalid string according to the empty_strs configuration item. Metric_alert_tag represents the server identifier, indicating which server the indicator data comes from, so this attribute must be valid information. When the metric_alert_tag attribute is an invalid string, it indicates that the indicator information is not output or the information cannot be obtained. At this time, this embodiment will obtain the machine name of the server currently collecting the indicator log and write it back to the metric_alert_tag attribute of the aggregated indicator.

[0100] In an optional embodiment, the indicator data storage module 150 is also used to convert the aggregate indicator data into a time series data format through the target standard protocol interface of the target time series database, and push the aggregate indicator data in the time series data format to the target time series database for storage.

[0101] Furthermore, the indicator data storage module 150 uses the remote write interface of prometheus to push the aggregated indicator data directly to the time series database of prometheus. In the push gateway push method in the related art, the pushed indicator can only be the current timestamp and cannot be changed. Compared with the related art, this embodiment uses the remote write method to customize the timestamp and realize the push of indicator data at any time.

[0102] After the indicator aggregation module completes the grouping and aggregation of indicator details within 1 minute, it generates a set of aggregated indicator data. It traverses the entire set and converts each indicator data in the set into prometheus time series data (TimeSeries). By specifying the remote write interface remote_write_addr of prometheus in the configuration item, the time series data (TimeSeries) is pushed to the prometheus time series database for storage.

[0103] Furthermore, the indicator data visualization module 160 is used to realize the visualization display of indicator data based on the open source software grafana. After installing the grafana software, in the data source configuration item in grafana, select the prometheus type data source, and configure the prometheus address for storing indicator data in the indicator aggregation module. Then create a dashboard in grafana, write a specific prometheus expression, and the corresponding indicator data can be displayed in a chart.

[0104] In summary, the present invention achieves the decoupling of indicator data from the business system by recording indicator data in the form of log files, so that the collection and storage of indicator data no longer depends on the business system itself, but is carried out independently through log files, which can reduce the performance burden of the business system because there is no need to process the push logic of indicator data within the business system. At the same time, through log files, indicator data can be viewed and analyzed in more detail, which is convenient for problem troubleshooting and performance monitoring. In addition, the present invention provides a reliable and universal log format specification, which helps to standardize the recording method of indicator data. The unified format makes indicator data easier to parse and process, improves the readability and operability of the data, and also facilitates data exchange between different business systems and tools.

[0105] Secondly, the indicator data collection module of the present invention adopts the mode of main thread plus N coroutines to collect indicator data, which can realize the simultaneous parallel collection of multiple log files. This parallel processing method significantly improves the efficiency of data collection, especially when processing a large number of log files, which can reduce the total time required for collection and improve the overall processing speed.

[0106] Furthermore, the indicator data collection module of the present invention triggers the indicator data collection operation at a fixed frequency through a timing scheduling algorithm, and can adjust the collection frequency according to actual needs, balance the server load and the real-time requirements of data, so that the business system can adjust the collection strategy according to different business scenarios and performance requirements, and optimize resource use. In the indicator aggregation process of the indicator data aggregation module, the present invention also automatically verifies invalid strings based on the empty_strs configuration item. If an invalid string is found, the server machine name of the current collection indicator log is automatically obtained and backfilled. This automatic verification and backfilling mechanism ensures the accuracy and integrity of the indicator data, and avoids inaccurate data or false alarms of monitoring alarms caused by invalid attribute values.

[0107] According to an embodiment of the present invention, a log-based indicator data processing method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0108] In this embodiment, a log-based indicator data processing method is provided, which is applied to Figure 1 In the log-based indicator data processing system shown, Figure 2 is a flow chart of a log-based indicator data processing method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0109] Step S201, start the main thread in the indicator data collection module, and obtain the log file path from the configuration item module.

[0110] Furthermore, this embodiment first stores the indicator data in the form of a log file through the indicator data specification module and adopts the standard JSON format to achieve decoupling from the business system, making the indicator data more standardized and readable. This is different from the traditional way of processing indicator data directly in the business system. After decoupling, it is convenient to independently manage and process indicator data, and the JSON format facilitates the exchange and parsing of data. This embodiment requires that the single-line log is stored in a standard JSON format key-value pair through the indicator data specification module, and clearly stipulates the fields that the indicator log must contain (such as metric_name indicator name, metric_type indicator type, metric_idc resource pool, etc.) and whether it is required and the remarks information. For example, metric_name, as a globally unique indicator name, helps to accurately identify different indicators; metric_type determines the indicator type (Gauge or Histogram), which is convenient for subsequent targeted processing.

[0111] This embodiment also implements precise control of the indicator log file path (metric_log_path can specify multiple files), indicator type (counter_metrics determines the counter type indicator), and collection frequency (read_interval controls timing scheduling) through various configuration items in the configuration item module, and can also handle invalid characters (empty_strs). Compared with related technologies, the collection process is more flexible and controllable. For example, the collection frequency can be adjusted according to actual needs to avoid excessive server load while ensuring data accuracy. This embodiment specifies the remotewrite interface address of Prometheus through the remote_write_addr configuration item to prepare for the subsequent push of aggregated data and ensure that the data can be accurately stored in the specified location.

[0112] For further information, see Figure 3 It is a flowchart of another log-based indicator data processing method according to an embodiment of the present invention. In this embodiment, when the system is initialized, the main thread in the indicator data collection module is first started. The main thread is responsible for overall coordination and management. After starting, it enters the ready state and waits for the operation of obtaining the log file path to be executed. The main thread reads the metric_log_path configuration item from the configuration item module. The configuration item stores the indicator log file path information to be collected in an array format.

[0113] Step S202, according to the number of the log file paths, a corresponding number of coroutines are created through the main thread; each of the coroutines is used to process a log file; the log file is obtained by storing the indicator data output by the business system in the form of a log file by the indicator data specification module.

[0114] Further, the main thread of this embodiment traverses the obtained log file path array and counts the number of valid paths. During the traversal process, the main thread will perform validity checks on each path, such as checking whether the file exists, whether it has read permissions, etc. If the path is invalid (such as the file does not exist or the permissions are insufficient), the error message is recorded and the path is skipped, and the coroutine creation count is not participated in. In this embodiment, the main thread creates N coroutines according to the number of valid log file paths N obtained by statistics. For example, if the number of valid paths is 3, 3 coroutines are created. When creating a coroutine, the main thread assigns a unique identifier (such as a coroutine ID) to each coroutine, and binds a valid log file path to the corresponding coroutine. After binding, each coroutine will be specifically responsible for processing the index data collection work in the log file bound thereto. These log files are stored after the business system outputs the index data in the format defined by the index data specification module, ensuring the consistency and standardization of the data format, which is convenient for the coroutine to perform subsequent processing.

[0115] Step S203: According to the data collection frequency configured in the configuration item module, the indicator data collection operation corresponding to each coroutine is executed through the indicator data collection module, and the indicator data details are obtained.

[0116] Furthermore, this embodiment adopts the main thread + N coroutine mode through the indicator data collection module, where N depends on the number of valid log file paths configured. After the main thread verifies the path, it creates a coroutine for each valid path to collect data in parallel, thereby improving the collection efficiency. Before starting the collection operation, each coroutine of this embodiment reads the read_interval configuration item from the configuration item module to obtain the currently set data collection frequency value. The coroutine of this embodiment periodically triggers the indicator data collection operation according to the acquired collection frequency under the control of the timing scheduling algorithm. When the collection operation is triggered, the coroutine starts to read the log file bound to it. During the reading process, the coroutine reads the file content line by line, extracts each line of text information, and attempts to convert it into jsonObject format. Whether the line of data is indicator data is determined by judging whether jsonObject contains the three key fields of metric_name, metric_value, and metric_timestamp. If it is indicator data, it is further parsed and processed; if not, the line is skipped and the next line is read. In this embodiment, for the rows identified as indicator data, when the coroutine reads the first indicator data, it parses its metric_timestamp field, formats the timestamp information to the minute level, and records it as the indicator start time metricStartTime of this collection work. Then, the coroutine continues to read subsequent indicator data by row, and for each indicator data, it determines whether the difference between its metric_timestamp and metricStartTime exceeds the specified time range (such as 1 minute). If it does not exceed 1 minute, the current indicator data is cached in the indicator record set (metricDetailList); if it exceeds 1 minute, it indicates that the current indicator data and the first indicator data do not belong to the same minute data, and the coroutine stops reading the file. After the coroutine stops reading the file, the reading position offset of the current file is updated and recorded in the corresponding file (through the .offset suffix file record corresponding to the log file), and at the same time, the set metricDetailList of all indicator detail records read this time is sent to the indicator data aggregation module. At this point, the coroutine completes a single indicator data collection work, waits for the next scheduling of the timing scheduling algorithm, and prepares for the next round of data collection.

[0117] Step S204: The indicator data aggregation module groups and aggregates the indicator data details according to preset rules to generate aggregated indicator data.

[0118] Further, the present embodiment continuously monitors the indicator data detail set (metricDetailList) from the indicator data collection module through the indicator data aggregation module. When a complete indicator data detail set within 1 minute is received, the aggregation module starts grouping and aggregation operations. When the grouping rule is executed, the present embodiment initializes an array propertyList of indicator attribute names, and adds the additional attribute names in the indicator details, metric_name, metric_idc, metric_job, metric_alert_tag, and metric_labels to the propertyList. Then, the elements in the propertyList are sorted in positive order, and then the sorted propertyList is traversed to obtain the attribute values ​​corresponding to the indicator and spliced ​​in order to obtain valueStr. Finally, the MD5 algorithm is used to convert valueStr into a 32-bit character key, and the key is used as the basis for the indicator detail grouping. Through this algorithm, it is ensured that only when the indicator attributes are completely consistent, the same key value will be obtained, so that the indicator data with the same attribute characteristics are grouped into the same group. When the aggregation operation is executed, the present embodiment performs aggregation operations according to the type of indicator data while grouping the indicators. First, according to the indicator name, it is determined whether the current indicator is a counter type indicator. The counter type indicator is determined by the counter_metrics configuration item. If the indicator name exists in the counter_metrics configuration item, it is a counter type indicator, otherwise it is determined to be an average type indicator. For counter type indicators, the counter count under its corresponding group in this embodiment performs cumulative counting (each time an indicator of this type appears, the count value is increased by 1), and the value of the indicator (metric_value) is migrated to metric_lables, so as to be grouped and displayed based on metric_value in the visualization module. For average value type indicators, the counter count under its corresponding group is also cumulatively counted, and the value of the indicator (metric_value) is accumulated to the total attribute under the group. For the initialization and information backfilling of aggregate indicators, the indicator data details of this embodiment initialize an aggregate indicator for each group after completing the grouping and aggregation operations. The metric_name, metric_idc, metric_job, metric_alert_tag and other information used for grouping are backfilled to the aggregate indicator, and the metric_timestamp of the aggregate indicator is set to the metricStartTime when the batch indicator is collected, so that the aggregate data represents the indicator aggregation information of 1 minute.For invalid string processing (server identification verification), this embodiment determines whether metric_alert_tag is an invalid string according to the empty_strs configuration item. Metric_alert_tag represents the server identification. If this attribute is an invalid string, it indicates that the indicator information may be missing or wrong in terms of server identification. At this time, the aggregation module will obtain the machine name of the server that is currently collecting the indicator log, and write it back to the metric_alert_tag attribute of the aggregated indicator to ensure the accuracy and validity of the server identification, so as to accurately track the source of the indicator data in the future. For the generation of aggregated indicator data, after the above series of operations, the aggregated indicator data is finally generated. For counter type indicators, count is used as the value of the aggregated indicator, indicating the total number of times the indicator appears within 1 minute; for average value type indicators, total / count is used as the value of the aggregated indicator, indicating the average value of the indicator within 1 minute. After these aggregated indicator data are prepared, they will be passed to the indicator data storage module for storage.

[0119] Step S205: storing the aggregated indicator data in a target time series database through the indicator data storage module.

[0120] Furthermore, after receiving the aggregated indicator data, the indicator data storage module first traverses the entire aggregated indicator data set. For each aggregated indicator data in the set, it is converted into the time series data (TimeSeries) format of Prometheus. This conversion process involves mapping and encapsulating each field in the aggregated indicator data according to the specification of TimeSeries to ensure that the data can be correctly stored in the time series database of Prometheus. For pushing data to the database, this embodiment uses the remote write interface remote_write_addr of Prometheus specified in the configuration item module, and the indicator data storage module uses the interface to push the converted time series data (TimeSeries) to the time series database of Prometheus for storage. During the push process, the storage module will establish a connection with the database, send data according to the interface specification of the database, and handle possible connection errors, data transmission errors, etc., to ensure that the data can be accurately and completely stored in the database, providing a reliable data foundation for subsequent query, analysis and visualization.

[0121] Step S206: Visually display the aggregated indicator data through the indicator data visualization module.

[0122] Further, in terms of Grafana configuration and data source connection, this embodiment first ensures that the Grafana software is installed and running. In Grafana, enter the data source configuration item, select the Prometheus type data source, and configure the Prometheus address for storing indicator data in step S205. This step establishes a connection between Grafana and the Prometheus time series database, so that Grafana can obtain the aggregated indicator data stored in the database. During the configuration process, it is necessary to enter the correct Prometheus server address, port number and other information, and perform a connection test to ensure a successful connection. After successfully configuring the data source, this embodiment creates a dashboard in Grafana. The dashboard is a container for displaying indicator data charts. Then, according to specific display requirements, write Prometheus expressions. These expressions are used to query specific aggregated indicator data from the Prometheus time series database and define how to display these data in the form of charts on the dashboard. For example, you can write an expression to calculate the average, maximum, minimum, etc. of a specific indicator within a certain time period, and display it as different types of charts such as line charts, bar charts, pie charts, etc., so that users can intuitively observe the changing trends, distribution and other information of the indicator data, thereby better understanding the operating status of the system and discovering potential problems or optimization directions.

[0123] In summary, the present invention achieves the decoupling of indicator data from the business system by recording indicator data in the form of log files, so that the collection and storage of indicator data no longer depends on the business system itself, but is carried out independently through log files, which can reduce the performance burden of the business system because there is no need to process the push logic of indicator data within the business system. At the same time, through log files, indicator data can be viewed and analyzed in more detail, which is convenient for problem troubleshooting and performance monitoring. In addition, the present invention provides a reliable and universal log format specification, which helps to standardize the recording method of indicator data. The unified format makes indicator data easier to parse and process, improves the readability and operability of the data, and also facilitates data exchange between different business systems and tools.

[0124] Secondly, the indicator data collection module of the present invention adopts the mode of main thread plus N coroutines to collect indicator data, which can realize the simultaneous parallel collection of multiple log files. This parallel processing method significantly improves the efficiency of data collection, especially when processing a large number of log files, which can reduce the total time required for collection and improve the overall processing speed.

[0125] Furthermore, the indicator data collection module of the present invention triggers the indicator data collection operation at a fixed frequency through a timing scheduling algorithm, and can adjust the collection frequency according to actual needs, balance the server load and the real-time requirements of data, so that the business system can adjust the collection strategy according to different business scenarios and performance requirements, and optimize resource use. In the indicator aggregation process of the indicator data aggregation module, the present invention also automatically verifies invalid strings based on the empty_strs configuration item. If an invalid string is found, the server machine name of the current collection indicator log is automatically obtained and backfilled. This automatic verification and backfilling mechanism ensures the accuracy and integrity of the indicator data, and avoids inaccurate data or false alarms of monitoring alarms caused by invalid attribute values.

[0126] The embodiment of the present invention also provides a computer device, see Figure 4 , Figure 4 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 4 As shown, the computer device includes: one or more processors 410, memory 420, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 4 A processor 410 is taken as an example.

[0127] The processor 410 may be a central processing unit, a network processor or a combination thereof. The processor 410 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable logic gate array, a general purpose array logic or any combination thereof.

[0128] The memory 420 stores instructions executable by at least one processor 410, so that at least one processor 410 executes the method shown in the above embodiment.

[0129] The memory 420 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 420 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 420 may optionally include a memory remotely arranged relative to the processor 410, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0130] The memory 420 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 420 may also include a combination of the above types of memory.

[0131] The computer device further includes a communication interface 430 for the computer device to communicate with other devices or a communication network.

[0132] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0133] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0134] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the defined scope.

Claims

1. A log-based indicator data processing system, characterized in that: The system includes: an indicator data specification module, a configuration item module, an indicator data collection module, an indicator data aggregation module, an indicator data storage module and an indicator data visualization module; The indicator data specification module is used to formulate a standard format for indicator data and instruct the indicator data output by the business system to be stored in the form of a log file; The configuration item module is used to provide dynamic configuration information for the indicator data collection module, the indicator data aggregation module and the indicator data storage module in the form of configuration files; The indicator data collection module is used to read the specified log files in parallel and extract the indicator data details within the specified time range into the memory; The indicator data aggregation module is used to group and aggregate the collected indicator data details according to preset rules to generate aggregated indicator data; The indicator data storage module is used to store the aggregated indicator data in the target time series database; The indicator data visualization module is used to visualize the aggregated indicator data through a target visualization tool.

2. The system according to claim 1, characterized in that The indicator data specification module is also used to indicate that the format of a single-line log record is the json data format, and to store the indicator data output by the business system in key-value pairs in key-value format; the storage field definition of the indicator data includes the field name, data type and field meaning.

3. The system according to claim 1, characterized in that The configuration items in the configuration item module include a metric_log_path configuration item, a counter_metrics configuration item, an empty_strs configuration item, a remote_write_addr configuration item, and a read_interval configuration item; The metric_log_path configuration item is used to specify the log file path corresponding to the metric data to be collected; The counter_metrics configuration item is used to specify the metric name of the counter type; The empty_strs configuration item is used to specify an invalid string set; The remote_write_addr configuration item is used to specify the remote write interface address of the target time series database; The read_interval configuration item is used to specify the timing scheduling frequency of the indicator collection module.

4. The system according to claim 3, characterized in that The indicator data collection module includes a main thread and coroutine collaboration unit, a timing scheduling unit, a data collection and filtering unit, and a data sending and recording unit; The main thread and coroutine collaboration unit is used to read multiple log files in parallel based on the mode of collaboration between the main thread and multiple coroutines; The timing scheduling unit is used to control the data collection frequency of each coroutine through the read_interval configuration item in the configuration item module to trigger the indicator data collection operation; The data collection and filtering unit is used to instruct each indicator data collection operation to obtain the indicator data details within the target time range, and filter the collected indicator data based on the target key field; The data sending and recording unit is used to record the current collection position and file inode number after each indicator data collection operation is completed, and send the collected indicator data details to the indicator data aggregation module.

5. The system according to claim 3, characterized in that The indicator data aggregation module includes a data grouping and aggregation unit and an indicator type differentiation unit; The data grouping and aggregation unit is used to group the indicator data details according to preset rules when receiving the indicator data details sent by the indicator data collection module, and aggregate the indicator data details under the same group to generate aggregated indicator data; The indicator type distinguishing unit is used to, before performing aggregation processing on the indicator data details, divide the indicator data details into a counter type and an average type based on the counter_metrics configuration item in the configuration item module, so as to perform aggregation operations respectively.

6. The system according to any one of claims 1 to 5, characterized in that: The indicator data storage module is also used to convert the aggregated indicator data into a time series data format through a target standard protocol interface of the target time series database, and push the aggregated indicator data in the time series data format to the target time series database for storage.

7. A log-based indicator data processing method, characterized in that: The method is applied to a log-based indicator data processing system according to any one of claims 1 to 6, and the method comprises: Start the main thread in the indicator data collection module and obtain the log file path from the configuration item module; According to the number of log file paths, a corresponding number of coroutines are created through the main thread; each coroutine is used to process a log file; the log file is obtained by storing the indicator data output by the business system in the form of a log file by the indicator data specification module; According to the data collection frequency configured in the configuration item module, the indicator data collection operation corresponding to each coroutine is executed through the indicator data collection module, and the indicator data details are obtained; The indicator data aggregation module groups and aggregates the indicator data details according to preset rules to generate aggregated indicator data; The aggregated indicator data is stored in a target time series database through the indicator data storage module; The aggregated indicator data is visualized through the indicator data visualization module.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the log-based indicator data processing method described in claim 7 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the log-based indicator data processing method described in claim 7.

10. A computer program product, characterized in that It includes computer instructions, which are used to enable a computer to execute the log-based indicator data processing method described in claim 7.

Images