Log-based index collection method, apparatus and device, and storage medium

Abstract

The invention relates to a log-based index collection method and device, equipment and a storage medium, and the method comprises the steps: monitoring a log directory of each node in real time, and reading a latest log file in the log directory; a log line of the latest log file is read, key monitoring fields in the log line are extracted based on a predefined field mapping relation table, and the key monitoring fields comprise request path fields; performing normalization processing on the request path field to obtain a normalized request path field; time sequence monitoring indexes are generated based on the key monitoring fields and the normalized request path fields, and the time sequence monitoring indexes at least comprise request counting indexes with the normalized request path fields as labels and response time statistical indexes; compared with the prior art, the technical scheme of the invention can realize real-time acquisition of the multi-node time sequence monitoring indexes, and improves the efficiency and accuracy of statistical analysis.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Web service operation and maintenance, and in particular to a log-based index collection method and device, equipment and a storage medium. BACKGROUND

[0002] Internet Information Services (IIS) is a mainstream Web server on the Windows platform and is widely used in the deployment and operation of enterprise internal systems and Internet applications. In actual operation and maintenance scenarios, IIS generates access logs containing rich information in real time during operation. These logs cover key data such as client IP, access URL (request path), request response time, HTTP status code, and are an important data source for operation and maintenance and performance analysis.

[0003] Existing IIS log analysis methods mainly include manual analysis, offline batch processing, and centralized log platforms. However, manual analysis requires operation and maintenance personnel to manually open log files to view data, which is not only inefficient but also cannot achieve real-time monitoring of system status. The offline batch processing scheme can process logs in batches, but has high data delay and cannot reflect the current system running status in a timely manner. Even a powerful centralized log platform is too cumbersome and wastes resources for scenarios that only need to monitor basic indicators such as queries per second and response time due to its complex deployment process and high server resource consumption.

[0004] Moreover, dynamic URLs in raw log data will cause the labels of indicators to be scattered, and the same type of request will be split into multiple independent indicators, which seriously affects the accuracy of statistical analysis and cannot provide reliable support for operation and decision-making. SUMMARY

[0005] The present application provides a log-based index collection method, device, equipment and storage medium, which can realize real-time collection of time series monitoring indicators of multiple nodes and improve the efficiency and accuracy of statistical analysis.

[0006] In a first aspect, the present application provides a log-based index collection method, comprising: monitoring the log directory of each node in real time, and reading the latest log file in the log directory; reading the log line of the latest log file, and extracting the key monitoring field in the log line based on a pre-defined field mapping relationship table, wherein the key monitoring field includes a request path field; performing normalization processing on the request path field to obtain a normalized request path field; and generating a time series based on the key monitoring field and the normalized request path field

[0007] The time series monitoring index includes at least a request count index and a response time statistical index labeled by the normalized request path field.

[0008] In a possible implementation, the log-based index collection method provided in the present application further includes: periodically pulling the time series monitoring index corresponding to all nodes to obtain an aggregated time series monitoring index; and performing multi-dimensional display and trend analysis on the aggregated time series monitoring index.

[0009] In a possible implementation, the periodically pulling the time series monitoring index corresponding to all nodes to obtain an aggregated time series monitoring index, and performing multi-dimensional display and trend analysis on the aggregated time series monitoring index specifically includes: accessing an HTTP interface exposed by each node according to a preset pulling period, and pulling the time series monitoring index corresponding to the node based on the HTTP interface; performing aggregation processing on the time series monitoring index of all nodes based on a preset aggregation rule to obtain an aggregated time series monitoring index, where the aggregation rule includes taking the normalized request path field and a node ID as label dimensions, and aggregating the request count index and the time statistical index under different label dimensions to obtain an aggregated time series monitoring index under different label dimensions, and the aggregated time series monitoring index includes an aggregated request count index and an aggregated time statistical index; obtaining a target label dimension selected by a user, obtaining a target aggregated time series monitoring index under the target label dimension, and performing visual processing on the target aggregated time series monitoring index; obtaining a historical target aggregated time series monitoring index corresponding to the target aggregated time series monitoring index, and generating a line chart based on the target aggregated time series monitoring index and the historical target aggregated time series monitoring index, where the line chart at least displays a change trend over time of a target aggregated request count index and a target aggregated time statistical index in the target aggregated time series monitoring index.

[0010] In a possible implementation, the reading of the log lines of the latest log file, the extraction of the key monitoring fields in the log lines based on a predefined field mapping relationship table specifically comprises: reading the log lines of the latest log file in real time, and detecting whether a current log line is empty or contains an annotation identifier; if yes, the current log line is skipped, and the next log line is read; if the current log line is detected as a valid data line, all valid data lines are integrated to obtain a valid data line set; a predefined field mapping relationship table is obtained, wherein the field mapping relationship table comprises key monitoring field names-key monitoring field indexes, and the field mapping relationship table is constructed based on the key monitoring field names and the corresponding key monitoring field indexes extracted from the field name sequence by recognizing an annotation line containing a preset identifier and parsing the field name sequence separated by spaces when reading the latest log file; the target field names corresponding to each valid data line in the valid data line set are split by a space delimiter, and the target field names are matched with the key monitoring field names in the field mapping relationship table respectively; if the matching is successful, the target key monitoring field indexes corresponding to the key monitoring field names of the matching success are obtained, the target key monitoring field values corresponding to the target key monitoring field indexes are extracted, and the key monitoring fields corresponding to each valid data line are obtained.

[0011] In a possible implementation, the time series monitoring index is generated based on the key monitoring fields and the normalized request path fields, wherein the time series monitoring index at least comprises a request count index and a response time statistical index labeled by the normalized request path fields, and specifically comprises: a monitoring label set is constructed based on the normalized request path fields and at least one server attribute label and at least one client attribute label selected from the key monitoring fields; a request count index for counting the frequency of request occurrence is generated based on the monitoring label set, and the request count index is obtained by counting and accumulating different dimension combinations of the monitoring label set; a response time statistical index for counting the request processing performance is generated based on the monitoring label set, and the response time statistical index is obtained by processing performance time series data obtained from the key monitoring fields based on the monitoring label set and calculating a statistical result based on a preset statistical algorithm; the request count index and the response time statistical index are associated and integrated based on a unified time reference to obtain time series monitoring data.

[0012] In a possible implementation, the normalization processing on the request path field to obtain a normalized request path field specifically includes: obtaining the request path field extracted from the key monitoring field, detecting whether the request path field contains a query string, if yes, removing the query string in the request path field to obtain a preprocessed request path field; replacing a numeric parameter in the preprocessed request path field with a first preset placeholder, and replacing an identification type parameter in the preprocessed request path field with a second preset placeholder to obtain the normalized request path field.

[0013] In a possible implementation, the log directory of each node is monitored in real time, and the latest log file in the log directory is read, specifically including: deploying a directory listening service on each node to detect the file change state under the log directory corresponding to each node in real time, where the file change state includes new log file creation; when detecting that the file change state of the log directory is new log file creation, triggering a latest log file identification process; reading the metadata information of all log files under the log directory, sorting all log files according to a preset sorting rule based on the metadata information, and selecting a log file with the highest priority in the sorting result as the latest log file.

[0014] In a second aspect, the present application provides a log-based index collection device, comprising:

[0015] a log file reading module, a key monitoring field extraction module, a normalization processing module and a time series monitoring index generation module; the log file reading module is configured to monitor the log directory of each node in real time and read the latest log file in the log directory; the key monitoring field extraction module is configured to read the log line of the latest log file, and extract the key monitoring field in the log line based on a pre-defined field mapping relationship table, where the key monitoring field includes a request path field; the normalization processing module is configured to normalize the request path field to obtain a normalized request path field; and the time series monitoring index generation module is configured to generate a time series monitoring index based on the key monitoring field and the normalized request path field, where the time series monitoring index at least includes a request count index and a response time statistical index with the normalized request path field as a label.

[0016] In a third aspect, the present application further provides a computer device including a memory and a processor, the memory has a computer program stored thereon, and the processor implements the above method when executing the computer program.

[0017] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium, which stores a computer program, and the computer program can implement the above method when executed by a processor.

[0018] The embodiments of the present application provide a log-based index collection method and device, equipment and storage medium, which have the following advantages compared with the prior art.

[0019] By monitoring the log directory of each node in real time, the latest log file in the log directory is read; the log line of the latest log file is read, and the key monitoring field in the log line is extracted based on a predefined field mapping relationship table, wherein the key monitoring field includes a request path field; the request path field is normalized to obtain a normalized request path field; and a time series monitoring index is generated based on the key monitoring field and the normalized request path field, wherein the time series monitoring index at least includes a request count index and a response time statistical index with the normalized request path field as a label; compared with the prior art, the technical solution of the present application solves the problems of low manual analysis efficiency, high offline batch processing delay and heavy centralized log platform by monitoring the log directory in real time, extracting the key monitoring field in the latest log file, normalizing the request path and generating the corresponding time series monitoring index, reduces resource consumption by light-weight deployment, avoids index dispersion caused by dynamic URL by request path normalization, guarantees the accuracy of statistical analysis, directly generates time series indexes without additional format conversion, improves visualization efficiency, and finally provides real-time, comprehensive and accurate monitoring data support for operation and maintenance personnel, helping to quickly grasp the system running state and make reliable decisions. BRIEF DESCRIPTION OF DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and serve to explain the principles of the present application together with the specification.

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, those skilled in the art can obtain other drawings from these drawings without creative effort.

[0022] One or more embodiments are exemplarily illustrated by pictures in the drawings corresponding thereto, and these exemplary illustrations do not constitute a limitation on the embodiments, and elements with the same reference numerals in the drawings represent similar elements, unless otherwise specified, and the drawings do not constitute a proportional limitation.

[0023] Figure 1is a flow diagram of an embodiment of a log-based index collection method provided by the present application;

[0024] Figure 2 is a structural diagram of an embodiment of a log-based index collection device provided by the present application;

[0025] Figure 3 is a structural diagram of a computer device provided by the present application. DETAILED DESCRIPTION

[0026] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0027] The following disclosure provides many different embodiments, or examples, for implementing different structures of the present application. For the purpose of simplicity, the elements and arrangements of the various examples are described in the following detailed description. These are, of course, merely examples and are not intended to limit the application. Additionally, the present application can be implemented in a wide variety of environments and contexts. Further, the description is not intended to limit the scope of the present application. For example, the present application can be implemented in any device, system, or method that is configured to perform the recited functions.

[0028] It should be understood that the terms "comprises" and "comprising" when used in this specification and the appended claims specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0029] It should also be understood that the terms used in the specification and the appended claims are intended to describe particular embodiments and do not intent to limit the present application. As used in the specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0030] It should be further understood that the term "and / or" as used in the specification and the appended claims means any one or more of the associated listed items, as well as all possible combinations of the items.

[0031] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrases "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0032] Example 1, see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of a log-based metric collection method provided in this application, as shown below. Figure 1 As shown, the method includes steps 101-104, as detailed below:

[0033] Step 101: Monitor the log directory of each node in real time and read the latest log file in the log directory.

[0034] In one embodiment, a directory monitoring service is deployed on each node to detect file changes in the log directory corresponding to each node in real time, wherein the file changes include the creation of new log files.

[0035] Specifically, when deploying the directory monitoring service on each node, the path of the log directory to be monitored for each node is first obtained, and the corresponding directory monitoring trigger rules are configured for each node. The log directory path and the directory monitoring trigger rules are used as log directory monitoring configuration parameters. Based on the log directory monitoring configuration parameters, the directory monitoring service is deployed on each node to detect the file change status under the target log directory in real time.

[0036] Example Description: In metric collection scenarios based on IIS logs, a lightweight directory listening service needs to be deployed on each IIS server (i.e., node). This directory listening service is used to detect the dynamic changes of the target log directory in real time. First, the log directory path of each node needs to be clearly defined, usually the default IIS log storage path or a custom path defined by the enterprise, to ensure that the listening service can accurately cover the storage location of all logs to be collected. After the directory listening service starts, it will detect the file change status in the log directory in real time in an event-driven mode, and pay special attention to the key event of new log file creation. This is because the IIS server will automatically rotate the logs according to preset rules and generate new log files, ensuring that no new data sources generated by log rotation are missed.

[0037] In an embodiment, when the file change state of the log directory is detected as a new log file creation, a latest log file identification process is triggered; metadata information of all log files under the log directory is read, all log files are sorted based on the metadata information according to a preset sorting rule, and a log file with the highest priority in the sorting result is selected as the latest log file.

[0038] Specifically, when a new log file under the target log directory is detected, a latest log file identification process is triggered; wherein the new log file is generated by the IIS server according to a preset rotation rule.

[0039] Specifically, after triggering the latest log file identification process, the directory monitoring service will first pause reading the current log file, and switch to a log file screening mode, and in the log file screening mode, read the metadata information of all log files under the target log directory, wherein the metadata information includes but is not limited to file creation time, file modification time, and file name identifier; all logs under the target log directory are sorted according to a preset sorting rule, such as descending order of file creation time, and descending order of file name sequence number, and the log file with the highest priority in the sorting result is selected as the latest log file.

[0040] Specifically, after selecting the log file with the highest priority in the sorting result, the readability of the log file is also verified, including detecting whether the log file is in a locked state, whether it is a valid IIS log format file, and if the verification is passed, the log file is determined as the latest log file to be read; if the verification fails, the next priority log file in the sorting result is selected for re-verification until a valid latest log file is confirmed.

[0041] Step 102: Read the log line of the latest log file, and extract the key monitoring field in the log line based on a predefined field mapping relationship table, wherein the key monitoring field includes a request path field.

[0042] In an embodiment, the log line of the latest log file is read in real time, and it is detected whether the current log line is empty or contains a comment identifier, if so, the current log line is skipped and the next log line is read, if the current log line is detected as a valid data line, all valid data lines are integrated to obtain a valid data line set.

[0043] Specifically, after determining the latest log file, the log line of the latest log file is processed in a streaming real-time reading mode, that is, starting from the current end position of the latest log file, the newly added log line is read in real time to avoid repeated reading of historical data.

[0044] Specifically, when reading the log lines of the latest log file, validity detection is also performed on each read log line, wherein the validity detection includes empty line filtering, comment line identification and valid line integration.

[0045] Specifically, for empty line filtering: if it is detected that the current log line is pure blank, i.e., without any character or containing only spaces and line breaks, the current log line is directly skipped to avoid invalid data occupying processing resources; for comment line identification: it is judged whether the current log line contains a comment identifier, i.e., starting with #, if yes, the current log line is determined to be a comment line, and the current log line is also directly skipped; for valid line integration: if the current log line is neither an empty line nor a comment line, it is determined that the current log line is a valid data line, which is temporarily stored in a memory cache queue, and all valid data lines in the memory cache queue are integrated to form a valid data line set.

[0046] In an embodiment, a predefined field mapping relationship table is obtained, wherein the field mapping relationship table includes key monitoring field name-key monitoring field index, and the field mapping relationship table is constructed based on the key monitoring field name and the corresponding key monitoring field index extracted from a field name sequence separated by spaces in an annotation line containing a preset identifier when reading the latest log file, and the key monitoring field name-key monitoring field index relationship table is constructed based on the extracted key monitoring field name and the corresponding key monitoring field index.

[0047] Specifically, when reading the log lines of the latest log file, a first annotation line containing a first preset identifier is read, wherein the first preset identifier is #Fields; after reading the first annotation line, the first annotation line is split by a space separator to remove the first preset identifier to obtain an ordered field name sequence; and the preset key monitoring fields are selected from the field name sequence, and the key monitoring field index of each key monitoring field name in each key monitoring field in the field name sequence is recorded, to construct a mapping relationship table of key monitoring field name-key monitoring field index with the key monitoring field name as the key and the corresponding key monitoring field index as the value.

[0048] Specifically, the preset key monitoring fields include but are not limited to server IP address field s-ip, request path field cs-uri-stem, response time field time-taken and client IP address field c-ip.

[0049] Preferably, the first annotation line containing the first preset identifier is in the standard format of the IIS log, and this line clearly defines the field order and name of all subsequent valid data lines.

[0050] In one embodiment, the target field name corresponding to each valid data row in the valid data row set is split by a space separator. The target field name is then matched with the key monitoring field name in the field mapping table. If a match is successful, the target key monitoring field index corresponding to the successfully matched key monitoring field name is obtained, and the target key monitoring field value corresponding to the target key monitoring field index is extracted to obtain the key monitoring field corresponding to each valid data row.

[0051] Specifically, for each data row in the set of valid data rows, the content of the row is split by a space delimiter to obtain a target field value sequence that completely corresponds to the field order of the first comment row containing the first preset identifier. The target field value sequence includes a field name sequence and a key monitoring field value sequence.

[0052] Specifically, the key monitoring field names in the field mapping table are traversed. If a key monitoring field name that matches the target field name exists, the value at the corresponding position of the target key monitoring field index is extracted from the key monitoring field value sequence of the target field value sequence according to the target key monitoring field index corresponding to the key monitoring field name in the mapping table, and used as the target key monitoring field value.

[0053] Specifically, the extraction results of all key monitoring fields in a valid data row, namely the key monitoring field values ​​corresponding to the server IP address field s-ip, request path field cs-uri-stem, response time field time-taken, and client IP address field c-ip, are associated and integrated to form a set of key monitoring fields corresponding to the valid data row; after traversing all valid data rows, the full set of key monitoring field data is obtained.

[0054] Step 103: Normalize the request path field to obtain a normalized request path field.

[0055] In one embodiment, a request path field extracted from the key monitoring field is obtained, and it is detected whether the request path field contains a query string. If so, the query string in the request path field is removed to obtain a preprocessed request path field.

[0056] Specifically, since the original format of the request path field may contain two parts: the path body and the query string, the query string can cause the same business path to be judged as different paths due to different parameters, interfering with the accuracy of metric statistics. Therefore, it is necessary to remove the query string from the request path field.

[0057] Specifically, when detecting whether the request path field contains a query string, the character sequence of the request path field is traversed to determine whether the character sequence contains the start identifier of the query string. If so, all characters before the start identifier are extracted as the path body, and the path body is used as the preprocessed request path field. If not, the request path field is used as the preprocessed request path field.

[0058] In one embodiment, the numeric parameters in the preprocessed request path field are replaced with a first preset placeholder, and the identifier parameters in the preprocessed request path field are replaced with a second preset placeholder to obtain a normalized request path field.

[0059] Specifically, since the preprocessed request path field may still contain dynamic parameter fragments, that is, the same business path may have different path formats due to different numbers and unique identifiers, such as / api / user / 123 and / api / user / 456, which are essentially user detail query paths; therefore, it is necessary to unify these dynamic fragments into preset placeholders through parameter replacement.

[0060] Specifically, the preprocessed request path field is split into multiple path segments according to a preset delimiter. Each path segment is then checked for pure numbers. If a path segment is detected as a pure number segment, it is determined to be a data parameter and replaced with the first placeholder.

[0061] Preferably, the preset separator is / ; the first placeholder is :id.

[0062] Specifically, a globally unique identifier is detected for each path segment. If the path segment conforms to the globally unique identifier format, the path segment is determined to be an identifier parameter and is replaced with a second placeholder.

[0063] Preferably, the preset separator is / ; the first placeholder is :id; and the second placeholder is :guid.

[0064] In one embodiment, the request path field is normalized to obtain a normalized request path field, which can unify the identification of similar business paths, reduce the label dimension, avoid the dispersion of indicators caused by dynamic parameters, and provide a unified dimension for subsequent business path statistics, response time and other indicators.

[0065] Step 104: Generate time-series monitoring metrics based on the key monitoring fields and the normalized request path fields, wherein the time-series monitoring metrics include at least a request count metric and a response time statistics metric labeled with the normalized request path fields.

[0066] In one embodiment, a monitoring tag set is constructed based on the normalized request path field and at least one server-side attribute tag and at least one client-side attribute tag selected from the key monitoring fields.

[0067] Specifically, the normalized request path field is used as the core tag. This tag is the core identifier of the business dimension. For example, / api / user / :id represents the user details query business, and / api / order / :guid represents the order query business. The unified normalized format avoids the dispersion of indicators caused by dynamic parameters.

[0068] Specifically, server-side attribute tags are extracted from key monitoring fields. The server-side attribute tag is the server IP address s-ip. This tag is used to distinguish different IIS nodes. For example, s-ip = 192.168.1.10 corresponds to node A in the cluster, and s-ip = 192.168.1.11 corresponds to node B, ensuring that subsequent statistical indicators can be calculated at the node level.

[0069] Specifically, client attribute tags are extracted from key monitoring fields. The client attribute tag is the client IP address (c-ip). This tag is used to distinguish the source of the request. For example, c-ip=10.0.0.5 corresponds to an internal office network request, and c-ip=203.0.113.8 corresponds to an external Internet request. This supports subsequent analysis of request characteristics based on the client source.

[0070] Specifically, the normalized request path field, server-side attribute tags, and client-side attribute tags are combined into a monitoring tag set, in the format of normalized request path-server-side attribute tags-client-side attribute tags; each monitoring tag set uniquely corresponds to a type of request scenario, laying the foundation for subsequent dimensional statistics of metrics.

[0071] In one embodiment, a request count indicator is generated based on the monitoring tag set to count the frequency of requests. The request count indicator is obtained by accumulating counts based on different dimensional combinations of the monitoring tag set.

[0072] Specifically, the core objective of the request count metric is to count the total number of requests under different tag dimensions, which is used to reflect key frequency indicators such as requests per second (QPS) and the total number of requests for a certain node.

[0073] Specifically, for request counting scenarios, an initial Counter type metric is established, and the constructed monitoring tag set is bound to the Counter type metric. That is, each monitoring tag set corresponds to an independent Counter. For each valid data row processed, representing an HTTP request, the system will match the monitoring tag set corresponding to the request and trigger the counter metric to accumulate. Each increment of 1 represents a new request in this scenario, resulting in a request count metric. The accumulated result of the request count metric is stored in real time with the timestamp, forming time-series data of timestamp-tag set-count value. Subsequently, the count increment within a certain time period can be queried through PromQL to calculate the total number of requests or requests per second (QPS) within a preset time period.

[0074] In one embodiment, a response time statistical indicator for statistically analyzing request processing performance is generated based on the monitoring tag set. The response time statistical indicator is calculated by processing performance time-series data obtained from the key monitoring fields according to the monitoring tag set and calculating statistical results based on a preset statistical algorithm.

[0075] Specifically, obtain the time-taken field value from the key monitoring fields, which is the response time of a single request. In IIS logs, this is usually in milliseconds. The time-taken field value is the performance time series data. For example, a time-taken of 200 for a certain request means that the response time is 200 milliseconds.

[0076] Specifically, for response time statistics scenarios, the initialization process involves configuring a preset statistical algorithm to calculate quantiles and averages, while binding the monitoring tag set to the Summary type metric to ensure statistics are performed by dimension.

[0077] Specifically, for each time-taken field value obtained, the system converts it into a uniform time unit and stores it in the Summary type metric of the corresponding monitoring tag set; the Summary type metric calculates all response time values ​​under the same monitoring tag set in real time to obtain the current statistical results and obtain the response time statistical metric.

[0078] Specifically, the statistical results of response time metrics will be updated with timestamps, forming time-series data of timestamp-label set-P50 / P95 / average. Grafana can then be used to display the percentile trend of response time and quickly locate performance bottlenecks.

[0079] In one embodiment, the request count metric and the response time metric are correlated and integrated based on a unified time reference to obtain time series monitoring data.

[0080] Specifically, the request time recorded in the log is used as a unified time benchmark. This time usually comes from the date and time fields in the IIS log. The system will convert it into a timestamp to ensure that the time dimension of all metrics is consistent.

[0081] Specifically, the request count metric and the response time metric are associated using a dual dimension of timestamp and monitoring tag set. That is, the count values ​​and response time statistics under the same timestamp and the same monitoring tag set will be grouped into a complete set of time-series data; the associated complete time-series data will be stored in the form of a time series.

[0082] In one embodiment, the process also involves periodically fetching the time-series monitoring metrics corresponding to all nodes to obtain aggregated time-series monitoring metrics; performing multi-dimensional display and trend analysis on the aggregated time-series monitoring metrics; this process involves accessing the HTTP interfaces exposed by each node according to a preset fetching cycle, and fetching the time-series monitoring metrics corresponding to each node based on the HTTP interfaces; and aggregating the set of time-series monitoring metrics for all nodes based on preset aggregation rules to obtain aggregated time-series monitoring metrics, wherein the aggregation rules include summarizing the request count metrics and time statistics metrics under different tag dimensions using the normalized request path field and node ID as tag dimensions to obtain different tags. The system implements aggregated time-series monitoring metrics under the target tag dimension, including aggregated request count metrics and aggregated time statistics metrics. It acquires the target tag dimension selected by the user, obtains the target aggregated time-series monitoring metrics under the target tag dimension, and performs visualization processing on the target aggregated time-series monitoring metrics. It also acquires the historical target aggregated time-series monitoring metrics corresponding to the target aggregated time-series monitoring metrics, and generates a line chart based on the target aggregated time-series monitoring metrics and the historical target aggregated time-series monitoring metrics. The line chart at least shows the time-varying trends of the target aggregated time-series monitoring metrics corresponding to the target aggregated request count metrics and the target aggregated time statistics metrics.

[0083] Specifically, based on the real-time monitoring requirements, a preset retrieval period is set in the indicator collection platform. The preset retrieval period is usually configured to be 10 seconds to 1 minute to ensure that the indicator delay is controllable. At the same time, the network addresses of all IIS nodes are entered, and each node exposes the / metri cs interface through the local HTTP port defined in the document. This interface will return the current time series monitoring indicators of the node in real time.

[0084] Example Description: The metrics collection platform initiates HTTP GET requests to the metrics interface of all nodes at preset intervals, such as sending requests in batches to node A (192.168.1.10:9184 / metrics) and node B (192.168.1.11:9184 / metrics) every 15 seconds to avoid congestion from single points of request. Each node returns time-series monitoring metrics in text format through the metrics interface. After receiving the data, the metrics collection platform temporarily stores the data according to the node dimension, forming a correspondence between nodes and time-series monitoring metrics, providing raw data for subsequent aggregation processing.

[0085] Specifically, the default aggregation dimension is the normalized request path field + node ID, ensuring that the aggregation result can reflect the overall situation of a certain business in the entire cluster, as well as distinguish the performance of a certain business on a specific node; the aggregation objects are two types of core indicators: request count indicators (Counter type) and response time statistics indicators (Summary type).

[0086] Specifically, for the request count metrics of all retrieved nodes, group them by the tag dimension of the normalized request path field uri + node IDs_ip, and sum the corresponding values ​​of the request count metrics within the same group. For example, if node A (s_ip = 192.168.1.10) has a request count of 101 for uri = " / api / user / :id", and node B (s_ip = 192.168.1.11) has a count of 89 for the same normalized request path field, then the "summary request count metric" for this tag dimension is 101 + 89 = 190. If it is necessary to summarize the same business across the entire cluster, then sum the requests by the normalized request path field to obtain the total number of requests for this business across the entire cluster.

[0087] Specifically, for response time statistics, the system groups the data by the tag dimension of the normalized request path field uri + node IDs_ip, and recalculates the response time statistics within the same group. For example, if the response time statistics for node A with the same uri are [0.1, 0.2, 0.3], and the response time statistics for node B with the same uri are [0.15, 0.25], the two groups of response time statistics are merged into [0.1, 0.2, 0.3, 0.15, 0.25] during aggregation, and the statistical values ​​such as P50, P95, and average value are recalculated to obtain the summary response time statistics for that tag dimension.

[0088] Specifically, the tag dimensions, aggregated request counts, aggregated response time statistics, and pull timestamps are integrated into aggregated time series monitoring metrics, which are stored in chronological order to form an aggregated metric time series library, supporting subsequent queries by dimension.

[0089] Specifically, the Grafana visualization platform provides a label dimension filter, allowing users to select target label dimensions from preset aggregation dimensions. For example, selecting "uri= / api / user / :id" (to view the entire cluster data for this service), or "uri= / api / user / :id+s_ip=192.168.1.10" (to view the data for this service on node A), or "s_ip=192.168.1.10" (to view all service data on node A). Based on the target dimension selected by the user, the visualization platform initiates a query request to the aggregated metric time series library. For example, if the user selects "uri= / api / user / :id", the system will query the aggregated request count and aggregated response time statistics for all nodes under that uri, returning the metric values ​​for the corresponding timestamps.

[0090] Specifically, the platform also visualizes the aggregated metrics retrieved; it generates bar charts for aggregated request counts to display requests over different time periods; it generates line charts for aggregated response times to display quantile changes; and it supports displaying multiple metrics on the same screen, such as overlaying request count and response time curves in the same chart, allowing users to intuitively see the relationship between request volume and performance, such as whether response time increases during peak request volumes.

[0091] Specifically, users can select a historical data time range on the visualization platform, such as the last 1 day, the last 7 days, or the last 30 days. Based on this range, the platform extracts the historical aggregated indicators corresponding to the target label dimension from the aggregated indicator time series library, such as the summary request count and summary response time P95 for the same time period each day within the last 7 days. The platform then aligns the current target aggregated indicator with the historical target aggregated indicator by time granularity. For example, if the granularity of the current target aggregated indicator is 15 seconds / point, the historical target aggregated indicator is also converted to 15-second points by time granularity to ensure time dimension consistency. Subsequently, the values ​​of the two types of indicators are associated in timestamp order to form structured data of time-current count-historical count-current response time-historical response time.

[0092] Specifically, the platform generates line charts based on the correlated structured data. The horizontal axis represents time, and the vertical axis is divided into left and right axes: the left axis represents request count, and the right axis represents response time. Different colored lines are used to display the current aggregated request count, the historical aggregated request count for the same period, the current aggregated response time (P95), and the historical aggregated response time (P95). By comparing the lines, users can quickly identify trends: for example, if the current request count line is higher than the historical line for the same period, it indicates an increase in business access volume; if the current response time line is higher than the historical line for the same period, it indicates that performance may have degraded, and further investigation is needed to determine the cause.

[0093] Example 2, see Figure 2 , Figure 2This is a schematic diagram of an embodiment of a log-based indicator collection device provided in this application. Corresponding to the above-described log-based indicator collection method, this application also provides a log-based indicator collection device. This log-based indicator collection device includes modules for executing the above-described log-based indicator collection method, and can be configured in a desktop computer, tablet computer, laptop computer, or other terminal. Specifically, the log-based indicator collection device includes a log file reading module 201, a key monitoring field extraction module 202, a normalization processing module 203, and a time-series monitoring indicator generation module 204.

[0094] The log file reading module 201 is used to monitor the log directory of each node in real time and read the latest log file in the log directory.

[0095] The key monitoring field extraction module 202 is used to read the log lines of the latest log file and extract the key monitoring fields in the log lines based on a predefined field mapping table, wherein the key monitoring fields include the request path field.

[0096] The normalization processing module 203 is used to normalize the request path field to obtain a normalized request path field.

[0097] The time series monitoring indicator generation module 204 is used to generate time series monitoring indicators based on the key monitoring field and the normalized request path field, wherein the time series monitoring indicators include at least a request count indicator and a response time statistics indicator labeled with the normalized request path field.

[0098] In one embodiment, the log-based indicator collection device provided in this application further includes an indicator analysis module.

[0099] In one embodiment, the indicator analysis module is used to periodically pull the time series monitoring indicators corresponding to all nodes to obtain aggregated time series monitoring indicators, and to perform multi-dimensional display and trend analysis on the aggregated time series monitoring indicators.

[0100] In one embodiment, the indicator analysis module is used to periodically pull the time-series monitoring indicators corresponding to all nodes to obtain aggregated time-series monitoring indicators, and to perform multi-dimensional display and trend analysis on the aggregated time-series monitoring indicators. Specifically, this includes: accessing the HTTP interfaces exposed by each node according to a preset pulling cycle, and pulling the time-series monitoring indicators corresponding to each node based on the HTTP interfaces; and aggregating the set of time-series monitoring indicators of all nodes pulled according to preset aggregation rules to obtain aggregated time-series monitoring indicators. The aggregation rules include summarizing the request count indicators and time statistics indicators under different label dimensions, using the normalized request path field and node ID as label dimensions. The process involves obtaining aggregated time-series monitoring metrics under different tag dimensions, wherein the aggregated time-series monitoring metrics include aggregated request count metrics and aggregated time statistics metrics; obtaining the target tag dimension selected by the user, obtaining the target aggregated time-series monitoring metrics under the target tag dimension, and visualizing the target aggregated time-series monitoring metrics; obtaining the historical target aggregated time-series monitoring metrics corresponding to the target aggregated time-series monitoring metrics, and generating a line chart based on the target aggregated time-series monitoring metrics and the historical target aggregated time-series monitoring metrics, wherein the line chart at least shows the time-varying trends of the target aggregated time-series monitoring metrics corresponding to the target aggregated request count metrics and the target aggregated time statistics metrics.

[0101] In one embodiment, the key monitoring field extraction module 202 is used to read the log lines of the latest log file and extract the key monitoring fields from the log lines based on a predefined field mapping table. Specifically, this includes: reading the log lines of the latest log file in real time and detecting whether the current log line is empty or contains a comment identifier; if so, skipping the current log line and continuing to read the next log line; if the current log line is detected as a valid data line, integrating all valid data lines to obtain a set of valid data lines; and obtaining a predefined field mapping table, wherein the field mapping table includes key monitoring field name - key monitoring field index. The field mapping table is obtained by identifying comment lines containing preset identifiers when reading the latest log file and parsing the comment lines... A sequence of field names separated by spaces is used to extract the key monitoring field names and key monitoring field indexes corresponding to preset key monitoring fields. Based on the extracted key monitoring field names and their corresponding key monitoring field indexes, a relational table containing key monitoring field names and key monitoring field indexes is constructed. The target field names corresponding to each valid data row in the valid data row set are split by space delimiters. The target field names are matched with the key monitoring field names in the field mapping relational table. If a match is successful, the target key monitoring field index corresponding to the successfully matched key monitoring field name is obtained, and the target key monitoring field value corresponding to the target key monitoring field index is extracted to obtain the key monitoring field corresponding to each valid data row.

[0102] In one embodiment, the time-series monitoring metric generation module 204 is used to generate time-series monitoring metrics based on the key monitoring fields and the normalized request path fields. The time-series monitoring metrics include at least a request count metric and a response time statistical metric tagged with the normalized request path field. Specifically, this includes: constructing a monitoring tag set based on the normalized request path field and at least one server-side attribute tag and at least one client-side attribute tag selected from the key monitoring fields; generating a request count metric for statistically analyzing request frequency based on the monitoring tag set, wherein the request count metric is obtained by accumulating counts based on different dimension combinations of the monitoring tag set; generating a response time statistical metric for statistically analyzing request processing performance based on the monitoring tag set, wherein the response time statistical metric is calculated by processing performance time-series data obtained from the key monitoring fields according to the monitoring tag set and calculating statistical results based on a preset statistical algorithm; and associating and integrating the request count metric and the response time statistical metric based on a unified time benchmark to obtain time-series monitoring data.

[0103] In one embodiment, the normalization processing module 203 is used to normalize the request path field to obtain a normalized request path field. Specifically, it includes: obtaining the request path field extracted from the key monitoring field; detecting whether the request path field contains a query string; if so, removing the query string from the request path field to obtain a preprocessed request path field; replacing the numeric parameters in the preprocessed request path field with a first preset placeholder; and replacing the identifier parameters in the preprocessed request path field with a second preset placeholder to obtain the normalized request path field.

[0104] In one embodiment, the log file reading module 201 is used to monitor the log directories of each node in real time and read the latest log files in the log directories. Specifically, it includes: deploying a directory monitoring service on each node to detect the file change status in the log directories of each node in real time, wherein the file change status includes the creation of new log files; when the file change status of the log directory is detected to be the creation of new log files, triggering the latest log file identification process; reading the metadata information of all log files in the log directory, sorting all log files according to a preset sorting rule based on the metadata information, and selecting the log file with the highest priority in the sorting result as the latest log file.

[0105] The log-based metric collection device described above can implement the log-based metric collection method of the above method embodiments. The options in the above method embodiments are also applicable to this embodiment, and will not be detailed here.

[0106] like Figure 3 As shown, Figure 3 This is a schematic diagram of the structure of a computer device provided in this application; it includes a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112 and the memory 113 communicate with each other through the communication bus 114, and the memory 113 is used to store computer programs.

[0107] In one embodiment of this application, the processor 111, when executing the program stored in the memory 113, implements the log-based metric collection method provided in any of the foregoing method embodiments.

[0108] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program may be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0109] Therefore, embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the log-based metric collection method provided in any of the foregoing method embodiments.

[0110] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, external hard drive, read-only memory (ROM), magnetic disk, or optical disk, or any other physical storage medium capable of storing program code. The computer-readable storage medium can be non-volatile or volatile.

[0111] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0112] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0113] The steps in the methods of this application embodiment can be adjusted, merged, or deleted according to actual needs. The units in the apparatus of this application embodiment can be merged, divided, or deleted according to actual needs. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0114] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0115] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0116] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Since these modifications and variations fall within the scope of the claims and their equivalents, this application also intends to include these modifications and variations.

[0117] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A log-based metric collection method, characterized in that, include: Monitor the log directories of each node in real time and read the latest log files in the log directories; Read the log lines of the latest log file, and extract the key monitoring fields from the log lines based on a predefined field mapping table, wherein the key monitoring fields include the request path field; The request path field is normalized to obtain a normalized request path field; Time-series monitoring metrics are generated based on the key monitoring fields and the normalized request path field, wherein the time-series monitoring metrics include at least a request count metric and a response time statistics metric labeled with the normalized request path field.

2. The method as described in claim 1 above, characterized in that, Also includes: Periodically retrieve the time series monitoring metrics corresponding to all nodes to obtain aggregated time series monitoring metrics; The aggregated time series monitoring indicators are displayed in multiple dimensions and their trends are analyzed.

3. The method as described in claim 2 above, characterized in that, The process involves periodically retrieving the time-series monitoring metrics corresponding to all nodes to obtain aggregated time-series monitoring metrics, and then performing multi-dimensional display and trend analysis on these aggregated time-series monitoring metrics, specifically including: Access the HTTP interface exposed by each node according to the preset retrieval period, and retrieve the time series monitoring indicators corresponding to each node based on the HTTP interface. Based on preset aggregation rules, the time series monitoring indicator set of all nodes is aggregated to obtain aggregated time series monitoring indicators. The aggregation rules include summarizing the request count indicator and the time statistics indicator under different label dimensions using the normalized request path field and node ID as label dimensions to obtain aggregated time series monitoring indicators under different label dimensions. The aggregated time series monitoring indicators include summarized request count indicators and summarized time statistics indicators. Obtain the target tag dimension selected by the user, obtain the target aggregated time series monitoring indicators under the target tag dimension, and perform visualization processing on the target aggregated time series monitoring indicators; Obtain the historical target aggregated time series monitoring indicators corresponding to the target aggregated time series monitoring indicators, and generate a line chart based on the target aggregated time series monitoring indicators and the historical target aggregated time series monitoring indicators. The line chart at least shows the changing trends of the target aggregated time series monitoring indicators and the target aggregated time statistics indicators over time.

4. The method as described in claim 1, characterized in that, The step of reading the log lines of the latest log file and extracting the key monitoring fields from the log lines based on a predefined field mapping table specifically includes: The system reads the latest log line from the log file in real time and checks whether the current log line is empty or contains a comment. If so, it skips the current log line and continues to read the next log line. If the current log line is detected as a valid data line, it integrates all valid data lines to obtain a set of valid data lines. Obtain a predefined field mapping table, wherein the field mapping table includes key monitoring field names and key monitoring field indexes. The field mapping table is constructed by identifying comment lines containing preset identifiers when reading the latest log file, parsing the space-separated field name sequence in the comment lines, extracting the key monitoring field names and key monitoring field indexes corresponding to preset key monitoring fields from the field name sequence, and constructing a relationship table containing key monitoring field names and key monitoring field indexes based on the extracted key monitoring field names and corresponding key monitoring field indexes. The target field name corresponding to each valid data row in the set of valid data rows is split by a space separator. The target field name is then matched with the key monitoring field name in the field mapping table. If a match is successful, the target key monitoring field index corresponding to the matched key monitoring field name is obtained, and the target key monitoring field value corresponding to the target key monitoring field index is extracted to obtain the key monitoring field corresponding to each valid data row.

5. The method as described in claim 4 above, characterized in that, The generation of time-series monitoring metrics based on the key monitoring fields and the normalized request path fields, wherein the time-series monitoring metrics include at least a request count metric and a response time statistical metric labeled with the normalized request path field, specifically including: Based on the normalized request path field and at least one server-side attribute tag and at least one client-side attribute tag selected from the key monitoring fields, a monitoring tag set is constructed; Based on the monitoring tag set, a request count indicator is generated to count the frequency of requests. The request count indicator is obtained by counting and accumulating different combinations of the monitoring tag set. Based on the monitoring tag set, a response time statistical indicator for statistically analyzing request processing performance is generated. The response time statistical indicator is calculated by processing the performance time-series data obtained from the key monitoring fields according to the monitoring tag set and calculating the statistical results based on a preset statistical algorithm. The request count metric and the response time metric are correlated and integrated based on a unified time benchmark to obtain time series monitoring data.

6. The method as described in claim 1, characterized in that, The normalization process for the request path field to obtain a normalized request path field specifically includes: Obtain the request path field extracted from the key monitoring fields, and check whether the request path field contains a query string. If so, remove the query string from the request path field to obtain the preprocessed request path field. The numeric parameters in the preprocessed request path field are replaced with a first preset placeholder, and the identifier parameters in the preprocessed request path field are replaced with a second preset placeholder to obtain the normalized request path field.

7. The method as described in claim 1, characterized in that, The real-time monitoring of the log directories of each node and reading the latest log files in the log directories specifically includes: Deploy a directory monitoring service on each node to detect file changes in the log directory of each node in real time, including the creation of new log files; When a file change in the log directory is detected as the creation of a new log file, the latest log file identification process is triggered. Read the metadata information of all log files in the log directory, sort all log files according to a preset sorting rule based on the metadata information, and select the log file with the highest priority in the sorting result as the latest log file.

8. A log-based metric collection device, characterized in that, include: Log file reading module, key monitoring field extraction module, normalization processing module, and time series monitoring indicator generation module; The log file reading module is used to monitor the log directory of each node in real time and read the latest log file in the log directory. The key monitoring field extraction module is used to read the log lines of the latest log file and extract the key monitoring fields in the log lines based on a predefined field mapping table. The key monitoring fields include the request path field. The normalization processing module is used to normalize the request path field to obtain a normalized request path field. The time series monitoring metric generation module is used to generate time series monitoring metrics based on the key monitoring field and the normalized request path field, wherein the time series monitoring metrics include at least a request count metric and a response time statistics metric labeled with the normalized request path field.

9. A computer device, characterized in that, The computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, can implement the method as described in any one of claims 1-7.

Images