Method and system for generating observable data through lightweight collection of WEB container logs

Abstract

The invention belongs to the technical field of computer technology server data processing, and discloses a method and a system for generating observable data through lightweight collection of WEB container logs. According to the method, a mode of expanding a filter of a WEB container is adopted, a user-defined filter is added to generate data in a set format, and the data is output to a log file; the method comprises the following steps: extracting data in a log, processing the data, and then calling a log storage module to store the data in a storage; creating an index according to the label, and maintaining a mapping relationship between the index and the original log content; and converting the log data into an observable view by utilizing the maintained index and data file storage structure. According to the method, lightweight design serves as the core, logs serve as carriers of original data, the resource cost of the data from generation, collection to display is reduced, the observability requirement is met, and the provided data can meet aggregation display of common various dimension index data.

Description

Technical Field

[0001] This invention belongs to the field of computer technology server-side data processing technology, and in particular relates to a lightweight method and system for collecting web container logs to generate observable data. Background Technology

[0002] A web container is a type of server software used to host web applications. It resides within the application server, providing a runtime environment for dynamic components, handling client requests and forwarding them to the appropriate server-side programs. It manages the lifecycle of the web application, request processing, sessions, etc., shielding it from underlying protocol details (such as thread safety, transactions, networking, and resources) so that business developers can focus on implementing business logic without needing to concern themselves with the underlying implementation details.

[0003] "Observability" is a crucial concept across multiple fields. Its core idea is to infer and understand the internal state of a system through information output from its external environment. In the field of computer software, the observability of web containers specifically refers to the ability to process data generated during web container runtime to obtain information reflecting the system's operational status. This information is typically presented visually to system administrators in the form of charts, text, or other visual aids. Observable data offers the following advantages: Rapid Fault Diagnosis and Troubleshooting: Assists in locating the root cause of problems when system anomalies occur. Understanding System Behavior: Provides in-depth understanding of the system's operation and resource consumption under real-world loads. Improving System Performance: Identifies optimization points to enhance user experience. Enhancing System Reliability: Proactively addresses potential problems before they impact users. Optimizing Resource Utilization: Enables reasonable capacity planning and cost optimization based on load and resource consumption.

[0004] Web container observation data is categorized into three types: Metrics, Logs, and Traces, each corresponding to different processing methods. Common components handle only one type of data. For example, Prometheus can work with Telegraf to collect Metrics data; SkyWalking uses its self-developed probe to collect Traces data and store it in Elasticsearch; ELKStack is used to collect and aggregate Logs information generated by services, providing a unified query solution. OpenTelemetry offers a unified method for collecting and exporting these three types of data, but the number of components it relies on and the cost remain relatively high.

[0005] Based on the above analysis, the problems and shortcomings of existing technologies are as follows: Web services use web containers as basic components. Common observability solutions typically require the deployment of many additional components to implement data collection, processing, storage, and display. Moreover, different dimensions of data such as logs, metrics, and traces usually require the deployment of different components, introducing significant resource and operational costs. Another method, built-in monitoring and display in web services, only provides real-time data queries, lacks persistence support, cannot retrospectively query historical data, and some cumulative metrics are reset after service restarts, making it difficult to meet long-term observation needs. Summary of the Invention

[0006] To overcome the problems existing in related technologies, this invention discloses a lightweight method and system for collecting web container logs to generate observable data. In particular, it relates to a method for obtaining observable data of web service containers based on logs, which can provide support and assistance for web service operation and maintenance.

[0007] The technical solution is as follows: This lightweight method for collecting web container logs and generating observable data includes the following steps: S1 uses an extended WEB container filter approach to add custom filters to generate data in a specified format and output the data to a log file; S2 extracts data from the logs, processes the data, and calls the log storage module to save the data to storage. S3, create an index based on the tags and maintain the mapping relationship between the index and the original log content; S4 uses maintained indexes and data file storage structures to transform log data into observable views.

[0008] In step S1, a custom filter is added to the web container to generate data in a specified format, and the data is output to a log file, including: Customize Class I and Class II filters for different containers; One type of filter is used to obtain request and response objects, retrieve TraceId and SpanId, and put them into the log context; record the HTTP Method type of the request, the request URI, and the start time and processing time at the end of the request; and print logs when the request enters the web container. The second type of filter is used to put elements into the queue according to the frequency of requests, activate tasks listening to the blocking queue, and the activated tasks obtain internal service status information and print the information to the log file.

[0009] Furthermore, in a type of filter-printed log, the data generated in the TRACE_REC_LOG type format includes: Log time [TraceId, ParentSapnId, SpanId] Log level [processing thread] Log type – [log subtype]: [HTTP request Method Request URI][Request start timestamp]; The second type filter generates data in METRIC_REC_LOG format.

[0010] In step S2, data is extracted from the logs and processed, including: obtaining log content of TRACE_REC_LOG and METRIC_REC_LOG types by periodically capturing incremental logs, collecting log information from multiple service nodes; and adding labels to the logs according to the service and configuration to which the logs belong.

[0011] Furthermore, after reading the log content, an asynchronous thread is used to process the log content and add tags according to the configuration items; For TRACE_REC_LOG type logs, the log position of the keyword is determined according to the configured regular expression, and the relative position of the keyword is determined by the predetermined space character; then, the keyword is only used as a data item in the query and no index is created, while the tag is used as the grouping basis to create an index for TRACE_REC_LOG type logs; For METRIC_REC_LOG type logs, the key and value are separated by the space character. The resulting key matches the content in the "{}" character. The tag information in the indicator data is extracted, and a METRIC_REC_LOG type log index is created based on the tag information.

[0012] In step S3, an index is created based on the tags, and the mapping relationship between the index and the original log content is maintained. This includes: receiving incoming data through a reserved HTTP interface, extracting the tag information of the data to build the index, and maintaining the position after writing the data block file in the index; specifically: The storage service first selects the root directory for storage according to the data category; then it groups the data files according to the time range, records summary information and keeps the grouped data in the group directory file. The summary information includes the disk space usage of each group, the start and end times of the record, and the number of block files contained therein. The files in the grouped directory include metadata files, index files, block files that store compressed data, and filter files for filtering. For link data, find the corresponding multiple block files according to the time range, traverse the index file in each block file, maintain the three Bloom filter files corresponding to the index file: TraceId, ParentSpanId, and SpanId, and update the content of the Bloom filter synchronously every time data is written to the data file. For log data, maintain corresponding Bloom filter files based on keywords; For indicator data, the index file points directly to the data block file, and then the required data is retrieved based on the #TYPE header type in the data file.

[0013] Furthermore, the metadata file contains the start and end time range, file ID, and compression information for each data file; The index file includes the correspondence between tag data, keyword data and data blocks. The tag metadata part records the content of the existing tag groups, the relationship data records the correspondence between tag groups and block file directories, and adds the corresponding time range information and specific data file information. The chunk files are stored in the chunks directory, named and saved according to the time of the collected log records. The size of a single chunk file is fixed. If the size exceeds the size of a single chunk file, it is split and saved to the next file.

[0014] In step S4, the log data is transformed into an observable view using the maintained index and data file storage structure, including: (1) Link data: By traversing the Bloom filter file through the time range, find the block file where the link ID may exist, obtain the detailed log content in it, filter it, and return it; Link data queries can quickly obtain complete data link results by searching by TraceId, ParentSpanId, and SpanId; (2) Indicator data: Based on the time range of the query, the corresponding data directory is obtained by querying the metadata in the metric directory. The block file information corresponding to the metric to be queried is obtained based on the index file in the directory. After loading the data, the information that meets the conditions is filtered out, and then the calculation is performed in memory. Constant or two-dimensional matrix data is generated and returned through sorting, aggregation and other methods. (3) Log content, providing abnormal log content display, QPS, interface response time sorting, and error rate index query; when querying, first obtain the corresponding data directory in the log directory according to the time range and the corresponding data block file according to the index file in the directory, use the Bloom filter maintained by the keyword specified in the collection stage to skip the block file that does not need to be queried, and then obtain the log information according to the inherent format of the row log and determine the log type.

[0015] Furthermore, in the log content, QPS is calculated based on the REQ-S type logs output by a filter, and interface response time is calculated based on the interface latency in the REQ-E type logs output by a filter. The interface error rate is calculated based on the status code portion of the REQ-E type logs output by a filter to classify errors.

[0016] Another object of the present invention is to provide a lightweight system for collecting web container logs and generating observable data. This system implements the aforementioned method for collecting web container logs and generating observable data. The system includes: The log generation module is used to add custom filters to generate data in a specified format by extending the filters of the web container, and then output the data to the log file. The log collection module is used to extract data from logs, process the data, and call the log storage module to save the data to storage. The log storage module is used to create indexes based on tags and maintain the mapping relationship between the indexes and the original log content; The log query module is used to transform log data into observable views by utilizing the maintained indexes and data file storage structure.

[0017] Combining all the above technical solutions, the beneficial effects of this invention are as follows: Web services typically focus on key metrics including CPU utilization, memory usage, disk I / O, response time (QPS), throughput, and error rate. Error logs are also needed in most scenarios to assist in problem localization. For tracing link data, observing inter-service calls is sufficient to meet basic requirements. Therefore, considering cost, a lightweight solution is adopted to achieve the acquisition and display of observable data, satisfying users' observation needs for web containers.

[0018] This invention focuses on lightweight design, using logs as the carrier of raw data. This reduces the resource costs of data generation, collection, and display, while fulfilling observability requirements. The provided data can aggregate and display commonly used dimensional metrics. Through a lightweight design, this invention achieves the observability requirements of web containers, reducing deployment and data storage costs.

[0019] This invention leverages existing key technologies, integrates them, and designs a lightweight product that can operate independently and possesses a certain degree of usability. This invention addresses the problem faced by small, low-resource-consuming services that desire observability but cannot afford higher monitoring costs or want their data hosted on a SaaS platform. This invention demonstrates that achieving observability for web containers does not necessarily require combining multiple components; basic observation requirements can be met through a low-cost solution. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure; Figure 1 This is a schematic diagram illustrating the principle of a lightweight system for collecting web container logs and generating observable data, as provided in an embodiment of the present invention. Figure 2 This is a flowchart of a lightweight method for collecting web container logs and generating observable data, provided in an embodiment of the present invention. Figure 3 This is a data directory structure diagram of the log storage module provided in this embodiment of the invention, which uses ordinary files as the basis. Figure 4 This is a schematic diagram of the process of collecting data using a filter, provided in an embodiment of the present invention. Figure 5 This is a schematic diagram of the data extraction index file provided in an embodiment of the present invention. Detailed Implementation

[0021] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0022] The innovation of this invention lies in its use of a custom data structure to store integrated indicator data, thereby meeting the observation requirements for data items of interest to web containers. This replaces the common implementation schemes that combine multiple components, thus reducing deployment resource costs.

[0023] Example 1: The lightweight system for collecting web container logs and generating observable data provided in this embodiment of the invention includes a log generation module, a log collection module, a log storage module, and a log query module.

[0024] The log generation module is used to add custom filters to generate data in a specified format by extending the filters of the web container, and then output the data to the log file. The log collection module is used to extract data from logs, process the data, and call the log storage module to save the data to storage. The log storage module is used to create indexes based on tags and maintain the mapping relationship between the indexes and the original log content; The log query module is used to transform log data into observable views by utilizing the maintained indexes and data file storage structure.

[0025] like Figure 1 The schematic diagram of the lightweight WEB container log collection and observable data generation system provided in this embodiment of the invention.

[0026] Example 2, as Figure 2 As shown in the figure, the lightweight method for collecting web container logs and generating observable data provided in this embodiment of the invention includes: S1 uses an extended WEB container filter approach to add custom filters to generate data in a specified format and output the data to a log file; S2 extracts data from the logs, processes the data, and calls the log storage module to save the data to storage. S3, create an index based on the tags and maintain the mapping relationship between the index and the original log content; S4 uses maintained indexes and data file storage structures to transform log data into observable views.

[0027] For example, in step S1, custom filters are developed for different containers. The functions to be implemented by one type of filter include: obtaining request and response objects, obtaining TraceId and SpanId (which can be done with the help of the embedded link tracing component), and putting them into the log context; recording the HTTP Method type of the request, the request URI, and the start time and end time of the request.

[0028] TraceId and SpanId are core concepts in distributed tracing systems.

[0029] TraceId: Trace ID, a globally unique identifier used to identify the complete trace path of a single request in a distributed system. It is generated when the request enters the system and runs through the entire processing flow.

[0030] SpanId: Span ID, a local identifier that indicates the call hierarchy and location of a request within a single service.

[0031] HTTP Method: The HTTP request type in network communication, common ones include GET, POST, PUT, DELETE, etc.

[0032] URI: Uniform Resource Identifier, is a string used to identify Internet resources, including two forms: Uniform Resource Locator (URL) and Uniform Resource Name (URN).

[0033] In some containers that support asynchronous processing, context information cannot be passed through ordinary filters (passing within the same thread). It is necessary to use the asynchronous support of the web container (passing information across threads through an object that holds the information) to maintain information such as start time.

[0034] As we know, web containers themselves have standard development specifications for asynchronous methods and provide user-defined objects to pass context information for asynchronous processing. For example, Tomcat provides the AsyncContext interface, which allows developers to access the original HttpServletRequest and HttpServletResponse objects in asynchronous operations.

[0035] When the request enters the container, the log is printed with the following content: 2025-05-28 08:17:23.926[c851885f,c851885f,c851885f] INFO [http-nio-8077-exec-10]TRACE_REC_LOG -[REQ-S]:[GET / user / all][1748391443926]; Format description: Log time [TraceId, ParentSapnId, SpanId] Log level [processing thread] Log type – [Log subtype]: [HTTP request method Request URI][Request start timestamp]; When the request ends and the response content is returned, print a log in the following format: 2025-05-28 08:17:24.074 [c851885f,c851885f,c851885f] INFO [http-nio-8077-exec-9]TRACE_REC_LOG - [REQ-E]:[GET / user / all 200]

[148] ; Format description: Log time [TraceId, ParentSapnId, SpanId] Log level [processing thread] Log type – [log subtype]: [http request method request URI][request processing duration]; The second type of filter internally maintains a blocking queue and a request counter. Based on the frequency of requests (the number of requests reaching a threshold within a unit of time), elements are added to the queue to activate the task of listening to the blocking queue. The task of activating the task of listening to the blocking queue includes: using the API provided by the "blocking queue" data structure itself, when the queue is empty, the calling thread will be blocked until an element is added. After an element is added, the originally blocked method will continue to execute, thus achieving the activation effect. Once activated, the task retrieves internal service status information (such as CPU utilization, memory usage, and thread count) and prints the information to a log file. The content is as follows: 2025-05-28 08:17:24.074 METRIC_REC_LOG - [S] # TYPE process_cpu_usage gauge process_cpu_usage 5.515744688934203E-4 # TYPE memory_used_bytes gauge memory_used_bytes{area="heap"} 3.36870912E8 # TYPE threads_states_threads gauge threads_states_threads{state="runnable",} 128.0 threads_states_threads{state="blocked",} 0.0 threads_states_threads{state="waiting",} 152.0 threads_states_threads{state="timed-waiting",} 180.0 threads_states_threads{state="new",} 0.0 threads_states_threads{state="terminated",} 0.0 2025-05-28 08:17:24.080 METRIC_REC_LOG - [E]; For example, in step S2, the log collection module obtains log content of type TRACE_REC_LOG and METRIC_REC_LOG by periodically capturing incremental logs, collecting log information from multiple service nodes. Labels are added to the logs according to their service and configuration. The processed logs are then saved by the log storage module.

[0036] For example, the log collection module implementation is given below. If the collector is deployed on the local machine, remote access permissions are not required; otherwise, the SSH user password is required for remote access to the log files.

[0037] Because log files refresh quickly, the collector was designed with real-time performance in mind. It uses a local method to listen for file change events (common operating systems like Linux and Windows provide interfaces for listening for file changes, and corresponding libraries exist for various programming languages, making it easy to implement file change monitoring). Remote access uses long polling (when a collection request is received, data is not immediately returned; the asynchronous processing capabilities of the container's Servlet 3.0+ specification are used to suspend the received request until the monitored log file changes, at which point the changed data is returned as the response content). The log collection module internally increments the current line count for each collected file, restarting the count from 0 if a line count rollback is detected. Each fetch is based on the previously retrieved line content and the incremental content of the current file's line count (incremental fetching; after each text content is retrieved, the current line count is updated to serve as the starting line index for the next fetch; the line index for each log file is persisted to a separate log file to prevent loss of index records after service restarts). When the log system is split by size or date, in addition to recording the number of lines each time, the current service also records the last processing time of the log. If a log rollback is detected, the previous log file is traced back, and the missing lines are filled in by rolling (a certain number of lines at a time) from the end of the time.

[0038] The log collection module here uses a non-blocking asynchronous programming framework to improve the system's processing power and response timeliness (the core advantage of the non-blocking asynchronous model is that it can carry a large number of I / O operations with a small number of threads, freeing up CPU resources from waiting by eliminating the waste of "waiting", and squeezing out hardware performance to the extreme, thereby achieving extremely high throughput). In addition, scenarios where log writing is too fast need to be considered, and flow control support needs to be added to control the amount of log content read each time. If the log writing exceeds a certain threshold, the collection will be delayed (this function is similar to the backpressure mechanism, the purpose of which is to prevent the receiver from being unable to complete processing due to the log generation speed being too fast, thus affecting stability). After listening for log changes, the current file line count is obtained. For example, on Linux, the command "wc -l " + path + " | awk '{print $1}'" can be used to count the file lines. Each time lines are read, they are read in a fixed length, such as 200 lines at a time, in a loop. Combined with the configured threshold (the threshold needs to take into account the type of log printing content, the average number of bytes per line, and network transmission bandwidth limitations, etc.), assuming the threshold is 1000 lines, if the change exceeds 1000 lines, only 1000 lines are returned, and the log is sent again after a fixed interval, such as 50ms). At this time, the log collection module will send a notification to the system maintenance personnel through an alarm.

[0039] After the log collection module reads the log content, it uses an asynchronous thread to process the log content, adding tags according to the configuration items to provide grouping and aggregation capabilities for queries.

[0040] For example, since the log collection module may read multiple log files, multiple different threads need to be started to prevent mutual blocking. In this case, asynchronous processing is needed to start a separate thread for processing. The asynchronous thread parses the original log text, matches it according to the configured regular expressions, and adds tags to those that meet the requirements. For TRACE_REC_LOG type logs, the log position of the keyword is determined according to the configured regular expression (for example, step S1 gives the format of TRACE_LOG: 2025-05-28 08:17:23.926 [c851885f,c851885f,c851885f] INFO [http-nio-8077-exec-10]TRACE_REC_LOG - [REQ-S]:[GET / user / all][1748391443926]; keywords are separated by space characters. Since the printed content is output according to the pre-set specifications, the above fixed format of the log can be organized, and each keyword can be separated by regular expressions). The fixed space characters determine the relative position of the keywords, improving the efficiency of subsequent row data retrieval. Keywords are only used as data items in the query and are not indexed, while tags are used as grouping criteria and are indexed.

[0041] For METRIC_REC_LOG type logs, the key and value are separated by a space character. The resulting key matches the content within the "{}" characters. Tag information is extracted from the metric data, and an index is created based on the tag information. For example, METRIC LOG in step S1. threads_states_threads{state="runnable"} 128.0; The output follows a predefined specification, with the format key {} value, separated by spaces. The text within the {} is then extracted, parsed into key-value pairs, and an index is created for the key.

[0042] For example, in step S3, the log storage module provides data persistence capabilities by receiving incoming data through a reserved HTTP interface, extracting the data's tag information to build an index, and maintaining the position of the data block file in the index.

[0043] The log storage module uses ordinary files as its foundation; the data directory structure can be found in the reference. Figure 3 Specifically, this includes: The storage service first selects the root directory for storage based on data categories. Then, it groups the data files for storage according to time ranges (configurable, default is 2h), records summary information, and keeps the grouped data in the group directory. The summary information includes the disk space usage of each group, the start and end times of the record, the number of block files contained, etc.

[0044] The files in the grouped directory are mainly divided into: metadata files, index files (directories), block files that store compressed data, and filter files used for filtering.

[0045] The metadata file contains the start and end time range, file ID, and compression information for each data file. The system performs queries based on time, and the search criteria must include a time range; therefore, the metadata can be used to locate the data block to be scanned.

[0046] The index file includes the correspondence between tag data, keyword data, and data blocks. The tag metadata section records the content of existing tag groups, and the relationship data records the correspondence between tag groups and block file directories, and adds the corresponding time range information and specific data file information.

[0047] Chunk files are stored in the chunks directory, named and saved chronologically according to the time of log recording, and support data compression. The size of a single chunk file is fixed (512MB for metric types, and 2MB by default for log types). This is because metrics typically observe data over a period of time, and the large number of metrics means that small single files would frequently trigger splitting. Log type files, on the other hand, have relatively independent data, with related data usually concentrated within a relatively short timeframe, such as milliseconds or seconds. When a chunk exceeds the size of a single block, it is split and saved to the next file. This allows for efficient loading of multiple independent files, improving retrieval efficiency.

[0048] The query must include a time range and tag information. This allows the corresponding block file directory to be quickly found based on the tags in the index file, and then the content of the block file to be queried can be quickly located based on the time range in the index.

[0049] For example, the content of the desired block file can be quickly located based on the time range in the index. The block files are divided by time, and the index maintains the block file names and their start and end times, sorted in order. During a query, the corresponding block file range can be quickly found based on the time range. For link data, multiple block files corresponding to the time range are found. The index files within each block file are traversed, and three Bloom filter files (TraceId, ParentSpanId, and SpanId) corresponding to the index files are maintained. Each time data is written to the data file, the contents of the Bloom filters are updated synchronously (the written TraceId and other data are fixed-length strings; the Bloom filter hash function algorithm is used to calculate the input link ID, and then written to the Bloom filter data file). Here, the Bloom filter is used to skip block files that do not contain the retrieved link ID.

[0050] If there are requirements for the richness of the link data itself, and the link information is complete, then it is necessary to extend the filter to print more information to the log for easy collection, such as message listening, processing of scheduled tasks, asynchronous processing of thread pools, etc.

[0051] For log data, corresponding Bloom filter files are maintained based on keywords to improve the efficiency of retrieving log content by keywords.

[0052] For metric data, the index file points directly to the data block file. The reason for not using a Bloom filter here is that metric data and log data are generated differently. Metrics always exist periodically in the block file. By first locating the specific data file based on the tag, and then quickly retrieving the required data based on the #TYPE header type in the data file, this ensures more efficient querying.

[0053] For example, such as Figure 4 The principle of using filters to collect data, such as Figure 5 The principle of data extraction index file.

[0054] For example, in step S4, the log query module obtains log data through the query endpoint provided by the data storage service of the log storage module, and performs conditional retrieval and aggregation to generate a result view. Depending on the different categories of collected data, the following different processing methods are adopted: (1) Link data: By traversing the Bloom filter file through the time range, find the block file where the link ID may exist (based on the characteristics of the Bloom filter, block files that do not store the link ID can be excluded), obtain the detailed log content in it, filter it and return it.

[0055] For example, after retrieving and filtering the detailed log content, the Bloom filter algorithm is used. Based on the link ID string, the Bloom filter API is called, passing in the link ID and the corresponding Bloom data file for the block file (the block file can be obtained by querying the index based on the time range, and the corresponding Bloom filter data file can be obtained based on the correspondence in the block file's index file). The Bloom filter algorithm is a widely used and publicly available algorithm. Its implementation can be obtained through various channels. The Bloom filter was proposed by Bloom in 1970. It is actually a long binary vector and a series of random mapping functions. Bloom filters can be used to check whether an element is in a set.

[0056] Link data query supports searching by TraceId, ParentSpanId, and SpanId to quickly obtain complete data link results. This assists in analyzing link operation status.

[0057] For example, in the complete data chain result, the combination of TraceId, ParentSpanId, and SpanId is unique, representing an independent task processing process, obtained by querying the keywords TraceId, ParentSpanId, and SpanId. The query process is performed according to the filtering and keyword matching method described above for traversing block files.

[0058] (2) Metric data: Based on the time range of the query, the corresponding data directory is obtained by querying the metadata in the metric directory (the log content generated by the second type of filter in step S1 is stored separately in the metric directory by the log storage module). The block file information corresponding to the metric to be queried is obtained based on the index file in the directory. After loading the data, the information that meets the conditions is filtered out, and then the calculation is performed in memory. Constant or two-dimensional matrix data is generated and returned through sorting, aggregation and other methods.

[0059] (3) Log content: Provides display of abnormal log content, and provides queries for metrics such as QPS, interface response time sorting, and error rate. When querying, first obtain the corresponding data directory in the log directory according to the metadata based on the time range, find the data block file pointed to by the index file in the directory, use the Bloom filter maintained by the keywords specified in the collection phase to skip the block files that do not need to be queried, and then obtain the log information by determining the log type according to the inherent format of the row log.

[0060] The QPS can be calculated based on the REQ-S type logs output by the filter in step S1 (the REQ-S logs are filtered line by line, and the number of records within the time range is the number of requests received). The interface response time can be calculated based on the interface latency in the REQ-E type logs output by the filter in S1 (the interface latency is obtained by finding the text within the last square brackets [] in the REQ-E type log). The interface error rate can also be calculated based on the status code portion of the REQ-E type logs output by the filter in S1 to classify errors.

[0061] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0062] The information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0063] Application Example. The lightweight web container log collection and observable data generation system provided in this embodiment includes: a log generation module. This invention is used in a flight management information system, where the flight management system adds filter implementation dependencies. After packaging and running, a log output strategy is configured. During runtime, request logs and service status logs are printed to a log file.

[0064] The log collection module requires configuring the address of the collection service, specifying the tags to be added to the logs, and specifying the regular expression matching format for the logs to be collected, so as to facilitate keyword extraction and grouping.

[0065] The log storage module allows you to set the data retention period, maximum disk space usage, and storage service memory usage limits. It also provides the deployment address for use by the log collection and log query services.

[0066] The log query module, after configuring the log storage service address, provides a visual web page for querying information such as CPU utilization, memory usage, disk I / O, response time, QPS, throughput, and error rate.

[0067] Scenario 1: During peak tourist season at an airport, the flight management system experienced increased response latency. Query service QPS analysis revealed that the number of requests during the same period had roughly doubled compared to historical data, with a significant increase in service memory and CPU utilization. This indicated a need to expand existing service resources to cope with the increased pressure on the system due to the increased flight volume.

[0068] Scenario 2: After using flight management for a period of time, users want to estimate the peak time periods for flight support tasks each day. In this case, they can query the QPS (Queries Per Second) over a daily period using a specific business interface (based on REQ-S logs), and obtain the peak time period information from the returned two-dimensional line chart to help rationally allocate personnel work.

[0069] Scenario 3: Users report that the flight management system is running slowly during a certain period. In this case, you can query the interfaces with slow response times based on the time range, obtain the link ID from a specific request-response log (based on REQ-E logs), and then query the time consumption of key nodes based on the link ID. Analysis will reveal that the slow processing time period involves external interface calls that take a long time.

[0070] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A lightweight method for collecting web container logs and generating observable data, characterized in that, The method includes the following steps: S1 uses an extended WEB container filter approach to add custom filters to generate data in a specified format and output the data to a log file; S2 extracts data from the logs, processes the data, and calls the log storage module to save the data to storage. S3, create an index based on the tags and maintain the mapping relationship between the index and the original log content; S4 uses maintained indexes and data file storage structures to transform log data into observable views.

2. The method for lightweight collection of web container logs to generate observable data according to claim 1, characterized in that, In step S1, a custom filter is added to the web container to generate data in a specified format, and the data is output to a log file, including: Customize Class I and Class II filters for different containers; One type of filter is used to obtain request and response objects, retrieve TraceId and SpanId, and put them into the log context; record the HTTP Method type of the request, the request URI, and the start time and processing time at the end of the request; and print logs when the request enters the web container. The second type of filter is used to put elements into the queue according to the frequency of requests, activate tasks listening to the blocking queue, and the activated tasks obtain internal service status information and print the information to the log file.

3. The method for lightweight collection of web container logs to generate observable data according to claim 2, characterized in that, In a type of filter-printed log, the data generated in the TRACE_REC_LOG type format includes: Log time [TraceId, ParentSapnId, SpanId] Log level [Processing thread] Log type – [Log subtype]: [HTTP request Method Request URI][Request start timestamp]; The second type filter generates data in METRIC_REC_LOG format.

4. The method for lightweight collection of web container logs to generate observable data according to claim 1, characterized in that, In step S2, data is extracted from the logs and processed, including: obtaining log content of TRACE_REC_LOG and METRIC_REC_LOG types by periodically capturing incremental logs, collecting log information from multiple service nodes; and adding labels to the logs according to the service and configuration to which the logs belong.

5. The method for lightweight collection of web container logs to generate observable data according to claim 4, characterized in that, After reading the log content, use an asynchronous thread to process the log content and add tags according to the configuration items; For TRACE_REC_LOG type logs, the log position of the keyword is determined according to the configured regular expression, and the relative position of the keyword is determined by the predetermined space character; then, the keyword is only used as a data item in the query and no index is created, while the tag is used as the grouping basis to create an index for TRACE_REC_LOG type logs; For METRIC_REC_LOG type logs, the key and value are separated by the space character. The resulting key matches the content in the "{}" character. The tag information in the indicator data is extracted, and a METRIC_REC_LOG type log index is created based on the tag information.

6. The method for lightweight collection of web container logs to generate observable data according to claim 1, characterized in that, In step S3, an index is created based on the tags, and the mapping relationship between the index and the original log content is maintained. This includes: receiving incoming data through a reserved HTTP interface, extracting the tag information of the data to build the index, and maintaining the position after writing the data block file in the index; specifically: The storage service first selects the root directory for storage according to the data category; then it groups the data files according to the time range, records summary information and keeps the grouped data in the group directory file. The summary information includes the disk space usage of each group, the start and end times of the record, and the number of block files contained therein. The files in the grouped directory include metadata files, index files, block files that store compressed data, and filter files for filtering. For link data, find the corresponding multiple block files according to the time range, traverse the index file in each block file, maintain the three Bloom filter files corresponding to the index file: TraceId, ParentSpanId, and SpanId, and update the content of the Bloom filter synchronously every time data is written to the data file. For log data, maintain corresponding Bloom filter files based on keywords; For indicator data, the index file points directly to the data block file, and then the required data is retrieved based on the #TYPE header type in the data file.

7. The method for lightweight collection of web container logs to generate observable data according to claim 6, characterized in that, The metadata file contains the start and end time range, file ID, and compression information for each data file; The index file includes the correspondence between tag data, keyword data and data blocks. The tag metadata part records the content of the existing tag groups, the relationship data records the correspondence between tag groups and block file directories, and adds the corresponding time range information and specific data file information. The chunk files are stored in the chunks directory, named and saved according to the time of the collected log records. The size of a single chunk file is fixed. If the size exceeds the size of a single chunk file, it is split and saved to the next file.

8. The method for lightweight collection of web container logs to generate observable data according to claim 1, characterized in that, In step S4, the log data is transformed into an observable view using the maintained index and data file storage structure, including: (1) Link data: By traversing the Bloom filter file through the time range, find the block file where the link ID may exist, obtain the detailed log content in it, filter it, and return it; Link data queries can quickly obtain complete data link results by searching by TraceId, ParentSpanId, and SpanId; (2) Indicator data: Based on the time range of the query, the corresponding data directory is obtained by querying the metadata in the metric directory. The block file information corresponding to the metric to be queried is obtained based on the index file in the directory. After loading the data, the information that meets the conditions is filtered out, and then the calculation is performed in memory. Constant or two-dimensional matrix data is generated and returned through sorting, aggregation and other methods. (3) Log content, providing abnormal log content display, QPS, interface response time sorting, and error rate index query; when querying, first obtain the corresponding data directory in the log directory according to the time range and the corresponding data block file according to the index file in the directory, use the Bloom filter maintained by the keyword specified in the collection stage to skip the block file that does not need to be queried, and then obtain the log information according to the inherent format of the row log and determine the log type.

9. The method for lightweight collection of web container logs to generate observable data according to claim 8, characterized in that, In the log content, QPS is calculated based on the REQ-S type logs output by a filter, and interface response time is calculated based on the interface latency in the REQ-E type logs output by a filter. The interface error rate is calculated based on the status code portion of the REQ-E type logs output by a filter to classify errors.

10. A lightweight system for collecting web container logs and generating observable data, characterized in that, This system implements the lightweight method for collecting web container logs and generating observable data as described in any one of claims 1-9, the system comprising: The log generation module is used to add custom filters to generate data in a specified format by extending the filters of the web container, and then output the data to the log file. The log collection module is used to extract data from logs, process the data, and call the log storage module to save the data to storage. The log storage module is used to create indexes based on tags and maintain the mapping relationship between the indexes and the original log content; The log query module is used to transform log data into observable views by utilizing the maintained indexes and data file storage structure.

Images