Lightweight full-stack monitoring method and device

Abstract

The embodiment of the invention discloses a lightweight full-stack monitoring method and equipment, which are applied to a monitoring system based on OpenResty and a preset database framework, and the method comprises the following steps: executing a plurality of monitoring acquisition tasks based on a Lua coroutine mechanism of OpenResty to obtain multi-dimensional monitoring data; wherein the multi-dimensional monitoring data comprises system monitoring data, application monitoring data and network monitoring data; performing data standardization processing on the multi-dimensional monitoring data to obtain time sequence monitoring data, and writing the time sequence monitoring data into a preset database; the method comprises the following steps: receiving a monitoring data query request uploaded by a client based on an API (Application Program Interface) of OpenResty; analyzing the query request, and querying matched time sequence monitoring data in a preset database according to the time range of the request and the data type; and performing aggregation processing on the matched monitoring data so as to return an aggregation result to the client through the API interface.

Description

Technical Field

[0001] This manual relates to the field of monitoring systems, and in particular to a lightweight full-stack monitoring method and device. Background Technology

[0002] Despite the increasing prevalence of distributed and cloud-native architectures, a large number of business systems still run in non-distributed environments such as single-server or dual-machine hot standby. Furthermore, with the widespread adoption of cloud computing, containerization, and microservice architectures, the complexity of modern IT infrastructure is growing exponentially, bringing unprecedented challenges to the field of system monitoring. Not only has the number of monitoring points increased dramatically from dozens to thousands or even tens of thousands, but the dependencies between services have also become increasingly complex. Simultaneously, application deployment environments are exhibiting significant diversification, with physical machines, virtual machines, containers, hybrid clouds, and edge computing nodes all becoming common deployment carriers. This places extremely high demands on the cross-environment adaptability of monitoring solutions. In resource-constrained scenarios such as edge computing and IoT devices, the monitoring system itself must have extremely low resource overhead to avoid competing with the monitored business for valuable computing, memory, and network resources. In addition, to achieve refined operation and maintenance, the granularity and frequency of monitoring data collection are constantly increasing, and the resulting massive amounts of time-series data pose a significant challenge to the write pressure, storage costs, and query performance of storage systems.

[0003] In addressing these issues, traditional heavyweight monitoring systems such as Zabbix and Nagios, while powerful, are complex to deploy and resource-intensive. Lightweight monitoring tools like Prometheus, while simple to deploy, typically require the use of multiple other tools to achieve comprehensive monitoring.

[0004] Therefore, a lightweight yet feature-rich monitoring solution is needed. Summary of the Invention

[0005] To address the aforementioned technical problems, this specification provides one or more embodiments of a lightweight full-stack monitoring method, device, and medium.

[0006] One or more embodiments of this specification employ the following technical solutions: This specification provides one or more embodiments of a lightweight full-stack monitoring method, the method including: Based on OpenResty's Lua coroutine mechanism, multiple monitoring and data collection tasks are executed to obtain multi-dimensional monitoring data; wherein, the multi-dimensional monitoring data includes, but is not limited to: system monitoring data, application monitoring data, and network monitoring data; The multidimensional monitoring data is subjected to data standardization processing to obtain time-series monitoring data, which is then written into the preset database. Based on the OpenResty API interface, it receives monitoring data query requests uploaded by the client; Parse the query request and query the time-series monitoring data that matches the data type in the preset database according to the time range and data type of the request; The matching monitoring data is aggregated, and the aggregation result is returned to the client via the API interface.

[0007] Optionally, in one or more embodiments of this specification, multiple monitoring and data acquisition tasks are executed based on OpenResty's Lua coroutine mechanism to obtain multi-dimensional monitoring data, specifically including: Lua coroutines are created based on the OpenResty timer; Based on the number of tasks for each type of monitoring and data collection task and the current monitoring requirements, the current task execution mode is determined; wherein, the task execution mode includes: serial mode and parallel mode; The collection objects corresponding to each type of monitoring and collection task are obtained. Based on the current task execution mode, the various types of monitoring and collection tasks are executed in the OpenResty Worker process through Lua coroutines to obtain multi-dimensional monitoring data corresponding to each collection object.

[0008] Optionally, in one or more embodiments of this specification, the collection objects corresponding to each type of monitoring and collection task are obtained, and based on the current task execution mode, the various types of monitoring and collection tasks are executed in the OpenResty Worker process through Lua coroutines to obtain multi-dimensional monitoring data corresponding to each collection object, specifically including: If the task execution mode is determined to be serial mode, then in a single Worker process of OpenResty, various types of monitoring and acquisition tasks are executed serially through Lua coroutines to obtain multi-dimensional monitoring data corresponding to each acquisition object. If the task execution mode is determined to be parallel mode, then based on the number of tasks of each type of monitoring and collection task and the current monitoring requirements, multiple Worker processes corresponding to the monitoring and collection task are determined. In the multiple Worker processes, various types of monitoring and acquisition tasks are executed in parallel through Lua coroutines to obtain multi-dimensional monitoring data corresponding to each acquisition object.

[0009] Optionally, in one or more embodiments of this specification, after executing multiple monitoring and data collection tasks based on OpenResty's Lua coroutine mechanism, the method further includes: Obtain the task execution time corresponding to each type of monitoring and data collection task, and compare the task execution time with the preset timeout threshold; If it is determined that the execution time of the task exceeds the preset timeout threshold, the monitoring and data collection task is stopped and an anomaly is marked.

[0010] Optionally, in one or more embodiments of this specification, before concurrently executing multiple monitoring and data acquisition tasks based on OpenResty's Lua coroutine mechanism to obtain multidimensional monitoring data, the method further includes: Based on the preset configuration information, the execution strategy of the monitoring and data collection task is determined and initialized; wherein, the configuration information includes at least: monitoring period, preset timeout threshold, and task execution mode; The connection pool between the OpenResty Worker process and the preset database is initialized, and a time-series data storage structure corresponding to the monitored data type is created in the preset database.

[0011] Optionally, in one or more embodiments of this specification, the multidimensional monitoring data is subjected to data standardization processing to obtain time-series monitoring data, and then written into the preset database, specifically including: Data extraction is performed on the multidimensional monitoring data to obtain key monitoring data from the multidimensional monitoring data; Determine the monitoring object and timestamp format corresponding to the key monitoring data, and perform time alignment on the key monitoring data according to the monitoring object and the timestamp format to obtain the time-series key monitoring data corresponding to the multi-dimensional monitoring data; The time-series key monitoring data is normalized and cleaned to obtain standardized time-series key monitoring data. The time-series key monitoring data is encapsulated based on a pre-defined data structure to obtain encapsulated time-series monitoring data; wherein, the time-series monitoring data has a timestamp tag and a data type tag; The time-series monitoring data is written into the preset database in key-value pair format; wherein, the key name of the key-value pair format is generated by combining the data name of the time-series monitoring data with the timestamp tag, and the key value is the corresponding time-series monitoring data.

[0012] Optionally, in one or more embodiments of this specification, the preset database includes at least: a first data layer and a second data layer; the first data layer is used to store high-frequency time-series monitoring data within a specified time range, and the second data layer is used to store long-term time-series monitoring data exceeding the specified time range; Specifically, writing to the preset database includes: The real-time collected time-series monitoring data is synchronously written to the first data layer; By using a preset aggregation time window, the time-series monitoring data in the first data layer is downsampled and aggregated to generate statistical feature data for each preset aggregation time window; The statistical feature data is written into an aggregated data sequence to store the aggregated data sequence in the second data layer, and the time-series monitoring data in the first data layer that exceeds the specified time range is deleted according to the specified time range.

[0013] Optionally, in one or more embodiments of this specification, parsing the query request and querying the time-series monitoring data matching the preset database according to the time range and data type of the request specifically includes: Parse the query request to obtain the target information of the query request; wherein, the target information includes: the time range of the request, the data type, and the data aggregation granularity; Based on the data aggregation granularity, the corresponding aggregated data sequence in the preset database is determined; based on the time range and data type of the request, it is matched with the corresponding aggregated data sequence to obtain matching time-series monitoring data.

[0014] Optionally, in one or more embodiments of this specification, the matching monitoring data is aggregated to return the aggregation result to the client via the API interface, specifically including: Update the aggregation time window of the matching time-series monitoring data according to the data aggregation granularity; Based on the updated aggregation time window, the matching time-series monitoring data is subjected to secondary aggregation processing to obtain the aggregation result, which is then returned to the client via the API interface.

[0015] This specification provides one or more embodiments of a lightweight full-stack monitoring device, the device comprising: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described above.

[0016] This specification provides one or more embodiments of a non-volatile computer storage medium storing computer-executable instructions, the computer-executable instructions being configured to execute any of the methods described above.

[0017] The above-described at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects: By leveraging OpenResty's Lua coroutine mechanism, multi-dimensional monitoring and data collection tasks can be executed with extremely low resource overhead. The lightweight nature of coroutines avoids the complex deployment and high resource consumption issues of traditional heavyweight monitoring systems such as Zabbix and Nagios. By standardizing multi-dimensional monitoring data and transforming it into structured time-series data, which is then written to a pre-built database, the problem of heterogeneous data formats is solved, and the speed of data writing and querying is greatly improved. Based on OpenResty's API interface, query requests are received and parsed, interacting directly with the high-performance pre-built database, enabling rapid responses to real-time data queries from clients. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 A flowchart illustrating a lightweight full-stack monitoring method provided in the embodiments of this specification; Figure 2 An overall framework diagram of a lightweight full-stack monitoring method provided in the embodiments of this specification; Figure 3 This is a schematic diagram of the structure of a lightweight full-stack monitoring device provided in the embodiments of this specification. Detailed Implementation

[0019] This specification provides a lightweight full-stack monitoring method and device through its embodiments.

[0020] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0021] like Figure 1 As shown, this specification provides a flowchart illustrating a lightweight full-stack monitoring method. Figure 2 As can be seen, in one or more embodiments of this specification, a lightweight full-stack monitoring method is applied to a monitoring system based on OpenResty and a pre-built database architecture, specifically including the following steps: S101: Based on the Lua coroutine mechanism of OpenResty, execute multiple monitoring and data collection tasks to obtain multi-dimensional monitoring data; wherein, the multi-dimensional monitoring data includes: system monitoring data, application monitoring data and network monitoring data.

[0022] OpenResty is a high-performance web platform based on the Nginx core extension. By bundling the Nginx core with numerous Lua libraries and third-party modules, it transforms Nginx from a simple HTTP and reverse proxy server into a powerful general-purpose web application server. In traditional monitoring data collection scenarios, serial collection often leads to excessively long total processing time or insufficient data timeliness, while multi-threaded collection suffers from high thread switching overhead and high resource consumption. To address these issues, this specification's embodiments utilize OpenResty's lightweight Lua coroutine concurrency mechanism. It encapsulates the collection tasks for system monitoring data, application monitoring data, and network monitoring data into independent coroutines. Leveraging OpenResty's inherited Nginx event-driven and non-blocking I / O model, it achieves efficient scheduling of multiple coroutines within a single process. Specifically, when a coroutine performs an I / O operation, it voluntarily relinquishes execution, allowing the event loop mechanism to schedule other ready coroutines to run, avoiding unnecessary waiting and resource waste. This process leverages the lock-free concurrency of Lua coroutines and the high-performance foundation of OpenResty to reduce the overall acquisition latency while ensuring the integrity of multi-dimensional monitoring data collection. It also supports horizontal scaling of the number of acquisition tasks without significantly increasing resource overhead, ultimately achieving real-time acquisition of multi-dimensional monitoring data in a lightweight architecture. For example, Figure 2 The monitoring and data collection tasks shown are executed based on the monitoring and data collection layer, and have the functions of system monitoring, application monitoring, and network monitoring. Specifically, system monitoring collects the usage of system resources such as CPU, memory, disk, and network; application monitoring is used to monitor the status and resource usage of Docker containers and processes; and network monitoring is used to monitor the online status of IP devices, supporting both HTTP and MQTT protocol detection.

[0023] Furthermore, in one or more embodiments of this specification, before concurrently executing multiple monitoring and data acquisition tasks based on OpenResty's Lua coroutine mechanism to obtain multidimensional monitoring data, the method further includes: First, the monitoring and data collection task execution strategy is initialized based on pre-configured information. This configuration information includes at least: monitoring period, pre-set timeout threshold, and task execution mode. The collection frequency is set to avoid excessively frequent collection leading to resource waste or excessively long collection intervals resulting in data loss. The pre-set timeout threshold sets the maximum execution time for each collection task, preventing individual tasks from getting stuck indefinitely due to blocking or anomalies, thus affecting the stability of the overall collection process. The task execution mode defines the core rules of coroutine scheduling to ensure that the execution of collection tasks meets business requirements. To ensure stable storage of monitoring data after collection, the connection pool between the OpenResty Worker process and the pre-set database needs to be initialized. This pre-set database can be a relational database or a non-relational database. Then, by reusing established database connections through the connection pool, the network overhead and performance loss caused by repeatedly creating and destroying connections after each collection task are avoided, improving the efficiency of writing data to the pre-set database and reducing the pressure on the pre-set database server caused by frequent connection establishment. Furthermore, considering the characteristics of multi-dimensional monitoring data, a corresponding time-series data storage structure is pre-created in the pre-set database, facilitating subsequent ordered storage based on the timestamps of the monitoring data.

[0024] Specifically, in one or more embodiments of this specification, multiple monitoring and data acquisition tasks are executed based on OpenResty's Lua coroutine mechanism to obtain multi-dimensional monitoring data, including: First, Lua coroutines are created based on OpenResty's timer functionality to serve as lightweight execution units for data acquisition tasks. Then, the current task execution mode is dynamically determined based on the quantity and scale of each type of monitoring task and the current monitoring requirements. These execution modes include a serial mode suitable for light loads or low-priority scenarios, and a parallel mode suitable for high loads or high real-time requirements. Next, the specific acquisition objects corresponding to each type of monitoring task are identified and obtained. Based on the determined task execution mode, Lua coroutines are scheduled within OpenResty's Worker process to efficiently execute these acquisition tasks. Ultimately, system monitoring data, application monitoring data, and network monitoring data from each acquisition object are successfully acquired, thus completing the collection of multi-dimensional monitoring data.

[0025] Specifically, in one or more embodiments of this specification, the collection objects corresponding to various types of monitoring and collection tasks are obtained, and based on the current task execution mode, various types of monitoring and collection tasks are executed in the OpenResty Worker process through Lua coroutines to obtain multi-dimensional monitoring data corresponding to each collection object, specifically including: If the task execution mode is determined to be serial, the system will sequentially execute various types of monitoring and acquisition tasks through Lua coroutines within a single Worker process in OpenResty, thereby acquiring system, application, and network monitoring data corresponding to each acquisition object in order. If the task execution mode is determined to be parallel, the system will further dynamically determine and allocate tasks to multiple Worker processes based on the number of acquisition tasks and the real-time requirements of the current monitoring. Subsequently, various acquisition tasks will be executed concurrently in these selected Worker processes through Lua coroutines, thereby acquiring multi-dimensional monitoring data corresponding to each acquisition object in parallel, thus significantly improving the overall efficiency and throughput of data acquisition.

[0026] Furthermore, in order to achieve timeout monitoring and automatic circuit breaking for data acquisition tasks, in one or more embodiments of this specification, after executing multiple monitoring and data acquisition tasks based on OpenResty's Lua coroutine mechanism, the method further includes the following process: The system obtains the execution time of each type of monitoring and data collection task and compares it with a preset timeout threshold. This preset timeout threshold is determined by an expert model based on the real-time requirements of the monitoring data, the normal response time of the data collection target, and the system's tolerance. If the task execution time exceeds the preset timeout threshold, a protection mechanism is immediately triggered. This mechanism includes stopping the monitoring and data collection task and marking it as abnormal. Stopping the task means the system actively interrupts its execution, immediately releasing the Lua coroutines, network connections, and other resources it occupies, preventing it from affecting other normal data collection tasks or even core business requests within the Worker process due to prolonged blocking. Marking the task as abnormal assigns a clear error identifier. This marker is crucial for subsequent data processing and system maintenance. It ensures that invalid data is not mistakenly reported and used as normal data, and also provides maintenance personnel with clear troubleshooting clues, enabling them to quickly locate performance bottlenecks or problematic data collection targets. This process solves the problem of data collection tasks being suspended for extended periods due to slow response from a single data collection target or network congestion, which in turn blocks the entire Lua coroutine scheduling and exhausts worker process resources. It also prevents a single data collection point's failure from propagating through the data collection chain, affecting the stability of the entire monitoring system and the collection of other normal indicators.

[0027] This specification's embodiments employ a low-resource-consumption design, using Lua coroutines to achieve non-blocking monitoring data acquisition. In a certain scenario, the main acquisition task currently runs on only one worker, but can be distributed to different workers as needed. All API tasks are executed and returned evenly by each worker. The design supports even distribution across workers in an odd-even distribution manner, and data is merged using a shared dictionary via OpenResty. In a feasible embodiment, the monitoring interval can be increased based on the number of workers. For example, if monitoring is performed every 60 seconds, worker1 executes in the first 60 seconds, worker2 executes in the second 60 seconds, i.e., interval-distributed monitoring, ensuring that time-consuming tasks are correctly included in the monitoring interval.

[0028] S102: Perform data standardization processing on the multidimensional monitoring data to obtain time-series monitoring data, and write it into the preset database.

[0029] After obtaining the multi-dimensional monitoring data as described above, the multi-dimensional monitoring data successfully acquired from each collection object will undergo data standardization processing to transform it into unified, storable time-series monitoring data, and then write it into a preset database. Specifically, in one or more embodiments of this specification, the multi-dimensional monitoring data is subjected to data standardization processing to obtain time-series monitoring data, and then written into a preset database, specifically including yix guoc: The raw data may contain redundant information, log headers, auxiliary descriptions, etc. To remove redundant content and retain only key data, data extraction is performed on the multi-dimensional monitoring data to obtain the key monitoring data. Since the data may come from different data sources, the precision of its timestamps and time zones may be inconsistent. Therefore, in this embodiment, the monitoring object and timestamp format corresponding to the key monitoring data are determined. Based on the monitoring object and timestamp format, the key monitoring data is time-aligned to ensure that all data points are assigned a unified and comparable time benchmark, thus obtaining the time-series key monitoring data corresponding to the multi-dimensional monitoring data. The time-series key monitoring data is normalized and cleaned to obtain standardized time-series key monitoring data. After obtaining the time-series key monitoring data, it is encapsulated according to a preset data structure to obtain encapsulated time-series monitoring data. This time-series monitoring data has a timestamp tag and a data type tag. The time-series monitoring data is written to a preset database in key-value pair format. The key name of the key-value pair is generated by combining the data name and timestamp tag of the time-series monitoring data, and the key value is the corresponding time-series monitoring data. By storing data in a key-value pair format, it is easier to retrieve all data points of a certain indicator within a specific time period based on key-value pair matching, thus improving query efficiency.

[0030] In one or more embodiments of this specification, to balance the need for high-performance access to the latest monitoring data with the need for cost-effective storage of massive amounts of historical data, the pre-configured database includes at least a first data layer and a second data layer. The first data layer stores high-frequency time-series monitoring data within a specified time range, enabling high-performance storage and ensuring low-latency read / write access to the latest data. The second data layer stores long-term time-series monitoring data beyond the specified time range, enabling date-based storage of historical monitoring data and supporting long-term retention. Specifically, writing to the pre-configured database includes the following processes: Real-time, fine-grained time-series monitoring data is synchronously written to the first data layer. Then, using preset aggregation time windows (e.g., 5 minutes, 1 hour), the time-series monitoring data in the first data layer is downsampled and aggregated to generate statistical characteristic data for each preset aggregation time window. This involves calculating the statistical characteristics of the original data within each time window, such as average, maximum, minimum, or sum, thus generating statistical characteristic data for each window. For example, if the first data layer stores CPU usage data per second, and the aggregation window is set to 5 minutes, the downsampling process will aggregate 300 original data points into one data point, whose value might be the average usage rate within those 5 minutes. The statistical characteristic data generated in the previous step is written into an aggregated data sequence, and this sequence is stored in the second data layer. Time-series monitoring data in the first data layer that exceeds a specified time range (e.g., 24 hours) is deleted based on this specified time range. By deleting expired original data, the system automatically reclaims storage space, preventing the first data layer from expanding indefinitely and ensuring it maintains high performance. The second data layer is used to store long-term historical data after downsampling, supporting offline queries that do not have high real-time requirements, such as trend analysis and capacity planning.

[0031] In a certain application scenario, a pre-configured database data storage layer is used, featuring two data layers: one for storing monitoring data for a recent time period, and the other for archiving data by date. During the operation of the monitoring system, such as... Figure 2 The diagram shows a timed task scheduling acquisition layer and a data processing layer. The acquisition layer collects data in memory, and the data processing layer aggregates the data into a pre-set database. When an API call is made, data from different layers is returned based on the API call.

[0032] S103: Based on the OpenResty API interface, it receives monitoring data query requests uploaded by the client.

[0033] like Figure 2The OpenResty API interface shown provides monitoring, health, and file APIs. The monitoring API provides a query interface for system and application monitoring data; the health API provides a query interface for health check status and history; and the file API provides a file management and ADB device management interface. Based on this OpenResty API interface, it can receive monitoring data query requests uploaded by clients, enabling various monitoring data query and statistical functions based on the data processing layer's data query capabilities.

[0034] S104: Parse the query request and query the time-series monitoring data that matches the data type in the preset database according to the time range and data type of the request.

[0035] After obtaining the query request uploaded via the API interface based on the above steps, the query request is parsed, and time-series monitoring data matching the requested time range and data type is retrieved from the preset database. Specifically, in one or more embodiments of this specification, parsing the query request and retrieving matching time-series monitoring data from the preset database based on the requested time range and data type includes the following steps: The query request is parsed using a structured parsing approach based on OpenResty's Lua scripts to obtain the target information. This target information includes the requested time range, data type, and data aggregation granularity. Based on the user-defined aggregation granularity, the target data level stored in the pre-configured database is located, resolving the compatibility issue between multi-precision storage and efficient querying of time-series data. Since the monitoring data in the pre-configured database employs a multi-level aggregation storage design, with different aggregation granularities corresponding to independent time-series data sequences, the corresponding aggregated data sequence in the pre-configured database is determined based on the data aggregation granularity of the target information. To accurately filter time-series data that meets the criteria from the target aggregated data sequence, the requested time range and data type in the target information are matched with the corresponding aggregated data sequence to obtain the matching time-series monitoring data.

[0036] S105: Aggregate the matching monitoring data to return the aggregation result to the client via the API interface.

[0037] Monitoring data is characterized by high-frequency collection and time-series density. If multiple indicators are queried, the data volume will increase exponentially. Directly returning the matching monitoring data to the client would result in high network transmission pressure and wasted storage and computing resources. Therefore, this specification's embodiments perform targeted aggregation processing on the time-series monitoring data matched from a pre-set database. Specifically, in one or more embodiments of this specification, the matching monitoring data is aggregated to return the aggregation results to the client via an API interface, including the following process: First, based on the data aggregation granularity in the query request, the aggregation time window of the matching time-series monitoring data is updated. Then, based on the updated aggregation time window, a second aggregation process is performed on the matching time-series monitoring data to obtain the aggregation result, which is then returned to the client via the API interface. This aggregation process resolves the problem of raw data redundancy, reducing system transmission and computational overhead, ensuring service performance, and directly delivering the query results to the user.

[0038] like Figure 3 As shown in the diagram, this specification provides a schematic diagram of the structure of a lightweight full-stack monitoring device. Figure 3 It can be seen that a lightweight full-stack monitoring device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described above.

[0039] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and non-volatile computer storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0040] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0041] The above description is merely one or more embodiments of this specification and is not intended to limit this specification. Various modifications and variations can be made to the one or more embodiments of this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of one or more embodiments of this specification should be included within the scope of the claims of this specification.

Claims

1. A lightweight full-stack monitoring method, characterized in that, The method is applied to a monitoring system based on an OpenResty and a preset database framework, and the method comprises the following steps: Based on the Lua coroutine mechanism of OpenResty, multiple monitoring collection tasks are executed to obtain multidimensional monitoring data; wherein the multidimensional monitoring data includes but is not limited to: system monitoring data, application monitoring data and network monitoring data; The multidimensional monitoring data is subjected to data standardization processing to obtain time series monitoring data, and the time series monitoring data is written into the preset database; Based on the API interface of OpenResty, a monitoring data query request uploaded by a client is received; The query request is parsed, and time series monitoring data matching the request in the preset database is queried according to the time range and data type of the request; The matching monitoring data is subjected to aggregation processing, and the aggregation result is returned to the client through the API interface.

2. The lightweight full-stack monitoring method of claim 1, wherein, Based on the Lua coroutine mechanism of OpenResty, multiple monitoring collection tasks are executed to obtain multidimensional monitoring data, specifically including: Based on the timer of the OpenResty, a Lua coroutine is created; Based on the number of tasks of each type of monitoring collection task and the current monitoring demand, the current task execution mode is determined; wherein the task execution mode includes: serial mode, parallel mode; Each type of monitoring collection task is executed in the Worker process of the OpenResty based on the current task execution mode through the Lua coroutine, and multidimensional monitoring data corresponding to each collection object is obtained.

3. The lightweight full-stack monitoring method of claim 2, wherein, Each type of monitoring collection task is executed in the Worker process of the OpenResty based on the current task execution mode through the Lua coroutine, and multidimensional monitoring data corresponding to each collection object is obtained, specifically including: If it is determined that the task execution mode is serial mode, then each type of monitoring collection task is executed in a single Worker process of the OpenResty through a Lua coroutine, and multidimensional monitoring data corresponding to each collection object is obtained; If it is determined that the task execution mode is parallel mode, then based on the number of tasks of each type of monitoring collection task and the current monitoring demand, a plurality of Worker processes corresponding to the monitoring collection task are determined; In the plurality of Worker processes, each type of monitoring collection task is executed in parallel through a Lua coroutine, and multidimensional monitoring data corresponding to each collection object is obtained.

4. The lightweight full-stack monitoring method of claim 1, wherein, Based on the Lua coroutine mechanism of OpenResty, after executing multiple monitoring collection tasks, the method further comprises the following steps: The task execution time corresponding to each type of monitoring collection task is obtained, and the task execution time is compared with a preset timeout threshold; If it is determined that the task execution time exceeds the preset timeout threshold, then the monitoring collection task is stopped and an exception is marked.

5. The lightweight full-stack monitoring method of claim 1, wherein, Based on the Lua coroutine mechanism of OpenResty, before concurrently executing multiple monitoring collection tasks to obtain multidimensional monitoring data, the method further comprises the following steps: Based on the preset configuration information, the execution strategy of the monitoring collection task is determined to be initialized; wherein the configuration information at least includes: monitoring period, preset timeout threshold, task execution mode; The connection pool of the OpenResty Worker process and the preset database is initialized, and a time series data storage structure corresponding to the monitoring data type is created in the preset database.

6. The lightweight full-stack monitoring method of claim 1, wherein, The multi-dimensional monitoring data is subjected to data standardization processing to obtain time series monitoring data, and is written into the preset database, specifically including: The multi-dimensional monitoring data is subjected to data extraction to obtain key monitoring data of the multi-dimensional monitoring data; The monitoring object and timestamp format corresponding to the key monitoring data are determined, and the key monitoring data is subjected to time alignment according to the monitoring object and the timestamp format, to obtain time series key monitoring data corresponding to the multi-dimensional monitoring data; The time series key monitoring data is subjected to normalization processing and data cleaning to obtain standardized time series key monitoring data; The time series key monitoring data is encapsulated based on a preset data structure to obtain encapsulated time series monitoring data; wherein the time series monitoring data has a timestamp label and a data type label; The time series monitoring data is written into the preset database in a key-value pair format; wherein the key name of the key-value pair format is generated by combining the data name and the timestamp label of the time series monitoring data, and the key value is the corresponding time series monitoring data.

7. The lightweight full-stack monitoring method of claim 1, wherein, The preset database at least includes: a first data layer and a second data layer; the first data layer is used to store high-frequency time series monitoring data within a specified time range, and the second data layer is used to store long-term time series monitoring data beyond the specified time range; Wherein, the writing into the preset database specifically includes: The real-time collected time series monitoring data is synchronously written into the first data layer; The time series monitoring data in the first data layer is subjected to down-sampling aggregation processing through a preset aggregation time window to generate statistical feature data of each preset aggregation time window; The statistical feature data is written into an aggregated data sequence to store the aggregated data sequence into the second data layer, and the time series monitoring data of the first data layer beyond the specified time range is deleted according to the specified time range.

8. The lightweight full-stack monitoring method of claim 7, wherein, The query request is parsed, and the matching time series monitoring data in the preset database is queried according to the requested time range and data type, specifically including: The query request is parsed to obtain target information of the query request; wherein the target information includes: requested time range, data type, data aggregation granularity; According to the data aggregation granularity, the corresponding aggregated data sequence in the preset database is determined; based on the requested time range and data type, the corresponding aggregated data sequence is matched to obtain matching time series monitoring data.

9. The lightweight full-stack monitoring method of claim 8, wherein, The matching monitoring data is subjected to aggregation processing to return the aggregation result to the client through the API interface, specifically including: According to the data aggregation granularity, the aggregation time window of the matching time series monitoring data is updated; According to the updated aggregation time window, the matched time sequence monitoring data is subjected to secondary aggregation processing, and an aggregation result is obtained, so as to return the aggregation result to the client through the API interface.

10. A lightweight full-stack monitoring device, comprising: at least one processor; and, a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

Images