Service monitoring method, system and device based on VM and storage medium

Abstract

The embodiment of the invention discloses a business monitoring method, system and equipment based on a VM (Virtual Machine) and a storage medium. According to the embodiment of the invention, the operation complexity is reduced by eliminating Elasticsearch and kafka, the storage query efficiency is improved by a VictoriaMetric database, and the learning, operation and maintenance workload of personnel is effectively reduced. High-throughput data writing and query can be quickly processed, millions of data points can be processed per second, meanwhile, the storage space requirement is remarkably reduced, and the quick query performance is kept. And under the same hardware resource, compared with other monitoring, higher performance and lower cost are provided. The method is completely compatible with a query language of Prometheus, and a storage rear end can be directly replaced by Prometheus. The method has the advantages of high performance, economy, high efficiency, high expandability and high usability, and is suitable for large-scale monitoring and log recording scenes.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of business monitoring, and in particular to a VM-based business monitoring method, system, device and storage medium. Background Art

[0002] VictoriaMetrics (VM) is a high-performance, scalable monitoring solution and time series database that is particularly suitable for processing and storing large amounts of monitoring data.

[0003] In the field of business monitoring, with the rapid development of information technology, enterprises have a growing demand for real-time monitoring and analysis of business data. One of the current mainstream technical solutions is to use the ELK stack, which is a combination of Elasticsearch, Logstash, and Kibana. The ELK stack provides a powerful platform for data collection, storage, search, and visualization, but it also has major limitations:

[0004] 1) Configuration and management complexity of the ELK stack

[0005] Each component of the ELK stack needs to be carefully configured and managed, which not only increases the complexity of the technology stack, but also increases the learning cost and operating cost:

[0006] Complex setup and configuration: The setup and configuration process of Elasticsearch, Logstash, and Kibana involves multiple steps. Elasticsearch requires configuration of clusters, indexes, mappings, and setting the appropriate number of shards and replicas, while Logstash requires configuration of input sources, filters, and output destinations. Kibana requires configuration of connections to Elasticsearch as well as configuration of dashboards and visualization components.

[0007] Steep usage threshold and learning curve: The learning curve of the ELK stack is very steep, and you need to master the Elasticsearch query language (such as Elasticsearch's DSL) to perform complex searches and analyses.

[0008] High operation and maintenance costs: As the business grows, the maintenance and expansion of the ELK stack becomes more complex and requires a professional operation and maintenance team to manage. For example, when the Elasticsearch cluster needs to be expanded, it may be necessary to reconfigure shards and replicas, as well as deal with data migration and other issues.

[0009] 2) Elasticsearch memory usage and server costs

[0010] Elasticsearch is written in Java, and Java applications usually require more memory to run. This means that in order to run Elasticsearch, a server with a higher memory configuration is required, which increases the cost.

[0011] High memory usage: Elasticsearch consumes a lot of memory resources when processing large amounts of data. This not only limits the amount of data that a single server can process, but also increases the demand for high-performance servers.

[0012] The cost of purchasing servers is high: Due to Elasticsearch's memory requirements, purchasing or renting servers that can meet its operating requirements can be very expensive, increasing the company's IT expenses.

[0013] 3) Kafka and Elasticsearch have low storage compression efficiency

[0014] Kafka and Elasticsearch may not use efficient compression algorithms when storing data, which can lead to increased storage costs.

[0015] Low storage compression efficiency: Kafka and Elasticsearch do not use efficient compression algorithms when storing data, resulting in increased storage costs.

[0016] The cost of storage is high: Due to low storage efficiency, enterprises may need to purchase more storage resources to save data, which will increase storage costs. In addition, more storage resources also mean higher energy consumption and maintenance costs.

[0017] Therefore, it has become an urgent need to invent a business monitoring method that can reduce the complexity of the technology stack, reduce the demand for memory and storage, and thus reduce the overall operating costs. Summary of the invention

[0018] To this end, the embodiments of the present invention provide a VM-based business monitoring method, system, device and storage medium to solve the technical problems of complex technology stack, high learning cost, extremely high memory occupancy and low storage compression efficiency when the prior art performs business monitoring through the ELK stack.

[0019] In order to achieve the above purpose, the embodiment of the present invention provides the following technical solutions:

[0020] According to a first aspect of an embodiment of the present invention, a VM-based business monitoring method is provided, the method being applied to VictoriaMetrics and comprising:

[0021] S1. Obtain a large amount of time series data, perform business tracking on the time series data and store them in VictoriaMetrics;

[0022] S2. Build an HTTP API interface to capture and collect cookie information and header information in HTTP requests, and store the cookie information and header information in VictoriaMetrics;

[0023] S3. Use the Go language to parse the data stored in VictoriaMetrics and the time series data after the new business is embedded, and compare the data change fields and change tables as well as the change time and change primary key;

[0024] S4. Obtain business monitoring requirements and use Grafana to write and execute PromQL query statements to analyze and visualize time series data stored in VictoriaMetrics.

[0025] Furthermore, a large amount of time series data is obtained, and business tracking is performed on the time series data and stored in VictoriaMetrics, including:

[0026] The time series data is processed using the indicator storage and query mechanism of VictoriaMetrics.

[0027] Furthermore, an HTTP API interface is built to capture and collect cookie information and header information in HTTP requests, including:

[0028] defining at least one API endpoint and determining access rights for the API endpoint, and receiving HTTP requests using the API endpoint;

[0029] Analyze the received HTTP request and extract the cookie and header information in the request.

[0030] Furthermore, the Go language is used to parse the data stored in VictoriaMetrics and the time series data after the new business tracking, and compare the data change fields and change tables as well as the change time and change primary key, including:

[0031] Use VictoriaMetrics' query API to retrieve stored time series data;

[0032] Compare the time series data after the new business is embedded with the stored data to detect data changes;

[0033] Through comparative analysis, the changed data fields are identified and a change table is generated for all the changed data fields;

[0034] The change time and the change primary key are obtained from the data change field.

[0035] Furthermore, we obtain business monitoring requirements and use Grafana to write and execute PromQL query statements to analyze and visualize the time series data stored in VictoriaMetrics, including:

[0036] Connect the Grafana dashboard to the VictoriaMetrics database to monitor and analyze data in real time.

[0037] According to a second aspect of an embodiment of the present invention, a VM-based service monitoring system is provided, the system comprising:

[0038] The business tracking module is used to obtain a large amount of time series data, perform business tracking on the time series data and store it in VictoriaMetrics;

[0039] An information capture module is used to build an HTTP API interface to capture and collect cookie information and header information in HTTP requests, and store the cookie information and header information in VictoriaMetrics;

[0040] The parsing module is used to use the Go language to parse the data stored in VictoriaMetrics and the time series data after the new business is embedded, and compare the data change fields and change tables as well as the change time and change primary key;

[0041] The monitoring module is used to obtain business monitoring requirements and use Grafana to write and execute PromQL query statements to analyze and visualize time series data stored in VictoriaMetrics.

[0042] According to a third aspect of an embodiment of the present invention, there is provided a VM-based service monitoring device, the device comprising: a processor and a memory;

[0043] The memory is used to store one or more program instructions;

[0044] The processor is used to run one or more program instructions to execute the steps of a VM-based business monitoring method as described in any one of the above items.

[0045] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of a VM-based business monitoring method as described in any one of the above items are implemented.

[0046] The embodiments of the present invention have the following advantages:

[0047] The embodiment of the present invention reduces the complexity of operation by eliminating Elasticsearch and kafka. The VictoriaMetrics database improves storage query efficiency and effectively reduces the workload of personnel learning and operation and maintenance. It can quickly handle high-throughput data writing and query, and can process millions of data points per second while significantly reducing storage space requirements and maintaining fast query performance. With the same hardware resources, it provides higher performance and lower cost than other monitoring. It is fully compatible with the query language of Prometheus, and the storage backend can be directly replaced with Prometheus. It has the advantages of high performance, cost-effectiveness, scalability and ease of use, and is suitable for large-scale monitoring and logging scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0049] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0050] Figure 1 A schematic diagram of the logical structure of a VM-based business monitoring system provided by an embodiment of the present invention;

[0051] Figure 2 A flow chart of a VM-based service monitoring method provided by an embodiment of the present invention;

[0052] Figure 3 A data flow diagram of a VM-based business monitoring method provided by an embodiment of the present invention;

[0053] Figure 4 A schematic diagram of the broadcast and multicast process in a VM-based service monitoring method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] In the field of business monitoring, with the rapid development of information technology, enterprises have a growing demand for real-time monitoring and analysis of business data. One of the current mainstream technical solutions is to use the ELK stack, which is a combination of Elasticsearch, Logstash, and Kibana. The ELK stack provides a powerful platform for data collection, storage, search, and visualization, but it also has major limitations.

[0056] Therefore, it has become an urgent need to invent a business monitoring method that can reduce the complexity of the technology stack, reduce the demand for memory and storage, and thus reduce the overall operating costs.

[0057] In order to solve the technical problems that the above-mentioned existing technology uses the ELK stack to monitor business, such as complex technology stack, high learning cost, extremely high memory usage and low storage compression efficiency.

[0058] refer to Figure 1 The embodiment of the present invention discloses a VM-based business monitoring system, which includes: a business tracking module 1; an information capturing module 2; a parsing module 3; and a monitoring module 4.

[0059] Corresponding to the above disclosed VM-based service monitoring system, the embodiment of the present invention further discloses a VM-based service monitoring method. The following describes in detail the VM-based service monitoring method disclosed in the embodiment of the present invention in combination with the above described VM-based service monitoring system.

[0060] refer to Figure 2 and Figure 3The present invention discloses a VM-based business monitoring method, which is applied to VictoriaMetrics and includes: S1, obtaining a large amount of time series data, performing business tracking on the time series data and storing them in VictoriaMetrics; S2, building an HTTP API interface to capture and collect cookie information and header information in HTTP requests, and storing the cookie information and header information in VictoriaMetrics; S3, using go language to parse the data stored in VictoriaMetrics and the time series data after new business tracking, and compare the data change fields and change tables as well as the change time and change primary key; S4, obtaining business monitoring requirements and using Grafana to write and execute PromQL query statements to analyze and visualize the time series data stored in VictoriaMetrics.

[0061] VictoriaMetrics is an open source database specifically designed for monitoring and alerting. It processes large-scale time series data through efficient metric storage and query mechanisms. Although VictoriaMetrics introduces a new technology stack when used in conjunction with Grafana, it can achieve a non-invasive, flexible, and multi-cluster monitoring solution.

[0062] VictoriaMetrics is a high-performance, cost-effective and scalable time series database (TSDB) widely used in monitoring and logging scenarios.

[0063] Furthermore, a large amount of time series data is obtained, and business points are tracked for the time series data and stored in VictoriaMetrics, including: using the indicator storage and query mechanism of VictoriaMetrics to process the time series data.

[0064] Furthermore, an HTTP API interface is constructed to capture and collect cookie information and header information in HTTP requests, including: defining at least one API endpoint and determining the access rights of the API endpoint, and receiving HTTP requests using the API endpoint; analyzing the received HTTP requests and extracting the cookie and header information in the requests.

[0065] Use HTTP API as a probe to read cookies and headers and store them in VictoriaMetrics.

[0066] Furthermore, the Go language is used to parse the stored data in VictoriaMetrics and the time series data after the new business is embedded, and compare the data change fields and change tables as well as the change time and change primary keys, including: using VictoriaMetrics' query API to obtain the stored time series data; comparing the time series data after the new business is embedded with the stored data to detect data changes; identifying the changed data change fields through comparative analysis and generating a change table for all data change fields; and obtaining the change time and change primary key from the data change fields.

[0067] Furthermore, business monitoring requirements are obtained and PromQL query statements are written and executed using Grafana to analyze and visualize the time series data stored in VictoriaMetrics, including: connecting to the VictoriaMetrics database through the Grafana dashboard to monitor and analyze the data in real time.

[0068] Grafana connects to VictoriaMetrics which is compatible with Prometheus to write analysis PromQL.

[0069] VictoriaMetrics is fully compatible with Prometheus's query language PromQL and can directly replace Prometheus as the storage backend without modifying existing business monitoring.

[0070] refer to Figure 4 , deploy monitoring probes on the front end to collect business data, process the collected data through the back-end module; store the processed data in the VictoriaMetrics database; connect to the VictoriaMetrics database through the Grafana dashboard to monitor and analyze the data in real time.

[0071] The front end includes obtaining business monitoring requirements, probe point placement, solving CORS problems and using products;

[0072] The backend includes lead tracking, extracting fields, and coordinating HTTP headers;

[0073] Storage includes Docker deployment, VM deployment, storage to VM, pre-compilation of VM statements, custom storage rules, and service termination.

[0074] The analysis includes docker deployment, grafana deployment, grafana linking to VM, analysis in conjunction with VM syntax, and regression business monitoring requirements.

[0075] The embodiments of the present invention have the following advantages:

[0076] 1) Designed to handle high throughput data writing and querying, capable of processing millions of data points per second;

[0077] 2) Use advanced compression algorithms to significantly reduce storage space requirements while maintaining fast query performance;

[0078] 3) Provides higher performance and lower cost than other monitoring systems with the same hardware resources;

[0079] 4) It is fully compatible with Prometheus' query language (PromQL) and can directly replace Prometheus as the storage backend without modifying the existing monitoring system;

[0080] 5) High performance, cost-effectiveness, scalability, and ease of use make it an ideal choice for time series data storage and query, especially suitable for large-scale monitoring and logging scenarios.

[0081] In addition, an embodiment of the present invention also provides a VM-based business monitoring device, which includes: a processor and a memory; the memory is used to store one or more program instructions; the processor is used to run one or more program instructions to execute the steps of a VM-based business monitoring method as described in any of the above items.

[0082] In addition, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of a VM-based business monitoring method as described in any of the above items are implemented.

[0083] In the embodiment of the present invention, the processor may be an integrated circuit chip having the signal processing capability. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0084] The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The processor reads the information in the storage medium and completes the steps of the above method in combination with its hardware.

[0085] The storage medium may be a memory, which may be, for example, a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memory.

[0086] Among them, the non-volatile memory can be a read-only memory (ROM), a programmable ROM (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory.

[0087] The volatile memory may be a random access memory (RAM) which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM).

[0088] The storage media described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0089] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the present invention can be implemented using a combination of hardware and software. When software is used, the corresponding functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. Storage media can be any available media that can be accessed by general or special-purpose computers.

[0090] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A VM-based business monitoring method, characterized in that: The method is applied to VictoriaMetrics and includes: S1. Obtain a large amount of time series data, perform business tracking on the time series data and store them in VictoriaMetrics; S2. Build an HTTP API interface to capture and collect cookie information and header information in HTTP requests, and store the cookie information and header information in VictoriaMetrics; S3. Use the Go language to parse the data stored in VictoriaMetrics and the time series data after the new business is embedded, and compare the data change fields and change tables as well as the change time and change primary key; S4. Obtain business monitoring requirements and use Grafana to write and execute PromQL query statements to analyze and visualize time series data stored in VictoriaMetrics.

2. A VM-based business monitoring method as claimed in claim 1, characterized in that: Obtain a large amount of time series data, perform business tracking on the time series data and store it in VictoriaMetrics, including: The time series data is processed using the indicator storage and query mechanism of VictoriaMetrics.

3. A VM-based business monitoring method as claimed in claim 2, characterized in that: Build an HTTP API interface to capture and collect cookie information and header information in HTTP requests, including: defining at least one API endpoint and determining access rights for the API endpoint, and receiving HTTP requests using the API endpoint; Analyze the received HTTP request and extract the cookie and header information in the request.

4. A VM-based business monitoring method as claimed in claim 3, characterized in that: Use the Go language to parse the data stored in VictoriaMetrics and the time series data after the new business tracking, and compare the data change fields and change tables as well as the change time and change primary key, including: Use VictoriaMetrics' query API to retrieve stored time series data; Compare the time series data after the new business tracking is completed with the stored data to detect data changes; Through comparative analysis, the changed data fields are identified and a change table is generated for all the changed data fields; The change time and the change primary key are obtained from the data change field.

5. A VM-based service monitoring method as claimed in claim 4, characterized in that: Obtain business monitoring requirements and use Grafana to write and execute PromQL query statements to analyze and visualize time series data stored in VictoriaMetrics, including: Connect the Grafana dashboard to the VictoriaMetrics database to monitor and analyze data in real time.

6. A VM-based business monitoring system, characterized in that: The system comprises: The business tracking module is used to obtain a large amount of time series data, perform business tracking on the time series data and store it in VictoriaMetrics; An information capture module is used to build an HTTP API interface to capture and collect cookie information and header information in HTTP requests, and store the cookie information and header information in VictoriaMetrics; The parsing module is used to use the Go language to parse the data stored in VictoriaMetrics and the time series data after the new business is embedded, and compare the data change fields and change tables as well as the change time and change primary key; The monitoring module is used to obtain business monitoring requirements and use Grafana to write and execute PromQL query statements to analyze and visualize time series data stored in VictoriaMetrics.

7. A VM-based business monitoring device, characterized in that: The device comprises: a processor and a memory; The memory is used to store one or more program instructions; The processor is used to run one or more program instructions to execute the steps of a VM-based business monitoring method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the VM-based service monitoring method as claimed in any one of claims 1 to 5 are implemented.

Images