Monitoring data processing method and device, equipment and storage medium
By using the Telegraf tool and standardized algorithms to process cloud product monitoring data, the problem of inconsistent monitoring data formats among multiple cloud providers was solved, achieving unified monitoring data processing and querying, and improving development efficiency.
Patent Information
- Application Number
- CN202310939524.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-27
- Publication Date
- 2025-01-28
AI Technical Summary
Existing technologies make it difficult to collect monitoring data for cloud products and to achieve unified monitoring of cloud products from multiple cloud providers.
The raw monitoring data is obtained using the data acquisition tool Telegraf. The data is then converted into adapted monitoring data with the same field names through an indicator adaptation table. The data is then calculated into standardized target monitoring data using a preset standardization algorithm and stored in a distributed database, providing a unified query interface.
It enables unified format conversion and querying of monitoring data from multiple cloud providers, improving development efficiency, saving development time, and providing a highly reusable monitoring data processing method.
Smart Images
Figure CN119377253A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of big data, and in particular to a monitoring data processing method, device, equipment and storage medium. Background Art
[0002] With the development of cloud computing, a variety of cloud products have emerged, including cloud server ecs, server load balancing slb, object storage oss, etc.
[0003] Traditional technologies for collecting monitoring data on device performance indicators are usually used to collect monitoring data on physical machines, but are difficult to collect monitoring data on cloud products.
[0004] Although some cloud vendors provide calling interfaces for cloud product monitoring data, the data formats of monitoring data of different cloud vendors are different, making it difficult to achieve unified monitoring of cloud products from multiple cloud vendors. Summary of the invention
[0005] The present application provides a monitoring data processing method, apparatus, device and storage medium to solve the problem that it is difficult to collect monitoring data of cloud products and difficult to achieve unified monitoring of cloud products of multiple cloud providers.
[0006] In a first aspect, the present application provides a monitoring data processing method, comprising:
[0007] Based on the collection tool telegraf, the original monitoring data of the monitored object is obtained; the monitored object is a cloud server, a physical server, a virtual machine, or a storage device:
[0008] Converting the original monitoring data into adapted monitoring data with the same field name according to the indicator adaptation table;
[0009] Calculating the adapted monitoring data according to a corresponding preset standardized algorithm so as to convert the adapted monitoring data into standardized target monitoring data;
[0010] Storing the target monitoring data in a distributed database;
[0011] In response to a data query request from a user terminal for a monitored object, target monitoring data of the monitored object is obtained from a distributed database, and the target monitoring data of the monitored object is sent to the user terminal.
[0012] Optionally, obtaining original monitoring data of the monitored object includes:
[0013] The collection tool telegraf configuration package is sent to the device corresponding to the monitored object, so that the device corresponding to the monitored object installs the collection tool telegraf, and the collection tool telegraf is used to collect the original monitoring data of the monitored object at a preset time interval, or the collection tool telegra is used to call the data interface of the monitored object to obtain the original monitoring data of the monitored object; the original monitoring data of the monitored object obtained by the collection tool telegraf is obtained.
[0014] Optionally, the collection tool telegraf sends the acquired original monitoring data of the monitored object to the message queue kafka cluster, and the acquiring the original monitoring data of the monitored object acquired by the collection tool telegraf includes: acquiring the original monitoring data of the monitored object acquired by the collection tool telegraf from the kafka cluster.
[0015] Optionally, converting the original monitoring data into adapted monitoring data with the same field name according to the indicator adaptation table includes:
[0016] Based on the identification of the monitored object and the type of original monitoring data, a corresponding adaptation strategy is determined from the indicator adaptation table; and according to the corresponding adaptation strategy, a field value corresponding to a preset field name is extracted to convert the original monitoring data into adapted monitoring data with the same field name.
[0017] Optionally, the calculating the adapted monitoring data according to a corresponding preset standardized algorithm so as to convert the adapted monitoring data into standardized target monitoring data includes:
[0018] The adapted monitoring data is converted into adapted monitoring data of preset standard units and formats; if it is determined that the monitoring data in the standard units and formats is a loss value, the big data processing engine is used to calculate the monitoring data in the standard units and formats according to the preset standardized algorithm corresponding to the loss value; if it is determined that the monitoring data in the standard units and formats is a cumulative value, the big data processing engine is used to calculate the monitoring data in the standard units and formats according to the preset standardized algorithm corresponding to the cumulative value.
[0019] Optionally, in response to a data query request of a user terminal for a monitored object, obtaining target monitoring data of the monitored object from a distributed database, and sending the target monitoring data of the monitored object to the user terminal includes:
[0020] Obtain the monitored object identifier and the target time range in the data query request; obtain the target monitoring data of the monitored object within the target time range from the distributed database based on the monitored object identifier and the target time range; send the target monitoring data of the monitored object within the target time range to the user terminal, so as to display the target monitoring data of the monitored object within the target time range in the form of a chart in the user terminal.
[0021] Optionally, after storing the target monitoring data in a distributed database, the method further includes:
[0022] In response to a data aggregation query request of a user terminal for at least one monitored object, target monitoring data of at least one monitored object is obtained from a distributed database; aggregation calculations are performed on the target monitoring data of each monitored object to obtain the aggregate calculation results of each monitored object; basic information corresponding to each monitored object is obtained from a relational database; the basic information corresponding to each monitored object and the aggregate calculation results of each monitored object are sent to the user terminal, so that the basic information corresponding to each monitored object and the aggregate calculation results of each monitored object are displayed in the user terminal in the form of a list.
[0023] In a second aspect, the present application provides a monitoring data processing device, comprising:
[0024] The acquisition module is used to acquire the original monitoring data of the monitored object based on the acquisition tool telegraf; the monitored object is a cloud server, a physical server, a virtual machine, or a storage device:
[0025] An adaptation module, used for converting the original monitoring data into adapted monitoring data with the same field name according to the indicator adaptation table;
[0026] A calculation module, used to calculate the adapted monitoring data according to a corresponding preset standardized algorithm, so as to convert the adapted monitoring data into standardized target monitoring data;
[0027] A storage module, used for storing the target monitoring data in a distributed database;
[0028] The query module is used to respond to a data query request of a monitored object from a user terminal, obtain the target monitoring data of the monitored object from a distributed database, and send the target monitoring data of the monitored object to the user terminal.
[0029] In a third aspect, the present application provides a data monitoring server, comprising: a processor, and a memory and a transceiver communicatively connected to the processor;
[0030] The memory stores computer-executable instructions; the transceiver is used to send and receive data;
[0031] The processor executes the computer-executable instructions stored in the memory to implement the monitoring data processing method described in any one of the above aspects.
[0032] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer execution instructions, and when the computer execution instructions are executed by a processor, they are used to implement the monitoring data processing method described in any of the above aspects.
[0033] In a fifth aspect, the present application provides a computer program product, comprising computer execution instructions, which, when executed by a processor, implement the monitoring data processing method described in any of the above aspects.
[0034] The monitoring data processing method, device, equipment and storage medium provided by the present application obtain the original monitoring data of the monitored object based on the collection tool telegraf; the monitored object is a cloud server, or a physical server, or a virtual machine, or a storage device: the original monitoring data is converted into the adapted monitoring data with the same field name according to the indicator adaptation table; the adapted monitoring data is calculated according to the corresponding preset standardized algorithm to convert the adapted monitoring data into standardized target monitoring data; the target monitoring data is stored in a distributed database; in response to a data query request of a monitored object by a user terminal, the target monitoring data of the monitored object is obtained from the distributed database, and the target monitoring data of the monitored object is sent to the user terminal. Through the indicator adaptation table and the calculation of the monitoring data in standard units and formats by the preset standardized algorithm, the monitoring data of different monitoring data formats of multiple cloud vendors and different physical server devices are universally adapted and converted into a data format that is easy to handle with a unified standard unit and format, achieving a high degree of reusability and improving development efficiency. In addition, a universal monitoring data query interface can be provided to obtain data query requests through the universal monitoring data query, thereby avoiding repeated development of monitoring data query interfaces for each monitored object, so that developers only need to focus on data collection and data processing processes, saving development time. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0036] Figure 1 A flow chart of a monitoring data processing method provided in an embodiment of the present application;
[0037] Figure 2 A monitoring data processing signaling diagram provided in an embodiment of the present application;
[0038] Figure 3A schematic diagram of an organizational structure provided for an embodiment of the present application;
[0039] Figure 4 A schematic diagram of the structure of a monitoring data processing device provided in an embodiment of the present application;
[0040] Figure 5 A schematic diagram of the structure of a data monitoring server provided in an embodiment of the present application.
[0041] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0042] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0043] The terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. In the description of the following embodiments, "multiple" means more than two, unless otherwise clearly and specifically defined.
[0044] First, the prior art involved in the present invention is described and analyzed in detail.
[0045] Currently, the monitoring data of different cloud vendors have their own data formats, and the monitoring query interface of traditional products needs to develop corresponding interfaces for each product separately to display relevant indicator information. Therefore, it is necessary to solve the problem of how to collect and process data from different cloud vendors and different devices and general query problems.
[0046] The inventor discovered during research that the open source collection tool telegraf has rich plug-ins, and the plug-ins in the collection tool telegraf can be used to collect various monitoring data of cloud products. The collection tool telegraf supports calling interfaces to forward data, so the collection tool telegraf can be used to obtain the original monitoring data of the monitored object; and because the data formats of the obtained original monitoring data are different, the original monitoring data can be standardized first, and the monitoring data in standard units and formats can be calculated to convert the monitoring data in each standard unit and format into target monitoring data of preset dimensions; since the standardized monitoring data of each cloud product is stored according to preset dimensions, a general query interface can be provided, so that users can call the general query interface to monitor each cloud product modified by each cloud business.
[0047] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0048] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.
[0049] Figure 1 The flow chart of the monitoring data processing method provided in the embodiment of the present application is that the monitoring data of cloud products is difficult to collect, and it is difficult to achieve unified monitoring of cloud products by multiple cloud vendors, and a monitoring data processing method is provided in the embodiment of the present application. The method in this embodiment is applied to a monitoring data processing device, and the monitoring data processing device can be located in a data monitoring server. Among them, the data monitoring server can be a digital computer or cloud server representing various forms. Such as laptop computers, cloud servers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers.
[0050] like Figure 2 As shown, the specific steps of this method are as follows:
[0051] Step S101: Acquire original monitoring data of the monitored object based on the collection tool telegraf.
[0052] The monitored objects are cloud servers, physical servers, virtual machines, or storage devices. They can also be bare metal (BMS) servers, network devices, object storage, file storage, load balancing slb, private network vpc, elastic public network eip, etc.
[0053] In the embodiment of the present application, the monitored object can be any system, device, service, etc. that can obtain monitoring data.
[0054] It should be understood that the collection tool telegraf can perform local collection or remote collection.
[0055] In the embodiment of the present application, a collection tool telegraf may be pre-installed in each monitored object, and the monitoring data of the monitored object may be collected using the collection tool telegraf in the device corresponding to the monitored object.
[0056] You can also install the collection tool telegraf in the device monitoring server and use the collection tool telegraf in the monitoring server to collect monitoring data of the monitored object.
[0057] The collection tool telegraf can also be pre-installed in the device corresponding to each monitored object. One device can correspond to multiple monitored objects, and one device can also correspond to one monitored object. This application does not limit this.
[0058] It should be understood that most monitoring indicator data, such as CPU, memory, disk, etc., can be collected and obtained through the relevant plug-ins that come with the collection tool telegraf.
[0059] Optionally, the collection tool telegraf can adopt a dual-engine architecture to support the collection of raw monitoring data for monitoring objects on the openstack platform and the k8s platform.
[0060] Step S102: Convert the original monitoring data into adapted monitoring data with the same field name according to the indicator adaptation table.
[0061] Among them, the indicator adaptation table may include: the adaptation strategy corresponding to each monitored object. In the embodiment of the present application, the field names of the same type of adapted monitoring data of the same type of monitored objects are the same. Exemplarily, the collected original monitoring data of the out-of-band temperature of the physical machine may include the field names: temperature, device temperature, temperature value, etc.; the device temperature may be determined as a preset field name corresponding to the field names such as temperature, device temperature, temperature value, etc., and stored in the corresponding adaptation strategy; after the original monitoring data is converted into the adapted monitoring data with the same field name, the adapted monitoring data of the out-of-band temperature of each physical machine includes: the preset field name: device temperature and the corresponding value, which is the same as the original monitoring data.
[0062] Specifically, the adaptation strategy corresponding to each monitored object can be determined based on the format of the original monitoring data of each monitored object and the preset field name, and the adaptation strategy corresponding to each monitored object can be added to the indicator adaptation table. After obtaining the original monitoring data of the monitored object, the original monitoring data can be converted into adapted monitoring data with the same field name according to the corresponding adaptation strategy in the adaptation strategy.
[0063] Step S103: Calculate the adapted monitoring data according to a corresponding preset standardized algorithm to convert the adapted monitoring data into standardized target monitoring data.
[0064] In the embodiment of the present application, the standardized target monitoring data is monitoring data with unified data units and statistical dimensions.
[0065] Specifically, the field value corresponding to the field name can be converted into units according to the preset standardization algorithm corresponding to the monitored object and the field name, and the field value after unit conversion can be numerically calculated according to the preset statistical dimension value, and the calculation result can be determined as the target monitoring data.
[0066] Step S104: store the target monitoring data in a distributed database.
[0067] This application does not limit the distributed database and the form of the distributed database used. By way of example, the target monitoring data can be stored in a mongoDB database cluster.
[0068] It should be understood that a distributed database system is a union of multiple autonomous computer systems located in different geographical locations, which uses a unified global model and query language in conjunction with distributed transaction management and concurrency control technologies to achieve sharing and centralized management of data resources. It has the advantages of high reliability, high performance, strong scalability, and balanced data distribution.
[0069] Step S105 : in response to a data query request of a monitored object from a user terminal, obtaining target monitoring data of the monitored object from a distributed database, and sending the target monitoring data of the monitored object to the user terminal.
[0070] Specifically, a data query request from a user terminal for a monitored object is obtained, target monitoring data of the monitored object is obtained from a distributed database in response to the data query request from the user terminal for the monitored object, and the target monitoring data of the monitored object is sent to the user terminal.
[0071] Optionally, the distributed database may be a mongoDB database cluster. The mongoDB cluster as a whole may include 5 nodes and may be deployed in a 5-shard, 2-copy mode, thereby ensuring high availability and fault tolerance of the cluster.
[0072] The monitoring data processing method improved by the embodiment of the present application obtains the original monitoring data of the monitored object based on the collection tool telegraf; the monitored object is a cloud server, or a physical server, or a virtual machine, or a storage device: the original monitoring data is converted into the adapted monitoring data with the same field name according to the indicator adaptation table; the adapted monitoring data is calculated according to the corresponding preset standardized algorithm to convert the adapted monitoring data into standardized target monitoring data; the target monitoring data is stored in a distributed database; in response to a data query request of a monitored object by a user terminal, the target monitoring data of the monitored object is obtained from the distributed database, and the target monitoring data of the monitored object is sent to the user terminal. Through the indicator adaptation table and the calculation of the monitoring data in standard units and formats by the preset standardized algorithm, the monitoring data of different monitoring data formats of multiple cloud vendors and different physical server devices are universally adapted and converted into a data format with a unified standard unit and format that is easy to handle, thereby achieving a high degree of reusability and improving development efficiency. In addition, a universal monitoring data query interface can be provided to obtain data query requests through the universal monitoring data query, thereby avoiding repeated development of monitoring data query interfaces for each monitored object, so that developers only need to focus on data collection and data processing processes, saving development time.
[0073] In an optional implementation, a collection tool telegraf may be pre-installed in a device corresponding to each monitored object. One device may correspond to multiple monitored objects, or one device may correspond to one monitored object. Specifically, the following steps may be included:
[0074] Step S201: Send the collection tool telegraf configuration package to the device corresponding to the monitored object, so that the collection tool telegraf is installed on the device corresponding to the monitored object.
[0075] Specifically, the device corresponding to the monitored object can use the collection tool telegraf to collect the original monitoring data of the monitored object at a preset time interval, or use the collection tool telegra to call the data interface of the monitored object to obtain the original monitoring data of the monitored object.
[0076] Step S202: Obtain the original monitoring data of the monitored object acquired by the collection tool telegraf.
[0077] In an embodiment of the present application, the original monitoring data of the monitored object acquired by the collection tool telegraf may be collected by the collection tool telegraf, or may be acquired by calling a data interface of the monitored object.
[0078] The monitoring data processing method provided in the embodiment of the present application sends the collection tool telegraf configuration package to the device corresponding to the monitored object, so that the device corresponding to the monitored object installs the collection tool telegraf, and uses the collection tool telegraf to collect the original monitoring data of the monitored object at a preset time interval, or uses the collection tool telegra to call the data interface of the monitored object to obtain the original monitoring data of the monitored object; obtain the original monitoring data of the monitored object obtained by the collection tool telegraf. The monitoring data can be collected directly or through the interface, that is, it can be collected remotely or remotely, which increases the scope of application of the monitoring data that can be collected, thereby increasing the applicability of the present method.
[0079] The monitoring data processing method provided in the above embodiment is described below with reference to a specific example. Figure 2 A monitoring data processing signaling diagram provided in an embodiment of the present application, such as Figure 2 As shown, the following steps are included:
[0080] Step S301: The collection tool telegraf sends the original monitoring data of the monitored object to the message queue kafka cluster.
[0081] Step S302: The data monitoring server obtains the original monitoring data of the monitored object obtained by the collection tool telegraf from the kafka cluster.
[0082] Specifically, the collection tool telegraf can be determined as the producer of the kafka cluster, and the data monitoring server can be determined as the consumer of the kafka cluster, so as to build a reliable real-time streaming pipeline for original monitoring data transmission between the collection tool telegraf and the data monitoring server.
[0083] In the embodiment of the present application, a kafka cluster is used to forward the original monitoring data of the monitored object, thereby avoiding the coupling between the collection tool telegraf and the data monitoring server, and reducing the pressure on the data monitoring server, that is, the load; and the nodes in the Kafka cluster can be dynamically added and deleted according to the amount of data.
[0084] Step S303: The data monitoring server determines a corresponding adaptation strategy from the indicator adaptation table based on the identifier of the monitored object and the type of the original monitoring data.
[0085] Step S304: The data monitoring server extracts the field value corresponding to the preset field name according to the corresponding adaptation strategy to convert the original monitoring data into adapted monitoring data with the same field name.
[0086] Specifically, the corresponding adaptation strategy can be determined from the indicator adaptation table based on the identification of the monitored object and the type of the original monitoring data. The adaptation strategy includes the preset field name corresponding to the type of the original monitoring data, and the original field name of the original monitoring data corresponding to the preset field name; the original monitoring data whose original field name corresponds to the preset field name is extracted, thereby obtaining the field value corresponding to the preset field name, and the preset field name and the corresponding field value are determined as the adapted monitoring data.
[0087] It should be understood that the original monitoring data may include data other than the field values corresponding to the preset field names, and the field values corresponding to the preset field names can be extracted according to the corresponding adaptation strategy.
[0088] In the embodiment of the present application, the corresponding adaptation strategy is determined from the indicator adaptation table based on the identification of the monitored object and the type of the original monitoring data; the field value corresponding to the preset field name is extracted according to the corresponding adaptation strategy to convert the original monitoring data into the adapted monitoring data with the same field name. The required fields can be filtered out from the original monitoring data, and the field names can be unified to facilitate the storage and reading of the monitoring data.
[0089] Step S305: The data monitoring server calculates the adapted monitoring data according to the corresponding preset standardized algorithm to convert the adapted monitoring data into standardized target monitoring data.
[0090] Optionally, it may include: converting the adapted monitoring data into adapted monitoring data in preset standard units and formats; if the monitoring data in the standard units and formats is determined to be a loss value, then using the big data processing engine to calculate the monitoring data in the standard units and formats according to a preset standardized algorithm corresponding to the loss value; if the monitoring data in the standard units and formats is determined to be a cumulative value, then using the big data processing engine to calculate the monitoring data in the standard units and formats according to a preset standardized algorithm corresponding to the cumulative value.
[0091] Exemplarily, the unit of duration may include: seconds and milliseconds, and the corresponding preset standard unit may be determined as seconds or milliseconds. If the preset standard unit corresponding to duration is determined as seconds, the unit of the field value of time may be converted from milliseconds to seconds based on the preset standardization algorithm corresponding to duration.
[0092] You can also determine a preset standard unit for temperature and set a preset standardization algorithm corresponding to the temperature to convert the unit of the temperature field value into the preset standard unit.
[0093] Exemplarily, the preset standard format of time can be determined as year-month-day; hour: minute: second. If the time data of the adapted monitoring data only includes hours and minutes, such as 16:10, the year, month and day can be determined based on the time of receiving the corresponding original monitoring data, and the value corresponding to the second is determined to be 00, and the standardized adapted monitoring data can be obtained, such as 2023-06-13; 16:10:00.
[0094] Specifically, after the adapted monitoring data is converted into the adapted monitoring data in a preset standard unit and format, the loss value or the accumulated value may also be calculated to unify the statistical dimension of the monitoring data.
[0095] It should be understood that the loss value is the difference between the predicted value and the actual value. The actual value can be calculated based on the predicted value and the loss value, and the actual value in a standard unit and format is determined as the standardized target monitoring data.
[0096] It should be understood that the loss value refers to the repetition of adding other values to one value. The actual value can be calculated based on the last accumulated value and the current accumulated value, and the actual value in the standard unit and format is determined as the standardized target monitoring data.
[0097] Optionally, the time dimension of the monitoring data may be unified. For example, the monitoring data of every minute and the monitoring data of every two minutes may be converted into the monitoring data of every five minutes.
[0098] In the embodiment of the present application, the adapted monitoring data is converted into the adapted monitoring data of the preset standard unit and format; if the monitoring data of the standard unit and format is determined to be a loss value, the big data processing engine is used to calculate the monitoring data of the standard unit and format according to the preset standardized algorithm corresponding to the loss value; if the monitoring data of the standard unit and format is determined to be an accumulated value, the big data processing engine is used to calculate the monitoring data of the standard unit and format according to the preset standardized algorithm corresponding to the accumulated value. The adapted monitoring data can be converted into standardized monitoring data, which can facilitate the calling and querying of the monitoring data, so that only a unified query interface can be provided to realize the query of the monitoring data of each monitored object.
[0099] Step S306: Store the target monitoring data in a distributed database.
[0100] Figure 3 A monitoring data query signaling diagram provided in an embodiment of the present application, such as Figure 3 As shown, the steps for monitoring data query are as follows:
[0101] The following is the general query process:
[0102] Step S307: The user terminal sends a data query request for a monitored object to the data monitoring server.
[0103] In the embodiment of the present application, the data query request may include the monitored object identifier and the target time range.
[0104] For example, the target time range may be within 1 hour, within 12 hours, within 7 days, etc., which is not limited in the present embodiment. The target time range may also be determined based on the user's selection of a date.
[0105] Step S308: The data monitoring server obtains target monitoring data of the monitored object within a target time range from the distributed database.
[0106] Specifically, the monitored object identifier and the target time range in the data query request may be obtained, and based on the monitored object identifier and the target time range, the target monitoring data of the monitored object within the target time range may be obtained from the distributed database.
[0107] Step S309: The data monitoring server sends the target monitoring data of the monitored object within the target time range to the user terminal.
[0108] Among them, the target monitoring data may include: CPU utilization (%), memory utilization (%), storage utilization (%), disk read and write IOPS (times / second), network inflow and outflow rate (Byte / s), disk space (Byte), IOPS (times / second), current total number of connections, network traffic (Byto / s), the total number and proportion of each process status, etc.
[0109] Step S310: The user terminal displays the target monitoring data of the monitored object within the target time range in the form of a chart.
[0110] For example, the CPU usage (%), memory usage (%), storage usage (%), disk read / write IOPS (times / second), and network inflow and outflow rate (Byte / s) can be displayed in the form of a line graph; the total number and proportion of each process status can be displayed in the form of a table. The process status includes: blocked, dead process, idle, etc.
[0111] In an embodiment of the present application, a general query interface is provided, which can display the target monitoring data of a monitored object within a target time range in the form of a chart in a user terminal, so that the user can know the target monitoring data of the monitored object within the target time range, thereby improving the user experience.
[0112] The following is the aggregation query process:
[0113] Step S311: The user terminal sends a data aggregation query request to the data monitoring server.
[0114] Step S312: The data monitoring server obtains basic information corresponding to each monitored object from the relational database.
[0115] The basic information may include the ID of the monitored object, cloud service provider, resource pool, available domain, number of CPUs, memory size, system disk size, etc.
[0116] Step S313: The data monitoring server obtains target monitoring data of at least one monitored object from the distributed database.
[0117] Specifically, in response to a data aggregation query request from a user terminal for at least one monitored object, target monitoring data of at least one monitored object is obtained from a distributed database.
[0118] Step S314: The data monitoring server performs aggregation calculations on the target monitoring data of each monitored object to obtain the aggregation calculation results of each monitored object.
[0119] The aggregate calculation results of the monitored objects may include: average CPU utilization, CPU peak utilization, average memory utilization, memory peak utilization, cloud hard disk utilization, etc.
[0120] Step S315: The data monitoring server sends the basic information corresponding to each monitored object and the aggregate calculation result of each monitored object to the user terminal.
[0121] Specifically, the basic information corresponding to each monitored object and the aggregate calculation results of each monitored object are sent to the user terminal, so that the basic information corresponding to each monitored object and the aggregate calculation results of each monitored object are displayed in the user terminal in the form of a list.
[0122] Optionally, the aggregate calculation results of each monitored object can be aggregated again to display the average CPU utilization, average memory utilization, cloud hard disk utilization, etc. of the total of each monitored object in the form of a line graph and a table in the user terminal.
[0123] Optionally, it may also be determined whether to issue an alarm based on the target monitoring data of each monitored object, and the aggregate calculation result of the alarm information may be displayed in the user terminal.
[0124] Step S316: The user terminal displays the basic information corresponding to each monitored object and the aggregate calculation results of each monitored object in the form of a list.
[0125] In an embodiment of the present application, an aggregate query interface is provided, which can display the basic information corresponding to each monitored object and the aggregate calculation results of each monitored object in the form of a list in the user terminal, so that the user can obtain the basic information corresponding to each monitored object and the aggregate calculation results of each monitored object, thereby improving the user experience.
[0126] Figure 4 The structure diagram of the monitoring data processing device provided in the embodiment of the present application is shown in FIG. The monitoring data processing device provided in the embodiment of the present application can execute the processing flow provided in the monitoring data processing method embodiment. Figure 4 As shown, the monitoring data processing device 50 includes: an acquisition module 501, an adaptation module 502, a calculation module 503, a storage module 504 and a query module 505.
[0127] Specifically, the acquisition module 501 is used to acquire the original monitoring data of the monitored object based on the collection tool telegraf; the monitored object is a cloud server, or a physical server, or a virtual machine, or a storage device:
[0128] An adaptation module 502, used to convert the original monitoring data into adapted monitoring data with the same field name according to the indicator adaptation table;
[0129] A calculation module 503, used to calculate the adapted monitoring data according to a corresponding preset standardized algorithm, so as to convert the adapted monitoring data into standardized target monitoring data;
[0130] Storage module 504, used to store target monitoring data in a distributed database;
[0131] The query module 505 is used to respond to a data query request of a monitored object from a user terminal, obtain target monitoring data of the monitored object from a distributed database, and send the target monitoring data of the monitored object to the user terminal.
[0132] The device provided in the embodiments of the present application can be specifically used to execute the method embodiments provided in the above embodiments, and the specific functions will not be repeated here.
[0133] Optionally, the acquisition module 501 is specifically used to: send the collection tool telegraf configuration package to the device corresponding to the monitored object, so that the collection tool telegraf is installed on the device corresponding to the monitored object, and use the collection tool telegraf to collect the original monitoring data of the monitored object at a preset time interval, or use the collection tool telegra to call the data interface of the monitored object to obtain the original monitoring data of the monitored object; obtain the original monitoring data of the monitored object obtained by the collection tool telegraf.
[0134] Optionally, the collection tool telegraf sends the acquired original monitoring data of the monitored object to the message queue kafka cluster, and the acquisition module 501 is specifically used to: acquire the original monitoring data of the monitored object acquired by the collection tool telegraf from the kafka cluster.
[0135] Optionally, the adaptation module 502 is specifically used to: determine the corresponding adaptation strategy from the indicator adaptation table based on the identification of the monitored object and the type of original monitoring data; extract the field value corresponding to the preset field name according to the corresponding adaptation strategy to convert the original monitoring data into adapted monitoring data with the same field name.
[0136] Optionally, the calculation module 503 is specifically used to: convert the adapted monitoring data into adapted monitoring data in preset standard units and formats; if the monitoring data in the standard units and formats is determined to be a loss value, then the big data processing engine is used to calculate the monitoring data in the standard units and formats according to a preset standardized algorithm corresponding to the loss value; if the monitoring data in the standard units and formats is determined to be a cumulative value, then the big data processing engine is used to calculate the monitoring data in the standard units and formats according to a preset standardized algorithm corresponding to the cumulative value.
[0137] Optionally, the query module 505 is specifically used to: obtain the monitored object identifier and the target time range in the data query request; obtain the target monitoring data of the monitored object within the target time range from the distributed database based on the monitored object identifier and the target time range; send the target monitoring data of the monitored object within the target time range to the user terminal, so as to display the target monitoring data of the monitored object within the target time range in the form of a chart in the user terminal.
[0138] Optionally, the query module 505 is also used to: in response to a data aggregation query request from a user terminal for at least one monitored object, obtain target monitoring data of at least one monitored object from a distributed database; perform aggregation calculations on the target monitoring data of each monitored object respectively to obtain the aggregation calculation results of each monitored object; obtain basic information corresponding to each monitored object from a relational database; send the basic information corresponding to each monitored object and the aggregation calculation results of each monitored object to the user terminal, so as to display the basic information corresponding to each monitored object and the aggregation calculation results of each monitored object in the form of a list in the user terminal.
[0139] The device provided in the embodiment of the present application can be specifically used to execute the above method embodiment, and the specific functions will not be repeated here.
[0140] Figure 5 A schematic diagram of the structure of the data monitoring server provided in the embodiment of the present application, such as Figure 5As shown, the present application also provides a data monitoring server 60, including: a processor 601, and a memory 602 and a transceiver 603 that are communicatively connected to the processor 601. Among them, the memory 602 stores computer execution instructions; the transceiver 603 is used to send and receive data; the processor 601 executes the computer execution instructions stored in the memory 602 to implement the method provided in any embodiment of the present application.
[0141] Specifically, the program may include a program code, and the program code includes a computer-executable instruction. The memory 602 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory. Among them, the computer-executable instruction is stored in the memory 602 and is configured to be executed by the processor 601 to implement the method provided in any embodiment of the present application. The relevant description can be understood by referring to the relevant description and effect corresponding to the steps in the accompanying drawings, and no further elaboration is made here.
[0142] In the embodiment of the present application, the memory 602 and the processor 601 are connected via a bus. The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0143] An embodiment of the present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the method provided by any embodiment of the present application.
[0144] An embodiment of the present application also provides a computer program product, including computer execution instructions, which implement the method provided by any embodiment of the present application when executed by a processor.
[0145] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.
[0146] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0147] In addition, each functional module in each embodiment of the present application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or in the form of hardware plus software functional modules.
[0148] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable full-path trajectory fusion device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.
[0149] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0150] In addition, although each operation is described in a specific order, this should be understood as requiring such operation to be performed in the specific order shown or in a sequential order, or requiring that all illustrated operations should be performed to obtain desired results. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the application. Some features described in the context of a separate embodiment can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination mode.
[0151] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.
[0152] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.
Claims
1. A monitoring data processing method, characterized in that: include: Based on the collection tool telegraf, the original monitoring data of the monitored object is obtained; the monitored object is a cloud server, a physical server, a virtual machine, or a storage device: Converting the original monitoring data into adapted monitoring data with the same field name according to the indicator adaptation table; Calculating the adapted monitoring data according to a corresponding preset standardized algorithm so as to convert the adapted monitoring data into standardized target monitoring data; Storing the target monitoring data in a distributed database; In response to a data query request from a user terminal for a monitored object, target monitoring data of the monitored object is obtained from a distributed database, and the target monitoring data of the monitored object is sent to the user terminal.
2. The method according to claim 1, characterized in that The obtaining of original monitoring data of the monitored object includes: Send the collection tool telegraf configuration package to the device corresponding to the monitored object, so that the device corresponding to the monitored object installs the collection tool telegraf, and uses the collection tool telegraf to collect the original monitoring data of the monitored object at a preset time interval, or uses the collection tool telegra to call the data interface of the monitored object to obtain the original monitoring data of the monitored object; The original monitoring data of the monitored object acquired by the acquisition tool telegraf is obtained.
3. The method according to claim 2, characterized in that The acquisition tool telegraf sends the acquired original monitoring data of the monitored object to the message queue kafka cluster, and the acquisition of the original monitoring data of the monitored object acquired by the acquisition tool telegraf includes: The original monitoring data of the monitored object obtained by the collection tool telegraf is obtained from the kafka cluster.
4. The method according to claim 1, characterized in that The step of converting the original monitoring data into adapted monitoring data with the same field name according to the indicator adaptation table includes: Determining a corresponding adaptation strategy from the indicator adaptation table based on the identifier of the monitored object and the type of the original monitoring data; The field value corresponding to the preset field name is extracted according to the corresponding adaptation strategy to convert the original monitoring data into the adapted monitoring data with the same field name.
5. The method according to claim 1, characterized in that The step of calculating the adapted monitoring data according to a corresponding preset standardized algorithm so as to convert the adapted monitoring data into standardized target monitoring data includes: Converting the adapted monitoring data into the adapted monitoring data in a preset standard unit and format; If it is determined that the monitoring data in the standard unit and format is a loss value, the monitoring data in the standard unit and format is calculated using a big data processing engine according to a preset standardization algorithm corresponding to the loss value; If it is determined that the monitoring data in the standard unit and format is a cumulative value, the big data processing engine is used to calculate the monitoring data in the standard unit and format according to a preset standardization algorithm corresponding to the cumulative value.
6. The method according to claim 1, characterized in that The method of responding to a data query request of a monitored object from a user terminal, obtaining target monitoring data of the monitored object from a distributed database, and sending the target monitoring data of the monitored object to the user terminal includes: Obtain the monitored object identifier and target time range in the data query request; Acquire target monitoring data of the monitored object within the target time range from a distributed database based on the monitored object identifier and the target time range; The target monitoring data of the monitored object within the target time range is sent to the user terminal, so that the target monitoring data of the monitored object within the target time range is displayed in the user terminal in the form of a chart.
7. The method according to any one of claims 1 to 6, characterized in that: After storing the target monitoring data in the distributed database, the method further includes: In response to a data aggregation query request of a user terminal for at least one monitored object, obtaining target monitoring data of at least one monitored object from a distributed database; Performing aggregate calculations on the target monitoring data of each monitored object to obtain aggregate calculation results of each monitored object; Obtain basic information corresponding to each monitored object from the relational database; The basic information corresponding to each monitored object and the aggregate calculation results of each monitored object are sent to the user terminal, so that the basic information corresponding to each monitored object and the aggregate calculation results of each monitored object are displayed in the user terminal in the form of a list.
8. A monitoring data processing device, characterized in that: include: The acquisition module is used to acquire the original monitoring data of the monitored object based on the acquisition tool telegraf; the monitored object is a cloud server, a physical server, a virtual machine, or a storage device: An adaptation module, used for converting the original monitoring data into adapted monitoring data with the same field name according to the indicator adaptation table; A calculation module, used to calculate the adapted monitoring data according to a corresponding preset standardized algorithm, so as to convert the adapted monitoring data into standardized target monitoring data; A storage module, used for storing the target monitoring data in a distributed database; The query module is used to respond to a data query request of a monitored object from a user terminal, obtain the target monitoring data of the monitored object from a distributed database, and send the target monitoring data of the monitored object to the user terminal.
9. A data monitoring server, characterized in that: include: A processor, and a memory and a transceiver communicatively connected to the processor; The memory stores computer-executable instructions; the transceiver is used to send and receive data; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.