Application cluster monitoring data acquisition method and system based on multi-dimensional fusion

Abstract

The invention provides an application cluster monitoring data acquisition method and system based on multi-dimensional fusion, and relates to the technical field of computers, and the method comprises the steps: obtaining IT platform operation indexes of each application cluster; extracting the operation indexes to obtain characteristic indexes; collecting the characteristic indexes to obtain collection indexes; according to the acquisition index, determining an acquisition frequency and an acquisition state to obtain state information of the operation index; according to the state information, collecting original data, and performing data encapsulation and integration processing on the original data to obtain encapsulated and integrated data; and summarizing the packaged and integrated data to obtain summarized data. Omnibearing data acquisition is carried out on the overall condition of each application cluster and application operation key indexes, acquisition strategies and acquisition frequencies of various indexes of the cluster are obtained, and multi-dimensional fusion monitoring of the overall utilization rate of each index of an application cluster IT platform and the application key feature operation condition is realized.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method for collecting application cluster monitoring data based on multi-dimensional fusion. Background Art

[0002] An important part of system operation and maintenance monitoring is the collection of basic operating characteristics of the IT platform. Traditional IT platform operation and maintenance monitoring mostly collects some basic load indicators from each server through a timing program, such as CPU utilization, memory utilization, SWAP utilization, etc. These monitoring information can reflect the load utilization of a single server to a certain extent, but for the cluster operation mode of the Fengyun-3 satellite application, the load indicator of a single server cannot reflect the overall cluster resource utilization of the application. In order to accurately reflect the overall load of each application cluster, it is necessary to collect the key attribute monitoring information of the computing, storage, network and related applications of each application operation cluster of the Fengyun-3 satellite in real time, formulate the collection strategy and collection frequency of various cluster indicators, and realize the multi-dimensional integrated monitoring of the overall utilization rate of various indicators of the application cluster IT platform and the operation status of key application characteristics. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide an application cluster monitoring data collection method and system based on multi-dimensional fusion, which conducts all-round data collection on the overall situation of each application cluster and key indicators of application operation, obtains the collection strategy and collection frequency of various indicators of the cluster, and realizes multi-dimensional fusion monitoring of the overall utilization rate of various indicators of the application cluster IT platform and the operation status of key application features.

[0004] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0005] In a first aspect, a method for collecting application cluster monitoring data based on multi-dimensional fusion is provided, the method comprising:

[0006] Obtain the IT platform operation indicators of each application cluster;

[0007] Extracting the operation indicators to obtain characteristic indicators;

[0008] Collecting characteristic indicators to obtain collection indicators;

[0009] According to the collected indicators, determine the collection frequency and collection status to obtain the status information of the operating indicators;

[0010] According to the status information, the original data is collected, and data encapsulation and integration processing is performed on the original data to obtain encapsulated and integrated data;

[0011] The packaged integrated data are aggregated to obtain aggregated data.

[0012] Furthermore, the IT platform operation indicators of each application cluster are obtained, including:

[0013] By summarizing the computing, storage, and network indicators of all nodes in multiple clusters, the overall operation indicators of the application cluster can be obtained.

[0014] Furthermore, the operation indicators are extracted to obtain characteristic indicators, including:

[0015] By configuring the collection plug-in, key characteristic indicators are captured, including middleware information, middleware service status, application database information, network communication between database high-availability nodes, application key port status, resource scheduling job queue status, resource scheduling service status, resource scheduling database status and monitoring data aggregation component status.

[0016] Furthermore, the characteristic indicators are collected to obtain collection indicators, including:

[0017] By applying the characteristic indicators, the application operation characteristics of different indicators are obtained;

[0018] According to the application operation characteristics of different indicators, corresponding collection strategies are adopted to collect data to obtain collection indicators.

[0019] Furthermore, according to the collection index, the collection frequency and the collection state are determined to obtain the state information of the operation index, including:

[0020] According to the collected indicators, the collection frequency and collection status are obtained. The collection frequency of CPU, memory, network, storage and application key characteristics is 30 seconds.

[0021] According to the indicator frequency and collection status, a concurrent strategy is adopted to read the status information of each running indicator to obtain the status information of each indicator.

[0022] Further, according to the state information, the original data is collected, and the original data is packaged and integrated to obtain packaged and integrated data, including:

[0023] According to the state information, various types of raw data are collected, and the raw data are preprocessed to obtain preprocessed data;

[0024] Encapsulate and verify the preprocessed data to obtain verified data;

[0025] The verified data is transformed and integrated to obtain packaged and integrated data.

[0026] Further, the packaged and integrated data is summarized to obtain summary data, including:

[0027] Filter, transform and calculate the packaged and integrated data to obtain processed data;

[0028] The processed data is aggregated to obtain summary data.

[0029] In a second aspect, an application cluster monitoring data collection system based on multi-dimensional fusion includes:

[0030] The acquisition module is used to obtain the operation indicators of each application cluster IT platform; extract the operation indicators to obtain characteristic indicators; and collect the characteristic indicators to obtain collection indicators.

[0031] The processing module is used to determine the collection frequency and collection status according to the collection indicators to obtain the status information of the operation indicators; collect the original data according to the status information, and perform data packaging and integration processing on the original data to obtain packaged and integrated data; summarize the packaged and integrated data to obtain summarized data.

[0032] According to a third aspect, a computing device includes:

[0033] one or more processors;

[0034] The storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method described.

[0035] In a fourth aspect, a computer-readable storage medium stores a program, and when the program is executed by a processor, the method described is implemented.

[0036] The above solution of the present invention includes at least the following beneficial effects:

[0037] The computing, storage and network indicators of all nodes in multiple clusters are summarized to ensure the comprehensiveness of the data. By configuring the collection plug-in to capture key characteristic indicators, including middleware information, database status, and resource scheduling status, the accuracy of data collection is improved. According to different collection indicators, corresponding collection strategies are adopted, and a reasonable collection frequency is set to ensure the efficiency and real-time performance of data collection. The status information of each operating indicator is read using a concurrent strategy to improve the efficiency of data collection. According to different application operation characteristics, corresponding collection strategies are adopted, which reflects good flexibility. By encapsulating and integrating the original data, and supporting data filtering, conversion and calculation processing, it can adapt to the monitoring needs of application clusters of different scales and complexities. Through all-round data collection and indicator integration, a comprehensive analysis of the application cluster operation status can be performed. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1It is a flowchart of an application cluster monitoring data collection method based on multi-dimensional fusion provided by an embodiment of the present invention.

[0039] Figure 2 It is a structural block diagram of an application cluster monitoring data collection method based on multi-dimensional fusion provided by an embodiment of the present invention.

[0040] Figure 3 It is a standardized data encapsulation and integration processing flow chart of an application cluster monitoring data collection method based on multi-dimensional fusion provided by an embodiment of the present invention.

[0041] Figure 4 It is a schematic diagram of streaming data aggregation processing of an application cluster monitoring data collection method based on multi-dimensional fusion provided by an embodiment of the present invention.

[0042] Figure 5 It is a schematic diagram of an application cluster monitoring data acquisition system based on multi-dimensional fusion provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0044] like Figures 1 to 4 As shown, an embodiment of the present invention proposes an application cluster monitoring data collection method based on multi-dimensional fusion, and the method includes the following steps:

[0045] Step 11, obtaining the IT platform operation indicators of each application cluster;

[0046] Step 12, extracting the operation index to obtain the characteristic index;

[0047] Step 13, collecting characteristic indicators to obtain collection indicators;

[0048] Step 14, determining the collection frequency and collection status according to the collection index to obtain status information of the operation index;

[0049] Step 15, collecting original data according to the state information, and performing data encapsulation and integration processing on the original data to obtain encapsulated and integrated data;

[0050] Step 16, summarizing the packaged and integrated data to obtain summary data.

[0051] In an embodiment of the present invention, the operating indicators of each application cluster IT platform can be obtained through the operating status of each application cluster IT platform; by extracting key feature indicators and collecting data according to the collection strategy of feature indicators, different collection indicators are obtained, and the original data is packaged and integrated, the quality and availability of the data can be improved, and the packaged and integrated data can be summarized to form a more comprehensive and systematic data set; the cluster's various indicator collection strategies and collection frequencies can achieve multi-dimensional integrated monitoring of the overall utilization rate of various indicators of the application cluster IT platform and the operation status of key application features.

[0052] In a preferred embodiment of the present invention, the above step 11 includes:

[0053] By summarizing the computing, storage, and network indicators of all nodes in multiple clusters, the overall operation indicators of the application cluster can be obtained.

[0054] In the embodiment of the present invention, the overall operation status of the IT platform of the application cluster can be comprehensively and in real time. Based on the summarized operation indicators, resource allocation and optimization can be performed more accurately. By real-time monitoring and summarizing the operation indicators, data support is provided for the overall operation. In specific applications, the IT platform operation indicators are indicators that can reflect the overall operation of the application cluster; for example, by formulating the IT platform operation indicators of the FY-3F / G satellite application cluster, the computing, storage, and network indicators of all nodes in the multi-cluster are summarized and counted, and the overall operation indicators of the application cluster are obtained.

[0055] In a preferred embodiment of the present invention, the above step 12 includes:

[0056] By configuring the collection plug-in, key characteristic indicators are captured, including middleware information, middleware service status, application database information, network communication between database high-availability nodes, application key port status, resource scheduling job queue status, resource scheduling service status, resource scheduling database status and monitoring data aggregation component status.

[0057] In an embodiment of the present invention, a dedicated collection plug-in is configured to capture and collect characteristic indicators, such as middleware information, middleware service status, application database information, network communication between database high-availability nodes, application key port status, resource scheduling job queue status, resource scheduling service status, resource scheduling database status, and monitoring data aggregation component status; the collection plug-in is a tool that can automatically acquire data, and it simulates the browser behavior to automatically access web pages and extract characteristic data; the collection plug-in can capture key characteristic indicators in real time, obtain data on the overall load situation based on real-time data, and realize scientific evaluation of real-time data.

[0058] In a preferred embodiment of the present invention, the above step 13 includes:

[0059] By applying the characteristic indicators, the application operation characteristics of different indicators are obtained;

[0060] According to the application operation characteristics of different indicators, corresponding collection strategies are adopted to collect data to obtain collection indicators.

[0061] In the embodiment of the present invention, each key characteristic indicator is analyzed to obtain the normal fluctuation range, abnormal manifestation and influencing factors of the characteristic indicator, and the application operation characteristics of each key characteristic indicator are obtained. According to different application operation characteristics, corresponding collection strategies are adopted for collection. The collection strategies are:

[0062] A. Cluster computing index collection strategy: The Fengyun-3F / G satellite product generation cluster has a total of 10 computing nodes. Collect the CPU value of each computing node, summarize the data within 10 minutes, and calculate the average and peak CPU values ​​of the cluster at each minute; cluster CPU average value characteristics, the cluster has a total of 10 computing nodes, let the data of node j in the ith minute be D ij and D ij+1 , summarizing the collected data within 10 minutes, then the average data point A in the i-th minute i It is expressed as:

[0063]

[0064] When 10 average data points are obtained, it means that the average characteristic data of the cluster CPU per minute within 10 minutes is:

[0065]

[0066]

[0067] Cluster CPU peak characteristics, summarizing the collected data within 10 minutes, the maximum value M in the i-th minute i , expressed as:

[0068]

[0069] When 10 maximum value data points are obtained, it means that the cluster CPU peak characteristic data per minute within 10 minutes is:

[0070]

[0071]

[0072] B. Cluster network indicator collection strategy: Similar to the calculation method of CPU usage, the network traffic of the entire cluster is represented by summarizing the data of each node. For example, the traffic information of the 10 computing node network cards in the Fengyun-3F / G satellite product generation cluster is obtained, and each server obtains the two key indicators of the network card's uplink sending data and downlink receiving data every 30 seconds;

[0073] Average numerical characteristics of the cluster network. The cluster has 10 computing nodes. The uplink data u i j and downlink data d i , j, where i represents the server index from 1 to 10, j represents the time point index from 1 to 2 within the minute, and there are data for two time points every minute. and The formulas for the average values ​​of the uplink and downlink data of the i-th server at the t-th minute are:

[0074]

[0075]

[0076] and The average values ​​of uplink and downlink data of all servers in the tth minute are:

[0077]

[0078]

[0079] Cluster network peak characteristics, get the maximum uplink and downlink data of all servers per minute:

[0080]

[0081]

[0082] Among them, M U (t) and M D (t) and represent the peak values ​​of uplink and downlink data of all servers in the tth minute, respectively.

[0083] C. Group storage indicator collection strategy: Storage indicators need to retain the original data, and storage mainly collects capacity usage, such as the used storage space and the remaining available space.

[0084] D. Application key characteristic index collection strategy: For example, the collection strategy of the key indicators of Fengyun-3F / G satellite applications, middleware information, middleware service status, application database information, database high-availability node network communication, application key port status, resource scheduling job queue status, resource scheduling service status, resource scheduling database status and monitoring data aggregation component status, the steps include:

[0085] Determine the information storage location or command: Determine the log storage location or database, resource scheduling software, and middleware command where the above key indicators are located;

[0086] Clarify the naming rules and output format: Clarify the naming rules for each log, such as date and time stamp, log level, and service name; determine the format of the log, such as plain text or JSON; the status class needs to clarify the output format and content of each command;

[0087] Determine keywords: clarify all status keywords, key ports, and process keywords that need to be collected;

[0088] Formulate a log collection strategy: enable and configure the log collection interval for each application; set the log collection frequency to 30 seconds; limit the frequency of log collection and sending, with a maximum of 140 logs collected and sent per second;

[0089] Configure the collection file: configure the log collection startup mode true or false; limit the frequency of log collection and sending, the maximum number of logs collected and sent per second; the location of the collected log file; match the log format; obtain log keyword information; obtain application key process commands and status, etc.

[0090] In a preferred embodiment of the present invention, the above step 14 includes:

[0091] According to the collected indicators, the frequency and status of collected indicators are obtained. The frequency of collected indicators for CPU, memory, network, storage and application key features is 30 seconds.

[0092] According to the indicator frequency and collection status, a concurrent strategy is adopted to read the status information of each running indicator to obtain the status information of each indicator.

[0093] In the embodiment of the present invention, according to the collection frequency, the changes of indicators can be captured, thereby providing real-time operation status information; the concurrent strategy allows the status information of multiple indicators to be read simultaneously, thereby improving the data processing efficiency. By regularly monitoring key indicators, the unstable factors or potential failure points of the system can be discovered in time; the status information of each indicator is obtained in real time to improve the overall performance and efficiency of the system.

[0094] In a specific embodiment of the present invention, the collection status and collection frequency of each indicator are clarified, and the concurrent strategy is adopted to efficiently read the status information of each operating indicator, wherein the frequency of collecting indicators of CPU, memory, network, storage, and application key features is 30 seconds; the specific process of concurrent strategy monitoring data is:

[0095] The collection component has formulated a variety of collection strategies based on different types of data. General indicators, such as CPU, memory, network and disk, are executed periodically according to their own configuration files, without the need for active calls by the collection component. The collection of application data is sent to the task queue at a fixed time according to the application configuration file, and multiple periodic or one-time task modules obtain tasks from the queue and execute them, avoiding delays in the next task due to the collection of some application data;

[0096] Lightweight threads are lightweight abstractions on top of threads, allowing multiple functions or methods to be executed in parallel in the same address space at a very low cost. Compared to threads, they are cheaper to create and destroy, and their scheduling is independent of threads;

[0097] The indicator collector is responsible for monitoring information collection, running periodically and sending the results to different channels of the specified data pipeline. Each indicator collector can have its own configuration file and run in its own independent lightweight thread. The lightweight thread implements a low-cost and high-concurrency collection operation strategy, and the time interval between each round of data collection can reach the second level.

[0098] In a preferred embodiment of the present invention, the above step 15 includes:

[0099] According to the state information, various types of raw data are collected, and the raw data are preprocessed to obtain preprocessed data;

[0100] Encapsulate and verify the preprocessed data to obtain verified data;

[0101] The verified data is transformed and integrated to obtain packaged and integrated data.

[0102] In the embodiment of the present invention, the preprocessing step can clean the noise, outliers and missing values ​​in the original data, thereby improving the accuracy and reliability of the data; the packaging and verification process ensures the integrity and consistency of the data and reduces data errors and redundancy; the conversion and integration steps convert the data into a unified format, which helps to eliminate format differences between different data sources and improve the readability and usability of the data; the process operations of preprocessing, packaging and verification, conversion and integration can reduce duplication of work and improve the speed and efficiency of data processing.

[0103] In a specific embodiment of the present invention, Figure 3 As shown in the figure, the specific process of encapsulating and integrating data is as follows:

[0104] Data collection: Collect various raw data from various monitoring indicator collectors;

[0105] Preprocessing: Preprocess the collected raw data, including cleaning, normalization and verification;

[0106] Unified packaging: The pre-processed data is uniformly packaged according to the preset data model. During the packaging process, appropriate key-value pairs are assigned to each data point to ensure the integrity and readability of the data;

[0107] Compliance check: Perform compliance check on the encapsulated data to ensure that all data complies with the predetermined standards and specifications, including data type check, necessary field verification, and value range confirmation;

[0108] Convert to JSON format: Convert the data for compliance verification into standard JSON format;

[0109] Data integration: Integrate the converted JSON data to form a complete data set, including data of all monitoring indicators, so as to facilitate unified processing and analysis;

[0110] Data transmission: The integrated JSON data is transmitted to the data collection and aggregation module through the data pipeline. The data pipeline can be a message queue, API call or other data transmission mechanism to ensure that the data reaches the target location safely and reliably.

[0111] In a preferred embodiment of the present invention, the above step 16 includes:

[0112] Filter, transform and calculate the packaged and integrated data to obtain processed data;

[0113] The processed data is aggregated to obtain summary data.

[0114] In the embodiment of the present invention, invalid, erroneous or redundant data can be eliminated through filtering processing, and the data can be further processed by conversion and calculation processing. During the processing, the data can be standardized and normalized to ensure the consistency and comparability of the data; the summary data provides comprehensive and processed information, which provides strong support for business decisions. By filtering, converting and calculating the data, the size and complexity of the data set can be reduced, and the aggregation and summary can simplify the representation of the data, realize the integrated subscription, effective distribution and persistent storage of the data, and realize the unified management of monitoring data.

[0115] In a specific embodiment of the present invention, data aggregation management is performed through an aggregation component; in specific applications, by subscribing to a channel specified in a pipeline, the aggregation component can receive data streams in real time, filter, convert, and calculate the data, and perform data aggregation processing on the processed data, such as Figure 4 to obtain summary data.

[0116] like Figure 5 As shown: An application cluster monitoring data acquisition system 50 based on multi-dimensional fusion, comprising:

[0117] The acquisition module 51 is used to acquire the operation indicators of each application cluster IT platform; extract the operation indicators to obtain characteristic indicators; and collect the characteristic indicators to obtain collection indicators.

[0118] The processing module 52 is used to determine the collection frequency and collection status according to the collection indicators to obtain the status information of the operation indicators; collect the original data according to the status information, and perform data packaging and integration processing on the original data to obtain packaged and integrated data; summarize the packaged and integrated data to obtain summarized data.

[0119] It should be noted that the device is a device corresponding to the above method, and all implementation methods in the above method embodiment are applicable to this embodiment and can achieve the same technical effect.

[0120] The embodiment of the present invention further provides a computing device, comprising: a processor, a memory storing a computer program, wherein when the computer program is executed by the processor, the method described above is executed. All implementations in the above method embodiment are applicable to this embodiment and can achieve the same technical effect.

[0121] The embodiment of the present invention also provides a computer-readable storage medium storing instructions, which, when executed on a computer, enable the computer to execute the method described above. All implementations in the above method embodiment are applicable to this embodiment and can achieve the same technical effect.

[0122] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for collecting application cluster monitoring data based on multi-dimensional fusion, It is characterized in that The method comprises: Obtain the IT platform operation indicators of each application cluster; Extracting the operation indicators to obtain characteristic indicators; Collecting characteristic indicators to obtain collection indicators; According to the collected indicators, determine the collection frequency and collection status to obtain the status information of the operating indicators; According to the status information, the original data is collected, and data encapsulation and integration processing is performed on the original data to obtain encapsulated and integrated data; The packaged integrated data are aggregated to obtain aggregated data.

2. According to the multi-dimensional fusion-based application cluster monitoring data collection method of claim 1, It is characterized in that Obtain the IT platform operation indicators of each application cluster, including: By summarizing the computing, storage, and network indicators of all nodes in multiple clusters, the overall operation indicators of the application cluster can be obtained.

3. The application cluster monitoring data collection method based on multi-dimensional fusion according to claim 2, It is characterized in that The operating indicators are extracted to obtain characteristic indicators, including: By configuring the collection plug-in, key characteristic indicators are captured, including middleware information, middleware service status, application database information, network communication between database high-availability nodes, application key port status, resource scheduling job queue status, resource scheduling service status, resource scheduling database status and monitoring data aggregation component status.

4. According to the multi-dimensional fusion-based application cluster monitoring data collection method of claim 3, It is characterized in that The characteristic indicators are collected to obtain collection indicators, including: By applying the characteristic indicators, the application operation characteristics of different indicators are obtained; According to the application operation characteristics of different indicators, corresponding collection strategies are adopted to collect data to obtain collection indicators.

5. According to the multi-dimensional fusion-based application cluster monitoring data collection method of claim 4, It is characterized in that According to the collection indicators, determine the collection frequency and collection status to obtain the status information of the operation indicators, including: According to the collected indicators, the collection frequency and collection status are obtained. The collection frequency of CPU, memory, network, storage and application key characteristics is 30 seconds. According to the indicator frequency and collection status, a concurrent strategy is adopted to read the status information of each running indicator to obtain the status information of each indicator.

6. The application cluster monitoring data collection method based on multi-dimensional fusion according to claim 5, It is characterized in that According to the status information, the original data is collected and the original data is packaged and integrated to obtain packaged and integrated data, including: According to the state information, various types of raw data are collected, and the raw data are preprocessed to obtain preprocessed data; Encapsulate and verify the preprocessed data to obtain verified data; The verified data is transformed and integrated to obtain packaged and integrated data.

7. The application cluster monitoring data collection method based on multi-dimensional fusion according to claim 6, It is characterized in that Summarize the packaged and integrated data to obtain summary data, including: Filter, transform and calculate the packaged and integrated data to obtain processed data; The processed data is aggregated to obtain summary data.

8. An application cluster monitoring data acquisition system based on multi-dimensional fusion, It is characterized in that include: The acquisition module is used to obtain the operating indicators of each application cluster IT platform; Extracting the operation indicators to obtain characteristic indicators; The characteristic indicators are collected to obtain collection indicators. The processing module is used to determine the collection frequency and collection status according to the collection indicators to obtain the status information of the operation indicators; According to the status information, the original data is collected, and data encapsulation and integration processing is performed on the original data to obtain encapsulated and integrated data; The packaged integrated data are aggregated to obtain aggregated data.

9. A computing device, It is characterized in that include: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images