Unified monitoring platform based on application system portrait and its implementation method

Abstract

The present invention discloses a unified monitoring platform based on application system portrait and an implementation method, which belongs to the field of monitoring management of multiple cloud platforms. The technical problem to be solved by the present invention is that the monitoring systems accumulated due to repeated construction for historical reasons are scattered, the monitoring data are of many types, the format is not unified, and there is a lack of unified view display, which cannot assist operation and maintenance personnel in fault analysis. The technical solution is: the monitoring platform includes a data extraction processing module, which is used to directly collect data from the original monitoring system or business system and process it; an application system portrait module, which is used for system portrait, and the system portrait processing data is displayed on the Web end of the monitoring platform according to business needs; a data collection module, which is used to cover the monitoring data of the original cloud platform manufacturers, and the data not included in the original cloud platforms are newly collected according to the specifications; a data standardization processing module, which is used to realize the access, analysis, screening, filtering and storage of log data and indicator data.

Description

Technical Field

[0001] The present invention relates to the field of monitoring and management of multiple cloud platforms, and in particular to a unified monitoring platform based on application system portraits and an implementation method thereof. Background Art

[0002] The construction of the government cloud includes public clouds for external use and private clouds for internal office use. During the construction of the government cloud, many cloud management platforms are purchased from various cloud service providers, and the corresponding resource management and monitoring platforms are also built and provided by various cloud service providers. There are both general basic monitoring, such as hardware, network, storage resource monitoring, etc., and special monitoring, such as business system monitoring, database monitoring, and various middleware monitoring. Each system has an independent management and display method, resulting in a lack of a unified view to summarize various types of data when a fault occurs, and it is impossible to effectively assist operation and maintenance personnel in fault analysis. Operation and maintenance personnel cannot intuitively judge the correlation between abnormal indicators, and need to check each system repeatedly, manually integrate data and analyze and judge, which is inefficient. Summary of the invention

[0003] The technical task of the present invention is to provide a unified monitoring platform and implementation method based on application system portraits to solve the problem that the monitoring systems accumulated due to repeated construction due to historical reasons are scattered, the monitoring data are of many types, the format is not unified, there is a lack of unified view presentation, and it is impossible to effectively assist operation and maintenance personnel in fault analysis.

[0004] The technical task of the present invention is achieved in the following manner: a unified monitoring platform based on application system portraits, the monitoring platform includes:

[0005] Data extraction and processing module, used to directly collect data from the original monitoring system or business system and process it;

[0006] The application system portrait module is used for system portrait. The system portrait processing data is displayed on the monitoring platform Web according to business needs, and sent to the alarm work order system according to business definitions;

[0007] The data collection module is used to cover the monitoring data of the original cloud platform vendors. The data not included in the original cloud platforms are newly collected according to the specifications;

[0008] The data standardization processing module is used to access, parse, screen, filter and store log data and indicator data.

[0009] Preferably, the working process of the data extraction processing module is as follows:

[0010] (1) Extract data from the original monitoring system through the data acquisition interface;

[0011] (2) Data collection, formatting and classification storage are performed on the extracted data, and data cleaning, deduplication and compression are performed;

[0012] (3) Signature processing of various types of data;

[0013] (4) Send the processed data to the application system portrait model for processing;

[0014] (5) Analyze and summarize the data based on business and time dimensions;

[0015] (6) The processed data is sent to the alarm work order system for alarm judgment and analysis, and displayed on the unified monitoring platform page according to the monitoring business requirements.

[0016] Preferably, the application system portrait module uses the collected information to perform classification and labeling, as follows:

[0017] Combine the asset data and business call relationship data of CMDB for association processing: The details are as follows:

[0018] By collecting asset configuration data and business relationship data from CMDB, we can obtain the multi-level topological relationship of the business system.

[0019] Associate key indicators of different objects with specific nodes of the business topology;

[0020] Through the real-time aggregation calculation of the stream processing framework, various key indicator data on the business topology map are finally gathered, making the report configurable and the monitoring scenario customized, thus getting rid of the dependence of system developers on changes in monitoring scenarios;

[0021] After modeling, labeling is performed: various hardware and software are labeled according to physical attributes, configuration attributes, operation data attributes, operation and maintenance data attributes and intelligent attributes to form an application portrait system; among them, hardware includes networks, cloud servers and cloud service physical hosts; software includes operating systems, databases and middleware.

[0022] Preferably, the labeling process is divided into manual labeling and machine labeling;

[0023] Among them, manual labeling is to set the label dictionary according to operation and maintenance experience and specifications, and is formed by querying the data association dictionary when collecting data. For example, the equipment manufacturer information is a physical and unmodifiable attribute, and the IP address of the cloud server is a configuration attribute;

[0024] Machine labeling is done by labeling the relationship between performance data and application access. For example, the device (software) is extracted and clustered based on log information, performance data, and failure frequency, and then labeled, such as stability and access frequency. Machine labeling is corrected manually.

[0025] Preferably, the labeled data is sorted by label, number, and value, and aggregated and normalized according to the analysis scenario to form ordered and standardized data.

[0026] Preferably, the Web client uses basic tables, reports and Echart components to display data in customized scenarios, providing intuitive, vivid, interactive and highly customizable data visualization charts;

[0027] Among them, data visualization charts support the mixing of charts and graphs, and apply relevant data to precipitate analytical reports of various dimensions; data visualization charts include line charts, bar charts, scatter charts, pie charts, K-line charts, box plots for statistics, maps for geographic data visualization, heat maps and line charts, relationship diagrams for data relationship visualization, treemaps, parallel coordinates for multidimensional data visualization, funnel charts and dashboards for BI.

[0028] Preferably, the information collected by the data collection module includes basic command information, platform information and middleware information; the basic command information includes curl, ping and snmp; the platform information includes openstack, vmware and ceph; the middleware information includes oracle, mysql and Nginx.

[0029] A method for implementing a unified monitoring platform based on application system portraits, the steps of the implementation method are as follows:

[0030] S1. Define standard interfaces to achieve data standardization and display standardization of existing cloud platforms; specifically: achieve unified monitoring and management of multiple platforms in the government cloud by defining unified interfaces and data collection; including comprehensive monitoring and management of IT resources such as load balancing devices, network devices, servers, storage devices, security devices, databases, middleware and application software;

[0031] S2. According to customized standards, Inspur's data lake products are connected with the data of existing cloud platforms to collect historical data of cloud platforms that need to be supervised. The collected data includes basic platform information and software and hardware performance data; the data is distributed to the application system portrait module and the alarm work order system for processing;

[0032] S3. Deploy the application system profiling module to aggregate and calculate the collected data and label it. After labeling, the data is distributed to the data standardization processing module and the alarm work order system.

[0033] S4. Deploy the web terminal to achieve unified display of data.

[0034] As a preferred option, we use the open source component skywalking and Inspur's data lake product to collect massive amounts of call chain log information and store it in Elasticsearch. We use the stream processing framework to analyze the upstream and downstream relationships of the call chain logs and the aggregate calculation of key indicators in real time. Finally, we use Echarts to customize the front end to display the call chain topology relationship and support drilling down to specific services to view key indicator trends and alarm information.

[0035] A computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the method for implementing a unified monitoring platform based on application system portraits as described above is implemented.

[0036] The unified monitoring platform based on application system portrait and implementation method of the present invention have the following advantages:

[0037] (1) The present invention is mainly used to solve the problems of duplicate construction, scattered existing systems, single monitoring indicator analysis that cannot perform business correlation analysis, and insufficient scenario analysis, so as to achieve unified monitoring and efficient operation and maintenance;

[0038] (ii) The present invention forms a unified view of data scattered in multiple systems, establishes an indicator analysis from a business perspective, solves the problem of single display of indicators, realizes in-depth interactive analysis of indicators, and analyzes scenario customization and scenario solidification functions; uses mature data visualization tools and business topology diagrams to realize a unified monitoring view of cross-system indicators, realizes the association of business indicators and basic indicators through business system portraits, realizes in-depth interactive analysis capabilities by calling link tracking, and realizes customized monitoring in proprietary scenarios;

[0039] (III) The present invention displays the key indicators of different systems in a unified interface, and associates real-time alarm data to provide alarm prompts for business nodes. It realizes the centralized control and management of multi-system indicator data, enriches the monitoring indicator system, supports end-to-end monitoring from the perspective of technical architecture and business processes, and provides all-round three-dimensional monitoring from business, application, function, and system levels, so that operation and maintenance personnel can quickly obtain information that can be used for fault (alarm) processing decisions, so that system faults can be restored as soon as possible, and operation and maintenance efficiency can be improved;

[0040] (IV) The unified monitoring platform in the present invention realizes end-to-end monitoring through unified data control and display, and realizes all-round three-dimensional monitoring from business to application, function, and system level; the network resources, hardware resources, software resources, database resources, etc. involved in all business systems are included in the unified operation and maintenance monitoring platform, and by eliminating the differences in management software and data collection methods, various data sources are unified in management, standardization, processing, display, user login, and authority control, and finally realize standardized, automated, and intelligent large-scale operation and maintenance management;

[0041] (V) The unified web view realizes standardized display and is customizable. Operation and maintenance personnel do not need to switch back and forth between multiple systems to manually obtain data, which can shorten the time for locating problems.

[0042] (VI) For the application system portrait module, the business system portrait is implemented according to business needs, integrating the business architecture topology and key business monitoring indicators, and displaying the data in a centralized and related manner. Based on the system-component-server, one-click context-related query of operation and maintenance data is performed, and trend comparison is performed to achieve end-to-end all-round and three-dimensional monitoring;

[0043] (VII) The present invention realizes complete single call tracking, visualized call relationship statistics display and real-time dynamic association with performance indicators, fast retrieval of single call chain logs, and can realize drill-down analysis of abnormal call chain information with front-end customized development. Drill-down can display the call chain detail log of key indicator abnormalities. The call chain information displayed in time series can intuitively find the node with call abnormality;

[0044] (VIII) The present invention customizes monitoring scenarios in the government cloud and conducts portrait analysis of business systems, thereby realizing centralized management and governance of multi-system indicator data, enriching the monitoring indicator system, and supporting end-to-end monitoring from the perspective of technical architecture and business processes, and all-round three-dimensional monitoring from business, to application, to function, and then to the system level. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The present invention is further described below in conjunction with the accompanying drawings.

[0046] Attached Figure 1 This is a schematic diagram of a unified monitoring platform based on application system portraits;

[0047] Attached Figure 2 Schematic diagram of the working process of the data extraction processing module. DETAILED DESCRIPTION

[0048] The unified monitoring platform based on application system profiling and the implementation method of the present invention are described in detail below with reference to the accompanying drawings and specific embodiments of the specification.

[0049] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0050] Embodiment 1:

[0051] As attached Figure 1 As shown, the unified monitoring platform based on application system portrait of the present invention includes:

[0052] Data extraction and processing module, used to directly collect data from the original monitoring system or business system and process it;

[0053] The application system portrait module is used for system portrait. The system portrait processing data is displayed on the monitoring platform Web according to business needs, and sent to the alarm work order system according to business definitions;

[0054] The data collection module is used to cover the monitoring data of the original cloud platform vendors. The data not included in the original cloud platforms are newly collected according to the specifications;

[0055] The data standardization processing module is used to access, analyze, screen, filter and store log data and indicator data. It has the advantages of adapting to various data sources such as log files, system events, indicators, databases, etc. The collected data is formatted and stored in the database of the unified monitoring platform. The network resources, hardware resources, software resources, database resources, etc. involved in all business systems are included in the unified operation and maintenance monitoring platform, and by eliminating the differences in management software and data collection methods, unified management, unified standardization and unified processing are achieved for various data sources.

[0056] As attached Figure 2 As shown, the working process of the data extraction processing module in this embodiment is as follows:

[0057] (1) Extract data from the original monitoring system through the data acquisition interface;

[0058] (2) Data collection, formatting and classification storage are performed on the extracted data, and data cleaning, deduplication and compression are performed;

[0059] (3) Signature processing of various types of data;

[0060] (4) Send the processed data to the application system portrait model for processing;

[0061] (5) Analyze and summarize the data based on business and time dimensions;

[0062] (6) The processed data is sent to the alarm work order system for alarm judgment and analysis, and displayed on the unified monitoring platform page according to the monitoring business requirements.

[0063] The application system profiling module in this embodiment uses the collected information to perform classification and labeling processing, as follows:

[0064] (I) Combine CMDB's asset data and business call relationship data for correlation processing: The details are as follows:

[0065] ①. Obtain the multi-level topological relationship of the business system by collecting the asset configuration data and business relationship data of CMDB;

[0066] ②Associate the key indicators of different objects to specific nodes of the business topology;

[0067] ③. Through the real-time aggregation calculation of the stream processing framework, various key indicator data on the business topology map are finally gathered, making the report configurable and the monitoring scene customized, getting rid of the dependence of system developers on changes in monitoring scenes;

[0068] (ii) Labeling after modeling: Labeling the application portrait system for various hardware and software according to physical attributes, configuration attributes, operation data attributes, operation and maintenance data attributes and intelligent attributes; hardware includes network, cloud server and cloud service physical host; software includes operating system, database and middleware.

[0069] The labeling process in this embodiment is divided into manual labeling and machine labeling;

[0070] Among them, manual labeling is to set the label dictionary according to operation and maintenance experience and specifications, and is formed by querying the data association dictionary when collecting data. For example, the equipment manufacturer information is a physical and unmodifiable attribute, and the IP address of the cloud server is a configuration attribute;

[0071] Machine labeling is done by labeling the relationship between performance data and application access. For example, the device (software) is extracted and clustered based on log information, performance data, and failure frequency, and then labeled, such as stability and access frequency. Machine labeling is corrected manually.

[0072] The labeled data is sorted by label, number and value, aggregated and normalized according to the analysis scenario to form orderly and standardized data.

[0073] The Web end in this embodiment uses basic tables, reports and Echart components to display data in customized scenarios, providing intuitive, vivid, interactive and highly customizable data visualization charts;

[0074] Among them, data visualization charts support the mixing of charts and graphs, and apply relevant data to precipitate analytical reports of various dimensions; data visualization charts include line charts, bar charts, scatter charts, pie charts, K-line charts, box plots for statistics, maps for geographic data visualization, heat maps and line charts, relationship diagrams for data relationship visualization, treemaps, parallel coordinates for multidimensional data visualization, funnel charts and dashboards for BI.

[0075] The information collected by the data collection module in this embodiment includes basic command information, platform information and middleware information; the basic command information includes curl, ping and snmp; the platform information includes openstack, vmware and ceph; the middleware information includes oracle, mysql and Nginx.

[0076] Embodiment 2:

[0077] The implementation method of the unified monitoring platform based on application system portrait of the present invention has the following steps:

[0078] S1. Define standard interfaces to achieve data standardization and display standardization of existing cloud platforms; specifically: achieve unified monitoring and management of multiple platforms in the government cloud by defining unified interfaces and data collection; including comprehensive monitoring and management of IT resources such as load balancing devices, network devices, servers, storage devices, security devices, databases, middleware and application software;

[0079] S2. According to customized standards, Inspur's data lake products are connected with the data of existing cloud platforms to collect historical data of cloud platforms that need to be supervised. The collected data includes basic platform information and software and hardware performance data; the data is distributed to the application system portrait module and the alarm work order system for processing;

[0080] S3. Deploy the application system profiling module to aggregate and calculate the collected data and label it. After labeling, the data is distributed to the data standardization processing module and the alarm work order system.

[0081] S4. Deploy the web terminal to achieve unified display of data.

[0082] Among them, the open source component skywalking and Inspur's data lake product are used to collect massive amounts of call chain log information and store it in Elasticsearch. The stream processing framework is used to analyze the upstream and downstream relationships of the call chain logs and the aggregate calculation of key indicators in real time. Finally, the Echarts custom-developed front end is used to display the call chain topology relationship, and it supports drilling down to specific services to view key indicator trends and alarm information.

[0083] Embodiment 3:

[0084] There are more than ninety business systems and platforms managed by a provincial department, involving multiple departments and units. Each business system has a wide range of users, wide coverage, high importance, and many collaborative units. It uses multiple cloud service providers such as Inspur, Ping An Cloud, and Ziguang Government Cloud. It has the characteristics of complex management, wide technical fields, and high security requirements. These scenarios have increased the difficulty and tediousness of unified system operation and maintenance, problem location, problem solving, stable system operation, and developer problem repair. During the use of multiple operation and maintenance monitoring platforms, fault location often requires multi-person collaborative meeting troubleshooting. The data visualization tool in the fault location scenario optimizes the collaborative work method during troubleshooting. Through the unified and centralized visualization view in this system, the scope of the problem is quickly narrowed, making troubleshooting more targeted, reducing the time of blind arguments, improving operation and maintenance efficiency, and improving operation and maintenance quality and satisfaction.

[0085] Through the real-time collection of business, application, service, equipment and other indicators, all-round and three-dimensional monitoring data is formed. The visualization tools display this multi-dimensional information in a panoramic view, which improves the efficiency of locating problems and shortens the time for troubleshooting.

[0086] In the process of operation and maintenance management, it can effectively solve the problems encountered by users in the application process according to their needs and strengthen the management and control of the business platform. Its value is mainly reflected in the following aspects:

[0087] ①. Accurate positioning: Based on the analysis of multiple dimensions such as the topology structure, call chain, and transactions within the application, it helps users quickly understand the application running status and locate the root cause of the problem.

[0088] ②. AIOps: AI conducts autonomous detection of thresholds, and based on anomaly detection algorithms and learning of baseline performance data, provides timely performance warnings and performance bottleneck predictions, providing data support for users to optimize resources.

[0089] Embodiment 4:

[0090] The embodiment of the present invention further provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions are loaded by a processor, so that the processor executes the implementation method of the unified monitoring platform based on the application system profile in any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, on which the software program code that implements the functions of any of the above embodiments is stored, and the computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0091] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0092] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer by a communication network.

[0093] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0094] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A unified monitoring platform based on application system portrait, characterized in that: The monitoring platform includes: Data extraction and processing module, used to directly collect data from the original monitoring system or business system and process it; The application system portrait module is used for system portrait. The system portrait processing data is displayed on the Web side of the monitoring platform according to business needs; The data collection module is used to cover the monitoring data of the original cloud platform vendors. The data not included in the original cloud platforms are newly collected according to the specifications; Data standardization processing module, used to access, analyze, screen, filter and store log data and indicator data; Among them, the working process of the data extraction processing module is as follows: (1) Extract data from the original monitoring system through the data acquisition interface; (2) Perform data collection, formatting and classification storage on the extracted data, and perform data cleaning, deduplication and compression processing; (3) Signature processing of various types of data; (4) Send the processed data to the application system portrait model for processing; (5) Analyze and summarize the data based on business and time dimensions; (6) Send the processed data to the alarm work order system for alarm judgment and analysis, and display it on the unified monitoring platform page according to monitoring business requirements; The application system portrait module uses the collected information to classify and label it, as follows: Combine the asset data and business call relationship data of CMDB for association processing: The details are as follows: By collecting asset configuration data and business relationship data from CMDB, we can obtain the multi-level topological relationship of the business system. Associate key indicators of different objects with specific nodes of the business topology; Through real-time aggregation calculations of the stream processing framework, various key indicator data on the business topology map are finally gathered; After modeling, labeling is performed: the application portrait system is labeled for various hardware and software according to physical attributes, configuration attributes, operation data attributes, operation and maintenance data attributes, and intelligent attributes; hardware includes networks, cloud servers, and cloud service physical hosts; software includes operating systems, databases, and middleware; Labeling processing is divided into manual labeling and machine labeling; Among them, manual labeling is to set the label dictionary based on operation and maintenance experience and specifications, and to query the data association dictionary when collecting data; Machine labeling is done by labeling performance data and application access associations. Machine labeling is corrected manually. The labeled data is sorted by label, number and value, and aggregated and normalized according to the analysis scenario to form orderly and standardized data; The Web end uses basic tables, reports and Echart components to display data in customized scenarios, providing intuitive, vivid, interactive and highly customizable data visualization charts; Among them, data visualization charts support the mixing of charts and graphs, and apply related data to precipitate analytical reports of various dimensions; data visualization charts include line charts, bar charts, scatter charts, pie charts, K-line charts, box charts for statistics, maps for geographic data visualization, heat maps and line charts, relationship diagrams for data relationship visualization, treemaps, parallel coordinates for multidimensional data visualization, funnel charts and dashboards for BI; The information collected by the data collection module includes basic command information, platform information and middleware information; basic command information includes curl, ping and snmp; platform information includes openstack, vmware and ceph; middleware information includes oracle, mysql and Nginx.

2. The method for implementing the unified monitoring platform based on application system portraits as described in claim 1 is characterized in that: The implementation steps are as follows: S1. Define standard interfaces to achieve data standardization and display standardization of existing cloud platforms; S2. According to customized standards, Inspur's data lake products are connected with the data of existing cloud platforms to collect historical data of cloud platforms that need to be supervised. The collected data includes basic platform information and software and hardware performance data; the data is distributed to the application system portrait module and the alarm work order system for processing; S3, deploy the application system profiling module to aggregate and label the collected data; After data is labeled, it is distributed to the data standardization processing module and the alarm work order system; S4. Deploy the web terminal to achieve unified display of data; Among them, the open source component skywalking and Inspur's data lake product are used to collect massive amounts of call chain log information and store it in Elasticsearch. The stream processing framework is used to analyze the upstream and downstream relationships of the call chain logs and the aggregate calculation of key indicators in real time. Finally, the Echarts custom-developed front end is used to display the call chain topology relationship, and it supports drilling down to specific services to view key indicator trends and alarm information.

3. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer execution instructions. When the processor executes the computer execution, the implementation method of the unified monitoring platform based on application system profiling as described in claim 2 is implemented.

Images