A visual processing method and device of an application monitoring link and a storage medium

Abstract

The present disclosure relates to a visual processing method and device of an application monitoring link and a storage medium. The method comprises: processing acquired monitoring data according to a pre-configured alarm rule, judging whether the monitoring data triggers alarm information, the alarm information containing a tracking identifier uniquely corresponding to a single complete monitoring link; if the alarm information is triggered, acquiring the complete monitoring link triggering the alarm information according to the tracking identifier in the alarm information; and visually displaying the complete monitoring link corresponding to the alarm information. The embodiment of the present disclosure labels and associates data from the source of collection to the whole process of data processing to the target data generation. Compared with tracking nodes, tracking data can more accurately analyze the abnormal reason, and the backtracking efficiency is higher, realizing backtracking of the complete monitoring link and rapid fault judgment.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of application monitoring, in particular to a method and device for visualizing an application monitoring link and a storage medium, to realize visualizing an application monitoring link. BACKGROUND

[0002] When an application system is online, its running state usually needs to be monitored to find abnormalities in the running process in time.

[0003] The commonly used application monitoring solutions in the prior art mainly include Prometheus and Zabbix, both of which use a database to store collected data, so their performance and functions are limited. The relational database used by Zabbix greatly limits its collection performance, and although Prometheus uses a self-developed high-performance time series database, it can achieve 10 million data storage per second, but it does not support storage and query of a large amount of historical data. In addition, the existing application monitoring solutions have limited and non-expandable data types that can be collected, processed and stored. Prometheus can only collect index type data and only supports performance and availability monitoring, and does not have functions such as log monitoring. Zabbix is more suitable for monitoring physical machine environments and has limited monitoring for cloud environments. The existing application monitoring solutions all analyze and arrange the collected data, and in the case of meeting the alarm rules, trigger an alarm. This process establishes a one-way link from the monitoring source data (data input into the application monitoring system) to the alarm information (output from the application monitoring system), and abnormal index data will trigger an alarm. Through the alarm information, the operation and maintenance personnel can know the abnormal situation of the application system. However, the same abnormal situation may be caused by different abnormalities, and only according to the one-way path established by the existing application monitoring solutions, it is impossible to trace back the monitoring source data from the alarm information, thereby causing difficulties in quickly troubleshooting and solving application problems. SUMMARY

[0004] The present disclosure provides a method, device and storage medium for visualizing an application monitoring link, which can ensure normal operation of the monitoring link process, and realize traceability and visualization of the data link by marking and establishing a correlation between the source data and the target data. The technical solution of the present disclosure includes the following:

[0005] According to a first aspect of the embodiments of the present disclosure, a method for visualizing an application monitoring link is provided, comprising:

[0006] According to the pre-configured alarm rule, the monitoring data obtained is processed to determine whether the monitoring data triggers an alarm information, and the alarm information contains a tracking identifier uniquely corresponding to a single complete monitoring link;

[0007] If the alarm information is triggered, a complete monitoring link triggering the alarm information is obtained according to a tracking identifier in the alarm information; the complete monitoring link includes a monitoring link from monitoring source data to the alarm information, in the complete monitoring link, a conversion node divided from a node is determined according to conversion information of the node, and a uniform data storage object is used to record a conversion rule and an association relationship between monitoring data of a current conversion node and a next conversion node in a single complete monitoring link, the association relationship includes reason data of conversion of the monitoring data between the conversion nodes;

[0008] The complete monitoring link corresponding to the alarm information is visually displayed.

[0009] In another embodiment of the method, the monitoring data obtained is processed according to the pre-configured alarm rule to determine whether the monitoring data triggers the alarm information, including:

[0010] The alarm rule configured by a user in a configuration platform front end is obtained;

[0011] The configuration platform back end is controlled to store configuration information including the configuration rule into a first database, and synchronize the configuration information to a second database, the first database is of a relational type, and the second database is of a column type database management system;

[0012] The configuration platform back end is controlled to generate a distributed timing task according to the configuration information, the distributed timing task is used to collect monitoring source data at a timing, and a data association object is stored into a distributed message system after being generated based on the collected monitoring source data;

[0013] The data association object in the distributed message system is real-time monitored, and when the data association object is consumed, the monitoring data obtained is processed to determine whether the monitoring data triggers the alarm information.

[0014] In another embodiment of the method, the data association object in the distributed message system is real-time monitored, and when the data association object is consumed, the monitoring data obtained is processed, including:

[0015] Based on an analysis result of the data association object, an association relationship object is constructed, and the association relationship object, source data or result data are respectively converted into a uniform data transfer object;

[0016] The data transfer object is stored into a data storage topic in the distributed message system;

[0017] When the data correlation object contains result data, extracting the result data recorded in the data correlation object, the name of the subject, converting the result data into a data entity object and storing the data entity object in the corresponding subject in the distributed message system; the subject includes business data categories and alarm message categories, and different categories of subjects have respective subject names;

[0018] Real-time monitoring of the data storage subject in the distributed message system, and when the data relay object is consumed, storing information associated with the consumption of the data relay object in the second database.

[0019] Another embodiment of the method further includes:

[0020] Real-time monitoring of the subject under the business data category in the distributed message system;

[0021] When the data entity object is consumed, obtaining result data obtained after the data entity object is executed by a business logic process;

[0022] Generating a data correlation object of the next conversion node based on the result data, and storing the data correlation object in the correlation relationship subject in the distributed message system.

[0023] Another embodiment of the method further includes:

[0024] Real-time monitoring of the subject under the alarm message category in the distributed message system;

[0025] The trigger alarm information includes: when the data relay object is consumed, if the monitoring data meets the alarm rule, the alarm information is triggered;

[0026] The visual display of the complete monitoring link corresponding to the alarm information includes: drawing the complete monitoring link of the alarm information; and visually displaying the drawn complete monitoring link.

[0027] Another embodiment of the method, the drawing of the complete monitoring link of the alarm information includes:

[0028] Based on the tracking identifier corresponding to the alarm information, the correlation relationship object involved in the monitoring link triggering the alarm information is obtained, including: starting from the starting node data correlation object of the monitoring link and traversing to the next conversion node until the sequence formed by the data correlation object with empty result data is obtained as the complete monitoring link of the alarm information.

[0029] Another embodiment of the method further includes:

[0030] Synchronizing the configuration information stored in the first database at regular intervals.

[0031] In another embodiment of the method, the data of any node in the link is recorded in the form of a binary tuple (u, v) to record the source data of the data flowing to the current node and the data of the current node.

[0032] According to a second aspect of the embodiments of the present disclosure, a visual processing device for monitoring a link is also provided, and the device comprises:

[0033] A monitoring data acquisition module is configured to process the acquired monitoring data according to a pre-configured alarm rule, and determine whether the monitoring data triggers an alarm information, wherein the alarm information contains a tracking identifier uniquely corresponding to a single complete monitoring link.

[0034] A link analysis processing module is configured to, if the alarm information is triggered, acquire a complete monitoring link triggering the alarm information according to the tracking identifier in the alarm information; the complete monitoring link comprises a monitoring link from monitoring source data to the alarm information, in the complete monitoring link, a conversion node divided from a node is determined according to conversion information of the monitoring data in the node, and a unified data storage object is used to record a conversion rule and an associated relationship between the monitoring data of the current conversion node and the next conversion node in a single complete monitoring link, and the associated relationship comprises reason data for conversion of the monitoring data between the conversion nodes.

[0035] A link display module is configured to visually display the complete monitoring link corresponding to the alarm information.

[0036] According to a third aspect of the embodiments of the present disclosure, a computer device is also provided, and the computer device comprises:

[0037] At least one processor;

[0038] A memory for storing instructions executable by the processor;

[0039] The processor is configured to execute the instructions to implement the method according to any one of the first aspect of the present disclosure.

[0040] According to a fourth aspect of the embodiments of the present disclosure, a computer readable storage medium is also provided, and when instructions in the computer readable storage medium are executed by a processor of an electronic device, the electronic device can execute the method according to any one of the first aspect of the present disclosure.

[0041] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is also provided, and the computer program product comprises a computer program, and the computer program is executed by a processor to implement the method according to any one of the first aspect of the present disclosure.

[0042] The technical solutions provided by the embodiments of the present disclosure at least bring the following beneficial effects:

[0043] In the embodiment of the present disclosure, the unified data storage object is used to record the monitoring data of different nodes in the complete monitoring link from the monitoring data source to the alarm information, and the association between the monitoring data and the next node data. The data in the node is associated with a unique tracking ID (Identity document, identification or identification), and the serial number state is continuously updated with the stage conversion of the monitoring link. The serial number uniquely records the node where the monitoring data is located and the change reason, which can be used for tracing the complete monitoring link and quickly determining the fault. From a new angle, the present disclosure marks the nodes for backtracking on the conventional monitoring link, and marks and associates the data from the source of collection to the entire process of target data generation. Compared with tracking the nodes, tracking the data can more accurately analyze the abnormal reason, and the backtracking efficiency is higher, which realizes tracing the complete monitoring link and quickly determining the fault.

[0044] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0045] The accompanying drawings incorporated in the specification and forming a part of it illustrate embodiments consistent with the present disclosure and together with the description serve to explain the principles of the present disclosure, and do not limit the present disclosure.

[0046] Figure 1 FIG. 1 is a flowchart of a visual processing method of a monitoring link according to an exemplary embodiment.

[0047] Figure 2 FIG. 1 is a flowchart of a visual processing method of a monitoring link according to an exemplary embodiment.

[0048] Figure 3 FIG. 1 is a flowchart of a visual processing method of a monitoring link according to an exemplary embodiment.

[0049] Figure 4 FIG. 1 is a flowchart of a visual processing method of a monitoring link according to an exemplary embodiment.

[0050] Figure 5 FIG. 1 is a schematic diagram of the implementation principle of a data acquisition microservice in the method provided by the present disclosure.

[0051] Figure 6 FIG. 1 is a schematic block diagram of a visual processing device of a monitoring link according to an exemplary embodiment.

[0052] Figure 7is a schematic diagram of an internal structure of a computer device according to an exemplary embodiment. DETAILED DESCRIPTION

[0053] In order for those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings.

[0054] It should be noted that the terms "first", "second", and the like in the specification and claims of the present disclosure and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Rather, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims. The terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, product or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or further includes elements inherent to such a process, method, product or device. Without more limitations, it does not exclude the presence of other same or equivalent elements in the process, method, product or device including the elements. For example, if the words first, second, etc. are used to represent names, they do not represent any particular order.

[0055] It should also be noted that the user information (including but not limited to user device information, user personal information, collected device information, etc.) and data (including but not limited to data for display, analyzed data, etc.) involved in the present disclosure are all information and data authorized by the user or authorized by all parties.

[0056] The processing flow of the application monitoring link visualization processing method provided by the embodiments of the present application can be as follows Figure 1The whole technical solution can include data storage, implementation of the data storage module, implementation of the data association module, and visualization of the monitoring link. One of the innovations of the present disclosure is that, under the condition of ensuring the normal operation of the monitoring link process, the traceability and visualization of the data link are realized by marking the source data and the target data and establishing the association relationship. From a new angle, the present disclosure implementation marks the nodes of the backtracking on the existing monitoring link, marks and associates the whole process from the source of the collected data to the target data generated by the data processing, and the tracking of the data can more accurately analyze the abnormal reasons compared to the tracking of the nodes, and the efficiency of the backtracking is also higher. The terminal involved in the present embodiment can include but is not limited to various personal computers, notebook computers, smart phones, tablet computers, wearable devices, vehicle-mounted devices, medical devices, and the like. The server or system involved in the present disclosure can be a single server, a server cluster, a distributed system, a cloud processing platform, a server containing a blockchain node, and a combination of the above devices.

[0057] The present specification provides a visualization processing method of a monitoring link. In the complete monitoring link from the source data to the alarm information, the monitoring data is divided into different nodes (storing the current monitoring data and using a unique data ID for identification) according to the conversion of the monitoring data, the conversion rule and the association relationship between the current node monitoring data and the next node in the link are recorded using a unified data storage object (the reason for the conversion of the monitoring data between the nodes is recorded). A unique tracking ID is associated with a single monitoring link, and the sequence of the node data ID formed in the complete monitoring link can be obtained according to the ID, which completely records the conversion process between the monitoring source data and the alarm information, and can be used for backtracking and rapid fault judgment. The present embodiment of the present disclosure will be described below in the implementation scene of the style migration of the video of the mobile terminal.

[0058] One implementation process of the visualization processing method of the monitoring link provided by the present disclosure is as follows. Figure 1The entire application monitoring system can analyze and process the obtained monitoring source data according to the alarm rules configured by the user, and then determine whether to trigger an alarm (the alarm information contains a tracking ID), obtain the complete monitoring link triggering the alarm based on the tracking ID in the alarm information, and visualize the complete monitoring link. The process involves a configuration platform, a data acquisition microservice, an alarm and monitoring link drawing microservice, and multiple sub-modules such as a data correlation analysis task, a business logic processing task, and a data storage task that can be implemented based on Flink (an open source stream processing framework). The implementation of the sub-modules and their interaction process can be realized through channels such as MySQL (an open source relational database), Redis (an open source key-value database), ClickHouse (an open source data analysis database), XXL-Job (an open source lightweight distributed scheduling platform), and Plusar (an open source distributed message system).

[0059] Figure 2 is a flowchart of a visualization processing method of an application monitoring link according to an example embodiment, as shown in Figure 2 The method comprises the following steps.

[0060] S202: According to the pre-configured alarm rule, the obtained monitoring data is processed to determine whether the monitoring data triggers an alarm information, and the alarm information contains a tracking identifier corresponding to a single complete monitoring link;

[0061] S204: If the alarm information is triggered, the complete monitoring link triggering the alarm information is obtained according to the tracking identifier in the alarm information; the complete monitoring link includes a monitoring link from monitoring source data to alarm information, and in the complete monitoring link, the conversion nodes divided from the nodes are determined according to the conversion information of the monitoring data in the nodes, and a unified data storage object is used to record the conversion rule and the correlation between the monitoring data of the current conversion node and the next conversion node in a single complete monitoring link, and the correlation includes reason data for conversion of the monitoring data between the conversion nodes;

[0062] S206: Visualize the complete monitoring link corresponding to the alarm information.

[0063] The specific flow of one embodiment can include the following steps (the description of the following examples or embodiments does not limit the scheme provided by the present disclosure to completely execute all or part of the steps or all execution actions in a certain step of an example or embodiment) :

[0064] 1. The user configures an alarm rule in the configuration platform front end (step 1 marked in Figure 1 );

[0065] 2. The configuration platform backend will store configuration information in MySQL and Redis, and synchronize the configuration information to ClickHouse (an open source data analysis database) Figure 1 Steps 2, 3, and 4 marked by Zhongbiao;

[0066] 3. The configuration platform backend generates XXL-Job timing tasks based on user configuration information Figure 1 Step 5 marked by Zhongbiao;

[0067] 4. The timing task calls the data acquisition microservice to collect monitoring source data at regular intervals, generates data association objects, and stores them in Pulsar (Topic: dataCombine) Figure 1 Steps 6 and 7 marked by Zhongbiao; Topic is a core concept in TDMQ Pulsar, usually used to centrally classify and manage various messages produced by the system, such as related messages to business data can be placed in a Topic for other consumers to subscribe to. In actual application scenarios, a Topic often represents a business aggregation, and developers decide how to design different Topics based on their own system design and data architecture design. For ease of description, Topic can be referred to as topic or queue;

[0068] 5. Flink (open source stream processing framework) data association analysis task real-time listens to Pulsar (Topic: dataCombine), consumes data association object messages Figure 1 Step 8 marked by Zhongbiao and performs the following operations:

[0069] 1) Analyze the data association object, construct the association relationship object, convert the association relationship object, source data or result data into a unified data transfer object, and store it in Pulsar (open source distributed message system) (Topic: dataStorage) Figure 1 Step 9 marked by Zhongbiao;

[0070] 2) Extract the result data and topic (including business data classification and alarm message classification) name recorded in the data association object, convert the result data to a data entity object and store it in Pulsar (Topic: topicName) Figure 1 Step 12 marked by Zhongbiao;

[0071] 6. Flink data storage task real-time listens to Pulsar (Topic: dataStorage), consumes data transfer object messages, and stores them in ClickHouse Figure 1 Steps 10 and 11 marked by Zhongbiao;

[0072] 7. Flink business logic processing task stores configuration information in Redis, real-time listens to Pulsar (business data Topic), consumes data entity object message, executes business logic processing on data entity object to obtain result data, and then generates data association object of next node and stores in Pulsar (Topic: dataCombine) Figure 1 Steps 13, 14 and 15 in the foregoing embodiment are shown in FIG. 13.

[0073] 8. Alarm and monitoring link drawing microservice real-time listens to Pulsar (alarm message classification Topic), consumes data transfer object message, issues alarm and draws monitoring link.

[0074] Figure 3 FIG. 13 is a flowchart showing part of steps in a method for visualizing application monitoring link according to an example embodiment. Figure 3 As shown in FIG. 13, the method comprises: processing the obtained monitoring data, and judging whether the monitoring data triggers alarm information according to a preconfigured alarm rule, which comprises:

[0075] S302: obtaining alarm rule configured by a user in a configuration platform front end;

[0076] S304: controlling the configuration platform back end to store configuration information including the configuration rule in a first database, and synchronizing the configuration information to a second database, wherein the first database is a relational database, and the second database is a columnar database management system;

[0077] S306: controlling the configuration platform back end to generate a distributed timing task according to the configuration information, wherein the distributed timing task is used to collect monitoring source data at a timing, and store data association object in a distributed message system after generating the data association object based on the collected monitoring source data;

[0078] S308: real-time listening to the data association object in the distributed message system, and processing the obtained monitoring data when the data association object is consumed, and judging whether the monitoring data triggers alarm information.

[0079] The first database can include the foregoing MySQL and Redis. The second database can include ClickHouse.

[0080] The data of each node is stored in the corresponding topic in a certain structure. Generally, the data structure has source data ID and result data. The Flink task consumes the message, assigns the ID, processes the result data, tracks the ID (the tracking ID of the source data when the source data is not empty, and assigns if not), establishes the association relationship during processing, and can store the association relationship in Pulsar. The data structure in some embodiments of the present disclosure includes the source data and the result data of different conversion nodes. If the source data is data whose source data is null, a tracking ID is assigned to the data when the data is stored.

[0081] The data message is consumed by the task of Flink, and the ID is assigned to the related data. The monitoring data of the current node is processed as result data. The result data with source data can use the tracking ID of the source data, and if the source data is null, an ID is assigned to the source data for tracking.

[0082] The tracking ID marked on the source data and the result data of different conversion nodes automatically generates the association relationship between the data. The association relationship is generated when the data message is consumed, and the processed data is further stored in another Topic of Pulsar. The relationship generated during processing is also stored at the same time.

[0083] Based on the examples and Figure 1 shown in the present disclosure, Figure 4 According to an example embodiment, a flowchart of a part of steps of a visual processing method of an application monitoring link is shown, as Figure 4 shown, the real-time monitoring of the data association object in the distributed message system is performed, and when the data association object is consumed, the obtained monitoring data is processed, including:

[0084] S402: Based on the analysis result of the data association object, an association relationship object is constructed, and the association relationship object, the source data or the result data is converted into a unified data transfer object;

[0085] S404: The data transfer object is stored in the data storage topic in the distributed message system;

[0086] S406: When the data association object contains result data, the result data recorded in the data association object and the name of the topic are extracted, the result data is converted into a data entity object, and the data entity object is stored in the corresponding topic in the distributed message system; the topics shown include business data classification and alarm message classification, and different classifications have respective topic names.

[0087] S408: Real-time monitoring of the data storage topic in the distributed message system, and when the data transfer object is consumed, storing information associated with the consumption of the data transfer object in the second database.

[0088] Based on the examples and Figure 1 As shown in the examples of the present disclosure, another embodiment of the method of the present disclosure can further include:

[0089] Real-time monitoring of the topic under the business data classification in the distributed message system;

[0090] When the data entity object is consumed, obtaining result data obtained after the business logic processing of the data entity object is performed;

[0091] Based on the result data, generating a data association object of the next conversion node, and storing the data association object in the association relationship topic in the distributed message system.

[0092] Based on the examples and Figure 1 As shown in the examples of the present disclosure, another embodiment of the method of the present disclosure can further include:

[0093] Real-time monitoring of the topic under the alarm message classification in the distributed message system;

[0094] The trigger alarm information includes: when the data transfer object is consumed, if the monitoring data satisfies the alarm rule, the alarm information is triggered;

[0095] The visual display of the complete monitoring link corresponding to the alarm information includes: drawing the complete monitoring link of the alarm information; and visually displaying the drawn complete monitoring link.

[0096] Based on the examples and Figure 1 As shown in the examples of the present disclosure, another embodiment of the method of the present disclosure, the drawing of the complete monitoring link of the alarm information can include:

[0097] Based on the tracking identifier corresponding to the alarm information, the association relationship object involved in the monitoring link triggering the alarm information is obtained, including: a sequence formed by traversing from the starting node data association object of the monitoring link to the next conversion node until the result data of the data association object is empty as the complete monitoring link of the alarm information.

[0098] Based on the examples and Figure 1 As shown in the examples of the present disclosure, another embodiment of the method of the present disclosure can further include:

[0099] Synchronization of the configuration information stored in the first database at a time.

[0100] In another embodiment of the method of the present disclosure, the data of any node in the link is monitored to record the source data of the data flowing to the current node and the data of the current node in the form of a two-tuple (u, v).

[0101] The source data of the data flowing to the current node and the data of the current node can be recorded in a unified data storage object in the form of a two-tuple (u, v). The data of any node in the link is completely monitored and recorded in the form of the two-tuple (u of the monitoring system source data and v of the alarm information are empty). Therefore, the complete monitoring link is recorded in the form of an undirected graph, and the monitoring source data can be traced back to the alarm information, and the alarm information can be traced back to the monitoring source data.

[0102] The configuration platform involved in the present disclosure is a terminal of the application monitoring system, which can be used to manage the configuration information of the application monitoring system. The front end of the configuration platform can be realized based on the open source components Vue and AntDesign, and the back end is realized based on the open source frameworks SpringCloud and SpringBoot. The front end of the monitoring platform provides a Web page for users to configure various monitoring tasks (such as the collection frequency of the data acquisition microservice, the alarm triggering condition, the recipient of the alarm information, etc.), and the back end of the monitoring platform will store the configuration information set by the user into the MySQL database for persistent storage, into the Redis cache for the Flink business logic processing task to read regularly, and into the ClickHouse for subsequent query and use when constructing data association objects. The monitoring configuration information is all constructed based on the configuration base class (see Table 1). When the configuration information changes, the latest configuration information is stored using MySQL, and all versions of the configuration information are recorded using ClickHouse.

[0103] Table 1 Configuration base class field description

[0104]

[0105]

[0106] Figure 5 is an implementation principle schematic diagram of a data acquisition microservice in the method provided by the present disclosure. The data acquisition microservice accepts the scheduling of a timing task, acquires the source data of the input application monitoring link from the data source, and the specific steps are as follows (as shown in Figure 5 ):

[0107] 1. Get the configuration information cfg;

[0108] 2. Collect raw data from multiple data sources and construct data entity object obj (based on entity base class implementation, field description see Table 2) through REST API (Application Insights, Log Analytics), API Server (Kubernetes) or JDBC (MySQL);

[0109] 3. Construct data association object dataCombineObj (with a unique traceId, field description see Table 3):

[0110] 1) dataComebineObj.origin = null

[0111] 2) dataComebineObj.config = cfg

[0112] 3) dataComebineObj.result = obj

[0113] 4) dataComebineObj.topic is set to the data source name

[0114] and stored in Pulsar (Topic: dataCombine).

[0115] Table 2 Entity base class field description

[0116]

[0117]

[0118] Table 3 Data association object field description

[0119]

[0120] The data association analysis task involved in the present disclosure can be implemented based on Flink, real-time listening to Pulsar (Topic: dataCombine), consuming data association object (dataCombineObj) messages and can perform the following analysis processing:

[0121] 1. Construct association relationship object connectionObj (field description see Table 4) according to dataComebineObj:

[0122] 1) connectionObj.traceId = dataCombineObj.traceId

[0123] 2) connectionObj.originId = dataCombineObj.origin.i (if dataCombineObj.origin is null, connectionObj.originId = null)

[0124] 3) connectionObj.configId = dataCombineObj.config.i (if dataCombineObj.config is null, connectionObj.configId = null)

[0125] 4) connectionObj.resultId = dataCombineObj.result.i (if dataCombineObj.result is null, connectionObj.resultId = null);

[0126] 2. Construct data transfer object transformDTO (see Table 5 for field description) according to connectionObj:

[0127] 1) transformDTO.traceId = connectionObj.traceId

[0128] 2) transformDTO = dataStorage

[0129] 3) transformDTO.obj = connectionObj

[0130] and store in Pulsar (Topic: dataStorage);

[0131] 3. If dataCombineObj.originNeedStorage = true, construct data transfer object transformDTO_origin according to dataCombineobj.origin:

[0132] 1) transformDTO_origin.traceId = dataCombineObj.traceId

[0133] 2) transformDTO_origin.topic = dataStorage

[0134] 3) transformDTO_origin.obj = dataCombineObj.origin

[0135] and store in Pulsar (Topic: dataStorage) ;

[0136] 4. If dataCombineObj.resultNeedStorage = true, construct a data transfer object transformDTO_result according to dataCombineobj.result:

[0137] 1) transformDTO_result.traceId = dataCombineObj.traceId

[0138] 2) transformDTO_result.topic = dataStorage

[0139] 3) transformDTO_result.obj = dataCombineObj.result

[0140] and store in Pulsar (Topic: dataStorage) ;

[0141] 5. Construct a data entity object dataObj according to dataCombineObj.result, and store in Pulsar (Topic: dataCombineObj.topic).

[0142] Table 4: Field description of association relationship object

[0143]

[0144]

[0145] Table 5: Field description of data transfer object

[0146]

[0147] The business logic processing task described in the present disclosure can be implemented based on Flink, and the following functions are mainly implemented:

[0148] 1. Synchronize the configuration information config_new stored in Redis at regular intervals;

[0149] 2. Real-time monitoring of Pulsar business data class Topic;

[0150] 3. Consumer service class data entity object message (dataObj);

[0151] 4. Perform service logic analysis on the data entity object to obtain result data result_new, and generate a data association object dataCombine_new of the next node:

[0152] 1) dataCombine_new.traceId = dataObj.traceId

[0153] 2) dataCombine_new.origin = dataObj

[0154] 3) dataCombine_new.config = config_new

[0155] 4) dataComebine_new.result = result_new

[0156] 5) dataComebine.topic = dataCombine

[0157] and stored in Pulsar (Topic: dataCombine).

[0158] The data storage task involved in the present disclosure can be implemented based on Flink, for real-time monitoring of Pulsar (Topic: dataStorage), consuming data transfer object messages and storing in ClickHouse database. The data storage task processes each piece of data, including checking whether the necessary fields id and dataTimestamp are correct, and setting insertTime. A ClickHouse Sink object is added to the Flink to store the processed data in ClickHouse. ClickHouse adopts a two-shard two-replica cluster mode, the database adopts an Atomic engine, the data table adopts a ReplicatedVersionedCollapsingMergeTree table engine, a sign field is added as a marker field, a version field is added as a version field, and a committed field is additionally added as a commit marker. A Distributed distributed table is created for operation. The ClickHouse Sink class realizes the EXACTLY_ONCE semantic based on the ReplicatedVersionedCollapsingMergeTree table engine, and the object fields sign=1, version=1, and committed=0 are set for pre-saving; sign=-1, version=1 are added for deleting the original saved data, sign=1, version=2, and committed=1 are added for saving the final data when a transaction is committed; and sign=-1, version=1 are added for deleting the original saved data when a transaction is rolled back.

[0159] The alarm and monitoring link mapping microservice involved in the present disclosure can real-time monitor Pulsar alarm message type Topic, consume alarm type data entity object messages, and send alarm information containing monitoring link tracking identifier (traceId) through email, telephone and other ways. Based on the traceId, the associated relationship objects involved in the monitoring link triggering the alarm can be obtained. The data association object with null originId is the starting node of the monitoring link, and the sequence formed by traversing from the current node to the data association object with null resultId is the complete monitoring link. According to the sequence, the open source Echarts component is used to realize monitoring link visualization.

[0160] In the embodiment scheme of the present disclosure, under the condition of ensuring the normal operation of the monitoring link process, the traceable and visualized data link can be realized by marking and establishing the association relationship between the source data and the target data, using the establishment and storage of the trace ID of the data and the establishment and storage of the association relationship between the data.

[0161] It can be understood that each embodiment of the above method is described in a progressive manner in the specification, and the same / similar parts between each embodiment can be referred to each other, and each embodiment mainly describes the difference from other embodiments, and the related part can be referred to the description of other method embodiments.

[0162] It should be understood that, although each step in the flowchart involved in the drawings is shown in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the drawings can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least part of other steps or other steps or stages.

[0163] It can be understood that the same / similar parts between each embodiment of the above method in the specification can be referred to each other, and each embodiment mainly describes the difference from other embodiments, and the related part can be referred to the description of other method embodiments.

[0164] Based on the description of the above-mentioned application monitoring link visualization processing embodiment, the disclosure also provides an application monitoring link visualization processing device. The device can include a system (including a distributed system), software (application), module, component, server, client, etc. using the method described in the embodiments of the present specification, and combining the necessary implementation hardware. Based on the same innovative idea, the device in one or more embodiments provided by the embodiments of the present disclosure is described in the following embodiments. Since the implementation scheme of the device to solve the problem is similar to the method, the implementation of the specific device in the embodiments of the present specification can be referred to the implementation of the foregoing method, and the repeated part will not be described. The term "unit" or "module" used below can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware, or a combination of software and hardware implementation is also possible and conceived.

[0165] Figure 6 is a schematic block diagram of an application monitoring link visualization processing device according to an exemplary embodiment. The device can be a server, such as an application monitoring system or an alarm system, or a module, component, device, computer program product, etc. integrated in the server. For details, please refer to Figure 6 The device 100 can include:

[0166] The monitoring data acquisition module 602 can be configured to process the acquired monitoring data according to a preconfigured alarm rule, and determine whether the monitoring data triggers alarm information, wherein the alarm information contains a tracking identifier uniquely corresponding to a single complete monitoring link.

[0167] The link analysis processing module 604 can be configured to acquire a complete monitoring link triggering the alarm information according to the tracking identifier in the alarm information if the alarm information is triggered; the complete monitoring link includes a monitoring link from monitoring source data to the alarm information, and in the complete monitoring link, a conversion node divided from a node is determined according to conversion information of the monitoring data at the node, and a uniform data storage object is used to record a conversion rule and an association relationship between the monitoring data of a current conversion node and a next conversion node in a single complete monitoring link, and the association relationship includes reason data of conversion of the monitoring data between the conversion nodes.

[0168] The link display module 606 can be configured to visually display the complete monitoring link corresponding to the alarm information.

[0169] The modules in the apparatus can be all or partially implemented by software, hardware and combinations thereof. The modules can be embedded in or independent of a processor in a computer device in a hardware form, or stored in a memory in a computer device in a software form, so as to be called and executed by a processor to perform operations corresponding to the modules.

[0170] In one embodiment, a computer device is provided, which can be a server, and an internal structure diagram of the computer device can be as shown in Figure 7 The computer device includes a processor, a memory and a network interface connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a plurality of databases. The internal memory provides an environment for running of the operating system and the computer program in the non-volatile storage medium. The plurality of databases of the computer device are configured to store various association object data. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement steps of any method embodiment of the present disclosure.

[0171] Those skilled in the art can understand that Figure 7 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. A specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0172] In an exemplary embodiment, a computer readable storage medium, for example, a nonvolatile storage medium, including instructions, which can be executed by a processor to complete the above method, is also provided. The storage medium can be a computer readable storage medium, for example, the computer readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, a graphene storage device, etc.

[0173] In an exemplary embodiment, a computer program product including instructions, which can be executed by a processor of a computer device to complete the above method, is also provided.

[0174] In an exemplary embodiment, a computer program product including instructions, which can be executed by a processor of a computer device to complete the above method, is also provided.

[0175] Each of the embodiments in the specification is described in a progressive manner, and the same and similar parts of each of the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, for the hardware + program type embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0176] It should be noted that the above-mentioned apparatus, computer device, storage medium, etc. according to the description of the method embodiment can also include other embodiments, and the specific implementation manner can be referred to the description of the related method embodiment. Meanwhile, the features of the mutual combination of each method and the apparatus, device, server embodiment constitute a new embodiment, which still belongs to the implementation range covered by the present disclosure, and is not described here.

[0177] For the convenience of description, the above apparatus is described as various modules respectively described in function. Of course, when one or more of the present specification is implemented, the functions of each module can be implemented in the same or more software and / or hardware, or the modules implementing the same function can be combined to realize a combination of sub-modules or sub-units. The above-described apparatus embodiment is only schematic, for example, the division of modules or units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the coupling, communication connection, etc. between the displayed or described apparatus or units can be directly and / or indirectly coupled / connected in a manner, which can be through some standard or self-defined interface, protocol, etc., which is electric, mechanical or other form.

[0178] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. It is intended that the disclosure be construed as including any paterns, uses, or adaptations of the features disclosed herein and including modifications apparent to those skilled in the art with the ordinary skill and self-constructive aids to which the disclosure pertains. The specification and examples are to be regarded as exemplary only, with the true scope and spirit of the disclosure being indicated by the following claims.

[0179] It is to be understood that the present disclosure is not limited to the precise construction described above and shown in the attached drawings, and that various modifications and changes can be made without departing from the scope thereof.

Claims

1. A method for visualizing processing of an application monitoring link, characterized by, The method comprises the following steps: According to the pre-configured alarm rule, the obtained monitoring data is processed to determine whether the monitoring data triggers an alarm information, and the alarm information contains a tracking identifier uniquely corresponding to a complete monitoring link; If the alarm information is triggered, the complete monitoring link triggering the alarm information is obtained according to the tracking identifier in the alarm information; The complete monitoring link includes a monitoring link from monitoring source data to alarm information, and in the complete monitoring link, a conversion node is determined from a node according to the conversion information of the monitoring data, and a unified data storage object is used to record the conversion rule and the associated relationship between the monitoring data of the current conversion node and the next conversion node in a single complete monitoring link, and the associated relationship includes reason data for converting the monitoring data between conversion nodes; The complete monitoring link corresponding to the alarm information is visually displayed; The method comprises the following steps: Obtain the alarm rule configured by the user in the configuration platform front end; Control the configuration platform back end to store the configuration information including the alarm rule into a first database, and synchronize the configuration information to a second database, the type of the first database is a relational database, and the type of the second database is a columnar database management system; Control the configuration platform back end to generate a distributed timing task according to the configuration information, the distributed timing task is used to collect monitoring source data regularly, and a data association object is generated based on the collected monitoring source data and stored in a distributed message system; Real-time monitoring of the data association object in the distributed message system, when the data association object is consumed, the obtained monitoring data is processed to determine whether the monitoring data triggers an alarm information; The method comprises the following steps: Based on the analysis result of the data association object, an associated relationship object is constructed, and the associated relationship object, source data or result data are converted into a unified data transfer object; The data transfer object is stored in the data storage topic in the distributed message system; When the data association object contains result data, the result data recorded in the data association object and the name of the topic are extracted, the result data is converted into a data entity object and stored in the corresponding topic in the distributed message system; the topic includes business data classification and alarm message classification, and different classifications have their own corresponding topic names; Real-time monitoring of the data storage topic in the distributed message system, when the data transfer object is consumed, the information associated with the consumption of the data transfer object is stored in the second database.

2. The method of claim 1, wherein, The method further comprises the following steps: Real-time monitoring of the topic under the business data classification in the distributed message system; When the data entity object is consumed, the result data obtained after the data entity object is processed by business logic is obtained. Generate a data correlation object of the next conversion node based on the result data, and store the data correlation object into a correlation relationship topic in the distributed message system.

3. The method of claim 1, wherein, Further comprising: Real-time monitoring of the topics under the alarm message classification in the distributed message system; The trigger alarm information includes: when the data relay object is consumed, if the monitoring data meets the alarm rule, the alarm information is triggered; The visual display of the complete monitoring link corresponding to the alarm information includes: drawing the complete monitoring link of the alarm information; and visually displaying the drawn complete monitoring link.

4. The method of claim 3, wherein, The drawing of the complete monitoring link of the alarm information includes: Based on the tracking identifier corresponding to the alarm information, the correlation relationship objects involved in the monitoring link triggering the alarm information are obtained, including: a sequence formed by traversing from the starting node data correlation object of the monitoring link to the next conversion node until the data correlation object with empty result data is obtained as the complete monitoring link of the alarm information.

5. The method of claim 1, wherein, Further comprising: Synchronize the configuration information stored in the first database.

6. The method of claim 1, wherein, The data of any node in the monitoring link is recorded in the form of a binary tuple (u, v) to record the source data of the data flowing to the current node and the data of the current node.

7. An application monitoring link visualization processing apparatus, characterized by, Including: A monitoring data acquisition module, configured to process the acquired monitoring data according to a pre-configured alarm rule, and determine whether the monitoring data triggers alarm information, wherein the alarm information contains a tracking identifier uniquely corresponding to a single complete monitoring link; A link analysis processing module, configured to, if the alarm information is triggered, obtain the complete monitoring link triggering the alarm information according to the tracking identifier in the alarm information; The complete monitoring link includes the monitoring link from the monitoring source data to the alarm information, in which the conversion nodes are determined according to the conversion information of the nodes in the monitoring data, and a unified data storage object is used to record the conversion rule and the correlation relationship between the monitoring data of the current conversion node and the next conversion node in a single complete monitoring link, wherein the correlation relationship includes reason data for conversion of the monitoring data between conversion nodes; A link display module, configured to visually display the complete monitoring link corresponding to the alarm information; The visual processing device of the application monitoring link is used to obtain the alarm rule configured by the user in the front end of the configuration platform; The visual processing device of the application monitoring link is also used to control the configuration platform back end to store the configuration information including the alarm rule into the first database, and synchronize the configuration information to the second database, wherein the type of the first database is a relational database, and the type of the second database is a columnar database management system; The visual processing device of the application monitoring link is also used to control the configuration platform back end to generate a distributed timing task based on the configuration information, wherein the distributed timing task is used to collect monitoring source data at regular intervals, and store the data correlation object generated based on the collected monitoring source data into the distributed message system; The visual processing device of the application monitoring link is also used to control the configuration platform back end to generate a distributed timing task based on the configuration information, wherein the distributed timing task is used to collect monitoring source data at regular intervals, and store the data correlation object generated based on the collected monitoring source data into the distributed message system; The application further comprises a visual processing device for monitoring the link, which is configured to listen to data correlation objects in the distributed message system in real time, process the obtained monitoring data when the data correlation objects are consumed, and determine whether the monitoring data triggers an alarm information; The application further comprises a visual processing device for monitoring the link, which is configured to construct a correlation relationship object based on the analysis result of the data correlation objects, and convert the correlation relationship object, source data or result data into a unified data transfer object; The application further comprises a visual processing device for monitoring the link, which is configured to store the data transfer object in a data storage topic in the distributed message system; The application further comprises a visual processing device for monitoring the link, which is configured to extract result data recorded in the data correlation object and the name of the topic when the data correlation object contains the result data, convert the result data into a data entity object, and store the data entity object in a corresponding topic in the distributed message system; the topic includes business data classification and alarm message classification, and different classifications of topics have corresponding topic names; The application further comprises a visual processing device for monitoring the link, which is configured to listen to the data storage topic in the distributed message system in real time, and store information associated with the consumption of the data transfer object in the second database when the data transfer object is consumed.

8. A computer device, characterized in that, The application further comprises a visual processing device for monitoring the link, which is configured to listen to the data storage topic in the distributed message system in real time, and store information associated with the consumption of the data transfer object in the second database when the data transfer object is consumed. The application further comprises a visual processing device for monitoring the link, which is configured to listen to the data storage topic in the distributed message system in real time, and store information associated with the consumption of the data transfer object in the second database when the data transfer object is consumed. The application further comprises a visual processing device for monitoring the link, which is configured to listen to the data storage topic in the distributed message system in real time, and store information associated with the consumption of the data transfer object in the second database when the data transfer object is consumed. The application further comprises a visual processing device for monitoring the link, which is configured to listen to the data storage topic in the distributed message system in real time, and store information associated with the consumption of the data transfer object in the second database when the data transfer object is consumed.

9. A computer-readable storage medium, characterized in that, The application further comprises a visual processing device for monitoring the link, which is configured to listen to the data storage topic in the distributed message system in real time, and store information associated with the consumption of the data transfer object in the second database when the data transfer object is consumed.

10. A computer program product comprising a computer program, characterized in that, The application further comprises a visual processing device for monitoring the link, which is configured to listen to the data storage topic in the distributed message system in real time, and store information associated with the consumption of the data transfer object in the second database when the data transfer object is consumed.

Images