A business monitoring method, device, equipment and storage medium based on graph data
By obtaining and analyzing the historical log data of the business system, generating a visual monitoring model, and monitoring it in combination with preset alarm strategies, the problems of low monitoring efficiency and difficulty in fault location in the existing technology are solved, and efficient multi-dimensional monitoring and Fault location.
Patent Information
- Application Number
- CN202110799078.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-07-15
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2041-07-15
AI Technical Summary
The prior art is difficult to achieve efficient monitoring and fault location of business systems, especially in complex transaction links, and it is difficult to quickly and accurately locate the root cause of the problem.
By obtaining the historical log data of the target business, extracting the association relationships of nodes in different dimensions, generating a visual monitoring model, and combining preset alarm strategies to conduct real-time monitoring and alarms, multi-dimensional visual monitoring of the business transaction link is realized.
It improves the efficiency of business monitoring and the accuracy of fault location, solves the problem that existing monitoring systems can only implement single application indicator monitoring, and realizes visual monitoring of different dimensions on the transaction link from the business level.
Smart Images

Figure CN113536054B_ABST
Abstract
Description
Technical Field
[0001] This article belongs to the field of financial technology, and specifically relates to a business monitoring method, device, equipment and storage medium based on graph data. Background Art
[0002] In recent years, with the rapid development of Internet technology, especially in the financial field, as the application architecture gradually transforms to service-oriented, the call levels between business systems are getting deeper and deeper, and the deployment is becoming more and more complex. It is often difficult to achieve real-time monitoring of business operations. When a business system fails, how to quickly analyze the problem from the complex call relationship and accurately locate the root cause of the failure is a key issue in current operation and maintenance work. At present, the monitoring and fault location of business systems are mainly the following: monitoring the key applications used in the core business, considering the failure problem when the application runs independently. Since the business operation is based on the execution of applications on the transaction link, when the business operation fails, a link on the link alarms, which leads to the alarm of upstream applications layer by layer. When locating the fault, the applications on the business transaction link are excluded one by one, and it is difficult to accurately and quickly locate the root cause of the problem. Therefore, how to improve the monitoring efficiency of the business system has become a technical problem that needs to be solved urgently. Summary of the invention
[0003] In view of the above problems in the prior art, the purpose of this article is to provide a business monitoring method, device, equipment and storage medium based on graph data, which can improve the efficiency of business system monitoring.
[0004] In order to solve the above technical problems, the specific technical solutions of this article are as follows:
[0005] On the one hand, this article provides a business monitoring method based on graph data, the method comprising:
[0006] Obtain historical log data for the target business;
[0007] Extracting and obtaining association relationships between nodes of different dimensions in the target business transaction link according to the historical log data;
[0008] Generate association relationship graph data of nodes of different dimensions according to the association relationship;
[0009] Constructing and generating a visual monitoring model of nodes of different dimensions in the target business transaction link according to the graph data;
[0010] According to the visual monitoring model and in combination with a preset alarm strategy, the operation data of the target business is monitored and alarmed.
[0011] Furthermore, the nodes of different dimensions include part or all of the applications, microservice clusters, microservices, containers and databases in the target business transaction link.
[0012] Furthermore, extracting and obtaining the association relationship of nodes of different dimensions in the target business transaction link according to the historical log data includes:
[0013] Determine all applications in the target business and the mutual access relationship between different applications according to the target business operation log in the historical log data;
[0014] Determine the microservice clusters in each application and the mutual access relationship between different microservice clusters according to the target business operation log and all the applications;
[0015] According to the microservice cluster, obtain the microservice call log corresponding to each microservice cluster in the historical log data;
[0016] Determine the mutual access relationship between different microservices in each microservice cluster according to the microservice call log;
[0017] According to the mutual access relationship between different applications, the mutual access relationship between different microservice clusters and the mutual access relationship between the different microservices, a first association relationship between nodes of different dimensions in the target business transaction link is formed.
[0018] Further, all applications in the target business and the mutual access relationship between different applications are determined according to the target business operation log in the historical log data, including:
[0019] Extracting the source address and destination address of the service operation from the target service operation log;
[0020] All applications in the target service and the mutual access relationship between different applications are determined according to the source address and the destination address.
[0021] Furthermore, the extracting and obtaining the association relationship of nodes of different dimensions in the target business transaction link according to the historical log data also includes:
[0022] According to the microservice, obtain the data interface call log corresponding to the microservice in the historical log data;
[0023] Determine the corresponding relationship between the microservice and the container, and the mutual access relationship between different containers according to the data interface call log;
[0024] Obtaining a data capture log corresponding to the container in the historical log data;
[0025] Determining a correspondence between the container and the database according to the data capture log;
[0026] According to the correspondence between the microservice and the container, the mutual access relationship between different containers, and the correspondence between the container and the database, a second association relationship between nodes of different dimensions in the target business transaction link is determined.
[0027] Furthermore, generating association relationship graph data of nodes of different dimensions according to the association relationship includes:
[0028] Generate association relationship text data of nodes of different dimensions according to the association relationship;
[0029] According to the text data, association relationship graph data of nodes of different dimensions are converted and generated.
[0030] Furthermore, after generating association relationship graph data of nodes of different dimensions according to the association relationship, the method further includes:
[0031] The converted graph data is stored in the graph database.
[0032] Furthermore, constructing and generating a visual monitoring model of nodes of different dimensions in the target business transaction link according to the graph data includes:
[0033] Constructing a visual initial monitoring model based on multi-dimensional nodes, wherein the visual initial monitoring model includes node types and node associated attributes;
[0034] Determine the graph data corresponding to each dimensional node according to the node type, and load the graph data into the visual initial monitoring model;
[0035] According to the node association attributes, the visual initial monitoring model after loading the graph data is configured to obtain a visual monitoring model of nodes of different dimensions in the target business transaction link that has been constructed.
[0036] Optionally, the node association attributes include a node type identifier, a graph data shape, and a mutual visit identifier.
[0037] Furthermore, the monitoring and alarming of the operation data of the target business according to the visual monitoring model and in combination with a preset alarm strategy includes:
[0038] Acquire the operation data of the target business in real time, wherein the operation data includes at least one of the following: the operation time of nodes of different dimensions, the CPU usage rate of nodes of different dimensions, and the time consumed for database calls;
[0039] Determine the alarm information of nodes of different dimensions according to the operation data and the preset alarm strategy;
[0040] According to the alarm information, a corresponding alarm prompt is displayed in the visual monitoring model.
[0041] Furthermore, the determining of the alarm information of nodes of different dimensions according to the operating data and the preset alarm strategy includes:
[0042] Get multiple sets of running data for a specified time period;
[0043] Determine abnormal operation data of nodes of different dimensions according to the multiple groups of operation data and preset thresholds;
[0044] Determining abnormal proportions of nodes of different dimensions according to the abnormal operation data and the multiple groups of operation data;
[0045] According to the abnormal proportion and the preset alarm strategy, alarm information of nodes of different dimensions is generated.
[0046] Optionally, generating alarm information of nodes of different dimensions according to the abnormal proportion and the preset alarm strategy includes:
[0047] When the abnormal ratio is less than the first threshold, no alarm information is generated;
[0048] When the abnormal ratio is between the first threshold and the second threshold, generating a first alarm information;
[0049] When the abnormal proportion is greater than the second threshold, second alarm information is generated, wherein the first threshold is less than the second threshold.
[0050] On the other hand, this document also provides a business monitoring device based on graph data, the device comprising:
[0051] A historical log data acquisition module is used to acquire historical log data for a target business;
[0052] An association relationship extraction module, used to extract the association relationships of nodes of different dimensions in the target business transaction link according to the historical log data;
[0053] A graph data generation module, used to generate association relationship graph data of nodes of different dimensions according to the association relationship;
[0054] A visualization model building module, used to build and generate a visualization monitoring model of nodes of different dimensions in the target business transaction link according to the graph data;
[0055] The monitoring and alarm module is used to monitor and alarm the operating data of the target business according to the visual monitoring model and the preset alarm strategy.
[0056] On the other hand, this document also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described above when executing the computer program.
[0057] Finally, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.
[0058] By adopting the above technical scheme, a graph-based business monitoring method, device, equipment and storage medium described in this article extracts the correlation relationships of different dimensions in the target business transaction link through the historical log data of the target business, and displays the correlation relationships through a visual monitoring model, thereby solving the problem that the existing monitoring system can only realize indicator monitoring of a single application, and realizes visual monitoring of different dimensions on the transaction link from the business level, improving the efficiency of fault location, thereby improving the efficiency of business monitoring.
[0059] In order to make the above and other purposes, features and advantages of this article more obvious and easy to understand, the following specifically cites preferred embodiments and describes them in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0060] In order to more clearly illustrate the technical solutions in the embodiments of this article or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of this article. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0061] Figure 1 A schematic diagram of an implementation environment of the method provided in the embodiments of this article is shown;
[0062] Figure 2 A schematic diagram of the steps of a business monitoring method based on graph data provided by an embodiment of this invention is shown;
[0063] Figure 3 A schematic diagram of the first association relationship acquisition step in the embodiment of this article is shown;
[0064] Figure 4 An example diagram of application-level association relationships in the embodiments of this document is shown;
[0065] Figure 5 A schematic diagram of the second association relationship acquisition step in the embodiment of this article is shown;
[0066] Figure 6A schematic diagram of the steps of constructing a visual monitoring model in an embodiment of this article is shown;
[0067] Figure 7 A schematic diagram of the alarm prompting steps in the embodiment of this article is shown;
[0068] Figure 8 A schematic diagram of the structure of a business monitoring device based on graph data provided by an embodiment of this invention is shown;
[0069] Fig. 9 A schematic diagram of the computer device structure provided in the embodiments of this article is shown.
[0070] Description of the accompanying symbols:
[0071] 10. Target business;
[0072] 20. Server;
[0073] 30. Monitoring device;
[0074] 100. Historical log data acquisition module;
[0075] 200. Association relationship extraction module;
[0076] 300. Graph data generation module;
[0077] 400. Visual model building module;
[0078] 500, monitoring and alarm module;
[0079] 902. Computer equipment;
[0080] 904, processor;
[0081] 906. Memory;
[0082] 908, driving mechanism;
[0083] 910, input / output module;
[0084] 912. Input device;
[0085] 914. Output device;
[0086] 916. Presentation equipment;
[0087] 918. Graphical user interface;
[0088] 920, network interface;
[0089] 922, communication link;
[0090] 924. Communication bus. DETAILED DESCRIPTION
[0091] The following will be combined with the drawings in the embodiments of this article to clearly and completely describe the technical solutions in the embodiments of this article. Obviously, the described embodiments are only part of the embodiments of this article, not all of the embodiments. Based on the embodiments of this article, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this article.
[0092] It should be noted that the terms "first", "second", etc. in the specification and claims of this article and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this article described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.
[0093] In the prior art, the monitoring of business systems is mainly based on the operation of a single application, and the failure problem when the application is running independently is considered. Since the business operation is based on the execution of the application on the transaction link, when the business operation fails, an alarm is triggered at a certain link on the link, which leads to alarms at all levels of upstream applications. When locating the fault, the applications on the business transaction link are excluded one by one, which makes it difficult to accurately and quickly locate the root cause of the problem.
[0094] In order to solve the above problems, the embodiment of this specification provides a business monitoring method, which is implemented based on graph data, such as Figure 1 As shown, it is a schematic diagram of the implementation environment of the method, which may include a target business 10, a server 20 and a monitoring device 30. The target business 10 may be any business in the business system. The log information of the target business 10 when executed is stored in the server 20. The monitoring device 30 obtains the log information of the target business 10 in the server 20, extracts the association relationship of the nodes of different dimensions in the target business transaction link, and then constructs a visualization monitoring model of the nodes of different dimensions in the target business transaction link based on the association relationship, so as to perform real-time monitoring of the target business according to the established visualization monitoring model. This paper can quickly and accurately locate faults and improve the efficiency of monitoring by real-time monitoring of nodes of different dimensions in the target business.
[0095] The server 20 can realize the storage and extraction of log information, and can be a physical device with a specific hardware structure, or can be a software structure that can realize the corresponding storage and extraction functions. In some other embodiments, the server 20 can also be a distributed server, which is not limited in the embodiments of this specification.
[0096] Specifically, the embodiments of this document provide a business monitoring method based on graph data, which can improve the efficiency of business monitoring. Figure 2 It is a step diagram of a business monitoring method based on graph data provided in the embodiments of this article. This specification provides method operation steps as described in the embodiments or flowcharts, but may include more or fewer operation steps based on conventional or non-creative labor. The order of steps listed in the embodiments is only one way of executing the steps among many orders, and does not represent the only order of execution. When the actual system or device product is executed, it can be executed in the order or in parallel according to the method shown in the embodiments or drawings. Specifically, Figure 2 As shown, the method may include:
[0097] S101: Obtain historical log data for a target business;
[0098] S102: extracting and obtaining association relationships between nodes of different dimensions in the target business transaction link according to the historical log data;
[0099] S103: Generate association relationship graph data of nodes of different dimensions according to the association relationship;
[0100] S104: constructing and generating a visual monitoring model of nodes of different dimensions in the target business transaction link according to the graph data;
[0101] S105: According to the visual monitoring model and in combination with a preset alarm strategy, monitoring and alarming are performed on the operation data of the target business.
[0102] It can be understood that the embodiments of this specification analyze the historical log data of the target business to determine the association relationship of nodes of different dimensions in the target business transaction link. The nodes of different dimensions can be nodes of different granularities, such as application level, microservice level, etc. By determining the association relationship of nodes of different dimensions, the execution relationship of the business functions in the target business transaction link can be obtained, thereby achieving accurate monitoring of nodes in each dimension. Then, the association relationship graph data of nodes of different dimensions is generated through the determined association relationship, and a visual monitoring model of nodes of different dimensions in the target business transaction link is constructed based on the graph data, thereby achieving visual monitoring of nodes of different dimensions of the target business, and setting different alarm strategies in different dimensions as needed, so as to achieve accurate and rapid positioning of faults and improve the efficiency and reliability of monitoring.
[0103] The historical log data may be historical log data of successful operation of the target business, which ensures that the information in the historical log data is reliable and guarantees the accuracy of the extracted information.
[0104] The association relationship may be a relationship between different nodes in the same dimension, such as a mutual access relationship (execution sequence), a calling relationship, etc., or a calling relationship, a subordinate relationship, etc. between nodes in different dimensions.
[0105] In some embodiments, the transaction link of a business will flow through several applications from initiation to completion. The business can be, for example, banking and financial business such as quick payment and bank-to-securities transfer. In order to realize a single bank-to-securities transfer transaction, it will flow through several applications on the transaction link. Each application is responsible for implementing different functions and cooperates with each other to complete a transaction, including the mutual visit relationship or call relationship between different applications; at the same time, an application will provide support for multiple different businesses, so that a many-to-many correspondence relationship will be formed between the business and the application. Therefore, the different dimensional nodes include part or all of the applications, microservice clusters, microservices, containers and databases in the target business transaction link.
[0106] Among them, the target business transaction link requires the cooperation and collaboration of multiple applications to be implemented. Each application can include at least one microservice cluster to implement a specific single function, and each microservice cluster can be further divided into multiple microservices. Each microservice runs in its own process, focusing on a single business capability and using lightweight mechanisms for communication, thereby realizing the extraction of transaction links from macro to micro.
[0107] Each microservice function is built around the business function and can be independently deployed through a fully automatic deployment mechanism. Microservices do not require centralized management and can be written in different programming languages and use different data storage technologies. Therefore, the function implementation of microservices needs to rely on other microservices. Assuming that microservice A depends on microservice B and microservice C, and microservice B and microservice C may continue to rely on other microservices, on such a call chain, if a certain microservice is unavailable or has a high delay, the request to call microservice A will be blocked, occupying the system's CPU, IO and other resources. When this type of request is increasing, and the system resources occupied are increasing, it will cause system bottlenecks to appear, causing other requests to be unavailable, and ultimately causing the business system to crash. Therefore, in the embodiments of this specification, the business system can improve the free integration and collaboration efficiency between different microservices through container technology (such as Docker technology).
[0108] Therefore, when the microservice is called and executed, the calling or mutual access relationship between different containers can be determined. At the same time, the target business must also involve the data processing process during operation. This article can also monitor the call status of the database, so as to achieve comprehensive visual monitoring of the target business and improve the comprehensiveness and accuracy of the monitoring.
[0109] In the embodiments of this specification, Figure 3 As shown, the extracting and obtaining the mutual visit relationship of nodes of different dimensions in the target business transaction link according to the historical log data includes:
[0110] S201: determining all applications in the target business and the mutual access relationship between different applications according to the target business operation log in the historical log data;
[0111] S202: Determine, according to the target business operation log and all the applications, the microservice cluster in each application and the mutual access relationship between different microservice clusters;
[0112] S203: According to the microservice cluster, obtain the microservice call log corresponding to each microservice cluster in the historical log data;
[0113] S204: Determine the mutual access relationship between different microservices in each microservice cluster according to the microservice call log;
[0114] S205: According to the mutual access relationship between different applications, the mutual access relationship between different microservice clusters, and the mutual access relationship between the different microservices, a first association relationship between nodes of different dimensions in the target business transaction link is formed.
[0115] It can be understood that the target business operation log contains the application information of the target business and the execution relationship between different applications. Of course, a log message may involve relatively little information and cannot fully display all applications on the target business. Therefore, a large number of operation logs can be obtained to ensure the integrity of the application information extraction.
[0116] The mutual access relationship may be the execution sequence between different applications, that is, the sequence of implementation of different functions on the target business transaction link.
[0117] Further, all applications in the target business and the mutual access relationship between different applications are determined according to the target business operation log in the historical log data, including:
[0118] Extracting the source address and destination address of the service operation from the target service operation log;
[0119] All applications in the target service and the mutual access relationship between different applications are determined according to the source address and the destination address.
[0120] The source address and the destination address may be the mutual access relationship between different servers, that is, the server corresponding to the source address is used to locate the server corresponding to the destination address, and each server corresponds to the corresponding application. Therefore, by determining the source address and the destination address, different applications and the mutual access relationship between different applications can be determined.
[0121] In some other embodiments, there may be other ways to determine different applications, such as determining each keyword in the operation log by word segmentation (such as jieba tool), determining the corresponding application based on the keyword, or directly determining the corresponding application by keyword extraction. The specific determination method is not limited in this specification.
[0122] When all applications are determined, the microservice clusters under each application can be further determined, and then the mutual visits between different microservice clusters under the same application can be determined, so that a two-level association relationship between application and microservice cluster can be established.
[0123] For example, for a bank-to-securities transfer transaction, by collecting its successful operation logs, for example, a single log data is as follows: {"service":"PPSA.Personal Pay ACS-com.icbc.stars.transaction.common.CommonService__1_0_0","method":"execute","application":"F-PRSA","service_group":"PPSA_AC","provider":"1.1.1.1:35060","consumer":"2.2.2.2","success":4,"failure":0,"max_concurrent":2,"elapsed":156,"max_elapsed":46,"timeout":0,"execute_limit":0,"exhaust":0,"unkown":0,"node_active":3,"no_pr ovider":0,"consumer_group":"BOOM_AC","time":"2020-09-16T14:27:44.000Z","type":"consumer","activate_percent":0.0};From the above logs, it can be determined that consumer 2.2.2.2 accessed provider 1.1.1.1, and the mutual access relationship between the application where 2.2.2.2 is located and the application where 1.1.1.1 is located can be obtained. By collecting a large number of logs in the bank-securities transfer transaction link business, multiple groups of mutual visits between the source IP application and the destination IP are formed, and then the mutual access relationship between applications in a specific transaction link is formed. Furthermore, in the above log, consumer 2.2.2.2 accessed provider 1.1.1.1. By taking the "service_group" field and the "consumer_group" field in the log, we can determine the mutual access relationship in different microservice clusters under the application, that is, the microservice cluster corresponding to the service_group and the microservice cluster corresponding to the consumer_group.
[0124] Specifically, Figure 4As shown in the figure, it is a schematic diagram of the application mutual access relationship in the bank-securities transfer transaction obtained by extracting the operation log. The lines between different applications in the figure represent the mutual access or calling relationship between them. Among them, F5 and SLB (Server Load Balance) are both server load balancing, which can realize load balancing in bank-securities transfer transactions. BOOM is the UnionPay transfer access layer, which is used to connect to the UnionPay transfer service. NOS is the storage database, which is used to obtain transfer-related data. HSM is an encryption machine, which is used to encrypt data. ATP is a customer agreement query, which is used to determine whether the user has signed the relevant agreement to carry out subsequent personal account settlement business. PRAS (Personal Retail Application System) is a personal financial business, which is used to provide settlement services for personal financial business. The DSR gateway is used to transmit the relevant data of BOOM and PRSA to the host. The host service delivery in the host realizes the transmission and transfer of data. PRAS personal finance is the settlement work of personal financial business for the received data. The bank-securities transfer settlement for personal financial business is completed through the collaboration between the above applications.
[0125] Furthermore, the microservice call log is obtained through the determined microservice cluster, and then the mutual access relationship of multiple microservices under the microservice cluster is analyzed. For example, the microservice call log is as follows: {"spanName":"com.atp.tx.channel.services.payprechkchanservinf__1_0 / serve","traceId":"6074fd22d1c6c88f581c3b7ca4d8f5eb","spanId":"ab41925a07eaebc2","spanMatchId":null,"timestamp":"1618279714092173","createTime":null,"duration":33470,"appName":"f-boom-ac-online","monitorConfBean":null,"parentId":"581c3b7ca4d8f5eb","durationStr":"33.470ms". The parentId field and spanId field are extracted. The parentId field is the parent node, and the spanId is the child node. By collecting a large number of such logs in the bank-securities transfer transaction link business, the mutual access relationship of the microservice call chain is formed.
[0126] On the basis of determining the first association relationship of nodes of different dimensions in the target business transaction link, the mutual visit or call relationship of nodes of smaller granularity may be further determined. As an option, Figure 5 As shown, the extracting and obtaining the association relationship of nodes of different dimensions in the target business transaction link according to the historical log data also includes:
[0127] S301: According to the microservice, obtain a data interface call log corresponding to the microservice in the historical log data;
[0128] S302: Determine the corresponding relationship between the microservice and the container, and the mutual access relationship between different containers according to the data interface call log;
[0129] S303: Obtaining a data capture log corresponding to the container in the historical log data;
[0130] S304: Determine the corresponding relationship between the container and the database according to the data capture log;
[0131] S305: Determine a second association relationship between nodes of different dimensions in the target business transaction link according to the correspondence between the microservice and the container, the mutual access relationship between different containers, and the correspondence between the container and the database.
[0132] It can be understood that the inclusion relationship between the microservice and the container can be determined through the data interface call log, and the business function of the corresponding microservice is realized through the container. Therefore, in the business system of this specification, the container technology is configured in advance so that the function implementation of each microservice can correspond to a container. When the container is called, it will receive the call instruction through the corresponding interface. Therefore, the corresponding relationship between the microservice and the container can be obtained through the data interface call log, and then the mutual access relationship between different containers can be analyzed.
[0133] Furthermore, while the container implements the corresponding microservice functions, the mutual access relationship between the database and the container can also be determined through the data capture log, which can also be understood as the connection relationship between the container and the database. When performing data capture, the source address and destination address in the traffic packet are captured by the packet capture device deployed on the switch. The source address and destination address can be understood as the database address. Therefore, by obtaining a large number of data capture logs, the mutual access relationship between all containers and databases can be extracted.
[0134] The first association relationship and the second association relationship of the nodes of different dimensions in the target business transaction link obtained through the above steps can determine the association relationship of the nodes of different dimensions in the target business transaction link. In a further embodiment, the association relationship graph data of the nodes of different dimensions is generated according to the association relationship, including:
[0135] Generate association relationship text data of nodes of different dimensions according to the association relationship;
[0136] According to the text data, association relationship graph data of nodes of different dimensions are converted and generated.
[0137] The association relationship text data can directly express the association relationship between nodes of different dimensions. However, for a single business, the number of nodes of different dimensions on the transaction link and the complexity of the association relationship are relatively high. When a fault occurs, it is difficult to directly locate the fault. It is difficult to have an intuitive feeling only through the document form. Therefore, the document data can be converted into a graph data form, and stored and displayed in the form of graph data. For example, a node-relationship storage form can be adopted. The method of converting text data into graph data is a common technical means for text conversion, and is not limited in the embodiments of this specification.
[0138] In a further embodiment, a preprocessing process of the text data may be included before the text data is converted, such as cleaning the text data, obtaining optimized text data by deduplication, deletion of abnormal data, etc., to ensure the reliability and accuracy of the text data.
[0139] In a further embodiment, the graph data obtained after the conversion is stored in the graph database (such as a Neo4j graph database), for example, in a node-relationship storage format.
[0140] After obtaining the graph data of nodes of different dimensions on the target business transaction link, a visual monitoring model can be constructed. As an option, Figure 6 As shown, the visualization monitoring model of nodes of different dimensions in the target business transaction link is constructed and generated according to the graph data, including:
[0141] S401: constructing a visual initial monitoring model based on multi-dimensional nodes, wherein the visual initial monitoring model includes node types and node-related attributes;
[0142] S402: Determine the graph data corresponding to each dimensional node according to the node type, and load the graph data into the visual initial monitoring model;
[0143] S403: According to the node association attributes, the visual initial monitoring model after loading the graph data is configured to obtain a constructed visual monitoring model of nodes of different dimensions in the target business transaction link.
[0144] Among them, the node type can be a node type of different dimensions, and the visual monitoring model can be a multi-layer display layer set according to nodes of different dimensions. Each display layer includes nodes of the same dimension and their associations. Different display layers can establish corresponding associations, such as by connecting lines. This article intuitively displays the operating conditions of different levels in a transaction link of the target business from different dimensions, thereby meeting the needs of operation and maintenance personnel at different levels. For example, they are the mutual access relationship of applications flowing through a transaction link, the mutual access relationship between microservice clusters, the mutual access relationship of microservice call chains, and the mutual access relationship between the container initiator and the container callee in the microservice call chain; in terms of logical display, it is displayed from coarse to fine, layer by layer.
[0145] The node association attributes include node type identification, graph data shape, and mutual visit identification. For example, different node types use different colors, styles, etc., so that the operating conditions of different nodes can be quickly distinguished. The graph data shape can be the performance status of each display layer, which further improves the visualization recognition. The mutual visit representation can be the association relationship between different nodes, which can be represented by lines of different colors, thicknesses, etc., so that targeted monitoring can be achieved. It solves the problem that the existing monitoring system can only monitor the indicators of a single application and cannot realize the visualization of a type of business transaction link. Compared with the existing operation and maintenance method that relies on the traditional monitoring platform, it simplifies the existing fault location process of repeatedly querying the holographic monitoring platform, and can directly locate the fault through the visual monitoring model, reducing the operation and maintenance pressure of the operation and maintenance personnel.
[0146] On the basis of obtaining the visual monitoring model, the operation of the target business can be monitored in real time. As an option, Figure 7 As shown, the operation data of the target business is monitored and alarmed according to the visual monitoring model and the preset alarm strategy, including:
[0147] S501: Acquire the operation data of the target business in real time, where the operation data includes at least one of the following: the operation time of nodes of different dimensions, the CPU usage rate of nodes of different dimensions, and the time consumption of database calls;
[0148] S502: Determine the alarm information of nodes of different dimensions according to the operation data and the preset alarm strategy;
[0149] S503: Displaying corresponding alarm prompts in the visual monitoring model according to the alarm information.
[0150] It can be understood that different monitoring indicators can be set for different nodes, so that independent monitoring of nodes in different dimensions can be achieved, which improves the accuracy and comprehensiveness of control. Of course, it is also possible to monitor only some important nodes, which can reduce the load pressure brought by full monitoring and improve the stability of monitoring.
[0151] In actual work, operation data can be collected according to a specified period, such as a set of data every 3 minutes or 5 minutes, etc. This can reduce the data storage pressure of real-time collection. After the collection, the abnormal points can be determined, or a certain amount of data (or within a specified time period) can be collected for unified analysis, which is not limited in the embodiments of this specification.
[0152] For example, for example, monitoring indicators are set at the application level. Different monitoring indicators can be set for different applications, including the application running time threshold, CPU usage threshold, and database call time threshold. When at least one indicator in the running data of the application exceeds the corresponding threshold, it means that the application is in an abnormal state and a corresponding alarm prompt needs to be given. When an alarm is given, the alarm can be given directly at the node position in the visual monitoring model, which can facilitate the operation and maintenance personnel to quickly determine the abnormal location, thereby improving the timeliness of fault resolution. Of course, a separate alarm section can also be set to realize the centralized display of alarms, which is convenient for operation and maintenance personnel to collect alarm information.
[0153] Although the calls between nodes involved in a single target business are clearer, when monitoring the entire business system, each business should be visually monitored and displayed. There are also situations where nodes are shared between different businesses, so the monitoring of the entire business system will be very complicated. When an alarm is prompted, the use of a single sampling point for alarm will cause the alarm to be too frequent, affecting the judgment of the operation and maintenance personnel. Therefore, according to the operation data and the preset alarm strategy, the alarm information of nodes of different dimensions is determined, including:
[0154] Get multiple sets of running data for a specified time period;
[0155] Determine abnormal operation data of nodes of different dimensions according to the multiple groups of operation data and preset thresholds;
[0156] Determining abnormal proportions of nodes of different dimensions according to the abnormal operation data and the multiple groups of operation data;
[0157] According to the abnormal proportion and the preset alarm strategy, alarm information of nodes of different dimensions is generated.
[0158] By setting a specified time period for alarm, the timeliness of the alarm can be ensured, and multiple services can be monitored simultaneously. Since multiple sampling points are included in the specified time period, the analysis of the abnormal situation of the sampling points determines whether to issue an alarm. For example, when there are many abnormal points, it means that the probability of node abnormality in the corresponding specified time period is high, and an alarm prompt can be issued. Alarming by abnormal ratio can ensure the reliability of the alarm, and at the same time reduce the probability of short-term high-speed alarms, further ensuring the credibility of the alarm.
[0159] In a further embodiment, generating alarm information of nodes of different dimensions according to the abnormal proportion and the preset alarm strategy includes:
[0160] When the abnormal ratio is less than the first threshold, no alarm information is generated;
[0161] When the abnormal ratio is between the first threshold and the second threshold, generating a first alarm information;
[0162] When the abnormal proportion is greater than the second threshold, second alarm information is generated, wherein the first threshold is less than the second threshold.
[0163] It can be understood that the abnormal ratio can reflect the abnormal condition of the corresponding node. When the abnormal ratio is large, it means that the number of node abnormalities is large in the corresponding time period, and a higher level of alarm is required to increase the attention of operation and maintenance personnel.
[0164] Exemplarily, 100 sampling points are collected within a specified time period, and the abnormal situation of each sampling point is analyzed through the monitoring indicators in the above steps to determine the abnormal proportion within the specified time period. Assuming that the first threshold is 20% and the second threshold is 60%, the abnormal proportion is compared with 20% and 60% to determine different alarm levels. Optionally, the first alarm information can be a text message reminder, an email reminder, and a color change of a node in a visual monitoring model, etc. The second alarm information can be a voice reminder and a highlight flashing of a node in a visual monitoring model, etc. The intensity of the second alarm information is greater than the intensity of the second alarm information. The specific alarm method is not limited in this specification.
[0165] The graph data-based business monitoring method provided in this paper extracts the association relationships of different dimensions in the target business transaction link through the historical log data of the target business, and displays the association relationships through a visual monitoring model. When the target business is running, it realizes the monitoring of nodes of different dimensions through its operating data and preset alarm strategies, and issues alarms in the visual monitoring model, thereby solving the problem that the existing monitoring system can only realize the indicator monitoring of a single application, and realizes the visual monitoring of different dimensions on the transaction link from the business level, which improves the efficiency of fault location, thereby improving the efficiency of business monitoring.
[0166] Based on the same inventive concept, this article also provides a business monitoring device based on graph data, such as Figure 8 As shown, the device comprises:
[0167] The historical log data acquisition module 100 is used to acquire historical log data for a target business;
[0168] The association relationship extraction module 200 is used to extract the association relationship of nodes of different dimensions in the target business transaction link according to the historical log data;
[0169] A graph data generation module 300, used to generate association relationship graph data of nodes of different dimensions according to the association relationship;
[0170] A visualization model building module 400 is used to build and generate a visualization monitoring model of nodes of different dimensions in the target business transaction link according to the graph data;
[0171] The monitoring and alarm module 500 is used to monitor and alarm the operation data of the target business according to the visual monitoring model and in combination with a preset alarm strategy.
[0172] The beneficial effects obtained by the above-mentioned device are consistent with the beneficial effects obtained by the above-mentioned method, and will not be described in detail in the embodiments of this specification.
[0173] It should be noted that the business monitoring method and device based on graph data provided in the embodiments of this specification can be used in the financial field for business system operation and maintenance monitoring strategies, and can also be used in any field other than the financial field. The application field of the business monitoring method and device based on graph data provided in the embodiments of this specification is not limited.
[0174] like Fig. 9As shown, a computer device provided in an embodiment of the present invention, the computer device 902 may include one or more processors 904, such as one or more central processing units (CPUs), and each processing unit may implement one or more hardware threads. The computer device 902 may also include any memory 906, which is used to store any kind of information such as code, settings, data, etc. Non-restrictive, for example, the memory 906 may include any one or more combinations of the following: any type of RAM, any type of ROM, flash memory device, hard disk, optical disk, etc. More generally, any memory may use any technology to store information. Further, any memory may provide volatile or non-volatile retention of information. Further, any memory may represent a fixed or removable component of the computer device 902. In one case, when the processor 904 executes an associated instruction stored in any memory or a combination of memories, the computer device 902 may perform any operation of the associated instruction. The computer device 902 also includes one or more drive mechanisms 908 for interacting with any memory, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.
[0175] The computer device 902 may also include an input / output module 910 (I / O) for receiving various inputs (via input devices 912) and for providing various outputs (via output devices 914). A specific output mechanism may include a presentation device 916 and an associated graphical user interface (GUI) 918. In other embodiments, the input / output module 910 (I / O), the input device 912, and the output device 914 may not be included, and the computer device 902 may be used as a computer device in a network. The computer device 902 may also include one or more network interfaces 920 for exchanging data with other devices via one or more communication links 922. One or more communication buses 924 couple the components described above together.
[0176] The communication link 922 may be implemented in any manner, for example, through a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 922 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc. governed by any protocol or combination of protocols.
[0177] Corresponds to Figure 2-Figure 7 The method in this article also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are executed.
[0178] The embodiment of the present invention also provides a computer readable instruction, wherein when the processor executes the instruction, the program therein causes the processor to execute the following Figures 2 to 7The method shown.
[0179] It should be understood that in the various embodiments of this document, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this document.
[0180] It should also be understood that in the embodiments of this document, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this document generally indicates that the associated objects before and after are in an "or" relationship.
[0181] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this article.
[0182] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0183] In the several embodiments provided herein, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or can be electrical, mechanical or other forms of connection.
[0184] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this article.
[0185] In addition, each functional unit in each embodiment of this invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional unit.
[0186] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this article is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of this article. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0187] Specific embodiments are used in this article to illustrate the principles and implementation methods of this article. The description of the above embodiments is only used to help understand the methods and core ideas of this article. At the same time, for general technicians in this field, according to the ideas of this article, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on this article.
Claims
1. A business monitoring method based on graph data, It is characterized in that The method comprises: Obtain historical log data for the target business; Extracting and obtaining association relationships of nodes of different dimensions in the target business transaction link according to the historical log data, wherein the nodes of different dimensions include a part or all of applications, microservice clusters, microservices, containers, and databases in the target business transaction link; Generate association relationship graph data of nodes of different dimensions according to the association relationship; Constructing and generating a visual monitoring model of nodes of different dimensions in the target business transaction link according to the graph data; According to the visual monitoring model and in combination with a preset alarm strategy, monitoring and alarming are performed on the operation data of the target business; The step of monitoring and alarming the operation data of the target business according to the visual monitoring model and in combination with a preset alarm strategy includes: Acquire the operation data of the target business in real time, wherein the operation data includes at least one of the following: the operation time of nodes of different dimensions, the CPU usage rate of nodes of different dimensions, and the time consumed for database calls; Determine the alarm information of nodes of different dimensions according to the operation data and the preset alarm strategy; According to the alarm information, a corresponding alarm prompt is displayed in the visual monitoring model; The step of constructing and generating a visualization monitoring model of nodes of different dimensions in the target business transaction link according to the graph data includes: constructing a visualization initial monitoring model based on multi-dimensional nodes, the visualization initial monitoring model including node types and node association attributes; determining the graph data corresponding to each dimensional node according to the node type, and loading the graph data into the visualization initial monitoring model; configuring the visualization initial monitoring model after loading the graph data according to the node association attributes, so as to obtain a visualization monitoring model of nodes of different dimensions in the target business transaction link that has been constructed; The method of determining the alarm information of nodes of different dimensions based on the operating data and the preset alarm strategy includes: obtaining multiple groups of operating data for a specified time period; determining abnormal operating data of nodes of different dimensions based on the multiple groups of operating data and a preset threshold; determining the abnormal proportion of nodes of different dimensions based on the abnormal operating data and the multiple groups of operating data; and generating alarm information for nodes of different dimensions based on the abnormal proportion and the preset alarm strategy.
2. The service monitoring method according to claim 1, It is characterized in that The extracting and obtaining the association relationship of nodes of different dimensions in the target business transaction link according to the historical log data includes: Determine all applications in the target business and the mutual access relationship between different applications according to the target business operation log in the historical log data; Determine the microservice clusters in each application and the mutual access relationship between different microservice clusters according to the target business operation log and all the applications; According to the microservice cluster, obtain the microservice call log corresponding to each microservice cluster in the historical log data; Determine the mutual access relationship between different microservices in each microservice cluster according to the microservice call log; According to the mutual access relationship between different applications, the mutual access relationship between different microservice clusters and the mutual access relationship between the different microservices, a first association relationship between nodes of different dimensions in the target business transaction link is formed.
3. The service monitoring method according to claim 2, It is characterized in that Determining all applications in the target business and the mutual access relationship between different applications according to the target business operation log in the historical log data, including: Extracting the source address and destination address of the service operation from the target service operation log; All applications in the target service and the mutual access relationship between different applications are determined according to the source address and the destination address.
4. The service monitoring method according to claim 2, It is characterized in that The extracting and obtaining the association relationship of nodes of different dimensions in the target business transaction link according to the historical log data also includes: According to the microservice, obtain the data interface call log corresponding to the microservice in the historical log data; Determine the corresponding relationship between the microservice and the container, and the mutual access relationship between different containers according to the data interface call log; Obtaining a data capture log corresponding to the container in the historical log data; Determining a correspondence between the container and the database according to the data capture log; According to the correspondence between the microservice and the container, the mutual access relationship between different containers, and the correspondence between the container and the database, a second association relationship between nodes of different dimensions in the target business transaction link is determined.
5. The service monitoring method according to claim 1, It is characterized in that The generating of association relationship graph data of nodes of different dimensions according to the association relationship includes: Generate association relationship text data of nodes of different dimensions according to the association relationship; According to the text data, association relationship graph data of nodes of different dimensions are converted and generated.
6. The service monitoring method according to claim 1, It is characterized in that After generating association relationship graph data of nodes of different dimensions according to the association relationship, the following further comprises: The converted graph data is stored in the graph database.
7. The service monitoring method according to claim 1, It is characterized in that The node association attributes include a node type identifier, a graph data shape, and a mutual visit identifier.
8. The service monitoring method according to claim 1, It is characterized in that Generating alarm information of nodes of different dimensions according to the abnormal proportion and the preset alarm strategy includes: When the abnormal ratio is less than the first threshold, no alarm information is generated; When the abnormal ratio is between the first threshold and the second threshold, generating a first alarm information; When the abnormal proportion is greater than the second threshold, second alarm information is generated, wherein the first threshold is less than the second threshold.
9. A business monitoring device based on graph data, It is characterized in that The device comprises: A historical log data acquisition module is used to acquire historical log data for a target business; An association relationship extraction module is used to extract the association relationship of different dimensional nodes in the target business transaction link according to the historical log data, and the different dimensional nodes include a part or all of the application, microservice cluster, microservice, container and database in the target business transaction link; A graph data generation module, used to generate association relationship graph data of nodes of different dimensions according to the association relationship; A visualization model building module, used to build and generate a visualization monitoring model of nodes of different dimensions in the target business transaction link according to the graph data; A monitoring and alarm module, used to monitor and alarm the operation data of the target business according to the visual monitoring model and in combination with a preset alarm strategy; The monitoring alarm module is also used to obtain the operation data of the target business in real time, and the operation data includes at least one of the following: the operation time of nodes of different dimensions, the CPU usage rate of nodes of different dimensions, and the database call time; determine the alarm information of nodes of different dimensions according to the operation data and the preset alarm strategy; and display the corresponding alarm prompt in the visual monitoring model according to the alarm information; The visualization model construction module is also used to construct a visualization initial monitoring model based on multi-dimensional nodes, wherein the visualization initial monitoring model includes node types and node association attributes; according to the node type, determine the graph data corresponding to each dimensional node, and load the graph data into the visualization initial monitoring model; according to the node association attributes, configure the visualization initial monitoring model after loading the graph data to obtain the visualization monitoring model of different dimensional nodes in the target business transaction link that has been constructed; The monitoring and alarm module is also used to obtain multiple groups of operating data in a specified time period; determine the abnormal operating data of nodes of different dimensions based on the multiple groups of operating data and preset thresholds; determine the abnormal proportions of nodes of different dimensions based on the abnormal operating data and the multiple groups of operating data; and generate alarm information for nodes of different dimensions based on the abnormal proportions and the preset alarm strategy.
10. A computer device comprising a memory, a processor and a computer program stored in the memory and running on the processor, It is characterized in that When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.
11. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.
