Distributed database monitoring method and device based on full-link monitoring

Abstract

The present invention belongs to the field of big data technology. The present invention provides a distributed database monitoring method and device based on full-link monitoring. The distributed database monitoring method based on full-link monitoring includes: generating a link corresponding to the transaction request in response to a transaction request; collecting link information and distributed database operation indicators; generating a topological relationship corresponding to the link based on the information and operation indicators; and monitoring the distributed database based on the topological relationship. From the perspective of SQL analysis, the present invention can help operation and maintenance personnel find that when a SQL query is slow overall, it can be associated with which specific links take a long time, and then drill down for analysis, first from the surface to the line, find a typical link, and then from the line to the point, so as to analyze the cause of the slow SQL link.

Description

Technical Field

[0001] The present application relates to the field of big data technology, in particular to the field of full-link monitoring technology, and specifically to a distributed database monitoring method and device based on full-link monitoring. Background Art

[0002] With the maturity of distributed architecture, a large number of enterprise-level applications adopt distributed and cloud computing technologies. There are often tens of thousands of nodes running in enterprise production, and the calling relationships between different types of nodes are intricate. These nodes are often cross-application, cross-team, and cross-park. Once an error occurs in a certain link, it is very difficult for development and operation and maintenance personnel to locate the problem. They often need to use massive discrete logs to troubleshoot and analyze, which is inefficient. The emergence of distributed tracing technology allows R&D and operation and maintenance personnel to better control the calling situation between nodes, and to more efficiently and conveniently troubleshoot online problems, making full-link monitoring possible.

[0003] With the help of full-link monitoring technology, you can clearly see the flow of each transaction, including the execution of query statements when accessing relational databases. For example, how many times did a transaction access the database, which database was accessed, what was the database query statement (SQL statement) for each access, how long it took to execute each statement, whether there was an error, etc. These are all from the perspective of the link, that is, from the perspective of the transaction to observe the execution of SQL. However, in actual operation and maintenance monitoring, it is often not enough for enterprises to analyze only from a certain point or a certain line. Usually the problem is not sporadic. For example, if a certain SQL is not written efficiently and takes a long time to execute (slow SQL), then all transactions that call this SQL will take a long time.

[0004] To sum up, how to monitor abnormal SQL, associate it with links, and combine points, lines and surfaces to facilitate SQL analysis has become an urgent problem to be solved. Summary of the invention

[0005] The present invention belongs to the field of big data technology. Aiming at the problems in the prior art, the present invention starts from the link perspective. When a link is found to have slow SQL or abnormal SQL, the present invention can grasp the overall operation status of the query statement from point to surface, so as to clarify whether it is a point problem or a surface problem. On the other hand, from the perspective of SQL analysis, when operation and maintenance personnel find that a certain SQL query is slow overall, they can associate it with which specific links take a long time, and then drill down for analysis, first from surface to line, find a typical link, and then from line to point, so as to analyze in detail the cause of the slow SQL link.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In response to the transaction request, generating a link corresponding to the transaction request;

[0008] Collecting information of the link and operating indicators of the distributed database;

[0009] Generate a topological relationship corresponding to the link according to the information and the operation indicator;

[0010] The distributed database is monitored according to the topological relationship.

[0011] In one embodiment, the collecting of the information of the link and the operating indicators of the distributed database includes:

[0012] Burying points on the link;

[0013] The tag ID of the link, the call ID of the current stage, the call ID of the previous stage and the operation index are collected according to the tracking point.

[0014] In one embodiment, generating the topological relationship corresponding to the link according to the information and the operation indicator includes:

[0015] The topological relationship is generated according to the tag ID of the link, the call ID of the current stage, the call ID of the previous stage and the operation index.

[0016] In one embodiment, monitoring the distributed database according to the topological relationship includes:

[0017] Calculate the corresponding hash code according to the topological relationship;

[0018] Taking the hash code as a dimension, performing aggregate calculation on the distributed database at a preset time to generate an average SQL execution time, a sum of SQL execution times, a sum of SQL success times, and a sum of SQL exception times;

[0019] The distributed database is monitored according to the average SQL execution time, the sum of SQL execution times, the sum of SQL success times and the sum of SQL exception times.

[0020] In a second aspect, the present invention provides a distributed database monitoring device based on full-link monitoring, the device comprising:

[0021] A link generation unit, configured to generate a link corresponding to the transaction request in response to the transaction request;

[0022] An information collection unit, used to collect information about the link and operating indicators of a distributed database;

[0023] A topology relationship generating unit, configured to generate a topology relationship corresponding to the link according to the information and the operation indicator;

[0024] An abnormality monitoring unit is used to monitor the distributed database according to the topological relationship.

[0025] In one embodiment, the information collection unit includes:

[0026] A tracking module, used for tracking points in the link;

[0027] The information collection module is used to collect the tag ID of the link, the call ID of the current stage, the call ID of the previous stage and the operation index according to the tracking point.

[0028] In one embodiment, the topology relationship generating unit is specifically configured to generate the topology relationship according to the tag ID of the link, the call ID of the current stage, the call ID of the previous stage, and the operation index.

[0029] In one embodiment, the abnormality monitoring unit includes:

[0030] A hash code calculation module, used to calculate the corresponding hash code according to the topological relationship;

[0031] An aggregation calculation module, used to perform aggregation calculation on the distributed database with the hash code as a dimension and at a preset time to generate an average SQL execution time, a sum of SQL execution times, a sum of SQL success times, and a sum of SQL exception times;

[0032] The database monitoring module is used to monitor the distributed database according to the average SQL execution time, the sum of SQL execution times, the sum of SQL success times and the sum of SQL exception times.

[0033] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, steps of a distributed database monitoring method based on full-link monitoring are implemented.

[0034] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a distributed database monitoring method based on full-link monitoring.

[0035] From the above description, it can be seen that an embodiment of the present invention provides a distributed database monitoring method and device based on full-link monitoring. First, in response to a transaction request, a link corresponding to the transaction request is generated; then, information on the link and operating indicators of the distributed database are collected; a topological relationship corresponding to the link is generated based on the information and operating indicators; and finally, the distributed database is monitored based on the topological relationship.

[0036] In order to make up for the shortcomings of the existing full-link monitoring solution in abnormal SQL monitoring and analysis, the distributed database monitoring method and device for full-link monitoring proposed in the present invention facilitate enterprises to combine points, lines and surfaces, comprehensively grasp the system SQL execution status, and better analyze SQL problems, specifically:

[0037] 1. The monitoring of slow SQL or erroneous SQL in the prior art only provides monitoring and analysis from the SQL perspective. The present invention associates the monitoring (point) of SQL with the link topology (surface), which facilitates a comprehensive review of the system operation status under a distributed architecture. Operation and maintenance personnel can start from a certain problematic SQL, grasp the dependency relationship of the problematic SQL from a macro perspective, and evaluate the impact scope of the problematic SQL.

[0038] 2. The present invention associates the monitoring (point) of SQL with the link topology (surface), and then associates it to the specific link (line) through the topology. By combining the points, lines and surfaces, the problematic SQL can eventually be associated with the specific link, so that the problem site can be traced back for a full analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 Schematic diagram of the process of a distributed database monitoring method based on full-link monitoring in an embodiment of the present invention;

[0041] Figure 2 200 is a flowchart of step 200 in an embodiment of the present invention;

[0042] Figure 3 300 is a flowchart of an embodiment of the present invention;

[0043] Figure 4 4 is a flow chart of step 400 in an embodiment of the present invention;

[0044] Figure 5 It is a block diagram of a distributed database monitoring system based on full-link monitoring in a specific application example of the present invention;

[0045] Figure 6 It is a structural schematic diagram of a link acquisition device in a specific application example of the present invention;

[0046] Figure 7 It is a structural schematic diagram of a message transfer device in a specific application example of the present invention;

[0047] Figure 8 It is a schematic diagram of the structure of a topology analysis device in a specific application example of the present invention;

[0048] Fig. 9 It is a structural schematic diagram of a polymer computing device in a specific application example of the present invention;

[0049] Fig.10 It is a structural schematic diagram of a detailed storage device in a specific application example of the present invention;

[0050] Fig.11 It is a structural schematic diagram of a visual monitoring device in a specific application example of the present invention;

[0051] Fig.12 It is a flow chart of a distributed database monitoring method based on full-link monitoring in a specific application example of the present invention;

[0052] Fig.13 Schematic diagram of the structure of a distributed database monitoring device based on full-link monitoring in an embodiment of the present invention;

[0053] Fig.14 It is a structural schematic diagram of an information collection unit in an embodiment of the present invention;

[0054] Fig.15 It is a structural schematic diagram of an abnormal monitoring unit in an embodiment of the present invention;

[0055] Fig.16 FIG. 4 is a schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] The embodiment of the present invention provides a specific implementation of a distributed database monitoring method based on full-link monitoring, see Figure 1 , which specifically includes the following contents:

[0058] Step 100: In response to a transaction request, generate a link corresponding to the transaction request.

[0059] It is understandable that the link in step 100 refers to a data link. Specifically, a link context is generated starting from a transaction request, and the link context needs to be transparently transmitted along the entire link to connect the links in series, and a complete link is generated based on the link context.

[0060] Step 200: Collect information about the link and operating indicators of the distributed database.

[0061] The information in step 200 includes: TRACEID (ID marking a call chain), SPANID (ID marking the current call in a link), PARENTSPANID (ID of the call in the previous stage, this field of the request entry is empty); the operation indicators are divided into indicators and details, the indicators include: whether it is successful, execution time, the details include SQL statement, TRACEID, hashcode of SQL statement, database address, whether there is an error, whether it is slow SQL, and SQL error stack.

[0062] Step 300: Generate a topological relationship corresponding to the link according to the information and the operation indicator;

[0063] In computer science, topology refers to the abstract way in which computers and various devices on a distributed network are interconnected. It does not care about the details of each computer or device, but cares about their interconnection relationships, and tries to express these relationships through graphs.

[0064] When step 300 is implemented, specifically, the links are connected in series according to SPANID and PARENTSPANID. First, it is determined whether the link is complete. If it is incomplete (for example, the link cannot be connected and the data is missing), it is not calculated and directly discarded. The complete link is processed into a topological relationship, such as interface A calls interface B, and interface B calls interface C. Then the interface names are spliced ​​and the hashcode is calculated.

[0065] Step 400: Monitor the distributed database according to the topological relationship.

[0066] It is understandable that the monitoring object of distributed tracing technology is each request (or transaction). By tracking the flow of requests, cross-process, cross-node, and cross-network operation monitoring (monitoring such as network time consumption, node execution time consumption, request flow, etc.) is realized, so it is also called full-link monitoring. According to the latest full-link monitoring standard (OPENTRACING), the principle of distributed tracing technology is to generate an ID (i.e., TRACEID) at the entry node of the request to mark a request. TRACEID will be transmitted through the entire call link, thereby connecting the distributed nodes in series. Each stage of a request is marked by generating a SPANID to indicate a step in the link. In addition, since the links are to be connected in series, it is also necessary to know the SPANID of the previous stage, i.e., PARENTSPANID. Therefore, PARENTSPANID is also transmitted through the network protocol stack, so that the entire link can be connected in series, and the call status of the entire link can be monitored.

[0067] From the above description, it can be seen that the embodiment of the present invention provides a distributed database monitoring method and device based on full-link monitoring. First, in response to a transaction request, a link corresponding to the transaction request is generated; then, the information of the link and the operating indicators of the distributed database are collected; the topological relationship corresponding to the link is generated according to the information and the operating indicators; finally, the distributed database is monitored according to the topological relationship. The present invention intercepts database access, collects SQL execution information, and reports it to the real-time analysis database for aggregation calculation. By calculating a hashcode for the topological structure of each link, the link and the topological structure are associated; the hashcode of the SQL statement is calculated and associated with the link. Finally, SQL and the topological structure are associated, so as to comprehensively examine the SQL execution of the system from point to surface.

[0068] In one embodiment, see Figure 2 , step 200 further comprises:

[0069] Step 201: embedding a point in the link;

[0070] The tracking technology is a commonly used data collection method for website analysis. It is a term in the field of data collection (especially the field of user behavior data collection), which refers to the relevant technologies and implementation processes for capturing, processing and sending specific user behaviors or events. For example, the number of times a user clicks on a certain icon, the length of time a certain video is watched, and so on. The essence of the tracking technology is to first monitor the events during the operation of the software application, and then judge and capture the events that need attention when they occur. Pay special attention to the need to clarify the time point and judgment conditions of the event. If you encounter any unclear points here, you need to communicate clearly with the developer to avoid differences between the collected data and the ideal. For example: you expect to collect the effective exposure number of a certain advertisement of a certain app. The judgment condition for effective exposure is that the stay time exceeds 1 second and the advertising content is effectively loaded.

[0071] Data collection is divided into three types: primary, intermediate, and advanced. Data collection is a good way to collect data in a private deployment.

[0072] Primary: embed statistical codes at key points of product and service conversion, and ensure that data collection is not repeated (such as purchase button click rate) based on their independent IDs;

[0073] Intermediate: Multiple codes are implanted to track the user's series of behaviors on each interface of the platform. The events are independent of each other (such as opening the product details page - selecting the product model - adding to the shopping cart - placing an order - completing the purchase);

[0074] Advanced: Cooperate with company engineering and ETL to collect and analyze all user behaviors, establish user portraits, and restore user behavior models as the basis for product analysis and optimization.

[0075] Step 202: Collect the tag ID of the link, the call ID of the current stage, the call ID of the previous stage and the operation index according to the tracking point.

[0076] Specifically, we intercept access to the database by tracking points and collect SQL execution information, including indicators and details, including: whether it is successful, execution time; SQL statement, TRACEID, SQL statement hashcode, database address, whether there is an error,

[0077] No, it is a slow SQL statement or SQL error stack. Determine by comparing the configured slow SQL time with the actual execution time (the default slow SQL time is 1 second).

[0078] In one embodiment, see Figure 3 , step 300 further comprises:

[0079] Step 301: Generate the topological relationship according to the tag ID of the link, the call ID of the current stage, the call ID of the previous stage and the operation index.

[0080] Specifically, each stage of a transaction can be marked with a SPANID (a link node, which can be a link ID, a link ID that can be a tag ID of the link, a call ID of the current stage, or a tag of a call ID of the previous stage) to indicate a step of the link. In addition, if you want to connect the links in series, you also need to know the SPANID of the previous stage, that is, the PARENT SPANID. Therefore, the PARENT SPANID is also transmitted through the network protocol stack so that the entire link can be connected in series to form a topological relationship. It should be noted that the operating indicators here refer to the entire SQL operating indicators, that is, the average execution time, number of times, success rate, and number of failures of SQL. That is, in the process of forming a topological relationship, the operating indicators of each SPANID need to be limited, so that the operation and maintenance personnel can evaluate whether there is a problem.

[0081] In one embodiment, see Figure 4 , step 400 further comprises:

[0082] Step 401: Calculate the corresponding hash code according to the topological relationship;

[0083] It is understandable that hashcode is used to improve the efficiency of accessing objects in HashMap. The hash algorithm maps a binary value of any length to a smaller binary value of fixed length. This small binary value is called a hash code (hash value). The hash value is a unique and extremely compact numerical identification form for a piece of data.

[0084] Step 402: Taking the hash code as a dimension, performing aggregate calculation on the distributed database at a preset time to generate an average SQL execution time, a sum of SQL execution times, a sum of SQL success times, and a sum of SQL exception times;

[0085] Specifically, aggregate functions in SQL (aggregate functions perform calculations on a set of values ​​and return a single value) can be used to perform calculations on distributed databases. It should be noted that all aggregate functions are deterministic. Whenever they are called with a given set of input values, the same value is returned. Aggregate functions can be applied to the SELECT or HAVING clause of a query statement, but cannot be used in a WHERE statement because WHERE is used to filter row records one by one.

[0086] Step 403: Monitor the distributed database according to the average SQL execution time, the sum of SQL execution times, the sum of SQL success times, and the sum of SQL exception times.

[0087] When step 403 is implemented, the queried data (average SQL execution time, sum of SQL execution times, sum of SQL success times, and sum of SQL exception times) can be visualized (various charts) for analysis by development and operation and maintenance personnel. The abnormal SQL analysis page displays abnormal SQL and its operation indicators, that is, SQL that exceeds 1s (slow SQL, 1s is the default value) and erroneous SQL. On the association analysis page, the association between specific SQL and topology is displayed, and all topological structures associated with a certain SQL statement are displayed. Then, based on the topology, it is associated with a specific link list (that is, links with the same topological structure), so that operation and maintenance personnel can conduct precise analysis from the link perspective. On the link analysis page, if there is database access in the link, the overall operation indicators of the SQL statement (the average execution time, number of times, success rate, and number of failures of this SQL) are provided to facilitate operation and maintenance personnel to evaluate whether there is a problem, such as a large difference between the execution time and the average value.

[0088] To further illustrate the present solution, the present invention also provides a specific application example of a distributed database monitoring method based on full-link monitoring, which specifically includes the following contents.

[0089] See also Figure 5 The present invention also provides a distributed database monitoring system based on full-link monitoring, which includes: a link collection device 1, a message transfer device 2, a topology analysis device 3, an aggregate calculation device 4, a detailed storage device 5, and a visual monitoring device 6. Device 1 is connected to device 2; device 2 is connected to device 3; device 2 is connected to device 4; device 2 is connected to device 5; device 3 is connected to device 5, device 3 is connected to device 6, and device 5 is connected to device 6. Specifically:

[0090] Link collection device 1: collects link information and SQL operation indicators in a buried point manner and reports them to device 2.

[0091] Message transfer device 2: a message transfer station for devices 3, 4, and 5 to subscribe to.

[0092] Topology analysis device 3: subscribes to link information and processes it, and the processed data is stored in device 5.

[0093] Aggregate calculation device 4: subscribes to the indicator data in device 2, performs aggregate calculation and stores it.

[0094] Detail storage device 5: subscribes to link data from device 2 and stores it, receives and stores the relationship reported by device 3, including: the relationship between SQL and link, the corresponding relationship between the hashcode of SQL statement and SQL statement, and the relationship between link and topology.

[0095] Visual monitoring device 6: query various monitoring data from device 4 and device 5, and perform visual processing and display.

[0096] Figure 6 Schematic diagram of the internal structure of the link collection device 1 in the present invention, Figure 6 As shown, the link collection device 1 includes: a link collection unit 11, an SQL information collection unit 12, a link reporting unit 13, and an SQL information reporting unit 14, wherein:

[0097] Link collection unit 11: Generate link context from the request entry. Link context needs to be transparently transmitted along the entire link to connect the links in series. The main fields are: TRACEID (ID marking a call chain), SPANID (ID marking the current call in a link), PARENTSPANID (ID of the previous stage call, this field is empty at the request entry). Unit 11 also needs to collect some information to assist in problem analysis, such as interface information, time consumption, network information, etc. at each stage.

[0098] SQL information collection unit 12: intercepts access to the database by means of embedding points, collects SQL execution information, including indicators and details. The indicators include: whether it is successful, execution time, and the details include SQL statement, TRACEID, SQL statement hashcode, database address, whether there is an error, whether it is slow SQL, and SQL error stack. Whether it is slow SQL is determined by comparing the configured slow SQL time with the actual execution time. The slow SQL time defaults to 1 second.

[0099] Link reporting unit 13: reports the link information collected in each stage to device 2.

[0100] SQL information reporting unit 14: reports the indicators collected by unit 12 (whether it is successful, execution time), the relationship between SQL and the link (TRACEID, hashcode of SQL statement, database address, whether there is an error, whether it is slow SQL, SQL error stack), the hashcode of each SQL statement and the corresponding relationship between the SQL statement (hashcode of SQL statement, SQL statement) to device 2.

[0101] Figure 7 Schematic diagram of the internal structure of the message transfer device 2 in the present invention. Figure 7 As shown, the message transfer device 2 includes a message receiving unit 21 and a message subscription unit 22, wherein:

[0102] Message receiving unit 21: adopts kafka message middleware, receives data reported by device 1, including 4 TOPICs: link, SQL operation index, SQL and link relationship, SQL statement hashcode and SQL statement correspondence.

[0103] Message subscription unit 22: Kafka is responsible for storing data in the file system and providing a network interface through which data consumers can subscribe to the data of the specified TOPIC.

[0104] Figure 8 is a schematic diagram of the internal structure of the topology analysis device 3 in the present invention, such as Figure 8 As shown, the topology analysis device 3 includes a data subscription unit 31, a data cache unit 32, a topology calculation unit 33, and a data reporting unit 34, wherein:

[0105] The data subscription unit 31 subscribes to the link TOPIC in the device 2 , ie, the link data reported by the unit 13 .

[0106] Data cache unit 32: caches link data for a period of time (default is five minutes) in the form of key-value pairs in memory, using TRACEID as key and link data of each stage as value. After timeout, it is taken out from the cache and the topology calculation unit 33 is called for calculation.

[0107] Topology calculation unit 33: Connect the links in series according to SPANID and PARENTSPANID. First, determine whether the link is complete. If it is incomplete (for example, the link cannot be connected or data is missing), it will be discarded without calculation. The complete link is processed into a topological relationship, such as interface A calling interface B, and interface B calling interface C. Then concatenate the interface names and calculate the hashcode.

[0108] The data reporting unit 34 reports the relationship between the topology hashcode and the TRACEID to the detail storage device 5, that is, the relationship details between the link and the topology.

[0109] Fig. 9 Schematic diagram of the internal structure of the aggregation computing device 4 in the present invention. The aggregation computing device 4 includes a data pulling unit 41, an aggregation computing unit 42, a data storage unit 43, and a data query unit 44, wherein:

[0110] Data pulling unit 41: uses the real-time computing database Druid to consume the data of the SQL operation index TOPIC from device 2.

[0111] Aggregate calculation unit 42: Druid performs aggregate calculations on a minute-by-minute basis for different types of indicators, taking the hashcode of SQL statements as the dimension. The calculations mainly include: averaging the SQL execution time, summing the number of SQL executions, summing the number of SQL successes, and summing the number of SQL exceptions.

[0112] The data storage unit 43 stores the aggregated calculated data in the file system.

[0113] Data query unit 44: provides an interface for external access and query of stored data.

[0114] Fig.10 Schematic diagram of the internal structure of the detailed storage device 5 in the present invention, such as Fig.10 As shown, the detailed storage device 5 includes a data subscription unit 51, a data receiving unit 52, a data storage unit 53, and a data query unit 54:

[0115] Data subscription unit 51: Device 5 uses Elastic Search database to actively subscribe to the data of these TOPICs from device 2: links, SQL and link relationships, SQL statement hashcode and SQL statement correspondence, and call data storage unit 53 to store in the specified table in the file system.

[0116] Data receiving unit 52: The Elastic Search database receives the relationship between the topology hashcode and TRACEID reported by the data reporting unit 34, that is, the relationship details between the link and the topology, and calls the data storage unit 53 to store it in the file system.

[0117] Data storage unit 53: Elastic Search database establishes 5 tables to store detailed data: link detail table, SQL and link relationship table, SQL statement hashcode and SQL statement correspondence table, link and topology relationship table.

[0118] Data query unit 54: provides a network interface to query detailed data in a specified table.

[0119] Fig.11 Schematic diagram of the internal structure of the visual monitoring device 6 in the present invention. Fig.11 As shown, the visual monitoring device 6 includes a data query unit 61 and a visual display unit 62:

[0120] Data query unit 61: Query data from device 4 and device 5. Query the operation index of a certain SQL from device 4. Query detailed data from device 5, such as abnormal SQL list (slow and error), topology list corresponding to a certain SQL, link list corresponding to a certain topology structure, and actual SQL statement corresponding to the hashcode of the SQL statement.

[0121] Visualization display unit 62: Visualize the queried data for analysis by development and operation and maintenance personnel. Display abnormal SQL and its operation indicators, for example: SQL that exceeds 1s (slow SQL, 1s is the default value) and erroneous SQL. On the association analysis page, the association between the specific SQL and the topology is displayed, and all topological structures associated with a certain SQL statement are displayed. Then, based on the topology, it is associated with a specific link list (i.e., links with the same topological structure), so that operation and maintenance personnel can perform accurate analysis from the link perspective. On the link analysis page, if there is database access in the link, the overall operation indicators of the SQL statement (the average execution time, number of times, success rate, and number of failures of this SQL) are provided to facilitate operation and maintenance personnel to evaluate whether there is a problem, such as the execution time is significantly different from the average value.

[0122] Based on the above-mentioned distributed database monitoring system based on full-link monitoring, the distributed database monitoring method based on full-link monitoring provided in this specific application example specifically includes the following steps, see Fig.12 .

[0123] Step S101: The link collection device 1 collects necessary information, error link data and SQL operation data by means of embedding points. The collected data is processed and sent to the message transfer device 2, which is mainly divided into details and indicators.

[0124] Step S102: The aggregate calculation device 4 consumes SQL operation indicator data from the message transfer device 2 and performs aggregate calculation on the indicators.

[0125] Step S103 : the topology analysis device 3 consumes link data from the device 2 , calculates the topological structure of the link, and stores the link and topological relationship in the detailed storage device 5 .

[0126] Step S104: the detail storage device 5 receives the detail data reported by the topology analysis device 3, actively subscribes to the detail data in the message transfer device 2, and stores these data in different tables.

[0127] Step S105: The visual monitoring device 6 queries data from the aggregate computing device 4 and the detailed storage device 5, and processes and displays the data to facilitate monitoring and analysis by the operation and maintenance personnel.

[0128] From the above description, it can be seen that the embodiment of the present invention provides a distributed database monitoring method based on full-link monitoring. In response to the problems in the prior art, enterprises need to combine points, lines and surfaces, and have a solution that can correlate link monitoring and abnormal SQL monitoring to better monitor and analyze the execution of database queries in the system. For example, from the link perspective, when a link is found to have slow SQL or abnormal SQL, the overall operation of this query statement can be grasped from point to surface, so as to clarify whether it is a point problem or a surface problem; from the SQL analysis perspective, when the operation and maintenance personnel find that a SQL query is slow overall, they can associate it with which specific links take a long time, and then drill down for analysis, first from surface to line, find a typical link, and then from line to point, and conduct a detailed analysis of the reasons for the slow SQL link.

[0129] Based on the same inventive concept, the embodiments of the present application also provide a distributed database monitoring device based on full-link monitoring, which can be used to implement the methods described in the above embodiments, such as the following embodiments. Since the principle of solving the problem by the distributed database monitoring device based on full-link monitoring is similar to that of the distributed database monitoring method based on full-link monitoring, the implementation of the distributed database monitoring device based on full-link monitoring can refer to the implementation of the distributed database monitoring method based on full-link monitoring, and the repeated parts will not be repeated. As used below, the terms "unit" or "module" can be a combination of software and / or hardware that implements predetermined functions. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0130] The embodiment of the present invention provides a specific implementation of a distributed database monitoring device based on full-link monitoring that can implement a distributed database monitoring method based on full-link monitoring, see Fig.13 The distributed database monitoring device based on full-link monitoring specifically includes the following contents:

[0131] The link generation unit 10 is used to generate a link corresponding to the transaction request in response to the transaction request;

[0132] An information collection unit 20, used to collect information of the link and operating indicators of the distributed database;

[0133] A topology relationship generating unit 30, configured to generate a topology relationship corresponding to the link according to the information and the operation indicator;

[0134] The abnormality monitoring unit 40 is used to monitor the distributed database according to the topological relationship.

[0135] In one embodiment, see Fig.14 , the information collection unit 20 includes:

[0136] A tracking module 201 is used to track the link;

[0137] The information collection module 202 is used to collect the tag ID of the link, the call ID of the current stage, the call ID of the previous stage and the operation index according to the tracking point.

[0138] In one embodiment, the topology relationship generating unit 30 is specifically configured to generate the topology relationship according to the tag ID of the link, the call ID of the current stage, the call ID of the previous stage, and the operation index.

[0139] In one embodiment, see Fig.15 , the abnormality monitoring unit 40 includes:

[0140] A hash code calculation module 401, used to calculate the corresponding hash code according to the topological relationship;

[0141] An aggregate calculation module 402 is used to perform aggregate calculation on the distributed database based on the hash code as a dimension and a preset time to generate an average SQL execution time, a sum of SQL execution times, a sum of SQL success times, and a sum of SQL exception times;

[0142] The database monitoring module 403 is used to monitor the distributed database according to the average SQL execution time, the sum of SQL execution times, the sum of SQL success times and the sum of SQL exception times.

[0143] From the above description, it can be seen that the embodiment of the present invention provides a distributed database monitoring device based on full-link monitoring, which calculates the investment methods of similar customers based on the natural attributes of the customers, obtains the recommended products of the combination fund, and uses the median algorithm to calculate the proportion of the product investment amount based on the customer's income and expenditure, thereby calculating the recommended purchase amount of the combination product. The present invention not only solves the confusion of account managers or customers in selecting products, but also clarifies the investment cycle and investment amount, and provides customers with a stable investment method. The investment method of the wealth management combination product also reduces the impact of market fluctuations and balances product returns and investment risks. In addition, since the recommended products are automatically pushed to customers on a regular basis, the bank's stickiness to customers is also increased.

[0144] The embodiments of the present application also provide a specific implementation of an electronic device that can implement all steps in the distributed database monitoring method based on full-link monitoring in the above embodiments, see Fig.16 , electronic equipment specifically includes the following:

[0145] Processor (processor) 1201, memory (memory) 1202, communication interface (CommunicationsInterface) 1203 and bus 1204;

[0146] The processor 1201, the memory 1202, and the communication interface 1203 communicate with each other through the bus 1204; the communication interface 1203 is used to realize information transmission between the server device and the client device and other related devices;

[0147] The processor 1201 is used to call the computer program in the memory 1202. When the processor executes the computer program, all steps in the distributed database monitoring method based on full-link monitoring in the above embodiment are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0148] Step 100: In response to a transaction request, generating a link corresponding to the transaction request;

[0149] Step 200: Collecting information of the link and operating indicators of the distributed database;

[0150] Step 300: Generate a topological relationship corresponding to the link according to the information and the operation indicator;

[0151] Step 400: Monitor the distributed database according to the topological relationship.

[0152] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all steps in the distributed database monitoring method based on full-link monitoring in the above embodiments. A computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, all steps of the distributed database monitoring method based on full-link monitoring in the above embodiments are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0153] Step 100: In response to a transaction request, generating a link corresponding to the transaction request;

[0154] Step 200: Collecting information of the link and operating indicators of the distributed database;

[0155] Step 300: Generate a topological relationship corresponding to the link according to the information and the operation indicator;

[0156] Step 400: Monitor the distributed database according to the topological relationship.

[0157] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the hardware + program embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0158] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0159] Although the present application provides method operation steps such as embodiments or flow charts, more or fewer operation steps may be included based on conventional or non-creative labor. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent the only execution order. When the actual device or client product is executed, it can be executed in the order of the method shown in the embodiments or the drawings or in parallel (for example, in a parallel processor or multi-threaded processing environment).

[0160] For the convenience of description, the above devices are described in various modules according to their functions. Of course, when implementing the embodiments of this specification, the functions of each module can be implemented in the same or more software and / or hardware, or the module implementing the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0161] Those skilled in the art also know that, in addition to implementing the controller in a purely computer-readable program code, the controller can be made to implement the same function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules for implementing the method and structures within the hardware component.

[0162] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0163] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0164] The present specification embodiments may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present specification embodiments may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0165] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. In the description of this specification, the description of the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of this specification. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, in the absence of contradiction, a person skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0166] The above is only an example of the embodiment of the present specification and is not intended to limit the embodiment of the present specification. For those skilled in the art, the embodiment of the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the embodiment of the present specification shall be included in the scope of the claims of the embodiment of the present specification.

Claims

1. A distributed database monitoring method based on full-link monitoring, characterized in that: include: In response to the transaction request, generating a link corresponding to the transaction request; Collecting information of the link and operating indicators of the distributed database; The information of the link includes: an ID marking a call chain, an ID marking the current call in a link, and an ID of the previous stage call; the operation indicators are divided into indicators and details, the indicators include: whether it is successful, execution time, and the details include: SQL statement, ID marking a call chain, hashcode of SQL statement, database address, whether there is an error, whether it is a slow SQL, and SQL error stack; Generate a topological relationship corresponding to the link according to the information and the operation indicator; The distributed database is monitored according to the topological relationship.

2. The distributed database monitoring method based on full-link monitoring according to claim 1 is characterized in that: The collecting of the information of the link and the operating indicators of the distributed database includes: Burying points on the link; The tag ID of the link, the call ID of the current stage, the call ID of the previous stage and the operation index are collected according to the tracking point.

3. The distributed database monitoring method based on full-link monitoring as claimed in claim 2 is characterized in that: Generating a topological relationship corresponding to the link according to the information and the operation indicator includes: The topological relationship is generated according to the tag ID of the link, the call ID of the current stage, the call ID of the previous stage and the operation index.

4. The distributed database monitoring method based on full-link monitoring according to claim 1 is characterized in that: Monitoring the distributed database according to the topological relationship includes: Calculate the corresponding hash code according to the topological relationship; Taking the hash code as a dimension, performing aggregate calculation on the distributed database at a preset time to generate an average SQL execution time, a sum of SQL execution times, a sum of SQL success times, and a sum of SQL exception times; The distributed database is monitored according to the average SQL execution time, the sum of SQL execution times, the sum of SQL success times and the sum of SQL exception times.

5. A distributed database monitoring device based on full-link monitoring, characterized in that: include: A link generation unit, configured to generate a link corresponding to the transaction request in response to the transaction request; The information collection unit is used to collect the information of the link and the operation index of the distributed database; the information of the link includes: an ID marking a call chain, an ID marking the current call in a link, and an ID of the previous stage call; the operation index is divided into indicators and details, the indicators include: whether it is successful, the execution time, and the details include: SQL statement, ID marking a call chain, hashcode of SQL statement, database address, whether there is an error, whether it is a slow SQL, and SQL error stack; A topology relationship generating unit, configured to generate a topology relationship corresponding to the link according to the information and the operation indicator; An abnormality monitoring unit is used to monitor the distributed database according to the topological relationship.

6. The distributed database monitoring device based on full-link monitoring according to claim 5, characterized in that: The information collection unit comprises: A tracking module, used for tracking points in the link; The information collection module is used to collect the tag ID of the link, the call ID of the current stage, the call ID of the previous stage and the operation index according to the tracking point.

7. The distributed database monitoring device based on full-link monitoring according to claim 6, characterized in that: The topology relationship generating unit is specifically configured to generate the topology relationship according to the tag ID of the link, the call ID of the current stage, the call ID of the previous stage, and the operation index.

8. The distributed database monitoring device based on full-link monitoring according to claim 5, characterized in that: The abnormality monitoring unit comprises: A hash code calculation module, used to calculate the corresponding hash code according to the topological relationship; An aggregation calculation module, used to perform aggregation calculation on the distributed database with the hash code as a dimension and at a preset time to generate an average SQL execution time, a sum of SQL execution times, a sum of SQL success times, and a sum of SQL exception times; The database monitoring module is used to monitor the distributed database according to the average SQL execution time, the sum of SQL execution times, the sum of SQL success times and the sum of SQL exception times.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the distributed database monitoring method based on full-link monitoring as described in any one of claims 1 to 4 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the distributed database monitoring method based on full-link monitoring described in any one of claims 1 to 4 are implemented.

Images