Link data acquisition and analysis method, computer device and computer readable storage medium

Abstract

The invention provides a link data acquisition and analysis method, a computer device and a computer readable storage medium, and the method comprises the steps: before executing a target database, reloading an interface before executing a used database connection pool SQL, judging whether the database currently executed by micro-service is the target database, and if yes, executing the target database; if yes, creating a span sub-chain in the call chain of the executed current thread, and setting the attribute of the span sub-chain; opening a span object, and storing the span object in a variable of the currently executed thread; and after executing the target database, reloading the interface before executing the used database connection pool SQL, and judging whether the database currently executed by the micro-service is the target database or not, if so, obtaining the created span object from the variables of the currently executed thread, ending the operation of the span object, and cleaning the variables of the previously executed thread. The invention further provides a computer device and a computer readable storage medium for implementing the method. According to the invention, the link data of the target database can be collected and analyzed.

Description

Technical Field

[0001] The present invention relates to the technical field of link data processing, and in particular, to a link data collection and analysis method, as well as a computer device and a computer-readable storage medium for implementing the method. Background Art

[0002] Microservice architecture is a service architecture based on cloud service implementation, which allows a single application to contain many small components or services that are loosely coupled and can be deployed separately. With the in-depth application of microservice architecture, various scenarios may encounter various abnormal situations when applying microservice architecture. Once an abnormal situation occurs, the most direct and reliable way to discover the abnormal problem is to check the time consumption and call error in the microservice call link.

[0003] Microservice call link monitoring is an intuitive way to view the service interface call performance. Microservice link monitoring through a distributed tracing system (Zipkin) is an important technical means. However, this method cannot support all databases. For example, for a specific database, such as the DAMO database, the existing method cannot perform link data collection, and monitoring of link data for a specific database is very necessary in full-link monitoring. Since it is difficult for existing technologies to collect link data from multiple different databases through a distributed tracing system, the collected link data is incomplete, which brings obstacles to the abnormal analysis of the microservice architecture, affecting the accuracy and efficiency of the abnormal analysis of the microservice architecture. Summary of the invention

[0004] The first object of the present invention is to provide a link data collection and analysis method that can collect and analyze link data from multiple databases.

[0005] The second object of the present invention is to provide a computer device for implementing the above-mentioned link data collection and analysis method.

[0006] The third object of the present invention is to provide a computer-readable storage medium for implementing the above-mentioned link data collection and analysis method.

[0007] To achieve the first purpose of the present invention, the link data collection and analysis method provided by the present invention includes reloading the SQL execution pre-interface of the database connection pool used before executing the target database, judging whether the database currently executed by the microservice is the target database, and if so, creating a new span subchain in the call chain of the current thread being executed, and setting the properties of the span subchain; opening the span object, and saving the span object in the variable of the currently executing thread; after executing the target database, reloading the SQL execution post-interface of the database connection pool used, and judging whether the database currently executed by the microservice is the target database, and if so, obtaining the created span object from the variable of the currently executing thread, and ending the running of the span object, and clearing the variables of the previously executed thread.

[0008] As can be seen from the above scheme, the present invention creates a span subchain on the currently executing thread and sets the properties of the span subchain. After executing the target database, the created span object is obtained from the variable of the currently executing thread to obtain the corresponding link data. In this way, for the originally incompatible database type, the corresponding link data can be obtained through the above method, thereby providing data support for accurate and efficient analysis of the anomalies of the microservice architecture.

[0009] A preferred solution is that creating a span subchain in the call chain of the currently executed thread includes: using a Zipkin tracing module to create a span subchain for the target database.

[0010] It can be seen that the above method can monitor the microservice link through the distributed tracking system (Zipkin), so that the link data can be collected and analyzed without fundamentally changing the monitoring method.

[0011] A further solution is that setting the attributes of the span subchain includes: setting a model for the span subchain to collect link data, the model including at least one of the following information: caller service name, caller IP, caller port, database name, database IP, database port, SQL statement summary, complete SQL statement, thread number for executing SQL, execution error information, SQL type, link ID, call time, and execution time.

[0012] It can be seen that by pre-setting the link model for storing SQL execution, the link data of SQL is collected according to the attributes of the predefined model based on the execution characteristics of SQL in the target database, and is associated and stored with the link data called by the microservice.

[0013] A further solution is that the SQL statement summary is a statement with a preset number of characters intercepted from the complete SQL statement.

[0014] A further solution is that the method also includes: displaying the collected link data: obtaining the time period to be displayed, selecting the target service under the current monitoring environment, and selecting the instance corresponding to the target service, obtaining the link list of the corresponding tracking ID from the call chain index; parsing all link data corresponding to the tracking ID, parsing the data according to a pre-set model, and saving it in the memory object of the model.

[0015] A further solution is to parse the data according to a preset model and then send the parsed data to a front-end device, which calls a chain component to display the link data graphically.

[0016] It can be seen that when the front-end device displays the microservice link, it also displays the SQL link, thereby displaying a more complete microservice link call situation, the performance of the analyzed SQL, and the execution error situation. By covering the link of the target database SQL execution in the microservice link monitoring, including the complete SQL statement executed in the target database, the execution time of the SQL statement, the number of executions, the number of errors and other data can be obtained, so as to facilitate the investigation of whether some time-consuming or abnormal call chains are caused by the database SQL execution. By performing real-time analysis of the SQL link data, the overall performance and error situation of the current SQL can be perceived.

[0017] A further solution is that the method also includes: displaying the collected link data: determining the display time period, selecting the target service under the current monitoring environment, and selecting the instance corresponding to the target service, and obtaining the corresponding link list from the call chain index; and separately counting the following data per unit time: the number of SQL executions, the average SQL execution time, the number of executions of the same SQL and the time ranking, and the SQL execution error details.

[0018] It can be seen that the above method can analyze and display the status of the link, allowing operation and maintenance personnel to quickly understand the operation status of the target database.

[0019] A further solution is that obtaining the corresponding link list from the call chain index includes: obtaining the corresponding link list from the call chain index according to the service name, service instance and preset time period of the service to be called.

[0020] To achieve the above second purpose, the computer device provided by the present invention includes a processor and a memory, the memory stores a computer program, and when the computer program is executed by the processor, each step of the above link data collection and analysis method is implemented.

[0021] In order to achieve the third objective mentioned above, the present invention provides a computer readable storage medium storing a computer program, and when the computer program is executed by a processor, each step of the link data collection and analysis method mentioned above is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a flow chart of the data collection process of the link data collection and analysis method embodiment of the present invention.

[0023] Figure 2 It is a flowchart of a first implementation mode of the data display process of the link data collection and analysis method embodiment of the present invention.

[0024] Figure 3 It is a flow chart of a second implementation mode of the data display process of the link data collection and analysis method embodiment of the present invention.

[0025] The present invention is further described below in conjunction with the accompanying drawings and embodiments. DETAILED DESCRIPTION

[0026] The link data collection and analysis method of the present invention can collect and analyze the call link data of the microservice architecture for various data sets, especially for the link data of the DAMO database. The link data collection and analysis method of the present invention can be implemented on a computer device, the computer device is provided with a processor and a memory, the memory stores a computer program, and the processor implements the above-mentioned link data collection and analysis method by executing the computer program.

[0027] Link data collection and analysis method embodiment: This embodiment is mainly aimed at the target database, such as the DAMO database, to collect and analyze link data in the microservice call. In the technology of collecting microservice link data based on the distributed tracking system (Zipkin), the collection of SQL execution link data is added in the link collection of the microservice by reloading the preset interface. The data collection format is a predefined model, so as to monitor the SQL link data in detail, and ensure that the link belongs to the sub-chain in the corresponding complete link through program control, and then save the collected data to the link index corresponding to Zipkin, and use ElasticSearch to store the link data. Finally, the collected link data is analyzed, and the collected link data and analysis results are displayed through the front-end device. Preferably, they are displayed in the form of charts, so that users can intuitively understand whether there is an abnormality in the call of the microservice.

[0028] Combine the following Figure 1The process of link data collection is introduced. This embodiment expands the code based on the druid database connection pool framework so that the microservice call can support the collection of link data for the target database.

[0029] First, in the processing before SQL execution, execute step S11 to reload the SQL pre-execution interface of the database connection pool used, and then execute step S12 to determine whether the currently executed data is the target data, for example, whether it is the DAMO database. If the database currently executed by the microservice is the target database, execute step S13. If the currently executed database is not the target database, the currently executed database may be a traditional microservice call that can support the database, and the link data can be collected in a traditional way.

[0030] If, after step S12, it is determined that the currently executing database is the target database, step S13 is executed to create a new span subchain (span subchain) in the call chain of the currently executing thread. For example, a new span subchain for the target database is created using Zipkin's tracing module (brave.Tracer).

[0031] Next, step S14 is executed to set the attributes of the span subchain. Specifically, the related attributes of the span subchain on SQL are set. The set related attributes are shown in Table 1 below.

[0032] Table 1

[0033] As can be seen from Table 1, when setting the properties of the span subchain, it is necessary to set the data collection model of the span subchain. The model is set according to the parameters of Table 1, that is, the model contains at least one of the following information: caller service name, caller IP, caller port, database name, database IP, database port, SQL statement summary, complete SQL statement, thread number for executing SQL, execution error information, SQL type, link ID, call time, and execution time. Among them, the SQL statement summary is a statement with a preset number of characters intercepted from the complete SQL statement, such as intercepting the first 256 characters of the complete SQL statement.

[0034] Next, step S15 is executed to open the span object and execute the target database. The microservice starts to call the corresponding component and saves the span object in the variable of the currently executing thread.

[0035] After SQL is executed, this embodiment needs to execute step S16 to reload the database connection pool SQL execution interface used, and then execute step S17 to determine whether the currently executed data is the target data, for example, whether it is the DAMO database. If the database currently executed by the microservice is the target database, execute step S18. If the currently executed database is not the target database, the currently executed database may be a traditional microservice call that can support the database, and the link data can be collected in a traditional way.

[0036] In step S18, it is necessary to obtain the previously created span object from the variables of the currently executing thread, and after obtaining the span object, execute step S19 to end the operation of the span object and clean up the variables in the currently executing thread. The above method can obtain the link data when the microservice architecture is called, and the collected link data can be analyzed later.

[0037] After analyzing the link data, the collected and analyzed data will be displayed through the front-end device so that the user can intuitively understand the link data. This embodiment can display the link data in two different ways. Figure 2 The first display method is to display data for a specific link. First, execute step S21 to obtain the time period to be displayed. For example, the user needs to select a time period for displaying link data. Preferably, the time period to be displayed is a fixed time period, for example, the default is 5 minutes. Then, execute step S22 to select the target service to be displayed in the current monitoring environment, and execute step S23 to select the instance corresponding to the target service, which can be the IP and port of the service, etc.

[0038] Then, step S24 is executed to obtain a link list of the trace ID from the call chain index according to the service name, service instance, determined time period, trace ID and other parameters of the called service. Then, step S25 is executed to parse all link data corresponding to the trace ID when a trace ID is selected, and parse the data for the SQL subchain according to a pre-set model, and save the parsed data in the memory object of the model.

[0039] Finally, execute step S26, pass the parsed data to the call chain component of the front-end device and display it graphically. At this time, the front-end device displays the link data in a graphical way, and the user can intuitively see the occupancy time of each link in the entire link, thereby determining whether there is any abnormality in the link data.

[0040] Another display method is based on SQL links. For details, see Figure 3First, step S31 is executed to determine the time period to be displayed. For example, the user needs to select a time period for displaying link data. Preferably, the time period to be displayed is a fixed time period, for example, the default is 5 minutes. Then, step S32 is executed to select the target service to be displayed in the current monitoring environment, and step S33 is executed to select the instance corresponding to the target service, which can be the IP and port of the service, etc.

[0041] Then, step S34 is executed to obtain the corresponding link list from the call chain index according to the service name, service instance, pre-set time period, trace ID and other parameters of the service to be called. Then, step S35 is executed to count multiple data in a unit time, for example, the unit time is one minute, and the counted data includes at least one of the following: the number of executions of SQL statements, the average execution time of SQL statements, the number of executions of the same SQL statement and the ranking of the time consumption, the details of SQL statement execution errors, etc.

[0042] Finally, execute step S36, transmit the parsed data to the front-end device and display it graphically. In this way, the front-end device displays the link data in a graphical way, and the user can intuitively see the occupancy time of each link in the entire link, thereby determining whether there is any abnormality in the link data.

[0043] This embodiment predefines a data collection model for storing SQL execution, collects data for SQL link data according to the attributes of the predefined model based on the execution characteristics of SQL in the target database, and associates and stores it with the link data called by the microservice. In addition, the front-end device can synchronously display the link data of SQL when displaying the microservice link, thereby displaying a more complete link call situation of the microservice, and can also analyze the performance and execution error situation of SQL. By covering the link of the target database SQL execution in the microservice link monitoring, including obtaining the complete SQL statement executed in the target database, the execution time of the SQL statement, the number of executions, the number of errors and other data, it is convenient to check whether some time-consuming or abnormal call chains are caused by the execution of database SQL statements. By performing real-time analysis on the link data of SQL, the overall performance and error situation of the current SQL can be perceived. In this way, the operation of various types of databases used by microservices can be monitored, and the compatibility of link data collection and analysis when microservices call multiple servers can be improved.

[0044] Computer device embodiment: The computer device of this embodiment can be an independently operating electronic device, which has a processor, a memory, and a computer program stored in the memory and executable on the processor, such as an information processing program for implementing the above-mentioned information processing method. When the processor executes the computer program, each step of the above-mentioned link data collection and analysis method is implemented.

[0045] For example, a computer program may be divided into one or more modules, one or more modules are stored in a memory and executed by a processor to complete each module of the present invention. One or more modules may be a series of computer program instruction segments capable of completing a specific function, and the instruction segments are used to describe the execution process of the computer program in a terminal device.

[0046] It should be noted that the terminal device may be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that the schematic diagram of the present invention is only an example of a terminal device and does not constitute a limitation on the terminal device. The terminal device may include more or fewer components than shown in the figure, or may combine certain components, or different components. For example, the terminal device may also include an input / output device, a network access device, a bus, etc.

[0047] The processor referred to in the present invention may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal device, and uses various interfaces and lines to connect various parts of the entire terminal device.

[0048] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the mobile phone, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0049] Computer readable storage medium: If the computer program stored in the computer device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, each step of the above-mentioned link data collection and analysis method can be implemented.

[0050] The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. Computer readable media may include: any entity or device capable of carrying computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0051] Finally, it should be emphasized that the above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various changes and modifications. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A link data collection and analysis method, characterized in that: include: Before executing the target database, reload the SQL pre-execution interface of the database connection pool used to determine whether the database currently executed by the microservice is the target database. If so, create a new span subchain in the call chain of the current thread being executed, and set the properties of the span subchain; Open the span object and save the span object in a variable of the currently executing thread; After executing the target database, the database connection pool SQL post-execution interface used is reloaded, and it is determined whether the database currently executed by the microservice is the target database. If so, the span object that has been created is obtained from the variables of the currently executed thread, and the span object is terminated, and the variables of the previously executed thread are cleaned up.

2. The link data collection and analysis method according to claim 1, characterized in that: Creating the span subchain in the call chain of the currently executed thread includes: using a Zipkin tracking module to create a span subchain for the target database.

3. The link data collection and analysis method according to claim 1, characterized in that: Setting the properties of the span subchain includes: setting a model for the span subchain to collect link data, the model containing at least one of the following information: caller service name, caller IP, caller port, database name, database IP, database port, SQL statement summary, complete SQL statement, thread number for executing SQL, execution error information, SQL type, link ID, call time, and execution time.

4. The link data collection and analysis method according to claim 3 is characterized in that: The SQL statement summary is a statement with a preset number of characters extracted from the complete SQL statement.

5. The link data collection and analysis method according to any one of claims 1 to 4, characterized in that: The method also includes: displaying the collected link data: Get the time period to be displayed, select the target service in the current monitoring environment, select the instance corresponding to the target service, and obtain the link list of the corresponding tracking ID from the call chain index; All link data corresponding to the tracking ID are parsed, the data is parsed according to a preset model, and stored in a memory object of the model.

6. The link data collection and analysis method according to claim 5, characterized in that: After parsing the data according to a preset model, the parsed data is sent to a front-end device, and the front-end device calls a chain component to graphically display the link data.

7. The link data collection and analysis method according to any one of claims 1 to 4, characterized in that: The method also includes: displaying the collected link data: Determine the time period for display, select the target service in the current monitoring environment, select the instance corresponding to the target service, and obtain the corresponding link list from the call chain index; The following data per unit time are collected separately: number of SQL executions, average SQL execution time, number of executions of the same SQL and ranking of execution time, and SQL execution error details.

8. The link data collection and analysis method according to claim 7, characterized in that: Acquiring the corresponding link list from the call chain index includes: acquiring the corresponding link list from the call chain index according to the service name, service instance and preset time period of the service to be called.

9. A computer device, characterized in that The method comprises a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, each step of the link data collection and analysis method as claimed in any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the link data collection and analysis method as described in any one of claims 1 to 8 is implemented.

Images