A method, device and storage medium for analyzing service topology of a distributed system

Abstract

The present invention discloses a service topology analysis method, device and storage medium for a distributed system. The service topology analysis method for the distributed system includes: collecting communication data during the service call process; the communication data is transmitted between various services in the form of real-time data stream; obtaining the real-time data of each service call through the real-time data stream, and storing it in the form of time series data; obtaining all time series data within the required query time period, and obtaining the topology data of each service through calculation. The present invention can obtain the latest call data to generate a real-time service topology map, helping technicians understand the latest system status; by using time series data to store call data, the call data of any time node can be accessed, so that the service topology map of any time range can be displayed, and further screening of topology data can be supported, so as to display a smaller range of topology maps.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed systems, and in particular to a service topology analysis method, device and storage medium for a distributed system. Background Art

[0002] With microservices becoming popular, it is common for a company to have hundreds or thousands of services. In this situation, technical personnel need to understand the calls and interdependencies between services in order to understand the operation of the entire system.

[0003] The technicians can obtain the request data between services by tracking the requests. The service topology analysis is based on this data. Currently, the following methods are mainly used for service topology analysis.

[0004] Scheduled full scan scans all call data within a certain period of time, counts the upstream and downstream information of each request, aggregates them into point-to-point relationships, and stores them in the database. When displaying, all data in the database is taken out, the relationship between points is depicted, and finally a service topology diagram is formed.

[0005] When the amount of call data is huge, this method takes a long time because it needs to scan the entire data. It usually needs to be run regularly, such as running yesterday's call data information today to form yesterday's service topology diagram for technical personnel to show. However, when technical personnel need to understand the real-time topology of the line, this solution cannot support it. And when technical personnel want to view fine-grained topology data (hourly or minute-level), the full scan solution cannot provide good support.

[0006] In view of this, the present invention patent is proposed. Summary of the invention

[0007] In order to solve the above problems, the present invention provides a service topology analysis method, device and storage medium for a distributed system. Specifically, the following technical solutions are adopted:

[0008] A service topology analysis method for a distributed system, comprising:

[0009] Collect communication data during service calls;

[0010] The communication data is transmitted between various services in the form of real-time data streams;

[0011] The real-time data of each service call is obtained through real-time data streams and stored in the form of time series data;

[0012] Obtain all time series data within the required query time period and obtain the topological data of each service through calculation.

[0013] As an optional implementation manner of the present invention, in a service topology analysis method for a distributed system described in the present invention, the storage in the form of time series data includes:

[0014] The timestamp is used as the horizontal axis, and the number of calls and / or the call duration from service Fi (i=1, 2, 3, ...) to Fj (j=1, 2, 3, ...) is used as the vertical axis;

[0015] A timestamp coordinate point is selected every set time interval T, and the real-time data corresponding to the set time interval T is obtained. The number of calls and / or call duration of the service Fi to Fj are calculated, and recorded at the corresponding coordinate point in the coordinate system from Fi to Fj.

[0016] As an optional implementation manner of the present invention, in a service topology analysis method for a distributed system described in the present invention, coordinate systems are established corresponding to the calling relationships of services Fi to Fj, and the horizontal and vertical axes of each coordinate system have the same data caliber.

[0017] As an optional implementation of the present invention, in a service topology analysis method for a distributed system described in the present invention, the coordinate points of all time series data of each service within the latest set time interval T on the coordinate system are obtained, and the time series data of each service are statistically calculated separately to obtain real-time topological data of each service.

[0018] As an optional implementation of the present invention, in a service topology analysis method for a distributed system described in the present invention, a historical time interval that needs to be queried is set, the coordinate points of all time series data of each service within the historical time interval on the coordinate system are obtained, the time series data of each service are statistically calculated separately, and the topological data of any time interval in the history of each service is obtained.

[0019] As an optional implementation of the present invention, in a service topology analysis method for a distributed system described in the present invention, the topology data of each service includes: the number of calls between services obtained by calculation, and / or the query rate per second, and / or the average time consumed.

[0020] As an optional implementation of the present invention, in a service topology analysis method for a distributed system described in the present invention, an inverted index is compiled for each stored time series data, topology information parameters displayed by the service are filtered through the inverted index, and a topology map is generated for display according to the filtered topology information parameters displayed by the service;

[0021] Optionally, the dimensions of the inverted index include service name, business line, and service type.

[0022] As an optional implementation manner of the present invention, in a service topology analysis method for a distributed system described in the present invention, the collection of communication data during the service call process includes: collecting communication data by embedding points in the service, and the embedding points support data collection for http, mysql, redis, and rpc protocols.

[0023] The present invention also provides a service topology analysis device for a distributed system, comprising:

[0024] Data collection module, collects communication data during service call process;

[0025] A data transmission module transmits the communication data between various services in the form of real-time data stream;

[0026] The data storage module obtains the real-time data of each service call through the real-time data stream and stores it in the form of time series data;

[0027] And the data analysis module obtains all time series data within the required query time period and obtains the topological data of each service through calculation.

[0028] The present invention also provides a storage medium storing a computer executable program, and when the computer executable program is executed, the service topology analysis method of a distributed system as described above is implemented.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] The real-time call data stream processed by the service topology analysis method of a distributed system of the present invention can obtain the latest call data to generate a real-time service topology map, helping technical personnel to understand the latest system status.

[0031] A service topology analysis method for a distributed system of the present invention uses time series data to store call data. We can access the call data of any time node, so that the service topology diagram of any time range can be displayed. It also supports further screening of the topology data to display a smaller range of topology diagrams. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Process of a service topology analysis method for a distributed system according to an embodiment of the present invention Figure 1 ;

[0033] Figure 2 A service topology diagram of a service topology analysis method for a distributed system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] To make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be described clearly and completely in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them.

[0035] Therefore, the following detailed description of the embodiments of the present invention is not intended to limit the scope of the invention claimed for protection, but merely represents some embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features and technical solutions in the embodiments may be combined with each other.

[0037] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0038] In the description of the present invention, it should be noted that the orientation or positional relationship indicated by the terms "upper", "lower", etc. is based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the invention product is usually placed when in use, or the orientation or positional relationship commonly understood by those skilled in the art. Such terms are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0039] See also Figure 1 As shown, a service topology analysis method for a distributed system in this embodiment includes:

[0040] Collect communication data during service calls;

[0041] The communication data is transmitted between various services in the form of real-time data streams;

[0042] The real-time data of each service call is obtained through real-time data streams and stored in the form of time series data;

[0043] Obtain all time series data within the required query time period and obtain the topological data of each service through calculation.

[0044] A data stream is an ordered sequence of bytes with a start and end point. It includes input streams and output streams. Real-time data streams are always running to extract data from the data source in real time and write it to the storage engine at the millisecond / second level, achieving real-time effects in data use and transmission.

[0045] Time series data refers to time series data. It is a data column recorded in chronological order. Each data in the same data column must be of the same caliber and must be comparable. Time series data can be a period number or a time point number.

[0046] Time series data has the following characteristics:

[0047] (1) Data is time-series and must have a timestamp: Networked devices continuously generate data according to a set cycle or triggered by external events. The time at which each data point is generated is very important for data calculation and analysis and must be recorded.

[0048] (2) Data is structured: Data from web crawlers, Weibo, and WeChat are all unstructured and can be text, pictures, videos, etc. However, data generated by IoT devices are often structured and numerical. For example, the current and voltage collected by smart meters can be represented by 4-byte standard floating-point numbers.

[0049] (3) Data are rarely updated: The data generated by networked devices are machine log data, which is generally not allowed and there is no need to modify it. There are very few scenarios where the collected raw data needs to be modified. However, for a typical information or Internet application, records can definitely be modified or deleted.

[0050] (4) The data source is unique: The data collected by one IoT device is completely independent of the data collected by another device. The data of a device must be generated by this device and cannot be generated by humans or other devices. In other words, the data of a device has only one producer and the data source is unique.

[0051] (5) Compared with Internet applications, there are more writes and fewer reads: For Internet applications, a data record is often written once and read many times. For example, a Weibo post or a WeChat public account article is written once, but it may be read by millions of people. However, the data generated by IoT devices is different. For the generated data, it is generally read automatically by calculation and analysis programs, and the number of calculations and analyses is not large. Only when analyzing scenarios such as accidents will people actively look at the original data.

[0052] (6) Users are concerned about trends over a period of time: For a bank record, or a Weibo or WeChat message, each one is very important to its users. However, for IoT data, the changes between each data point are not large, and are generally gradual. People are more concerned about a period of time, such as the trend of data changes in the past 5 minutes or the past hour, and generally do not pay attention to the data value at a specific time point.

[0053] (7) Data has a retention period: The collected data generally has a retention strategy based on time, such as only retaining it for one day, one week, one month, one year or even longer. In order to save storage space, it is best for the system to automatically delete it.

[0054] (8) Data query and analysis is often based on a time period and a group of devices: For IoT data, when doing calculations and analysis, the time range must be specified, not just a point in time or the entire history. Moreover, it is often necessary to analyze the data collected from a subset of IoT devices based on the analysis dimension, such as devices in a certain geographical area, devices of a certain model or batch, devices of a certain manufacturer, etc.

[0055] (9) In addition to storage and query, real-time analysis and computing operations are often required: For most Internet big data applications, offline analysis is more common. Even if there is real-time analysis, the requirements for real-time analysis are not high. For example, user profiling can be performed after accumulating a certain amount of user behavior data. However, for IoT applications, the requirements for real-time data computing are often very high, because real-time alarms need to be issued based on the calculation results to avoid accidents.

[0056] (10) Traffic is stable and predictable: Given the number of IoT devices and the frequency of data collection, the required bandwidth and traffic, as well as the size of new data generated each day, can be estimated more accurately.

[0057] (11) Peculiarities of data processing: Compared with the typical Internet, there are different data processing requirements. For example, if you want to check a certain quantity collected by a device at a specific time, but the actual collection time of the sensor is not at this point in time, then interpolation processing is often required. There are also many scenarios that require complex mathematical function calculations based on the collected quantity.

[0058] (12) Huge amount of data: Take smart meters as an example. A smart meter collects data every 15 minutes and automatically generates 96 records every day. There are nearly 500 million smart meters in China, generating nearly 50 billion records every day. A connected car collects data and sends it to the cloud every 10 to 15 seconds. A car can easily generate 1,000 records a day. If all 200 million cars in China are connected to the Internet, 200 billion records will be generated every day. Within 5 years, the data generated by IoT devices will account for more than 90% of the world's total data.

[0059] The real-time call data stream processed by the service topology analysis method of a distributed system in this embodiment can obtain the latest call data to generate a real-time service topology map, helping technical personnel understand the latest system status.

[0060] A service topology analysis method for a distributed system in this embodiment stores call data by using time series data. We can access the call data of any time node, so that we can display the service topology map of any time range, and also support further screening of the topology data, so as to display a smaller range of topology maps.

[0061] As an optional implementation of this embodiment, a service topology analysis method for a distributed system described in this embodiment, wherein the storage in the form of time series data includes:

[0062] The timestamp is used as the horizontal axis, and the number of calls and / or the call duration from service Fi (i=1, 2, 3, ...) to Fj (j=1, 2, 3, ...) is used as the vertical axis;

[0063] A timestamp coordinate point is selected every set time interval T, and the real-time data corresponding to the set time interval T is obtained. The number of calls and / or call duration of the service Fi to Fj are calculated, and recorded at the corresponding coordinate point in the coordinate system from Fi to Fj.

[0064] We store the collected data in the form of time series data, such as Figure 2 As shown in the figure, we take a timestamp as the horizontal axis at regular intervals, and on the vertical axis we can record the total number of calls from the FA service to the FB service during this period. Similarly, we can add another point to record the total call time from the FA service to the FB service. In this way, we add a point every once in a while, and a series of data with the same caliber is a series of time series data. By diverging in sequence, we can store the data of the FB service calling the FC service and the FC service calling the FD service in the form of time series.

[0065] Furthermore, in the service topology analysis method of a distributed system described in this embodiment, coordinate systems are established for the call relationship between services Fi and Fj, and the horizontal and vertical axes of each coordinate system have the same data caliber. Caliber = a standard / rule. For example, for a data warehouse, data caliber refers to the data collection standard of certain fields, such as the source or calculation method of a certain field should be consistent, then this is the standard; for an indicator, caliber is the calculation logic of this indicator, that is, the calculation standard.

[0066] The service topology analysis method of the distributed system of this embodiment can realize real-time topology data analysis of services, and can also realize diachronic topology data analysis of services. The specific implementation method is as follows:

[0067] The service topology analysis method of a distributed system described in this embodiment obtains the coordinate points of all time series data of each service within the latest set time interval T on the coordinate system, statistically calculates the time series data of each service respectively, and obtains real-time topology data of each service.

[0068] The service topology analysis method of a distributed system described in this embodiment sets the historical time interval that needs to be queried, obtains the coordinate points of all time series data of each service within the historical time interval on the coordinate system, statistically calculates the time series data of each service respectively, and obtains the topological data of any time interval in the history of each service.

[0069] Specifically, in a distributed system service topology analysis method described in this embodiment, the topology data of each service includes: the number of calls between services, and / or the query rate per second, and / or the average time consumed by calculation. The processing end of this embodiment obtains real-time data through a real-time data stream, and then parses the data to obtain various information such as the request initiator, the request receiver, the request initiation time, the request time consumption, and the request attributes.

[0070] Therefore, when we need to obtain real-time topology information, we obtain all the time series data points in the corresponding time period, and through calculation, we can obtain the number of calls between services, the query rate per second QPS, the average time consumption and other data. In this way, we can view the real-time topology data, and also support viewing the topology data at any time in history.

[0071] Furthermore, as an optional implementation of the present embodiment, a service topology analysis method for a distributed system described in the present embodiment compiles an inverted index for each stored time series data, filters the topology information parameters displayed by the service through the inverted index, and generates a topology map for display according to the filtered topology information parameters displayed by the service.

[0072] Optionally, the dimensions of the inverted index include service name, business line, and service type.

[0073] An inverted index is compiled for each time series data. The dimensions of the inverted index can be service name, business line, and service type. Through the inverted index, we can further refine the display of topological data, such as only displaying the topology diagram of a certain service or the topology diagram of services on a certain business line. A more flexible display method can better meet the needs of developers.

[0074] As an optional implementation of this embodiment, a service topology analysis method for a distributed system described in this embodiment, the collection of communication data during the service call process includes: collecting communication data by embedding points in the service, and the embedding points support data collection for http, mysql, redis, and rpc protocols.

[0075] Tracking, also known as tracking, is to collect data by tracking (writing code), adding a piece of code where the user behavior data needs to be monitored. We can call it the Capture mode, which tracks certain points on the client / server side, collects relevant data to the cloud, and finally presents it on the cloud. Tracking data after data verification is very accurate and stable, suitable for monitoring and analysis, and is a very effective method for non-exploratory analysis; and tracking can often add more business attributes, which is convenient for product managers to decompose and drill down the business attributes of events, and can well cut into behavior analysis from business logic and understand the business ideas behind the behavior.

[0076] This embodiment also provides a service topology analysis device for a distributed system, including:

[0077] Data collection module, which collects communication data during the service call process;

[0078] A data transmission module transmits the communication data between various services in the form of real-time data stream;

[0079] The data storage module obtains the real-time data of each service call through the real-time data stream and stores it in the form of time series data;

[0080] And the data analysis module obtains all time series data within the required query time period and obtains the topological data of each service through calculation.

[0081] A data stream is an ordered sequence of bytes with a start and end point. It includes input streams and output streams. Real-time data streams are always running to extract data from the data source in real time and write it to the storage engine at the millisecond / second level, achieving real-time effects in data use and transmission.

[0082] Time series data refers to time series data. It is a data column recorded in chronological order. Each data in the same data column must be of the same caliber and must be comparable. Time series data can be a period number or a time point number.

[0083] Time series data has the following characteristics:

[0084] (1) Data is time-series and must have a timestamp: Networked devices continuously generate data according to a set cycle or triggered by external events. The time at which each data point is generated is very important for data calculation and analysis and must be recorded.

[0085] (2) Data is structured: Data from web crawlers, Weibo, and WeChat are all unstructured and can be text, pictures, videos, etc. However, data generated by IoT devices are often structured and numerical. For example, the current and voltage collected by smart meters can be represented by 4-byte standard floating-point numbers.

[0086] (3) Data are rarely updated: The data generated by networked devices are machine log data, which is generally not allowed and there is no need to modify it. There are very few scenarios where the collected raw data needs to be modified. However, for a typical information or Internet application, records can definitely be modified or deleted.

[0087] (4) The data source is unique: The data collected by one IoT device is completely independent of the data collected by another device. The data of a device must be generated by this device and cannot be generated by humans or other devices. In other words, the data of a device has only one producer and the data source is unique.

[0088] (5) Compared with Internet applications, there are more writes and fewer reads: For Internet applications, a data record is often written once and read many times. For example, a Weibo post or a WeChat public account article is written once, but it may be read by millions of people. However, the data generated by IoT devices is different. For the generated data, it is generally read automatically by calculation and analysis programs, and the number of calculations and analyses is not large. Only when analyzing scenarios such as accidents will people actively look at the original data.

[0089] (6) Users are concerned about trends over a period of time: For a bank record, or a Weibo or WeChat message, each one is very important to its users. However, for IoT data, the changes between each data point are not large, and are generally gradual. People are more concerned about a period of time, such as the trend of data changes in the past 5 minutes or the past hour, and generally do not pay attention to the data value at a specific time point.

[0090] (7) Data has a retention period: The collected data generally has a retention strategy based on time, such as only retaining it for one day, one week, one month, one year or even longer. In order to save storage space, it is best for the system to automatically delete it.

[0091] (8) Data query and analysis is often based on a time period and a group of devices: For IoT data, when doing calculations and analysis, the time range must be specified, not just a single point in time or the entire history. Moreover, it is often necessary to analyze the data collected from a subset of IoT devices based on the analysis dimension, such as devices in a certain geographic area, devices of a certain model or batch, devices from a certain manufacturer, and so on.

[0092] (9) In addition to storage and query, real-time analysis and computing operations are often required: For most Internet big data applications, offline analysis is more common. Even if there is real-time analysis, the requirements for real-time analysis are not high. For example, user profiling can be performed after accumulating a certain amount of user behavior data. However, for IoT applications, the requirements for real-time data computing are often very high, because real-time alarms need to be issued based on the calculation results to avoid accidents.

[0093] (10) Traffic is stable and predictable: Given the number of IoT devices and the frequency of data collection, the required bandwidth and traffic, as well as the size of new data generated each day, can be estimated more accurately.

[0094] (11) Peculiarities of data processing: Compared with the typical Internet, there are different data processing requirements. For example, if you want to check a certain quantity collected by a device at a specific time, but the actual collection time of the sensor is not at this point in time, then interpolation processing is often required. There are also many scenarios that require complex mathematical function calculations based on the collected quantity.

[0095] (12) Huge amount of data: Take smart meters as an example. A smart meter collects data every 15 minutes and automatically generates 96 records every day. There are nearly 500 million smart meters in China, generating nearly 50 billion records every day. A connected car collects data and sends it to the cloud every 10 to 15 seconds. A car can easily generate 1,000 records a day. If all 200 million cars in China are connected to the Internet, 200 billion records will be generated every day. Within 5 years, the data generated by IoT devices will account for more than 90% of the world's total data.

[0096] A real-time call data stream processed by a service topology analysis device of a distributed system in this embodiment can obtain the latest call data to generate a real-time service topology map, helping technical personnel understand the latest system status.

[0097] A service topology analysis device for a distributed system of this embodiment stores call data by using time series data. We can access the call data of any time node, so that we can display the service topology map of any time range, and also support further screening of the topology data, so as to display a topology map of a smaller range.

[0098] As an optional implementation of this embodiment, in a service topology analysis device for a distributed system described in this embodiment, the data storage module adopts a time series data storage method to include:

[0099] The timestamp is used as the horizontal axis, and the number of calls and / or the call duration from service Fi (i=1, 2, 3, ...) to Fj (j=1, 2, 3, ...) is used as the vertical axis;

[0100] A timestamp coordinate point is selected every set time interval T, and the real-time data corresponding to the set time interval T is obtained. The number of calls and / or call duration of service Fi to Fj are calculated, and recorded at the corresponding coordinate point in the coordinate system from Fi to Fj.

[0101] For the collected data, the data storage module uses time series data to store, such as Figure 2 As shown in the figure, we take a timestamp as the horizontal axis at regular intervals, and on the vertical axis we can record the total number of calls from the FA service to the FB service during this period. Similarly, we can add another point to record the total call time from the FA service to the FB service. In this way, we add a point every once in a while, and a series of data with the same caliber is a series of time series data. By diverging in sequence, we can store the data of the FB service calling the FC service and the FC service calling the FD service in the form of time series.

[0102] Furthermore, in the service topology analysis device of a distributed system described in this embodiment, the data storage module establishes coordinate systems corresponding to the calling relationship of services Fi to Fj, and the horizontal and vertical axes of each coordinate system have the same data caliber. Caliber = a standard / rule. For example, for a data warehouse, data caliber refers to the data collection standard of certain fields, such as the source or calculation method of a certain field should be consistent, then this is the standard; for an indicator, the caliber is the calculation logic of this indicator, that is, the calculation standard.

[0103] The service topology analysis device of the distributed system of this embodiment can realize real-time topology data analysis of services, and can also realize diachronic topology data analysis of services. The specific implementation method is as follows:

[0104] In the service topology analysis device of a distributed system described in this embodiment, the data analysis module obtains the coordinate points of all time series data of each service within the latest set time interval T on the coordinate system, and statistically calculates the time series data of each service to obtain real-time topology data of each service.

[0105] A service topology analysis device for a distributed system described in this embodiment sets the historical time interval that needs to be queried. The data analysis module obtains the coordinate points of all time series data of each service within the historical time interval on the coordinate system, and statistically calculates the time series data of each service to obtain the topological data of any time interval in the history of each service.

[0106] Specifically, in a service topology analysis device of a distributed system described in this embodiment, the data analysis module obtains the topology data of each service by calculating the number of calls between services, and / or the query rate per second, and / or the average time consumption. The processing end of this embodiment obtains real-time data through a real-time data stream, and then parses the data to obtain various information such as the request initiator, the request receiver, the request initiation time, the request time consumption, and the request attributes.

[0107] Therefore, when we need to obtain real-time topology information, we obtain all the time series data points in the corresponding time period, and through calculation, we can obtain the number of calls between services, the query rate per second QPS, the average time consumption and other data. In this way, we can view the real-time topology data, and also support viewing the topology data at any time in history.

[0108] Furthermore, as an optional implementation of the present embodiment, a service topology analysis device of a distributed system described in the present embodiment includes an index module, which compiles an inverted index for each stored time series data, filters the topology information parameters displayed by the service through the inverted index, and generates a topology map for display according to the filtered topology information parameters displayed by the service.

[0109] Optionally, the dimensions of the inverted index include service name, business line, and service type.

[0110] The index module compiles an inverted index for each time series data. The dimensions of the inverted index can be service name, business line, and service type. Through the inverted index, we can further refine the display of topological data, such as only displaying the topology diagram of a certain service or the topology diagram of services on a certain business line. A more flexible display method can better meet the needs of developers.

[0111] As an optional implementation of this embodiment, a service topology analysis device for a distributed system described in this embodiment, the data acquisition module collects communication data in the service call process including: collecting communication data by embedding points in the service, and the embedding points support data collection for http, mysql, redis, and rpc protocols.

[0112] Tracking, also known as tracking, is to collect data by tracking (writing code), adding a piece of code where the user behavior data needs to be monitored. We can call it the Capture mode, which tracks certain points on the client / server side, collects relevant data to the cloud, and finally presents it on the cloud. Tracking data after data verification is very accurate and stable, suitable for monitoring and analysis, and is a very effective method for non-exploratory analysis; and tracking can often add more business attributes, which is convenient for product managers to decompose and drill down the business attributes of events, and can well cut into behavior analysis from business logic and understand the business ideas behind the behavior.

[0113] This embodiment also provides a storage medium storing a computer executable program. When the computer executable program is executed, the service topology analysis method of a distributed system is implemented.

[0114] The storage medium described in the present embodiment may include a data signal propagated in a baseband or as a part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0115] This embodiment further provides an electronic device, including a processor and a memory, wherein the memory is used to store a computer executable program, and when the computer program is executed by the processor, the processor executes the service topology analysis method of a distributed system.

[0116] The electronic device is presented in the form of a general-purpose computing device. The processor may be one or more than one and work in coordination. The present invention does not exclude distributed processing, that is, the processor may be dispersed in different physical devices. The electronic device of the present invention is not limited to a single entity, but may also be the sum of multiple physical devices.

[0117] The memory stores a computer executable program, which is usually a machine-readable code. The computer-readable program can be executed by the processor to enable the electronic device to perform the method of the present invention, or at least some steps in the method.

[0118] The memory includes a volatile memory, such as a random access memory unit (RAM) and / or a cache memory unit, and may also be a non-volatile memory, such as a read-only memory unit (ROM).

[0119] It should be understood that the electronic device of the present invention may also include elements or components not shown in the above examples. For example, some electronic devices also include display units such as display screens, and some electronic devices also include human-computer interaction elements such as buttons, keyboards, etc. As long as the electronic device can execute the computer-readable program in the memory to implement the method of the present invention or at least part of the steps of the method, it can be considered as an electronic device covered by the present invention.

[0120] Through the above description of the implementation mode, it is easy for those skilled in the art to understand that the present invention can be implemented by hardware capable of executing a specific computer program, such as the system of the present invention, and the electronic processing unit, server, client, mobile phone, control unit, processor, etc. contained in the system. The present invention can also be implemented by computer software that executes the method of the present invention, such as control software executed by a microprocessor, an electronic control unit, a client, a server, etc. However, it should be noted that the computer software that executes the method of the present invention is not limited to being executed by one or a specific hardware entity, and it can also be implemented in a distributed manner by unspecified specific hardware. For computer software, the software product can be stored in a computer-readable storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), and can also be distributed and stored on the network, as long as it can enable the electronic device to execute the method according to the present invention.

[0121] The above embodiments are only used to illustrate the present invention and are not intended to limit the technical solutions described in the present invention. Although the present invention has been described in detail with reference to the above embodiments, the present invention is not limited to the above specific implementation methods. Therefore, any modification or equivalent replacement of the present invention; and all technical solutions and improvements thereof that do not depart from the spirit and scope of the invention are included in the scope of the claims of the present invention.

Claims

1. A service topology analysis method for a distributed system, characterized in that: include: Collect communication data during service calls; The communication data is transmitted between various services in the form of real-time data streams; The real-time data of each service call is obtained through real-time data streams and stored in the form of time series data; Obtain all time series data within the required query period, and obtain the topological data of each service through calculation; The storage in the manner of time series data includes: The timestamp is used as the horizontal axis, and the number of calls or the call duration from service Fi to Fj is used as the vertical axis, where i=1, 2, 3, ..., j=1, 2, 3, ...; Select a timestamp coordinate point every set time interval T, obtain the real-time data corresponding to the set time interval T, calculate the number of calls or call duration of service Fi to Fj, and record it at the corresponding coordinate point on the coordinate system from Fi to Fj; The collected data is stored in the form of time series data. A timestamp is taken as the horizontal axis at regular intervals. The total number of calls from FA service to FB service within this time period is recorded on the vertical axis. The total call time from FA service to FB service is recorded on the coordinate axis. A point is marked every once in a while. A series of data with the same caliber is a string of time series data. The data of FB service calling FC service and FC service calling FD service is stored in the form of time series. Corresponding to the calling relationship of service Fi to Fj, coordinate systems are established respectively, and the horizontal and vertical axes of each coordinate system are All have the same data caliber; Get the coordinate points of all time series data within the latest set time interval T of each service on the coordinate system. Calculate the time series data of each service separately to obtain the real-time topological data of each service; Set the historical time interval to be queried to obtain the historical time interval of each service in the coordinate system. The coordinate points of all time series data are counted and calculated for each service, and the topological data of each service at any time interval in history is obtained.

2. A method for analyzing service topology of a distributed system according to claim 1, characterized in that: The topology data of each service includes: the number of calls between services obtained by calculation, and / or the query rate per second, and / or the average time consumed.

3. The service topology analysis method of a distributed system according to claim 1, characterized in that: An inverted index is compiled for each piece of time series data stored, and the topology information parameters displayed by the filtering service are used to generate a topology map for display according to the topology information parameters displayed by the filtering service; The dimensions of the inverted index include service name, business line, and service type.

4. The service topology analysis method of a distributed system according to claim 1, characterized in that: The communication data collected during the service call process includes: collecting communication data by embedding points in the service, and the embedding points support data collection for http, mysql, redis, and rpc protocols.

5. A service topology analysis device for a distributed system, characterized in that: include: Data collection module, collects communication data during service call process; A data transmission module transmits the communication data between various services in the form of real-time data stream; The data storage module obtains the real-time data of each service call through the real-time data stream and stores it in the form of time series data; And the data analysis module obtains all time series data within the required query time period and obtains the topological data of each service through calculation; The data storage module adopts the method of storing time series data, including: The timestamp is used as the horizontal axis, and the number of calls or the call duration from service Fi to Fj is used as the vertical axis, where i=1, 2, 3, ..., j=1, 2, 3, ...; Select a timestamp coordinate point every set time interval T, obtain the real-time data corresponding to the set time interval T, calculate the number of calls or call duration of service Fi to Fj, and record it at the corresponding coordinate point on the coordinate system from Fi to Fj; The data storage module establishes coordinate systems corresponding to the calling relationship of services Fi to Fj, and the horizontal and vertical axes of each coordinate system have the same data caliber; The collected data is stored in the form of time series data. A timestamp is taken as the horizontal axis at regular intervals. The total number of calls from FA service to FB service within this time period is recorded on the vertical axis. The total call time from FA service to FB service is recorded on the coordinate axis. A point is marked every once in a while. A series of data with the same caliber is a string of time series data. The data of FB service calling FC service and FC service calling FD service is stored in the form of time series. The data analysis module obtains the coordinate points of all time series data within the latest set time interval T of each service on the coordinate system, and statistically calculates the time series data of each service to obtain the real-time topological data of each service; Set the historical time interval to be queried. The data analysis module obtains the coordinate points of all time series data of each service within the historical time interval on the coordinate system, calculates the time series data of each service separately, and obtains the topological data of each service at any time interval in history.

6. A storage medium storing a computer executable program, characterized in that: When the computer executable program is executed, a service topology analysis method for a distributed system as described in any one of claims 1 to 4 is implemented.

Images