Microservice system observability system based on knowledge graph

Abstract

The invention discloses a knowledge graph-based microservice system observability method. The method comprises the following steps of S1, collecting an online microservice call log; s2, establishing a knowledge graph model; s3, constructing a knowledge graph by the graph database; and S4, performing observability by using a knowledge graph. According to the method, the service, the interface and the dependency relationship are modeled through a method for constructing the knowledge graph, so that knowledge inference and treatment can be carried out.

Description

Technical Field

[0001] The present invention relates to the fields of microservices and distributed systems, and in particular to an observability method for a microservice system based on a knowledge graph. Background Art

[0002] Microservice architecture is suitable for large-scale distributed systems. The entire application of microservice architecture is divided into multiple services, and the services communicate with each other through the network. After the services are split, almost all functions will involve multiple services, which will cause the following problems: (1) Service stability decreases. The probability of service problems increases, and the services associated with a service failure will also fail; (2) Problem troubleshooting is difficult. Due to the complex mesh calling relationship between services, it is very difficult to locate the fault point; (3) Service release, maintenance, and governance are difficult. Whether the service interface is still in use and the rationality of the relationship between services make service governance very difficult.

[0003] The observability method is a method for observing the internal state of a system based on output information. The present invention provides a method for observability of a microservice system based on a knowledge graph. The method constructs a microservice cluster network through a knowledge graph, observes the state of microservices according to the observability method, and performs global and local governance of microservices based on the observed state. Based on this, the present invention discloses a method for observability of a microservice system based on a knowledge graph to solve the problem of difficult microservice governance. Summary of the invention

[0004] The present invention aims at the deficiencies in the prior art and provides a knowledge graph-based microservice system observability method to solve the difficulties in microservice governance and meet actual usage requirements.

[0005] To solve the above problems, the technical solutions adopted by the present invention are as follows:

[0006] A knowledge graph-based microservice system observability method includes the following steps:

[0007] S1. Collect online microservice call logs

[0008] Collect historical microservice call logs. The specific data collected include: whether it is a gateway service, service name, interface name, called service name, called interface name, and first call time. Each service generally has multiple interfaces available for calling. For example, the login service has a text message sending interface, a text message verification interface, a login interface, a logout interface, etc.

[0009] An example of a collected data entry could be: falseloginService, login, passwordService, verifyPassword, 2022-02-01, 10:11:50; the specific explanation is:

[0010] (1) false means that the login service is not a gateway service, which means that the network layer does not first pass the login behavior initiated by the user through the app to the loginService service;

[0011] (2) The login interface login of the login service loginService calls the password verification interface verifyPassword of the password service passwordService;

[0012] (3) 2022-02-01 10:11:50 is the first call time, indicating that the login interface of the login service calls the verification password interface of the password service from then on, that is, it starts to rely on the verification password interface from this time point;

[0013] S2. Build a knowledge graph model

[0014] The knowledge graph defines two types of entity nodes: service nodes and interface nodes. The attributes of service nodes include name: value service name, unique mark of service; service type: gateway type, service type; gateway is the entry network service initiated by network program, and service is the non-entry network service; creation time: this time is the first time the service appears in the collected log; the attributes of interface nodes include name: here we select service name.interface name as the name of the interface node. For example, the name of service name a and interface name 1 is expressed as a.1. This encoding can confirm the unique interface by name, that is, each interface node represents a unique interface;

[0015] Interface type: gateway interface, service interface; the gateway interface type indicates that this interface is a gateway service interface; the service interface indicates that this interface is a non-gateway service interface;

[0016] Creation time: This is the time when the interface first appears in the collected logs;

[0017] The knowledge graph defines three types of relationship nodes: the relationship between services (service dependency), the relationship between interfaces (interface dependency), and the relationship between services and interfaces (with dependency). All three types are unidirectional relationships.

[0018] The attributes of the relationship between services (service dependency) include name: here we choose the calling service name_dependent service name as the relationship name. For example, service a depends on service b, and the name is a_b.

[0019] Dependency start time: such as 2022-02-01 10:11:50, which means that service a depends on service b from this time on; the attributes of the relationship between interfaces (interface dependency) include name: here, the name of the calling interface node_the name of the dependent interface node is selected as the name of the relationship. For example, if a.1 depends on b.1, it is recorded as a.1_b.1; Dependency start time: such as 2022-02-0110:11:50, which means that interface a.1 depends on interface b.1 from this time on;

[0020] The properties of the relationship (with dependencies) between services and interfaces are:

[0021] (1) Name: Here, the name of the interface* is selected as the name of the relationship. For example, if a depends on interface a.1, it is recorded as a.1*.

[0022] (2) Interface start time: For example, 2022-02-01 10:11:50, which means that service a has added interface a.1 since this time;

[0023] S3, graph database to build knowledge graph

[0024] You can choose a graph knowledge database to build a knowledge graph, such as neo4j. The construction method is:

[0025] First, build the entity node, create corresponding graph elements for the services and interfaces in the S1 log, and obtain the corresponding graph element numbers;

[0026] Rebuild the relationship node again, and parse the log of S1 to find out the service dependency, interface dependency, and included dependency; for example, falseloginService, login, passwordService, verifyPassword, 2022-02-0110:11:50, the service dependency is loginService depends on passwordService; the interface dependency is loginService.login depends on passwordService.verifyPassword; the included dependency is loginService has loginService.login;

[0027] An example of a graph constructed is Figure 2 ;

[0028] S4. Using knowledge graphs for observability

[0029] 1. Global observability.

[0030] (1) Maximum global network depth

[0031] All gateway nodes can be regarded as starting nodes, and nodes without out-degree can be regarded as end nodes. The maximum path length from all gateway nodes to the end node can be calculated, and the maximum path lengths of all nodes are sorted. For the path lengths ranked at the top, the service and interface governance solution should be adopted. For users, the deepest call link has a high probability of failure.

[0032] (2) Global abnormal service automatically locates the root cause service

[0033] If a service is known to behave abnormally, then check whether the services it depends on are also abnormal. If the dependent services are abnormal, continue searching until the abnormal service at the lowest level is found. If the dependent services are normal, then the service that failed this time is the service. This way, the root cause of the problem can be identified.

[0034] (3) Globally identify new dependencies

[0035] It can automatically list the dependencies of services and interfaces added daily. These are changes in the network. For these new dependencies, it may be that the system has abnormal behavior and has called logic that has not been reached before. It is necessary to manually check and confirm whether the system is working normally.

[0036] 2. Local service observability

[0037] Service governance can be performed manually. When a service is selected, the knowledge graph can make the following inferences. Based on the inferred knowledge, manual service stability governance and optimization can be performed.

[0038] (1) Dependent services

[0039] (2) All dependent interfaces

[0040] (3) Gateway interface affected by services

[0041] (4) Whether there is a self-loop in the service

[0042] (5) Impacted core interfaces

[0043] (6) Dependent interfaces added after a certain point in time

[0044] 3. Local interface observability

[0045] Interface management can be done manually. Select an interface and the knowledge graph can make the following inferences. Based on the inferred knowledge, manual interface stability management and optimization can be performed.

[0046] (1) Display link depth (maximum distance to the gateway)

[0047] (2) Affected core interfaces

[0048] (3) Core services affected

[0049] (4) Affected gateway interface

[0050] (5) Whether the interface has any self-loop.

[0051] Compared with the prior art, the implementation effects of the present invention are as follows:

[0052] The present invention provides a knowledge graph-based microservice system observability method to solve the governance problems of microservices and distributed systems. The method models services, interfaces, and dependencies by constructing a knowledge graph, thereby enabling knowledge inference and governance.

[0053] The advantage of this method is that it can build a knowledge graph based on the system call log without manual intervention, and use it to describe the observability of the microservice cluster, thereby completing the governance of microservices. This method can reduce the difficulty of microservice governance. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a schematic diagram of the process of the present invention;

[0055] Figure 2 A schematic diagram of constructing a knowledge graph for the graph database of the present invention. DETAILED DESCRIPTION

[0056] The content of the present invention will be described below with reference to specific embodiments.

[0057] like Figure 1 and Figure 2 As shown in the figure, a knowledge graph-based microservice system observability method includes the following steps:

[0058] S1. Collect online microservice call logs

[0059] Collect historical microservice call logs. The specific data collected include: whether it is a gateway service, service name, interface name, called service name, called interface name, and first call time. Each service generally has multiple interfaces available for calling. For example, the login service has a text message sending interface, a text message verification interface, a login interface, a logout interface, etc.

[0060] An example of a collected data entry could be: falseloginService, login, passwordService, verifyPassword, 2022-02-01, 10:11:50; the specific explanation is:

[0061] (1) false means that the login service is not a gateway service, which means that the network layer does not first pass the login behavior initiated by the user through the app to the loginService service;

[0062] (2) The login interface login of the login service loginService calls the password verification interface verifyPassword of the password service passwordService;

[0063] (3) 2022-02-01 10:11:50 is the first call time, indicating that the login interface of the login service calls the verification password interface of the password service from then on, that is, it starts to rely on the verification password interface from this time point;

[0064] S2. Build a knowledge graph model

[0065] The knowledge graph defines two types of entity nodes: service nodes and interface nodes. The attributes of service nodes include name: value service name, unique mark of service; service type: gateway type, service type; gateway is the entry network service initiated by network program, and service is the non-entry network service; creation time: this time is the first time the service appears in the collected log; the attributes of interface nodes include name: here we select service name.interface name as the name of the interface node. For example, the name of service name a and interface name 1 is expressed as a.1. This encoding can confirm the unique interface by name, that is, each interface node represents a unique interface;

[0066] Interface type: gateway interface, service interface; the gateway interface type indicates that this interface is a gateway service interface; the service interface indicates that this interface is a non-gateway service interface;

[0067] Creation time: This is the time when the interface first appears in the collected logs;

[0068] The knowledge graph defines three types of relationship nodes: the relationship between services (service dependency), the relationship between interfaces (interface dependency), and the relationship between services and interfaces (with dependency). All three types are unidirectional relationships.

[0069] The attributes of the relationship between services (service dependency) include name: here we choose the calling service name_dependent service name as the relationship name. For example, service a depends on service b, and the name is a_b.

[0070] Dependency start time: such as 2022-02-01 10:11:50, which means that service a depends on service b from this time on; the attributes of the relationship between interfaces (interface dependency) include name: here, the name of the calling interface node_the name of the dependent interface node is selected as the name of the relationship. For example, if a.1 depends on b.1, it is recorded as a.1_b.1; Dependency start time: such as 2022-02-0110:11:50, which means that interface a.1 depends on interface b.1 from this time on;

[0071] The properties of the relationship (with dependencies) between services and interfaces are:

[0072] (1) Name: Here, the name of the interface* is selected as the name of the relationship. For example, if a depends on interface a.1, it is recorded as a.1*.

[0073] (2) Interface start time: For example, 2022-02-01 10:11:50, which means that service a has added interface a.1 since this time;

[0074] S3, graph database to build knowledge graph

[0075] You can choose a graph knowledge database to build a knowledge graph, such as neo4j. The construction method is:

[0076] First, build the entity node, create corresponding graph elements for the services and interfaces in the S1 log, and obtain the corresponding graph element numbers;

[0077] Rebuild the relationship node again, and parse the log of S1 to find out the service dependency, interface dependency, and included dependency; for example, falseloginService, login, passwordService, verifyPassword, 2022-02-0110:11:50, the service dependency is loginService depends on passwordService; the interface dependency is loginService.login depends on passwordService.verifyPassword; the included dependency is loginService has loginService.login;

[0078] An example of a graph constructed is Figure 2 ;

[0079] S4. Using knowledge graphs for observability

[0080] 1. Global observability.

[0081] (1) Maximum global network depth

[0082] All gateway nodes can be regarded as starting nodes, and nodes without out-degree can be regarded as end nodes. The maximum path length from all gateway nodes to the end node can be calculated, and the maximum path lengths of all nodes are sorted. For the path lengths ranked at the top, the service and interface governance solution should be adopted. For users, the deepest call link has a high probability of failure.

[0083] (2) Global abnormal service automatically locates the root cause service

[0084] If a service is known to behave abnormally, then check whether the services it depends on are also abnormal. If the dependent services are abnormal, continue searching until the abnormal service at the lowest level is found. If the dependent services are normal, then the service that failed this time is the service. This way, the root cause of the problem can be identified.

[0085] (3) Globally identify new dependencies

[0086] It can automatically list the dependencies of services and interfaces added daily. These are changes in the network. For these new dependencies, it may be that the system has abnormal behavior and has called logic that has not been reached before. It is necessary to manually check and confirm whether the system is working normally.

[0087] 2. Local service observability

[0088] Service governance can be performed manually. When a service is selected, the knowledge graph can make the following inferences. Based on the inferred knowledge, manual service stability governance and optimization can be performed.

[0089] (1) Dependent services

[0090] (2) All dependent interfaces

[0091] (3) Gateway interface affected by services

[0092] (4) Whether there is a self-loop in the service

[0093] (5) Impacted core interfaces

[0094] (6) Dependent interfaces added after a certain point in time

[0095] 3. Local interface observability

[0096] Interface management can be done manually. Select an interface and the knowledge graph can make the following inferences. Based on the inferred knowledge, manual interface stability management and optimization can be performed.

[0097] (1) Display link depth (maximum distance to the gateway)

[0098] (2) Affected core interfaces

[0099] (3) Core services affected

[0100] (4) Affected gateway interface

[0101] (5) Whether the interface has any self-loop.

[0102] The present invention provides a knowledge graph-based microservice system observability method to solve the governance problems of microservices and distributed systems. The method models services, interfaces, and dependencies by constructing a knowledge graph, thereby enabling knowledge inference and governance.

[0103] The advantage of this method is that it can build a knowledge graph based on the system call log without manual intervention, and use it to describe the observability of the microservice cluster, thereby completing the governance of microservices. This method can reduce the difficulty of microservice governance.

[0104] In summary, the microservice observability method constructed by knowledge graph has the advantages of automated construction, rapid location of microservice failures, and optimization of microservice network structure. It reduces the difficulty of microservice governance, enhances the timeliness and maintainability of service governance, and promotes the sustainable development of distributed software.

[0105] The above content is a detailed description of the present invention in combination with specific embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be regarded as falling within the scope of protection of the present invention.

Claims

1. A knowledge graph-based microservice system observability method, which has the following characteristics: The following steps are involved: S1. Collect online microservice call logs Collect historical microservice call logs. The specific data collected include: whether it is a gateway service, service name, interface name, called service name, called interface name, and first call time. Each service generally has multiple interfaces available for calling. For example, the login service has a text message sending interface, a text message verification interface, a login interface, a logout interface, etc. S2. Build a knowledge graph model The knowledge graph defines two types of entity nodes: service nodes and interface nodes. The attributes of service nodes include name: value service name, unique mark of service; service type: gateway type, service type; gateway is the entry network service initiated by network program, and service is the non-entry network service; creation time: this time is the first time the service appears in the collected log; the attributes of interface nodes include name: here we select service name.interface name as the name of the interface node. For example, the name of service name a and interface name 1 is expressed as a.

1. This encoding can confirm the unique interface by name, that is, each interface node represents a unique interface; Interface type: gateway interface, service interface; the gateway interface type indicates that this interface is a gateway service interface; the service interface indicates that this interface is a non-gateway service interface; Creation time: This is the time when the interface first appears in the collected logs; The knowledge graph defines three types of relationship nodes: the relationship between services (service dependency), the relationship between interfaces (interface dependency), and the relationship between services and interfaces (with dependency). All three types are unidirectional relationships. The attributes of the relationship between services (service dependency) include name: here we choose the calling service name_dependent service name as the relationship name. For example, service a depends on service b, and the name is a_b. Dependency start time: such as 2022-02-01 10:11:50, which means that service a depends on service b from this time on; the attributes of the relationship between interfaces (interface dependency) include name: here, the name of the calling interface node_the name of the dependent interface node is selected as the name of the relationship. For example, if a.1 depends on b.1, it is recorded as a.1_b.1; Dependency start time: such as 2022-02-01 10:11:50, which means that interface a.1 depends on interface b.1 from this time on; The properties of the relationship (with dependencies) between services and interfaces are: (1) Name: Here, the name of the interface* is selected as the name of the relationship. For example, if a depends on interface a.1, it is recorded as a.1*. (2) Interface start time: For example, 2022-02-01 10:11:50, which means that service a has added interface a.1 since this time; S3, graph database to build knowledge graph You can choose a graph knowledge database to build a knowledge graph, such as neo4j. The construction method is: First, build the entity node, create corresponding graph elements for the services and interfaces in the S1 log, and obtain the corresponding graph element numbers; Rebuild the relationship node again, and parse the log of S1 to find out the service dependency, interface dependency, and included dependency; such as falseloginService, login, passwordService, verifyPassword, 2022-02-0110:11:50, the service dependency is loginService depends on passwordService; the interface dependency is loginService.login depends on passwordService.verifyPassword; the dependency is loginService has loginService.login; the constructed graph example is shown in Figure 2; S4. Using knowledge graphs for observability 1. Global observability. (1) Maximum global network depth All gateway nodes can be regarded as starting nodes, and nodes without out-degree can be regarded as end nodes. The maximum path length from all gateway nodes to the end node can be calculated, and the maximum path lengths of all nodes are sorted. For the path lengths ranked at the top, the service and interface governance solution should be adopted. For users, the deepest call link has a high probability of failure. (2) Global abnormal service automatically locates the root cause service If a service is known to behave abnormally, then check whether the services it depends on are also abnormal. If the dependent services are abnormal, continue searching until the abnormal service at the lowest level is found. If the dependent services are normal, then the service that failed this time is the service. This way, the root cause of the problem can be identified. (3) Globally identify new dependencies It can automatically list the dependencies of services and interfaces added daily. These are changes in the network. For these new dependencies, it may be that the system has abnormal behavior and has called logic that has not been reached before. It is necessary to manually check and confirm whether the system is working normally.

2. Local service observability Service governance can be performed manually. When a service is selected, the knowledge graph can make the following inferences. Based on the inferred knowledge, manual service stability governance and optimization can be performed. (1) Dependent services (2) All dependent interfaces (3) Gateway interface affected by services (4) Whether there is a self-loop in the service (5) Impacted core interfaces (6) Dependent interfaces added after a certain point in time 3. Local interface observability Interface management can be done manually. Select an interface and the knowledge graph can make the following inferences. Based on the inferred knowledge, manual interface stability management and optimization can be performed. (1) Display link depth (maximum distance to the gateway) (2) Affected core interfaces (3) Core services affected (4) Affected gateway interface (5) Whether the interface has any self-loop.

Images