Multi-cloud service dependency analysis method and device and computer equipment

Abstract

The invention relates to a multi-cloud service dependency analysis method and device and computer equipment. The method comprises the steps of obtaining service discovery information and service calling data of a plurality of cloud environments, constructing a service dependency graph according to the service discovery information and the service calling data, determining the importance of each service based on the service dependency graph, screening out TopN target services based on the importance, and identifying dependency services and strong connectivity components of the TopN target services. By adopting the method, the service distribution and interaction conditions on different cloud platforms can be comprehensively known, the interaction mode and the dependency relationship among the services can be intuitively and clearly determined by constructing the service dependency graph, and the efficiency of dependency relationship analysis among multiple cloud services can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of cloud computing technology, and in particular to a multi-cloud service dependency analysis method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the popularity of cloud computing and the rise of multi-cloud strategies, enterprises are increasingly inclined to deploy their businesses on different cloud platforms to fully utilize the superior resources of each cloud service provider and improve business flexibility and disaster recovery capabilities. However, the complexity of multi-cloud environments also brings unprecedented challenges, especially in service dependency management.

[0003] Existing service dependency analysis and methods are mostly targeted at microservices and often rely on manual configuration and analysis, which makes it difficult to adapt to the rapidly changing microservice architecture and dynamic service instances, making dependency analysis inefficient. Summary of the invention

[0004] Based on this, it is necessary to provide an efficient multi-cloud service dependency analysis method, apparatus, computer equipment, computer-readable storage medium and computer program product to address the above technical problems.

[0005] In a first aspect, the present application provides a multi-cloud service dependency analysis method, comprising:

[0006] Obtain service discovery information and service call data for multiple cloud environments;

[0007] Build a service dependency graph based on service discovery information and service call data;

[0008] Based on the service dependency graph, determine the importance of each service, and select the TopN target services based on the importance;

[0009] Identify the dependent services and strongly connected components of the TopN target services.

[0010] In one embodiment, the method further comprises:

[0011] Integrate the service discovery information, service call data, dependent services and strongly connected components of the target service within a preset time period to obtain the load time series data of the target service;

[0012] Using the load time series data of each target service as input, the trained load prediction model is called to obtain the load prediction data of each service;

[0013] The load prediction model is obtained by training the long short-term memory network based on the historical load time series data of each service.

[0014] In one embodiment, the method further comprises:

[0015] Receive a service dependency result display message sent by a client, where the service dependency result display message carries a service identifier;

[0016] Obtaining a service dependency graph and load prediction data corresponding to the service identifier;

[0017] The service dependency graph is displayed in the form of an interactive graph through a preset visualization tool;

[0018] Call the preset chart tool to display the load forecast data in the form of a chart.

[0019] In one embodiment, building a service dependency graph based on service discovery information and service call data includes:

[0020] Based on the service discovery information and service call data, an edge set of service call relations is constructed;

[0021] Create an initial service dependency graph based on the preset graph database;

[0022] Based on the edge set, the nodes and edges of the initial service dependency graph are created to obtain the service dependency graph; each edge represents a service call, and the nodes connected at both ends of the edge are the calling service party and the called service party respectively.

[0023] In one embodiment, after constructing a service dependency graph based on the service discovery information and the service call data, the method further includes:

[0024] Monitor cluster events, log streams, and alarm data in different cloud environments to obtain service monitoring data;

[0025] Based on service monitoring data, dynamically adjust and update the service dependency graph.

[0026] In one embodiment, identifying dependent services and strongly connected components of TopN target services includes:

[0027] For each target service, use the shortest path algorithm to find the shortest dependency path from the target service to the remaining target services;

[0028] Based on the shortest dependency path of each service, get the dependent services of the TopN target services;

[0029] For each target service, traverse the graph through depth-first search, record the access order of each target service during the traversal process and the lowest index when backtracking;

[0030] Based on the visit order of each target service and the lowest index when backtracking, strongly connected components are identified.

[0031] In one embodiment, obtaining service discovery information of multiple cloud environments includes:

[0032] Trigger the service discovery mechanism, call the preset data acquisition interface, traverse the nodes and service objects under the specified namespace, and obtain the metadata of the nodes and service objects;

[0033] Query instance information of different cloud services;

[0034] Integrate the metadata of nodes and service objects, as well as instance information of different cloud services, to obtain service discovery information for multiple cloud environments.

[0035] In a second aspect, the present application also provides a multi-cloud service dependency analysis device. The device includes:

[0036] A data acquisition module, used to obtain service discovery information and service call data of multiple cloud environments;

[0037] A dependency graph building module is used to build a service dependency graph based on service discovery information and service call data;

[0038] The service screening module is used to determine the importance of each service based on the service dependency graph, and screen out the TopN target services based on the importance;

[0039] The data identification module is used to identify the dependent services and strongly connected components of the TopN target services.

[0040] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps in any one of the above-mentioned multi-cloud service dependency analysis method embodiments are implemented.

[0041] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in any one of the above-mentioned multi-cloud service dependency analysis method embodiments are implemented.

[0042] In a fifth aspect, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps in any one of the above-mentioned multi-cloud service dependency analysis method embodiments.

[0043] The above-mentioned multi-cloud service dependency analysis method, device, computer equipment, computer-readable storage medium and computer program product, by obtaining service discovery information and service call data in a multi-cloud environment, comprehensively understand the service distribution and interaction on different cloud platforms, and then build an accurate service dependency graph based on the service discovery information and service call data, which can intuitively clarify the interaction mode and dependency relationship between each service. Then, through the service dependency graph, the importance of each service in the system is quantified, and the TopN target services are selected based on the service importance, so that managers can focus on those services that have the greatest impact on the business and ensure the stable operation of key services. Finally, by identifying the dependent services and strongly connected components of the TopN target services, it is beneficial to optimize the dependency relationship between services, reduce the coupling degree between services, improve the flexibility and scalability of the system, and also help reduce large-scale service interruptions caused by single point failures. The above-mentioned scheme can effectively improve the efficiency of dependency analysis between multiple cloud services. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 An application environment diagram of a multi-cloud service dependency analysis method in one embodiment;

[0046] Figure 2 1 is a flowchart of a multi-cloud service dependency analysis method in one embodiment;

[0047] Figure 3 It is a flowchart of a multi-cloud service dependency analysis method in another embodiment;

[0048] Figure 4 A schematic diagram of a flow chart of a data display step in an embodiment;

[0049] Figure 5 A schematic diagram of a process for constructing a service dependency graph in one embodiment;

[0050] Figure 6 1 is a flowchart of a multi-cloud service dependency analysis method in yet another embodiment;

[0051] Figure 7 A flowchart of the steps of identifying dependent services and strongly connected components in one embodiment;

[0052] Figure 8is a structural block diagram of a multi-cloud service dependency analysis device in one embodiment;

[0053] Fig. 9 It is a structural block diagram of a multi-cloud service dependency analysis device in another embodiment;

[0054] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] The data lineage analysis method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 through a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers.

[0057] Specifically, an operator may send a data analysis instruction to the server 104 through the terminal 102. The server 104 connects to the target cluster by executing a preset script, and then obtains service discovery information and service call data of multiple cloud environments. A service dependency graph is constructed based on the service discovery information and service call data. Next, the importance of each service is determined based on the service dependency graph, and the TopN target services are screened out based on the importance. Finally, the dependent services and strongly connected components of the TopN target services are identified.

[0058] Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0059] In an exemplary embodiment, Figure 2 As shown in the figure, a multi-cloud service dependency analysis method is provided, which is applied to Figure 1The server 104 in FIG. 1 is used as an example to illustrate the method, which includes the following steps S200 to S800. Among them:

[0060] S200, obtaining service discovery information and service call data of multiple cloud environments.

[0061] The cloud environment can be understood as a cloud platform. Service discovery information refers to information used to locate and identify other service instances in a distributed system, usually including the service name, label, address (IP and port), health status, and version number. Service call data records detailed information about calls between services, including but not limited to call time, caller, callee, request parameters, response results, and time consumption.

[0062] In the specific implementation, the developer can develop a Python script to connect the server to the service registry (such as Consul, Eureka, Kubernetes Service Discovery, etc.) and the log system (such as ELK Stack, Prometheus, Grafana, etc.) in each cloud environment. Subsequently, through API calls or log collection tools, the latest service discovery information is regularly pulled from the service registry, and the latest service call data is pulled from the log system. After pulling the service discovery information and service call data, the obtained raw data can be parsed into a unified format, such as JSON or CSV format, and then the parsed data is stored in a database or data warehouse. It can be understood that the acquisition of service discovery information and service call data can be based on a preset script, and can be acquired in real time at a fixed time, or after acquiring the data at a fixed time, the data is stored in a database or data warehouse, and the stored service discovery information and service call data are obtained from the database or data warehouse after the analysis instruction sent by the response terminal.

[0063] S400, constructing a service dependency graph according to the service discovery information and the service call data.

[0064] The service dependency graph is a graphical representation method that shows the dependency relationship between services in the system. Each node represents a service, and the edge represents the call relationship between services.

[0065] After obtaining service discovery information and service call data, the server can remove invalid or erroneous data to ensure data accuracy and consistency. Subsequently, based on the service call data, the caller and callee of each service are extracted, the dependency relationship between services is established, and a service dependency graph is constructed using a graph database (such as Neo4j) or a graph algorithm library (such as NetworkX). Each service is a node, and each call is an edge.

[0066] S600, based on the service dependency graph, determine the importance of each service, and screen out TopN target services based on the importance.

[0067] Service importance is an indicator to measure the importance of a service in the system. The factors usually considered include call frequency, number of dependent services, service type, etc. The TopN target services are sorted according to service importance, and the top N services are selected as key management objects. In this embodiment, N can be 10. It can be understood that in other embodiments, N can also be 8, 12, or other values.

[0068] In specific implementation, the server can use graph algorithms (such as PageRank, BetweennessCentrality, etc.) based on the service dependency graph, taking into account the service's in-degree, out-degree, intermediate nodes and other features, to calculate the importance of each service.

[0069] Taking the use of the PageRank algorithm to quantify the relative importance or influence of each service as an example, services with high importance may be allocated more resource backups, or be restored first in the event of a failure, to ensure the stable operation and service quality of the overall system.

[0070] The specific formula is as follows:

[0071]

[0072] Among them, PR(u) represents the PageRank value of node u; d is the damping factor, usually 0.85; N is the total number of nodes in the graph; Bu is the set of all nodes pointing to node u, which can be understood as the dependency frequency of node u; W(v) is the weight of node v, and the request frequency, dependency, response time, etc. of the node (i.e., service) can be used as the weight; ∑w∈BvW(w) is the sum of the outbound link weights of node v, that is, the sum of the weights of all nodes that call node v.

[0073] The adjusted algorithm introduces the weight factor W(v), which allows the algorithm to adjust according to the importance or type of the service, so that the PageRank value better reflects the actual importance of the service, improving the accuracy and fairness of the algorithm.

[0074] After obtaining the importance of each service, the services can be sorted according to the calculated service importance to generate a service list sorted in descending order of importance. Then, according to business needs, the first 10 services are selected from the sorted service list as target services. Furthermore, the first N services can be selected from the sorted service list as target services.

[0075] S800: Identify dependent services and strongly connected components of the TopN target services.

[0076] Dependent services refer to other services that a service depends on, including directly dependent services and indirectly dependent services. Strongly connected components are an important concept in graph theory, which refers to a type of subgraph in a directed graph. Specifically, a strongly connected component refers to a largest subgraph in a directed graph, in which there is a mutually reachable path between any two vertices. If all vertices in a subgraph are strongly connected, then the subgraph is called a strongly connected component.

[0077] In specific implementation, the server can determine the direct dependent services and indirect dependent services of the TopN target services from the service dependency graph, and then use graph algorithms (such as Tarjan algorithm or Kosaraju algorithm) to detect the strongly connected components in the service dependency graph. Furthermore, the composition of each strongly connected component can be analyzed to identify key services and potential risk points. The identified dependent services and strongly connected component information are stored in a database or configuration file for subsequent monitoring and optimization.

[0078] In the above multi-cloud service dependency analysis method, by obtaining service discovery information and service call data in a multi-cloud environment, the service distribution and interaction on different cloud platforms can be fully understood, and then an accurate service dependency graph can be constructed based on the service discovery information and service call data, which can intuitively clarify the interaction mode and dependency relationship between each service. Then, through the service dependency graph, the importance of each service in the system is quantified, and then the TopN target services are selected based on the service importance, so that managers can focus on those services that have the greatest impact on the business and ensure the stable operation of key services. Finally, by identifying the dependent services and strongly connected components of the TopN target services, it is beneficial to optimize the dependency relationship between services, reduce the coupling between services, improve the flexibility and scalability of the system, and also help reduce large-scale service interruptions caused by single point failures. The above scheme can effectively improve the efficiency of dependency analysis between multiple cloud services.

[0079] In some embodiments, service discovery information of multiple cloud environments is obtained, including: triggering a service discovery mechanism, calling a preset data acquisition interface, traversing the nodes and service objects under a specified namespace, obtaining metadata of the nodes and service objects, querying instance information of different cloud services, integrating metadata of the nodes and service objects, and instance information of different cloud services, to obtain service discovery information of multiple cloud environments.

[0080] Service discovery is a mechanism for automatically detecting and locating services in a distributed system. A namespace is a logical container for organizing and isolating services, which can contain multiple nodes and service objects. A service object is a service defined in a namespace, which can be a microservice or a functional module. Metadata is detailed information describing nodes and service objects, such as name, ID, address, port, health status, etc.

[0081] In specific implementation, the server can execute Python scripts. Develop a Python script and use the Kubernetes Python client library (such as the kubernetes SDK) to connect to the target Kubernetes cluster. Call the API interfaces list_namespaced_pod and list_namespaced_service to traverse all Pods (nodes) and Service objects (service objects) under the specified namespace; for non-Kubernetes environments, you can query AWS EC2 instances, AzureVMs, or GCP Compute Engine instance information. Subsequently, parse the metadata of the Pod and Service, including the service name, label, IP address, port, etc., and store this information in the data structure in memory or directly write it into the database as service discovery information.

[0082] In addition, developers can also configure the Prometheus server, create or modify the Prometheus configuration file, and add monitoring targets for Kubernetes services, such as through ServiceMonitor or PodMonitor resources. Integrate the Jaeger client in the service code, insert tracking code at key locations according to the service call logic, and record the start, end, success or failure status, and duration of the service call. You can also use log collection tools such as Fluentd or Logstash, configure log parsing rules, and extract service call link information (i.e. service call data) from the service log. Furthermore, the server can forward the service call data to the central log storage system.

[0083] In this embodiment, the service discovery mechanism is automatically triggered to collect data, and the specified namespace is traversed to discover the metadata of nodes and service objects to obtain more comprehensive and detailed service discovery information. By monitoring targets and log collection tools, more comprehensive service call data can be collected.

[0084] like Figure 3 As shown, in some embodiments, after S800, the method further includes:

[0085] S900, integrates the service discovery information, service call data, dependent services and strongly connected components of the target service within a preset time period, obtains the load time series data of the target service, takes the load time series data of each target service as input, calls the trained load prediction model, and predicts the load prediction data of each target service.

[0086] In this embodiment, the load prediction model is obtained by training a long short-term memory network based on the historical load time series data of each service.

[0087] Load time series data refers to data integrated based on service discovery information, service call data, dependent services, and strongly connected components. In addition to the above data, it also includes service request frequency, dependency, response time and other data.

[0088] In practical applications, the training process of the load prediction model can be as follows: collect the historical load time series data of the top 10 target services in the historical time period, including service dependency analysis results, service request rate, response time, resource utilization and other information, and then preprocess the historical load time series data, including data cleaning, data standardization and other processing, and then determine the preprocessed historical load time series data as the training data set of the model. Using Python and TensorFlow or PyTorch framework, the input data of the model is the historical load time series data of the top 10 target services to train an LSTM (Long Short-Term Memory, long short-term memory network) model, and the output data is the load forecast data of each target service every day in the future, such as a week. Subsequently, through cross-validation, error analysis and other methods, adjust the model parameters until the prediction accuracy requirements are met to obtain the load prediction model. It can be understood that the load data can be predicted within the next 3 days, one month, or two months, depending on the actual situation, and is not limited here.

[0089] In specific implementation, after identifying the dependent services and strongly connected components of the target service, the service discovery information, service call data, dependent services and strongly connected components of the target service within a preset time period can be integrated to obtain the load time series data of the target service. Then, the load time series data of each target service is used as input to call the trained load prediction model to predict the load data of each target service in the next week.

[0090] In this embodiment, by training the load prediction model to predict the future load of the target service, computing resources can be planned and allocated in advance to ensure that there are sufficient resources to process requests during high-load periods, avoiding service performance degradation or downtime caused by insufficient resources. Measures can also be taken in time to prevent system overload and improve system stability and reliability.

[0091] like Figure 4 As shown, in some embodiments, after S800, the method further includes:

[0092] S940: Receive a service dependency result display message sent by the client, where the service dependency result display message carries a service identifier.

[0093] The service ID can be a unique name, ID, or other unique identifier of the service. When the user wants to view the service dependency analysis results intuitively, the client can send a service dependency result display message to the server. The server parses the received message and extracts the service ID. Verify the validity of the service ID to ensure that the request comes from a legitimate client. At the same time, record the request log, including request time, client IP, service ID, and other information for subsequent auditing and troubleshooting.

[0094] S942, obtaining a service dependency graph and load prediction data corresponding to the service identifier.

[0095] Subsequently, according to the service identifier, the corresponding service dependency graph and load prediction data are retrieved from the database or data warehouse. Furthermore, the retrieved data can be formatted into a format suitable for display, such as JSON, XML, etc. In order to improve the response speed, the commonly used service dependency graph and load prediction data can be cached in the memory or cache system (such as Redis).

[0096] S944, through the preset visualization tool, the service dependency graph is displayed in the form of an interactive graph.

[0097] The formatted service dependency graph data is transmitted to the front-end visualization tool, such as D3.js, ECharts, Graphviz, etc. The front-end visualization tool generates an interactive graph based on the received data. The interactive graph should support functions such as hovering the mouse to display detailed dependency information and clicking a node to expand more details.

[0098] S946, calling a preset chart tool to display the load forecast data in the form of a chart.

[0099] Next, the formatted load forecast data is transmitted to the front-end charting tool, such as Highcharts, Chart.js, Apache ECharts, etc. The front-end charting tool generates charts based on the received data, supports users to switch to view the forecasted load of each day according to the timeline, and provides abnormal warning function, and issues a warning when the forecast value exceeds the threshold. Among them, common chart types include line charts, bar charts, pie charts, etc. Furthermore, the server can also realize the dynamic update function of the chart, such as regularly refreshing data and supporting users to select different time ranges.

[0100] In this embodiment, through interactive diagrams and dynamic charts, users can more intuitively understand service dependencies and future load conditions, and perform operations such as viewing and scaling charts as needed.

[0101] like Figure 5 As shown, in some embodiments, S400 includes:

[0102] S420: Construct an edge set of service call relationships based on the service discovery information and the service call data.

[0103] An edge set is a set of edges that represent the relationship between nodes. Here, the edge set represents the call relationship between services. After the server obtains service discovery information and service call data, it can remove invalid or erroneous data, such as records of failed calls, duplicate records, etc. It traverses all call records, extracts the names of the caller and the callee, and then uses a tuple (caller, callee) to represent an edge. All extracted edges are stored in a list or set to form an edge set.

[0104] S440: Create an initial service dependency graph based on a preset graph database.

[0105] The initial service dependency graph is an empty graph that stores the dependencies between services.

[0106] Following the previous step, the server selects a suitable graph database, such as Neo4j, installs and configures the graph database, and ensures that it is running properly. Then, it connects to the graph database, creates a new graph instance, and defines the graph schema, including the types of nodes and edges. Next, it creates an empty service dependency graph in the graph database, ready for subsequent additions of nodes and edges.

[0107] S460, based on the edge set, create nodes and edges of the initial service dependency graph to obtain the service dependency graph; each edge represents a service call, and the nodes connected at both ends of the edge are the calling service party and the called service party respectively.

[0108] A node is the basic unit in a graph, representing a service. An edge is a line connecting two nodes, representing the call relationship between services.

[0109] After creating the initial service dependency graph, the server traverses all the edges in the edge set and extracts the service names of the caller and the server. For each service name, if there is no corresponding node in the graph, a new node is created and added to the initial service dependency graph. Then, according to each edge in the edge set, a corresponding edge is created and added to the graph. The starting point of each edge is the node of the caller service and the end point is the node of the callee service. Next, set properties for the nodes and edges. For example, the nodes can set properties such as the service name and service instance address, and the edges can set properties such as the number of calls and call time. Further optimize the structure of the graph as needed, such as merging duplicate edges and adjusting node layout. The created service dependency graph is persisted in the graph database for subsequent query and analysis.

[0110] In this embodiment, by constructing a service dependency graph, the calling relationship between services can be fully understood, helping managers to quickly locate problems and optimize resource allocation. In addition, the service dependency graph can intuitively display the dependency relationship between services, which helps to quickly identify the fault point and the scope of impact, and improve the efficiency of troubleshooting.

[0111] like Figure 6 As shown, in some embodiments, after S400, the method further includes:

[0112] S500, monitor cluster events, log streams and alarm data in different cloud environments, obtain service monitoring data, and dynamically adjust and update the service dependency graph based on the service monitoring data.

[0113] Cluster events refer to various events that occur in the cluster, including but not limited to node joining or leaving, service startup or shutdown, resource allocation changes, etc. Log streams refer to log data generated during service operation, including application logs, system logs, security logs, etc. Alarm data refers to various alarm information triggered in the system, such as performance alarms, security alarms, resource usage alarms, etc. Service monitoring data is data used to monitor the status and performance of services by integrating cluster events, log streams, and alarm data.

[0114] In specific implementation, after building the service dependency graph, the server can regularly pull the latest cluster events, log streams, and alarm data from the cluster management system, log collection system, and alarm system through API calls or log collection tools. Subsequently, it extracts key information from the pulled data, such as event type, log level, alarm type, timestamp, etc., and then automatically updates the service dependency graph based on real-time service monitoring data to reflect the current service status and dependencies.

[0115] Exemplarily, taking the updating of nodes and edges in the service dependency graph according to cluster events as an example, if a service stops running, remove the corresponding node; if a service starts, add the corresponding node. According to the log stream, identify new service call relationships and add or update the corresponding edges. According to the alarm data, mark the target service or edge with problems, such as marking it in red or adding comments. Furthermore, use a graph database (such as Neo4j) or a graph algorithm library (such as NetworkX) to update the service dependency graph. Persist the updated graph in the graph database for subsequent query and analysis.

[0116] In this embodiment, by real-time monitoring of cluster events, log streams and alarm data, and real-time updating of the service dependency graph, the current service status and dependency relationships can be reflected promptly and accurately.

[0117] like Figure 7 As shown, in some embodiments, S800 includes:

[0118] S820: For each target service, find out the shortest dependency path from the target service to the remaining target services by using the shortest path algorithm.

[0119] The shortest dependency path refers to the shortest path of dependency relationships between services. The dependency path indicates how one service depends on other services.

[0120] After the server builds the service dependency graph, the Dijkstra algorithm or Floyd-Warshall algorithm can be used to find the shortest dependency path from each target service to the remaining target services. Taking the Dijkstra algorithm as an example, the shortest dependency path can be found in the following way:

[0121] a. Initialization

[0122] Treat the target service as a node, set the shortest path estimate dist[v] of all nodes to infinity, except the starting node dist[A]=0; create a set of unvisited nodes.

[0123] b. Algorithm process

[0124] Select the node u with the smallest shortest path estimate from the set of unvisited nodes (initially A);

[0125] For each neighbor v of u:

[0126] Calculate the new path cost from u to v: new_cost = dist[u] + weight(u,v)·P(u,v);

[0127] If new_cost < dist[v], then update the shortest path estimate of v: dist[v] = new_cost;

[0128] Repeat the above steps until the set of unvisited nodes is empty.

[0129] Subsequently, the shortest path can be constructed by recording the predecessor node of each node. weight(u, v) represents the cost or time of a service call, and the performance evaluation metric P(u, v) represents the performance of the call, such as latency or reliability. By considering the performance evaluation metric P(u, v), the adjusted algorithm can more comprehensively evaluate the "cost" of a path, rather than just distance or weight. This enables the algorithm to adapt to different network environments and performance requirements, for example, making more reasonable path selections in terms of network latency or reliability.

[0130] S840, Based on the shortest dependency paths of each target service, obtain the dependent services of the top N target services.

[0131] After obtaining the shortest dependency paths of each target service, extract the dependent services of each target service from the shortest paths.

[0132] S860, For each target service, traverse the graph through depth - first search, and record the access order of each target service and the lowest index during backtracking.

[0133] Depth - first search is a graph traversal algorithm. Specifically, it starts from a node and searches as deep as possible along a path through recursion or a stack until it can no longer continue, and then backtracks.

[0134] During specific implementation, the access order and the lowest index can be initialized for each node. Subsequently, starting from each target service node, use DFS to traverse the graph. During the traversal process, record the access order of each node, and during the backtracking process, update the lowest index of each node.

[0135] S880, Based on the access order of each target service and the lowest index during backtracking, identify strongly connected components.

[0136] After the server obtains the access order of each target service and the lowest index during backtracking, use the Tarjan algorithm, combined with the access order and the lowest index, to identify the strongly connected components in the graph. Specifically, during the DFS traversal process, use a stack to record the nodes on the current path. When the access order of a node is equal to its lowest index, pop all the nodes from the stack, and these nodes form a strongly connected component. The specific implementation steps can be as follows:

[0137] a. Initialization

[0138] Use depth-first search to maintain disc (discovery time) and low (earliest discovery time that can be achieved) for each node. Use a stack to maintain the nodes on the current DFS path.

[0139] b. Algorithm process

[0140] If the node has not been visited, perform a DFS visit on it and update disc and low.

[0141] For each neighbor node, if the neighbor node is in the stack, update the low value.

[0142] If the low value is found to be equal to the disc value, the current node is the root of a strongly connected component, and all nodes in the stack are popped to form the SCC.

[0143] Update low value:

[0144] If the neighbor node is in the stack, low[u]=Min(low[u],disc[v]);

[0145] If the node has not been visited, low[u] = Min(low[u], low[v]).

[0146] In this embodiment, by calculating the shortest dependency path from each target service to other services, the dependent services of the target service can be determined. The DFS traversal can cover all nodes in the graph to ensure that there are no omissions. The Tarjan algorithm can accurately identify the strongly connected components in the graph, providing a reliable basis for subsequent dependency analysis.

[0147] In order to make a clearer description of the data lineage analysis method provided by the present application, a specific embodiment is described below, and the specific embodiment includes the following steps:

[0148] S1, obtains service discovery information and service call data of multiple cloud environments.

[0149] S2, based on the service discovery information and service call data, builds the edge set of the service call relationship, creates the initial service dependency graph according to the preset graph database, and creates the nodes and edges of the initial service dependency graph based on the edge set to obtain the service dependency graph.

[0150] Each edge represents a service call, and the nodes connected at both ends of the edge are the calling service party and the called service party respectively.

[0151] S3, based on the service dependency graph, determines the importance of each service, and selects the TopN target services based on the importance.

[0152] S4, for each target service, find out the shortest dependency path from the target service to the remaining target services through the shortest path algorithm, and obtain the dependency services of the TopN target services based on the shortest dependency paths of each target service.

[0153] S5, for each target service, traverse the graph through depth-first search, record the access order of each target service during the traversal and the lowest index when backtracking, and identify the strongly connected components based on the access order of each target service and the lowest index when backtracking.

[0154] S6, integrating the service discovery information, service call data, dependent services and strongly connected components of the target service within a preset time period to obtain the load time series data of the target service.

[0155] S7, taking the load time series data of each target service as input, calling the trained load prediction model, and predicting the load prediction data of each target service.

[0156] S6 monitors cluster events, log streams, and alarm data in different cloud environments to obtain service monitoring data. Based on the service monitoring data, the service dependency graph is dynamically adjusted and updated.

[0157] S6, receiving the service dependency result display message sent by the client, the service dependency result display message carries the service identifier, and displays the service dependency graph in the form of an interactive graph through a preset visualization tool, and calls a preset chart tool to display the load forecast data in the form of a chart.

[0158] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0159] Based on the same inventive concept, the embodiment of the present application also provides a data lineage analysis device for implementing the data lineage analysis method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more data lineage analysis device embodiments provided below can refer to the limitations of the data lineage analysis method above, and will not be repeated here.

[0160] In an exemplary embodiment, Figure 8 As shown, a multi-cloud service dependency analysis device 800 is provided, including: a data acquisition module 810, a dependency graph construction module 820, a service screening module 830 and a data identification module 840, wherein:

[0161] The data acquisition module 810 is used to acquire service discovery information and service call data of multiple cloud environments.

[0162] The dependency graph construction module 820 is used to construct a service dependency graph according to the service discovery information and the service call data.

[0163] The service screening module 830 is used to determine the importance of each service based on the service dependency graph, and screen out the TopN target services based on the importance.

[0164] The data identification module 840 is used to identify the dependent services and strongly connected components of the TopN target services.

[0165] like Fig. 9 As shown, in some embodiments, the device also includes a load prediction module 850, which is used to integrate the service discovery information, service call data, dependent services and strongly connected components of the target service within a preset time period to obtain the load time series data of the target service, and use the load time series data of each target service as input to call a trained load prediction model to obtain the load prediction data of each service, wherein the load prediction model is obtained by training a long short-term memory network based on the historical load time series data of each service.

[0166] like Fig. 9 As shown, in some embodiments, the device also includes a data display module 860, which is used to receive a service dependency result display message sent by the client, the service dependency result display message carries a service identifier, obtains a service dependency graph and load prediction data corresponding to the service identifier, and displays the service dependency graph in the form of an interactive graph through a preset visualization tool, and calls a preset chart tool to display the load prediction data in the form of a chart.

[0167] In some embodiments, the dependency graph construction module 820 is also used to construct an edge set of service call relationships based on service discovery information and service call data, create an initial service dependency graph according to a preset graph database, and create nodes and edges of the initial service dependency graph based on the edge set to obtain a service dependency graph; each edge represents a service call, and the nodes connected at both ends of the edge are the calling service party and the called service party respectively.

[0168] like Fig. 9As shown, in some embodiments, the device also includes a data update module 870 for monitoring cluster events, log streams and alarm data in different cloud environments, obtaining service monitoring data, and dynamically adjusting and updating the service dependency graph based on the service monitoring data.

[0169] In some embodiments, the data identification module 840 is also used to find out the shortest dependency path from the target service to the remaining target services through the shortest path algorithm for each target service, obtain the dependency services of the TopN target services based on the shortest dependency path of each service, and traverse the graph for each target service through depth-first search to record the access order of each target service during the traversal process and the lowest index when backtracking, and identify strongly connected components based on the access order of each target service and the lowest index when backtracking.

[0170] In some embodiments, the data acquisition module 810 is also used to trigger the service discovery mechanism, call a preset data acquisition interface, traverse the nodes and service objects under the specified namespace, obtain the metadata of the nodes and service objects, query the instance information of different cloud services, integrate the metadata of the nodes and service objects, and the instance information of different cloud services, and obtain service discovery information for multiple cloud environments.

[0171] Each module in the above-mentioned data lineage analysis device can be implemented in whole or in part by software, hardware and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.

[0172] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig.10 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as SQL query statements, stored process data and data lineage relationships. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data lineage analysis method is implemented.

[0173] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0174] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps in any one of the above-mentioned data lineage analysis method embodiments are implemented.

[0175] In one embodiment, when the processor executes the computer program, it also implements the steps in any one of the above data lineage analysis method embodiments.

[0176] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in any one of the above-mentioned data lineage analysis method embodiments are implemented.

[0177] In one embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, implements the steps in any one of the above multi-cloud service dependency analysis method embodiments.

[0178] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0179] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0180] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0181] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A multi-cloud service dependency analysis method, characterized in that: The method comprises: Obtain service discovery information and service call data for multiple cloud environments; Constructing a service dependency graph according to the service discovery information and the service call data; Based on the service dependency graph, determine the importance of each service, and select the TopN target services based on the importance; Identify dependent services and strongly connected components of the TopN target services.

2. The method according to claim 1, characterized in that The method further comprises: Integrate the service discovery information, service call data, dependent services and strongly connected components of the target service within a preset time period to obtain load time series data of the target service; Taking the load time series data of each target service as input, calling the trained load prediction model to obtain the load prediction data of each service; The load prediction model is obtained by training a long short-term memory network based on the historical load time series data of each service.

3. The method according to claim 2, characterized in that The method further comprises: Receiving a service dependency result display message sent by a client, wherein the service dependency result display message carries a service identifier; Obtaining a service dependency graph and load prediction data corresponding to the service identifier; The service dependency graph is displayed in the form of an interactive graph by using a preset visualization tool; A preset chart tool is called to display the load forecast data in the form of a chart.

4. The method according to any one of claims 1 to 3, characterized in that: The step of constructing a service dependency graph according to the service discovery information and the service call data includes: Constructing an edge set of service call relations based on the service discovery information and the service call data; Create an initial service dependency graph based on the preset graph database; Based on the edge set, nodes and edges of the initial service dependency graph are created to obtain a service dependency graph; each edge represents a service call, and the nodes connected at both ends of the edge are the calling service party and the called service party respectively.

5. The method according to claim 4, characterized in that After constructing the service dependency graph according to the service discovery information and the service call data, the method further includes: Monitor cluster events, log streams, and alarm data in different cloud environments to obtain service monitoring data; Based on the service monitoring data, the service dependency graph is dynamically adjusted and updated.

6. The method according to any one of claims 1 to 3, characterized in that: The identifying the dependent services and strongly connected components of the TopN target services includes: For each target service, find the shortest dependency path from the target service to the remaining target services using the shortest path algorithm; Based on the shortest dependency path of each service, get the dependent services of the TopN target services; For each target service, traverse the graph through depth-first search, record the access order of each target service during the traversal process and the lowest index when backtracking; Based on the visit order of each target service and the lowest index when backtracking, strongly connected components are identified.

7. The method according to any one of claims 1 to 3, characterized in that: Get service discovery information for multiple cloud environments, including: Trigger the service discovery mechanism, call the preset data acquisition interface, traverse the nodes and service objects under the specified namespace, and obtain the metadata of the nodes and service objects; Query instance information of different cloud services; The metadata of the node and the service object and the instance information of the different cloud services are integrated to obtain service discovery information of multiple cloud environments.

8. A multi-cloud service dependency analysis device, characterized in that: The device comprises: A data acquisition module, used to obtain service discovery information and service call data of multiple cloud environments; A dependency graph construction module, used to construct a service dependency graph according to the service discovery information and the service call data; A service screening module, used to determine the importance of each service based on the service dependency graph, and screen out the TopN target services based on the importance; The data identification module is used to identify dependent services and strongly connected components of the TopN target services.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images