Fault locating method, apparatus, device, and storage medium

Abstract

Embodiments of the present application provide a fault positioning method, device, equipment and storage medium, the method is applied to an electronic device in a Kubernetes cluster, business alarm information in the Kubernetes cluster is obtained; according to the business alarm information, Access Log log information in the Kubernetes cluster and Pod information in the Kubernetes cluster are determined, according to the Access Log log information and the Pod information in the Kubernetes cluster, non-200 response code nine-tuple requests in the Kubernetes cluster are determined, the non-200 response code nine-tuple requests are taken as abnormal requests, and the technical means of determining the fault information corresponding to the abnormal requests are achieved, and the technical effect of improving fault analysis efficiency is achieved.

Description

Technical Field

[0001] This application relates to the field of intelligent operation and maintenance technology, and in particular to a fault location method, device, equipment, and storage medium. Background Technology

[0002] In recent years, with the continuous advancement of the comprehensive cloud migration initiative, various applications are increasingly shifting from on-premises server deployments to Kubernetes clusters in the cloud. Along with the hybrid deployment of cluster services and the continuous increase in business volume, intelligent operations and maintenance (O&M) methods, which monitor and analyze cluster status, application status, and business requests, can significantly reduce the burden on O&M personnel and generate widespread benefits.

[0003] When encountering a decline in business access success rate, operations and maintenance personnel still need to manually check various data such as cluster monitoring indicators, and combine their own experience to analyze and locate the fault.

[0004] However, the above process is cumbersome and time-consuming, resulting in low efficiency in fault analysis. Summary of the Invention

[0005] This application provides a fault location method, apparatus, device, and storage medium to achieve the technical effect of improving fault analysis efficiency.

[0006] In a first aspect, embodiments of this application provide a fault location method applied to electronic devices in a Kubernetes cluster, the method comprising:

[0007] Obtain service alarm information from the Kubernetes cluster, the service alarm information including: time information and cluster name;

[0008] Based on the business alarm information, determine the Access Log information in the Kubernetes cluster and the Pod information in the Kubernetes cluster;

[0009] Based on the Access Log information and the Pod information in the Kubernetes cluster, determine the nine-tuple requests in the Kubernetes cluster that do not have a 200 response code;

[0010] The non-200 response code nine-tuple request is treated as an abnormal request, and the fault information corresponding to the abnormal request is determined.

[0011] In one possible implementation, determining the nine-tuple request with a non-200 response code in the Kubernetes cluster based on the Access Log information and the Pod information in the Kubernetes cluster includes:

[0012] Determine the seven-tuple of the Access Log information, which includes: the time the request occurred, the Pod Ip of the request to reach the load balancer, the Pod of the load balancer to which the request reached, the status code of the request, the request path, the Pod Ip of the backend service, and the total time of the request.

[0013] Based on the seven-tuple and the Pod information, determine the nine-tuple request sequence of the Kubernetes cluster;

[0014] Identify the non-200 response codes in the nine-tuple request sequence.

[0015] In one possible implementation, the nine-tuple request includes: the request sent by the client to the cluster, the time when the request occurred, the status code of the request, the Pod to which the request reached, the Internet Protocol IP address of the load balancer Pod to which the request reached, the Pod name of the backend service, the namespace to which the Pod of the backend service belongs, the IP address of the business node where the Pod of the backend service is located, and the total time of the entire request.

[0016] In one possible implementation, determining the Access Log information and Pod information in the Kubernetes cluster based on the service alarm information includes:

[0017] Based on the aforementioned business alarm information, generate Elastic Search query statements and Prometheus query statements;

[0018] Based on the Elastic Search query statement, query the Access Log information in the Kubernetes cluster;

[0019] Based on the Prometheus query, extract Pod information from the start time to the end time in the time information of the Kubernetes cluster.

[0020] In one possible implementation, an Elastic Search query is generated based on the business alarm information, including:

[0021] Based on the preset log query template, the time information, and the cluster name, the ElasticSearch query statement is generated. The log query template includes: start time, end time, cluster name, AND condition, OR condition, number of queries per query, and time sorting.

[0022] In one possible implementation, a Prometheus query statement is generated based on the business alarm information, including:

[0023] Based on the preset indicator query template, the time information, and the cluster name, the Prometheus query statement is generated. The indicator query template includes: start time, end time, and cluster name.

[0024] In one possible implementation, the Access Log information includes log information extracted from Elastic Search from the start time to the end time in the time information.

[0025] Secondly, embodiments of this application provide a fault location device applied to electronic devices in a Kubernetes cluster, the device comprising:

[0026] The acquisition module is used to acquire business alarm information in the Kubernetes cluster, including: time information and cluster name;

[0027] The first determining module is used to determine the Access Log information in the Kubernetes cluster and the Pod information in the Kubernetes cluster based on the business alarm information.

[0028] The second determining module is used to determine the nine-tuple request with a non-200 response code in the Kubernetes cluster based on the Access Log information and the Pod information in the Kubernetes cluster.

[0029] The third determining module is used to treat the non-200 response code nine-tuple request as an abnormal request and determine the fault information corresponding to the abnormal request.

[0030] In one possible implementation, the second determining module is specifically used for:

[0031] Determine the seven-tuple of the Access Log information, which includes: the time the request occurred, the Pod Ip of the request to reach the load balancer, the Pod of the load balancer to which the request reached, the status code of the request, the request path, the Pod Ip of the backend service, and the total time of the request.

[0032] Based on the seven-tuple and the Pod information, determine the nine-tuple request sequence of the Kubernetes cluster;

[0033] Identify the non-200 response codes in the nine-tuple request sequence.

[0034] In one possible implementation, the nine-tuple request includes: the request sent by the client to the cluster, the time when the request occurred, the status code of the request, the Pod to which the request reached, the Internet Protocol IP address of the load balancer Pod to which the request reached, the Pod name of the backend service, the namespace to which the Pod of the backend service belongs, the IP address of the business node where the Pod of the backend service is located, and the total time of the entire request.

[0035] In one possible implementation, the first determining module is specifically used for:

[0036] Based on the aforementioned business alarm information, generate Elastic Search query statements and Prometheus query statements;

[0037] Based on the Elastic Search query statement, query the Access Log information in the Kubernetes cluster;

[0038] Based on the Prometheus query, extract Pod information from the start time to the end time in the time information of the Kubernetes cluster.

[0039] In one possible implementation, the first determining module generates an Elastic Search query statement based on the business alarm information, specifically for:

[0040] Based on the preset log query template, the time information, and the cluster name, the ElasticSearch query statement is generated. The log query template includes: start time, end time, cluster name, AND condition, OR condition, number of queries per query, and time sorting.

[0041] In one possible implementation, the first determining module generates a Prometheus query statement based on the service alarm information, specifically for:

[0042] Based on the preset indicator query template, the time information, and the cluster name, the Prometheus query statement is generated. The indicator query template includes: start time, end time, and cluster name.

[0043] In one possible implementation, the Access Log information includes log information extracted from Elastic Search from the start time to the end time in the time information.

[0044] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0045] The memory stores computer-executed instructions;

[0046] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0047] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0048] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0049] The fault location method, apparatus, device, and storage medium provided in this application embodiment are applied to electronic devices in a Kubernetes cluster. The method includes: obtaining service alarm information in the Kubernetes cluster, the service alarm information including: time information and cluster name; determining Access Log information and Pod information in the Kubernetes cluster based on the service alarm information; determining nine-tuple requests with non-200 response codes in the Kubernetes cluster based on the Access Log information and Pod information in the Kubernetes cluster; treating nine-tuple requests with non-200 response codes as abnormal requests; and determining the fault information corresponding to the abnormal requests. These technical means achieve the technical effect of improving fault analysis efficiency. Attached Figure Description

[0050] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0051] Figure 1 Flowchart of the fault location method provided in this application Figure 1 ;

[0052] Figure 2 Flowchart of the fault location method provided in this application Figure 2 ;

[0053] Figure 3 Flowchart of the fault location method provided in this application Figure 3 ;

[0054] Figure 4 Flowchart of the fault location method provided in this application Figure 4 ;

[0055] Figure 5 A schematic diagram of the fault location device provided in this application;

[0056] Figure 6 A schematic diagram of the structure of the electronic device provided in this application.

[0057] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0058] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0059] First, the terms used in this application will be explained:

[0060] 1) Kubernetes:

[0061] Kubernetes (K8S) is an open-source container orchestration platform. It can orchestrate, schedule, and manage containers (Pods) packaged for business applications. A Kubernetes cluster consists of two parts: the control plane (management nodes) and the data plane (business nodes). A Pod is the unit of scheduling in Kubernetes, which is a package of one or more containers.

[0062] 2) Management Node:

[0063] The master node, or main node, is the node in a Kubernetes cluster responsible for managing and scheduling the entire cluster. It consists of multiple machines, with one master and multiple backups. Core Kubernetes components are deployed on the management node, such as the Ingress Controller, API Server, Controller Manager, Scheduler, and the distributed key-value store ETCD. It is responsible for controlling cluster communication, scheduling and orchestrating cluster resources, and storing cluster data and status information.

[0064] 3) Business Nodes:

[0065] In short, a Node is a node in a Kubernetes cluster responsible for running business applications. Besides the business Pod, it also includes components such as Kubelet and Kube-proxy, which are responsible for tasks such as creating, starting, stopping, communicating with, and load balancing the containers corresponding to the Pod.

[0066] 4), Nginx Ingress Controller:

[0067] Ingress, or Layer 7 load balancer, is deployed as a Pod on the management node of the Kubernetes cluster. Client requests first reach the Ingress Controller load balancer in the cluster, which then proxies the request to the Pod of the backend service and outputs corresponding logs to the access log based on the request.

[0068] 5) Access Log:

[0069] This refers to the log file of the Nginx Ingress Controller. When the Nginx Ingress Controller forwards requests, it records the requests in this file for operations personnel to analyze client request behavior. For each request, an Access Log is generated. The format of the Access Log is: <$remote_addr-$remote_user$time_local"$request"$status$body_bytes_sent"$http_referer"$http_user_agent"$request_length$request_time$proxy_upstream_name$upstream_addr>;

[0070] In this context, `$remote_addr` refers to the client address; `$remote_user` refers to the client username; `$time_local` refers to the access time; `$request` refers to the request URI; `$status` refers to the request status code; `$body_bytes_sent` refers to the size of the file content sent to the client; `$http_referer` refers to the URL referrer; `$http_user_agent` refers to information such as the user's terminal browser; `$request_length` refers to the request length; `$request_time` refers to the total request time; `$proxy_upstream_name` refers to the backend service to which the request is forwarded; and `$upstream_addr` refers to the backend address to which the request is forwarded.

[0071] 6) Elastic Search:

[0072] Elastic Search is an open-source, highly scalable, distributed full-text search engine that provides an API for fast full-text search. By connecting the logs of the Kubernetes cluster to Elastic Search, operations personnel can use the application programming interface (API) to quickly search and analyze logs.

[0073] 7), Prometheus:

[0074] This refers to an open-source service monitoring system and time-based database that provides a general data model and convenient interfaces for data collection, storage, and querying. By writing PromQL query statements, it is possible to query and aggregate monitoring metric data.

[0075] 8) Nine-tuple request:

[0076] This refers to a nine-tuple containing complete request information, whose expression is requestInfo: <OccurTime,Status,IngressPod,IngressIp,UpstreamPod,UpstreamPodNs,URI,UpstreamNodeIp,RequesTime> ;

[0077] In this context, requestInfo represents the request sent by the client to the cluster; OccurTime represents the time when the request occurred; Status represents the status code of the request; IngressPod represents the Nginx IngressController Pod to which the request reached, and the request is forwarded from this load balancer Pod to the backend service Pod; IngressIp represents the IP of the Nginx Ingress Controller Pod to which the request reached; UpstreamPod represents the name of the backend service Pod; UpstreamPodNs represents the namespace to which the backend service Pod belongs; UpstreamNodeIp represents the IP of the business node where the backend service Pod is located; URI represents the request path; and RequestTime represents the total time of the entire request.

[0078] Secondly, the technical background involved in this application will be explained:

[0079] In recent years, with the continuous advancement of the comprehensive cloud migration initiative, various applications are increasingly shifting from on-premises server deployments to Kubernetes clusters in the cloud. Along with the hybrid deployment of cluster services and the continuous increase in business volume, intelligent operations and maintenance (O&M) methods, which monitor and analyze cluster status, application status, and business requests, can significantly reduce the burden on O&M personnel and generate widespread benefits.

[0080] When encountering a decline in business access success rate, operations and maintenance personnel still need to manually check various data such as cluster monitoring indicators, and combine their own experience to analyze and locate the fault.

[0081] However, the above process is cumbersome and time-consuming, resulting in low efficiency in fault analysis.

[0082] This application provides a fault location method, specifically involving a method that generates query conditions based on templates, automatically extracts Kubernetes cluster component logs reported to Elastic Search and automatically extracts Prometheus metric information based on the query conditions, and uses the captured logs and metric information to quickly extract abnormal logs and generate the nine-tuple requests involved in the abnormality in a rule-based manner, thereby accurately achieving rapid fault location and facilitating further analysis by operations and maintenance personnel.

[0083] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0084] Figure 1 Flowchart of the fault location method provided in this application Figure 1 ,like Figure 1 As shown, this method is applied to electronic devices in a Kubernetes cluster, including:

[0085] The electronic device can be an analysis device for fault location.

[0086] Step 11: Obtain business alarm information from the Kubernetes cluster;

[0087] The business alarm information includes: time information and cluster name;

[0088] In this step, when a service alarm occurs in the Kubernetes cluster, the electronic devices in the Kubernetes cluster can obtain the service alarm and extract basic information such as time information and cluster name from the service alarm information.

[0089] In one possible implementation, the expression for the business alarm information `basicInfo` can be...<alertInfo.t1,alertInfo.t2,alertInfo.cluster> .

[0090] Among them, alertInfo is the business alarm, t1 is the start time in the time information, t2 is the end time in the time information, and cluster is the cluster name.

[0091] Step 12: Based on the business alarm information, determine the Access Log information and Pod information in the Kubernetes cluster;

[0092] In this step, a Pod in Kubernetes is the unit for running business applications. Based on the time information and cluster name in the business alarm information, you can view the information of the Pods in the cluster at the relevant time.

[0093] Furthermore, in a Kubernetes cluster, Access Log information is typically recorded by Ingress Nginx or a service proxy. Based on business alert information, you can view the logs of the relevant Ingress or proxy, analyze the request status code, response time, etc., to help further confirm the problem.

[0094] Optionally, the Pod information in the Kubernetes cluster is the information of all Pods from the start time to the end time corresponding to the above time information.

[0095] Step 13: Based on the Access Log information and the Pod information in the Kubernetes cluster, identify the nine-tuple requests in the Kubernetes cluster that do not have a 200 response code.

[0096] In this step, based on the Access Log information, the seven-tuple of the Access Log information can be determined first. Then, based on the seven-tuple and the Pod information in the Kubernetes cluster, the nine-tuple request sequence of the Kubernetes cluster is determined. This nine-tuple request sequence contains multiple nine-tuple requests.

[0097] Then, extract the non-200 response code nine-tuple requests from the nine-tuple request sequence.

[0098] Optionally, a nine-tuple request with a non-200 response code usually represents an error or abnormal status and serves as a fault indication signal, requiring close attention in a production environment; this is mainly because it usually indicates that a problem occurred during the request processing, resulting in a failure to return a successful response.

[0099] In addition, the nine-tuple request includes: the request sent by the client to the cluster, the time when the request occurred, the status code of the request, the Pod to which the request reached, the Internet Protocol IP address of the load balancer Pod to which the request reached, the Pod name of the backend service, the namespace to which the Pod of the backend service belongs, the IP address of the business node where the Pod of the backend service is located, and the total time of the entire request.

[0100] By combining Access Log information with Pod information in the Kubernetes cluster, nine-tuple requests with non-200 response codes can be effectively identified and analyzed, thereby locating and resolving issues in the cluster.

[0101] Step 14: Treat non-200 response code nine-tuple requests as abnormal requests and determine the fault information corresponding to the abnormal requests.

[0102] In this step, requests with non-200 response codes and nine-tuples are treated as abnormal requests, and these abnormal requests are parsed to determine the corresponding fault information.

[0103] For example, the following example illustrates a nine-tuple request with a non-200 response code (such as 4xx or 5xx error codes):

[0104] 1. Client error (4xx response code):

[0105] 1) 400 Bad Request: The request format is incorrect, usually because the request data sent by the client is invalid or missing necessary parameters.

[0106] 2) 401 Unauthorized: The request did not provide valid authentication credentials.

[0107] 3) 403 Forbidden: The requested resource is forbidden to access, even if the correct authentication credentials are provided.

[0108] 4) 404 Not Found: The requested resource does not exist, possibly due to an incorrect URL.

[0109] 5) 408 Request Timeout: The request timed out, and the client waited for the server response for longer than expected.

[0110] 2. Server error (5xx response code):

[0111] 1) 500 Internal Server Error: The server encountered an unexpected error that prevented it from completing the request. This is usually due to a server bug or resource limitation issue.

[0112] 2) 502 Bad Gateway: This usually indicates that the proxy or gateway server cannot connect to the backend server.

[0113] 3) 503 Service Unavailable: The service is unavailable, possibly due to server overload, resource exhaustion, or the service being temporarily unavailable.

[0114] 4) 504 Gateway Timeout: The gateway or proxy server timed out while waiting for a response from the backend server.

[0115] The fault location method provided in this application embodiment is applied to electronic devices in a Kubernetes cluster. The method includes: obtaining service alarm information in the Kubernetes cluster, the service alarm information including: time information and cluster name; determining Access Log information and Pod information in the Kubernetes cluster based on the service alarm information; determining nine-tuple requests with non-200 response codes in the Kubernetes cluster based on the Access Log information and Pod information in the Kubernetes cluster; treating nine-tuple requests with non-200 response codes as abnormal requests; and determining the fault information corresponding to the abnormal requests. This technical means achieves the technical effect of improving fault analysis efficiency.

[0116] Based on the above embodiments, Figure 2 Flowchart of the fault location method provided in this application Figure 2 ,like Figure 2 As shown, step 12 above may include the following implementation:

[0117] Step 21: Generate Elastic Search and Prometheus query statements based on the business alarm information;

[0118] Before this step, you can pre-generate log query templates (e.g., Elastic Search log query templates) and metric query templates (Prometheus metric query templates).

[0119] Among them, 1) the Elastic Search log query template is used to generate specific Elastic Search query statements; 2) the Prometheus metric query template is used to generate Prometheus query statements.

[0120] Optionally, in step 21, generating an Elastic Search query statement based on business alarm information can be implemented as follows: generating an Elastic Search query statement based on a preset log query template, time information, and cluster name;

[0121] The log query template includes: start time, end time, cluster name, AND condition, OR condition, number of queries per query, and time sorting.

[0122] In this implementation, when a business alarm occurs in the Kubernetes cluster, based on the time information, cluster name, and other conditions in the business alarm information, an Elastic Search query statement `queryes` is generated using the log query condition template `template es`. The formula is as follows:

[0123] Query es =template es (t1, t2, cluster, list(cond AND ), list(cond OR ));

[0124] Among them, cond AND The set of conditions for the query; cond OR For the query or set of conditions.

[0125] Optionally, in step 21, generating a Prometheus query statement based on the business alarm information can be implemented as follows: generating a Prometheus query statement based on a preset metric query template, time information, and cluster name.

[0126] The metric query template includes: start time, end time, and cluster name.

[0127] In this implementation, when a business alarm occurs in the Kubernetes cluster, based on the time information, cluster name, and other conditions in the business alarm information, a Prometheus query statement `queryprom` is generated using the metric query template `template prom`. The formula is as follows:

[0128] query prom=template prom(t1, t2, cluster).

[0129] Step 22: Query the Access Log information in the Kubernetes cluster using the Elastic Search query statement;

[0130] In this step, use the Elastic Search query to retrieve the Access Log information of the cluster's Nginx IngressController component (i.e., the load balancer).

[0131] Optionally, the Access Log information includes: log information extracted from ElasticSearch from the start time to the end time in the time information.

[0132] In one possible implementation, the log information extracted by Elastic Search from the start time to the end time is denoted as `logs`, as shown in the following formula:

[0133] Logs=[Log1, Log2, Log3,...Logn];

[0134] For example, the formula for logi can be:

[0135] Logi =<OccurTimei,IngressIpi,IngressPodi,accesslogi> .

[0136] Specifically, logi represents the log information tuple of the i-th request; OccurTimei represents the time when the log information tuple occurred; IngressIpi represents the IP of the Nginx Ingress Controller Pod to which the i-th request arrived; IngressPodi represents the name of the Nginx Ingress Controller Pod to which the i-th request arrived; and accesslogi represents the Access Log corresponding to the i-th request.

[0137] Step 23: Extract Pod information from the Kubernetes cluster, from the start time to the end time, based on the Prometheus query.

[0138] In this step, Prometheus queries are used to extract information about all Pods in the cluster from the start time to the end time.

[0139] In one possible implementation, the Pod information extracted from the Kubernetes cluster from the start time to the end time is called Pods, as shown in the following formula:

[0140] Pods=[Pod1, Pod2, Pod3,...Podn];

[0141] Taking Podi as an example, its formula can be:

[0142] Podi =<Pod_ipi,nsi,Pod_namei,host_ipi> .

[0143] Specifically, Podi represents the tuple of metrics information for the i-th Pod; Pod_ipi represents the IP address of the i-th Pod; nsi represents the namespace to which the i-th Pod belongs; Pod_namei represents the name of the i-th Pod; and host_ipi represents the IP address of the node where the i-th Pod resides.

[0144] The fault location method provided in this application generates ElasticSearch and Prometheus query statements based on business alarm information. The ElasticSearch query retrieves Access Log information from the Kubernetes cluster, while the Prometheus query extracts Pod information from the Kubernetes cluster, showing the start and end times from the time information. This method can generate query conditions based on templates to extract cluster log and metric information.

[0145] Based on the above embodiments, Figure 3 Flowchart of the fault location method provided in this application Figure 3 ,like Figure 3 As shown, step 13 above may include the following implementation:

[0146] Step 31: Determine the seven-tuple of Access Log information;

[0147] The seven-tuple includes: the request status code (Status), the request path (URI), the Pod Ip of the backend service (UpstreamNodeIp), the total request time (RequesTime), the time the request occurred (OccurTime), the Pod Ip of the load balancer (e.g., Nginx Ingress Controller) to which the request arrived (IngressIp), the Pod of the Nginx Ingress Controller to which the request arrived (IngressPod), and the request being forwarded from that load balancer Pod to the Pod of the backend service.

[0148] In this step, a rule-based approach is used to further analyze the Access Log information, extracting the request status code, request path, backend service Pod IP, and total request time from the Access Log information to obtain a seven-tuple of log information.

[0149] The formula for the seven-tuple of log information is as follows:

[0150] Logs_extracted=[lge1, lge 2, lge 3,...lge n];

[0151] Taking lge i as an example, its formula can be:

[0152] lge i=<OccurTimei,IngressIpi,IngressPodi,Statusi,URIi,UpsteamPodIpi,RequestTimei> .

[0153] Specifically, lge i represents the seven-tuple of log information for the i-th request; Statusi represents the status code of the i-th request; URIi represents the request path of the i-th request; UpstreamPodIpi represents the backend service Pod Ip to which the i-th request is forwarded by the NginxIngress Controller; and RequestTimei represents the total time consumed by the i-th request.

[0154] Step 32: Determine the nine-tuple request sequence for the Kubernetes cluster based on the seven-tuple and Pod information;

[0155] In this step, all the seven-tuples and Pod information obtained above are matched and concatenated to generate the cluster's nine-tuple request Info.

[0156] Optionally, the formula for the nine-tuple request sequence requestsInfos is as follows:

[0157] requestsInfos=[requestsInfo1, requestsInfo2,...requestsInfon].

[0158] Specifically, for any nine-tuple request:

[0159] requestsInfo = <OccurTime,Status,IngressPod,IngressIp,UpstreamPod,UpstreamPodNs,URI,UpstreamNodeIp,RequesTime> .

[0160] Step 33: Identify the non-200 response codes in the nine-tuple request sequence.

[0161] In this step, the nine-tuple request sequence is filtered according to the response code of each request, and the nine-tuple requests that do not have a 200 response code are extracted, which are the requests involved in the exception.

[0162] The formula is as follows:

[0163] AbnormalRequestInfo is equal to requestInfo (requestInfo.Start is not equal to 200).

[0164] The fault location method provided in this application determines the seven-tuple of Access Log information, which includes: the request status code, request path, Pod IP of the backend service, and total request time. Based on the seven-tuple and Pod information, it determines the nine-tuple request sequence of the Kubernetes cluster; and identifies the nine-tuple requests in the nine-tuple request sequence that do not have a 200 response code. This method can quickly extract abnormal logs and the nine-tuple requests involved in the abnormality, thereby assisting operations and maintenance personnel in further analysis and completing rapid fault location.

[0165] Based on the above embodiments, Figure 4 Flowchart of the fault location method provided in this application Figure 4 ,like Figure 4 As shown, one overall embodiment of this fault location method can be:

[0166] Step 1, Begin;

[0167] Step 2: Extracting business alarm information, which is performed in steps 3 and 7 respectively;

[0168] In this embodiment, when a service alarm occurs, basic information such as start time, end time, and cluster name are extracted from the service alarm information.

[0169] Step 3: Generate a log query template;

[0170] In this embodiment, an Elastic Search query template is generated, which mainly includes start time, end time, cluster name, AND condition, OR condition, number of queries per query, and time sorting.

[0171] Step 4: Generate Elastic Search query statements;

[0172] In this embodiment, based on the time, cluster name, and other conditions in the alarm information, an Elastic Search query statement is generated using a log query condition template.

[0173] Step 5: Query Elastic Search log information;

[0174] In this embodiment, an Elastic Search query is used to query the Access Log information of the Nginx IngressController component in the cluster;

[0175] Step 6: Extract the seven-tuple from the log information and proceed to step 10;

[0176] In this embodiment, a rule-based approach is used to further mine the log information in step 3, extracting the request status code, request path, backend service Pod IP, and total request time from the Access Log to obtain a seven-tuple of log information.

[0177] Step 7: Generate indicator query template;

[0178] In this embodiment, a Prometheus query template is generated, which mainly includes start time, end time, and cluster name.

[0179] Step 8: Generate the Prometheus query statement;

[0180] In this embodiment, a Prometheus query statement is generated based on the time, cluster name, and other conditions in the alarm information and the indicator query template.

[0181] Step 9: Extract Pod information;

[0182] In this embodiment, a Prometheus query is used to extract all Pod information from the start time to the end time of the cluster;

[0183] Step 10: Match and generate nine-tuple requests;

[0184] In this embodiment, all Pod information obtained in steps G and H is matched and concatenated with the log information seven-tuple to generate a cluster nine-tuple request.

[0185] Step 11: Extract abnormal nine-tuple requests;

[0186] In this embodiment, for the nine-tuple request sequence, the response code of each request is filtered out, and the nine-tuple requests with non-200 response codes are extracted to obtain the abnormal requests involved.

[0187] Step 12, End.

[0188] The fault location method involved in this application can generate query conditions based on templates, extract cluster log information and indicator information, and use a rule-based approach to quickly extract abnormal logs and the nine-tuple requests involved in the abnormality, thereby assisting operation and maintenance personnel in further analysis and completing rapid fault location.

[0189] Figure 5 A schematic diagram of the fault location device provided in this application is shown below. Figure 5 As shown, this fault location device is applied to electronic devices in a Kubernetes cluster and includes:

[0190] The acquisition module 51 is used to acquire business alarm information in the Kubernetes cluster. The business alarm information includes: time information and cluster name.

[0191] The first determination module 52 is used to determine the Access Log information and Pod information in the Kubernetes cluster based on the business alarm information.

[0192] The second determining module 53 is used to determine the nine-tuple request in the Kubernetes cluster that is not a 200 response code based on the Access Log information and the Pod information in the Kubernetes cluster.

[0193] The third determination module 54 is used to treat non-200 response code nine-tuple requests as abnormal requests and determine the fault information corresponding to the abnormal requests.

[0194] In one possible implementation, the second determining module 53 is specifically used for:

[0195] Determine the seven-tuple of Access Log information. The seven-tuple includes: the time when the request occurred, the Pod Ip of the request to reach the load balancer, the Pod of the load balancer to which the request reached, the status code of the request, the request path, the Pod Ip of the backend service, and the total time of the request.

[0196] Based on the seven-tuple and Pod information, determine the nine-tuple request sequence of the Kubernetes cluster;

[0197] Identify 9-tuple requests in the 9-tuple request sequence that do not have a 200 response code.

[0198] In one possible implementation, the nine-tuple request includes: the request sent by the client to the cluster, the time when the request occurred, the status code of the request, the Pod to which the request reached, the Internet Protocol IP address of the load balancer Pod to which the request reached, the Pod name of the backend service, the namespace to which the Pod of the backend service belongs, the IP address of the business node where the Pod of the backend service is located, and the total time of the entire request.

[0199] In one possible implementation, the first determining module 52 is specifically used for:

[0200] Based on the business alarm information, generate Elastic Search query statements and Prometheus query statements;

[0201] Use an Elastic Search query to retrieve Access Log information from a Kubernetes cluster.

[0202] Based on the Prometheus query, extract Pod information from the Kubernetes cluster, from the start time to the end time in the time information.

[0203] In one possible implementation, the first determining module 52 generates an ElasticSearch query statement based on the business alarm information, specifically for:

[0204] Based on the preset log query template, time information, and cluster name, generate an Elastic Search query statement. The log query template includes: start time, end time, cluster name, AND condition, OR condition, number of queries per query, and time sorting.

[0205] In one possible implementation, the first determining module 52 generates a Prometheus query statement based on the service alarm information, specifically for:

[0206] Based on the preset metric query template, time information, and cluster name, a Prometheus query statement is generated. The metric query template includes: start time, end time, and cluster name.

[0207] In one possible implementation, the Access Log information includes log information extracted from Elastic Search from the start time to the end time in the time information.

[0208] The fault location device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0209] Figure 6 A schematic diagram of the structure of the electronic device provided in this application. Figure 6 As shown, the electronic device includes at least one processor 61 and a memory 62.

[0210] Optionally, the device 6 also includes a communication component 63. The processor 61, memory 62, and communication component 63 are connected via a bus 64.

[0211] In a specific implementation, at least one processor 61 executes computer execution instructions stored in memory 62, causing at least one processor 61 to perform the above-described method.

[0212] The specific implementation process of processor 61 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0213] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0214] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0215] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0216] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0217] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0218] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0219] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0220] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0221] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0222] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0223] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0224] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0225] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A fault location method, characterized in that, The method, applied to electronic devices in a Kubernetes cluster, includes: Obtain service alarm information from the Kubernetes cluster, the service alarm information including: time information and cluster name; Based on the business alarm information, determine the Access Log information in the Kubernetes cluster and the container Pod information in the Kubernetes cluster; Based on the Access Log information and the Pod information in the Kubernetes cluster, determine the nine-tuple requests in the Kubernetes cluster that do not have a 200 response code; The non-200 response code nine-tuple request is treated as an abnormal request, and the fault information corresponding to the abnormal request is determined. The step of determining the nine-tuple requests with non-200 response codes in the Kubernetes cluster based on the Access Log information and the Pod information in the Kubernetes cluster includes: Determine the seven-tuple of the Access Log information, which includes: the time when the request occurred, the Internet Protocol address (IP) of the load balancer Pod to which the request arrived, the Pod of the load balancer to which the request arrived, the status code of the request, the request path, the Pod IP of the backend service, and the total time of the request. Based on the seven-tuple and the Pod information, determine the nine-tuple request sequence of the Kubernetes cluster; Identify the non-200 response code nine-tuple requests in the nine-tuple request sequence; The nine-tuple request includes: the request sent by the client to the cluster, the time when the request occurred, the status code of the request, the Pod to which the request reached, the Internet Protocol IP address of the load balancer Pod to which the request reached, the Pod name of the backend service, the namespace to which the Pod of the backend service belongs, the IP address of the business node where the Pod of the backend service is located, and the total time of the entire request.

2. The method according to claim 1, characterized in that, The step of determining the Access Log information and Pod information in the Kubernetes cluster based on the service alarm information includes: Based on the aforementioned business alarm information, generate Elastic Search query statements and Prometheus query statements; Based on the Elastic Search query statement, query the Access Log information in the Kubernetes cluster; Based on the Prometheus query, extract Pod information from the start time to the end time in the time information of the Kubernetes cluster.

3. The method according to claim 2, characterized in that, Based on the aforementioned business alarm information, generate an ElasticSearch query statement, including: Based on the preset log query template, the time information, and the cluster name, the Elastic Search query statement is generated. The log query template includes: start time, end time, cluster name, AND condition, OR condition, number of queries per query, and time sorting.

4. The method according to claim 2, characterized in that, Based on the aforementioned business alarm information, a Prometheus query statement is generated, including: Based on the preset indicator query template, the time information, and the cluster name, the Prometheus query statement is generated. The indicator query template includes: start time, end time, and cluster name.

5. The method according to claim 2, characterized in that, The Access Log information includes log information extracted from Elastic Search from the start time to the end time in the time information.

6. A fault location device, characterized in that, An electronic device used in a Kubernetes cluster, the device comprising: The acquisition module is used to acquire business alarm information in the Kubernetes cluster, including: time information and cluster name; The first determining module is used to determine the AccessLog information in the Kubernetes cluster and the Pod information in the Kubernetes cluster based on the business alarm information. The second determining module is used to determine the nine-tuple request with a non-200 response code in the Kubernetes cluster based on the Access Log information and the Pod information in the Kubernetes cluster. The third determining module is used to treat the non-200 response code nine-tuple request as an abnormal request and determine the fault information corresponding to the abnormal request. The second determining module, based on the Access Log information and the Pod information in the Kubernetes cluster, determines the nine-tuple requests in the Kubernetes cluster that do not have a 200 response code, specifically for: Determine the seven-tuple of the Access Log information, which includes: the time when the request occurred, the Internet Protocol address (IP) of the load balancer Pod to which the request arrived, the Pod of the load balancer to which the request arrived, the status code of the request, the request path, the Pod IP of the backend service, and the total time of the request. Based on the seven-tuple and the Pod information, determine the nine-tuple request sequence of the Kubernetes cluster; Identify the non-200 response code nine-tuple requests in the nine-tuple request sequence; The nine-tuple request includes: the request sent by the client to the cluster, the time when the request occurred, the status code of the request, the Pod to which the request reached, the Internet Protocol IP address of the load balancer Pod to which the request reached, the Pod name of the backend service, the namespace to which the Pod of the backend service belongs, the IP address of the business node where the Pod of the backend service is located, and the total time of the entire request.

7. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-5.

Images