Fault positioning method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a fault positioning method and device, equipment and a storage medium, and the method is applied to electronic equipment in a Kubernetes cluster, and comprises the steps: obtaining service alarm information in the Kubernetes cluster; according to the technical means, the method comprises the steps that service alarm information is sent to a Kubernetes cluster, Access Log information in the Kubernetes cluster and Pod information in the Kubernetes cluster are determined according to the service alarm information, a nine-tuple request of a non-200 response code in the Kubernetes cluster is determined according to the Access Log information and the Pod information in the Kubernetes cluster, the nine-tuple request of the non-200 response code serves as an abnormal request, and fault information corresponding to the abnormal request is determined. The technical effect of improving the fault analysis efficiency is achieved.

Description

Technical Field

[0001] The present application relates to the field of intelligent operation and maintenance technology, and in particular to a fault location method, device, equipment, and storage medium. Background Art

[0002] In recent years, with the continuous advancement of cloud migration, various applications are constantly moving from off-cloud server deployment to on-cloud Kubernetes clusters. With the hybrid deployment of cluster services and the continuous increase in business volume, the monitoring and analysis of cluster status, application status, and business requests through intelligent operation and maintenance can greatly reduce the burden on operation and maintenance personnel and generate extensive benefits.

[0003] When the business access success rate drops, the operation and maintenance personnel still need to manually view various data such as the cluster monitoring indicators, and combine their own experience to analyze and locate the fault.

[0004] However, the above process is cumbersome and time-consuming, resulting in low efficiency of fault analysis. Summary of the invention

[0005] The embodiments of the present application provide a fault location method, apparatus, device, and storage medium to achieve the technical effect of improving the efficiency of fault analysis.

[0006] In a first aspect, an embodiment of the present application provides a fault location method, which is applied to an electronic device in a Kubernetes cluster, and the method includes:

[0007] Obtaining service alarm information in the Kubernetes cluster, where the service alarm information includes: time information and cluster name;

[0008] Determine the Access Log information in the Kubernetes cluster and the Pod information in the Kubernetes cluster according to the business alarm information;

[0009] Determine, according to the Access Log information and the Pod information in the Kubernetes cluster, a nine-tuple request with a non-200 response code in the Kubernetes cluster;

[0010] The nine-tuple request with a non-200 response code is regarded as an abnormal request, and fault information corresponding to the abnormal request is determined.

[0011] In a possible implementation, determining, according to the Access Log information and the Pod information in the Kubernetes cluster, a nine-tuple request with a non-200 response code in the Kubernetes cluster includes:

[0012] Determine the seven-tuple of the Access Log log information, the seven-tuple including: the time when the request occurs, the Pod IP of the load balancer to which the request arrives, the Pod of the load balancer to which the request arrives, the status code of the request, the request path, the Pod IP of the backend service, and the total time of the request;

[0013] Determine a nine-tuple request sequence of the Kubernetes cluster according to the seven-tuple and the Pod information;

[0014] A nine-tuple request having a non-200 response code in the nine-tuple request sequence is determined.

[0015] In one possible implementation, the nine-tuple request includes: the request sent by the client to the cluster, the time when the request occurs, the status code of the request, the Pod to which the request arrives, the Internet Protocol IP address of the load balancer Pod at which the request reaches, the Pod name of the backend service, the namespace to which the Pod of the backend service belongs, the business node IP where the Pod of the backend service is located, and the total time of the entire request.

[0016] In a possible implementation, determining the Access Log information in the Kubernetes cluster and the Pod information in the Kubernetes cluster according to the business alarm information includes:

[0017] Generate an Elastic Search query statement and a Prometheus query statement according to the business alarm information;

[0018] According to the Elastic Search query statement, query the Access Log information in the Kubernetes cluster;

[0019] According to the Prometheus query statement, extract the Pod information in the Kubernetes cluster from the start time to the end time in the time information.

[0020] In a possible implementation, generating an Elastic Search query statement according to the service alarm information includes:

[0021] The ElasticSearch query statement is generated according to a preset log query template, the time information, and the cluster name, wherein the log query template includes: start time, end time, cluster name, and condition, or condition, the number of queries each time, and time sorting.

[0022] In a possible implementation, generating a Prometheus query statement according to the business alarm information includes:

[0023] The Prometheus query statement is generated according to a preset indicator query template, the time information, and the cluster name, wherein the indicator query template includes: start time, end time, and cluster name.

[0024] In a possible implementation, the Access Log log information includes: log information extracted from Elastic Search from the start time to the end time in the time information.

[0025] In a second aspect, an embodiment of the present application provides a fault location device, which is applied to an electronic device in a Kubernetes cluster, and the device includes:

[0026] An acquisition module is used to acquire business alarm information in the Kubernetes cluster, where the business alarm information includes: time information and cluster name;

[0027] A first determination module is used to determine the Access Log information in the Kubernetes cluster and the Pod information in the Kubernetes cluster according to the business alarm information;

[0028] A second determination module is used to determine a nine-tuple request with a non-200 response code in the Kubernetes cluster according to the Access Log information and the Pod information in the Kubernetes cluster;

[0029] The third determination module is used to treat the nine-tuple request with a non-200 response code as an abnormal request and determine the fault information corresponding to the abnormal request.

[0030] In a possible implementation manner, the second determining module is specifically configured to:

[0031] Determine the seven-tuple of the Access Log log information, the seven-tuple including: the time when the request occurs, the Pod IP of the load balancer to which the request arrives, the Pod of the load balancer to which the request arrives, the status code of the request, the request path, the Pod IP of the backend service, and the total time of the request;

[0032] Determine a nine-tuple request sequence of the Kubernetes cluster according to the seven-tuple and the Pod information;

[0033] A nine-tuple request having a non-200 response code in the nine-tuple request sequence is determined.

[0034] In one possible implementation, the nine-tuple request includes: the request sent by the client to the cluster, the time when the request occurs, the status code of the request, the Pod to which the request arrives, the Internet Protocol IP address of the load balancer Pod at which the request reaches, the Pod name of the backend service, the namespace to which the Pod of the backend service belongs, the business node IP where the Pod of the backend service is located, and the total time of the entire request.

[0035] In a possible implementation manner, the first determining module is specifically configured to:

[0036] Generate an Elastic Search query statement and a Prometheus query statement according to the business alarm information;

[0037] According to the Elastic Search query statement, query the Access Log information in the Kubernetes cluster;

[0038] According to the Prometheus query statement, extract the Pod information in the Kubernetes cluster from the start time to the end time in the time information.

[0039] In a possible implementation, the first determining module generates an Elastic Search query statement according to the service alarm information, specifically for:

[0040] The ElasticSearch query statement is generated according to a preset log query template, the time information, and the cluster name, wherein the log query template includes: start time, end time, cluster name, and condition, or condition, the number of queries each time, and time sorting.

[0041] In a possible implementation, the first determination module generates a Prometheus query statement according to the service alarm information, specifically for:

[0042] The Prometheus query statement is generated according to a preset indicator query template, the time information, and the cluster name, wherein the indicator query template includes: start time, end time, and cluster name.

[0043] In a possible implementation, the Access Log log information includes: log information extracted from Elastic Search from the start time to the end time in the time information.

[0044] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0045] The memory stores computer-executable instructions;

[0046] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0047] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0048] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0049] The fault location method, apparatus, device, and storage medium provided in the embodiments of the present application are applied to electronic devices in a Kubernetes cluster. The method includes: obtaining business alarm information in the Kubernetes cluster, the business alarm information including: time information and cluster name; determining Access Log information in the Kubernetes cluster and Pod information in the Kubernetes cluster according to the business alarm information, determining nine-tuple requests with non-200 response codes in the Kubernetes cluster according to the Access Log information and the Pod information in the Kubernetes cluster, treating the nine-tuple requests with non-200 response codes as abnormal requests, and determining technical means for fault information corresponding to the abnormal requests, thereby achieving the technical effect of improving the efficiency of fault analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0051] Figure 1 Schematic diagram of the fault location method provided in this application Figure 1 ;

[0052] Figure 2 Schematic diagram of the fault location method provided in this application Figure 2 ;

[0053] Figure 3 Schematic diagram of the fault location method provided in this application Figure 3 ;

[0054] Figure 4 Schematic diagram of the fault location method provided in this application Figure 4 ;

[0055] Figure 5 A schematic diagram of the structure of the fault location device provided in this application;

[0056] Figure 6 A schematic diagram of the structure of the electronic device provided in this application.

[0057] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0058] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0059] First, the terms used in this application are explained:

[0060] 1) Kubernetes:

[0061] Refers to an open source container scheduling and orchestration platform, referred to as K8S; it can orchestrate, schedule and manage containers (English abbreviation: Pod) packaged with business applications; Kubernetes cluster consists of two parts, namely the control plane (management node) and the data plane (business node); Pod refers to the scheduling unit in Kubernetes, which is the encapsulation of one or more containers;

[0062] 2) Management Node:

[0063] The master node (English: Master) refers to the node in the Kubernetes cluster that is responsible for managing and scheduling the entire cluster. It consists of multiple machines, one master and multiple backups. The management node deploys Kubernetes core components, such as the ingress controller (English: Ingress CONtroller), API server (English: Server), controller manager (English: Controller-manager), scheduler (English: Scheduler), distributed key-value storage system ETCD, etc., which are responsible for controlling cluster communication, scheduling and arranging cluster resources, storing cluster data and status information, etc.;

[0064] 3) Business nodes:

[0065] Node refers to the node in the Kubernetes cluster that is responsible for running business applications. In addition to business Pods, it also includes components such as Kubelet and Kube-proxy, which are responsible for tasks such as creating, starting, stopping, communicating, and load balancing the containers corresponding to the Pods;

[0066] 4), Nginx Ingress Controller:

[0067] Ingress refers to the seven-layer load balancer used in the Kubernetes cluster, which is deployed on the management node in the Kubernetes cluster in the form of Pod. The client's request will first go to the cluster's Ingress Controller load balancer, and then the load balancer will proxy the request to the Pod of the backend service and output the corresponding log to the access log (English: access log) according to the request.

[0068] 5) Access Log:

[0069] Refers to the log file of Nginx Ingress Controller. When Nginx Ingress Controller forwards requests, it records the request log in the file for operation and maintenance personnel to analyze the access behavior of client requests. For each request, an Access Log is generated. The format of Access Log is: <$remote_addr-$remote_user$time_local"$request"$status$body_bytes_sent"$http_referer""$http_user_agent"$request_length$request_time$proxy_upstream_name$upstream_addr>;

[0070] Among them, $remote_addr refers to the client address; $remote_user refers to the client user name; $time_local refers to the access time; $request refers to the requested URI; $status refers to the status code of the request; $body_bytes_sent refers to the size of the file content sent to the client; $http_referer refers to the URL jump source; $http_user_agent refers to the user terminal browser and other information; $request_length refers to the length of the request; $request_time refers to the total time of the entire request; $proxy_upstream_name refers to the backend service to which the request is forwarded; $upstream_addr refers to the backend address to which the request is forwarded

[0071] 6) Elastic Search:

[0072] Refers to an open source, highly scalable, distributed full-text search engine that provides an API interface to achieve fast full-text search. By connecting the logs of the Kubernetes cluster to Elastic Search, it can help operation and maintenance personnel call the application programming interface (API) for fast log search and analysis;

[0073] 7) Prometheus:

[0074] Refers to an open source service monitoring system and time database, which provides a common data model and a quick data collection, storage and query interface. By writing PromQL data query statements, it is possible to query and aggregate monitoring indicator data;

[0075] 8), Nine-tuple request:

[0076] Refers to a nine-tuple containing complete request information, and its expression requestInfo is: <OccurTime,Status,IngressPod,IngressIp,UpstreamPod,UpstreamPodNs,URI,UpstreamNodeIp,RequesTime> ;

[0077] Among them, requestInfo represents the request sent by the client to the cluster; OccurTime represents the time when the request occurs; Status represents the status code of the request; IngressPod represents the Pod of the Nginx IngressController to which the request arrives, and the request is forwarded from the load balancer Pod to the Pod of the backend service; IngressIp represents the IP of the Nginx Ingress Controller Pod to which the request arrives; UpstreamPod represents the Pod name of the backend service; UpstreamPodNs represents the namespace to which the backend service Pod belongs; UpstreamNodeIp represents the business node IP where the backend service Pod is located; URI represents the request path; RequestTime represents the total time of the entire request.

[0078] Secondly, the technical background involved in this application is explained:

[0079] In recent years, with the continuous advancement of cloud migration, various applications are constantly moving from off-cloud server deployment to on-cloud Kubernetes clusters. With the hybrid deployment of cluster services and the continuous increase in business volume, the monitoring and analysis of cluster status, application status, and business requests through intelligent operation and maintenance can greatly reduce the burden on operation and maintenance personnel and generate extensive benefits.

[0080] When the business access success rate drops, the operation and maintenance personnel still need to manually view various data such as the cluster monitoring indicators, and combine their own experience to analyze and locate the fault.

[0081] However, the above process is cumbersome and time-consuming, resulting in low efficiency of fault analysis.

[0082] The present application provides a fault location method, which specifically involves generating query conditions based on templates, automatically extracting Kubernetes cluster component logs reported to Elastic Search according to the query conditions, automatically extracting Prometheus indicator information, and using the captured logs and indicator information, using a rule-based approach to quickly extract abnormal logs and generate nine-tuple requests involved in the abnormality, thereby accurately and quickly locating the fault and facilitating further analysis by operation and maintenance personnel.

[0083] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0084] Figure 1 Schematic diagram of the fault location method provided in this application Figure 1 ,like Figure 1 As shown, the method is applied to electronic devices in a Kubernetes cluster, including:

[0085] The electronic device may be an analysis device for fault location.

[0086] Step 11: Get the business alarm information in the Kubernetes cluster.

[0087] Among them, the business alarm information includes: time information, cluster name;

[0088] In this step, when a service alarm occurs in the Kubernetes cluster, the electronic equipment in the Kubernetes cluster can obtain the service alarm and extract basic information such as time information and cluster name in the service alarm information.

[0089] In a possible implementation, the expression of the service alarm information basicInfo can be<alertInfo.t1,alertInfo.t2,alertInfo.cluster> .

[0090] Among them, alertInfo is the business alarm, t1 is the start time in the time information, t2 is the end time in the time information, and cluster is the cluster name.

[0091] Step 12: Determine the Access Log information and Pod information in the Kubernetes cluster based on the business alarm information;

[0092] In this step, the Pod in Kubernetes is the unit that runs the business application. Based on the time information and cluster name in the business alarm information, you can view the information of the Pod in the cluster at the relevant time.

[0093] Also, in a Kubernetes cluster, Access Log information is usually recorded by Ingress Nginx or service proxy. Based on the business alarm information, you can view the logs of the relevant Ingress or proxy, analyze the request status code, response time, etc., to help further confirm the problem.

[0094] Optionally, the Pod information in the Kubernetes cluster is all Pod information from the start time to the end time corresponding to the above time information.

[0095] Step 13: According to the Access Log information and the Pod information in the Kubernetes cluster, determine the nine-tuple request with a non-200 response code in the Kubernetes cluster;

[0096] In this step, based on the Access Log information, the seven-tuple of the Access Log information can be determined first, and then based on the seven-tuple and the Pod information in the Kubernetes cluster, the nine-tuple request sequence of the Kubernetes cluster can be determined, and the nine-tuple request sequence includes multiple nine-tuple requests.

[0097] Then, the nine-tuple request with a non-200 response code is extracted from the nine-tuple request sequence.

[0098] Optional, non-200 response code nine-tuple request usually represents an error or abnormal state, as a fault indication signal, need to be paid special attention to in the production environment; mainly because it usually means that there is a problem in the request processing, resulting in no successful response being returned.

[0099] In addition, the nine-tuple request includes: the request sent by the client to the cluster, the time when the request occurred, the status code of the request, the Pod to which the request reaches, the Internet Protocol IP address of the load balancer Pod when the request reaches the load balancer Pod, the Pod name of the backend service, the namespace to which the Pod of the backend service belongs, the business node IP where the Pod of the backend service is located, and the total time of the entire request.

[0100] By combining Access Log information with Pod information in the Kubernetes cluster, you can effectively identify and analyze nine-tuple requests with non-200 response codes, thereby locating and solving problems in the cluster.

[0101] Step 14: Treat the nine-tuple request with a non-200 response code as an abnormal request, and determine the fault information corresponding to the abnormal request.

[0102] In this step, the nine-tuple request with a non-200 response code is regarded as an abnormal request, and the abnormal request is parsed to determine the corresponding fault information.

[0103] For example, the following uses a non-200 response code nine-tuple request (such as 4xx, 5xx error code) as an example:

[0104] 1. Client error (4xx response code):

[0105] 1) 400 Bad Request: The request format is incorrect. Usually, the request data sent by the client is invalid or lacks necessary parameters.

[0106] 2)401Unauthorized: The request did not provide valid authentication credentials.

[0107] 3)403Forbidden: The requested resource is prohibited from access, even if the correct authentication credentials are provided.

[0108] 4)404 Not Found: The requested resource does not exist, possibly because the requested URL is incorrect.

[0109] 5)408Request Timeout: The request timed out and the client waited longer than expected for the server to respond.

[0110] 2. Server error (5xx response code):

[0111] 1) 500 Internal Server Error: The server encountered an unexpected error and was unable to complete the request. This is usually a bug or resource limitation issue on the server.

[0112] 2)502Bad Gateway: Usually means that the proxy or gateway server cannot connect to the backend server.

[0113] 3)503Service Unavailable: The service is unavailable. The server may be overloaded, resources are exhausted, or the service is temporarily unavailable.

[0114] 4)504Gateway Timeout: The gateway or proxy server times out while waiting for a response from the backend server.

[0115] The fault locating method provided in the embodiment of the present application is applied to an electronic device in a Kubernetes cluster, and the method includes: obtaining business alarm information in the Kubernetes cluster, the business alarm information including: time information and cluster name; determining Access Log information in the Kubernetes cluster and Pod information in the Kubernetes cluster according to the business alarm information, determining nine-tuple requests with non-200 response codes in the Kubernetes cluster according to the Access Log information and the Pod information in the Kubernetes cluster, treating the nine-tuple requests with non-200 response codes as abnormal requests, and determining technical means for fault information corresponding to the abnormal requests, thereby achieving the technical effect of improving the efficiency of fault analysis.

[0116] Based on the above embodiments, Figure 2 Schematic diagram of the fault location method provided in this application Figure 2 ,like Figure 2 As shown, the above step 12 may include the following implementations:

[0117] Step 21: Generate Elastic Search query statements and Prometheus query statements based on business alarm information;

[0118] Before this step, a log query template (eg, Elastic Search log query template) and an indicator query template (Prometheus indicator query template) may be generated in advance.

[0119] Among them, 1) the Elastic Search log query template is used to generate specific ElasticSearch query statements; 2) the Prometheus indicator query template is used to generate Prometheus query statements.

[0120] Optionally, in step 21, generating an Elastic Search query statement according to the business alarm information may be implemented as follows: generating an Elastic Search query statement according to a preset log query template, time information, and cluster name;

[0121] The log query template includes: start time, end time, cluster name, and condition, or condition, number of queries per time, and time sorting.

[0122] In this implementation, when a business alarm occurs in the Kubernetes cluster, an Elastic Search query statement queryes is generated based on the log query condition template template es according to the time information, cluster name and other conditions in the business alarm information. The formula is:

[0123] Query es =template es (t1, t2, cluster, list(cond AND ), list(cond OR ));

[0124] Among them, cond AND is the query and condition set; cond OR A collection of conditions or conditions for a query.

[0125] Optionally, in step 21, generating a Prometheus query statement according to the business alarm information may be implemented as follows: generating a Prometheus query statement according to a preset indicator query template, time information, and cluster name;

[0126] The indicator query template includes: start time, end time, cluster name

[0127] In this implementation, when a business alarm occurs in the Kubernetes cluster, a Prometheus query statement queryprom is generated based on the indicator query template template prom according to the time information, cluster name and other conditions in the business alarm information. The formula is:

[0128] query prom=template prom(t1, t2, cluster).

[0129] Step 22: Query the Access Log information in the Kubernetes cluster according to the Elastic Search query statement;

[0130] In this step, use the Elastic Search query statement to query the Access Log information of the cluster Nginx IngressController component (that is, the load balancer).

[0131] Optionally, the Access Log log information includes: log information extracted from ElasticSearch from the start time to the end time in the time information.

[0132] In a possible implementation, the log information extracted from the start time to the end time in Elastic Search is logs, that is, the formula is as follows:

[0133] Logs=[Log1, Log2, Log3,...Logn];

[0134] Among them, taking logi as an example, the formula can be:

[0135] Logi=<OccurTimei,IngressIpi,IngressPodi,accesslogi> .

[0136] Specifically, logi represents the log information tuple of the i-th request; OccurTimei represents the time when the log information tuple occurs; IngressIpi represents the IP of the Nginx Ingress Controller Pod to which the i-th request arrives; IngressPodi represents the name of the Nginx Ingress Controller Pod to which the i-th request arrives; accesslogi represents the Access Log corresponding to the i-th request.

[0137] Step 23: According to the Prometheus query statement, extract the Pod information from the start time to the end time in the time information in the Kubernetes cluster.

[0138] In this step, use the Prometheus query statement to extract all Pod information of the cluster from the start time to the end time.

[0139] In a possible implementation, the Pod information extracted from the start time to the end time in the Kubernetes cluster is Pods, that is, the formula is as follows:

[0140] Pods=[Pod1, Pod2, Pod3,...Podn];

[0141] Among them, taking Podi as an example, the formula can be:

[0142] Podi=<Pod_ipi,nsi,Pod_namei,host_ipi> .

[0143] Specifically, Podi represents the indicator information tuple of the i-th Pod; Pod_ipi represents the IP of the i-th Pod; nsi represents the namespace to which the i-th Pod belongs; Pod_namei represents the i-th Pod name; host_ipi represents the IP of the node where the i-th Pod is located.

[0144] The fault location method provided in the embodiment of the present application generates ElasticSearch query statements and Prometheus query statements based on business alarm information, queries the Access Log information in the Kubernetes cluster based on the Elastic Search query statement, and extracts the Pod information from the start time to the end time in the time information in the Kubernetes cluster based on the Prometheus query statement. The above method can generate query conditions based on templates and extract cluster log information and indicator information.

[0145] Based on the above embodiments, Figure 3 Schematic diagram of the fault location method provided in this application Figure 3 ,like Figure 3 As shown, the above step 13 may include the following implementation:

[0146] Step 31, determine the seven-tuple of Access Log information;

[0147] The seven-tuple includes: the request status code (Status), the request path (URI), the Pod Ip (UpstreamNodeIp) of the backend service, the total request time (RequestTime), the time when the request occurred (OccurTime), the Pod Ip (IngressIp) of the load balancer (such as Nginx Ingress Controller) where the request arrives, the Pod (IngressPod) of the NginxIngress Controller where the request arrives, and the request is forwarded from the load balancer Pod to the Pod of the backend service;

[0148] In this step, the Access Log information is further analyzed in a rule-based manner to extract the status code of the request, the request path, the Pod IP of the backend service, and the total request time in the Access Log information to obtain a seven-tuple of the log information.

[0149] The formula for the seven-tuple of log information is:

[0150] Logs_extracted=[lge1, lge 2, lge 3,...lge n];

[0151] Among them, taking lge i as an example, the formula can be:

[0152] lge i=<OccurTimei,IngressIpi,IngressPodi,Statusi,URIi,UpsteamPodIpi,RequestTimei> .

[0153] Specifically, lge i represents the seven-tuple of log information of the i-th request; Statusi represents the status code of the i-th request; URIi represents the request path of the i-th request; UpsteamPodIpi represents the backend service Pod Ip to which the i-th request is forwarded by NginxIngress Controller; RequestTimei represents the total time consumed by the i-th request.

[0154] Step 32: Determine the nine-tuple request sequence of the Kubernetes cluster based on the seven-tuple and Pod information;

[0155] In this step, all the seven-tuples and Pod information obtained above are matched and spliced ​​to generate the nine-tuple request requestsInfo of the cluster.

[0156] Optionally, the formula of the nine-tuple request sequence requestsInfos is as follows:

[0157] requestsInfos=[requestsInfo1, requestsInfo2,...requestsInfon].

[0158] Specifically, for any nine-tuple request:

[0159] requestsInfo= <OccurTime,Status,IngressPod,IngressIp,UpstreamPod,UpstreamPodNs,URI,UpstreamNodeIp,RequesTime> .

[0160] Step 33: Determine the nine-tuple request with a non-200 response code in the nine-tuple request sequence.

[0161] In this step, the nine-tuple request sequence is screened according to the response code of each request, and the nine-tuple requests with non-200 response codes are extracted, that is, the requests involved in the exception.

[0162] The formula is as follows:

[0163] AbnormalRequestInfo is requestInfo (requestInfo.Start is not equal to 200).

[0164] The fault location method provided in the embodiment of the present application determines the seven-tuple of Access Log log information, the seven-tuple includes: the status code of the request, the request path, the Pod IP of the backend service and the total request time; determines the nine-tuple request sequence of the Kubernetes cluster according to the seven-tuple and the Pod information; determines the nine-tuple request with a non-200 response code in the nine-tuple request sequence. The above method can quickly extract the abnormal log and the nine-tuple request involved in the abnormality, thereby assisting the operation and maintenance personnel to further analyze and complete the rapid location of the fault.

[0165] Based on the above embodiments, Figure 4 Schematic diagram of the fault location method provided in this application Figure 4 ,like Figure 4 As shown, an overall embodiment of the fault location method may be:

[0166] Step 1. Start;

[0167] Step 2: Extraction of business alarm information, executing steps 3 and 7 respectively;

[0168] In this embodiment, when a service alarm occurs, basic information such as the start time, end time, and cluster name in the service alarm information is extracted;

[0169] Step 3: Generate a log query template;

[0170] In this embodiment, an Elastic Search query template is generated, which mainly includes the start time, end time, cluster name, and condition, or condition, the number of queries per time, and time sorting;

[0171] Step 4: Generate Elastic Search query statements;

[0172] In this embodiment, according to the time, cluster name and other conditions in the alarm information, an Elastic Search query statement is generated based on the log query condition template;

[0173] Step 5. Query Elastic Search log information;

[0174] In this embodiment, the Elastic Search query statement is used to query the Access Log information of the Nginx IngressController component of the cluster;

[0175] Step 6: Extract the seven-tuple in the log information and execute step 10;

[0176] In this embodiment, a rule-based approach is adopted to further mine the log information in step 3, extract the status code of the request, the request path, the Pod IP of the backend service, and the total request time in the Access Log, and obtain a seven-tuple of the log information;

[0177] Step 7: Generate an indicator query template;

[0178] In this embodiment, a Prometheus query template is generated, which mainly includes the start time, end time, and cluster name;

[0179] Step 8. Generate Prometheus query statements;

[0180] In this embodiment, according to the time, cluster name and other conditions in the alarm information, a Prometheus query statement is generated based on the indicator query template;

[0181] Step 9. Extract Pod information;

[0182] In this embodiment, a Prometheus query statement is used to extract all Pod information of the cluster from the start time to the end time;

[0183] Step 10: Match and generate nine-tuple requests;

[0184] In this embodiment, all Pod information obtained in step G and step H are matched and spliced ​​with the seven-tuple of log information to generate a cluster nine-tuple request;

[0185] Step 11: Extract abnormal nine-tuple requests;

[0186] In this embodiment, the nine-tuple request sequence is screened according to the response code of each request, and the nine-tuple requests with non-200 response codes are extracted to obtain the abnormal requests involved.

[0187] Step 12. End.

[0188] The fault location method involved in the present application can generate query conditions based on templates, extract cluster log information and indicator information, and use a rule-based approach to quickly extract exception logs and the nine-tuple requests involved in the exception, thereby assisting operation and maintenance personnel in further analysis and completing rapid fault location.

[0189] Figure 5 A schematic diagram of the structure of the fault location device provided in this application, such as Figure 5 As shown, the fault location device is applied to electronic equipment in a Kubernetes cluster, including:

[0190] The acquisition module 51 is used to obtain the business alarm information in the Kubernetes cluster, and the business alarm information includes: time information and cluster name;

[0191] The first determination module 52 is used to determine the Access Log information in the Kubernetes cluster and the Pod information in the Kubernetes cluster according to the business alarm information;

[0192] The second determination module 53 is used to determine the nine-tuple request with a non-200 response code in the Kubernetes cluster according to the Access Log information and the Pod information in the Kubernetes cluster;

[0193] The third determining module 54 is configured to treat the nine-tuple request with a non-200 response code as an abnormal request and determine the fault information corresponding to the abnormal request.

[0194] In a possible implementation manner, the second determining module 53 is specifically configured to:

[0195] Determine the seven-tuple of Access Log information, which includes: the time when the request occurred, the Pod IP of the load balancer to which the request arrived, the Pod of the load balancer to which the request arrived, the status code of the request, the request path, the Pod IP of the backend service, and the total request time;

[0196] Determine the nine-tuple request sequence of the Kubernetes cluster based on the seven-tuple and Pod information;

[0197] Identify the nine-tuple requests with non-200 response codes in the nine-tuple request sequence.

[0198] In one possible implementation, the nine-tuple request includes: the request sent by the client to the cluster, the time when the request occurs, the status code of the request, the Pod to which the request reaches, the Internet Protocol IP address of the load balancer Pod when the request reaches the load balancer Pod, the Pod name of the backend service, the namespace to which the Pod of the backend service belongs, the business node IP where the Pod of the backend service is located, and the total time of the entire request.

[0199] In a possible implementation manner, the first determining module 52 is specifically configured to:

[0200] Generate Elastic Search query statements and Prometheus query statements based on business alarm information;

[0201] Query the Access Log information in the Kubernetes cluster based on the Elastic Search query statement;

[0202] According to the Prometheus query statement, extract the Pod information from the start time to the end time in the time information in the Kubernetes cluster.

[0203] In a possible implementation, the first determination module 52 generates an ElasticSearch query statement according to the service alarm information, specifically for:

[0204] Generate an Elastic Search query statement based on the preset log query template, time information, and cluster name. The log query template includes: start time, end time, cluster name, and condition, or condition, number of queries per time, and time sorting.

[0205] In a possible implementation, the first determination module 52 generates a Prometheus query statement according to the service alarm information, specifically for:

[0206] Generate a Prometheus query statement based on the preset indicator query template, time information, and cluster name. The indicator query template includes: start time, end time, and cluster name.

[0207] In a possible implementation, the Access Log log information includes: log information extracted from Elastic Search from the start time to the end time in the time information.

[0208] The fault location device provided in this embodiment can execute the method provided in the above method embodiment, and its implementation principle and technical effect are similar, which will not be described in detail in this embodiment.

[0209] Figure 6 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 6 As shown, the electronic device includes: at least one processor 61 and a memory 62.

[0210] Optionally, the device 6 further includes a communication component 63 , wherein the processor 61 , the memory 62 and the communication component 63 are connected via a bus 64 .

[0211] In a specific implementation process, at least one processor 61 executes the computer-executable instructions stored in the memory 62, so that at least one processor 61 executes the above method.

[0212] The specific implementation process of the processor 61 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0213] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0214] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0215] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0216] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0217] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0218] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0219] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0220] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0221] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0222] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0223] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0224] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0225] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A fault location method, characterized in that: Applied to an electronic device in a Kubernetes cluster, the method comprises: Obtaining service alarm information in the Kubernetes cluster, where the service alarm information includes: time information and cluster name; Determine the Access Log information in the Kubernetes cluster and the container Pod information in the Kubernetes cluster according to the business alarm information; Determine, according to the Access Log information and the Pod information in the Kubernetes cluster, a nine-tuple request with a non-200 response code in the Kubernetes cluster; The nine-tuple request with a non-200 response code is regarded as an abnormal request, and fault information corresponding to the abnormal request is determined.

2. The method according to claim 1, characterized in that The determining, according to the Access Log information and the Pod information in the Kubernetes cluster, a nine-tuple request with a non-200 response code in the Kubernetes cluster includes: Determine the seven-tuple of the Access Log log information, the seven-tuple including: the time when the request occurs, the Internet Protocol address IP of the load balancer Pod where the request arrives, the Pod of the load balancer where the request arrives, the status code of the request, the request path, the Pod IP of the backend service, and the total time of the request; Determine a nine-tuple request sequence of the Kubernetes cluster according to the seven-tuple and the Pod information; A nine-tuple request having a non-200 response code in the nine-tuple request sequence is determined.

3. The method according to claim 2, characterized in that The nine-tuple request includes: the request sent by the client to the cluster, the time when the request occurs, the status code of the request, the Pod to which the request arrives, the Internet Protocol IP address of the load balancer Pod when the request reaches the load balancer Pod, the Pod name of the backend service, the namespace to which the Pod of the backend service belongs, the business node IP where the Pod of the backend service is located, and the total time of the entire request.

4. The method according to any one of claims 1 to 3, characterized in that: Determining the Access Log information in the Kubernetes cluster and the Pod information in the Kubernetes cluster according to the business alarm information includes: Generate an Elastic Search query statement and a Prometheus query statement according to the business alarm information; According to the Elastic Search query statement, query the Access Log information in the Kubernetes cluster; According to the Prometheus query statement, extract the Pod information in the Kubernetes cluster from the start time to the end time in the time information.

5. The method according to claim 4, characterized in that Generate an ElasticSearch query statement based on the business alarm information, including: The Elastic Search query statement is generated according to a preset log query template, the time information, and the cluster name, wherein the log query template includes: a start time, an end time, a cluster name, an AND condition, an OR condition, a number of queries per time, and a time sort.

6. The method according to claim 4, characterized in that Generate a Prometheus query statement based on the business alarm information, including: The Prometheus query statement is generated according to a preset indicator query template, the time information, and the cluster name, wherein the indicator query template includes: start time, end time, and cluster name.

7. The method according to claim 4, characterized in that The Access Log information includes: log information extracted from Elastic Search from the start time to the end time in the time information.

8. A fault location device, characterized in that: An electronic device applied to a Kubernetes cluster, the device comprising: An acquisition module is used to acquire business alarm information in the Kubernetes cluster, where the business alarm information includes: time information and cluster name; A first determination module is used to determine the AccessLog log information in the Kubernetes cluster and the Pod information in the Kubernetes cluster according to the business alarm information; A second determination module is used to determine a nine-tuple request with a non-200 response code in the Kubernetes cluster according to the Access Log information and the Pod information in the Kubernetes cluster; The third determination module is used to treat the nine-tuple request with a non-200 response code as an abnormal request and determine the fault information corresponding to the abnormal request.

9. An electronic device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images