Component analysis method and related device

Abstract

The present application discloses a component analysis method and related devices, the component analysis method comprising: receiving abnormal information of a first indicator of a first component; judging whether the first component has not called other components or whether the first indicators of the other components called by the first component have not been abnormal; wherein the number of the other components is at least one; if so, judging that the abnormality of the first indicator is caused by the first component; otherwise, obtaining a second component called by the first component and having an abnormality in the first indicator, taking the second component as the first component, and returning to the step of judging whether the first component has not called other components or whether the first indicators of the other components called by the first component have not been abnormal. In the above manner, the present application can locate the root cause of the system or component abnormality.

Description

Technical Field

[0001] The present application belongs to the field of Internet technology, and specifically relates to a component analysis method and related devices. Background Art

[0002] With the continuous development of computer and Internet technologies, the emergence of containerization technology has effectively solved the problems of environmental isolation and resource constraints between different components, and accelerated the large-scale deployment, dynamic migration and elastic scaling of components in distributed computing environments. However, at the same time, the deployment location and number of instances of these components in distributed environments have become uncertain. The complex and high-frequency dynamic migration and elastic scaling of components, as well as the massive concurrent access of users, have brought severe challenges to the stability of the system. The uncertainty of the component deployment location and the number of instances has increased the difficulty of system troubleshooting.

[0003] At present, common troubleshooting systems in distributed computing environments often find it difficult to accurately locate the root cause of system or component anomalies, and the troubleshooting time is long, resulting in reduced system availability. Summary of the invention

[0004] The present application provides a component analysis method and related devices to accurately locate the root cause of system or component abnormalities.

[0005] To solve the above technical problems, a technical solution adopted in the present application is: providing a component analysis method, including: receiving abnormal information of a first indicator of a first component; judging whether the first component has not called other components or whether the first indicators of the other components called by the first component have not had any abnormalities; wherein the number of the other components is at least one; if so, judging that the abnormality of the first indicator is caused by the first component; otherwise, obtaining a second component called by the first component and in which the first indicator has an abnormality, taking the second component as the first component, and returning to the step of judging whether the first component has not called other components or whether the first indicators of the other components called by the first component have not had any abnormalities.

[0006] To solve the above technical problems, a technical solution adopted in the present application is: to provide a component analysis device, including: a receiving module, used to receive abnormal information of a first indicator of a first component; a judgment module, used to judge whether the first component has not called other components or whether the first indicators of the other components called by the first component have not been abnormal; wherein the number of the other components is at least one; a first judgment module, connected to the judgment module, used to judge that the abnormality of the first indicator is caused by the first component when the judgment result of the judgment module is yes; a second judgment module, connected to the judgment module, used to obtain the second component called by the first component and the first indicator of which is abnormal when the judgment result of the judgment module is no, take the second component as the first component, and return to the step of judging whether the first component has not called other components or whether the first indicators of the other components called by the first component have not been abnormal.

[0007] To solve the above technical problems, a technical solution adopted in the present application is: to provide an electronic device, comprising a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the component analysis method described in any of the above embodiments.

[0008] In order to solve the above technical problems, another technical solution adopted by the present application is: providing a storage device storing program instructions that can be executed by a processor, wherein the program instructions are used to implement the component analysis method described in any of the above embodiments.

[0009] Different from the existing technology, the beneficial effect of the present application is that the present application can traverse the components on the link with the calling relationship and locate the root cause component that causes the exception, so as to improve the troubleshooting efficiency, shorten the fault recovery time, and improve the system availability. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work, among which:

[0011] Figure 1 This is a flow chart of an implementation method of a component analysis method of the present application;

[0012] Figure 2 This is a flow chart of another embodiment of the component analysis method of the present application;

[0013] Figure 3aA schematic diagram of the structure of an implementation method after weight matrix initialization;

[0014] Figure 3b It is a structural schematic diagram of an implementation method of performing statistics based on the first time sub-segment;

[0015] Figure 3c It is a structural schematic diagram of an implementation method after statistics are performed based on all time sub-segments;

[0016] Figure 4 This is a flow chart of another embodiment of the component analysis method of the present application;

[0017] Figure 5 This is a flow chart of another embodiment of the component analysis method of the present application;

[0018] Figure 6 This is a schematic structural diagram of an embodiment of a component analysis device of the present application;

[0019] Figure 7 This is a schematic diagram of the structure of an embodiment of an electronic device of the present application;

[0020] Figure 8 This is a schematic structural diagram of an implementation of a storage device of the present application. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0022] In large-scale clusters, there are many components and complex business call links. Although most faults can be discovered based on conventional indicator threshold monitoring and log monitoring, it is difficult to locate the root cause of the fault, resulting in a long time for fault recovery. The existing solution has the following problems:

[0023] 1) The lack of output specifications for component indicators and logs, or the lack of an effective specification guarantee mechanism, leads to different indicators, indicator attribute labels, and log data structures of different components. Even the same-name indicators have different measurement methods, numerical units, and semantics in different components, resulting in the need for the troubleshooting system to independently model and analyze different components. The root cause analysis and correlation analysis of faults for different components become complicated and difficult, increasing the complexity and difficulty of troubleshooting system design and implementation.

[0024] 2) Component crashes, freezes, and other failures rely on offline debugging and analysis, and the efficiency of fault location is low.

[0025] 3) Based on conventional indicator threshold monitoring, it is difficult to detect abnormal trends in indicator data, and it is difficult to provide early warning of faults.

[0026] 4) Lack of alarm filtering and root cause location based on the call link. When an upstream component or business fails, an alarm storm is likely to occur, making it difficult to locate the root cause of the fault.

[0027] In order to solve at least one of the problems in the above existing solutions, this solution will pre-set the output specifications of each indicator of each component before performing the normal component analysis method, and all components must comply with this specification when outputting indicator data. Regarding the normalization of component indicator data, specifically, it is agreed that the indicator data output by the component is time series data, which complies with the metrics indicator structure and type specifications; and the component can provide a GET request interface based on the http protocol, or has the function of actively pushing indicator data, so that the component returns various indicator data when it receives an indicator data return request or meets a certain condition. At the same time, the specification constrains the output of indicators in the following aspects:

[0028] 1) Standardize the semantics of indicators, that is, the semantics of each indicator is clear, and indicators with ambiguous semantics are refined or uniformly agreed upon. For example, for the memory usage of a component, if the name of the indicator is defined as "memory_usage", it is unclear whether the indicator includes cache, and different people will have different understandings, which will cause semantic ambiguity. At the same time, the measurement unit of the indicator is also unclear. Different components may use different measurement units for the indicator, which will eventually lead to confusion or errors in statistical calculations when the troubleshooting system performs statistical analysis. Therefore, it is necessary to refine and clarify to ensure that the measurement unit of the same indicator in different components is the same. For example, the memory usage of a component is subdivided into "memory_rss_usage_bytes" (which means: physical memory usage, measured in bytes) and "memory_cache_usage_bytes" (which means: cache memory usage, measured in bytes).

[0029] 2) Normalization of indicator dimensions. All components must output necessary indicator data. The indicators output by a component must at least include the four major categories of golden indicators of the monitoring system: delay, traffic, errors, and saturation; secondly, it must include custom indicator items for business-specific scenarios, such as the number of tasks stranded in the component task queue, the average stranded duration, etc., which can be customized according to business scenarios and needs. Among them, delay represents the specific processing delay of the business; traffic represents the number of calls made by the business per unit time; errors represent the number of errors in business calls, success rate, etc.; saturation represents the proportion of resources used by the application. The normalization of indicator dimensions is to meet the most basic data requirements for the troubleshooting system to analyze, alarm, and locate faults on components.

[0030] 3) Standardization of indicator attribute labels. Each specific indicator must contain necessary attribute labels (for example, component labels and indicator labels), followed by business-related custom attribute labels. For example, for a component indicator, it is usually necessary to include indicator attribute labels such as component name or container name, host IP or container IP. The standardization of indicator attribute labels is to meet the basic and detailed statistics and analysis of indicators by the troubleshooting system. For example, all components report the "memory_rss_usage_bytes" indicator item. In order to be able to statistically analyze the trend changes of different components with respect to this indicator, the attribute label of this indicator item must include the indicator attribute label of the component name, or other indicator attribute labels that can uniquely identify each component.

[0031] 4) Normalization of indicator values. Metrics provides five basic measurement types, namely Gauges, Counters, Histograms, Timers, and Meters (TPS counters). The measurement type of the indicator value must meet the preset requirements, that is, it must be selected from these five measurement types. At the same time, the indicator value must comply with the value specification of the indicator measurement type; for example, for a Counter type indicator, its indicator value must be greater than or equal to 0, and can only increase monotonically over time.

[0032] 5) Standardization of error code semantics. The same error code may represent different errors in different components, which means that error codes also have semantic ambiguity. In order to standardize the semantics of error codes, this application stipulates a list of error codes and clarifies the meaning of each error code. When a component needs to report indicator data of related errors in the process of processing business logic, it must report it according to the error code with corresponding semantics. If the error code with corresponding semantics is missing, the error code list needs to be supplemented.

[0033] The standardization of the indicator data of the above components ensures that the indicator data output by the components are isomorphic, which facilitates the unified modeling and analysis of these indicator data by the troubleshooting system. However, how to ensure that the specifications are well implemented is still a big challenge. For this reason, the business system implements an intelligent component to ensure the effective implementation of the specifications. The intelligent component is actually implemented as a service manager, which implements the abstract management of services. The business logic of specific services is abstracted into plug-ins, which are implemented by developers of different services. Different service plug-ins are combined with intelligent components to form a fully functional service component. In addition to the abstract management of services, the intelligent component also implements the collection of basic indicator data in the above specifications. By decoupling business logic from basic functions, the intelligent component ensures the standardization of basic indicator data in dimensions and semantics between different service components, and ensures that the indicator data between different components are isomorphic. In the specific application process, the indicator data output by each component meets the above specifications without the need for conversion processing by other servers, so as to reduce the computing load of the troubleshooting system and improve the working efficiency of the troubleshooting system.

[0034] During the operation of the system, the indicator data of each component can be collected. Since all components naturally implement the output of basic indicators and provide metrics interfaces, the collection of component indicators can be implemented based on Prometheus, which pulls indicator data through the component's metrics interface and stores it persistently.

[0035] In addition, log data can also be collected during system operation. Configure the docker runtime root directory of each machine as a unified designated directory. For component container logs, use DaemonSet type to deploy filebeat for log collection. DaemonSet is a management object for kubernetes container orchestration. Components deployed with the DeamonSet type will be automatically deployed to each machine that meets the machine label selection, and there is one service instance on each machine. For text logs customized by components, use SideCar to deploy filebeat for log collection. The so-called SideCar deployment method is to orchestrate the filebeat container and the component container in a pod management object. The filebeat container and the component container share the log directory through a shared data volume, so that filebeat can monitor and collect the text logs of the component. After the component container logs and business text logs are collected by filebeat, they are directly transferred to ElasticSearch for storage, and can be retrieved and displayed through Kibina later.

[0036] Generally speaking, during system operation, abnormal events such as component crash, component stuck, component startup abnormality, etc. may occur. The monitoring methods for the above abnormal events can be:

[0037] For component crash events, it is necessary to capture the crash exception during the component operation process, set the program exception capture and processing module at the program entry position of the component, and when the program crashes, it will trigger the exception capture and processing module to execute the corresponding processing logic.

[0038] For component stuck events, it is necessary to monitor whether the program's threads or coroutines are deadlocked or blocked. In the component's service management, session management is performed for each request; under normal circumstances, the service plug-in should return success or failure within the timeout period for each session. If one or more session requests do not return success or failure results within the timeout period, it is determined that the current service is stuck.

[0039] For component startup exception events, we need to capture various exceptions during the startup process that cause the component to fail to execute normally as expected. Common startup exception causes include insufficient memory, configuration file errors, network exceptions, database exceptions, etc. During the program startup process, the component will detect related exceptions in the environment and configuration. If an exception is found, it will be determined as a startup exception event. Otherwise, the initialization interface of the service plug-in will be executed. If the service startup is abnormal, the error code and the description of the cause of the exception will be returned through the initialization interface.

[0040] After the above abnormal events are monitored, event data related to the abnormal events can be collected; generally speaking, event data consists of three parts: basic information of the abnormal event (i.e., the attribute label of the abnormal event), data information associated with the abnormal event, and session parameters at the time when the abnormal event occurred. The basic information of the event is data that describes some basic attributes of the event itself. The data information associated with the abnormal event is some data associated with the root cause of the event, such as the crash or stuck stack, and a description of the cause of the startup exception. The session parameters at the time of the abnormal event are the session parameters that the component is processing at the time when the failure occurs. The collection methods of different event data vary, specifically:

[0041] 1) Collection of basic information of abnormal events. The basic information of abnormal events includes the abnormal event id, abnormal event type, service id, cluster name, service pod name, abnormal event occurrence time and metadata. The metadata of abnormal events is a text in json format, which includes the host address, the text name of the abnormal event associated data, and other extensible information. When an abnormal event occurs, the component generates basic event information according to the configuration of the service, and calls the event management interface of the troubleshooting system to append the event information.

[0042] 2) Collection of data information associated with abnormal events. Abnormal event-related data includes stack information at the time of crash and freeze events, and description information of the abnormal cause when the abnormal event is started. When the component detects the occurrence of an abnormal event, the processing module corresponding to the abnormal event will collect the abnormal event-related data. Specifically, for crash events, the component captures the stack of program panic exceptions through the "runtime.Stack" method of the "runtime" package, and writes it to a file with a naming convention of "event type_service id_event id". For freeze events, the component obtains the stack information of the current process based on the pstack tool, and writes it to a file with a naming convention of "event type_service id_event id". For startup abnormal events, the component writes the detected abnormal information or the abnormal information returned by the service plug-in initialization interface to a file with a naming convention of "event type_service id_event id". Finally, the log collection tool filebeat will monitor the changes in the content of the file, read it and transmit it to ElasticSearch for persistent storage.

[0043] 3) Collection of session parameters at the time of abnormal event. When a crash or freeze occurs, the component will also read the session parameters being processed in the session queue. If the session parameters contain multimedia data such as audio, video, and pictures, they will be base64 encoded, and then the abnormal event id will be used as the key and the json object array composed of the session parameters will be used as the value to store the session parameters at the time of the abnormal event in the database.

[0044] After determining the above specifications, the normal monitoring and analysis process can be carried out on multiple components in the system. Figure 1 , Figure 1 This is a flow chart of an implementation method of a component analysis method of the present application. The component analysis method specifically includes:

[0045] S101: Receive abnormal information of a first indicator of a first component.

[0046] Specifically, the processor can receive time series data information of multiple indicators uploaded by multiple components in the system; when it is determined that a certain indicator of a current component is abnormal based on the time series data information, the component where the abnormality occurs can be called the first component, and the certain indicator where the abnormality occurs can be called the first indicator. In some cases, there may be multiple first components, and there may be multiple first indicators of the first components. The following analysis process of steps S102-S104 can be performed for the first indicator of each first component where the abnormality occurs. Alternatively, the processor may obtain the first component and the first indicator based on the abnormal event after receiving the reporting information of the abnormal event. Alternatively, the processor may obtain the first component and the first indicator based on the reporting information of the error code after receiving the reporting information of the error code.

[0047] Generally speaking, the abnormality of the first indicator of the first component may be caused by the first component itself, or may be caused by other components in the call link of the first component. The purpose of subsequent steps S102-S104 is to determine the location of the root cause component that causes the abnormality of the first indicator of the first component.

[0048] S102: Determine whether the first component has not called other components or whether first indicators of other components called by the first component have not been abnormal; wherein the number of other components is at least one.

[0049] Specifically, in the APM system (application performance management system), the full-link trace_id tracking of the session request can obtain the call relationship between the components, which is essentially a directed graph. When a service exception occurs, the root cause analysis module of the troubleshooting system obtains the adjacency matrix of the component call relationship directed graph from the APM system to obtain an n×n matrix, where n represents the number of components; based on the adjacency matrix, the call relationship of the first component is obtained, that is, the call link where the first component is located is obtained. Among them, the definition f1(i,j)=1 indicates that component i calls component j, and f1(i,j)=0 indicates that there is no call relationship between component i and component j, where 1≤i,j≤n. In addition, the call referred to in this application is a one-way call, that is, after component i calls component j, component j returns information to component i, which is not called a call. The definition e(k,m)=1 indicates that the m indicator of component k is abnormal, and e(k,m)=0 indicates that the m indicator of component k is not abnormal, where 1≤k≤n.

[0050] When the abnormal information of the m index of component i is received in step S101, the judgment process of step S102 is expressed by the formula: Is it equal to 0 or Is it equal to 0?

[0051] S103: If yes, determine that the abnormality of the first indicator of the first component is caused by the first component.

[0052] Specifically, the meaning of the above if is: if the first component does not call other components, or the first component calls other components but the first indicators of other called components are normal, it can be determined that the abnormality of the first indicator of the first component is caused by the first component itself.

[0053] In an application scenario, assume that the following calling relationship is obtained based on the adjacency matrix: component A calls component B, component B calls component C, and component A calls component D; and after step S101, it is found that the m index of component A is abnormal, but after step S102, it is found that the m indexes of component B and component D called by component A are not abnormal, then it can be determined that the abnormality of the m index of component A is caused by component A itself.

[0054] S103: Otherwise, obtain the second component that is called by the first component and whose first indicator is abnormal, take the second component as the first component, and return to the step of determining whether the first component has not called other components, or whether the first indicators of other components called by the first component have not been abnormal.

[0055] Specifically, the above-mentioned otherwise means: if the first component calls other components and the first indicator of at least one of the called other components is abnormal, then the second component called by the first component and the first indicator of which is abnormal is judged to determine whether the root cause of the abnormality occurs in the second component.

[0056] In an application scenario, assume that the following calling relationship is obtained based on the adjacency matrix: component A calls component B, component B calls component C, and component A calls component D; and after step S101, it is found that the m index of component A is abnormal, but after step S102, it is found that the m index of component B called by component A is abnormal, and the m index of component D called by component A is not abnormal, then component B is obtained at this time, and the root cause of component B is determined. The process of determining the root cause of component B is similar to that of component A, that is, the calling relationship of component B is obtained based on the adjacency matrix, and then it is determined whether component B has not called other components or whether the m indexes of other components called by component B have not been abnormal. This cycle is repeated until the location of the root cause component is found.

[0057] Through the above design method, the present application can traverse the components on the links with calling relationships and locate the root cause components that cause the exceptions, so as to improve troubleshooting efficiency, shorten fault recovery time, and improve system availability.

[0058] In one embodiment, each component generally has time series indicators of many dimensions, and some indicators of some components have certain correlations, that is, when an indicator of one component is abnormal, another indicator of another component may also be abnormal at the same time. The troubleshooting system of the present application can cluster historical abnormal items based on historical abnormal time series data through cluster analysis to discover the relationship between many abnormal items. In the subsequent system monitoring process, only one representative indicator can be selected from multiple indicators clustered into one category as the system monitoring indicator, thereby simplifying the complexity of monitoring and the difficulty of root cause location. For details, please refer to Figure 2 , Figure 2 This is a flow chart of another embodiment of the component analysis method of the present application, the component analysis method comprising:

[0059] S201: Obtain abnormal time series data of multiple abnormal items that have abnormalities within a first preset time period; wherein the attribute labels of the abnormal items include component labels and indicator labels, and at least one of the component labels and indicator labels of different abnormal items is different.

[0060] For example, assuming that currently in the first preset time period, components A and B have abnormalities, component A has two indicators a and b, component B also has two indicators a and b, then the abnormal items include Aa, Ab, Ba and Bb.

[0061] S202: Divide the first preset time period into a plurality of time sub-segments, perform statistics based on each time sub-segment, increase the autocorrelation weight of the abnormal item where the abnormality occurs by a preset value, and increase the mutual correlation weight of two abnormal items where the abnormality occurs simultaneously by a preset value.

[0062] Optionally, in this embodiment, each time sub-segment can be counted with the help of a weight matrix. Specifically, before the above step S202, it includes: initializing the weight matrix; wherein the number of the multiple abnormal items is N, the weight matrix is ​​an N-row and N-column matrix, and the element in the i-th row and j-th column of the weight matrix represents the correlation weight between the i-th abnormal item and the j-th abnormal item; when the i-th abnormal item is the same as the j-th abnormal item, the correlation weight is the autocorrelation weight; when the i-th abnormal item is different from the j-th abnormal item, the correlation weight is the cross-correlation weight.

[0063] For example, when the abnormal items include Aa, Ab, Ba, and Bb, Figure 3a As shown, Figure 3a The schematic diagram of the structure of an implementation method after the weight matrix is ​​initialized. After the weight matrix is ​​initialized, the initial value of each weight can be 0; and optionally, the preset value can be a positive integer such as 1. The first preset time period can be divided into a plurality of time sub-segments, and the length of each time sub-segment is the same, and the end time of the previous time sub-segment in two adjacent time sub-segments coincides with the start time of the next time sub-segment.

[0064] like Figure 3b As shown, Figure 3b It is a structural schematic diagram of an implementation method of performing statistics based on the first time sub-segment. Figure 3b The first preset time period is divided into nine time sub-segments, and Figure 3b The circle in the middle indicates that an abnormality has occurred in the indicator of the corresponding component within the time sub-segment. Taking the first time sub-segment as an example, the abnormal items are Aa and Ba. At this time, the autocorrelation weight corresponding to Aa in the weight matrix is ​​+1, the autocorrelation weight corresponding to Ba is +1, and the mutual correlation weight related to Aa and Ba is +1. Similarly, taking the third time sub-segment as an example, the abnormal item is Bb. At this time, the autocorrelation weight corresponding to Bb in the weight matrix is ​​+1. Taking the sixth time sub-segment as an example, the abnormal items include Aa, Ab and Ba. At this time, the autocorrelation weights corresponding to Aa, Ab and Ba in the weight matrix are +1, the mutual correlation weight related to Aa and Ab is +1, the mutual correlation weight related to Aa and Ba is +1, and the mutual correlation weight related to Ab and Ba is +1. The final statistical results can be shown as follows Figure 3c As shown, Figure 3c It is a structural schematic diagram of an implementation method after statistics are performed based on all time sub-segments.

[0065] S203: For each abnormal item, in response to the autocorrelation weight of the current abnormal item exceeding the clustering threshold, and the mutual correlation weight related to the current abnormal item is within a preset range, the two abnormal items corresponding to the current mutual correlation weight are clustered into one category; wherein the preset range is related to the autocorrelation weight and the fluctuation coefficient.

[0066] Specifically, the autocorrelation weight p of the t index of component i can be obtained from the statistical weight matrix: it , if p it >δ, and its cross-correlation weight with the m index of component j is P(it,jm)≤p it ±ε, the t index of component i and the m index of component j are clustered into one category; where δ is the clustering threshold, ε is the fluctuation coefficient, and p it ±ε is the preset range.

[0067] For example, Figure 3c Take the abnormal item Aa as an example, and its related abnormal items include Ab and Ba. At this time, it is necessary to determine whether Aa and Ab can be clustered into one category, and whether Aa and Ba can be clustered into one category. Assuming that when it is determined that Aa and Ab can be clustered into one category, and Aa and Ba can be clustered into one category, Aa, Ab and Ba can be directly clustered into one category.

[0068] It should be noted that, in this embodiment, the above Figure 2 The cluster analysis in Figure 1Before or after the link analysis, or the above Figure 2 The cluster analysis in Figure 1 The link analysis in Figure 2 Cluster analysis in Figure 1 When the link analysis is performed before, one can be selected from the set of clusters as a monitoring object; when the abnormal information of the monitoring object (i.e. Figure 1 After the abnormal information of the first indicator of the first component in the first component is obtained, other indicator items that are clustered into the same category as the first indicator of the first component but are not monitored can be obtained. When it is determined that other indicator items are abnormal, the other indicator items are taken as the first component and the first indicator, and the same Figure 1 The process of step S102-step S104 is performed to determine the root cause component; that is, the component information and indicator information of other abnormal items are obtained, the component information and indicator information of other abnormal items are used as the first component and the first indicator, and then the process of step S102-step S104 is performed.

[0069] In another implementation, a component has many dimensions of timing indicators. Some of the indicators have certain correlations, and even certain causal relationships. For example, a large number of requests leads to increased interface delays. The troubleshooting system finds the correlations between indicators of different dimensions of the same component through correlation analysis, and takes a representative indicator among the strongly correlated indicators as the system's monitoring indicator, thereby simplifying the complexity of monitoring and the difficulty of root cause location. The specific implementation method is as follows, please refer to Figure 4 , Figure 4 This is a flow chart of another embodiment of the component analysis method of the present application, which specifically includes:

[0070] S301: Acquire first historical time series data of multiple indicators of the same component within a second preset time period; wherein, the component has an abnormality or no abnormality has occurred within the second preset time period.

[0071] Specifically, the first historical time series data may include abnormal data or non-abnormal data, and each indicator of the same component at the same time has corresponding first historical time series data.

[0072] S302: Obtain a correlation coefficient between two indicators based on first historical time series data of any two indicators of the same component.

[0073] Specifically, the correlation coefficient can be calculated according to the following formula:

[0074]

[0075] Among them, r xy is the correlation coefficient between indicator x and indicator y, n is the number of indicator data, x iis the index value of index x at the i-th moment, y i is the index value of index y at the i-th moment.

[0076] S303: In response to the absolute value of the correlation coefficient being greater than or equal to the first threshold, there is a strong correlation between the two indicators related to the correlation coefficient.

[0077] Optionally, the first threshold may be 0.8 or the like.

[0078] It should be noted that, in this embodiment, the above Figure 4 The correlation analysis in Figure 1 Before or after the link analysis, or separately Figure 1 The link analysis in Figure 4 Correlation analysis in Figure 1 When the link analysis is performed before, one can be selected from the set with strong correlation as the monitoring object; when the abnormal information of the monitoring object (i.e. Figure 1 After the abnormal information of the first indicator of the first component in the first component is obtained, other indicators that have a strong correlation with the first indicator of the first component but are not monitored can be obtained. The other indicators of the first component are also monitored. Figure 1 The process from step S102 to step S104 is performed to determine the root cause component; that is, the other indicators of the first component are used as the first indicator of the first component, and the process goes to step S102 to step S104.

[0079] In another embodiment, in a large-scale distributed cluster, indicators tend to show stable trend changes in time series, especially flow and saturation related indicators. For these indicators with obvious trend changes, a prediction model can be established by fitting historical data, and the value at a certain moment in the future can be predicted based on the prediction model. Conventional data fitting relies on a large amount of historical data and has a large amount of calculation. This application proposes trend prediction based on an improved moving average method, which is suitable for trend prediction within a nearby time period and has the characteristics of small amount of calculation and high real-time performance. For details, please refer to Figure 5 , Figure 5 This is a flow chart of another embodiment of the component analysis method of the present application. The component analysis method includes:

[0080] S401: Setting a time window based on a current moment in a current cycle, and obtaining second historical time series data of a target indicator of a target component in a same time window in multiple historical cycles before the current cycle.

[0081] Specifically, assuming that the current moment in the current cycle is t, the time window is defined as [t-Δt, t+Δt]. The size of the specific time window has little to do with the future moment to be predicted, as long as the future moment is around t+Δt. Slide the time window with a certain time period T (generally a day) to obtain the second historical time series data in the same time window in each historical cycle. For example, in the i-th time period T i The time window is [ti×T-Δt, ti×T+Δt], and the second historical time series data m in the time window is obtained. i1 ,…,m ip , where p is the number of time series indicators in the time window.

[0082] S402: Obtain an average of multiple second historical time series data at the same time based on all second historical time series data.

[0083] Specifically, it is expressed as follows:

[0084]

[0085] in, represents the mean, n represents the number of historical cycles, m iz Represents the time series indicator value at the zth moment in the ith historical period.

[0086] S403: Fit all mean values ​​at different times to obtain a fitting function.

[0087] Specifically, the mean values ​​at multiple moments may be subjected to linear or polynomial fitting to obtain the fitting function f2(t).

[0088] S404: Obtain a predicted value for a future moment after the current moment based on the fitting function and a plurality of second historical time series data before the current moment in the current cycle.

[0089] Specifically, the specific implementation process of the above step S404 can be:

[0090] A. Obtain the deviation based on the mean of the multiple second historical time series data before the current moment in the current cycle and the corresponding moment. Specifically, take the time series indicators m1,…,m in the time window [t-Δt, t] v , where v is the number of time series indicators in the time window [t-Δt, t], and the deviations between these time series data and the corresponding mean are calculated, which can be expressed as follows:

[0091]

[0092] Among them, δ is the deviation, m Jis the time series index value at the jth moment in the [t-Δt, t] time window, is the average value at the jth moment in multiple historical periods obtained in step S402.

[0093] B. Obtain the function value at the future moment based on the fitting function, and take the sum of the function value and the deviation as the predicted value.

[0094] Specifically, the future time t1 can be substituted into the fitting function f2(t) to obtain the function value f2(t1), and the sum of f2(t1) and δ is taken as the predicted value.

[0095] S405: Obtain a trend determination result of the target indicator based on the predicted value.

[0096] Specifically, the implementation process of the above step S405 can be: determine whether the predicted value is outside the first credible interval and whether the deviation is outside the second credible interval; if the predicted value is outside the first credible interval or the deviation is outside the second credible interval, determine that the trend of the target indicator is abnormal; otherwise, determine that the trend of the target indicator is normal.

[0097] It should be noted that, in this embodiment, the above Figure 5 Trend forecasting analysis can be found in Figure 1 Before or after the link analysis, or separately Figure 1 The link analysis in is performed.

[0098] In another embodiment, after an abnormal event occurs, fault log retrieval can be performed based on the abnormal event. Specifically, after the event occurs, when log retrieval is needed to assist in troubleshooting, the troubleshooting system queries the basic event information from the database based on the abnormal event id, and obtains information such as the time t when the event occurred, the service id, the event type, the log file name, and the service pod name. Then, based on the basic information of the abnormal event, the troubleshooting system calls the query interface in the ElasticSearch SDK to retrieve the container logs, component business logs, and crash or stuck stacks related to the event. The default retrieval is for related logs and stacks within the time period of [t-10min, t+10min], and supports users to independently select the start and end time of the retrieval. The troubleshooting system displays these log information on the log retrieval page.

[0099] In addition, when a crash or freeze occurs, the troubleshooting system collects the session parameters that are being processed at the time of the exception. The component crash or freeze is very likely to be caused by these session requests. Therefore, the online reproduction of the fault is mainly based on the playback of the session requests that are being processed at the time of the event. It also supports user-defined request playback. The specific process is as follows:

[0100] In response to receiving an online reproduction request for an abnormal event, based on the attribute tag of the abnormal event (for example, the id of the abnormal event), the session parameters at the time when the abnormal event occurred are obtained; the session parameters are converted into a first call request (for example, a gRPC or http call request), and the first call request is replayed. Users can replay these call requests one by one or in batches.

[0101] In response to the first call request failing to be replayed, receiving a user-constructed custom request parameter, generating a second call request based on the custom request parameter, and replaying the second call request. Specifically, the user can construct a custom request parameter based on a web protocol form, and replay the call requests generated by the custom request parameter one by one or in batches.

[0102] In addition, after an abnormal event occurs, it can also be debugged. The specific process can be as follows: the instance deployer queries the service id, cluster name, and service pod name associated with the event from the database of the troubleshooting system based on the event id. The instance deployer obtains the deployment orchestration file of the service component from the CD system based on the service id, cluster name, and service pod name. The instance deployer modifies the deployment orchestration file of the service component, modifies the node selection label to the specified target debugging machine, modifies the environment variables, prohibits the component from automatically registering and reporting service information to the LB, and keeps other information unchanged. The container startup command is changed to bash. Based on the modified service component orchestration file, the deployment of the service component debugging instance is completed through the API of the CD system. The instance deployer obtains the information of the service pod that has been started through the CD system query API, calls the management API of the CD system, and executes the exec operation to start the service process. At this point, a container instance with bash as the main process and the service process already started is started. The debugging terminal provides users with a web terminal that is easy to debug. After the container instance of the service is started, the debugging terminal first obtains the information of the service container through the CD system, then calls the docker API to obtain the http stream, and implements the execution and echo of the debugging operation based on xtermjs and web socket. After debugging is completed, the instance deployer calls the CD offline interface to complete the recovery of the debugged instance and resources.

[0103] See also Figure 6 , Figure 6 The schematic diagram of the structure of an embodiment of the component analysis device of the present application is as follows. The component analysis device comprises a receiving module 20 , a judging module 22 , a first judging module 24 and a second judging module 26 .

[0104] Among them, the receiving module 20 is used to receive the abnormal information of the first indicator of the first component. The judgment module 22 is used to judge whether the first component has not called other components or whether the first indicators of other components called by the first component have not been abnormal; wherein the number of other components is at least one. The first judgment module 24 is connected to the judgment module 22, and is used to judge that the abnormality of the first indicator is caused by the first component when the judgment result of the judgment module 22 is yes; the second judgment module 26 is connected to the judgment module 22, and is used to obtain the second component called by the first component and the first indicator is abnormal when the judgment result of the judgment module 22 is no, and use the second component as the first component, and return to the step of judging whether the first component has not called other components or whether the first indicators of other components called by the first component have not been abnormal.

[0105] In one embodiment, the component analysis device provided by the present application further includes a first acquisition module, a statistical module and a clustering module; wherein the first acquisition module is used to obtain abnormal time series data of multiple abnormal items that have abnormalities within a first preset time period; wherein the attribute label of the abnormal item includes a component label and an indicator label, and at least one of the component labels and indicator labels of different abnormal items is different. The statistical module is connected to the first acquisition module, and is used to divide the first preset time period into multiple time sub-segments, perform statistics based on each time sub-segment, increase the autocorrelation weight of the abnormal item that has abnormalities by a preset value, and increase the mutual correlation weight of two abnormal items that have abnormalities at the same time by a preset value. The clustering module is connected to the statistical module, and is used for each abnormal item, in response to the autocorrelation weight of the current abnormal item exceeding the clustering threshold, and the mutual correlation weight related to the current abnormal item is within a preset range, then the two abnormal items corresponding to the current mutual correlation weight are clustered into one category; wherein the preset range is related to the autocorrelation weight and the fluctuation coefficient.

[0106] Optionally, an initialization module may be included between the first acquisition module and the statistical module, for initializing a weight matrix; wherein the number of the multiple abnormal items is N, the weight matrix is ​​an N-row and N-column matrix, and the element in the i-th row and j-th column of the weight matrix represents the correlation weight between the i-th abnormal item and the j-th abnormal item; when the i-th abnormal item is the same as the j-th abnormal item, the correlation weight is the autocorrelation weight; when the i-th abnormal item is different from the j-th abnormal item, the correlation weight is the cross-correlation weight.

[0107] In another embodiment, the component analysis device provided by the present application further includes: a second acquisition module, a third acquisition module and a correlation determination module; wherein the second acquisition module is used to obtain the first historical time series data of multiple indicators of the same component within a second preset time period; wherein the component has an abnormality or no abnormality within the second preset time period. The third acquisition module is connected to the second acquisition module, and is used to obtain the correlation coefficient between two indicators based on the first historical time series data of any two indicators of the same component. The correlation determination module is connected to the third acquisition module, and is used to respond to the absolute value of the correlation coefficient being greater than or equal to the first threshold value, and the two indicators related to the correlation coefficient have a strong correlation relationship.

[0108] In another embodiment, the component analysis device provided by the present application also includes a fourth acquisition module, a fifth acquisition module, a fitting module, a prediction module and a trend determination module. Among them, the fourth acquisition module is used to set a time window based on the current moment in the current cycle, and obtain the second historical time series data of the target indicator of the target component in the same time window in multiple historical cycles before the current cycle. The fifth acquisition module is connected to the fourth acquisition module, and is used to obtain the mean of multiple second historical time series data at the same moment based on all second historical time series data. The fitting module is connected to the fifth acquisition module, and is used to fit all mean values ​​at different moments to obtain a fitting function. The prediction module is connected to the fitting module, and is used to obtain the predicted value of the future moment after the current moment based on the fitting function and the multiple second historical time series data before the current moment in the current cycle. The trend determination module is connected to the prediction module, and is used to obtain the trend determination result of the target indicator based on the predicted value.

[0109] Optionally, the above-mentioned prediction module is specifically used to obtain the deviation based on multiple second historical time series data before the current moment in the current cycle and the mean of the corresponding moment; obtain the function value at the future moment based on the fitting function, and use the sum of the function value and the deviation as the predicted value.

[0110] Alternatively, the trend determination module is specifically used to determine whether the predicted value is outside the first credible interval and whether the deviation is outside the second credible interval; if the predicted value is outside the first credible interval or the deviation is outside the second credible interval, the target indicator trend is determined to be abnormal; otherwise, the target indicator trend is determined to be normal.

[0111] In another embodiment, the component analysis device provided by the present application also includes: a setting module, which is used to set the output specifications of each indicator of each component; wherein the semantics of each indicator is clear, the measurement unit of the same indicator in different components is the same, and the measurement type of the indicator meets the preset requirements; and / or, the indicators output by each component include at least delay, traffic, error, saturation, and custom indicators corresponding to the business; and / or, the attribute label of each indicator includes a component label, an indicator label, and a custom attribute label related to the business; and / or, setting the output specifications of each error code reported by the component; wherein the semantics of each error code is clear; and / or, setting the output specifications of each abnormal event occurring in the component; wherein the abnormal event includes at least one of a component crash, a component freeze, and a component startup abnormality; the output data of the abnormal event includes the attribute label of the abnormal event, data information associated with the abnormal event, and the session parameters at the time when the abnormal event occurs.

[0112] In another embodiment, the component analysis device provided by the present application also includes an online playback module, which is used to obtain the session parameters at the time when the abnormal event occurred based on the attribute tags of the abnormal event in response to receiving an online reproduction request for the abnormal event; convert the session parameters into a first call request, and replay the first call request; in response to the first call request being unable to be replayed, receive custom request parameters constructed by the user, generate a second call request based on the custom request parameters, and replay the second call request.

[0113] See also Figure 7 , Figure 7The schematic diagram of the structure of an embodiment of an electronic device of the present application. The electronic device comprises: a memory 32 and a processor 30 coupled to each other, wherein program instructions are stored in the memory 32, and the processor 30 is used to execute the program instructions to implement any of the above-mentioned component analysis methods. Specifically, the electronic device includes but is not limited to: a desktop computer, a laptop computer, a tablet computer, a server, etc., which are not limited here. In addition, the processor 30 can also be referred to as a CPU (Center Processing Unit). The processor 30 may be an integrated circuit chip with signal processing capabilities. The processor 30 may also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. In addition, the processor 30 may be implemented by an integrated circuit chip.

[0114] See also Figure 8 , Figure 8 This is a schematic diagram of the structure of an embodiment of a storage device of the present application. The storage device 40 stores program instructions 400 that can be executed by a processor, and the program instructions 400 are used to implement any of the above-mentioned component analysis methods.

[0115] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0116] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0117] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0118] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0119] The above description is only an embodiment of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A component analysis method, characterized in that: include: Receiving abnormal information of a first indicator of a first component; Determine whether the first component does not call other components or whether first indicators of the other components called by the first component are not abnormal; wherein the number of the other components is at least one; If yes, it is determined that the abnormality of the first indicator is caused by the first component; Otherwise, obtain a second component that is called by the first component and in which the first indicator is abnormal, use the second component as the first component, and return to the step of determining whether the first component does not call other components or whether the first indicators of the other components called by the first component are not abnormal; Wherein, the component analysis method further includes: obtaining abnormal time series data of multiple abnormal items that have abnormalities within a first preset time period; wherein the attribute label of the abnormal item includes a component label and an indicator label, and at least one of the component label and the indicator label of different abnormal items is different; dividing the first preset time period into multiple time sub-segments, performing statistics based on each of the time sub-segments, increasing the autocorrelation weight of the abnormal item that has abnormalities by a preset value, and increasing the mutual correlation weight of two abnormal items that have abnormalities at the same time by the preset value; for each of the abnormal items, in response to the autocorrelation weight of the current abnormal item exceeding the clustering threshold, and the mutual correlation weight related to the current abnormal item being within a preset range, the two abnormal items corresponding to the current mutual correlation weight are clustered into one category; wherein the preset range is related to the autocorrelation weight and the fluctuation coefficient; Before the step of receiving the abnormal information of the first indicator of the first component, it includes: setting the output specification of each indicator of each component; wherein, the semantics of each indicator is clear, the measurement unit of the same indicator in different components is the same, and the measurement type of the indicator meets the preset requirements; and / or, the indicators output by each component at least include delay, traffic, error, saturation, and custom indicators corresponding to the business; and / or, the attribute label of each indicator includes a component label, an indicator label and a custom attribute label related to the business; and / or, setting the output specification of each error code reported by the component; wherein, the semantics of each error code is clear; and / or, setting the output specification of each abnormal event occurring in the component; wherein, the abnormal event includes at least one of component crash, component freeze, and component startup abnormality; the output data of the abnormal event includes the attribute label of the abnormal event, the data information associated with the abnormal event, and the session parameters at the time when the abnormal event occurs.

2. The component analysis method according to claim 1, characterized in that: Before the step of performing statistics based on each of the time sub-segments, the method includes: Initialize a weight matrix; wherein the number of the multiple abnormal items is N, the weight matrix is ​​an N-row and N-column matrix, and the element in the i-th row and j-th column in the weight matrix represents the correlation weight between the i-th abnormal item and the j-th abnormal item; when the i-th abnormal item is the same as the j-th abnormal item, the correlation weight is the autocorrelation weight; when the i-th abnormal item is different from the j-th abnormal item, the correlation weight is the cross-correlation weight.

3. The component analysis method according to claim 1, characterized in that: Also includes: Acquire first historical time series data of multiple indicators of the same component within a second preset time period; wherein the component has an abnormality or no abnormality occurs within the second preset time period; Obtaining a correlation coefficient between two indicators based on first historical time series data of any two indicators of the same component; In response to the absolute value of the correlation coefficient being greater than or equal to the first threshold, there is a strong correlation between the two indicators related to the correlation coefficient.

4. The component analysis method according to claim 1, characterized in that: Also includes: A time window is set based on a current moment in a current cycle, and second historical time series data of a target indicator of a target component in the same time window in multiple historical cycles before the current cycle is obtained; Obtaining an average of a plurality of the second historical time series data at the same time based on all the second historical time series data; Fitting all the mean values ​​at different moments to obtain a fitting function; Obtaining a predicted value at a future moment after the current moment based on the fitting function and a plurality of second historical time series data before the current moment in the current cycle; A trend determination result of the target indicator is obtained based on the predicted value.

5. The component analysis method according to claim 4, characterized in that: The step of obtaining a predicted value at a future moment after the current moment based on the fitting function and the second historical time series data before the current moment in the current cycle includes: Obtaining a deviation based on a plurality of second historical time series data before the current moment in the current period and the mean value at the corresponding moment; A function value at a future time is obtained based on the fitting function, and the sum of the function value and the deviation is used as the predicted value.

6. The component analysis method according to claim 5, characterized in that: The step of obtaining the trend determination result of the target indicator based on the predicted value includes: Determining whether the predicted value is outside a first credible interval and whether the deviation is outside a second credible interval; If the predicted value is outside the first credible interval or the deviation is outside the second credible interval, the target indicator trend is determined to be abnormal; otherwise, the target indicator trend is determined to be normal.

7. The component analysis method according to claim 1, characterized in that: Also includes: In response to receiving an online reproduction request for an abnormal event, obtaining a session parameter at a time when the abnormal event occurs based on an attribute tag of the abnormal event; Converting the session parameters into a first call request, and playing back the first call request; In response to the first call request being unable to be replayed, receiving a user-defined request parameter constructed by a user, generating a second call request based on the user-defined request parameter, and replaying the second call request.

8. A component analysis device, characterized in that: include: A receiving module, used for receiving abnormal information of a first indicator of a first component; A judgment module, used to judge whether the first component does not call other components or whether the first indicators of the other components called by the first component are not abnormal; wherein the number of the other components is at least one; A first determination module, connected to the determination module, configured to determine that the abnormality of the first indicator is caused by the first component when the determination result of the determination module is yes; A second determination module is connected to the determination module and is used for, when the determination result of the determination module is negative, obtaining a second component called by the first component and having an abnormality in the first indicator, taking the second component as the first component, and returning to the step of determining whether the first component has not called other components or whether the first indicators of the other components called by the first component have not all been abnormal; Wherein, the component analysis device also includes a first acquisition module, a statistical module and a clustering module; the first acquisition module is used to obtain abnormal time series data of multiple abnormal items that occur abnormally within a first preset time period; wherein, the attribute label of the abnormal item includes a component label and an indicator label, and at least one of the component labels and indicator labels of different abnormal items is different; the statistical module is connected to the first acquisition module, and is used to divide the first preset time period into multiple time sub-segments, perform statistics based on each time sub-segment, increase the autocorrelation weight of the abnormal item that occurs abnormally by a preset value, and increase the mutual correlation weight of two abnormal items that occur abnormally at the same time by a preset value; the clustering module is connected to the statistical module, and is used for each abnormal item, in response to the autocorrelation weight of the current abnormal item exceeding the clustering threshold, and the mutual correlation weight related to the current abnormal item is within a preset range, then the two abnormal items corresponding to the current mutual correlation weight are clustered into one category; wherein the preset range is related to the autocorrelation weight and the fluctuation coefficient; Wherein, the component analysis device also includes a setting module, which is used to set the output specifications of each indicator of each component; wherein, the semantics of each indicator is clear, the measurement unit of the same indicator in different components is the same, and the measurement type of the indicator meets the preset requirements; and / or, the indicators output by each component include at least delay, traffic, error, saturation, and custom indicators corresponding to the business; and / or, the attribute label of each indicator includes a component label, an indicator label, and a custom attribute label related to the business; and / or, the output specifications of each error code reported by the component are set; wherein the semantics of each error code are clear; and / or, the output specifications of each abnormal event occurring in the component are set; wherein the abnormal event includes at least one of a component crash, a component freeze, and a component startup abnormality; the output data of the abnormal event includes the attribute label of the abnormal event, data information associated with the abnormal event, and the session parameters at the time when the abnormal event occurs.

9. An electronic device, characterized in that: The method comprises a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the component analysis method according to any one of claims 1 to 7.

10. A storage device, characterized in that: Program instructions that can be executed by a processor are stored, and the program instructions are used to implement the component analysis method according to any one of claims 1 to 7.

Images