Root cause failure determination method and apparatus, electronic device, and storage medium

Abstract

The application relates to the technical field of reliability testing, and provides a root cause failure determination method and device, electronic equipment and a storage medium. The method comprises the following steps: in response to concurrent injection of failures when a reliability test task is executed, triggering a cloud stress test task associated with the reliability test task, and collecting running data when the cloud stress test task is executed; mapping a target dependency graph into a directed acyclic graph structure of a Bayesian network; calculating marginal posterior probabilities of each failure with respect to observed abnormal evidence according to the running data and the directed acyclic graph structure; calculating marginal causal effects of each failure on a page index by using a backdoor criterion; and determining a root cause failure from each failure according to the marginal posterior probabilities and the marginal causal effects. The application can establish a quantifiable causal chain between failure injection, interface abnormality and page failure, reduce the dependence of root cause positioning on artificial experience, and improve the accuracy and efficiency of root cause positioning in a multi-failure concurrent scenario.

Description

Technical Field

[0001] This application belongs to the field of reliability testing technology, and in particular relates to a root cause failure determination method, apparatus, electronic device and storage medium. Background Technology

[0002] With the deepening of industrial informatization, system stability faces severe challenges. Reliability testing, which can actively inject faults to verify the fault tolerance of a system, is an important means of ensuring system stability. Meanwhile, cloud-based fault testing technology can simulate real user requests to monitor the system end-to-end from the user's perspective. However, in scenarios with multiple concurrent faults, the independent impact of each fault on interface and page anomalies is difficult to decouple effectively due to the simultaneous action of multiple faults on the system. This leads to confounding biases in reliability assessment results, making it impossible to establish a quantifiable causal chain between fault injection and user experience degradation. Summary of the Invention

[0003] In view of the shortcomings of the prior art, this application provides a root cause fault determination method, apparatus, electronic device and storage medium, which at least solves the problem in the related art that it is impossible to establish a quantifiable causal chain between fault injection and user experience degradation.

[0004] In a first aspect, this application provides a method for determining root cause failures, including: In response to concurrent fault injection during the execution of reliability testing tasks, trigger cloud testing tasks associated with the reliability testing tasks and collect runtime data during the execution of cloud testing tasks. The target dependency graph is mapped to a directed acyclic graph structure of a Bayesian network. The target dependency graph represents the dependency relationship between services, interfaces and pages. In the directed acyclic graph structure, the fault node is the parent node, the interface node is the intermediate node, the page node is the child node, and the edge direction represents the causal direction of fault propagation. Based on the operational data and the structure of the directed acyclic graph, calculate the marginal posterior probability of each fault relative to the observed anomalous evidence. The marginal causal effect of each fault on page metrics is calculated using the backdoor criterion. Based on the marginal posterior probability and marginal causal effect, the root cause fault is identified from each fault.

[0005] In one embodiment of this application, the marginal posterior probability of each fault relative to observed anomalous evidence is calculated based on operational data and the directed acyclic graph structure, including: Determine the prior probability of each fault; construct a first conditional probability table of interface nodes relative to fault nodes and a second conditional probability table of page nodes relative to interface nodes based on the interface call success rate, response latency distribution, and page load dependency; when abnormal evidence is observed from the runtime data, determine the joint posterior probability of the fault combination state based on the prior probability of each fault node, the first conditional probability table, and the second conditional probability table; determine the marginal posterior probability of each fault based on the joint posterior probability.

[0006] In one embodiment of this application, the marginal causal effect of each fault on page metrics is calculated using the backdoor criterion, including: Each fault is sequentially designated as a target fault. The injection states of other faults besides the target faults are fixed. The first expected value of the target fault on the page metrics under the first preset condition and the second expected value of the target fault on the page metrics under the second preset condition are obtained respectively. Based on the first and second expected values, the marginal causal effect of the target fault is determined.

[0007] In one embodiment of this application, the root cause fault is identified from each fault based on marginal posterior probability and marginal causality, including: The root cause comprehensive score of each fault is determined based on the marginal posterior probability and marginal causal effect; the root cause fault is then identified from the faults based on the root cause comprehensive score.

[0008] In one embodiment of this application, after identifying the root cause fault from among the faults based on marginal posterior probability and marginal causality, the method further includes: Generate a fault propagation path; and / or, obtain at least one of interface health, page flow integrity, and response time; determine a target comprehensive score based on at least one of interface health, page flow integrity, and response time; generate test conclusions for the reliability test task based on the comparison between the target comprehensive score and a preset score threshold; and execute pipeline control operations based on the test conclusions and preset blocking strategies.

[0009] In one embodiment of this application, before mapping the target dependency graph to a directed acyclic graph structure of a Bayesian network, the method further includes: Parse historical pages and iterate through the test data to extract the data requests initiated by each page and establish the first mapping relationship from page nodes to interface nodes; obtain the service instance corresponding to the interface through the service registry center and / or parse the link tracing data of the server to establish the second mapping relationship from interface nodes to service nodes; construct the target dependency graph based on the first and second mapping relationships.

[0010] In one embodiment of this application, the method further includes: Initialize fault parameters, including adjustment step size and indicator thresholds, including warning thresholds and failure thresholds; if the running data collected for a preset number of consecutive times meets the preset conditions, the fault intensity is increased by adjusting step size; if any collected running data exceeds the warning threshold, the fault intensity is decreased by adjusting step size; if any collected running data exceeds the failure threshold, the preset target operation is executed.

[0011] Secondly, this application also provides a root cause fault determination apparatus, comprising: The task execution module is configured to respond to fault concurrency injection when executing a reliability test task, trigger the cloud dialing test task associated with the reliability test task, and collect the running data when executing the cloud dialing test task. The graph mapping module is configured to map the target dependency graph into a directed acyclic graph structure of a Bayesian network. The target dependency graph represents the dependency relationship between services, interfaces and pages. In the directed acyclic graph structure, the fault node is the parent node, the interface node is the intermediate node, the page node is the child node, and the edge direction represents the causal direction of fault propagation. The probability calculation module is configured to calculate the marginal posterior probability of each fault relative to the observed anomalous evidence based on the running data and the directed acyclic graph structure. The fault determination module is configured to calculate the marginal causal effect of each fault on page metrics using the backdoor criterion, and to identify the root cause fault from each fault based on the marginal posterior probability and the marginal causal effect.

[0012] Thirdly, this application also provides an electronic device, which includes: one or more processors; and a storage device for storing one or more programs, which, when executed by one or more processors, cause the electronic device to perform the steps of the root cause fault determination method described above.

[0013] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer's processor, causes the computer to implement the steps of the above-described root cause fault determination method.

[0014] The beneficial effects of this technical solution are as follows: This application constructs a target dependency graph containing dependencies between services, interfaces, and pages, and maps this target dependency graph to a directed acyclic graph structure of a Bayesian network. Based on the runtime data during cloud testing and the directed acyclic graph structure, the marginal posterior probability of each fault relative to observed anomalous evidence is calculated. Simultaneously, the marginal causal effect of each fault on page metrics is calculated using the backdoor criterion. Based on the marginal posterior probability and marginal causal effect, the root cause fault is identified from each fault. Since the edge direction of the directed acyclic graph structure represents the causal direction of fault propagation, the marginal posterior probability calculation of the Bayesian network can quantify the correlation strength between fault nodes and anomalous evidence. The intervention calculation of the backdoor criterion can eliminate confounding paths between multiple faults, thereby accurately separating the independent impact of each fault on page performance metrics in multi-fault concurrent scenarios, ensuring that the root cause localization result has both probabilistic and causal basis. This establishes a quantifiable causal chain from fault injection to interface anomaly to page failure, reducing the reliance on human experience in root cause localization and improving the accuracy and efficiency of root cause localization in scenarios with multiple concurrent faults.

[0015] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings: Figure 1 This is a schematic flowchart illustrating a root cause failure determination method according to an exemplary embodiment of this application; Figure 2 This is a schematic diagram illustrating the structural relationships between services, interfaces, and pages, as shown in an exemplary embodiment of this application. Figure 3 This is a schematic diagram illustrating the structure of a root cause fault determination device according to an exemplary embodiment of this application; Figure 4 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0017] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0018] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. The drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the shape, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0019] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the present application. However, it will be apparent to those skilled in the art that embodiments of the present application may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the present application.

[0020] Distributed systems typically consist of numerous services, interfaces, and front-end pages. A single user operation often requires navigating a multi-level call chain involving pages, interfaces, and services. Reliability testing, which involves proactively injecting faults into the system to verify its fault tolerance, is a crucial means of ensuring system stability. In typical reliability testing scenarios, testers can inject faults such as network latency, CPU overload, and a full database connection pool into a controlled environment to observe the system's performance under fault conditions and assess its resilience.

[0021] However, in actual reliability testing practices, fault injection and observation results are usually only simply aligned on a timeline, lacking an effective causal inference mechanism. When an anomaly occurs in the system during fault injection, it is difficult to determine which fault directly caused which interface to malfunction, and consequently which page to fail. The root causes of this technical problem are: First, the observed data only focuses on internal system metrics such as CPU utilization, memory usage, and error logs, lacking end-to-end verification of the real user behavior chain, making the impact of faults on actual business availability unobservable; Second, the transmission relationship between faults, interfaces, and pages is not explicitly modeled as structured data with causal direction, and the observed anomalies and injected faults only have a temporal correlation, making it impossible to distinguish between causal relationships and pseudo-correlation relationships; Third, a fault may indirectly affect other interfaces by affecting a certain service, forming a hybrid path, causing the independent impact of each fault to be mixed together, distorting the evaluation results.

[0022] In view of this, this application provides a root cause fault determination method, apparatus, electronic device and storage medium to at least solve the problem in the related art that it is impossible to establish a quantifiable causal chain between fault injection and user experience degradation.

[0023] Please see Figure 1 , Figure 1 This is a flowchart illustrating a root cause failure determination method as shown in an exemplary embodiment of this application. Figure 1 As shown, in an exemplary embodiment, the root cause fault determination method includes steps S110 to S140, and each step is described in detail below.

[0024] S110, in response to fault concurrency injection during the execution of reliability testing tasks, triggers cloud dial-up testing tasks associated with the reliability testing tasks and collects runtime data during the execution of cloud dial-up testing tasks. It is understandable that concurrent fault injection refers to the operation of injecting multiple faults into one or more services in a controlled environment during the execution of a reliability testing task. Multiple faults can act on the system simultaneously or within overlapping time windows.

[0025] Cloud probing tasks refer to observation tasks that simulate real users initiating access requests through a probing engine. Cloud probing tasks include two types of subtasks: interface probing subtasks and page traversal probing subtasks. Interface probing subtasks are used to send data requests to the target system's interfaces and collect response data at the interface level. Page traversal probing subtasks are used to simulate users traversing and accessing pages through a headless browser and collect loading data at the page level.

[0026] When performing reliability tasks, concurrent faults are injected into the target application in a controlled environment, i.e., concurrent fault operations occur. The controlled environment can be a test environment or a pre-production environment. Association rules can be pre-established between reliability test tasks and cloud testing tasks; one reliability test task is bound to at least one cloud testing task.

[0027] In response to concurrent fault injection during reliability testing, a cloud probing task associated with the reliability testing task can be triggered during the fault injection period or within a configurable time window after the injection, and runtime data during the execution of the cloud probing task can be collected. At this time, the interface probing subtask concurrently sends HTTP or HTTPS requests based on a high-performance Hypertext Transfer Protocol (HTTP) or Hypertext Transfer Protocol Secure (HTTPS) client library, collecting runtime data at the interface level. Runtime data includes, but is not limited to, request time, response time, HTTP status codes, business error codes, success assertion rate, and response body size.

[0028] The page traversal testing subtask launches a headless browser instance based on the Playwright headless browser automation framework, simulating user behavior to traverse one or more system pages and collect page-level runtime data, including but not limited to maximum content rendering, DOMContentLoaded time, load events, script errors, network errors, traversal step status, element reachability, and screenshots.

[0029] Furthermore, it can receive precise timestamps for the start and end of a fault, as well as raw test logs with precise timestamps. It aligns fault injection time, API request time, and page execution time precisely at the millisecond level, forming a unified time-series dataset. By placing runtime data from different data sources under the same time benchmark, the chronological relationship between fault injection events and API anomaly events / page failure events can be accurately determined, providing a reliable observational evidence basis for subsequent causal inference.

[0030] S120 maps the target dependency graph to a directed acyclic graph structure of a Bayesian network; Among them, the target dependency graph represents the dependency relationships between services, interfaces, and pages. In the directed acyclic graph structure, the fault node is the parent node, the interface node is the intermediate node, the page node is the child node, and the edge direction represents the causal direction of fault propagation. When mapping the target dependency graph to a directed acyclic graph (DAG) structure, faulty nodes (i.e., fault injection points) can be set as parent nodes, and faults such as network latency and CPU overload can be injected into them. Affected interface nodes can be set as intermediate nodes, and ultimately affected page nodes can be set as child nodes. The edge direction of the DAG represents the causal direction of fault propagation, that is, the edge direction is from the faulty node to the interface node, and then from the interface node to the page node.

[0031] It's important to note that in the target dependency graph, the direction of dependency declarations is that the page depends on the interface, and the interface depends on the service. However, the causal direction of fault propagation is that the fault acts on the service, the service's exception propagates to the interface, and the interface's exception propagates to the page—these two directions are opposite. Therefore, when mapping the target dependency graph to a directed acyclic graph (DAG) structure, the edge directions must be set according to the causal propagation direction to ensure that the DAG structure accurately reflects the fault propagation path within the system.

[0032] In some embodiments, before mapping the target dependency graph to a directed acyclic graph structure of a Bayesian network, the method further includes: Parse historical pages and iterate through the test data to extract the data requests initiated by each page and establish the first mapping relationship from page nodes to interface nodes; obtain the service instance corresponding to the interface through the service registry center and / or parse the link tracing data of the server to establish the second mapping relationship from interface nodes to service nodes; construct the target dependency graph based on the first and second mapping relationships.

[0033] Specifically, see Figure 2 The target dependency graph is structurally represented as a three-tier dependency graph of pages, interfaces, and services, from top to bottom, comprising a page layer, an interface layer, and a service layer. The page layer includes page nodes, such as page A, page B, and page C, with horizontal directed connections between these nodes representing the user's business traversal order. The interface layer includes interface nodes, such as interface A, interface B, interface C, interface D, interface E, and interface F. The service layer includes service nodes, with horizontal directed connections between these nodes representing the service call chain. In terms of inter-layer mapping relationships, page → interface is a one-to-many dependency, meaning a page can depend on multiple interfaces; for example, both interface A and interface B are depended on by page A. Interface → service is a many-to-one mapping, meaning multiple interfaces can map to the same service node.

[0034] When establishing the first mapping relationship, historical page traversal test data can be parsed to extract all HTTP requests initiated by each page. For example, if parsing historical page traversal test data reveals that page A initiated requests to interface A and interface B, then a directed mapping relationship is established from page A to interface A and from page A to interface B, that is, the first mapping relationship from page node to interface node.

[0035] When establishing a second mapping relationship, the service instance corresponding to the interface can be obtained from the service registry, for example, by querying the list of service instances corresponding to interface A from the service registry; alternatively, the server's trace data (TraceID) can be parsed to reconstruct the service to which the interface call belongs, thereby establishing a second mapping relationship from the interface node to the service node. These two methods for establishing a second mapping relationship can be used individually or in combination. For example, when the information in the service registry is incomplete, trace data can be used as a supplementary source to make the mapping relationship more complete.

[0036] Furthermore, if the system lacks historical page traversal testing data, a fault-free full probing test can be performed during the test execution phase to capture page requests in real time, dynamically generate an initial dependency graph, and continuously revise this graph based on anomaly analysis results in subsequent tests. This allows the construction of the target dependency graph to be independent of the existence of historical data.

[0037] As an optional implementation, key interfaces can be automatically marked based on request timing: requests completed before the DOMContentLoaded event or initiated before the load event are marked as key interfaces, such as the product list interface; requests initiated after the load event are marked as non-key interfaces, such as recommendation interfaces and statistics interfaces. Simultaneously, page paths corresponding to core business processes can be marked as critical paths based on business importance and traffic volume, such as the "browse products, add to cart, checkout" path in an e-commerce system. During reliability testing, only page traversal and key interface probing on critical paths can be performed, skipping non-critical pages and non-key interfaces, thus shortening testing time while ensuring evaluation accuracy. This concentrates testing resources on the business links most critical to user experience, avoiding to some extent the problems of high resource consumption and low testing efficiency associated with full-scale probing.

[0038] In this way, constructing the target dependency graph as described above can improve the efficiency and error-prone nature of manually sorting out service dependencies, providing accurate topological input for the directed acyclic graph structure of Bayesian networks. Furthermore, since the target dependency graph records the mapping relationships between pages, interfaces, and services, it can also be used to identify critical paths and key interfaces, providing a screening basis for targeted probing and ensuring that subsequent causal inference and probing resource focus are both based on accurate dependencies.

[0039] S130, Calculate the marginal posterior probability of each fault relative to the observed anomalous evidence based on the operational data and the directed acyclic graph structure; In some embodiments, the marginal posterior probability of each fault relative to observed anomalous evidence is calculated based on operational data and the directed acyclic graph structure, including: Determine the prior probability of each fault; construct a first conditional probability table of interface nodes relative to fault nodes and a second conditional probability table of page nodes relative to interface nodes based on the interface call success rate, response latency distribution, and page load dependencies; when abnormal evidence is observed from the runtime data, determine the joint posterior probability of the fault combination state based on the prior probability of each fault node, the first conditional probability table, and the second conditional probability table; determine the marginal posterior probability of each fault based on the joint posterior probability.

[0040] Specifically, when determining the prior probability of each fault, each fault (i.e., fault node) can be set based on historical fault data or expert experience. The prior probability P( ).

[0041] Furthermore, for scenarios without historical data during the initial test, the prior probability of each fault can be initialized as a uniformly distributed probability P( )=1 / n, and Bayesian updates are performed based on historical observations in subsequent iterative tests.

[0042] Based on API call success rate, response latency distribution, and page load dependencies, the following conditional probability table is constructed: Interface Node Relative to its set of all possible faulty nodes First conditional probability table , which represents the conditional probability that the interface corresponding to the interface node will experience a timeout or error under any combination of fault states; Page Nodes Relative to its set of all interface nodes ( The second conditional probability table , which represents the conditional probability of page loading failure or exception under a combination of interface response states.

[0043] When the cloud testing task collects observation evidence E (such as abnormal events like interface timeout, error codes, and page loading failures), for each fault... According to Bayes' theorem, the joint posterior probability of the combination of fault states given evidence E is:

[0044] in, For joint posterior probabilities, Let be the prior probabilities of each fault. Indicates an injection failure. This indicates that no fault has been injected, and the denominator is a normalized constant over all possible combinations of fault states. The joint likelihood is given for all specific combinations of fault states, and this joint likelihood can be expressed by the first conditional probability table. Second conditional probability table It is calculated using the chain rule.

[0045] Furthermore, to quantify the independent contribution of each target fault in a concurrent scenario, the marginal posterior probability of the target fault is obtained by summing all possible states of other faults in the joint posterior probability:

[0046] in, This represents the specific combinations of values ​​for all faults except the i-th fault. The summation symbol represents the summation over all 2... n 1 The remaining fault combinations are accumulated. This marginal posterior probability... The fault is reflected after the observation of abnormal evidence. In a specific state (usually taking) This is the independent confidence level when a fault is injected. This calculation allows multiple concurrently injected faults to be ranked from highest to lowest posterior probability, serving as an important basis for root cause localization.

[0047] S140: Calculate the marginal causal effect of each fault on page metrics using the backdoor criterion, and identify the root cause fault from each fault based on the marginal posterior probability and marginal causal effect.

[0048] In some embodiments, the marginal causal effect of each fault on page metrics is calculated using the backdoor criterion, including: Each fault is sequentially designated as a target fault. The injection states of other faults besides the target faults are fixed. The first expected value of the target fault on the page metrics under the first preset condition and the second expected value of the target fault on the page metrics under the second preset condition are obtained respectively. Based on the first and second expected values, the marginal causal effect of the target fault is determined.

[0049] For scenarios involving concurrent injection of multiple faults, this method aims to accurately quantify individual target faults. The independent contribution can be calculated using the backdoor criterion.

[0050] Specifically, Indicates fixing the remaining faults Forced injection failure while keeping the injection state unchanged. ; This indicates that other faults are fixed in the same way. Cancel or close the fault while keeping the state unchanged. Based on this, the target fault is calculated. Marginal causal effect on page metrics Y (such as response time):

[0051] in, The marginal causal effect of the target fault on the page metric Y, The first expected value, The second expected value, Other combinations of fault states actually injected during testing.

[0052] It should be noted that this formula quantifies the marginal contribution of the target fault in the current concurrent environment. When there is a significant synergistic effect among faults, by traversing multiple tests of different fault subsets, the overall system effect is decomposed into main effects and interaction effects. The interaction effects are weighted and allocated according to the marginal contribution ratio of each fault, and finally, the independent contribution ranking of each fault is output. In this way, relevant but non-causal interference from other concurrent faults transmitted through shared services can be eliminated, achieving effect decoupling in multi-fault scenarios.

[0053] In some embodiments, the root cause fault is identified from among the faults based on the marginal posterior probability and the marginal causal effect, including: The root cause comprehensive score of each fault is determined based on the marginal posterior probability and marginal causal effect; the root cause fault is then identified from the faults based on the root cause comprehensive score.

[0054] Calculate each fault separately normalized value of marginal posterior probability Normalized value of marginal causality Normalized value of marginal posterior probability Normalized value of marginal causality The root cause score is obtained by weighted summation:

[0055] in, Root cause comprehensive score, Preset weighting coefficients are used to adjust the degree of preference between statistical confidence and causal truth (default values ​​are acceptable). =0.5).

[0056] Root cause comprehensive score The highest level of fault classification is root cause fault; In addition, if the root cause comprehensive score If the highest score is lower than the preset effective root cause threshold, the current system is determined to be abnormal without a clear root cause fault, and it is indicated that there may be uninjected external interference or insufficient observation data.

[0057] In some embodiments, after identifying the root cause fault from among the faults based on the marginal posterior probability and marginal causal effect, the method further includes: Generate a fault propagation path; and / or, Obtain at least one of the following: interface health, page flow completeness, and response time; determine the target comprehensive score based on at least one of these three scores; generate the test conclusion for the reliability test task based on the comparison between the target comprehensive score and a preset score threshold; and execute pipeline control operations based on the test conclusion and a preset blocking strategy.

[0058] Once the root cause of the fault is identified, the interface with the greatest causal effect can be selected as the critical fault interface, and the page with the greatest impact can be selected as the first affected page. The complete causal fault propagation chain from the root cause fault → critical fault interface → first affected page → subsequent cascading failure pages can be output.

[0059] In addition, at least one of the following can be obtained: interface health, page flow completeness, and response time. Each of these parameters is then scored. To better understand this implementation, the following example illustrates the acquisition of interface health, page flow completeness, and response time: Each of the following criteria—interface health, page flow completeness, and response time—is scored to obtain an interface score, a page score, and a response time score. Then, based on these scores, a target overall score is determined.

[0060] Where S is the overall target score. , , These are the weighting coefficients. The weights can be customized according to business needs.

[0061] Next, based on the comparison between the target comprehensive score and the preset scoring threshold, the test conclusion for the reliability test task is generated. For example, if the target comprehensive score is greater than or equal to 90 points, the test conclusion is "pass"; if the target comprehensive score is greater than or equal to 70 points but less than 90 points, the test conclusion is "warning"; if the target comprehensive score is less than 70 points, the test conclusion is "failure". Simultaneously, interpretable decision-making criteria are output, including the failure type, root cause interface, failure page, affected link, and affected scope, and a standard test report is generated.

[0062] After generating the test results, these results can be retrieved, and corresponding pipeline control operations can be executed according to the preset blocking strategy. The preset blocking strategy can be configured as follows: failures of core business paths will result in immediate blocking with a veto, such as immediately blocking the pipeline when the login path fails; failures of non-core paths will only generate alarms or reduce the scoring weight, without directly blocking the pipeline.

[0063] In some examples, when integrating with a Jenkins CI / CD (Continuous Integration / Continuous Delivery) pipeline, a reliability testing phase can be added to the Jenkins pipeline. After testing, the test results and overall score are sent back to Jenkins. If the score is greater than or equal to 90, the pipeline continues execution, allowing code merging and release. If the score is greater than or equal to 70 but less than 90, the pipeline issues a warning, notifying testers for manual review. If the score is less than 70, the pipeline automatically terminates, marks the build as failed, and sends a detailed test report to the relevant developers. Furthermore, defect tickets can be automatically created, with root causes and reproduction steps indicated, for subsequent secondary verification and comparison with historical tasks.

[0064] In this way, the evaluation results of reliability testing are no longer a simple pass or fail conclusion, but can include interpretable evidence from multiple dimensions such as fault type, root cause interface, failure page, impact chain, and impact scope. At the same time, the test results can be automatically integrated into the release pipeline to automatically verify the system's resilience before release, reduce the probability of defective code going live, and improve the problem of uninterpretable evaluation results and difficulty in realizing the value of testing in related technologies.

[0065] In some embodiments, the method further includes: Initialize fault parameters, including adjustment step size and indicator thresholds, including warning thresholds and failure thresholds; if the running data collected for a preset number of consecutive times meets the preset conditions, the fault intensity is increased by adjusting step size; if any collected running data exceeds the warning threshold, the fault intensity is decreased by adjusting step size; if any collected running data exceeds the failure threshold, the preset target operation is executed.

[0066] Specifically, when initializing fault parameters, in addition to adjusting the step size and metric thresholds, the initial fault strength and maximum fault strength can also be initialized. In a specific example, the initial fault strength is set to an initial network latency of 100ms, the adjustment step size is increased by 50ms each time, the maximum fault strength is the maximum network latency of 2000ms, and the metric thresholds include an interface error rate less than or equal to 5% and a page completion rate greater than or equal to 90%. The warning threshold can be set as the boundary corresponding to an interface error rate of 5% and a page completion rate of 90%, while the failure threshold can be set as a more lenient boundary than the warning threshold, such as the boundary corresponding to an interface error rate of 10% and a page completion rate of 80%.

[0067] During fault injection, real-time metrics from cloud testing are collected every 5 seconds as operational data. If the operational data collected for three consecutive times all meet the preset conditions, i.e., all metrics are within the warning threshold, the system is considered to be operating stably under the current fault intensity. The fault intensity is then increased by adjusting the step size, for example, increasing the network latency from 100ms to 150ms, to further probe the system's resilience boundary. If any collected operational data exceeds the warning threshold, for example, the interface error rate reaches 6%, the system is considered to be approaching its capacity limit. The fault intensity is then reduced by adjusting the step size to bring the system back to a stable operating range. If any collected operational data exceeds the failure threshold, for example, the page completion rate drops to 75%, the system is considered to be unable to provide normal service. The preset target operation is executed, i.e., the fault is immediately revoked, the test is terminated, and an alarm is triggered.

[0068] When the fault intensity increases to the maximum fault intensity, or when the indicator exceeds the warning threshold for the first time, the current fault intensity is recorded as the system's resilience boundary. In a specific example, if the network latency fault intensity gradually increases to 2000ms while all system indicators still meet the standards, 2000ms is recorded as the system's resilience boundary for that fault type; or if the interface error rate first exceeds the warning threshold when the fault intensity increases to 850ms, then 850ms is recorded as the resilience boundary. Recording the resilience boundary provides a quantitative basis for system capacity planning and fault recovery capability assessment.

[0069] In this way, the fault intensity can be adaptively adjusted according to the real-time performance of the system, achieving dynamic optimization between exposing hidden dangers and preventing system collapse. This improves the blindness of manually configuring fault intensity and allows the resilience boundary of the system to be quantitatively obtained.

[0070] According to the technical solution provided in this application, a target dependency graph containing dependencies between services, interfaces, and pages is constructed, and this target dependency graph is mapped to a directed acyclic graph structure of a Bayesian network. Based on the running data during cloud testing and the directed acyclic graph structure, the marginal posterior probability of each fault relative to observed abnormal evidence is calculated. Simultaneously, the marginal causal effect of each fault on page metrics is calculated using the backdoor criterion. Based on the marginal posterior probability and marginal causal effect, the root cause fault is identified from each fault. Since the edge direction of the directed acyclic graph structure represents the causal direction of fault propagation, the marginal posterior probability calculation of the Bayesian network can quantify the correlation strength between fault nodes and abnormal evidence. The intervention calculation of the backdoor criterion can eliminate mixed paths between multiple faults, thereby accurately separating the independent impact of each fault on page performance metrics in multi-fault concurrent scenarios, making the root cause localization result have both probabilistic and causal basis. Thus, a quantifiable causal chain from fault injection to interface anomaly to page failure is established, reducing the dependence of root cause localization on human experience and improving the accuracy and efficiency of root cause localization in multi-fault concurrent scenarios.

[0071] It should be understood that the order of the steps in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the process of the embodiments of this application.

[0072] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0073] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0074] Figure 3 This is a schematic diagram illustrating the structure of a root cause fault determination device according to an exemplary embodiment of this application. Figure 3 As shown, the exemplary root cause fault determination device includes: The task execution module 310 is configured to respond to fault concurrency injection when executing a reliability test task, trigger a cloud dial-up test task associated with the reliability test task, and collect runtime data when executing the cloud dial-up test task. The graph mapping module 320 is configured to map the target dependency graph into a directed acyclic graph structure of a Bayesian network. The target dependency graph represents the dependency relationship between services, interfaces and pages. In the directed acyclic graph structure, the fault node is the parent node, the interface node is the intermediate node, the page node is the child node, and the edge direction represents the causal direction of fault propagation. The probability calculation module 330 is configured to calculate the marginal posterior probability of each fault relative to the observed anomalous evidence based on the running data and the directed acyclic graph structure. The fault determination module 340 is configured to calculate the marginal causal effect of each fault on page metrics using the backdoor criterion, and to identify the root cause fault from each fault based on the marginal posterior probability and the marginal causal effect.

[0075] In some embodiments, the probability calculation module 330 is further configured to determine the prior probability of each fault; construct a first conditional probability table of interface nodes relative to fault nodes and a second conditional probability table of page nodes relative to interface nodes based on the interface call success rate, response latency distribution, and page load dependency; when abnormal evidence is observed from the running data, determine the joint posterior probability of the fault combination state based on the prior probability of each fault node, the first conditional probability table, and the second conditional probability table; and determine the marginal posterior probability of each fault based on the joint posterior probability.

[0076] In some embodiments, the fault determination module 340 is further configured to sequentially treat each fault as a target fault, fix the injection state of other faults besides the target fault, obtain the first expected value of the target fault on the page metric under the first preset condition, and the second expected value of the target fault on the page metric under the second preset condition; and determine the marginal causal effect of the target fault based on the first expected value and the second expected value.

[0077] In some embodiments, the fault determination module 340 is further configured to determine the root cause comprehensive score of each fault based on the marginal posterior probability and the marginal causal effect; and to determine the root cause fault from among the faults based on the root cause comprehensive score.

[0078] In some embodiments, the fault determination module 340 is further configured to, after determining the root cause fault from among the faults based on the marginal posterior probability and marginal causal effect, also include: Generate a fault propagation path; and / or, obtain at least one of interface health, page flow integrity, and response time; determine a target comprehensive score based on at least one of interface health, page flow integrity, and response time; generate test conclusions for the reliability test task based on the comparison between the target comprehensive score and a preset score threshold; and execute pipeline control operations based on the test conclusions and preset blocking strategies.

[0079] In some embodiments, the graph mapping module 320 is further configured to parse historical page traversal test data, extract data requests initiated by each page, and establish a first mapping relationship from page nodes to interface nodes; obtain the service instance corresponding to the interface through the service registry center, and / or parse the link tracing data of the server, and establish a second mapping relationship from interface nodes to service nodes; and construct a target dependency graph based on the first mapping relationship and the second mapping relationship.

[0080] In some embodiments, the map mapping module 320 is further configured to initialize fault parameters, including adjustment step size and indicator threshold, the indicator threshold including warning threshold and failure threshold; if the running data collected consecutively for a preset number of times all meet the preset conditions, the fault intensity is increased by adjustment step size; if any collected running data exceeds the warning threshold, the fault intensity is decreased by adjustment step size; if any collected running data exceeds the failure threshold, a preset target operation is executed.

[0081] Embodiments of this application also provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, which, when executed by one or more processors, cause the electronic device to implement the methods provided in the above embodiments.

[0082] Figure 4 A schematic diagram of a computer system suitable for implementing the embodiments of this application is shown. It should be noted that... Figure 4 The computer system 400 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0083] like Figure 4 As shown, the computer system 400 includes a Central Processing Unit (CPU) 401, which can perform various appropriate actions and processes based on a program stored in Read-Only Memory (ROM) 402 or a program loaded from Storage Section 408 into Random Access Memory (RAM) 403, such as performing the methods described in the above embodiments. The RAM 403 also stores various programs and data required for system operation. The CPU 401, ROM 402, and RAM 403 are interconnected via a bus 404. An Input / Output (I / O) interface 405 is also connected to the bus 404.

[0084] The following components are connected to I / O interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to I / O interface 405 as needed. Removable media 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 410 as needed so that computer programs read from them can be installed into storage section 408 as needed.

[0085] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by central processing unit (CPU) 401, it performs various functions defined in the system of this application.

[0086] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0087] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation that may be implemented in systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0088] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0089] Another aspect of this application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer's processor, causes the computer to perform the method as described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not assembled into the electronic device.

[0090] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods described in the various embodiments above.

[0091] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the steps of this application.

Claims

1. A method for determining root cause faults, characterized in that, include: In response to a concurrent fault injection during the execution of a reliability test task, a cloud dial-up test task associated with the reliability test task is triggered, and runtime data during the execution of the cloud dial-up test task is collected. The target dependency graph is mapped to a directed acyclic graph structure of a Bayesian network, wherein the target dependency graph represents the dependency relationship between services, interfaces and pages, and the fault node is the parent node, the interface node is the intermediate node, the page node is the child node, and the edge direction represents the causal direction of fault propagation. Based on the operational data and the directed acyclic graph structure, calculate the marginal posterior probability of each fault relative to the observed anomalous evidence; The marginal causal effect of each fault on page metrics is calculated using the backdoor criterion. Based on the marginal posterior probability and the marginal causal effect, the root cause fault is identified from each fault.

2. The method according to claim 1, characterized in that, The step of calculating the marginal posterior probability of each fault relative to the observed anomalous evidence based on the operational data and the directed acyclic graph structure includes: Determine the prior probability of each fault; Based on the interface call success rate, response latency distribution, and page loading dependency, a first conditional probability table of the interface node relative to the faulty node is constructed, and a second conditional probability table of the page node relative to the interface node is constructed. When abnormal evidence is observed from the operational data, the joint posterior probability of the fault combination state is determined based on the prior probability of each fault node, the first conditional probability table, and the second conditional probability table. Based on the joint posterior probability, the marginal posterior probability of each fault is determined.

3. The method according to claim 1, characterized in that, The calculation of the marginal causal effect of each fault on page metrics using the backdoor criterion includes: Each of the aforementioned faults is sequentially designated as a target fault. The injection status of other faults besides the target faults is fixed. The first expected value of the target faults on the page metrics under the first preset condition and the second expected value of the target faults on the page metrics under the second preset condition are obtained respectively. The marginal causal effect of the target fault is determined based on the first expected value and the second expected value.

4. The method according to claim 1, characterized in that, The step of identifying the root cause fault from among the faults based on the marginal posterior probability and the marginal causal effect includes: The root cause comprehensive score for each fault is determined based on the marginal posterior probability and the marginal causal effect. Based on the root cause comprehensive score, the root cause fault is determined from each of the faults.

5. The method according to claim 1 or 4, characterized in that, After determining the root cause fault from among the faults based on the marginal posterior probability and the marginal causal effect, the method further includes: Generate a fault propagation path; and / or, Obtain at least one of the following: API health, page flow completeness, and response time; The target comprehensive score is determined based on at least one of the following: interface health, page flow completeness, and response time. Based on the comparison between the target comprehensive score and the preset score threshold, the test conclusion of the reliability test task is generated; Execute pipeline control operations based on the test results and the preset blocking strategy.

6. The method according to claim 1, characterized in that, Before mapping the target dependency graph to a directed acyclic graph structure of a Bayesian network, the method further includes: Analyze the test data from the historical pages, extract the data requests initiated by each page, and establish the first mapping relationship between page nodes and interface nodes. Obtain the service instance corresponding to the interface through the service registry center, and / or parse the link tracing data of the server to establish a second mapping relationship from the interface node to the service node; The target dependency graph is constructed based on the first mapping relationship and the second mapping relationship.

7. The method according to claim 1, characterized in that, The method further includes: Initialize fault parameters, which include adjustment step size and indicator thresholds, including warning thresholds and failure thresholds; If the operating data collected for a preset number of consecutive times all meet the preset conditions, the fault intensity is increased according to the adjustment step size. If any of the collected operational data exceeds the warning threshold, the fault intensity is reduced according to the adjustment step size. If any of the collected running data exceeds the failure threshold, then a preset target operation is executed.

8. A root cause fault determination device, characterized in that, include: The task execution module is configured to respond to fault concurrency injection when executing a reliability test task, trigger a cloud dialing test task associated with the reliability test task, and collect the running data when executing the cloud dialing test task. The graph mapping module is configured to map the target dependency graph into a directed acyclic graph structure of a Bayesian network, wherein the target dependency graph represents the dependency relationship between services, interfaces and pages, and the directed acyclic graph structure has fault nodes as parent nodes, interface nodes as intermediate nodes, page nodes as child nodes, and edge directions representing the causal direction of fault propagation. The probability calculation module is configured to calculate the marginal posterior probability of each fault relative to the observed anomalous evidence based on the running data and the directed acyclic graph structure. The fault determination module is configured to calculate the marginal causal effect of each fault on page metrics using the backdoor criterion, and to determine the root cause fault from each fault based on the marginal posterior probability and the marginal causal effect.

9. An electronic device, characterized in that, include: One or more processors and a memory, the memory storing a computer program that, when executed by the one or more processors, causes the device to perform the steps of the root cause fault determination method as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that, It stores a computer program that, when executed by one or more processors, causes the device to perform the steps of the root cause fault determination method as described in any one of claims 1 to 7.

Images