Failure diagnosis method

Abstract

The invention relates to the technical field of computer software, and discloses a failure diagnosis method, which is suitable for a software automatic execution process, and comprises the following steps: when the software automatic execution process fails to execute, obtaining a page picture failed to execute; generating a fusion feature corresponding to the page picture; according to the execution action corresponding to the page picture, the execution result corresponding to the page picture and the fusion feature corresponding to the page picture, constructing a causal graph; determining an execution failure reason corresponding to the page picture according to the causal graph; and determining a recovery strategy according to the execution failure reason, and executing the recovery strategy so as to continue the software automatic execution process. Therefore, when the software automatic execution process fails to execute, the execution failure reason can be automatically diagnosed, and corresponding recovery measures are taken for recovery, so that normal execution of the process is ensured.

Description

Technical Field

[0001] This application relates to the field of computer software technology, and in particular to a method for diagnosing failures. Background Technology

[0002] Existing software automation technologies mainly fall into three categories: selector-based automation tools (such as Selenium and Playwright), image recognition-based tools (such as Sikuli), and AI-based tools (such as Browser-use). However, various anomalies often occur during the execution of automated software processes, leading to the failure of automated steps. Selector-based and image recognition-based tools do not automatically resume execution after failure, while AI-based tools typically attempt to find the cause of failure by forward-looking analysis of the failed steps. In practice, this approach has an extremely low success rate. Therefore, it is necessary to improve the accuracy of failure diagnosis and timely automatic recovery when automated software processes fail. Summary of the Invention

[0003] This application implements a failure diagnosis method that can automatically diagnose the cause of failure when the software automated execution process fails, and take corresponding recovery measures to ensure the normal execution of the process.

[0004] In a first aspect, embodiments of this application provide a failure diagnosis method applicable to automated software execution processes, the method comprising: Step 1: When the software automation process fails, obtain the image of the page where the execution failed; Step 2: Generate the fusion features corresponding to the page image; Step 3: Construct a cause-effect graph based on the execution action corresponding to the page image, the execution result corresponding to the page image, and the fusion feature corresponding to the page image; determine the reason for the execution failure corresponding to the page image based on the cause-effect graph; Step 4: Determine the recovery strategy based on the reason for the execution failure, and execute the recovery strategy to continue the software automation execution process.

[0005] Using the above method, when the software automation execution process fails, the system acquires the page image of the failed page; generates the corresponding fusion feature of the page image; constructs a cause-effect graph based on the execution action, execution result, and fusion feature of the page image; determines the cause of execution failure based on the cause-effect graph; determines a recovery strategy based on the cause of failure, and executes the recovery strategy to continue the software automation execution process. In this way, when the software automation execution process fails, the system can automatically diagnose the cause of failure and take corresponding recovery measures to ensure the normal execution of the process.

[0006] In one optional implementation, generating the fusion feature corresponding to the page image includes: determining the visual feature, the text feature, and the structural feature corresponding to the page image; and using an attention mechanism to fuse the visual feature, the text feature, and the structural feature corresponding to the page image to obtain the fusion feature corresponding to the page image.

[0007] Using the above method, when the software automation execution process fails, the page images of the page images that have failed are obtained from multiple perspectives. The multimodal features of the page images are then fused to obtain the fused features corresponding to the page images. This fully extracts the information from the page images and improves the accuracy of failure diagnosis.

[0008] In one optional implementation, a causal graph is constructed based on the execution action corresponding to the page image, the execution result corresponding to the page image, and the fusion feature corresponding to the page image. This includes: determining the execution intent corresponding to the page image; constructing node information corresponding to the execution intent, the execution action, the execution result, and the fusion feature in the causal graph; constructing first edge information between the node corresponding to the execution intent and the node corresponding to the execution action; constructing second edge information between the node corresponding to the execution action and the node corresponding to the execution result; and constructing third edge information between the node corresponding to the fusion feature and the node corresponding to the execution action, thereby obtaining the causal graph.

[0009] Using the above method, a cause-effect graph is constructed based on the execution action corresponding to the page image, the execution result corresponding to the page image, and the fusion feature corresponding to the page image. This allows for a full understanding of the execution intent in the page image corresponding to the execution failure, as well as the semantic information in the page image, further improving the accuracy of the cause-effect graph construction and providing a foundation for subsequent failure diagnosis.

[0010] In one optional implementation, determining the execution failure reason corresponding to the page image based on the cause-effect graph includes: determining a preliminary failure reason from a set of preset execution failure reasons based on the failure phenomenon in the execution result and the cause-effect graph; performing counterfactual reasoning verification on the preliminary failure reason; and if the verification passes, the preliminary failure reason is the execution failure reason corresponding to the page image.

[0011] By using the above method, the identified preliminary failure reasons are verified through counterfactual reasoning. The verified preliminary failure reasons are then identified as the reasons for the execution failure corresponding to the page images, thereby improving the accuracy of determining the failure reasons corresponding to the page images.

[0012] In one optional implementation, a preliminary failure cause is determined from a set of preset failure causes based on the failure phenomena in the execution results and the causal graph. This includes: using Bayesian reasoning based on the failure phenomena in the execution results, the causal graph, and any failure cause to calculate the probability of each failure cause; and selecting the failure cause with the highest probability among all failure causes as the preliminary failure cause.

[0013] In one optional implementation, the counterfactual reasoning verification of the initial failure cause includes: determining a preset posterior probability distribution corresponding to each execution failure cause based on the causal graph; determining an intervention measure corresponding to the initial failure cause; determining the effectiveness of the intervention measure; and determining the counterfactual probability of the counterfactual reasoning based on the posterior probability distribution and the effectiveness of the intervention measure, wherein the counterfactual probability is used to determine whether the verification passes.

[0014] By using the above method, the posterior probability distribution corresponding to each preset failure cause and the effectiveness of the intervention measures corresponding to the initial failure cause are determined based on the cause-effect graph. The counterfactual probability of the initial failure cause is determined, and the verification of the initial failure cause is determined based on the counterfactual probability, which further improves the accuracy of the verification.

[0015] In one optional implementation, determining the counterfactual probability of counterfactual reasoning based on the posterior probability distribution and the effectiveness of the intervention includes: determining the first effectiveness of the intervention from historical statistical data, determining the second effectiveness of the intervention from the causal graph, and determining the third effectiveness of the intervention from simulation results; and determining the counterfactual probability of counterfactual reasoning based on the posterior probability distribution, the first effectiveness, the second effectiveness, the third effectiveness, and the corresponding weights.

[0016] In one optional implementation, the step of determining a recovery strategy based on the execution failure reason and executing the recovery strategy to continue the software automation execution process includes: determining the execution failure reason corresponding to the failure phenomenon based on the failure phenomenon in the execution result, and determining a recovery strategy based on the execution failure reason corresponding to the failure phenomenon; if the counterfactual probability is less than a preset threshold, then executing the recovery strategy to continue the software automation execution process; if the counterfactual probability is greater than or equal to the preset threshold, then prioritizing the execution of the recovery strategy to continue the software automation execution process; if the recovery strategy fails, then determining the intervention measure corresponding to the initial failure reason as the recovery strategy, and executing the recovery strategy to continue the software automation execution process.

[0017] Using the above method, when the counterfactual probability is less than a preset threshold, it indicates that the determined preliminary cause of failure is unreliable. If the intervention measures corresponding to the preliminary cause of failure are directly applied for recovery, the success rate of recovery is low. Therefore, it is necessary to redetermine the cause of execution failure and the corresponding recovery strategy based on the phenomenon of execution failure, and then execute the recovery strategy to continue the automated software execution process, thereby improving the success rate of recovery. When the counterfactual probability is greater than or equal to the preset threshold, the determined preliminary cause of failure is reliable. However, this application determines the cause of failure from multiple perspectives and prioritizes executing the recovery strategy corresponding to the redetermined cause of execution failure based on the phenomenon of execution failure to continue the automated software execution process. If the recovery strategy fails, the intervention measures corresponding to the preliminary cause of failure are determined as the recovery strategy, and the recovery strategy is executed to continue the automated software execution process. This improves the success rate of recovery.

[0018] In one optional implementation, determining the execution failure cause corresponding to the failure phenomenon based on the failure phenomenon in the execution result, and determining the recovery strategy based on the execution failure cause corresponding to the failure phenomenon, includes: determining the probability distribution of multiple execution failure causes based on the failure phenomenon in the execution result and a historical case library; performing importance sampling on the probability distribution to determine the failure cause corresponding to the failure phenomenon in the execution result; determining the case set corresponding to the failure cause from the historical case library based on the failure cause; calculating the historical success rate of the recovery strategy corresponding to the case set, and determining the recovery strategy with the highest historical success rate as the target recovery strategy.

[0019] Using the above method, the failure phenomena in the execution results are matched with the historical case library, and the recovery strategy corresponding to the failure reason is determined based on the historical success rate of the recovery strategy in the set of successfully matched cases. By determining the recovery strategy with the highest historical success rate as the target recovery strategy, the software automated execution process is further continued by executing the target recovery strategy, thereby improving the success rate of recovery.

[0020] In one optional implementation, steps 2 to 4 are executed by a preset model in the software automated execution process; the loss function value of the preset model is obtained based on the multimodal understanding loss corresponding to step 2, the causal consistency loss corresponding to step 3, the recovery loss corresponding to step 4, and the corresponding weights. The multimodal understanding loss is determined based on the time taken to execute step 2, the causal consistency loss is determined based on the time taken to execute step 3, and the recovery loss is determined based on the time taken to execute step 4.

[0021] In one optional implementation, the loss function value further includes a complexity regularization term and a corresponding weight, wherein the complexity regularization term is determined based on the complexity of the preset model, the actual total inference time of the preset model, the target inference time, the actual memory usage, and the available memory.

[0022] Secondly, embodiments of this application provide a failure diagnosis device, the device comprising: The acquisition module is used to retrieve the page image of the failed execution when the software automation process fails. The generation module is used to generate the fusion features corresponding to the page image; The construction module is used to construct a cause-effect graph based on the execution action corresponding to the page image, the execution result corresponding to the page image, and the fusion feature corresponding to the page image. The determination module is used to determine the execution failure reason corresponding to the page image based on the cause-effect graph; The recovery module is used to determine a recovery strategy based on the reason for the execution failure, and execute the recovery strategy to continue the software automation execution process.

[0023] In one optional implementation, the generation module is specifically used to determine the visual features, text features, and structural features corresponding to the page image; and to fuse the visual features, text features, and structural features corresponding to the page image using an attention mechanism to obtain the fused features corresponding to the page image.

[0024] In one optional implementation, the construction module is specifically used to: determine the execution intent corresponding to the page image; construct node information corresponding to the execution intent, node information corresponding to the execution action, node information corresponding to the execution result, and node information corresponding to the fusion feature in the causal graph; construct first edge information between the node corresponding to the execution intent and the node corresponding to the execution action; construct second edge information between the node corresponding to the execution action and the node corresponding to the execution result; and construct third edge information between the node corresponding to the fusion feature and the node corresponding to the execution action, thereby obtaining the causal graph.

[0025] In one optional implementation, the determining module is specifically used to determine a preliminary failure reason from a set of preset execution failure reasons based on the failure phenomenon in the execution result and the cause-effect graph; to perform counterfactual reasoning verification on the preliminary failure reason; and if the verification passes, the preliminary failure reason is the execution failure reason corresponding to the page image.

[0026] In an optional implementation, the determining module is further configured to, based on the failure phenomenon in the execution result, the causal graph, and any execution failure cause, use Bayesian reasoning to calculate the probability of each execution failure cause; and select the execution failure cause with the highest probability among all execution failure causes as the preliminary failure cause.

[0027] In an optional implementation, the determining module is further configured to: determine the posterior probability distribution corresponding to each preset execution failure cause based on the causal graph; determine the intervention measures corresponding to the initial failure causes; determine the effectiveness of the intervention measures; and determine the counterfactual probability of counterfactual reasoning based on the posterior probability distribution and the effectiveness of the intervention measures, wherein the counterfactual probability is used to determine whether the verification passes.

[0028] In an optional implementation, the determining module is further configured to: determine a first effectiveness of the intervention from historical statistical data; determine a second effectiveness of the intervention from the causal graph; determine a third effectiveness of the intervention from simulation results; and determine the counterfactual probability of counterfactual reasoning based on the posterior probability distribution, the first effectiveness, the second effectiveness, the third effectiveness, and the corresponding weights.

[0029] In one optional implementation, the recovery module is specifically configured to: determine the execution failure cause corresponding to the failure phenomenon based on the failure phenomenon in the execution result, and determine a recovery strategy based on the execution failure cause corresponding to the failure phenomenon; if the counterfactual probability is less than a preset threshold, then execute the recovery strategy to continue the software automated execution process; if the counterfactual probability is greater than or equal to the preset threshold, then prioritize the execution of the recovery strategy to continue the software automated execution process; if the recovery strategy fails, then determine the intervention measures corresponding to the initial failure cause as the recovery strategy, execute the recovery strategy to continue the software automated execution process.

[0030] In an optional implementation, the recovery module is further configured to determine the probability distribution of multiple execution failure reasons based on the failure phenomena in the execution results and the historical case library; perform importance sampling on the probability distribution to determine the failure reasons corresponding to the failure phenomena in the execution results; determine the case set corresponding to the failure reasons from the historical case library based on the failure reasons; calculate the historical success rate of the recovery strategies corresponding to the case set, and determine the recovery strategy with the highest historical success rate as the target recovery strategy.

[0031] In one optional implementation, the steps of the generation module, the construction module, the determination module, and the recovery module are executed by a preset model in the software automated execution flow; The loss function value of the preset model is obtained based on the multimodal understanding loss corresponding to the generation module, the causal consistency loss corresponding to the construction module and the determination module, the recovery loss corresponding to the recovery module, and the corresponding weights. The multimodal understanding loss is determined based on the execution time of the generation module, the causal consistency loss is determined based on the execution time of the construction module and the determination module, and the recovery loss is determined based on the execution time of the recovery module.

[0032] In one optional implementation, the loss function value further includes a complexity regularization term and a corresponding weight, wherein the complexity regularization term is determined based on the complexity of the preset model, the actual total inference time of the preset model, the target inference time, the actual memory usage, and the available memory.

[0033] Thirdly, embodiments of the present invention provide a failure diagnosis device, the device comprising: a memory for storing a computer program; and a processor for executing the method described in the first aspect according to the obtained program when executing the computer program stored in the memory.

[0034] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, wherein when a computer reads and executes the computer program, the method described in the first aspect is performed.

[0035] Fifthly, embodiments of the present invention provide a computer program product that, when read and executed by a computer, causes the method described in the first aspect to be executed. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 A schematic diagram illustrating a scenario for a failure diagnosis method provided in an embodiment of this application; Figure 2 The flowchart is a method for diagnosing failures provided in an embodiment of this application. Figure 3 A schematic diagram illustrating a method for generating fusion features provided in an embodiment of this application; Figure 4 This is the flowchart corresponding to the construction of the cause-effect graph provided in the embodiments of this application; Figure 5 This is a schematic diagram illustrating the construction of a cause-effect graph provided in an embodiment of this application; Figure 6 A flowchart for verifying the counterfactual reasoning of the initial cause of failure provided in this application; Figure 7 A flowchart corresponding to another failure diagnosis method provided in the embodiments of this application; Figure 8 This is a schematic diagram of the structure of the failure diagnosis device provided in the embodiments of this application; Figure 9 This is a schematic diagram of the structure of a failure diagnosis device provided in an embodiment of this application. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0039] Based on the exemplary embodiments shown in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this application. Furthermore, although the disclosures in this application are presented by way of one or more exemplary examples, it should be understood that each aspect of these disclosures can constitute a complete technical solution on its own.

[0040] Before introducing the failure diagnosis method provided in the embodiments of this application, the background technology of the embodiments of this application will be introduced in detail for ease of understanding.

[0041] Existing software automation technologies mainly fall into three categories: selector-based automation tools (such as Selenium and Playwright), image recognition-based tools (such as Sikuli), and AI-based tools (such as Browser-use). Existing selector-based and image recognition-based automation tools cannot handle execution failures during the software automation process. AI-based tools, when encountering execution failures, typically work backwards from the failed step to find the cause of the failure. For example, if the failed step is the inability to find the "Add to Cart" button, the AI-based tool's goal is to find that button. Therefore, if the "Add to Cart" button is not found on the current page, the AI-based tool will search for it on other pages. Thus, even if the "Add to Cart" button is successfully clicked, the current failure is not actually resolved; the execution process is not properly resumed. Since AI-based tools search for the cause of failure by forward-looking steps based on the failure process, the failure diagnosis process is resource-intensive and usually takes a long time. In addition, because existing diagnostic methods lack semantic understanding, they cannot truly understand the user's operational intent, resulting in a low accuracy rate of failure diagnosis and thus failing to take accurate recovery strategies when execution fails.

[0042] Based on this, embodiments of this application provide a failure diagnosis method. When the automated execution process of a software fails, the method acquires the page image of the failed execution; generates a fusion feature corresponding to the page image; constructs a cause-effect graph based on the execution action, execution result, and fusion feature corresponding to the page image; determines the cause of execution failure corresponding to the page image based on the cause-effect graph; determines a recovery strategy based on the cause of execution failure, and executes the recovery strategy to continue the automated execution process of the software. In this way, when the automated execution process of a software fails, the cause of execution failure can be automatically diagnosed, and corresponding recovery measures can be taken to ensure the normal execution of the process.

[0043] The following is a brief introduction to the application scenarios to which the technical solutions of the embodiments of this application are applicable. It should be noted that the application scenarios described below are only for illustrating the embodiments of this application and are not intended to limit the scope. In specific implementation, the technical solutions provided by the embodiments of this application can be flexibly applied according to actual needs.

[0044] Figure 1 An exemplary schematic diagram of a failure diagnosis method provided in an embodiment of this application is shown, such as... Figure 1 As shown, this application scenario includes at least one terminal device 101 and at least one server 102. The terminal device 101 can be a mobile phone, tablet computer, laptop computer, desktop computer, etc., but is not limited to these. A failure diagnosis client can be installed on the terminal device 101. The client can be software (such as a browser, instant messaging software, etc.), or a webpage, mini-program, etc. In this embodiment, the user terminal device 101 can use the aforementioned failure diagnosis client to upload the page image of the execution failure obtained when the software automated execution process fails, and transmit the page image of the execution failure to the server 102 for analysis.

[0045] Furthermore, server 102 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. In this embodiment, the server 102 may be equipped with a failure diagnosis device corresponding to the client, used to process failure diagnosis tasks initiated by the user on the client related to the failure diagnosis method on the terminal device 101. According to the method provided in this embodiment, the page image that failed to execute is processed to obtain the reason for the failure and the corresponding recovery strategy, thereby improving the accuracy of failure diagnosis and the automated recovery process.

[0046] In some scenarios, terminal device 101 can access the network and communicate with server 102 through cellular mobile communication technology, which may include 5th generation mobile networks (5G) technology.

[0047] In other scenarios, terminal device 101 can access the network and communicate with server 102 via short-range wireless communication, which may include Wireless Fidelity (Wi-Fi) technology.

[0048] It should be noted that, Figure 1 The examples shown are merely illustrative; in reality, the number of terminal devices 101 and servers 102 is not limited and is not specifically limited in this embodiment.

[0049] To further illustrate the technical solutions provided in the embodiments of this application, a detailed description is provided below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiments of this application provide method operation steps as shown in the following embodiments or drawings, the method may include more or fewer operation steps based on conventional or non-inventive methods. In steps where there is no logically necessary causal relationship, the execution order of these steps is not limited to the execution order provided in the embodiments of this application. In actual processing or when the device executes the method, it may be executed in the order shown in the embodiments or drawings, or in combination.

[0050] Figure 2 This is a flowchart illustrating a failure diagnosis method provided in an embodiment of this application. This method is applicable to automated software execution processes, which can be executed by a failure diagnosis device, such as... Figure 2 The failure diagnosis device mounted on server 102 shown is used to improve the accuracy of failure diagnosis, such as... Figure 2 As shown, the process includes the following steps: Step 201: When the software automation execution process fails, obtain the page image of the failure.

[0051] For example, in an e-commerce website's shopping automation scenario, the automated process includes: searching for a specific product on Taobao, selecting the first search result, adding it to the shopping cart, and proceeding to checkout. If the "Add to Cart" button is obscured by dynamic content, the software's automated execution process will fail when attempting to add the product to the cart, and the corresponding page image will be retrieved.

[0052] Step 202: The failure diagnosis device generates the fusion features corresponding to the page image.

[0053] For example, the failure diagnosis device determines the visual features, text features, and structural features corresponding to the page image; it then uses an attention mechanism to fuse these features to obtain the fused features corresponding to the page image. Figure 3As shown, the failure diagnosis device can use CNN to extract image features from page images to determine the corresponding visual features; it can use BERT to extract text information from page images to determine the corresponding text features; and it can use GNN to extract structural information (such as a DOM tree) from page images to determine the corresponding structural features. Taking the example of the page image corresponding to the failure when the "Add to Cart" button is obscured by dynamic content, the structural information of the corresponding page includes the page structure of the dynamically obscured pop-up and the structure of the current shopping page. The dimensions of the determined visual features, text features, and structural features corresponding to the page image can be the same or different. For example, visual features can be 512-dimensional, text features 768-dimensional, and structural features 256-dimensional. After determining the visual features, text features, and structural features corresponding to the page image, the failure diagnosis device concatenates these features, uses an attention mechanism to determine their weights, and generates a fused feature based on the visual features, text features, and structural features of the page image and their corresponding weights.

[0054] Below is a code example for generating the fusion features corresponding to the page image: Determine visual features: def extract_visual_features(screenshot): """Using ResNet-50 to extract 512-dimensional visual features"" features = resnet50(screenshot) visual_vector = global_avg_pool(features) return visual_vector # shape: (512,) Identify text features: def extract_text_features(dom_text): """Using BERT to extract 768-dimensional text features"" text_embedding = bert_encoder(dom_text) text_vector = text_embedding[0] # [CLS] token return text_vector # shape: (768,) Determine structural features: def extract_structure_features(dom_tree): """Using GNN to extract 256-dimensional structural features""" graph = build_dom_graph(dom_tree) structure_vector = gcn_encoder(graph) return structure_vector # shape: (256,) Generate fusion features: def attention_fusion(visual, text, structure): Input: Feature vectors of three modalities; Output: Unified fused representation S(e) ∈ 256""" # Splicing features concat_features = concatenate([visual, text, structure]) # 1536 dimensions #Calculate attention weights attention_scores = attention_network(concat_features) attention_weights = softmax(attention_scores) # 3 weights, sum to 1 # Weighted fusion fused_representation = ( attention_weights[0] linear_v(visual) + attention_weights[1] linear_t(text) + attention_weights[2] linear_s(structure) #256 return fused_representation Step 203: The failure diagnosis device constructs a cause-effect graph based on the execution action corresponding to the page image, the execution result corresponding to the page image, and the fusion features corresponding to the page image; and determines the reason for the execution failure corresponding to the page image based on the cause-effect graph.

[0055] Figure 4 The flowchart corresponding to the construction of the cause-effect graph provided in the embodiments of this application is as follows: Figure 4 As shown, the process includes: Step 401: Determine the execution intent corresponding to the page image.

[0056] For example, taking the page image corresponding to the failure of the "Add to Cart" button being obscured by dynamic content as an example, the execution intent of the page image is to add the selected item to the cart.

[0057] Step 402: Construct the node information corresponding to the execution intent, the execution action, the execution result, and the fusion feature in the causal graph.

[0058] Step 403: Construct the first-side information between the node corresponding to the execution intent and the node corresponding to the execution action.

[0059] For example, such as Figure 5 As shown, the first edge information between the node corresponding to the execution intent and the node corresponding to the execution action is... Figure 5 The directed arrows between the nodes corresponding to the execution intent and the nodes corresponding to the execution actions indicate a strong causal relationship between them.

[0060] Step 404: Construct the second-side information between the node corresponding to the execution action and the node corresponding to the execution result.

[0061] For example, the second-side information between the node corresponding to the action and the node corresponding to the result is: Figure 5 The directed arrows between the nodes corresponding to the actions and the nodes corresponding to the results indicate a strong causal relationship between them.

[0062] Step 405: Construct the third-side information between the nodes corresponding to the fusion features and the nodes corresponding to the execution actions to obtain the causal graph.

[0063] For example, the third-side information between the node corresponding to the fused feature and the node corresponding to the executed action is: Figure 5The directed arrows between the nodes corresponding to the fused features and the nodes corresponding to the execution actions indicate a weak causal relationship between them.

[0064] For example, the constructed causal graph also includes nodes corresponding to failure causes, nodes corresponding to fusion features and nodes corresponding to failure causes have a weak causal relationship, and nodes corresponding to failure causes and nodes corresponding to execution results have a negative causal relationship.

[0065] The following is a code example for building a cause-effect graph: class CausalGraph: def __init__(self): self.nodes = {} # Causal nodes self.edges = {} # Causal edges def build_causal_model(self, action_sequence): "Construct a causal graph G=(V,E,Φ,Ψ), where V: a set of nodes {Intent, Action, Context, Result, Failure}; E: a set of directed edges representing causal relationships; Φ: node feature functions; Ψ: edge weight functions (causal strength)." for action in action_sequence: # Create a node intent_node = self.create_node("intent", action.intent) action_node = self.create_node("action", action) result_node = self.create_node("result", action.result) # Add causal edges self.add_edge(intent_node, action_node, weight=0.8) self.add_edge(action_node, result_node, weight=0.9) # Handle the impact of fusion features, where context is the fusion feature generated in step 202, and action.context is the action corresponding to the failure.

[0066] context_node = self.create_node("context", action.context)

[0067] self.add_edge(context_node, action_node, weight=0.6)

[0068] Furthermore, the failure diagnosis device determines the execution failure reason corresponding to the page image based on the cause-effect graph, including: determining the preliminary failure reason from the preset execution failure reasons based on the failure phenomenon in the execution result and the cause-effect graph; verifying the preliminary failure reason through counterfactual reasoning; if the verification passes, the preliminary failure reason is the execution failure reason corresponding to the page image.

[0069] Specifically, the failure diagnosis device uses Bayesian reasoning to calculate the probability of each failure reason based on the failure phenomena, the causal graph, and any failure cause in the execution results. The failure cause with the highest probability is selected as the initial failure cause. These failure causes include: element selector failure, page incomplete loading, element obscuration, insufficient permissions, network latency, etc. The Bayesian formula used in the Bayesian reasoning satisfies: ,in, For the i-th reason of failure, This refers to failures that occur during the execution process. As a feature of fusion, For the total probability, Let be the probability of the i-th reason for execution failure.

[0070] The following is a code example that uses Bayesian reasoning to determine the initial cause of failure: def causal_diagnosis(failure_event, causal_graph): "Using Bayesian inference to diagnose the reasons for failure" hypotheses = { H1: "Element selector is invalid". H2: "Page not fully loaded", "H3": "Element is obscured", H4: "Insufficient permissions", H5: "Network Latency" } posteriors = {} for h in hypotheses: likelihood = compute_likelihood(failure_event, h, causal_graph) prior = get_prior_probability(h) evidence = compute_evidence(failure_event) posteriors[h] = (likelihood prior) / evidence return max(posteriors, key=posteriors.get) Furthermore, the failure diagnosis device can perform counterfactual reasoning to verify the accuracy of the initial failure cause. The principle of counterfactual reasoning is to assume that a certain diagnostic result (i.e., the initial failure cause) is correct, then when we implement the corresponding intervention measures, the problem should be solved.

[0071] Figure 6 The flowchart provided in this application for verifying the counterfactual reasoning of the initial cause of failure is as follows: Figure 6 As shown, the process includes: Step 601: Determine the posterior probability distribution corresponding to each preset execution failure reason based on the cause-effect graph.

[0072] Step 602: Determine the intervention measures corresponding to the initial cause of failure.

[0073] Step 603: Determine the effectiveness of the intervention.

[0074] Step 604: Based on the posterior probability distribution and the effectiveness of the intervention measures, determine the counterfactual probability of the counterfactual reasoning. The counterfactual probability is used to determine whether the verification passes.

[0075] For example, the counterfactual probability of counterfactual reasoning is determined based on the posterior probability distribution and the effectiveness of the intervention measures, including: determining the first effectiveness of the intervention measures from historical statistical data, determining the second effectiveness of the intervention measures from causal graph reasoning, and determining the third effectiveness of the intervention measures from simulation results; and determining the counterfactual probability of counterfactual reasoning based on the posterior probability distribution, the first effectiveness, the second effectiveness, the third effectiveness, and the corresponding weights. When the counterfactual probability is greater than or equal to a preset threshold, it indicates that the counterfactual verification has passed, and the initial failure reason is determined to be the execution failure reason corresponding to the page image. The threshold can be set according to actual conditions; for example, it can be set to 0.8, which is not limited here. The counterfactual probability of counterfactual reasoning satisfies the following formula: ,in, For first validity, The weight corresponding to the first validity period. For second validity, The weight corresponding to the second validity period. For third validity, The weight corresponding to the third validity. Let be the probability corresponding to the i-th cause of failure in the posterior probability distribution.

[0076] The following is a code example for performing counterfactual diagnosis: # Constructing counterfactual scenarios counterfactual_action = generate_intervention(diagnosis) # Simulate execution predicted_outcome = simulate_intervention( counterfactual_action, causal_graph ) # Calculate the counterfactual probability success_probability = compute_counterfactual_probability( predicted_outcome, causal_graph ) return success_probability > 0.8 def generate_intervention(diagnosis): "Intervention measures are generated based on the diagnosis." interventions = { "Element selector invalid": {"action": "use_alternative_selector"}, "Page not fully loaded": {"action": "wait_for_element"}, "Element is obscured": {"action": "scroll_to_element"}, "Insufficient permissions": {"action": "request_permission"}, "Network latency": {"action": "retry_with_timeout"} } return interventions.get(diagnosis) The following is a code example for calculating the probability of an inverse event: Input: predicted_outcome: The predicted outcome object, containing the intervention and the expected state; causal_graph: A causal graph containing the causal relationships between nodes and edges; Output: success_probability: The counterfactual probability: [0,1] # Step 1: Obtain the posterior probability distribution from the cause-effect graph posterior_dist = causal_graph.get_posterior_distribution() # For example: {"H1": 0.35, "H2": 0.45, "H3": 0.15, "H4": 0.05} # Step 2: Determine the intervention measures intervention = predicted_outcome.intervention_action # Step 3: Calculate the weighted success probability counterfactual_prob = 0.0 for hypothesis, posterior_prob in posterior_dist.items(): # Check the effectiveness of this intervention for a specific cause. effectiveness = query_intervention_effectiveness( intervention = intervention, hypothesis = hypothesis data_source=["historical_db", "causal_graph", "simulation"] ) # Weighted summation counterfactual_prob += posterior_prob effectiveness return counterfactual_prob def query_intervention_effectiveness(intervention, hypothesis, data_source): """Query the effectiveness of the intervention, return: P(success|do(intervention), reason=hypothesis)""" #Historical data statistics (weight 0.5) historical_success_rate = historical_db.query( intervention = intervention, cause=hypothesis ) #Causal Graph Reasoning (Weight 0.3) causal_prediction = causal_graph.predict_intervention_effect( intervention = intervention, cause=hypothesis ) #Simulation (weight 0.2) simulation_result = simulator.run( intervention = intervention, initial_cause=hypothesis, n_trials=100 ) # Combining three types of effectiveness effectiveness = ( 0.5 historical_success_rate + 0.3 causal_prediction + 0.2 simulation_result.success_rate ) return effectiveness Step 204: The failure diagnosis device determines the recovery strategy based on the cause of the execution failure and executes the recovery strategy to continue the software automation execution process.

[0077] For example, when the counterfactual probability determined in step 203 is greater than or equal to a preset threshold, the intervention measures determined by counterfactual reasoning to verify the initial cause of failure can be identified as the corresponding recovery strategy, and the recovery strategy can be executed to continue the software automation execution process.

[0078] Steps 202-204 above are executed by a preset model in the software automated execution process. The loss function value of the preset model is obtained based on the multimodal understanding loss corresponding to step 202, the causal consistency loss corresponding to step 203, the recovery loss corresponding to step 204, and the corresponding weights. The multimodal understanding loss is determined based on the execution time of step 202, the causal consistency loss is determined based on the execution time of step 203, and the recovery loss is determined based on the execution time of step 204.

[0079] Optionally, the loss function value also includes a complexity regularization term and its corresponding weight. The complexity regularization term is determined based on the complexity of the preset model, the actual total inference time of the preset model, the target inference time, the actual memory usage, and the available memory. The loss function satisfies the following formula: ,in, For multimodal understanding loss, The weights corresponding to the multimodal understanding loss. For loss of causal consistency, The weights corresponding to the causal consistency loss. To recover the losses, To recover the weights corresponding to the loss, For complexity regularization, This represents the weight corresponding to the complexity regularization term. It should be noted that... , , , It can be set according to the actual situation. Set to 0.3, Set to 0.3, Set to 0.3, Set it to 0.1, without making any specific restrictions here.

[0080] For example, the complexity regularization term can be determined based on the complexity of the preset model, the complexity of the preset model's inference time, the complexity of its memory usage, and the corresponding weights. The complexity regularization term satisfies the following formula: ,in, To predetermine the complexity of the model, The weights are the weights corresponding to the predefined model complexity. To predetermine the complexity of the model's inference time, The weights are assigned to the model inference time complexity. For memory usage complexity, These are the weights corresponding to the memory usage complexity. It should be noted that... , , It can be set according to the actual situation. Set to 0.3, Set to 0.5. Set it to 0.2, without making any specific restrictions here.

[0081] The following sections will introduce the complexity of the preset model, the complexity of the preset model's inference time, and the complexity of its memory usage.

[0082] (1) Complexity of the preset model: The complexity of the preset model is determined based on the total number of parameters, baseline number of parameters, total number of floating-point operations, and baseline FLOPs involved in the preset model. The total number of parameters involved in the preset model is the sum of the parameters involved in CNN, BERT, GNN, and other layers. The baseline number of parameters can be set to 1 million as a benchmark. The total number of floating-point operations is the FLOPs of one forward propagation, and the baseline FLOPs can be 1 billion FLOPs. The complexity of the preset model satisfies the following formula: ,in, This refers to the total number of parameters involved in the preset model. For baseline parameter quantities, This represents the total number of floating-point operations. Baseline FLOPs.

[0083] (2) Complexity of the preset model inference time: Based on the actual total inference time and the target inference time of the preset model, the complexity of the preset model inference time is determined. The complexity of the preset model inference time satisfies the following formula: ,in, The actual total inference time of the preset model, Reasoning time for the target.

[0084] The following is a code example for calculating the complexity of the inference time of a preset model: def compute_inference_complexity(): """Calculate the time complexity of reasoning""" # Real-time measurement of the time consumed by each module timings = { "visual_features": 50, # Visual feature extraction 50ms "text_features": 80, # Text feature extraction takes 80ms "structure_features": 40, # Structural feature extraction 40ms "attention_fusion": 20, # Attention fusion 20ms "causal_diagnosis": 150, # Causal diagnosis 150ms "bayesian_inference": 300, # Bayesian inference 300ms "strategy_selection": 50, # Strategy selection time 50ms "overhead": 30 # Other overhead 30ms } T_actual = sum(timings.values()) # 720ms T_target = 600 # Target time: 600ms # Calculate the excess ratio R_inference = max(0, (T_actual - T_target) / T_target) # = max(0, (720 - 600) / 600) # = 0.20 (Exceeds 20%) return R_inference (3) Memory complexity: The memory complexity is determined based on the actual memory usage and available memory. The memory complexity satisfies the following formula: ,in, This represents the actual memory usage. Available memory.

[0085] Furthermore, it can be based on the loss function The value determines whether the preset model needs to be optimized. When the value is greater than or equal to the preset threshold, it indicates that the preset model needs to be optimized, therefore it is necessary to further determine which layers of the specific preset model need to be optimized; when If the result is less than the preset threshold, it means that the result of the preset model meets expectations and no optimization is needed.

[0086] Furthermore, when When the threshold is greater than or equal to a preset threshold, further optimization of the preset model is determined. A meta-learning mechanism can be used to identify the specific parts to be optimized. Specifically, the meta-information of the automated task execution, the data during the execution of the preset model, the data of the preset model execution results, performance indicators, and the context can be input into the meta-learning mechanism, which will output the data that needs to be updated for each part of the preset model. The meta-information of the automated task execution includes at least one of the following: a unique identifier corresponding to the task, the type of the task (e.g., e-commerce shopping, form filling), the domain name of the website (e.g., taobao.com), and the task execution timestamp. The data during the execution of the preset model includes: multimodal fusion layer data, causal layer data, and recovery layer data. The multimodal fusion layer data includes at least one of the following: visual features, text features, structural features, attention weights, and element localization accuracy. The causal layer data includes at least one of the following: the constructed causal graph, failure diagnosis results (preliminary failure reasons), diagnostic confidence, counterfactual probability, and logical consistency score. The recovery layer data includes: recovery time. The data for the pre-defined model execution results includes at least one of the following: whether the task was successful, the total time of the pre-defined model, the number of retries, whether a recovery layer was used, and the final sequence of actions executed. Performance metrics include at least one of the following: recovery accuracy, efficiency, robustness (number of successful recoverys / number of failures), and inference latency. The context includes at least one of the following: the structural complexity of page images, the degree of dynamic content, and network latency. The output of the meta-learning mechanism includes: updates to the multimodal fusion layer, the causal layer, the recovery layer, the loss function, and the parameters of the meta-learning model itself. Specifically, the multimodal fusion layer update includes updates to the weights corresponding to visual features, text features, and structural features. The causal layer update includes updates to the weights of the failure causes. The recovery layer update includes updates to the historical success rate corresponding to the failure causes. The loss function update includes: L... multimodal Corresponding weights, L causal Corresponding weights, L healing The corresponding weights and R complexity The corresponding weights are updated.

[0087] The following is a code example that uses a meta-learning mechanism to identify specific parts that need optimization: Input to the meta-learner: def __init__(self): # ===== Input 1: Meta-information for automated task execution ===== self.task_meta = { "task_id": "Unique identifier", "task_type": "Task type (e.g., e-commerce shopping, form filling)", "website_domain": "website domain (e.g., taobao.com)", "timestamp": "Execution timestamp" } # ===== Input 2: Data during the execution of the preset model ===== self.execution_trace = { #Multimodal fusion layer data "perception": { "visual_features": np.array([...]), # Visual features "text_features": np.array([...]), # Text features "structure_features": np.array([...]), # Structural features "attention_weights": [0.4, 0.35, 0.25], # Attention weights "element_location_accuracy": 0.92 # Element location accuracy }, # Causal layer data "causal": { "causal_graph": CausalGraph(...), # Causal graph structure "diagnosis_result": "Element is occluded", # Diagnosis failed "diagnosis_confidence": 0.85, # Diagnostic confidence level "counterfactual_prob": 0.78, # Counterfactual probability "logic_consistency_score": 0.95 # Logical consistency score }, # Recovery layer data "healing": { "recovery_strategy_used": "close_modal_then_retry", "recovery_success": True, "recovery_time_ms": 580, "alternative_strategies_tried": [ {"strategy": "wait_5s", "success": False}, {"strategy": "close_modal", "success": True} ] } } # ===== Input 3: Data of the preset model execution result ===== self.outcome = { "success": True, # Whether the task was successful "total_time_ms": 1850, # Total time elapsed "retry_count": 2, # Number of retries "used_self_healing": True, # Whether the recovery layer is used "final_action_sequence": [...] # The final sequence of actions to be executed } # ===== Input 4: Performance Metrics ===== self.performance_metrics = { "accuracy": 1.0, # Accuracy (success=1, failure=0) "efficiency": 0.72, #efficiency (target_time / actual_time) "robustness": 0.88, # Robustness (number of successful recoveries / number of failures) "latency": 0.58, # Inference delay (seconds) "resource_usage": { "cpu_percent": 45, "memory_mb": 3200, "gpu_utilization": 68 } } # ===== Input 5: Context ===== self.context = { "page_structure_complexity": 0.75, # Page structure complexity "dynamic_content_level": 0.6, # Dynamic content level "network_latency_ms": 120, # Network latency "browser_type": "chrome", "viewport_size": [1920, 1080] } Output of the meta-learner: class StrategyUpdate: def __init__(self): # ===== Output 1: Multimodal Fusion Layer Update ===== self.perception_updates = { "attention_weights_adjustment": { "visual": +0.02, # Increase visual feature weights "text": -0.01, # Reduce text feature weights "structure": -0.01 # Reduce the weight of structural features }, "feature_extractor_fine_tune": { "target_layer": "resnet50.layer4", "learning_rate": 0.0001, "update_direction": "gradient_vector" }, "update_location": "perception_layer.attention_network.weights" } # ===== Output 2: Causal Layer Update ===== self.causal_updates = { "causal_graph_structure": { "add_edge": ("Dynamic content loading", "Element visibility"), "edge_weight": 0.75, "reason": "This causal relationship was observed in 5 tasks." }, "prior_probability_adjustment": { "Element is obscured": 0.30, # Updated from 0.25 to 0.30 "Page not fully loaded": 0.28, # Updated from 0.30 to 0.28 "reason": "On taobao.com, elements being obscured is more common." }, "update_location": "causal_layer.causal_graph.priors" } # ===== Output 3: Recovery Layer Update ===== self.healing_updates = { "strategy_priority_reranking": { "Element is obscured": [ {"strategy": "close_modal", "priority": 1, "success_rate": 0.92}, {"strategy": "scroll_to_element", "priority": 2, "success_rate":0.75}, {"strategy": "wait_for_element", "priority": 3, "success_rate": 0.45} ] }, "new_strategy_generation": { "condition": "taobao.com + element is obscured", "strategy": "Close the pop-up window first, wait 1 second, then scroll to the element", "expected_success_rate": 0.95 }, "update_location": "healing_layer.strategy_db" } # ===== Output 4: Loss Function Update ===== self.global_controller_updates = { "loss_weight_adjustment": { L multimodal ": 0.30, # Remain unchanged L causal ": 0.31, # +0.01 (Good performance of the causal layer) L healing : 0.29, # -0.01 R complexity 0.10 # Remain unchanged }, "layer_credibility_scores": { "perception": 0.88, # Reliability of the multimodal fusion layer "causal": 0.92, # Causal layer credibility (highest) "healing": 0.85 #Healing layer reliability }, "update_location": "unified_controller.global_weights" } # ===== Output 5: Meta-learning model self-update===== self.meta_model_updates = { "performance_history_append": { "timestamp": "2025-11-14 10:23:15", "average_success_rate": 0.94, "average_latency": 0.58}, "meta_parameters_adjustment": { "learning_rate": 0.001, # Meta-learning rate "stability_threshold": 0.95, "rollback_buffer_size": 10 }, "update_location": "meta_learning_module.parameters" } Using the above method, when the software automation execution process fails, the system acquires the page image of the failed page; generates the corresponding fusion feature of the page image; constructs a cause-effect graph based on the execution action, execution result, and fusion feature of the page image; determines the cause of execution failure based on the cause-effect graph; determines a recovery strategy based on the cause of failure, and executes the recovery strategy to continue the software automation execution process. In this way, when the software automation execution process fails, the system can automatically diagnose the cause of failure and take corresponding recovery measures to ensure the normal execution of the process.

[0088] In one alternative implementation, Figure 7 The flowchart corresponding to another failure diagnosis method provided in the embodiments of this application is as follows: Figure 7 As shown, the process includes the following steps: Step 701: When the software automation execution process fails, obtain the page image of the execution failure.

[0089] Step 702: Generate the fusion features corresponding to the page images.

[0090] Step 703: Construct a cause-effect graph based on the execution action corresponding to the page image, the execution result corresponding to the page image, and the fusion feature corresponding to the page image; determine the preliminary execution failure reason corresponding to the page image based on the cause-effect graph.

[0091] Step 704: Verify the reasons for the initial execution failure by counterfactual reasoning and determine the counterfactual probability.

[0092] Step 705: Based on the failure phenomena in the execution results, determine the execution failure reasons corresponding to the failure phenomena, and determine the recovery strategy based on the execution failure reasons corresponding to the failure phenomena.

[0093] For example, the failure diagnosis device determines the probability distribution of multiple execution failure reasons based on the failure phenomena in the execution results and a historical case library; it performs importance sampling on the probability distribution to determine the execution failure reasons corresponding to the failure phenomena in the execution results; based on the execution failure reasons, it determines the case set corresponding to the execution failure reasons from the historical case library; it calculates the historical success rate of the recovery strategies corresponding to the case set, and determines the recovery strategy with the highest historical success rate as the target recovery strategy. The historical case library includes multiple successful cases and multiple failed cases. The failure diagnosis device determines the case set corresponding to the execution failure reasons from the historical case library, and the case set corresponding to the execution failure reasons includes multiple matching successful cases and multiple matching failed cases.

[0094] Furthermore, if the counterfactual probability determined in step 703 is less than a preset threshold, it indicates that the initial failure diagnosis is unreliable. Therefore, the recovery strategy corresponding to the failure determination in step 704 needs to be executed to continue the automated software execution process. If the counterfactual probability determined in step 703 is greater than or equal to a preset threshold, the recovery strategy corresponding to the failure determination in step 704 can be executed first to continue the automated software execution process. If the recovery strategy fails, the intervention measures corresponding to the initial failure determination in step 703 are determined as the recovery strategy, and the recovery strategy corresponding to the initial failure determination is executed to continue the automated software execution process.

[0095] Based on the same technical concept, embodiments of this application provide a failure diagnosis device 8000. Figure 8 This is a schematic diagram of the structure of the failure diagnosis device provided in the embodiments of this application, as shown below. Figure 8 As shown, the device 8000 includes: The acquisition module 801 is used to acquire the page image of the failed execution when the software automation execution process fails. Generation module 802 is used to generate the fusion features corresponding to the page image; Construction module 803 is used to construct a cause-effect graph based on the execution action corresponding to the page image, the execution result corresponding to the page image, and the fusion feature corresponding to the page image; The determination module 804 is used to determine the execution failure reason corresponding to the page image based on the cause-effect graph; The recovery module 805 is used to determine a recovery strategy based on the reason for the execution failure, and execute the recovery strategy to continue the software automation execution process.

[0096] In one optional implementation, the generation module 802 is specifically used to determine the visual features, text features, and structural features corresponding to the page image; and to fuse the visual features, text features, and structural features corresponding to the page image using an attention mechanism to obtain the fused features corresponding to the page image.

[0097] In one optional implementation, the construction module 803 is specifically used to: determine the execution intent corresponding to the page image; construct node information corresponding to the execution intent, node information corresponding to the execution action, node information corresponding to the execution result, and node information corresponding to the fusion feature in the causal graph; construct first edge information between the node corresponding to the execution intent and the node corresponding to the execution action; construct second edge information between the node corresponding to the execution action and the node corresponding to the execution result; and construct third edge information between the node corresponding to the fusion feature and the node corresponding to the execution action, thereby obtaining the causal graph.

[0098] In one optional implementation, the determining module 804 is specifically used to determine a preliminary failure reason from a set of preset execution failure reasons based on the failure phenomenon in the execution result and the cause-effect graph; to perform counterfactual reasoning verification on the preliminary failure reason; and if the verification passes, the preliminary failure reason is the execution failure reason corresponding to the page image.

[0099] In an optional implementation, the determining module 804 is further configured to, based on the failure phenomenon in the execution result, the causal graph, and any execution failure cause, use Bayesian reasoning to calculate the probability of each execution failure cause; and select the execution failure cause with the highest probability among all execution failure causes as the preliminary failure cause.

[0100] In an optional implementation, the determining module 804 is further configured to: determine the posterior probability distribution corresponding to each preset execution failure cause based on the causal graph; determine the intervention measures corresponding to the initial failure causes; determine the effectiveness of the intervention measures; and determine the counterfactual probability of counterfactual reasoning based on the posterior probability distribution and the effectiveness of the intervention measures, wherein the counterfactual probability is used to determine whether the verification passes.

[0101] In an optional implementation, the determining module 804 is further configured to: determine the first effectiveness of the intervention from historical statistical data; determine the second effectiveness of the intervention from the causal graph; determine the third effectiveness of the intervention from simulation results; and determine the counterfactual probability of counterfactual reasoning based on the posterior probability distribution, the first effectiveness, the second effectiveness, the third effectiveness, and the corresponding weights.

[0102] In one optional implementation, the recovery module 805 is specifically configured to: determine the execution failure cause corresponding to the failure phenomenon based on the failure phenomenon in the execution result, and determine a recovery strategy based on the execution failure cause corresponding to the failure phenomenon; if the counterfactual probability is less than a preset threshold, then execute the recovery strategy to continue the software automated execution process; if the counterfactual probability is greater than or equal to the preset threshold, then prioritize the execution of the recovery strategy to continue the software automated execution process; if the recovery strategy fails, then determine the intervention measures corresponding to the initial failure cause as the recovery strategy, execute the recovery strategy to continue the software automated execution process.

[0103] In an optional implementation, the recovery module 805 is further configured to determine the probability distribution of multiple execution failure reasons based on the failure phenomena in the execution results and the historical case library; perform importance sampling on the probability distribution to determine the failure reasons corresponding to the failure phenomena in the execution results; determine the case set corresponding to the failure reasons from the historical case library based on the failure reasons; calculate the historical success rate of the recovery strategies corresponding to the case set, and determine the recovery strategy with the highest historical success rate as the target recovery strategy.

[0104] In an optional implementation, the steps of the generation module 802, the construction module 803, the determination module 804, and the recovery module 805 are executed by a preset model in the software automated execution process; The loss function value of the preset model is obtained based on the multimodal understanding loss corresponding to the generation module 802, the causal consistency loss corresponding to the construction module 803 and the determination module 804, the recovery loss corresponding to the recovery module 805, and the corresponding weights. The multimodal understanding loss is determined based on the execution time of the generation module 802, the causal consistency loss is determined based on the execution time of the construction module 803 and the determination module 804, and the recovery loss is determined based on the execution time of the recovery module 805.

[0105] In one optional implementation, the loss function value further includes a complexity regularization term and a corresponding weight, wherein the complexity regularization term is determined based on the complexity of the preset model, the actual total inference time of the preset model, the target inference time, the actual memory usage, and the available memory.

[0106] Based on the same technical concept, embodiments of this application provide a failure diagnosis device 9000. Figure 9 This is a schematic diagram of the structure of a failure diagnosis device 9000 provided in an embodiment of this application, as shown below. Figure 9 As shown, the device 9000 includes at least one processor 901 and a memory 902 connected to the at least one processor 901. In this embodiment, the specific connection medium between the processor 901 and the memory 902 is not limited. Figure 9 Taking the connection between processor 901 and memory 902 via a bus as an example, the bus can be divided into address bus, data bus, control bus, etc. In this embodiment of the invention, memory 902 stores instructions that can be executed by at least one processor 901. By executing the instructions stored in memory 902, at least one processor 901 can implement the steps of the above-mentioned failure diagnosis method.

[0107] The processor 901 is the control center of the computer device, capable of connecting various parts of the device via various interfaces and lines. It performs resource configuration by running or executing instructions stored in the memory 902 and accessing data stored in the memory 902. Optionally, the processor 901 may include one or more processing units. The processor 901 may integrate an application processor and a modem processor. The application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may not be integrated into the processor 901. In some embodiments, the processor 901 and the memory 902 may be implemented on the same chip; in other embodiments, they may be implemented on separate chips.

[0108] The processor 901 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0109] Memory 902, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 902 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 902 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. In the embodiments of this application, memory 902 can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0110] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium. The computer program product includes computer program code, which, when executed on a computer, causes the computer to perform any of the failure diagnosis methods discussed above. Since the principle by which the above-described computer-readable storage medium solves the problem is similar to that of the failure diagnosis method, the implementation of the above-described computer-readable storage medium can be referred to the implementation of the method; repeated details will not be elaborated further.

[0111] Based on the same inventive concept, this application also provides a computer program product, which includes computer program code. When the computer program code is run on a computer, it causes the computer to execute any of the failure diagnosis methods discussed above. Since the principle of the above computer program product in solving the problem is similar to that of the failure diagnosis method, the implementation of the above computer program product can refer to the implementation of the method, and repeated details will not be described again.

[0112] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0113] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0114] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0115] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0116] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A failure diagnosis method characterized by comprising: The method is suitable for software automatic execution process, and the method comprises: Step 1: obtaining a page picture of execution failure when the software automatic execution process fails; Step 2: generating fusion features corresponding to the page picture; Step 3: constructing a causal diagram according to an execution action corresponding to the page picture, an execution result corresponding to the page picture and the fusion features corresponding to the page picture; determining an execution failure reason corresponding to the page picture according to the causal diagram; Step 4: determining a recovery strategy according to the execution failure reason, and executing the recovery strategy to continue the software automatic execution process.

2. The method of claim 1, wherein, The generating of the fusion features corresponding to the page picture comprises: determining visual features corresponding to the page picture, text features corresponding to the page picture and structure features corresponding to the page picture; fusing the visual features corresponding to the page picture, the text features corresponding to the page picture and the structure features corresponding to the page picture by using an attention mechanism to obtain the fusion features corresponding to the page picture.

3. The method of claim 1, wherein, The constructing of the causal diagram according to the execution action corresponding to the page picture, the execution result corresponding to the page picture and the fusion features corresponding to the page picture comprises: determining an execution intention corresponding to the page picture; constructing node information corresponding to the execution intention, node information corresponding to the execution action, node information corresponding to the execution result and node information corresponding to the fusion features in the causal diagram; constructing first edge information between a node corresponding to the execution intention and a node corresponding to the execution action; constructing second edge information between the node corresponding to the execution action and a node corresponding to the execution result; constructing third edge information between the node corresponding to the fusion features and the node corresponding to the execution action, thereby obtaining the causal diagram.

4. The method of claim 1, wherein, The determining of the execution failure reason corresponding to the page picture according to the causal diagram comprises: determining a preliminary failure reason from preset execution failure reasons according to a failure phenomenon in the execution result and the causal diagram; performing counterfactual reasoning verification on the preliminary failure reason, and if the verification is passed, the preliminary failure reason is the execution failure reason corresponding to the page picture.

5. The method of claim 4, wherein, The determining of the preliminary failure reason from preset execution failure reasons according to a failure phenomenon in the execution result and the causal diagram comprises: performing reasoning by using Bayes according to the failure phenomenon in the execution result, the causal diagram and any execution failure reason to calculate a probability of each execution failure reason; taking an execution failure reason with the largest probability in the execution failure reasons as the preliminary failure reason.

6. The method of claim 4, wherein, The counterfactual reasoning verification on the preliminary failure reason comprises: determining a posterior probability distribution corresponding to each execution failure reason according to the causal diagram; determining an intervention measure corresponding to the preliminary failure reason; determining effectiveness of the intervention measure; determining a counterfactual probability of counterfactual reasoning according to the posterior probability distribution and the effectiveness of the intervention measure, and the counterfactual probability is used to determine whether the verification is passed.

7. The method of claim 6, wherein, The step of determining the counterfactual probability of counterfactual reasoning according to the posterior probability distribution and the effectiveness of the intervention measure comprises: The first effectiveness of the intervention measure is determined from historical statistical data, the second effectiveness of the intervention measure is determined from the causal graph, and the third effectiveness of the intervention measure is determined from simulation results. The counterfactual probability of counterfactual reasoning is determined according to the posterior probability distribution, the first effectiveness, the second effectiveness, the third effectiveness, and corresponding weights.

8. The method of claim 6, wherein, The step of determining the recovery strategy according to the execution failure cause and executing the recovery strategy to continue the software automated execution process comprises: The execution failure cause corresponding to the failure phenomenon is determined according to the failure phenomenon in the execution result, and the recovery strategy is determined according to the execution failure cause corresponding to the failure phenomenon. If the counterfactual probability is less than a preset threshold, the recovery strategy is executed to continue the software automated execution process. If the counterfactual probability is greater than or equal to the preset threshold, the recovery strategy is preferentially executed to continue the software automated execution process, and if the recovery strategy fails to be executed, the intervention measure corresponding to the preliminary failure cause is determined as the recovery strategy, and the recovery strategy is executed to continue the software automated execution process.

9. The method of claim 8, wherein, The step of determining the recovery strategy according to the execution failure cause corresponding to the failure phenomenon in the execution result comprises: The probability distribution of multiple execution failure causes is determined according to the failure phenomenon in the execution result and a historical case library. The failure cause corresponding to the failure phenomenon in the execution result is determined by importance sampling of the probability distribution. The case set corresponding to the failure cause is determined from the historical case library according to the failure cause. The historical success rate of the corresponding recovery strategy in the case set is calculated, and the recovery strategy with the highest historical success rate is determined as the target recovery strategy.

10. The method according to any one of claims 1 to 9, characterized in that, The steps 2 to 4 are executed by a preset model in the software automated execution process. The loss function value of the preset model is obtained according to the multi-modal understanding loss corresponding to the step 2, the causal consistency loss corresponding to the step 3, the recovery loss corresponding to the step 4, and corresponding weights, the multi-modal understanding loss is determined according to the time consumption of executing the step 2, the causal consistency loss is determined according to the time consumption of executing the step 3, and the recovery loss is determined according to the time consumption of executing the step 4.

11. The method of claim 10, wherein, The loss function value further comprises a complexity regularization term and a corresponding weight, and the complexity regularization term is determined according to the complexity of the preset model, the actual total reasoning time of the preset model, the target reasoning time, the actual memory occupation, and the available memory.

Images