Fault processing system, method and equipment

Abstract

The embodiment of the invention discloses a fault processing system, method and equipment, and the method comprises the steps: obtaining service related information, which comprises business information, alarm source information and data source information; determining an agent scheduling strategy adapted to the service related information, wherein the agent scheduling strategy is used for guiding, scheduling and executing a target agent of the fault processing task; scheduling a target agent based on the agent scheduling strategy to analyze the service-related information, determining a fault event associated with the service-related information and a fault reason causing the fault event, and generating a repair strategy for the fault event based on the fault reason; and when it is monitored that the repair strategy is generated, the repair strategy is returned to the administrator, so that the administrator repairs the fault event according to the repair strategy, and the fault processing accuracy is improved.

Description

TECHNICAL FIELD

[0001] The present document relates to the technical field of Internet, and particularly relates to a fault processing system, method and device. BACKGROUND

[0002] With the deep integration of Internet into social operation, the scale of various businesses continues to develop, and the development team of the business also frequently changes, resulting in a higher and higher frequency of faults in the business scenario.

[0003] At present, after a fault occurs in the business scenario, an alarm is triggered, and then a manual troubleshooting method is used to locate and repair the fault. However, the manual effort is limited, and it is impossible to continuously and efficiently deal with the faults. Therefore, a technical solution is needed to more efficiently and continuously implement fault processing. SUMMARY

[0004] The purpose of the embodiments of the present specification is to provide a fault processing solution with more accurate processing results.

[0005] In order to achieve the above technical solutions, the embodiments of the present specification are implemented as follows: The fault processing system provided by the embodiments of the present specification comprises an access layer subsystem, a scheduling layer subsystem and a processing layer subsystem, wherein: a plurality of different access interfaces are arranged in the access layer subsystem, service related information is obtained through the plurality of different access interfaces, and the service related information comprises business related information, alarm source information and data source information; at least one scheduling unit is arranged in the scheduling layer subsystem to schedule corresponding resources; the at least one scheduling unit comprises an agent scheduling unit, which is configured to determine an agent scheduling strategy adapted to the service related information, and the agent scheduling strategy is used to guide the processing layer subsystem to schedule a target agent to execute a fault processing task; the processing layer subsystem comprises a main module and an auxiliary module, the auxiliary module is configured to assist the main module to execute a corresponding processing task, and the main module is configured to schedule a target agent based on the agent scheduling strategy, to analyze the service related information, determine a fault event associated with the service related information and a fault cause of the fault event, and generate a repair strategy for the fault event based on the fault cause; and the auxiliary module is configured to return the repair strategy to an administrator when it is monitored that the repair strategy is generated, so that the administrator repairs the fault event according to the repair strategy.

[0006] The embodiment of the present specification provides a fault processing method, which comprises: acquiring service related information, wherein the service related information comprises business information, alarm source information and data source information; determining an agent scheduling strategy adapted to the service related information, wherein the agent scheduling strategy is used to guide a target agent to perform a fault processing task; scheduling the target agent based on the agent scheduling strategy, so as to analyze the service related information, determine a fault event associated with the service related information and a fault cause of the fault event, and generate a repair strategy for the fault event based on the fault cause; and returning the repair strategy to an administrator in the case that the repair strategy is generated, so that the administrator repairs the fault event according to the repair strategy.

[0007] The embodiment of the present specification provides a fault processing device, which comprises: a processor; and a memory arranged to store computer executable instructions, wherein the executable instructions, when executed, cause the processor to: acquire service related information, wherein the service related information comprises business information, alarm source information and data source information; determine an agent scheduling strategy adapted to the service related information, wherein the agent scheduling strategy is used to guide a target agent to perform a fault processing task; schedule the target agent based on the agent scheduling strategy, so as to analyze the service related information, determine a fault event associated with the service related information and a fault cause of the fault event, and generate a repair strategy for the fault event based on the fault cause; and return the repair strategy to an administrator in the case that the repair strategy is generated, so that the administrator repairs the fault event according to the repair strategy.

[0008] The embodiment of the present specification further provides a storage medium for storing computer executable instructions, wherein the executable instructions, when executed by a processor, implement the following processes: acquiring service related information, wherein the service related information comprises business information, alarm source information and data source information; determining an agent scheduling strategy adapted to the service related information, wherein the agent scheduling strategy is used to guide a target agent to perform a fault processing task; scheduling the target agent based on the agent scheduling strategy, so as to analyze the service related information, determine a fault event associated with the service related information and a fault cause of the fault event, and generate a repair strategy for the fault event based on the fault cause; and returning the repair strategy to an administrator in the case that the repair strategy is generated, so that the administrator repairs the fault event according to the repair strategy.

[0009] The embodiment of the present specification further provides a computer program product comprising a computer program which, when executed by a processor, implements the following process: obtaining service related information, wherein the service related information comprises business information, alarm source information and data source information; determining an agent scheduling strategy adapted to the service related information, wherein the agent scheduling strategy is used to guide a target agent to perform a fault processing task; scheduling the target agent based on the agent scheduling strategy, to analyze the service related information, determine a fault event associated with the service related information and a fault cause of the fault event, and generate a repair strategy for the fault event based on the fault cause; and returning the repair strategy to an administrator to make the administrator repair the fault event according to the repair strategy, in a case where it is monitored that the repair strategy is generated. BRIEF DESCRIPTION OF DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present specification or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments described in the present specification, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings; Figure 1 A structural schematic diagram of an implementation environment to which a fault processing system provided by an embodiment of the present specification is applied; Figure 2 A structural schematic diagram of a fault processing system provided by an embodiment of the present specification; Figure 3 A schematic diagram of a multi-level label list provided by another embodiment of the present specification; Figure 4 A structural schematic diagram of a knowledge graph provided by still another embodiment of the present specification; Figure 5 A schematic diagram of an online data global snapshot at a fault occurrence moment provided by still another embodiment of the present specification; Figure 6 A structural schematic diagram of another fault processing system provided by still another embodiment of the present specification; Figure 7 A flow schematic diagram of a fault processing method provided by still another embodiment of the present specification; Figure 8 A schematic diagram of a fault processing device provided by still another embodiment of the present specification. DETAILED DESCRIPTION

[0011] The embodiment of the present specification provides a fault processing system, method and device.

[0012] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0013] This specification provides an automated fault handling mechanism. As enterprise business scale continues to grow, the size of backend system clusters increases rapidly, and development teams and personnel also change continuously. Major security incidents frequently occur on major internet platforms. Furthermore, troubleshooting online stability issues is becoming increasingly difficult. This involves not only the fine-grained location of node topologies in mesh systems but also the challenges of long communication links and difficult document maintenance, making it difficult to effectively guarantee the effectiveness of fault handling.

[0014] Currently, the alarm thresholds for critical nodes in system services, instances, and clusters are typically configured by operations and maintenance (O&M) personnel. When anomalies are detected, alarms are triggered, notifying the O&M agent for handling. Upon receiving the alarm, the O&M agent collects all nodes with anomalies along the entire network call dependency topology, calculates similarity using vector matching algorithms based on historical similar issues, selects the answer to the approximate problem, and outputs the results according to the prompt word output format specifications. Combining the results of intelligent analysis with the answers to historical similar problems, the final location and troubleshooting result is generated and sent to IT O&M personnel. O&M personnel then manually repair the fault based on the analysis results provided by the O&M agent. However, human resources are limited, making it impossible to continuously and efficiently handle fault handling issues.

[0015] To address this, the present invention proposes a superior fault handling scheme. By acquiring service-related information, a smart agent scheduling strategy adapted to this information is determined. This strategy guides and schedules target smart agents to perform fault handling tasks. Based on the scheduling strategy, the target smart agents are scheduled to analyze the service-related information, identify the associated fault events and their causes, and generate repair strategies based on these causes. Upon detecting the generation of a repair strategy, it is returned to the administrator, enabling the administrator to repair the fault events according to the strategy. Therefore, this automated fault handling mechanism allows for efficient fault event location and timely fault repair. Specific details can be found in the following embodiments.

[0016] The fault handling system provided in one or more embodiments of this specification is applicable to the fault handling implementation environment. (Refer to...) Figure 1The implementation environment includes at least: Client 100 and server 200. Server 200 can be configured with a fault handling system, including: Client 100 runs on a terminal device, which can be a mobile phone, personal computer, tablet computer, e-book reader, wearable device, device for information interaction based on AR (Augmented Reality) and VR (Virtual Reality), and laptop computer, etc. Client 100 can be installed on the terminal device. Client 100 can be an application, a browser, or a subroutine embedded in an application.

[0017] Server 200 runs on a server, which can be one or more servers, a server cluster consisting of several servers, or a cloud server of a cloud computing platform. Server 200, server 200 application, or subroutines within the application can be installed on the server. The fault handling system can be integrated into server 200, or server 200 can perform corresponding operations through any one or more subsystems in the fault handling system.

[0018] In addition, it may include a database 300, which may be set in the server on which the server 200 runs or outside the server on which the server 200 runs. The database 300 may store fault events associated with the storage service information, repair strategies generated for the fault events, and other related information.

[0019] In this implementation environment, server 200 can obtain service-related information through client 100, determine the intelligent agent scheduling strategy that is compatible with the service-related information, and schedule the target intelligent agent to analyze the service-related information based on the intelligent agent scheduling strategy, determine the fault events associated with the service-related information and the fault causes of the fault events, generate repair strategies for the fault events based on the fault causes, store the determined fault events and repair strategies in database 300, and return the repair strategies to the administrator so that the administrator can repair the fault events according to the repair strategies.

[0020] This specification provides a fault handling system that can be installed in a designated electronic device via software. The electronic device can be a terminal device or a server. The terminal device can be a mobile terminal device such as a mobile phone or tablet, or a computer device such as a laptop or desktop computer, or an IoT device (specifically, a smartwatch, in-vehicle device, etc.). The server can be a single server or a server cluster consisting of multiple servers. The server can be a backend server for financial services or online shopping services, or a backend server for a specific application, and can be configured according to actual needs.

[0021] In practical applications, this fault handling system can identify and repair fault events occurring in various business scenarios based on service-related information. These different business scenarios may include payment, shopping, or risk control scenarios, and potential fault events within these scenarios may include information system failures, data leaks, platform service interruptions, algorithm malfunctions, and content security incidents. The specific business scenarios and fault events supported by the system can be set according to actual conditions; this specification does not limit this.

[0022] like Figure 2 As shown, the fault handling system 210, due to its multiple functions, can be divided into several subsystems based on its different functions and division of labor. The fault handling subsystem 210 may include an access layer subsystem 211, a scheduling layer subsystem 212, and a processing layer subsystem 213. Different subsystems can interact through a transmission protocol, such as HTTP or RCP. Specifically, the fault handling system may include the following: The access layer subsystem 211 is equipped with multiple different access interfaces. Service-related information is obtained through these interfaces, including business-related information, alarm source information, and data source information.

[0023] Among them, service-related information refers to the relevant information required to perform fault handling services; business-related information refers to various types of data directly related to business activities carried out in the business scenario, and business-related information can support business decisions; alarm source information refers to the relevant data of the original source that triggers the alarm, and alarm source information can provide key basis for fault location, root cause analysis and handling; data source information is the underlying data that the fault handling system relies on when handling faults, and it directly determines the accuracy and reliability of the processing results.

[0024] For example, business-related information may include the business scenario name, troubleshooting thought process, and domain; alarm source information may include upstream alarm sources and external customer complaints; and data source information may include change queries and code queries.

[0025] In implementation, the fault handling system 210 can establish communication connections with business systems and devices in various business scenarios. These business systems and devices can be used to support the operation of business functions in these scenarios. The access layer subsystem 211, as the input to the fault handling model 210, can obtain service-related information for various business scenarios through its access interface. The access interface can be triggered in several ways. For example, the access layer subsystem 211 can respond to processing instructions initiated by business systems or devices and obtain service-related information through the access interface; alternatively, the fault handling system 210 can perform data scanning on the business systems and devices connected to it and obtain service-related information through the access interface of the access layer subsystem 211.

[0026] In addition, the access layer subsystem 211 can be associated with a configuration interface. The triggering method of the access interface and the scope of service-related information acquisition can be flexibly set through the configuration interface. Furthermore, this embodiment also sets data permissions. When accessing information that exceeds the data permissions, it is necessary to submit the data to the platform administrator for data evaluation. For example, if the obtained data source information is a data service channel already supported by the fault handling system, the information access can be achieved by configuring it on the configuration interface; if the obtained data source information is a new type of data service channel, it is necessary to submit the access to the platform administrator for evaluation, and access is only allowed when the evaluation result shows that access is permitted.

[0027] Optionally, business-related information may also include observable information. Observable information aims to help connected scenarios quickly generate visual reports. For example, observable information may include intelligent analysis volume, response / location timeliness, recall / precision rate, etc. Introducing demand information can help the fault handling system analyze service-related information more accurately, thereby improving the accuracy of identifying fault events.

[0028] The scheduling layer subsystem 212 is provided with at least one scheduling unit, through which corresponding resources can be scheduled; the at least one scheduling unit in the scheduling layer subsystem 212 may include an intelligent agent scheduling unit, which is configured to determine an intelligent agent scheduling strategy adapted to service-related information, and the intelligent agent scheduling strategy is used to guide the processing layer subsystem 213 to schedule the target intelligent agent to perform the fault handling task.

[0029] The scheduling layer subsystem 212 is mainly used to schedule various resources required by the fault handling system 210 during fault handling. The number of scheduling units in the scheduling layer subsystem 212 can be flexibly set according to the actual situation. Different scheduling units can be used to schedule different types of resources, including intelligent agents, test cases, experience knowledge, etc.

[0030] An intelligent agent is an entity capable of perceiving its environment, making reasoning decisions based on knowledge, and autonomously executing actions to achieve a specific goal. From a simple automation rule to a highly complex large-scale language model, each can be considered an intelligent agent of varying complexity. The agent scheduling unit in the scheduling layer subsystem 212 can be used to determine the target intelligent agent for performing the fault handling task from among the multiple intelligent agents included in the processing layer subsystem 213. However, the agent scheduling unit does not directly call the target intelligent agent to perform the fault handling task; instead, it determines the target intelligent agent, and then the processing layer subsystem calls the target intelligent agent to perform the fault handling task.

[0031] In implementation, the agent scheduling unit can determine an agent scheduling strategy that matches the service-related information based on the business-related information and data source information contained in the service-related information. The target agent pointed to in the agent scheduling strategy is not static. During the analysis of service-related information by the processing layer subsystem, the agent scheduling unit can flexibly adjust the target agent used to perform fault handling tasks according to the actual processing situation of the processing layer subsystem, so that the processing layer subsystem calls different target agents to analyze service-related information accordingly.

[0032] The processing layer subsystem 213 includes a main module and an auxiliary module. The auxiliary module is configured to assist the main module in performing corresponding processing tasks. The main module is configured to schedule target intelligent agents based on the intelligent agent scheduling strategy to analyze service-related information, determine the fault events associated with the service-related information and the causes of the fault events, and generate repair strategies for the fault events based on the causes of the fault events. The auxiliary module is configured to return the repair strategy to the administrator when it detects that a repair strategy has been generated, so that the administrator can repair the fault events according to the repair strategy.

[0033] The main module is the core processing module in the fault handling system. It is primarily used to identify fault events and pinpoint their root causes. The auxiliary modules assist the main module's processing and optimize its results. An administrator refers to a person or device with management authority over business systems or devices that generate service-related information. Administrators can be managers of business systems, change operators, etc.

[0034] In implementation, a communication connection is established between the main module and the agent scheduling unit. The main module can schedule target agents according to the agent scheduling strategy issued by the agent scheduling unit, so as to use the target agents to analyze service-related information, determine the fault events associated with the service-related information and the causes of the fault events, and generate repair strategies for the fault events based on the fault causes. The auxiliary module can monitor the fault handling process of the main module, and when it detects the generation of repair strategies, it returns the repair strategies to the administrator so that the administrator can repair the fault events according to the repair strategies.

[0035] This specification provides a fault handling system, comprising an access layer subsystem, a scheduling layer subsystem, and a processing layer subsystem. The access layer subsystem has multiple different access interfaces to acquire service-related information, including business-related information, alarm source information, and data source information. The scheduling layer subsystem has at least one scheduling unit to schedule corresponding resources. An agent scheduling unit in the scheduling layer subsystem determines an agent scheduling strategy adapted to the service-related information. This agent scheduling strategy guides the processing layer subsystem to schedule target agents to perform fault handling tasks. The processing layer subsystem includes a main module and an auxiliary module. The auxiliary module assists the main module in performing corresponding processing tasks. The main module schedules target agents based on the agent scheduling strategy to analyze service-related information, determine the fault events associated with the service-related information and the causes of the fault events, and generate repair strategies for the fault events based on the causes. Upon detecting the generation of a repair strategy, the auxiliary module returns the repair strategy to the administrator so that the administrator can repair the fault events according to the repair strategy. In this way, multiple different intelligent agents are designed to analyze service-related information from multiple dimensions. This not only considers fault events at the business level, but also further refines the fault branches in various scenarios, which can effectively improve the accuracy of the main module in analyzing service-related information, thereby further improving the accuracy of fault handling.

[0036] In the above or following embodiments, the main module in the processing layer subsystem 213 may specifically include the following: The main module can include multiple agents, each associated with a hierarchical label. The hierarchical label characterizes the depth of the agent's analysis of service-related information. In other words, the main module contains a top-down, multi-layered framework, with each layer corresponding to a hierarchical label. At least one agent within each layer is associated with that layer's corresponding hierarchical label. For example, hierarchical labels could include route identification, rapid location, and deep location. This indicates that the main module contains a three-layered framework, with each layer containing at least one invokeable agent. Through these agents across the three layers, service-related information can be analyzed in a progressively deeper manner to identify the associated fault events.

[0037] It should be noted that not every service-related information is analyzed using agents at all levels. Once the fault event and cause associated with the service-related information are identified using an agent at a certain level, the process will not continue to the next level.

[0038] In practical applications, the agent scheduling unit can monitor the process by which the main module schedules the target agent to analyze service-related information. When it detects that the target agent cannot determine the cause of the fault event, it can re-determine an agent scheduling strategy that is compatible with the service-related information for the main module.

[0039] In implementation, the agent scheduling unit also contains a top-down multi-layered framework. The hierarchy of this framework corresponds to the hierarchy within the main module. Specifically, the agent scheduling unit includes a route identification layer, a rapid localization layer, and a deep localization layer. Each layer contains agent scheduling logic related to the business scenario. After obtaining service-related information, the agent scheduling unit can, based on this information and the agent scheduling logic in the route identification layer, formulate a scheduling strategy for agents in the main module whose hierarchical label is "route identification." In other words, it determines the target agent from among the agents in the main module whose hierarchical label is "route identification," and whether the agent is compatible with the service-related information is determined by the agent scheduling logic within the route identification layer of the agent scheduling unit.

[0040] Taking the aforementioned three-layer framework as an example, the agent scheduling unit can first determine an agent scheduling strategy with the hierarchical label of identification route for the main module. The main module then schedules target agents with the hierarchical label of identification route based on this agent scheduling strategy. The agent scheduling unit monitors the analysis process of service-related information by the target agents with the hierarchical label of identification route. When it detects that the target agents with the hierarchical label of identification route cannot determine the cause of the fault event, the agent scheduling unit determines an agent scheduling strategy with the hierarchical label of rapid localization for the main module. The main module then schedules target agents with the hierarchical label of rapid localization based on this agent scheduling strategy. Finally, the agent scheduling unit monitors the analysis process of service-related information by the target agents with the hierarchical label of rapid localization. When it detects that the target agents with the hierarchical label of rapid localization cannot determine the cause of the fault event, the agent scheduling unit determines an agent scheduling strategy with the hierarchical label of deep localization for the main module. The main module then schedules target agents with the hierarchical label of deep localization based on this agent scheduling strategy.

[0041] In practical applications, it is possible to set only one agent in the main module with the label of route identification. This is because the differences between different business scenarios are not significant during shallow analysis. Using the same agent to analyze service-related information can reduce the workload of scheduling agents while ensuring the accuracy of the analysis, thereby improving the analysis efficiency.

[0042] The main module uses hierarchical labels to identify target agents for routes and analyzes service-related information. Simultaneously, the agent scheduling unit monitors the analysis results of these target agents. When the agent scheduling unit detects that the cause of the fault for a target agent with a hierarchical label indicating a route cannot be determined, it iterates to the rapid localization layer based on the current target agent's analysis results. Using the agent scheduling logic in the rapid localization layer, it re-determines a target agent that is compatible with the service-related information. The hierarchical label of this determined target agent is "rapid localization."

[0043] Based on this, the main module uses hierarchical labels to quickly locate target agents and analyze service-related information. Simultaneously, the agent scheduling unit monitors the analysis results of these target agents. When the agent scheduling unit detects that the cause of the fault for a target agent with hierarchical labels that are quickly located cannot be determined, it can iterate to the deep localization layer based on the analysis results of the current target agent. Using the agent scheduling logic in the deep localization layer, it then re-determines a target agent that is compatible with the service-related information. The hierarchical label of the determined target agent at this point is the deep localization layer label.

[0044] At this point, when the target agent's hierarchical label is "deep localization," it indicates that the analysis has reached the last level. If the fault event associated with the service-related information still cannot be determined, the service-related information can be fed back to humans for analysis to determine the fault event associated with the service-related information.

[0045] In practical applications, the main module needs to perform a lot of calculations to determine the target intelligent agent from multiple intelligent agents. In this embodiment, the intelligent agent scheduling unit determines the intelligent agent scheduling strategy, and the main module then schedules the corresponding target intelligent agent according to the intelligent agent scheduling strategy. This saves the workload of the main module itself in determining the target intelligent agent, thereby improving the efficiency of fault handling.

[0046] Based on this, the processing layer subsystem 213 schedules target agents according to the agent scheduling strategy. When the hierarchical label of the target agent is identification route or rapid location, the target agent is configured to perform risk detection on service-related information. Risk detection of service-related information aims to determine whether the fault events associated with the current service-related information are valid fault events. There are various methods for risk detection, such as analyzing the periodic patterns of monitoring curves to determine whether the current service-related information is a valid fault event, or matching historical noise similarity to determine whether the current service-related information is a valid fault event, etc.

[0047] When the detection results indicate that there is no risk related to the service, it means that the fault event associated with the service-related information is an invalid fault event, and the fault handling process for the service-related information can be terminated. Invalid alarm events refer to fault events where the alarm threshold configuration is unreasonable or that have no impact on business operations.

[0048] When the detection results indicate that there is a risk related to the service, it means that the fault event associated with the service-related information is a valid fault event. This can be explained according to different hierarchical labels as follows: When the hierarchical label of the target agent is identification route, the target agent can be used to analyze service-related information, determine the fault events associated with the service-related information and the causes of the fault events, and, if the cause of the fault cannot be determined, the target agent can be scheduled from at least one agent with the hierarchical label of fast location based on the agent scheduling strategy.

[0049] In implementation, when the target agent for identifying routes cannot determine the cause of a failure, further in-depth analysis of service-related information can be performed. Based on service-related information, the route can be routed from the current target agent to a sub-domain, and the agent corresponding to the sub-domain that matches the service-related information can be selected as the target agent. Sub-domains can include "business problems," "system engineering problems," "algorithm model problems," etc. The agent corresponding to the sub-domain can quickly identify the failure event and locate the cause of the failure through three functional points: change matching, scene noise reduction list matching, and thought chain whitelist matching.

[0050] When the hierarchical label of the target agent is fast localization, the target agent can be used to analyze service-related information, determine the fault events associated with the service-related information and the causes of the fault events, and, if the cause of the fault cannot be determined, the target agent can be scheduled from at least one agent with the hierarchical label of deep localization based on the agent scheduling strategy.

[0051] In implementation, when the target intelligent agent for rapid location using hierarchical labels cannot determine the cause of the fault, further in-depth analysis of service-related information can be performed. Based on service-related information, routing can be performed from the current sub-domain to the sub-dimensions within that sub-domain, and the intelligent agent corresponding to the sub-dimension that matches the service-related information can be selected as the target intelligent agent. These sub-dimensions can include "financial risk control," "content security," and "code analysis," etc. The fault handling logic for intelligent agents corresponding to different sub-dimensions is not entirely the same. The specific handling logic can be flexibly set according to the actual dimension division; this specification does not limit this in the embodiments.

[0052] When the target agent's hierarchical label is deep localization, the target agent can be used to split the fault handling task for service-related information into multiple sub-tasks, and a sub-target agent can be assigned to each sub-task according to a preset task priority list. The sub-target agent is configured to respond to the call of the target agent, execute the sub-task assigned to it by the target agent, determine the fault event associated with the service-related information and the fault cause of the fault event, and generate a repair strategy for the fault event based on the fault cause.

[0053] In this context, the target intelligent agent for deep localization, tagged with a hierarchical label, can utilize the Supervisor pattern to analyze service-related information. The Supervisor pattern is an architectural design pattern that establishes a coordination mechanism (i.e., the supervisor intelligent agent) to enable multiple specialized intelligent agents (i.e., worker intelligent agents) to collaborate and solve complex problems. The target intelligent agent, determined by the main module based on the agent scheduling strategy, is the "supervisor intelligent agent" in the Supervisor pattern. This target intelligent agent spontaneously calls the corresponding sub-target intelligent agents (i.e., worker intelligent agents), and the logic of the target intelligent agent calling the sub-target intelligent agents is independent of the agent scheduling unit. The target intelligent agent assigns multiple sub-tasks, broken down from the fault handling task, to the corresponding sub-target intelligent agents. These multiple sub-target intelligent agents collaborate to identify the fault events associated with the service-related information and the causes of those fault events, and then generate repair strategies based on the fault causes.

[0054] In this way, the Supervisor model allows the target agent to autonomously schedule multiple sub-target agents to collaboratively execute fault handling tasks. By introducing an independent supervision mechanism or role, additional verification and intervention are performed on core tasks or decision-making processes, thereby effectively reducing misjudgments and improving the accuracy of the main module's output processing results.

[0055] In implementation, a task priority list can be pre-set in the main module. The hierarchical label indicates the target agent for deep localization. After breaking down service-related fault handling tasks into multiple sub-tasks, these sub-tasks can be assigned to the corresponding sub-target agents according to the task priority list. Furthermore, a review agent can be set up in the main module. This review agent can collect relevant information during the task processing and dynamically adjust the task priority list based on the collected information to ensure its real-time performance.

[0056] In this embodiment, the intelligent agent with the hierarchical label of identifying routes may include scene routing intelligent agents, etc.; based on further analysis of scene routing intelligent agents, the intelligent agent with the hierarchical label of fast positioning may include business fast positioning intelligent agents, engineering fast positioning intelligent agents, algorithm fast positioning intelligent agents, etc.

[0057] Further in-depth analysis is conducted based on the business-based rapid location intelligence agent. Intelligence agents tagged as "deep location" can include "fund risk control intelligence agent," "content security intelligence agent," and "digital review intelligence agent," among others. Specifically, the fund risk control intelligence agent can analyze whether the strategies triggering penalties exhibit batch aggregation, what the key characteristics affecting strategy results are, whether there are aggregations, abnormal mutations, or anomalies in the feature value range, and whether there are changes in the dependent storage or model indicators of abnormal features, in order to identify fault events and their causes, and recommend corresponding remediation strategies. If the cause of the fault event cannot be determined, the relevant service information can be fed back to human analysts for further analysis to determine the cause.

[0058] Further in-depth analysis based on the engineering rapid location intelligence agent can be performed. Intelligence agents tagged as "deep location" can include "system performance intelligence agent," "storage anomaly intelligence agent," and "code analysis intelligence agent," among others. The system performance intelligence agent can analyze container resource metrics, system metrics, thread stack / stack snapshots, and other data to determine the fault event and its cause, and recommend corresponding remediation strategies, aided by error stack code parsing. The storage anomaly intelligence agent can analyze the abnormal stack of the business system to identify the storage type, and then determine the fault event and its cause based on storage FAQ documentation. If the cause of the fault event cannot be determined, the relevant service information can be fed back to human analysts for further analysis to determine the cause.

[0059] Based on algorithms, intelligent agents are quickly located for further in-depth analysis. Agents categorized as "deep-level localization" agents can include "capacity analysis agents" and "code analysis agents." Capacity analysis agents can provide emergency capabilities such as one-click capacity expansion / restart / data center traffic switching, identifying fault events on-site by analyzing key performance indicators like CPU / GPU / memory / wait queues. Code analysis agents can locate corresponding model code and logs based on abnormal stack traces during model execution, and match code change records to investigate fault events and their causes. If the cause of the fault event cannot be determined, the relevant service information can be fed back to human analysts for further analysis to determine the cause.

[0060] In the above or following embodiments, the auxiliary modules in the processing layer subsystem may include inspection agents, optimization agents, and snapshot agents, etc., wherein the specific functions of each agent are as follows: The inspection agent is configured to asynchronously detect whether the repaired fault event meets the repair requirements; when the repair requirements are met, a repair completion notification is sent to the administrator; when the repair requirements are not met, an analysis notification is sent to the main module so that the main module can re-analyze the service-related information.

[0061] In implementation, the inspection agent can asynchronously record processing results during the fault handling process of the main module, and asynchronously check whether alarm indicators have returned to normal. When the alarm indicator is detected to have returned to normal, it can promptly record and send a repair completion notification to the administrator; when the alarm indicator is detected to have not returned to normal, it can provide the detection results to the main model in the analysis notification. The main module can respond to the analysis notification, re-analyze the service-related information to redetermine the fault event and its cause, and regenerate the repair strategy. The optimization agent is configured to generate optimization requirements based on the fault cause of the fault event. The optimization requirements can be used to characterize the improvement needs of the service code; for example, the optimization requirements may include the phenomenon of the fault event, the exception code stack details, etc.

[0062] Among them, the optimization requirements are used to guide code administrators to adjust service code. The code administrator can be a separately configured application for intelligent code generation. This application can intelligently generate service code based on the optimization requirements and use the generated service code to replace the original service code in order to change the original execution logic in the scenario of the fault event.

[0063] Furthermore, in this embodiment, an output scheduling unit can be set in the scheduling layer subsystem. The output scheduling unit can generate an optimization request form associated with service-related information based on the optimization requirements output by the intelligent agent in the auxiliary module. The code administrator will scan for the generation of optimization request forms in real time. When a newly generated optimization request form is detected, service code is intelligently generated based on the optimization requirements in the optimization request form, and the generated service code is used to replace the original service code to change the original execution logic in the scenario to which the fault event belongs.

[0064] In this way, by adjusting the service code to change the native logic of the scenario to which the fault event belongs, the risk can be shifted to the left, reducing the probability of the fault event occurring, thereby indirectly improving the accuracy of fault handling.

[0065] In addition, the output scheduling unit can also provide a visualization output function. The output scheduling unit can convert the processing results of the main module into a visualization form (such as telephone reminders, email sending, group cards, etc.) and provide the visualization output results to the administrator.

[0066] The snapshot agent is configured to collect and store the historical processing records of the main module. The historical processing records include information on multiple historical fault events. Each historical fault event includes the processing process and results of the main module for the historical fault event. The processing process may include the processing results of agents at different levels for service-related information, the data and resources on which the target agent analyzes service-related information, etc. The processing results may include the fault event and the cause of the fault event.

[0067] In practical applications, the scheduling layer subsystem 212 may also include an evaluation scheduling unit, which contains evaluation cases. The evaluation cases contain service-related information and corresponding fault labeling reasons.

[0068] The evaluation scheduling unit is configured to evaluate the fault causes determined by the main module based on the service-related information contained in the test cases and the corresponding marked fault causes, and generate a fault handling score for the service-related information. The fault handling score is used to guide the logical adjustment of the target agent in the main module.

[0069] like Figure 3 As shown, service-related information can be manually tagged based on a pre-defined multi-level tag list, allowing for the generation of evaluation test cases across multiple business scenarios. These test cases serve as scoring criteria to evaluate the processing results of the main module. Each test case includes service-related information, the corresponding fault events, and the marked fault causes. Essentially, based on the same service-related information, the marked fault causes are used as the scoring criteria to evaluate the fault causes identified by the main module, generating a fault handling score for the service-related information.

[0070] In this embodiment, by introducing manually labeled test cases, automatic evaluation of the main module can be achieved, negative samples can be continuously identified and optimized, the deterioration of indicators such as recall rate and performance can be detected in a timely manner, and the logic of the main module can be adjusted in a timely manner to ensure the accuracy of fault handling.

[0071] Based on this, a knowledge scheduling unit can be set up in the scheduling layer subsystem, and an additional module can be set up in the processing layer subsystem to assist the main module in fault handling through the knowledge scheduling unit and the additional module.

[0072] The knowledge scheduling unit can build reference samples for the main module based on the test cases obtained from the evaluation scheduling unit and the historical processing records obtained from the snapshot unit; the additional module can store the reference samples generated by the knowledge scheduling unit and support the main module to call the reference samples to guide the confirmation of fault events and fault causes.

[0073] The reference samples can be carried in two ways: a Retrieval-Augmented Generation (RAG) and a Knowledge Graph (KG). A Retrieval-Augmented Generation (RAG) provides a general framework for retrieving unstructured text fragments based on vector retrieval and is often used as an auxiliary product in the field of retrieval-enhanced generation to assist large language models in performing tasks. A Knowledge Graph is a structured semantic network that organizes and represents knowledge graphically, aiming to describe concepts, things, and their interrelationships in the real world through entities, attributes, and their relationships.

[0074] In implementation, the knowledge scheduling unit can cluster historical fault events recorded in the evaluation test cases and historical processing records to obtain multiple clusters. A preset event threshold is set. When the number of historical fault events in a cluster exceeds the threshold, it indicates that the fault events are multidimensional and related. In this case, the historical fault events in the cluster are represented by a knowledge graph, which serves as a reference sample for the main module. When the number of historical fault events in a cluster does not exceed the threshold, it indicates that the fault events are unidimensional and highly cohesive. In this case, the historical fault events in the cluster are represented by a knowledge base, which serves as a reference sample for the main module.

[0075] like Figure 4 As shown, a knowledge graph can contain alarm events (i.e., fault events), applications, alarm classifications, etc. The generation process of the knowledge graph can be as follows: analyze the thought chain of the analysis process and the processing results, aggregate historical fault events with the same alarm indicators, similar investigation conclusions, and close occurrence times, and abstract them into incidents. Common fault patterns, noise clues, root cause types, and key clues (such as anomaly stacks, clustered distributions, anomaly upstream and downstream, etc.) of multiple historical fault events can be extracted and incrementally accumulated into the knowledge graph around the incident center.

[0076] Based on this, the knowledge base and knowledge graph generated by the knowledge scheduling unit can be stored as reference samples in an additional module of the processing layer subsystem, so as to use the reference samples to feed back into the main module. The knowledge base includes business background knowledge (risk control strategy system, penalty action definition, risk control event system, etc.), special scenario investigation thinking chain, noise list and other single-dimensional and relatively independent static knowledge, supporting real-time dynamic retrieval by the model; the knowledge graph includes the related system of the incident, key features, key actions, related scenarios, knowledge of the overall security plan, link call knowledge, policy lineage knowledge, etc.; when the main module analyzes service-related information, it can call the reference samples in the additional module to guide the confirmation of the fault events associated with the service-related information, as well as the causes of the fault events.

[0077] In addition, the extra modules also include the Model Content Protocol (MCP) service. The MCP service is responsible for providing external dynamic functions that the main module analysis depends on, including various atomic service capabilities such as related monitoring data query, emergency action execution, code detail query, and change record query.

[0078] It should be noted that the fault handling system can also include a dependency layer subsystem. This subsystem serves as the infrastructure of the overall technical framework, providing the runtime environment framework for the agents within the processing layer subsystem, such as DeRisk, and all dynamic data source services during the main module's processing. These include data sources for monitoring metrics, emergency response processes, end-to-end trace tracking, code repositories, application / business change data, RAG-dependent document library data, and APIs related to security risk control engine strategies and features. Through these data source services, a global online data snapshot can be created at the moment of the fault, such as... Figure 5 As shown, this online global snapshot of data is the data foundation for accurately locating the cause of a fault.

[0079] like Figure 6 As shown in the embodiments of this specification, another fault handling system is provided. This fault handling system can be divided into four layers from top to bottom: a unified access layer, an intelligent scheduling layer, a model analysis layer, and an external dependency layer. Each layer is functionally cohesive, and the layers interact through HTTP / RPC service calls. The unified access layer includes four sub-dimensions: business access, alarm source access, dependent data access, and observable report access. The intelligent scheduling layer includes four parts: agent scheduling, knowledge scheduling, evaluation scheduling, and conclusion output scheduling. The model analysis layer includes the main model, auxiliary models, and model-related services. The external dependency layer provides the runtime environment framework (DeRisk) for the model analysis layer agent, as well as all dynamic data source services from model analysis. The main workflow of this fault handling system is as follows: Step A02: The unified access layer comprehensively collects service-related information from the access fault handling system, and the model dependency layer provides dynamic data source services. Step A04: Agent scheduling in the intelligent scheduling layer triggers first-layer agent analysis by integrating service-related information from the unified access layer. Step A06: Based on the runtime environment framework provided by the external dependency layer, the main model in the model analysis layer uses the first-layer agent (i.e., the identification routing layer agent) determined by the agent scheduling to analyze service-related information. When the first-layer agent determines that there is no risk, it directly performs intelligent noise reduction and ends the current analysis process. Step A08: The Agent in the intelligent scheduling layer collects the analysis results of the main model. When the analysis results show that the cause of the fault has not been located, the process is recursively iterated to the second-layer Agent. Step A10: If the first-layer agent in the main model determines that there is a risk and the cause of the failure cannot be located, the second-layer agent (i.e., the fast location layer) determined by the routing agent scheduling will be used to continue the analysis. When the second-layer agent determines that there is no risk, intelligent noise reduction will be performed directly to end the current analysis process. Step A12: The Agent in the intelligent scheduling layer collects the analysis results of the main model. When the analysis results show that the cause of the fault has not been located, the process is recursively iterated to the third layer Agent. Step A14: If the first-layer agent in the main model determines that there is a risk and cannot locate the cause of the failure, then the third-layer agent (i.e., the deep positioning layer) determined by the routing to the agent scheduling is used to continue the analysis. The third-layer agent locates the cause of the failure event through the Supervisor mode and generates corresponding repair suggestions. Step A16: The auxiliary model in the model analysis layer determines the cause type based on the fault cause analyzed by the main model. If the cause type is a non-code cause, a repair suggestion is returned to the administrator so that the administrator can repair the fault event according to the repair strategy. If the cause type is a code cause, optimization requirements are generated. Step A18: The output scheduling in the intelligent scheduling layer can generate an optimization requirement form associated with service-related information based on the optimization requirement content provided by the auxiliary model. The optimization requirement form is used to guide the code administrator to adjust the service code. In addition, the output scheduling can also perform a visualization transformation on the output results of the main model. Step A20: The evaluation scheduling in the intelligent scheduling layer automatically evaluates the analysis results of the main model based on pre-labeled evaluation test cases. The evaluation results can be used as the basis for making logical adjustments to the main model. Step A22: The auxiliary model in the model analysis layer asynchronously records the full index snapshot data during the main model analysis process, and asynchronously checks whether the alarm index has recovered to the normal level. If it has recovered, a notification is sent in a timely manner. Step A24: The knowledge scheduling in the intelligent scheduling layer generates experiential knowledge as a reference sample for the main model based on the evaluation cases provided by the evaluation scheduling and the full index snapshot data provided by the auxiliary model. The experiential knowledge is carried in the form of knowledge base (RAG) and knowledge graph (KG). Step A26: The model-supporting service in the model analysis layer stores the knowledge base and knowledge graph generated by knowledge scheduling, so as to provide reference samples in the subsequent analysis of the main model.

[0080] This specification provides a fault handling system that, in addition to system configuration dimensions such as services, instances, and clusters, introduces a business dimension, broadening the scope of fault handling and reducing the occurrence of missed issues. Furthermore, at each dimension, multiple agents are used for layer-by-layer analysis, expanding the depth of fault handling and thus effectively improving its accuracy. By utilizing auxiliary models and knowledge scheduling, experiential knowledge is integrated into a knowledge graph or knowledge base, allowing it to feed back into the main model. The system automatically evaluates the main model's processing results based on manually labeled test cases and corrects the main model's processing logic accordingly. This effectively ensures the timeliness and accuracy of the main model's processing logic, further improving fault handling accuracy. Additionally, this embodiment can adjust application code based on the main model's processing results, modifying the code to eliminate some fault causes due to code issues, shifting risk to the left, reducing the probability of fault occurrence, and thus improving fault handling accuracy from the source.

[0081] like Figure 7 As shown in the embodiments of this specification, a fault handling method is provided. The execution subject of this method can be a fault handling system, which includes an access layer subsystem, a scheduling layer subsystem, and a processing layer subsystem. The functions of each subsystem can be found in the relevant content of the above embodiments, and will not be repeated here. The fault handling system can be set in a terminal device or a server. The terminal device can be a mobile terminal device such as a mobile phone or tablet computer, or a computer device such as a laptop or desktop computer, or an IoT device (specifically, a smartwatch, in-vehicle device, etc.). The server can be a single server or a server cluster composed of multiple servers. The server can be a backend server for financial business or online shopping business, or a backend server for an application. This embodiment uses a server as the execution subject for detailed description. For the case where the execution subject is a terminal device, please refer to the following server case handling, which will not be repeated here. The method may specifically include the following steps: In step S702, service-related information is obtained, including business information, alarm source information, and data source information; In step S704, an agent scheduling strategy adapted to service-related information is determined. The agent scheduling strategy is used to guide the target agent to perform the fault handling task. In step S706, the target intelligent agent is scheduled based on the intelligent agent scheduling strategy to analyze the service-related information, determine the fault events associated with the service-related information and the fault causes of the fault events, and generate a repair strategy for the fault events based on the fault causes. In step S708, if a repair strategy is detected, the repair strategy is returned to the administrator so that the administrator can repair the fault event according to the repair strategy.

[0082] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the fault handling method embodiments are basically similar to the system embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the system embodiments.

[0083] This specification provides a fault handling method. It involves acquiring service-related information, including business information, alarm source information, and data source information; determining an agent scheduling strategy adapted to the service-related information, which guides the scheduling of target agents to perform fault handling tasks; scheduling the target agents based on the agent scheduling strategy to analyze the service-related information, identify the fault events associated with the service-related information and the causes of the fault events, and generating repair strategies for the fault events based on the fault causes; and returning the repair strategies to the administrator when they are detected, so that the administrator can repair the fault events according to the repair strategies. This method designs multiple different agents to analyze service-related information from multiple dimensions, considering not only fault events at the business level but also further refining fault branches in various scenarios. This effectively improves the accuracy of the main module's analysis of service-related information, thereby further improving the accuracy of fault handling.

[0084] The above describes the fault handling system provided in the embodiments of this specification. Based on the same idea, the embodiments of this specification also provide a fault handling device, such as... Figure 8 As shown.

[0085] The fault handling device can provide terminal equipment or servers, etc., for the above embodiments.

[0086] Fault handling devices can vary significantly in configuration and performance, and may include a communication interface 802, a user interface 804, a processor 806, and a data storage 808. These components are interconnected and communicate with each other via a system bus, network, or other connection mechanism 810. The communication interface 802 enables the fault handling device 800 to communicate with other devices, access networks, and transmission networks via analog or digital modulation. For example, the communication interface 802 may include a chipset and antenna for wireless communication with a radio access network or access point. Furthermore, the communication interface 802 can be a wired interface such as Ethernet, Token Ring, or a USB port, or a wireless interface such as Wi-Fi, Bluetooth, Global Positioning System (GPS), or a wide-area wireless interface (e.g., WiMAX or LTE). Of course, the communication interface 802 may also support other forms of physical layer interfaces and standard or proprietary communication protocols. The communication interface 802 may also include multiple physical communication interfaces, such as Wi-Fi, Bluetooth, and wide-area wireless interfaces.

[0087] User interface 804 includes receiving user input and providing output to the user. Therefore, user interface 804 may include input components such as a keypad, keyboard, touch-sensitive or presence-sensitive panel, computer mouse, trackball, joystick, microphone, still camera, and video camera, and output components such as a display screen (which may be combined with a touch-sensitive panel), CRT, LCD, LED, display using DLP technology, printer, and other similar devices known or developed in the future. User interface 804 may also generate auditory output via speakers, speaker jacks, audio output ports, audio output devices, headphones, and other similar devices known or developed in the future. In some embodiments, user interface 804 may include software, circuitry, or other forms of logic capable of transmitting and receiving data from external user input / output devices. Additionally or alternatively, fault handling device 800 may support remote access from other devices via communication interface 802 or another physical interface (not shown). User interface 804 may be configured to receive user input, the position and movement of which may be indicated by indicators or cursors described herein. User interface 804 may also be configured as a display device for rendering or displaying text fragments.

[0088] The processor 806 may contain one or more general-purpose processors and / or special-purpose processors.

[0089] Data storage 808 may include one or more volatile and / or non-volatile storage components and may be integrated wholly or partially with processor 806. Data storage 808 may include removable and non-removable components.

[0090] Processor 806 is capable of executing program instructions 818 (e.g., compiled or uncompiled program logic and / or machine code) stored in data storage 808 to perform the various functions described herein. Data storage 808 may contain a non-transitory computer-readable medium on which program instructions are stored, which, when executed by fault handling device 800, enable fault handling device 800 to perform any methods, processes, or functions disclosed in this specification and / or the accompanying drawings. Execution of program instructions 818 by processor 806 may result in processor 806 using data 812.

[0091] For example, program instructions 818 may include an operating system 822 (e.g., an operating system kernel, device drivers, and / or other modules) installed on the fault handling device 800, and one or more application programs 820 (e.g., a browser, social application, or game application). Similarly, data 812 may include operating system data 816 and application data 814. Operating system data 816 is primarily accessible to the operating system 822, while application data 814 is primarily accessible to one or more application programs 820. Application data 814 may reside in a file system that is visible or hidden from the user on the fault handling device 800.

[0092] Application 820 can communicate with operating system 812 through one or more application programming interfaces (APIs). These APIs help application 820 read and / or write application data 814, transmit or receive information via communication interface 802, receive or display information on user interface 804, etc.

[0093] In some terminology, application 820 may be simply referred to as "app". Furthermore, application 820 can be downloaded to fault handling device 800 through one or more online app stores or app markets. However, the application can also be installed on fault handling device 800 in other ways, such as through a web browser or a physical interface on fault handling device 800 (e.g., a USB port).

[0094] Specifically, in this embodiment, the fault handling device 800 includes a data storage 808 and one or more program instructions 818, wherein one or more program instructions 818 are stored in the data storage 808, and one or more program instructions 818 are configured to be executed by one or more processors. The one or more program instructions include computer-executable instructions for performing the following: Obtain service-related information, including business information, alarm source information, and data source information; Determine an agent scheduling strategy that is compatible with the service-related information, the agent scheduling strategy being used to guide the target agent to schedule and execute the fault handling task; The target intelligent agent is scheduled based on the intelligent agent scheduling strategy to analyze the service-related information, determine the fault events associated with the service-related information and the fault causes of the fault events, and generate a repair strategy for the fault events based on the fault causes. If a repair strategy is detected, the repair strategy is returned to the administrator so that the administrator can repair the fault event according to the repair strategy.

[0095] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the fault handling device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0096] This specification provides a fault handling device that acquires service-related information, including business information, alarm source information, and data source information; determines an agent scheduling strategy adapted to the service-related information, which guides the scheduling of target agents to perform fault handling tasks; schedules target agents based on the agent scheduling strategy to analyze the service-related information, determine the fault events associated with the service-related information and the causes of the fault events, and generates repair strategies for the fault events based on the fault causes; and, upon detecting the generation of repair strategies, returns the repair strategies to the administrator so that the administrator can repair the fault events according to the repair strategies. In this way, multiple different agents are designed to analyze service-related information from multiple dimensions, considering not only fault events at the business level but also further refining fault branches in various scenarios, which can effectively improve the accuracy of the main module's analysis of service-related information, thereby further improving the accuracy of fault handling.

[0097] Furthermore, based on the above Figure 7 This specification also provides a storage medium for storing computer-executable instruction information in one or more embodiments. In one specific embodiment, the storage medium may be a USB flash drive, optical disc, hard disk, etc. When the computer-executable instruction information stored in the storage medium is executed by a processor, it can realize the following process: Obtain service-related information, including business information, alarm source information, and data source information; Determine an agent scheduling strategy that is compatible with the service-related information, the agent scheduling strategy being used to guide the target agent to schedule and execute the fault handling task; The target intelligent agent is scheduled based on the intelligent agent scheduling strategy to analyze the service-related information, determine the fault events associated with the service-related information and the fault causes of the fault events, and generate a repair strategy for the fault events based on the fault causes. If a repair strategy is detected, the repair strategy is returned to the administrator so that the administrator can repair the fault event according to the repair strategy.

[0098] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the above-described storage medium embodiment is basically similar to the method embodiment, so the description is relatively simple; relevant parts can be referred to the description of the method embodiment.

[0099] This specification provides a storage medium that acquires service-related information, including business information, alarm source information, and data source information; determines an agent scheduling strategy adapted to the service-related information, which guides the scheduling of target agents to perform fault handling tasks; schedules target agents based on the agent scheduling strategy to analyze the service-related information, determine the fault events associated with the service-related information and the causes of the fault events, and generates repair strategies for the fault events based on the fault causes; and, upon detecting the generation of a repair strategy, returns the repair strategy to the administrator so that the administrator can repair the fault events according to the repair strategy. In this way, multiple different agents are designed to analyze service-related information from multiple dimensions, considering not only fault events at the business level but also further refining fault branches in various scenarios, which can effectively improve the accuracy of the main module's analysis of service-related information, thereby further improving the accuracy of fault handling.

[0100] Furthermore, based on the above Figure 7 This specification also provides one or more embodiments of a computer program product, including a computer program, which, when executed by a processor, can perform the following processes: Obtain service-related information, including business information, alarm source information, and data source information; Determine an agent scheduling strategy that is compatible with the service-related information, the agent scheduling strategy being used to guide the target agent to schedule and execute the fault handling task; The target intelligent agent is scheduled based on the intelligent agent scheduling strategy to analyze the service-related information, determine the fault events associated with the service-related information and the fault causes of the fault events, and generate a repair strategy for the fault events based on the fault causes. If a repair strategy is detected, the repair strategy is returned to the administrator so that the administrator can repair the fault event according to the repair strategy.

[0101] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the above-described embodiment of a computer program product is relatively simple in description because it is fundamentally similar to the method embodiment; relevant parts can be referred to the description of the method embodiment.

[0102] This specification provides a computer program product that acquires service-related information, including business information, alarm source information, and data source information; determines an agent scheduling strategy adapted to the service-related information, which guides the scheduling of target agents to perform fault handling tasks; schedules target agents based on the agent scheduling strategy to analyze the service-related information, determine the fault events associated with the service-related information and the causes of the fault events, and generates repair strategies for the fault events based on the fault causes; and, upon detecting the generation of repair strategies, returns the repair strategies to the administrator so that the administrator can repair the fault events according to the repair strategies. In this way, multiple different agents are designed to analyze service-related information from multiple dimensions, considering not only fault events at the business level but also further refining fault branches in various scenarios, which can effectively improve the accuracy of the main module's analysis of service-related information, thereby further improving the accuracy of fault handling.

[0103] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than those shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous. Moreover, although one or more embodiments of this specification provide method steps as described in the embodiments or flowcharts, it is understood that the order of steps listed in the embodiments or flowcharts is merely one possible execution order among many steps and does not represent the only execution order. Therefore, when method steps are involved in the claims, adjustments to the order of those steps, or parallelism between steps, are also within the scope of protection of the claims.

[0104] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must also be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0105] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0106] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0107] For ease of description, the above apparatus is described by dividing it into various functional units. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0108] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0109] Embodiments in this specification are described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable parallel device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable parallel device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0110] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable fraud device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0111] These computer program instructions can also be loaded onto a computer or other programmable device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0112] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0113] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0114] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0115] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical or equivalent elements in the process, method, article, or apparatus that includes said element. Furthermore, "a," "an," and "the" are not specifically singular and may include plural forms. Ordinal numbers such as "first," "second," etc., do not necessarily indicate order; they are often used to distinguish objects. For example, "first server" and "second server" usually refer to two servers, described as "first server" and "second server" to differentiate them; however, sometimes these two servers may be the same server. Moreover, in this specification, unless explicitly stated otherwise, "receiving and sending data" does not necessarily mean direct receiving and sending; it can be indirect receiving and sending (i.e., receiving and sending indirectly through one or more entities). Similarly, in this specification, unless otherwise stated, the relationships between structures can be direct or indirect.

[0116] Furthermore, the specific terms used in this specification to describe embodiments, such as "an embodiment," "one embodiment," or "some embodiments," refer to a particular feature, structure, or characteristic related to at least one embodiment of this specification. Therefore, it should be emphasized and noted that "an embodiment," "one embodiment," or "an alternative embodiment" mentioned twice or more in different locations in this specification do not necessarily refer to the same embodiment. Moreover, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples, without contradiction.

[0117] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0118] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0119] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the method embodiments are basically similar to the system embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the system embodiments.

[0120] The above description is merely an embodiment of this specification and is not intended to limit this document. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims in this document.

Claims

1. A fault processing system, comprising an access layer subsystem, a dispatch layer subsystem and a processing layer subsystem, wherein: a plurality of different access interfaces are provided in the access layer subsystem, through which service related information is obtained, the service related information comprising service related information, alarm source information and data source information; at least one dispatch unit is provided in the dispatch layer subsystem to dispatch corresponding resources; the at least one dispatch unit comprises an agent dispatch unit configured to determine an agent dispatch strategy adapted to the service related information, the agent dispatch strategy being used to guide the processing layer subsystem to dispatch a target agent to execute a fault processing task; the processing layer subsystem comprises a main module and an auxiliary module, the auxiliary module being configured to assist the main module to execute a corresponding processing task, and the main module being configured to dispatch a target agent based on the agent dispatch strategy to analyze the service related information, determine a fault event associated with the service related information and a fault cause of the fault event, and generate a repair strategy for the fault event based on the fault cause; the auxiliary module is configured to return the repair strategy to an administrator when it is monitored that the repair strategy is generated, so that the administrator repairs the fault event according to the repair strategy. 2.The system of claim 1, wherein the agent dispatch unit monitors a process in which the main module dispatches a target agent to analyze the service related information, and when it is monitored that the target agent cannot determine a fault cause of a fault event, the agent dispatch unit re-determines an agent dispatch strategy adapted to the service related information for the main module. 3.The system of claim 1, wherein a plurality of agents are included in the main module, and each agent is associated with a level label, the level label comprising an identification routing, a quick positioning and a deep positioning, wherein: when the level label of the target agent is the identification routing or the quick positioning, the target agent is configured to perform risk detection on the service related information; and when a detection result indicates that the service related information is risk-free, the fault processing for the service related information is ended; when the detection result indicates that the service related information is risky, the target agent with the level label of the identification routing is configured to analyze the service related information, determine a fault event associated with the service related information and a fault cause of the fault event, and based on the agent dispatch strategy, dispatch a target agent from at least one agent with the level label of the quick positioning when the fault cause cannot be determined. When the detection result indicates that the service-related information is at risk, the target agent with the hierarchical label of quick positioning is configured to analyze the service-related information, determine the fault event associated with the service-related information and the fault cause of the fault event, and in the case where the fault cause cannot be determined, dispatch the target agent from at least one agent with a hierarchical label of deep positioning based on the agent scheduling strategy.

4. The system of claim 3, wherein: When the target agent has a hierarchical label of deep positioning, the target agent is configured to split the fault handling task for the service-related information into a plurality of sub-tasks, and assign a sub-target agent to each sub-task according to a preset task priority list; The sub-target agent is configured to execute the sub-task assigned by the target agent in response to a call from the target agent, to determine the fault event associated with the service-related information and the fault cause of the fault event, and generate a repair strategy for the fault event based on the fault cause.

5. The system of claim 3, wherein the agent with a hierarchical label of identification routing comprises a scenario routing agent, the agent with a hierarchical label of quick positioning comprises a business quick positioning agent, an engineering quick positioning agent, and an algorithm quick positioning agent, and the agent with a hierarchical label of deep positioning comprises a fund risk control agent, a content security agent, and a code analysis agent.

6. The system of claim 1, wherein the auxiliary module comprises a patrol agent, an optimization agent, and a snapshot agent, and wherein: The patrol agent is configured to asynchronously detect whether the repaired fault event meets the repair requirement; and when it is detected that the repair requirement is not met, send an analysis notification to the main module to cause the main module to re-analyze the service-related information; The optimization agent is configured to generate optimization requirement content for representing service code improvement requirements based on the fault cause of the fault event, the optimization requirement content being used to guide a code administrator to adjust the service code; The snapshot agent is configured to collect and store historical processing records of the main module, the historical processing records including a plurality of historical fault event information, each historical fault information including a processing process and a processing result of the main module for a historical fault event.

7. The system of claim 6, wherein the scheduling layer subsystem comprises an evaluation scheduling unit, the evaluation scheduling unit including an evaluation case, the evaluation case including service-related information and a corresponding marked fault cause, and wherein: The evaluation scheduling unit is configured to evaluate the determination of the fault cause by the main module based on the service-related information according to the service-related information and the corresponding marked fault cause included in the evaluation case, and generate a fault handling score for the service-related information, the fault handling score being used to guide logical adjustment of the target agent in the main module. 8.The system of claim 7, wherein the dispatch layer subsystem comprises a knowledge dispatch unit, and the processing layer subsystem comprises an additional module, and wherein: the knowledge dispatch unit is configured to construct a reference sample for the subject module based on the evaluation cases obtained from the evaluation dispatch unit and the historical processing records obtained from the snapshot unit, the reference sample being in a form of a knowledge base and a knowledge graph; and the additional module is configured to store the reference sample generated by the knowledge dispatch unit and support the subject module to call the reference sample to guide the confirmation of the fault event and the fault cause. 9.A fault processing method applied to a fault processing system, the method comprising: obtaining service related information, the service related information comprising business information, alarm source information, and data source information; determining an agent dispatch strategy adapted to the service related information, the agent dispatch strategy being used to guide a target agent to perform a fault processing task; dispatching the target agent based on the agent dispatch strategy to analyze the service related information, determine a fault event associated with the service related information and a fault cause causing the fault event, and generate a repair strategy for the fault event based on the fault cause; and in a case where the repair strategy is generated, returning the repair strategy to an administrator to make the administrator repair the fault event according to the repair strategy. 10.A fault processing device, the device comprising: a processor; and a memory arranged to store computer executable instructions that, when executed, cause the processor to: obtain service related information, the service related information comprising business information, alarm source information, and data source information; determine an agent dispatch strategy adapted to the service related information, the agent dispatch strategy being used to guide a target agent to perform a fault processing task; dispatch the target agent based on the agent dispatch strategy to analyze the service related information, determine a fault event associated with the service related information and a fault cause causing the fault event, and generate a repair strategy for the fault event based on the fault cause; and in a case where the repair strategy is generated, return the repair strategy to an administrator to make the administrator repair the fault event according to the repair strategy. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​

Images