Service scene-oriented resource fault root cause analysis system and method

Abstract

The invention discloses a service scene-oriented resource fault root cause analysis system. The system comprises a service link management module and a fault root cause analysis module, the service link management module is used for monitoring a service system, generating panoramic service link conditions, monitoring operation states of all service links and counting operation indexes; the service link management module is in signal connection with the fault root cause analysis module and is used for carrying out panoramic resource analysis on fault information generated by the service link management module after a service link reports an error, positioning a fault point and issuing fault related information; the invention further discloses a service scene-oriented resource fault root cause analysis method. According to the invention, deeper, comprehensive and accurate operation and maintenance management of the service system is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of business system operation and maintenance, and particularly to a resource fault root cause analysis system and method for business scenarios. Background Art

[0002] IT operation and maintenance management is the maintenance and management of enterprise information systems, equipment, operating environments, etc. Traditional IT operation and maintenance models in the field mostly take independent business application systems and infrastructure as management units, and separately construct operation and maintenance systems for business application systems and infrastructure. This is a flat and hierarchically fragmented operation and maintenance system, which is difficult to meet the current overall operation and maintenance management requirements. There are mainly the following problems.

[0003] First, when a business application system fails, it may be due to problems with the system's own modules or services, related middleware, shared services, associated resources (physical, virtual, network, container, etc.). It is difficult to locate problems with simple business application system or infrastructure operation and maintenance.

[0004] Second, when infrastructure resources fail, the associated business applications will also fail, repeatedly triggering multiple failures. When a large number of failure messages appear simultaneously, it is difficult for operation and maintenance personnel to intuitively find the key failure points and manual troubleshooting is required.

[0005] Third, the independent operation and maintenance management of business applications and infrastructure lacks the topological relationship between applications, services, devices, and networks, and it is impossible to conduct failure impact analysis and judge the failure impact range.

[0006] Fourth, when the business operation is not smooth, although it is successfully executed and not transformed into a failure, the execution time is long, the response is slow, and the processing capacity and index information cannot accurately reflect the association relationship, making it difficult to analyze and optimize.

[0007] Fifth, there are a large number of index data at each associated point, which requires high personal capabilities of operation and maintenance personnel and high learning costs. Summary of the Invention

[0008] The purpose of the present invention is to provide a resource fault root cause analysis system and method for business scenarios to solve the problems existing in the operation and maintenance of existing business systems, such as shallow operation and maintenance dimensions, incomplete fault merging algorithms, inaccurate fault troubleshooting and positioning, incomplete elimination of potential hazards, difficulty in excavating potential fault points, and high operation and maintenance learning costs, and to achieve more in-depth, comprehensive, and accurate operation and maintenance management of business systems.

[0009] To achieve the above object, the present invention provides a resource fault root cause analysis system for business scenarios, the system comprising: a business link management module and a fault root cause analysis module; the business link management module is used for monitoring the business system, generating a panoramic business link situation, monitoring the running status of all business links and statistical operation indicators; the business link management module is signal-connected to the fault root cause analysis module, and is used for performing panoramic resource analysis on the fault information generated by the business link management module after a business link error, locating the fault point, and publishing fault-related information.

[0010] A resource fault root cause analysis method for business scenarios, the method adopts a resource fault root cause analysis system for business scenarios as described above, the method comprising: step S1, the business link management module constructs a panoramic resource link of the business call scenario; step S2, the business link management module monitors the business system in real time, if the business system has no fault, it maintains the status quo; if the business system has a fault, it locates the fault information through the panoramic resource link and enters step S3; step S3, the fault root cause analysis module receives the fault information and analyzes the fault information to output a fault result.

[0011] Preferably, step S1 includes: step S11, building a CMDB configuration system based on each deployed business system and constructing a basic resource topology, each business system includes a plurality of configuration items; step S12, adding an independent global transaction code identifier to each business system one by one; step S13, generating a call link view of each business system, the call link view includes a plurality of link interfaces; wherein, the link interface is a refinement subordinate of the configuration item; step S14, associating the plurality of link interfaces of the call link view with the plurality of configuration items of the basic resource topology to generate a panoramic resource link.

[0012] Preferably, specifically, step S3 includes: step S31, the fault root cause analysis module receives the fault information and performs analysis on the status of each module and the status of each service based on the fault information; if there are abnormalities in each, the abnormal information is temporarily stored in the fault root cause analysis module and enters step S32; if not, it directly enters step S32; step S32, the fault root cause analysis module performs data operation monitoring based on the fault information, if there are abnormalities in each, the abnormal information is temporarily stored in the fault root cause analysis module and enters step S33; if not, it directly enters step S33; step S33, the fault root cause analysis module performs resource association monitoring based on the fault information, if there are abnormalities in each, the abnormal information is temporarily stored in the fault root cause analysis module and enters step S34; if not, it directly enters step S34; step S34, the fault root cause analysis module performs business application root cause monitoring based on all abnormal information, performs weight algorithm analysis, and outputs a fault result.

[0013] Preferably, specifically, the module status analysis in step S31 refers to the internal inspection of the self - operation of the business link management module and the root cause of failure analysis module by the root cause of failure analysis module; the analysis of each service status refers to the inspection of the operation of this method.

[0014] Preferably, in step S32, the data operation monitoring includes: relational database analysis, big data platform status analysis, cache database analysis, and message queue failure analysis.

[0015] Preferably, the resource association monitoring includes: cluster failure analysis, computing resource analysis, storage resource analysis, network resource analysis; the cluster failure analysis can analyze the aggregate, where the aggregate refers to the whole composed of multiple data / devices; the computing resource analysis can analyze the carrier of computing functions; the storage resource analysis can analyze the carrier of storage functions; the network resource analysis can analyze the carrier of network functions.

[0016] Preferably, the business application root cause monitoring includes: call chain tracing analysis, database SQL analysis, and record analysis.

[0017] Preferably, for the weight algorithm analysis, the weight value C of the configuration item or link interface is as follows: application is 1, load balancer is 2, database is 3, container is 4, ECS is 5, physical machine is 6, network device is 7; the failure level F is divided into 1 point, 2 points, and 3 points, and the calculation formula of the weight value W is W = C×F×α + β; where, α is the dynamic adjustment coefficient; β is the basic priority constant; if it is greater than or equal to 25, then this failure information aggregate is critical; if it is less than 25 and greater than or equal to 15, then this failure information aggregate is important; if it is less than or equal to 15, then this failure information aggregate is general.

[0018] Preferably, α = 1.2; β = 10.

[0019] In summary, compared with the prior art, a resource failure root cause analysis system for business scenarios provided by the present invention has the following beneficial effects:

[0020] First, the present invention is oriented to business scenarios, and the operation and maintenance object can reach the specific execution functions of the business system. Compared with other operation and maintenance that only reach the business system, the operation and maintenance dimension is deeper.

[0021] Second, the present invention incorporates various types of applications, associated software, infrastructure, and resources as a whole into unified operation and maintenance analysis, generates a business panoramic link, and the failure merging algorithm is more comprehensive.

[0022] Third, relying on the upper - layer failure information, the present invention conducts self - inspection scans on all associated software and infrastructure, and the failure troubleshooting and positioning are more accurate.

[0023] Fourth, relying on the upper-layer fault information which is not just fault information, the present invention monitors and analyzes the operation index data of each layer, scans the associated resources without fault information, and eliminates potential hazards more thoroughly. It can perform statistical analysis on the entire link of each business scenario of the business system, with the depth reaching the average business execution success rate and call duration, generate specific business views, and can deeply dig out potential fault points that have no fault information but are operating poorly, with a deeper operation and maintenance level.

[0024] Fifth, for general operation and maintenance personnel, the present invention can be competent for the operation and maintenance work in complex scenarios, reducing the learning cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a flowchart of a method for analyzing the root cause of resource faults for business scenarios proposed by the present invention.

[0026] Figure 2 It is a view of the call link of each business system generated under the transfer business process of the present invention.

[0027] Figure 3 Arrange the order of the interfaces of the corresponding links of the configuration items.

[0028] Figure 4 Based on the fault information, the present invention performs status analysis of each module and status analysis of each service.

[0029] Figure 5 The root cause analysis module of the present invention monitors the data operation based on the fault information.

[0030] Figure 6 The root cause analysis module of the present invention monitors the resource association based on the fault information.

[0031] Figure 7 The root cause analysis module monitors the root cause of business applications according to all abnormal information. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0032] The following will combine the accompanying drawings in the embodiments of the present invention Figure 1 ~the accompanying drawings Figure 7 to elaborate in detail on the technical solutions, structural features, achieved objectives, and effects in the embodiments of the present invention.

[0033] It should be noted that the accompanying drawings are in a very simplified form and all use non-precise scales, only for the purpose of conveniently and clearly assisting in explaining the embodiments of the present invention, and are not used to limit the limiting conditions of the embodiments of the present invention. Therefore, they do not have technical substance significance. Any modification of the structure, change of the proportional relationship, or adjustment of the size, without affecting the effects that the present invention can produce and the objectives that can be achieved, should still fall within the scope covered by the technical content disclosed by the present invention.

[0034] It should be noted that, in the present invention, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only the elements explicitly listed, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0035] The present invention proposes a resource fault root cause analysis system for business scenarios, which is used to achieve more in-depth, comprehensive and accurate operation and maintenance management of business systems; the system comprises: a business link management module and a fault root cause analysis module.

[0036] As described in the background technology, with the development of cloud computing, big data, distributed, microservices and other technologies, the business systems and equipment that originally operated independently are organically integrated together to better meet the development and use of the business. However, as the functions / performance of each business system become more powerful, the difficulty of operation and maintenance is also increasing. Operation and maintenance personnel need to collect the operating conditions of independent business systems, infrastructure, middleware, etc., determine the point and cause of the fault, and have high requirements on the ability of operation and maintenance management.

[0037] Specifically, the business link management module is used for business system monitoring, generating panoramic business link status, monitoring the operating status of all business links and statistical operating indicators. Among them, the panoramic business link refers to a comprehensive and detailed characterization and presentation of the complete process of a business system from start to end. It covers all components, services, data flows and their interactions involved in the business system, forming a visual and coherent business process map, like a detailed map, clearly showing the overall picture of the business. That is, the business link management module includes but is not limited to business indicator data collection, indicator data monitoring, business link generation, etc.

[0038] Specifically, the service link management module is connected to the fault root cause analysis module by signal, and is used to perform a panoramic resource analysis on the fault information generated by the service link management module after the service link reports an error, locate the fault point, and publish fault related information. That is, the fault root cause analysis module includes but is not limited to service system fault analysis, basic software fault analysis, basic equipment fault analysis, fault root cause analysis, etc.

[0039] Therefore, the service link management module and the fault root cause analysis module achieve the purpose of the present invention by operating in collaboration with each other.

[0040] In this regard, the present invention also proposes a method for analyzing the root cause of resource failures for business scenarios, such as Figure 1 shown. The method uses a system for analyzing the root cause of resource failures for business scenarios as described above. The method includes:

[0041] Step S1, the business link management module constructs a panoramic resource link for the business call scenario;

[0042] Step S2, the business link management module monitors the business system in real time. If the business system has no faults, it maintains the status quo; if the business system has a fault, it locates the fault information through the panoramic resource link and enters step S3;

[0043] Step S3, the root cause analysis module for faults receives the fault information and analyzes the fault information to output a fault result.

[0044] Specifically, the step S1 includes:

[0045] Step S11, build a CMDB configuration system (Configuration Management Database) based on each deployed business system and construct a basic resource topology. Each of the business systems includes multiple configuration items (CI, Configuration Item);

[0046] Among them, by building and deploying the CMDB configuration system, starting from each deployed business system, various associated software of the relevant business systems, the containers carried, the configuration of basic device network resources, etc. (i.e., the aforementioned configuration items) are entered into the CMDB configuration system to form a basic resource topology. Configuration items can be various IT resources, such as servers, network devices, software applications, databases, user accounts, etc.

[0047] The CMDB configuration system is a core tool for managing IT infrastructure and service configuration information and plays a crucial role in IT service management. The CMDB configuration system is a database system that centrally stores and manages all configuration items in the IT environment and their interrelationships. This CMDB configuration system records the detailed information of each configuration item, including its attributes, status, version, location, etc., as well as the association relationships between configuration items, such as dependency relationships, hierarchical relationships, etc. That is, the CMDB configuration system is a system that includes multiple configuration items.

[0048] Step S12, add independent global transaction code identifiers to each business system one by one;

[0049] Specifically, when the business system invokes a service, a global transaction code identifier is added. When each business system interface is invoked, this transaction code is passed until the business ends. In this way, the corresponding data transmitted by the same business system in the basic resource topology (i.e., the data transmitted through the link interface in step S3) can be clearly indicated.

[0050] Step S13: Generate a call link view for each business system, where the call link view includes multiple link interfaces.

[0051] For example, sample the data in the business system according to business calls, identify the business orders with successful execution, query the corresponding call links, identify the execution processes, and generate a business call link view corresponding to the order process by continuously superimposing the data of successful order processes. As Figure 2 shown, it shows multiple link interfaces involved in the transfer process, which are, in sequence, the mobile client service area, the mobile service area, the channel interaction platform, unified authentication, the capability center, access and egress, the enterprise customer bus (ESB), and unified payment. Because, as described in step S2, the global transaction code identifiers in the same business system are the same, these multiple link interfaces represent the transmission of various data of the business system. Therefore, the link interface is the smallest unit that carries the data of the business system. The link interface includes, but is not limited to, the specific composition of the database, the type of container, etc. It can be understood that the link interface is a refinement of the configuration item. It should be noted here that not every configuration item in a business system will always have a link interface in the construction of a basic resource topology. There may be cases where, for the convenience of operation, a combination of multiple link interfaces is regarded as an overall configuration item.

[0052] Step S14: Associate the multiple link interfaces of the call link view with the multiple configuration items of the basic resource topology to generate a panoramic resource link.

[0053] Associate each link interface of the business call link view with the basic resource topology in the CMDB configuration, and associate the configuration items required for the cooperation of this business link view, that is, related basic software, hosting containers, basic equipment resources, network conditions, etc., to obtain the panoramic resource link of this business call scenario.

[0054] For example, as Figure 3As shown, they are arranged in the order of the corresponding links of the configuration items. Application name: Channel interaction platform. Business call chain links: Payment information verification, transfer order details, update order payment status. Basic software: mysql (mysql is a widely used open-source relational database management system), RabbitMQ container (RabbitMQ is an open-source message queue middleware that implements the Advanced Message Queuing Protocol. Deploying RabbitMQ in a container can facilitate message passing and asynchronous communication between distributed systems): ECS; Basic device resources: Physical machine. This is only a partial display here and will not be elaborated.

[0055] Specifically, step S2 further includes: The business link management module can monitor the business system in real time at a certain interval. If the business system is fault-free, it remains unchanged; if a fault occurs in the business system, the fault information is located through the panoramic resource link, and step S3 is entered.

[0056] The purpose of step S2 is to perform real-time monitoring during business operation. When a fault occurs in the business system, the fault information is mapped to the business call link view to locate the service link interface or configuration item with problems, facilitating subsequent step S3 to analyze and read the corresponding fault information (which is essentially data). Among them, the fault information includes: data corruption during data transmission, database connection exception, database transaction processing exception, insufficient database server resources, etc.; they will not be elaborated one by one here.

[0057] Specifically, step S3 includes:

[0058] Step S31, the fault root cause analysis module receives the fault information and performs analysis of the status of each module and the status of each service based on the fault information; if there are abnormalities, the abnormal information is temporarily stored in the fault root cause analysis module, and step S32 is entered; if not, step S32 is directly entered;

[0059] Among them, the module status analysis refers to the internal inspection of the self-operation of the business link management module and the fault root cause analysis module by the fault root cause analysis module; the analysis of the status of each service refers to the inspection of the operation of this method. As Figure 4 shown, it shows the analysis process of this internal monitoring, including the status analysis of each module and each service. The analysis dimensions include process status, thread status, log analysis, and port occupancy. Step S31 is essentially a preliminary review of the self-operation basis of the method to ensure the accuracy of subsequent fault inspections.

[0060] The following explains the process status, thread status, log analysis, and port occupancy one by one.

[0061] A process is an execution instance of a program in an operating system and is the basic unit for the system to allocate and schedule resources. The process state reflects the stage and running status of the process during its life cycle.

[0062] A thread is an execution unit within a process. A process can contain multiple threads, and they share the resources of the process. The thread state describes the different stages and statuses of a thread during execution.

[0063] Logs are a series of events and information recorded by an application during its operation, including system messages, error messages, debugging information, etc. Log analysis is to collect, organize, filter, and parse these log data to obtain valuable information.

[0064] A port is the entrance and exit for a computer to communicate with the outside world and is used to distinguish different network applications. Port occupation means that a certain port has been bound and used by a certain process or service, and other processes cannot use this port for communication anymore.

[0065] Step S32: The root cause analysis module for faults monitors data operation based on the fault information. If there are abnormalities, the abnormal information is temporarily stored in the root cause analysis module for faults, and then step S33 is entered; if not, step S33 is directly entered;

[0066] As Figure 5 shown, among them, the data operation monitoring includes: relational database analysis, big data platform status analysis, cache database analysis, and message queue fault analysis; the essence of the above is to capture whether there are abnormalities during the process of data operation and transmission in the business system, as detailed in the following embodiment introduction; in specific applications, different analyses can be performed according to actual needs, which are not limited here. The following is only an example, and it can be selected according to the database / big data platform carried, etc. How to carry and select the database / big data platform, etc. are all existing technologies and will not be elaborated here.

[0067] (1) Relational database analysis

[0068] Database service status: Mainly focus on whether the database is running normally, such as whether the mysql process is in the running state and whether there are abnormal stops or crashes. If the service is abnormal, it may cause the application to be unable to connect to the database, resulting in problems such as data query and write failures.

[0069] Data connection situation: Monitor the number of connections to the database, including the current number of connections and the maximum connection number limit. Too many connections may exhaust the database resources and affect performance; too few connections may indicate problems with the application's connection to the database and the inability to obtain connections normally.

[0070] Tablespace Remaining Space: Check the usage of the database tablespace. Insufficient remaining space may cause data writing failures. Useless data should be cleared in time or the tablespace should be extended.

[0071] Buffer Pool Status: The hit rate of the buffer pool is a key indicator to measure its performance. The higher the hit rate, the greater the proportion of data read from memory by the database, the fewer disk I / O operations, and the better the database performance. If the buffer pool size is set unreasonably, or the data pages in the buffer pool are frequently swapped out, resulting in a low hit rate, it will increase the disk I / O burden and reduce the database response speed. Monitoring the usage of the buffer pool, such as memory occupancy, the number of data pages, and the hit rate, helps optimize database performance. For example, adjust the buffer pool size according to the actual business load.

[0072] SQL (SQL stands for Structured Query Language, a standard language for database management and query): Monitor and analyze the relevant SQL statements executed in the database. This includes aspects such as the execution time of SQL statements, execution plans, resource consumption (such as CPU, memory, disk I / O, etc.), and execution frequency.

[0073] (2) Big Data Platform Status Analysis

[0074] Hdfs (Hadoop Distributed File System): Monitor the file system status of HDFS, such as whether the distribution of data blocks is balanced and whether there are any missing data blocks.

[0075] Yarn (Yet Another Resource Negotiator, a cluster resource management system): Pay attention to the resource allocation of Yarn, such as the available memory and CPU resources in the cluster, and the resource occupancy of each application.

[0076] Zookeeper (ZooKeeper is an open-source distributed coordination service that provides an efficient and reliable coordination mechanism for distributed applications): Monitor the node status of Zookeeper to check if there are any offline nodes; observe whether the election mechanism of Zookeeper is normal to ensure the consistency and normal coordination function of the distributed system.

[0077] Hbase (Hadoop Database, that is, Hadoop database, which is a distributed, scalable NoSQL database built on top of the Hadoop distributed file system): Check the status of Hbase for faults; monitor the read and write performance of tables, such as read and write latency and throughput. If the read and write performance deteriorates, it may affect the access efficiency of applications to Hbase data.

[0078] Storm (Storm is an open-source distributed real-time computing system): Monitor the running status of Storm topologies for any task failures or backlogs; check the processing speed. If the processing speed is too slow, it may cause untimely data processing and affect the real-time calculation results.

[0079] (III) Cache Database Analysis

[0080] Service Alive Status: Used to determine whether the cache database service is running properly. That is, whether the process of the cache database is active and can respond to client requests normally. For example, for the Redis service, whether its main process is running, whether it can connect to the service port through the network and perform data access and storage operations, etc.

[0081] Connection Status: Similar to relational databases, the number of connections to the cache database also needs to be monitored. Excessive connections may lead to resource waste or performance issues.

[0082] Memory Usage: Monitor the amount of memory occupied by the cache database to see if it is approaching or exceeding the server's memory limit. Excessive memory usage may cause a decline in server performance or even system crashes.

[0083] (IV) Message Queue Fault Analysis

[0084] Connection Status: Similar to the above, the number of connections to the message queue also needs to be monitored. Excessive connections may lead to resource waste or performance issues.

[0085] Memory Usage: Similar to the above, monitor the amount of memory occupied by the message queue to see if it is approaching or exceeding the server's memory limit. Excessive memory usage may cause a decline in server performance or even system crashes.

[0086] Message Processing Status: For example, check if there is a large backlog of messages in the message queue. If the message backlog is severe, it may be that the producer is sending messages too fast or the consumer's ability to process messages is insufficient, and the production and consumption speeds need to be adjusted.

[0087] Queue Status: Covers various attributes and status information of the message queue, such as queue length (the number of messages backlogged in the queue), queue creation and deletion status, queue permission settings, queue consumer group situation, etc.

[0088] The above are specific examples under certain embodiments. Specifically, which analysis to select needs to be correspondingly selected according to the actual situation and will not be elaborated here.

[0089] Step S33: The fault root cause analysis module performs resource correlation monitoring based on the fault information. If there are abnormalities, the abnormal information is temporarily stored in the fault root cause analysis module and step S34 is entered; if not, step S34 is directly entered.

[0090] Such as Figure 6 shown, wherein, the resource correlation monitoring includes: cluster fault analysis, computing resource analysis, storage resource analysis, and network resource analysis.

[0091] The cluster fault analysis can analyze the aggregate, where the aggregate refers to the whole composed of multiple data / devices; for example, analyze the physical machine status of the cloud cluster, the data cluster status, the container cluster status, etc.

[0092] The computing resource analysis can analyze the carriers of computing functions, such as the physical machine status, virtual machine status, container status, etc.

[0093] The storage resource analysis can analyze the carriers of storage functions, such as the disk array status, storage controller status, etc.

[0094] The network resource analysis can analyze the carriers of network functions, such as the router status, switch status, etc.

[0095] The foregoing analysis mode covers the status monitoring and analysis of various resources from physical machines to virtual machines and containers, and from storage devices to network devices.

[0096] That is, steps S31 to S33 are specific link interfaces accurate to specific configuration items and corresponding bearing data for analysis, thereby facilitating the calculation of weights in step S34.

[0097] Step S34: The fault root cause analysis module performs business application root cause monitoring based on all abnormal information, performs weight algorithm analysis, and outputs the fault result.

[0098] Such as Figure 7 shown, specifically, the business application root cause monitoring includes: call chain tracing analysis, database SQL analysis, and record analysis.

[0099] The specific analysis objects of the call chain tracing analysis are hierarchical calls, interface response time, business processes, business typical analysis, etc.

[0100] The specific analysis objects of the database SQL analysis are time-consuming SQL statements, abnormal SQL statements, SQL optimization, etc.

[0101] The record analysis specifically takes the host operation logs, business system logs, I / O failure logs, alarm logs, etc. as the analysis objects; the logs here have a wider scope compared to the aforementioned log analysis.

[0102] Specifically, the weight algorithm analysis is as follows: According to the configuration items or link interfaces where failures occur, the corresponding weight values are superimposed. If it is greater than or equal to 25, the aggregate of the failure information is critical, and it needs to be immediately repaired at P0 (urgent), with the entire team collaborating to restore it within 1 hour; if it is less than 25 and greater than or equal to 15, the aggregate of the failure information is important, and it needs to be repaired at P1 (high) within 2 hours, with cross-departmental collaboration; if it is less than or equal to 15, the aggregate of the failure information is general, and it needs to be repaired at P2 (medium / low) within 4 hours, with conventional processes for handling.

[0103] For example, the following is an example of the weight value C of the configuration item or link interface: The application is 1, the load balancer is 2, the database is 3, the container is 4, the ECS is 5, the physical machine is 6, and the network device is 7. According to the system operation and maintenance requirements, the weights of the configuration items are now refined and classified to optimize the failure response priority.

[0104] Adopting the product weighting method, the weight value is dynamically calculated in combination with the failure level F, where the failure level F represents the failure level, divided into 1 point (general), 2 points (important), and 3 points (critical), and this score corresponds to the preset situation; the calculation formula for the weight value W is as follows:

[0105] W = C × F × α + β

[0106] α: Dynamic adjustment coefficient (default 1.2, can be adjusted according to business pressure);

[0107] β: Basic priority constant (default 10, to prevent the weight from being zero).

[0108] For example, in Scenario 1: A critical failure (F = 3) occurs in the network device (C = 7). W = 7 × 3 × 1.2 + 10 = 25.2 + 10 = 35.2 (P0 level). Response strategy: Activate the network-wide redundant link and prioritize restoring the core switch.

[0109] For example, in Scenario 2: A general failure (F = 1) occurs in the database (C = 3). W = 3 × 1 × 1.2 + 10 = 3.6 + 10 = 13.6 (P2 level). Response strategy: Repair during the low-traffic period at night to avoid affecting the business peak.

[0110] In addition, the failure information with a high weight is considered the most likely cause of the failure. When the weight values are the same, they are all considered the most likely causes of the failure and are presented to the operation and maintenance personnel together.

[0111] Although the content of the present invention has been described in detail through the above preferred embodiments, it should be recognized that the above description should not be considered as a limitation of the present invention. After those skilled in the art have read the above content, various modifications and alternatives to the present invention will be obvious. Therefore, the protection scope of the present invention should be defined by the appended claims.

Claims

1. A resource failure root cause analysis system for business scenarios, characterized in that: The system comprises: a service link management module and a fault root cause analysis module; The service link management module is used for service system monitoring, generating panoramic service link status, monitoring the operating status of all service links and statistical operating indicators; The service link management module is connected to the fault root cause analysis module by signal, and is used to perform a panoramic resource analysis on the fault information generated by the service link management module after the service link reports an error, locate the fault point, and publish fault-related information.

2. A resource failure root cause analysis method for business scenarios, characterized in that: The method adopts a service scenario-oriented resource fault root cause analysis system as claimed in claim 1, and the method comprises: Step S1, the business link management module constructs a panoramic resource link of the business call scenario; Step S2, the business link management module monitors the business system in real time. If the business system is fault-free, the status quo is maintained; if a fault occurs in the business system, the fault information is located through the panoramic resource link, and step S3 is entered; Step S3: The fault root cause analysis module receives the fault information, analyzes the fault information, and outputs the fault result.

3. A service scenario-oriented resource failure root cause analysis method according to claim 2, characterized in that: The step S1 comprises: Step S11, building a CMDB configuration system based on each deployed business system and constructing a basic resource topology, wherein each business system includes multiple configuration items; Step S12, adding independent global transaction code identifiers to each business system one by one; Step S13, generating a call chain view of each business system, wherein the call chain view includes multiple link interfaces; wherein the link interface is a detailed subordinate of the configuration item; Step S14, associating multiple link interfaces of the calling link view with multiple configuration items of the basic resource topology to generate a panoramic resource link.

4. The method for analyzing root causes of resource failures in a business scenario according to claim 3, characterized in that: Specifically, step S3 includes: Step S31, the fault root cause analysis module receives the fault information, and performs status analysis of each module and each service based on the fault information; if there is an abnormality, the abnormality information is temporarily stored in the fault root cause analysis module, and the process proceeds to step S32; if not, the process proceeds directly to step S32; Step S32, the fault root cause analysis module performs data operation monitoring based on the fault information. If there is an abnormality, the abnormality information is temporarily stored in the fault root cause analysis module, and the process proceeds to step S33; if not, the process proceeds directly to step S33; Step S33, the fault root cause analysis module performs resource association monitoring based on the fault information. If there is an abnormality, the abnormality information is temporarily stored in the fault root cause analysis module, and the process proceeds to step S34; if not, the process proceeds directly to step S34; Step S34, the fault root cause analysis module monitors the root causes of business applications based on all abnormal information, performs weighted algorithm analysis, and outputs fault results.

5. A service scenario-oriented resource failure root cause analysis method according to claim 4, characterized in that: Specifically, the module status analysis in step S31 refers to the internal inspection of the service link management module and the fault root cause analysis module by the fault root cause analysis module; and the analysis of each service status refers to the inspection of the operation of the method.

6. A service scenario-oriented resource failure root cause analysis method according to claim 5, characterized in that: In step S32, the data operation monitoring includes: relational database analysis, big data platform status analysis, cache database analysis and message queue fault analysis.

7. A service scenario-oriented resource failure root cause analysis method according to claim 6, characterized in that: The resource association monitoring includes: cluster fault analysis, computing resource analysis, storage resource analysis, and network resource analysis; the cluster fault analysis can analyze the aggregate, where the aggregate refers to the whole composed of multiple data / devices; the computing resource analysis can analyze the computing function carrier; the storage resource analysis can analyze the storage function carrier; The network resource analysis may analyze the network function bearer.

8. The service scenario-oriented resource failure root cause analysis method according to claim 7, characterized in that: The business application root cause monitoring includes: call chain tracking analysis, database SQL analysis, and record analysis.

9. The service scenario-oriented resource failure root cause analysis method according to claim 8, characterized in that: The weight algorithm analysis is as follows: the weight value C of the configuration item or link interface is 1 for application, 2 for load balancing, 3 for database, 4 for container, 5 for ECS, 6 for physical machine, and 7 for network equipment; the fault level F is divided into 1 point, 2 points, and 3 points, and the calculation formula of the weight value W is W=C×F×α+β; wherein α is the dynamic adjustment coefficient; β is the basic priority constant; if it is greater than or equal to 25, the fault information aggregate is fatal; if it is less than 25 and greater than or equal to 15, the fault information aggregate is important; if it is less than or equal to 15, the fault information aggregate is general.

10. The service scenario-oriented resource failure root cause analysis method according to claim 9, characterized in that: The α=1.2; β=10.

Images