Fault delimiting method and device, equipment, storage medium and product

Abstract

The invention discloses a fault delimiting method and device, equipment, a storage medium and a product, and relates to the technical field of computers, and the fault delimiting method comprises the steps that in response to abnormal alarm information sent by a cloud system, first hierarchical architecture information of the cloud system and first index data corresponding to each hierarchical architecture are acquired; performing anomaly judgment on the first index data to obtain a first anomaly result; and based on the first hierarchical architecture information, carrying out layer-by-layer convergence processing on the first abnormal result, and determining a first fault handling object of the cloud system and a corresponding first fault handling scheme. According to the method, the system on the cloud is divided into a plurality of hierarchical frameworks, the fault delimiting platform quickly defines the range and boundary of the abnormal problem by converging the abnormal results of the plurality of hierarchical frameworks layer by layer, and the fault handling object and the corresponding first fault handling scheme can be determined without determining the abnormal root cause, so that the fault handling efficiency is improved. Therefore, the fault processing efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a fault demarcation method, apparatus, device, storage medium and product. Background Art

[0002] In the field of modern information technology, system stability and reliability are crucial to business operations. With the development of technology, cloud computing and microservice architecture are becoming increasingly popular, and system architecture is becoming more and more complex, involving a wide range of technology stacks, including hardware, software, networks and other levels. As the system is split into hundreds or even thousands of microservices, a failure in any link may cause instability or performance degradation of the entire system. Therefore, how to quickly identify the cause of system failure and provide a solution is a technical problem that needs to be solved urgently.

[0003] Related technologies usually use the root cause location method to comprehensively analyze the performance of all relevant technology stacks of the system from top to bottom, determine the root cause of the fault, and then formulate targeted solutions. However, the root cause location in this method requires in-depth analysis of the root cause of the problem, involving multiple levels and components. The location process may require more data association and analysis, resulting in inefficient fault handling.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of this application is to provide a fault demarcation method, aiming to solve the technical problem of low efficiency in fault processing.

[0006] To achieve the above objectives, the present application proposes a fault demarcation method, which includes:

[0007] In response to the abnormal alarm information sent by the cloud system, obtaining the first-level architecture information of the cloud system and the first indicator data corresponding to each level architecture;

[0008] Performing abnormality judgment on the first indicator data to obtain a first abnormality result;

[0009] Based on the first hierarchical architecture information, the first abnormal result is aggregated layer by layer to determine a first fault handling object of the cloud system and a corresponding first fault handling solution.

[0010] Optionally, the step of performing layer-by-layer aggregation processing on the first abnormal result based on the first hierarchical architecture information to determine a first fault handling object of the cloud system and a corresponding first fault handling solution includes:

[0011] Based on the first hierarchical architecture information, determine the granularity information of each hierarchical architecture, wherein the granularity information refers to the size information of the resource scope involved in the hierarchical architecture in the cloud system;

[0012] Based on the granularity information, sorting the hierarchical structures from small to large to obtain sorted hierarchical structures;

[0013] Based on the first abnormal result, the abnormal result of each hierarchical structure after the sorting is judged layer by layer to obtain the maximum granularity hierarchical structure with abnormality, and the maximum granularity hierarchical structure with abnormality is used as the first fault handling object;

[0014] Based on the abnormal result of the first fault handling object, a corresponding first fault handling solution is determined.

[0015] Optionally, the step of performing abnormal result judgment on each hierarchical structure after sorting based on the first abnormal result to obtain the maximum granularity hierarchical structure with abnormalities includes:

[0016] Based on the sorted hierarchical structures, determining a first hierarchical structure;

[0017] Based on the abnormal result, determining whether the first hierarchical architecture has an abnormality;

[0018] If there is an exception in the first hierarchical structure, based on the first exception result, it is determined whether there is an exception in the next hierarchical structure, until there is no exception in the current hierarchical structure, and the previous level of the hierarchical structure without the exception is used as the maximum granularity hierarchical structure with the exception.

[0019] Optionally, the step of determining a corresponding fault handling solution based on the abnormal result of the fault handling object includes:

[0020] Acquire a fault handling solution library, wherein the fault handling solution library contains a handling solution for each abnormal result;

[0021] Based on the fault handling solution library, the abnormal results of the fault handling object are matched with solutions to obtain corresponding fault handling solutions.

[0022] Optionally, the step of performing abnormality judgment on the first indicator data to obtain a first abnormal result includes:

[0023] Determine the weight coefficients of different levels of architectural indicators;

[0024] Based on the weight coefficient, a health index is calculated for the first indicator data by using a preset time series anomaly detection algorithm to obtain a health index value of each level architecture;

[0025] Based on the health index value, a corresponding first abnormal result is determined.

[0026] Optionally, the cloud system includes an application layer and a service layer, and the fault demarcation method further includes:

[0027] In response to the abnormal alarm information sent by the cloud system, obtaining the second level architecture information of the application layer and the second indicator data corresponding to each level architecture;

[0028] Performing an abnormality judgment on the second indicator data to obtain a second abnormality result;

[0029] Based on the second hierarchical architecture information, the second abnormal result is aggregated layer by layer to determine a second fault handling object and a corresponding second fault handling solution of the application layer;

[0030] If the second fault handling solution is that the application layer cannot handle it, obtaining the third-level architecture information of the service layer and the third indicator data corresponding to each level architecture of the service layer;

[0031] Performing abnormality judgment on the third indicator data to obtain a third abnormal result;

[0032] Based on the third hierarchical architecture information, the third abnormal result is aggregated layer by layer to determine a third fault handling object and a corresponding third fault handling solution of the service layer.

[0033] In addition, to achieve the above purpose, the present application also proposes a fault demarcation device, the fault demarcation device comprising:

[0034] An acquisition module, configured to acquire first-level architecture information of the cloud system and first indicator data corresponding to each level of architecture in response to abnormal alarm information sent by the cloud system;

[0035] A judgment module, used for performing an abnormality judgment on the first indicator data to obtain a first abnormality result;

[0036] The aggregation module is used to perform layer-by-layer aggregation processing on the first abnormal result based on the first hierarchical architecture information, and determine the first fault handling object and the corresponding first fault handling plan of the cloud system.

[0037] In addition, to achieve the above objectives, the present application also proposes a fault delimitation device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the fault delimitation method described above.

[0038] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the fault delimitation method described above are implemented.

[0039] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the steps of the fault delimitation method described above are implemented.

[0040] One or more technical solutions proposed in this application have at least the following technical effects:

[0041] In contrast to the root cause location method usually adopted in related technologies, a comprehensive top-down analysis of all relevant technology stack performances of the system is performed to determine the root cause of the fault and then develop a targeted solution. However, the root cause location in this method requires an in-depth analysis of the root cause of the problem, involving multiple levels and components. The location process may require more data association and analysis, resulting in low efficiency in fault handling. In this application, the fault delimitation platform responds to the abnormal alarm information sent by the cloud system, obtains the first-level architecture information of the cloud system and the first indicator data corresponding to each level of architecture, and then performs an abnormal judgment on the first indicator data to obtain a first abnormal result. Finally, based on the first-level architecture information, the first abnormal result is aggregated layer by layer to determine the first fault handling object of the cloud system and the corresponding first fault handling solution. It is understandable that in the present application, the cloud system is divided into multiple hierarchical architectures. When a fault occurs, that is, when the abnormal alarm information sent by the cloud system is received, the fault demarcation platform quickly defines the scope and boundaries of the abnormal problem by aggregating the abnormal results of multiple hierarchical architectures layer by layer. After defining the scope and boundaries of the abnormal problem, the fault handling object and the corresponding first fault handling plan can be determined without determining the root cause of the abnormality, thereby improving the efficiency of fault handling. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0044] Figure 1 A flowchart of the first embodiment of the fault delimitation method of the present application is provided;

[0045] Figure 2 A logical flow chart of alarm triggering delimitation of the fault delimitation method of this application;

[0046] Figure 3 A schematic diagram of the layered architecture of the cloud system for the fault demarcation method of this application;

[0047] Figure 4 A flowchart of the second embodiment of the fault delimitation method of the present application is provided;

[0048] Figure 5 This is a schematic diagram of the overall demarcation process of the cloud system in the fault demarcation method of this application;

[0049] Figure 6 A schematic diagram of a path for application layer fault demarcation in the fault demarcation method of this application;

[0050] Figure 7 A schematic diagram of a path for service layer fault demarcation in the fault demarcation method of this application;

[0051] Figure 8 A schematic diagram of cloud service architecture deployment in the fault demarcation method of this application;

[0052] Fig. 9 This is a schematic diagram of the module structure of the fault delimiting device according to an embodiment of the present application;

[0053] Fig.10 Schematic diagram of the device structure of the hardware operating environment involved in the fault delimitation method in the embodiment of the present application.

[0054] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0055] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0056] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0057] The main solution of the embodiment of the present application is: in response to the abnormal alarm information sent by the cloud system, obtain the first-level architecture information of the cloud system and the first indicator data corresponding to each level architecture; perform abnormal judgment on the first indicator data to obtain a first abnormal result; based on the first-level architecture information, perform level-by-level aggregation processing on the first abnormal result to determine the first fault handling object of the cloud system and the corresponding first fault handling plan.

[0058] In this embodiment, the fault demarcation platform is used as the execution subject. For the convenience of description, it is hereinafter referred to as the "platform".

[0059] Since the root cause location method is usually used in related technologies, all relevant technology stacks of the system are comprehensively analyzed from top to bottom to determine the root cause of the fault and then develop targeted solutions. However, the root cause location in this method requires in-depth analysis of the root cause of the problem, involving multiple levels and components. The location process may require more data association and analysis, resulting in inefficient fault handling.

[0060] This application provides a solution to achieve fault demarcation of cloud systems, thereby improving the efficiency of fault handling.

[0061] From the above embodiments, it can be seen that the fault delimitation platform in the present application responds to the abnormal alarm information sent by the cloud system, obtains the first-level architecture information of the cloud system and the first indicator data corresponding to each level architecture, and then performs abnormal judgment on the first indicator data to obtain the first abnormal result. Finally, based on the first-level architecture information, the first abnormal result is aggregated layer by layer to determine the first fault handling object of the cloud system and the corresponding first fault handling plan. It can be understood that the cloud system is divided into multiple hierarchical architectures in the present application. When a fault occurs, that is, when the abnormal alarm information sent by the cloud system is received, the fault delimitation platform quickly defines the scope and boundaries of the abnormal problem by aggregating the abnormal results of multiple hierarchical architectures layer by layer. After defining the scope and boundaries of the abnormal problem, the fault handling object and the corresponding first fault handling plan can be determined without determining the root cause of the abnormality, thereby improving the efficiency of fault handling.

[0062] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a terminal system, etc. The following takes the fault delimiting device as an example to illustrate this embodiment and the following embodiments.

[0063] Based on this, the present application embodiment provides a fault demarcation method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the fault delimitation method of the present application.

[0064] In this embodiment, the fault demarcation method includes steps S100 to S300:

[0065] Step S100, in response to the abnormal alarm information sent by the cloud system, obtaining the first-level architecture information of the cloud system and the first indicator data corresponding to each level architecture;

[0066] In a specific implementation, the fault delimitation method is applied to a fault delimitation platform, and the fault delimitation platform is communicatively connected to the cloud system, wherein the fault delimitation platform is a system or service, which establishes a communication connection with the cloud system (i.e., a system running in a cloud environment) through a network, and such a connection enables the fault delimitation platform to receive information from the cloud system in order to perform fault delimitation of the cloud system. Fault delimitation generally refers to determining the approximate scope or area where a fault occurs in a complex system by analyzing the performance and behavior of the system. When an abnormality occurs in the cloud system, an alarm message will be generated, and the alarm message will be sent to the fault delimitation platform, and the platform needs to respond to these alarms for further analysis and processing.

[0067] In the specific implementation, refer to Figure 2 When an alarm occurs, the cloud system pushes the relevant alarms in real time through HTTP through the rule engine (and also pushes the corresponding recovery events), and automatically triggers the fault delimitation function. The specific process is as follows: For the alarms of the same application system within the same minute, only the first alarm A pushed in real time will trigger the delimitation function (generate a delimitation ID), and other alarms B and C do not need to trigger the delimitation repeatedly, but only need to be associated with this alarm A and the delimitation ID (that is, the delimitation ID corresponds to alarms A, B, and C). After the delimitation conclusion is obtained (no need to write back when there is no result), the result needs to be called The rule engine interface is written back to the event (add delimitation completed before the alarm description), and it is saved and displayed in the historical delimitation list. For alarms D and E after 1 minute, alarm D will trigger delimitation at this time, and E will be associated with D, and the above steps will be repeated. Similarly, the prophet event platform will also push recovery events in real time. When alarms A, B, and C all generate recovery events, it is considered that the fault corresponding to this delimitation conclusion has been restored. At this time, the status of the delimitation conclusion needs to be changed to restored.

[0068] In the specific implementation, this application divides the cloud system into multiple hierarchical architectures, each of which may contain different components or services. The fault demarcation platform needs to obtain information about these hierarchical architectures in order to understand the organizational structure of the system and the relationship between the components. Furthermore, in addition to the architectural information, the fault demarcation platform also needs to obtain indicator data related to these hierarchical architectures. These indicator data may include performance indicators, resource usage, error logs, etc. This indicator data is crucial for analyzing system status and identifying faults.

[0069] In the specific implementation, this application proposes an automated fault management process, in which the fault demarcation platform collects the system's architecture information and related indicator data after receiving abnormal alarms from the cloud system, so as to analyze and locate the fault. This process helps to quickly identify and solve problems in the cloud environment, thereby improving the reliability and stability of the system.

[0070] Step S200, performing abnormality judgment on the first indicator data to obtain a first abnormality result;

[0071] In the specific implementation, abnormality judgment refers to the platform using a specific algorithm or tool to analyze the first indicator data to identify any behavior or value that does not conform to the normal operation mode. Abnormality judgment methods include but are not limited to statistical analysis, machine learning models, threshold comparison and other methods.

[0072] In the specific implementation, after completing the abnormality judgment, the platform will generate a result, which indicates whether there is an abnormality in the analyzed indicator data. This result may be binary (i.e. abnormal or normal), or it may be a more detailed description indicating the specific abnormality type, severity or possible impact, which is not specifically limited here.

[0073] In a specific implementation, the platform performs abnormality judgment on the first indicator data to obtain a first abnormal result, including:

[0074] Determine the weight coefficients of indicators of different hierarchical structures; based on the weight coefficients, calculate the health index of the first indicator data through a preset time series anomaly detection algorithm to obtain the health index value of each hierarchical structure; based on the health index value, determine the corresponding first anomaly result.

[0075] In the specific implementation, the anomaly judgment of this application mainly starts from the high-availability architecture of the cloud system, combines various indicator data and time series anomaly detection algorithms (mainly single-indicator or multi-indicator time series anomaly detection algorithms) to obtain the health status of each level of architecture.

[0076] In another embodiment, in order to achieve decoupling of functional modules, the exception judgment logic can also be implemented by the business monitoring system and provide services through an interface. The fault delimitation platform does not assume the function of exception calculation, and the fault discovery platform (i.e., the business monitoring system) uniformly handles the exception calculation logic. The delimitation platform directly obtains the exception results through the interface, and then aggregates them according to the exception results to finally obtain the exception conclusion, effectively shortening the judgment path and reducing time consumption.

[0077] In the specific implementation, the platform first needs to determine the weight coefficients of the indicators of different hierarchical structures, where the weight coefficient is a numerical value that represents the importance of each indicator in the overall evaluation. Specifically, the determination of the weight coefficient can be achieved through expert scoring, data analysis or statistical methods, such as the analytic hierarchy process (AHP). Secondly, the platform can use the preset time series anomaly detection algorithm to analyze the collected first indicator data. The time series anomaly detection algorithm can be a statistical method (such as z-score, moving average, box plot, etc.) or a machine learning-based method (such as isolation forest, LSTM Autoencoder, etc.). The platform uses the above algorithm and the determined weight coefficient to calculate the health index for the indicator data of each level. The health index is a comprehensive indicator that reflects the performance or health status of each hierarchical architecture within a given time series. Finally, based on the calculated health index value, the platform determines whether there are anomalies in the architecture of each level of the system. This process may involve setting thresholds. When the health index exceeds or falls below these thresholds, the platform will mark it as an anomaly. The above process is a continuous process from data collection to anomaly detection, which aims to evaluate and ensure the stability and performance of the cloud system through quantitative analysis. Through this method, the platform can detect potential problems in a timely manner and take appropriate measures to repair or optimize them.

[0078] Step S300: Based on the first hierarchical architecture information, the first abnormal result is aggregated layer by layer to determine the first fault handling object of the cloud system and the corresponding first fault handling solution.

[0079] In a specific implementation, the layer-by-layer aggregation processing refers to starting from the lowest-level architecture information, analyzing and processing the abnormal results layer by layer in order to determine the scope of the fault and the corresponding processing plan. The present application quickly defines the scope and boundaries of the abnormal problem by aggregating the abnormal results of multiple-level architectures layer by layer. After defining the scope and boundaries of the abnormal problem, the fault handling object and the corresponding first fault handling plan can be determined without determining the root cause of the abnormality, thereby improving the efficiency of fault handling.

[0080] In the specific implementation, through the layer-by-layer convergence processing, the platform can determine the specific fault point or fault object in the system. The fault handling object may be a specific hardware component, a software service, an application, or any other system part that may cause the failure. Once the fault handling object is determined, a corresponding fault handling plan needs to be formulated. This plan should include steps on how to repair or mitigate the fault, which may include restarting the service, replacing hardware, updating software, adjusting configuration, and other measures.

[0081] In a specific implementation, the platform aggregates the first abnormal result layer by layer based on the first hierarchical architecture information, and determines the first fault handling object of the cloud system and the corresponding first fault handling solution, including:

[0082] Based on the first hierarchical architecture information, determine the granularity information of each hierarchical architecture, wherein the granularity information refers to the size information of the resource scope involved by the hierarchical architecture in the cloud system; based on the granularity information, sort the hierarchical architectures from small to large to obtain sorted hierarchical architectures; based on the first abnormal result, judge the abnormal results of each sorted hierarchical architecture layer by layer to obtain the maximum granularity hierarchical architecture with abnormalities, and use the maximum granularity hierarchical architecture with abnormalities as the first fault handling object;

[0083] In the specific implementation, the platform needs to analyze each layer of the cloud system architecture and determine the granularity information of each layer. Granularity information refers to the range of resources involved in the system at that layer, including the scale and complexity of hardware resources, software services, data flows, etc.

[0084] In the specific implementation, the platform sorts all hierarchical structures according to the size of the resource scope based on the determined granularity information. This sorting process is to help identify the hierarchical structure from the finest granularity (small scope) to the coarsest granularity (large scope). For example, refer to Figure 3 , starting from the deployment architecture diagram of the cloud system, with applications as the object, from top to bottom are: application, region, availability zone, cluster, instance. Different types of application deployment architectures have subtle differences. Applications are classified according to different architecture types. Here, it is necessary to rely on centralized management and constraints on application deployment types and clarify the deployment standards of various types of applications.

[0085] In the specific implementation, the platform uses the first abnormal result obtained previously to perform abnormal judgment on the sorted hierarchical architecture layer by layer. This process may involve analyzing the indicator data of each layer to determine whether there is an abnormality. In the layer-by-layer abnormality judgment process, the platform identifies the maximum granularity hierarchical architecture where the abnormality exists. This means that the problem is delimited at this level, and once the maximum granularity hierarchical architecture with the abnormality is determined, it will be used as the primary fault handling object. Due to the high availability characteristics, the corresponding standardized handling actions can be executed immediately without knowing the root cause, which greatly improves the efficiency of fault handling.

[0086] In a specific implementation, the platform determines the abnormal results of each hierarchical structure after sorting based on the first abnormal result, and obtains the maximum granularity hierarchical structure with abnormalities, including:

[0087] Based on the sorted hierarchical structures, a first hierarchical structure is determined; based on the abnormal result, whether the first hierarchical structure has an abnormality is judged; if the first hierarchical structure has an abnormality, based on the first abnormal result, whether the next hierarchical structure has an abnormality is judged, until the current hierarchical structure has no abnormality, then the previous level of the hierarchical structure without the abnormality is used as the maximum granularity hierarchical structure with the abnormality; based on the abnormal result of the first fault handling object, a corresponding first fault handling plan is determined.

[0088] In the specific implementation, first, the platform selects the lowest level from the previously sorted hierarchical architecture as the first-level architecture, and then uses the collected anomaly detection results to determine whether there is a problem or anomaly in this first-level architecture. If the first-level architecture does have an anomaly, then continue to go deeper into the next-level architecture, that is, based on the anomaly results of the first-level architecture, further determine whether the next-level architecture has an anomaly. The above process will continue, layer by layer, until the currently checked hierarchical architecture is found to have no anomalies. Once a level without anomalies is found, its immediate upper level architecture is identified as the highest granularity hierarchical architecture with anomalies. This means that the level with the largest possible impact of the problem has been located. Finally, after determining the highest granularity hierarchical architecture with anomalies, formulate a corresponding fault handling plan based on the anomaly results of this level. This plan should include specific steps, measures or strategies to solve or alleviate the identified anomalies.

[0089] In practice, the above process is a systematic, hierarchical approach to fault diagnosis that aims to accurately define the scope and boundaries of abnormal problems and develop appropriate solutions for each identified problem. This approach helps improve the efficiency and accuracy of fault response and ensures that the system can quickly return to normal operation.

[0090] In the specific implementation, the delimitation conclusions are aggregated from the bottom up based on the high availability architecture, and the final fault handling objects and recommended handling solutions are obtained by sorting according to the impact size (application>Region>AZ>cluster>instance). The specific application delimitation conclusion aggregation logic is shown in Table 1 below:

[0091] Table 1

[0092]

[0093] In the specific implementation, the purpose of fault demarcation is to analyze the fault, find effective and disposable objects for fault processing and finally restore the business. Based on the conclusion of demarcation, the fault demarcation module will automatically call the corresponding emergency disposal interface, and automatically create an emergency disposal task after pre-filling the relevant parameters. Users can directly perform one-click disposal and efficiently execute tasks without manually setting complex parameters, thereby shortening the time for fault resolution and further ensuring business continuity and stability.

[0094] In a specific implementation, the platform determines the steps of a corresponding fault handling solution based on the abnormal result of the fault handling object, including:

[0095] A fault handling solution library is obtained, wherein the fault handling solution library contains a solution for each abnormal result; based on the fault handling solution library, a solution is matched on the abnormal result of the fault handling object to obtain a corresponding fault handling solution.

[0096] In the specific implementation, first, the platform needs to access a pre-established fault handling solution library, which contains the corresponding solutions for various abnormal results. The fault handling solution library is an area for centralized storage and management of fault information, which is convenient for quickly finding and applying existing solutions when similar faults occur in devices or systems. Once the abnormal results of the fault handling object are determined, the platform will search for matching solutions in the fault handling solution library based on these results. The matching process may involve a detailed analysis of the abnormal results to ensure that the most appropriate solution is matched. Through the matching process, the system will provide one or more corresponding fault handling solutions for specific abnormal results. These solutions will guide the responsible personnel to repair or mitigate the faults to restore the normal operation of the cloud system.

[0097] In the specific implementation, the above process emphasizes the importance of systematic and preset solutions for fault handling, which can improve the efficiency and accuracy of troubleshooting, reduce system downtime, and ensure faster business recovery. The establishment and maintenance of a fault handling solution library is the key to ensure the success of this process, which usually requires continuous optimization by combining historical data, expert knowledge, and constantly updated technological developments.

[0098] In contrast to the root cause location method usually adopted in related technologies, a comprehensive top-down analysis of all relevant technology stack performances of the system is performed to determine the root cause of the fault and then develop a targeted solution. However, the root cause location in this method requires an in-depth analysis of the root cause of the problem, involving multiple levels and components. The location process may require more data association and analysis, resulting in low efficiency in fault handling. In this application, the fault delimitation platform responds to the abnormal alarm information sent by the cloud system, obtains the first-level architecture information of the cloud system and the first indicator data corresponding to each level of architecture, and then performs an abnormal judgment on the first indicator data to obtain a first abnormal result. Finally, based on the first-level architecture information, the first abnormal result is aggregated layer by layer to determine the first fault handling object of the cloud system and the corresponding first fault handling solution. It is understandable that in the present application, the cloud system is divided into multiple hierarchical architectures. When a fault occurs, that is, when the abnormal alarm information sent by the cloud system is received, the fault demarcation platform quickly defines the scope and boundaries of the abnormal problem by aggregating the abnormal results of multiple hierarchical architectures layer by layer. After defining the scope and boundaries of the abnormal problem, the fault handling object and the corresponding first fault handling plan can be determined without determining the root cause of the abnormality, thereby improving the efficiency of fault handling.

[0099] Based on the above first embodiment, the present application also proposes another embodiment, referring to Figure 4 The cloud system includes an application layer and a service layer, and the fault demarcation method further includes:

[0100] Step A100, in response to the abnormal alarm information sent by the cloud system, obtaining the second hierarchical architecture information of the application layer and the second indicator data corresponding to each hierarchical architecture;

[0101] It should be noted that, based on the architectural layering of the entire cloud system, this application also divides the fault delimitation into two parts: application layer delimitation and service layer delimitation according to the characteristics of layered decoupling of the technology stack. Among them, the service layer specifically includes the database and cloud service layer.

[0102] Understandably, referring to Figure 5 This application defines the application layer as the most important construction scenario, which is the first line of defense for emergency response to application system failures. This type of demarcation starts from the high-availability deployment architecture of the application, and divides the application from top to bottom into four levels: Region-AZ-Cluster-Instance (there may be slight differences between different applications) and uses them as the objects of disposal. The emergency response capability is mainly achieved through isolation and switching; when the overall application abnormality cannot be quickly recovered through the high-availability architecture, after eliminating problems such as program code logic, the application dependencies, that is, the abnormalities of the database or cloud service, should be considered. At this time, further demarcation of the cloud service layer is required.

[0103] Furthermore, when an abnormality occurs in a cloud service, it is necessary to delimit the cloud service with the cloud service as the core. At this time, the fault scope will be analyzed through the high-availability architecture of the cloud service to clarify the objects that can be dealt with and the means of dealing with them. The emergency response capability is guaranteed by the high-availability design of the cloud service. At the same time, the relevant conclusions will be written back to the affected application alarms to ensure that the causes of the application abnormalities are notified and handled in a timely manner.

[0104] In the specific implementation, application layer delimitation means: according to the high-availability architecture level of the application system (such as: Region-AZ-cluster / group-instance), combined with the business golden indicators and performance indicators of each layer of the architecture, the impact scope and disposable objects of application failure are clarified. The data sources and requirements of the specific application layer are referred to in Table 2 below:

[0105] Table 2

[0106]

[0107]

[0108] Step A200, performing abnormality judgment on the second indicator data to obtain a second abnormality result;

[0109] Step A300: Based on the second hierarchical architecture information, the second abnormal result is aggregated layer by layer to determine a second fault handling object and a corresponding second fault handling solution of the application layer;

[0110] In the specific implementation, the above fault demarcation for the application layer is the same as that in the first embodiment. For the specific process, refer to Figure 6 , it only further divides the entire cloud system into the application layer and the service layer, and performs the fault demarcation steps in a further divided order. Therefore, the specific steps refer to the first embodiment and are not described here.

[0111] Step A400: if the second fault handling solution is that the application layer cannot handle it, then obtain the third-level architecture information of the service layer and the third indicator data corresponding to each level architecture of the service layer;

[0112] It should be noted that if the second fault handling solution is that the application layer cannot handle it, it means that the fault demarcation range exceeds the application layer, or the fault problem is not in the application layer, so the same fault demarcation process is further performed on the service layer. Figure 7Specifically, service layer delimitation means: with the database and / or cloud service as the core, first check whether there are any abnormalities in the database and / or cloud service architecture, obtain the objects that can be dealt with, and then deal with them using the high-availability architecture capabilities of the database and / or cloud service; at the same time, it is also necessary to clearly indicate the scope of impact of the database and / or cloud service failure, such as a list of affected applications. For specific service layer data sources and requirements, please refer to Table 3 below:

[0113] Table 3

[0114]

[0115] In the specific implementation, the fault delimitation scenario of the service layer is also automatically triggered by the actual alarm. The alarm category is clarified through fields such as department, alarm level, alarm nature, monitoring tool, and event name to distinguish different observation scenarios. The specific implementation method is consistent with the trigger logic of the application layer delimitation.

[0116] In the specific implementation, refer to Figure 8 The deployment architecture of the service layer is obtained through the CMDB model to obtain the high-availability deployment architecture of cloud services. Different from the high-availability architecture of applications, which can be roughly summarized by four levels, the deployment of cloud services is more personalized, and it is necessary to communicate and clarify the architecture with the responsible parties of each cloud service.

[0117] Step A500, performing abnormality judgment on the third indicator data to obtain a third abnormality result;

[0118] Step A600: Based on the third hierarchical architecture information, the third abnormal result is aggregated layer by layer to determine the third fault handling object and the corresponding third fault handling solution of the service layer.

[0119] In the specific implementation, the abnormal judgment of service layer fault delimitation also starts from the deployment architecture to ensure that each layer or part in the architecture diagram is delimitable and tractable. The judgment of abnormality mainly comes from two aspects: 1. Event platform alarm: The cloud service responsible party formulates the alarm strategy and provides the matching conditions of relevant alarms. Such alarms can directly reveal the abnormal conditions of each layer of the cloud service high-availability deployment architecture. 2. Observability interface: The cloud service responsible party clarifies the calculation logic and provides the calculation capability and abnormal result return of the cloud service high-availability deployment architecture service capability indicator in the form of an interface.

[0120] In the specific implementation, the delimitation conclusion of the service layer is mainly composed of two parts: first, according to the high-availability architecture, it clearly points out the objects that can be handled for the fault, as well as the corresponding recommended handling solutions; second, it is necessary to determine the impact range of the fault, that is, which specific applications are affected by the current cloud service alarm. Similarly, the fault delimitation results of the cloud service layer will also be closely linked with the corresponding emergency handling actions to achieve the purpose of delimitation and handling, improve the efficiency of monitoring and emergency response, shorten the duration of faults, and reduce business losses.

[0121] In the specific implementation, the fault delimitation method of this application emphasizes a standardized fault handling process. This means that no matter which high-availability unit the fault occurs in, there are predefined observation indicators and standardized processing steps. This standardized handling method can ensure that when faced with various complex problems, operators can follow unified guidelines to reduce misjudgment and confusion, and improve the efficiency and consistency of problem solving. At the same time, standardized processing procedures can also promote knowledge inheritance and collaborative work between teams.

[0122] Furthermore, thanks to the division of high-availability units of the application system and the standardized rules and processing procedures, the demarcation judgment logic between each high-availability unit is independent of each other. Through the monitoring and demarcation of independent units, the boundaries of faults can be quickly defined in large-scale systems.

[0123] In the specific implementation, compared with the root cause location, the fault delimitation method of this application emphasizes that the application system must give full play to the high-availability design of the distributed architecture and the layered decoupling characteristics of the cloud infrastructure, and design the application into several independent high-availability units. When a fault occurs, the boundary of the fault is defined by observing the operation of each high-availability unit, and the object that needs to be handled is quickly identified and handled in a standardized manner without knowing the cause of the fault. Therefore, the fault delimitation method is suitable for large-scale distributed information system architectures in cloud environments, and can effectively ensure the timeliness of emergency response.

[0124] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the fault delimitation method of the present application. More simple transformations based on this technical concept are all within the protection scope of the present application.

[0125] This application also provides a fault demarcation device, please refer to Fig. 9 , the fault delimiting device comprises:

[0126] An acquisition module 10 is used to acquire first-level architecture information of the cloud system and first indicator data corresponding to each level architecture in response to abnormal alarm information sent by the cloud system;

[0127] A judgment module 20, configured to perform an abnormality judgment on the first indicator data to obtain a first abnormality result;

[0128] The aggregation module 30 is used to perform layer-by-layer aggregation processing on the first abnormal result based on the first hierarchical architecture information, and determine the first fault handling object and the corresponding first fault handling solution of the cloud system.

[0129] Optionally, the aggregation module 30 includes:

[0130] A granularity determination module, configured to determine granularity information of each hierarchical architecture based on the first hierarchical architecture information, wherein the granularity information refers to information on the size of a resource scope involved in the hierarchical architecture in the cloud system;

[0131] A sorting module, used to sort the hierarchical structures from small to large based on the granularity information to obtain sorted hierarchical structures;

[0132] A processing object determination module is used to determine the abnormal results of each hierarchical structure after the sorting based on the first abnormal result, obtain the maximum granularity hierarchical structure with abnormalities, and use the maximum granularity hierarchical structure with abnormalities as the first fault processing object;

[0133] The processing scheme determining module is used to determine a corresponding first fault processing scheme based on the abnormal result of the first fault processing object.

[0134] Optionally, the treatment object determination module includes:

[0135] A first-level structure determination module, configured to determine a first-level structure based on the sorted level structures;

[0136] An abnormality judgment module, used for judging whether there is an abnormality in the first hierarchical architecture based on the abnormal result;

[0137] The iterative judgment module is used to judge whether there is an abnormality in the next level architecture based on the first abnormality result if there is an abnormality in the first level architecture, until there is no abnormality in the current level architecture, and then the previous level of the level architecture without abnormality is used as the maximum granularity level architecture with abnormality.

[0138] Optionally, the treatment plan determination module includes:

[0139] A solution library acquisition module is used to acquire a fault handling solution library, wherein the fault handling solution library contains a handling solution for each abnormal result;

[0140] The solution matching module is used to match the abnormal results of the fault handling object based on the fault handling solution library to obtain a corresponding fault handling solution.

[0141] Optionally, the judging module 20 includes:

[0142] A weight coefficient determination module is used to determine the weight coefficients of different level architecture indicators;

[0143] A calculation module, configured to calculate the health index of the first indicator data based on the weight coefficient and by using a preset time series anomaly detection algorithm to obtain the health index value of each level architecture;

[0144] The abnormal result determination module is used to determine the corresponding first abnormal result based on the health index value.

[0145] Optionally, the fault delimiting device further includes:

[0146] An application layer architecture information acquisition module, used to obtain the second level architecture information of the application layer and the second indicator data corresponding to each level architecture in response to the abnormal alarm information sent by the cloud system;

[0147] A first abnormal result judgment module, used for judging the abnormality of the second indicator data to obtain a second abnormal result;

[0148] A first layer-by-layer aggregation module, configured to perform layer-by-layer aggregation processing on the second abnormal result based on the second hierarchical architecture information, and determine a second fault handling object and a corresponding second fault handling solution of the application layer;

[0149] A service layer architecture information acquisition module, if the second fault handling solution is that the application layer cannot handle it, then obtains the third-level architecture information of the service layer and the third indicator data corresponding to each level architecture of the service layer;

[0150] A second abnormal result judgment module, used for performing abnormal judgment on the third indicator data to obtain a third abnormal result;

[0151] The second layer-by-layer aggregation module is used to perform layer-by-layer aggregation processing on the third abnormal result based on the third hierarchical architecture information, and determine the third fault handling object and the corresponding third fault handling solution of the service layer.

[0152] The fault delimitation device provided by the present application adopts the fault delimitation method in the above embodiment, which can solve the technical problem of fault delimitation. Compared with the prior art, the beneficial effects of the fault delimitation device provided by the present application are the same as the beneficial effects of the fault delimitation method provided by the above embodiment, and other technical features in the fault delimitation device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0153] The present application provides a fault demarcation device, which includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the fault demarcation method in the above-mentioned embodiment 1.

[0154] Reference below Fig.10 , which shows a schematic diagram of the structure of a fault demarcation device suitable for implementing the embodiment of the present application. The fault demarcation device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.10 The fault demarcation device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0155] like Fig.10 As shown, the fault delimiting device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the fault delimiting device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the fault demarcation device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a fault demarcation device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0156] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0157] The fault delimitation device provided by the present application adopts the fault delimitation method in the above embodiment to solve the technical problem of fault delimitation. Compared with the prior art, the beneficial effects of the fault delimitation device provided by the present application are the same as the beneficial effects of the fault delimitation method provided by the above embodiment, and other technical features in the fault delimitation device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0158] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0159] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0160] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the fault delimitation method in the above-mentioned embodiment.

[0161] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0162] The computer-readable storage medium may be included in the fault demarcation device; or may exist independently without being assembled into the fault demarcation device.

[0163] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the fault demarcation device, the fault demarcation device is enabled to: demarcate the fault.

[0164] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0165] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0166] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0167] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned fault delimitation method, and can solve the technical problem of fault delimitation. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the fault delimitation method provided in the above-mentioned embodiment, and will not be repeated here.

[0168] The present application also provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, the steps of the fault delimitation method as described above are implemented.

[0169] The computer program product provided by the present application can solve the technical problem of fault delimitation. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the fault delimitation method provided by the above embodiment, which will not be described in detail here.

[0170] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A fault demarcation method, characterized in that: Applied to a fault demarcation platform, the fault demarcation platform is communicatively connected with a cloud system, and the fault demarcation method includes: In response to the abnormal alarm information sent by the cloud system, obtaining the first-level architecture information of the cloud system and the first indicator data corresponding to each level architecture; Performing abnormality judgment on the first indicator data to obtain a first abnormality result; Based on the first hierarchical architecture information, the first abnormal result is aggregated layer by layer to determine a first fault handling object of the cloud system and a corresponding first fault handling solution.

2. The fault demarcation method according to claim 1, characterized in that: The step of performing layer-by-layer aggregation processing on the first abnormal result based on the first hierarchical architecture information to determine a first fault handling object of the cloud system and a corresponding first fault handling solution includes: Based on the first hierarchical architecture information, determine the granularity information of each hierarchical architecture, wherein the granularity information refers to the size information of the resource scope involved in the hierarchical architecture in the cloud system; Based on the granularity information, sorting the hierarchical structures from small to large to obtain sorted hierarchical structures; Based on the first abnormal result, the abnormal result of each hierarchical structure after the sorting is judged layer by layer to obtain the maximum granularity hierarchical structure with abnormality, and the maximum granularity hierarchical structure with abnormality is used as the first fault handling object; Based on the abnormal result of the first fault handling object, a corresponding first fault handling solution is determined.

3. The fault demarcation method according to claim 2, characterized in that: The step of judging the abnormal results of each hierarchical structure after sorting based on the first abnormal result to obtain the maximum granularity hierarchical structure with abnormalities includes: Based on the sorted hierarchical structures, determining a first hierarchical structure; Based on the abnormal result, determining whether the first hierarchical architecture has an abnormality; If there is an exception in the first hierarchical structure, based on the first exception result, it is determined whether there is an exception in the next hierarchical structure, until there is no exception in the current hierarchical structure, and the previous level of the hierarchical structure without the exception is used as the maximum granularity hierarchical structure with the exception.

4. The fault demarcation method according to claim 2, characterized in that: The step of determining a corresponding fault handling solution based on the abnormal result of the fault handling object includes: Acquire a fault handling solution library, wherein the fault handling solution library contains a handling solution for each abnormal result; Based on the fault handling solution library, the abnormal results of the fault handling object are matched with solutions to obtain corresponding fault handling solutions.

5. The fault demarcation method according to claim 1, characterized in that: The step of performing abnormality judgment on the first indicator data to obtain a first abnormal result includes: Determine the weight coefficients of different levels of architectural indicators; Based on the weight coefficient, a health index is calculated for the first indicator data by using a preset time series anomaly detection algorithm to obtain a health index value of each level architecture; Based on the health index value, a corresponding first abnormal result is determined.

6. The fault demarcation method according to claim 1, characterized in that: The cloud system includes an application layer and a service layer, and the fault demarcation method further includes: In response to the abnormal alarm information sent by the cloud system, obtaining the second level architecture information of the application layer and the second indicator data corresponding to each level architecture; Performing an abnormality judgment on the second indicator data to obtain a second abnormality result; Based on the second hierarchical architecture information, the second abnormal result is aggregated layer by layer to determine a second fault handling object and a corresponding second fault handling solution of the application layer; If the second fault handling solution is that the application layer cannot handle it, obtaining the third-level architecture information of the service layer and the third indicator data corresponding to each level architecture of the service layer; Performing abnormality judgment on the third indicator data to obtain a third abnormal result; Based on the third hierarchical architecture information, the third abnormal result is aggregated layer by layer to determine a third fault handling object and a corresponding third fault handling solution of the service layer.

7. A fault demarcation device, characterized in that: The device comprises: An acquisition module, configured to acquire first-level architecture information of the cloud system and first indicator data corresponding to each level of architecture in response to abnormal alarm information sent by the cloud system; A judgment module, used for performing an abnormality judgment on the first indicator data to obtain a first abnormality result; The aggregation module is used to perform layer-by-layer aggregation processing on the first abnormal result based on the first hierarchical architecture information, and determine the first fault handling object and the corresponding first fault handling plan of the cloud system.

8. A fault demarcation device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the fault delimitation method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the fault delimitation method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the fault delimitation method according to any one of claims 1 to 6 are implemented.

Images