Distributed application operation and maintenance method, device, equipment and storage medium

Through the Zabbix distributed monitoring tool, the abnormalities in distributed applications are identified and repaired, and the problem of inefficient manual operation and maintenance in super-large-scale applications is solved, and automated repair and operation and maintenance efficiency are improved.

CN115473787BActive Publication Date: 2024-12-06SHANGHAI DONGPU INFORMATION TECH CO LTD
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202211025492.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-08-25
Publication Date
2024-12-06
Estimated Expiration
2042-08-25

AI Technical Summary

Technical Problem

In super-large-scale distributed applications, manual operation and maintenance efficiency is inefficient and it is difficult to deal with abnormal problems in a timely manner.

Method used

By calling Zabbix distributed monitoring tool to perform business monitoring on target distributed applications, identify exception alarm information, determine the exception application module and its corresponding target server, query and execute corresponding operation and maintenance scripts to repair exceptions. If the script is not queried, Distribute operation and maintenance work orders to the responsible engineer.

Benefits of technology

It realizes automated repair without manual intervention, improves operation and maintenance efficiency and accuracy, adapts to the expansion of distributed application scale, and ensures timely repair of abnormalities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115473787B_ABST
    Figure CN115473787B_ABST
Patent Text Reader

Abstract

The present application relates to the field of scaffold operation and maintenance, and discloses an operation and maintenance method, device, equipment and storage medium for distributed applications. The method includes: calling the Zabbix distributed monitoring tool to perform business monitoring on the target distributed application; responding to the abnormal alarm information sent by the Zabbix distributed monitoring tool, determining the abnormal application module in the target distributed application according to the abnormal alarm information; determining the target server node running the abnormal application module from the distributed cluster; identifying the target alarm type corresponding to the abnormal alarm information; determining the target script library corresponding to the abnormal application module, and querying the target operation and maintenance script of the target alarm type from the target script library; if the target operation and maintenance script is queried, calling the target server node to execute the target operation and maintenance script to repair the abnormal application module. The present invention can obtain the corresponding operation and maintenance script according to the abnormal alarm information to automatically repair the abnormal application module, thereby improving the operation and maintenance efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of infrastructure operation and maintenance, and in particular to an operation and maintenance method, device, equipment and storage medium for distributed applications. Background Art

[0002] Distributed Application refers to the way in which applications are distributed on different computers, and these different computers work together to complete a task through the network. In order to ensure that problems with external services of distributed applications can be handled in a timely manner, a corresponding monitoring system is needed to monitor and operate distributed applications.

[0003] In existing technologies, when an abnormality is detected in a distributed application, operation and maintenance engineers usually uniformly maintain the servers in each distributed cluster. However, in ultra-large-scale distributed applications, this manual operation and maintenance method will appear inefficient.

[0004] Application Contents

[0005] The embodiment of the present application provides an operation and maintenance method for distributed applications, which can obtain corresponding operation and maintenance scripts according to abnormal alarm information to automatically repair abnormal application modules, thereby solving the problem of low efficiency of manual operation and maintenance in ultra-large-scale distributed applications.

[0006] In a first aspect, an embodiment of the present application provides an operation and maintenance method for a distributed application, comprising: calling a Zabbix distributed monitoring tool to perform business monitoring on a target distributed application, wherein the target distributed application runs in a distributed cluster, and the distributed cluster includes multiple server nodes; in response to abnormal alarm information sent by the Zabbix distributed monitoring tool, determining an abnormal application module in the target distributed application where an abnormality occurs according to the abnormal alarm information; determining a target server node running the abnormal application module from the distributed cluster; identifying a target alarm type corresponding to the abnormal alarm information; determining a target script library corresponding to the abnormal application module, and querying a target operation and maintenance script of the target alarm type from the target script library; if the target operation and maintenance script is queried, calling the target server node to execute the target operation and maintenance script to repair the abnormal application module.

[0007] Based on the method provided in the embodiment of the present application, business monitoring of distributed applications is performed by using a distributed monitoring tool to obtain abnormal alarm information, and the abnormal application module and its corresponding target server are determined according to the abnormal alarm information, and the target alarm type corresponding to the abnormal alarm information is identified. Finally, the operation and maintenance script corresponding to the target alarm type is obtained and executed in the target server to repair the abnormal application module, without the need for human intervention, thereby improving the operation and maintenance efficiency and accuracy.

[0008] In one possible implementation, if the target operation and maintenance script is queried, then after calling the target server node to execute the target operation and maintenance script to repair the abnormal application module, it also includes: if the target operation and maintenance script is not queried, determining the target operation and maintenance engineer responsible for the target server; generating a target operation and maintenance work order containing abnormal alarm information in a preset work order platform; and dispatching the target operation and maintenance work order to the account of the target operation and maintenance engineer in the work order platform.

[0009] In this way, if the operation and maintenance script that can automatically repair the distributed application is not found, manual server operation and maintenance is performed to repair the distributed application anomaly, thereby ensuring that the anomaly can be repaired; at the same time, as the scale of distributed applications continues to expand, the one-to-many responsibility model of operation and maintenance engineers for servers is changed to a one-to-one responsibility model. By dispatching operation and maintenance work orders to the operation and maintenance engineers responsible for the abnormal server, the operation and maintenance efficiency is improved, and because the operation and maintenance engineers only need to be responsible for a single server, the accuracy of the operation and maintenance work is improved.

[0010] In one possible implementation, after a target operation and maintenance work order containing abnormal alarm information is generated in a preset work order platform, before the target operation and maintenance work order is dispatched to the account of a target operation and maintenance engineer in the work order platform, it also includes: obtaining an abnormality resolution strategy corresponding to the abnormal alarm information, and adding the abnormality resolution strategy to the target operation and maintenance work order.

[0011] In one possible implementation, obtaining the exception resolution strategy corresponding to the exception alarm information and adding the exception resolution strategy to the target operation and maintenance work order includes: determining the target policy library corresponding to the abnormal application module; querying the exception resolution strategy of the target alarm type from the target policy library, and adding the exception resolution strategy to the target operation and maintenance work order.

[0012] In this way, the exception resolution strategies corresponding to the abnormal alarm information are informed to engineers for reference, which improves the efficiency and accuracy of operation and maintenance.

[0013] In a possible implementation, identifying the target alarm type corresponding to the abnormal alarm information includes: calling a preset alarm type identification model to identify the abnormal alarm information, and obtaining the target alarm type corresponding to the abnormal alarm information.

[0014] In a possible implementation, before calling a preset alarm type recognition model to identify abnormal alarm information and obtaining a target alarm type corresponding to the abnormal alarm information, it also includes: obtaining a historical log file of a target distributed application, extracting multiple abnormal alarm information contained in the historical log file to construct an alarm information set; receiving annotation information input by a user, and annotating the corresponding alarm type for each abnormal alarm information in the alarm information set according to the annotation information; and performing classification recognition training on a preset initial network model based on the alarm information set to obtain an alarm type recognition model.

[0015] In this way, classification and identification are performed based on the data statistical characteristics of historical abnormal alarm information, so as to identify the accurate alarm type, and then obtain the accurate operation and maintenance script to accurately repair the abnormal application module in the distributed application.

[0016] In one possible implementation, in response to the abnormal alarm information sent by the Zabbix distributed monitoring tool, after determining the abnormal application module where the abnormality occurs in the target distributed application according to the abnormal alarm information, it also includes: counting the abnormal alarm information sent by the Zabbix distributed monitoring tool, and performing persistent storage of the counted abnormal alarm information based on a preset type of database.

[0017] In this way, various abnormal alarm information is persistently stored to facilitate subsequent abnormal query; and the above-mentioned alarm type recognition model can be continuously optimized and trained based on the persistently stored abnormal alarm information to make its recognition result more accurate, and then obtain accurate operation and maintenance scripts to accurately repair abnormal application modules in distributed applications.

[0018] In a second aspect, an embodiment of the present application provides an operation and maintenance device for a distributed application, including: a business monitoring module, which is used to call the Zabbix distributed monitoring tool to perform business monitoring on a target distributed application, wherein the target distributed application runs in a distributed cluster, and the distributed cluster includes multiple server nodes; an abnormality identification module, which is used to respond to the abnormal alarm information sent by the Zabbix distributed monitoring tool, and determine the abnormal application module in the target distributed application that has an abnormality according to the abnormal alarm information; a server determination module, which is used to determine the target server node running the abnormal application module from the distributed cluster; an alarm identification module, which is used to identify the target alarm type corresponding to the abnormal alarm information; a script query module, which is used to determine the target script library corresponding to the abnormal application module, and query the target operation and maintenance script of the target alarm type from the target script library; an abnormality repair module, which is used to call the target server node to execute the target operation and maintenance script to repair the abnormal application module if the target operation and maintenance script is queried.

[0019] In one possible implementation, the operation and maintenance device of the distributed application also includes: an engineer determination module, which is used to determine the target operation and maintenance engineer responsible for the target server if the target operation and maintenance script is not queried; a work order generation module, which is used to generate a target operation and maintenance work order containing abnormal alarm information in a preset work order platform; and a work order distribution module, which is used to distribute the target operation and maintenance work order to the account of the target operation and maintenance engineer in the work order platform.

[0020] In one possible implementation, the operation and maintenance device of a distributed application also includes: an engineer determination module, which is used to determine the target operation and maintenance engineer responsible for the target server if the target operation and maintenance script is not queried; a work order generation module, which is used to generate a target operation and maintenance work order containing abnormal alarm information in a preset work order platform; a strategy addition module, which is used to obtain the abnormality resolution strategy corresponding to the abnormal alarm information, and add the abnormality resolution strategy to the target operation and maintenance work order; and a work order dispatching module, which is used to dispatch the target operation and maintenance work order to the account of the target operation and maintenance engineer in the work order platform.

[0021] In one possible implementation, the policy adding module specifically includes: a policy library determination unit, used to determine the target policy library corresponding to the abnormal application module; a policy query unit, used to query the abnormality resolution strategy of the target alarm type from the target policy library, and add the abnormality resolution strategy to the target operation and maintenance work order.

[0022] In a possible implementation, the alarm identification module is specifically used to: call a preset alarm type identification model to identify abnormal alarm information, and obtain a target alarm type corresponding to the abnormal alarm information.

[0023] In one possible implementation, the alarm identification module specifically includes: a set construction unit, which obtains the historical log file of the target distributed application, and extracts multiple abnormal alarm information contained in the historical log file to construct an alarm information set; a data labeling unit, which is used to receive the labeling information input by the user, and label each abnormal alarm information in the alarm information set with the corresponding alarm type according to the labeling information; a model training unit, which is used to perform classification and recognition training on a preset initial network model based on the alarm information set to obtain an alarm type recognition model; a type recognition unit, which is used to call a preset alarm type recognition model to identify the abnormal alarm information, and obtain the target alarm type corresponding to the abnormal alarm information.

[0024] In a possible implementation, the operation and maintenance device of the distributed application also includes: a persistent storage module, which is used to count the abnormal alarm information sent by the Zabbix distributed monitoring tool, and perform persistent storage on the counted abnormal alarm information based on a preset type of database.

[0025] The technical effects of the second aspect and its various possible implementations are similar to the technical effects of the first aspect and its various possible implementations mentioned above, and will not be repeated here.

[0026] In a third aspect, an embodiment of the present application provides an operation and maintenance device for a distributed application, comprising: a memory and at least one processor, wherein instructions are stored in the memory; the at least one processor calls the instructions in the memory so that the operation and maintenance device for the distributed application executes each step of the above-mentioned operation and maintenance method for the distributed application.

[0027] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which instructions are stored, and when the computer-readable storage medium is run on a computer, the computer executes each step of the above-mentioned operation and maintenance method of the distributed application. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a flow chart of an embodiment of an operation and maintenance method for a distributed application provided in an embodiment of the present application;

[0029] Figure 2 This is a flow chart of another embodiment of an operation and maintenance method for a distributed application provided in an embodiment of the present application;

[0030] Figure 3 This is a flow chart of an embodiment of another distributed application operation and maintenance method provided in an embodiment of the present application;

[0031] Figure 4 A schematic diagram of the structure of an operation and maintenance device for a distributed application provided in an embodiment of the present application;

[0032] Figure 5 A schematic diagram of the structure of another distributed application operation and maintenance device provided in an embodiment of the present application;

[0033] Figure 6 A schematic diagram of the structure of an operation and maintenance device for a distributed application provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The embodiment of the present application provides an operation and maintenance method for distributed applications, which can obtain corresponding operation and maintenance scripts according to abnormal alarm information to automatically repair abnormal application modules, thereby solving the problem of low efficiency of manual operation and maintenance in ultra-large-scale distributed applications.

[0035] The technical scheme in the embodiment of the present application will be described below in conjunction with the accompanying drawings in the embodiment of the present application. Wherein, the terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable in appropriate circumstances, so that the embodiments described here can be implemented in a sequence other than the content illustrated or described here. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, the process, method, system, product or equipment comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0036] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0037] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0038] It is understandable that any part of the data acquisition or collection involved in this application has been authorized by the user.

[0039] It is understandable that the execution subject of the present application may be an operation and maintenance device of a distributed application, or may be a terminal or a server, which is not specifically limited here.

[0040] For ease of understanding, the present application embodiment takes the scheduling server as the execution subject as an example to illustrate the operation and maintenance method of the distributed application. The specific process of the present application embodiment is described below. Figure 1 , the embodiment of the present application provides an operation and maintenance method for a distributed application, including:

[0041] 101. Calling the Zabbix distributed monitoring tool to perform business monitoring on a target distributed application, wherein the target distributed application runs in a distributed cluster, and the distributed cluster includes multiple server nodes;

[0042] It can be understood that a distributed application is provided with services required by a distributed cluster including multiple server nodes. The distributed application is divided into multiple business application modules according to business type, and each business application module is supported by at least one server node in the distributed cluster. For example, a distributed application includes four business application modules A, B, C, and D, wherein business application module A is provided with service support by the first server node and the second server node, business application module B is provided with service support by the third server node, business application module C is provided with service support by the fourth server node, and business application module D is provided with service support by the fifth server node. The first server node, the second server node, the third server node, the fourth server node, and the fifth server node together constitute a distributed cluster.

[0043] Zabbix is ​​a monitoring tool that provides distributed system monitoring and network monitoring functions. It can monitor various network parameters, provide a flexible notification mechanism, and ensure server security and normal operation.

[0044] In a possible implementation, the distributed cluster includes multiple subclusters that deploy the same service to provide service support for the target distributed application at the same time, and each subcluster can provide complete business services for the target distributed application. In this way, when any subcluster has an abnormality, other subclusters in the distributed cluster can be switched to continue to provide service support for the target distributed application. At the same time, each subcluster can also be deployed in different geographical areas according to the geographical location, so as to provide high-speed target distributed application services to users in different geographical areas.

[0045] 102. In response to the abnormal alarm information sent by the Zabbix distributed monitoring tool, determine the abnormal application module in which the abnormality occurs in the target distributed application according to the abnormal alarm information;

[0046] Zabbix sends abnormal alarm information to the scheduling server based on the distributed subscription service system. The abnormal alarm information should at least include abnormal object information, and may optionally include abnormal description information, abnormal occurrence time, etc., which is not limited in the embodiments of the present application. The scheduling server locates the abnormal application module in the target distributed application according to the abnormal object information. For example, if the abnormal object information is the a service interface under the A application module, and the abnormal description information is the request response timeout, then the abnormal application module is the A application module. For another example, if the abnormal object information is the sample variable under the B application module, and the abnormal description information is that the sample variable cannot be obtained, then the abnormal application module is the B application module.

[0047] 103. Determine a target server node running an abnormal application module from the distributed cluster;

[0048] It is understandable that when deploying the target distributed application, each business application module corresponds to at least one server node providing services. The scheduling server determines the target server node (including the server address of the target server node) corresponding to the abnormal application module based on the corresponding deployment relationship between each business application module and each server node in the target distributed application.

[0049] 104. Identify the target alarm type corresponding to the abnormal alarm information;

[0050] Methods for identifying alarm types include, but are not limited to, keyword matching, clustering, data statistics and classification, which are not specifically limited in the embodiments of the present application.

[0051] In one possible design, the scheduling server matches the target alarm type corresponding to the abnormal alarm information based on a preset alarm type matching rule. In a specific implementation, all alarm types corresponding to the abnormal object are first obtained according to the abnormal object information, and then an alarm description list corresponding to each alarm type is obtained. The alarm description list includes at least one abnormal description information. The abnormal description information in the abnormal alarm information is fully matched in the alarm description list corresponding to each alarm type in turn to determine the target alarm type.

[0052] In another possible design, the scheduling server determines the target alarm type corresponding to the abnormal alarm information based on data statistics of historical abnormal alarm information of the target distributed application.

[0053] 105. Determine a target script library corresponding to the abnormal application module, and query a target operation and maintenance script of a target alarm type from the target script library;

[0054] It is understandable that the tester can pre-enumerate various abnormal problems that may occur in each application module under the target distributed application or count various abnormal problems that have occurred in the history of the target distributed application, and then design a corresponding operation and maintenance script for each abnormal problem. Among them, in order to improve the efficiency of script query, the operation and maintenance scripts can be classified and stored in multiple databases according to different application modules to obtain the script library corresponding to each application module. The programming language type of each operation and maintenance script is not specifically limited in the embodiments of the present application.

[0055] In one possible design, the scheduling server determines the target script logic library corresponding to the abnormal application module, and queries the script logic text corresponding to the target alarm type from the target script logic library, thereby inputting the script logic text into the preset script generation model to directly generate the target operation and maintenance script. It should be noted that the script generation model can be obtained by training the preset initial network model with a large number of public script codes containing annotated texts. The initial network model can be a recurrent neural network (RNN), Transformer, Generative Adversarial Nets (GAN) GAN, etc., which is not limited in the embodiments of the present application.

[0056] In this way, there is no need to generate operation and maintenance scripts for various abnormal problems in advance. It is only necessary to directly generate the corresponding operation and maintenance scripts according to the script logic text under the calculation of the generation algorithm, which reduces the script storage space, reduces the workload of test developers, and improves operation and maintenance efficiency.

[0057] 106. If the target operation and maintenance script is found, the target server node is called to execute the target operation and maintenance script to repair the abnormal application module.

[0058] In a specific implementation, the scheduling server accesses the target server node according to the server address of the target server node, and calls the target server node to obtain and execute the target operation and maintenance script from the target script library, thereby modifying the abnormal application module in the target distributed application.

[0059] In a possible design, the scheduling server also counts the abnormal alarm information sent by the Zabbix distributed monitoring tool, and performs persistent storage on the counted abnormal alarm information based on a preset type of database.

[0060] In this way, various abnormal alarm information is persistently stored to facilitate subsequent abnormal query; and the above-mentioned alarm type recognition model can be continuously optimized and trained based on the persistently stored abnormal alarm information to make its recognition result more accurate, and then obtain accurate operation and maintenance scripts to accurately repair abnormal application modules in distributed applications.

[0061] Based on the method provided in the embodiment of the present application, business monitoring of distributed applications is performed by using a distributed monitoring tool to obtain abnormal alarm information, and the abnormal application module and its corresponding target server are determined according to the abnormal alarm information, and the target alarm type corresponding to the abnormal alarm information is identified. Finally, the operation and maintenance script corresponding to the target alarm type is obtained and executed in the target server to repair the abnormal application module, without the need for human intervention, thereby improving the operation and maintenance efficiency and accuracy.

[0062] See also Figure 2 , the embodiment of the present application provides another distributed application operation and maintenance method, including:

[0063] 201. Calling a Zabbix distributed monitoring tool to perform business monitoring on a target distributed application, wherein the target distributed application runs in a distributed cluster, and the distributed cluster includes multiple server nodes;

[0064] 202. In response to the abnormal alarm information sent by the Zabbix distributed monitoring tool, determine the abnormal application module in which the abnormality occurs in the target distributed application according to the abnormal alarm information;

[0065] 203. Determine a target server node running an abnormal application module from the distributed cluster;

[0066] Among them, steps 201-203 are similar to the execution steps of the above steps 101-103, and will not be repeated here.

[0067] 204. Calling a preset alarm type recognition model to recognize abnormal alarm information, and obtaining a target alarm type corresponding to the abnormal alarm information;

[0068] It should be understood that the alarm type identification model is a classification model or a clustering model based on data statistical characteristics, and the embodiments of the present application do not specifically limit it.

[0069] In one possible design, the alarm type identification model is a clustering model obtained after unsupervised training. The scheduling server obtains the historical log files of the target distributed application, extracts multiple abnormal alarm information contained in the historical log files to construct an alarm information set, performs data clustering on the alarm information set based on a preset clustering algorithm to obtain multiple alarm information communities, receives manual alarm type annotations of the alarm information communities to obtain alarm information communities of multiple alarm types, and when the alarm type identification model is called to identify abnormal alarm information, the target alarm information community to which the abnormal alarm information belongs is calculated based on the clustering algorithm, and the target alarm type corresponding to the abnormal alarm information is obtained according to the alarm type annotated by the target alarm information community. The clustering algorithm includes but is not limited to K-means, KNN, DBSCAN, etc., and the embodiments of the present application do not limit it.

[0070] In one possible design, the alarm type recognition model is a classification model obtained after supervised training. The scheduling server obtains the historical log files of the target distributed application, extracts multiple abnormal alarm information contained in the historical log files to construct an alarm information set; receives the annotation information input by the user, and labels each abnormal alarm information in the alarm information set with the corresponding alarm type according to the annotation information; performs classification recognition training on a preset initial network model based on the alarm information set to obtain an alarm type recognition model. The initial network model includes but is not limited to a recurrent neural network (RNN), a Transformer, etc., which is not limited to the embodiments of the present application.

[0071] In a specific implementation, classification and recognition training is performed on a preset initial network model based on an alarm information set to obtain an alarm type recognition model, including: extracting a text feature matrix of the alarm information; reducing the dimension of the text feature matrix; performing dot product operations on the reduced text feature matrix and a pre-designed probability density matrix for each alarm type in sequence, and then normalizing the matrix to obtain a probability distribution of the alarm type for each alarm type, and determining an intermediate alarm type with the largest probability value in the probability distribution; calculating a loss result corresponding to the intermediate alarm type corresponding to the probability distribution based on a preset loss function, and adjusting the network parameters of the initial network model according to the loss result until the model converges to obtain an alarm type recognition model.

[0072] In this way, classification and identification are performed based on the data statistical characteristics of historical abnormal alarm information, so as to identify the accurate alarm type, and then obtain the accurate operation and maintenance script to accurately repair the abnormal application module in the distributed application.

[0073] In one possible design, the initial network model is Transformer. The scheduling server first pre-trains the Transformer based on a large-scale alarm information data set to obtain a target pre-trained model. Secondly, the target pre-trained model is migrated to the alarm information set of the target distributed application to perform classification training to obtain an alarm type recognition model.

[0074] In this way, pre-training is first performed based on a large-scale data set, and then the model is migrated to the actual application scenario for secondary training. The model convergence speed will be significantly improved, and only the network parameters of the pre-trained model need to be fine-tuned.

[0075] 205. Determine a target script library corresponding to the abnormal application module, and query a target operation and maintenance script of a target alarm type from the target script library;

[0076] 206. If the target operation and maintenance script is found, the target server node is called to execute the target operation and maintenance script to repair the abnormal application module.

[0077] Among them, steps 205-206 are similar to the execution steps of the above steps 105-106, and will not be repeated here.

[0078] Based on the method provided in the embodiment of the present application, classification and identification are performed according to the data statistical characteristics of historical abnormal alarm information, so as to identify the accurate alarm type, and then obtain the accurate operation and maintenance script to accurately repair the abnormal application module in the distributed application.

[0079] See also Figure 3 , the embodiment of the present application provides another distributed application operation and maintenance method, including:

[0080] 301. Calling a Zabbix distributed monitoring tool to perform business monitoring on a target distributed application, wherein the target distributed application runs in a distributed cluster, and the distributed cluster includes multiple server nodes;

[0081] 302. In response to the abnormal alarm information sent by the Zabbix distributed monitoring tool, determine the abnormal application module in which the abnormality occurs in the target distributed application according to the abnormal alarm information;

[0082] 303. Determine a target server node running an abnormal application module from the distributed cluster;

[0083] 304. Identify the target alarm type corresponding to the abnormal alarm information;

[0084] 305. Determine a target script library corresponding to the abnormal application module, and query a target operation and maintenance script of a target alarm type from the target script library;

[0085] 306. If the target operation and maintenance script is found, the target server node is called to execute the target operation and maintenance script to repair the abnormal application module;

[0086] Among them, steps 301-306 are similar to the execution steps of the above steps 101-106, and will not be repeated here.

[0087] 307. If the target operation and maintenance script is not found, determine the target operation and maintenance engineer responsible for the target server;

[0088] 308. Generate a target operation and maintenance work order containing abnormal alarm information in a preset work order platform;

[0089] 309. Dispatching the target operation and maintenance work order to the account of the target operation and maintenance engineer in the work order platform.

[0090] In an ultra-large-scale target distributed application, which includes multiple business application modules and a certain number of server nodes, multiple operation and maintenance engineers are required to maintain a certain number of server nodes respectively.

[0091] In one possible design, after a target operation and maintenance work order containing abnormal alarm information is generated in a preset work order platform, before the target operation and maintenance work order is dispatched to the account of a target operation and maintenance engineer in the work order platform, it also includes: obtaining an abnormality resolution strategy corresponding to the abnormal alarm information, and adding the abnormality resolution strategy to the target operation and maintenance work order.

[0092] In one possible design, obtaining the exception resolution strategy corresponding to the exception alarm information and adding the exception resolution strategy to the target operation and maintenance work order includes: determining the target policy library corresponding to the abnormal application module; querying the exception resolution strategy of the target alarm type from the target policy library, and adding the exception resolution strategy to the target operation and maintenance work order.

[0093] In this way, the exception resolution strategies corresponding to the abnormal alarm information are informed to engineers for reference, which improves the efficiency and accuracy of operation and maintenance.

[0094] Based on the method provided in the embodiment of the present application, if the operation and maintenance script that can automatically repair the distributed application is not found, manual server operation and maintenance is performed to repair the anomaly of the distributed application, thereby ensuring that the anomaly can be repaired; at the same time, as the scale of distributed applications continues to expand, the mode in which operation and maintenance engineers are responsible for one-to-many servers is changed to a one-to-one mode, and the operation and maintenance efficiency is improved by dispatching operation and maintenance work orders to the operation and maintenance engineers responsible for the abnormal server. In addition, since the operation and maintenance engineers only need to be responsible for a single server, the accuracy of the operation and maintenance work is improved.

[0095] The above describes the operation and maintenance method of the distributed application in the embodiment of the present application. The following describes the operation and maintenance device of the distributed application in the embodiment of the present application. Figure 4 , an embodiment of the present application provides an operation and maintenance device for a distributed application, including:

[0096] A business monitoring module 401 is used to call the Zabbix distributed monitoring tool to perform business monitoring on a target distributed application, wherein the target distributed application runs in a distributed cluster including multiple server nodes; an abnormality identification module 402 is used to respond to abnormal alarm information sent by the Zabbix distributed monitoring tool, and determine an abnormal application module in the target distributed application that has an abnormality according to the abnormal alarm information; a server determination module 403 is used to determine a target server node running the abnormal application module from the distributed cluster; an alarm identification module 404 is used to identify a target alarm type corresponding to the abnormal alarm information; a script query module 405 is used to determine a target script library corresponding to the abnormal application module, and query a target operation and maintenance script of the target alarm type from the target script library; an abnormality repair module 406 is used to call a target server node to execute the target operation and maintenance script to repair the abnormal application module if the target operation and maintenance script is found.

[0097] Based on the device provided in the embodiment of the present application, business monitoring of distributed applications is performed by using a distributed monitoring tool to obtain abnormal alarm information, and the abnormal application module and its corresponding target server are determined according to the abnormal alarm information, and the target alarm type corresponding to the abnormal alarm information is identified. Finally, the operation and maintenance script corresponding to the target alarm type is obtained and executed in the target server to repair the abnormal application module, without the need for human intervention, thereby improving the operation and maintenance efficiency and accuracy.

[0098] See also Figure 5 , the embodiment of the present application provides another distributed application operation and maintenance device, including:

[0099] A business monitoring module 401 is used to call the Zabbix distributed monitoring tool to perform business monitoring on a target distributed application, wherein the target distributed application runs in a distributed cluster including multiple server nodes; an abnormality identification module 402 is used to respond to abnormal alarm information sent by the Zabbix distributed monitoring tool, and determine an abnormal application module in the target distributed application that has an abnormality according to the abnormal alarm information; a server determination module 403 is used to determine a target server node running the abnormal application module from the distributed cluster; an alarm identification module 404 is used to identify a target alarm type corresponding to the abnormal alarm information; a script query module 405 is used to determine a target script library corresponding to the abnormal application module, and query a target operation and maintenance script of the target alarm type from the target script library; an abnormality repair module 406 is used to call a target server node to execute the target operation and maintenance script to repair the abnormal application module if the target operation and maintenance script is found.

[0100] In a possible design, the operation and maintenance device of the distributed application also includes:

[0101] The engineer determination module 407 is used to determine the target operation and maintenance engineer responsible for the target server if the target operation and maintenance script is not found; the work order generation module 408 is used to generate a target operation and maintenance work order containing abnormal alarm information in a preset work order platform; the strategy addition module 409 is used to obtain the abnormality resolution strategy corresponding to the abnormal alarm information, and add the abnormality resolution strategy to the target operation and maintenance work order; the work order distribution module 410 is used to distribute the target operation and maintenance work order to the account of the target operation and maintenance engineer in the work order platform.

[0102] In a possible design, the operation and maintenance device of the distributed application also includes:

[0103] The persistent storage module 411 is used to count the abnormal alarm information sent by the Zabbix distributed monitoring tool, and perform persistent storage on the counted abnormal alarm information based on a preset type of database.

[0104] In one possible design, the alarm identification module 404 specifically includes: a set construction unit 4041, which obtains the historical log file of the target distributed application, and extracts multiple abnormal alarm information contained in the historical log file to construct an alarm information set; a data labeling unit 4042, which is used to receive the labeling information input by the user, and label the corresponding alarm type for each abnormal alarm information in the alarm information set according to the labeling information; a model training unit 4043, which is used to perform classification and recognition training on a preset initial network model based on the alarm information set to obtain an alarm type recognition model; a type recognition unit 4044, which is used to call a preset alarm type recognition model to identify the abnormal alarm information, and obtain the target alarm type corresponding to the abnormal alarm information.

[0105] In one possible design, the policy adding module 409 specifically includes: a policy library determination unit 4091, used to determine the target policy library corresponding to the abnormal application module; a policy query unit 4092, used to query the abnormality resolution strategy of the target alarm type from the target policy library, and add the abnormality resolution strategy to the target operation and maintenance work order.

[0106] Based on the device provided in the embodiment of the present application, the modular design allows the hardware of each part of the operation and maintenance device of the distributed application to focus on the realization of a certain function, thereby maximizing the performance of the hardware. At the same time, the modular design also reduces the coupling between the modules of the device, making maintenance more convenient.

[0107] above Figures 4 to 5 The operation and maintenance device of the distributed application in the embodiment of the present application is described in detail from the perspective of modular functional entities. The operation and maintenance equipment of the distributed application in the embodiment of the present application is described in detail from the perspective of hardware processing.

[0108] Figure 6 It is a structural diagram of an operation and maintenance device for a distributed application provided in an embodiment of the present application. The operation and maintenance device 600 for the distributed application may have relatively large differences due to different configurations or performances, and may include one or more processors 610 (for example, one or more processors) and a memory 620, and one or more storage media 630 (for example, one or more massive storage devices) storing application programs 633 or data 632. Among them, the memory 620 and the storage medium 630 may be short-term storage or persistent storage. The program stored in the storage medium 630 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the operation and maintenance device 600 for the distributed application. Furthermore, the processor 610 may be configured to communicate with the storage medium 630 to execute a series of instruction operations in the storage medium 630 on the operation and maintenance device 600 for the distributed application.

[0109] The distributed application operation and maintenance device 600 may also include one or more power supplies 640, one or more wired or wireless network interfaces 650, one or more input and output interfaces 660, and / or one or more operating systems 631, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will appreciate that Figure 6 The illustrated structure of the operation and maintenance equipment for distributed applications does not constitute a limitation on the operation and maintenance equipment for distributed applications, and may include more or fewer components than illustrated, or a combination of certain components, or a different arrangement of components.

[0110] The present application also provides an operation and maintenance device for a distributed application, wherein the computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes each step of the operation and maintenance method for the distributed application in the above-mentioned embodiments.

[0111] The present application also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes each step of the operation and maintenance method of the distributed application.

[0112] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0113] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0114] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0115] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A distributed application operation and maintenance method, It is characterized in that The operation and maintenance method of the distributed application includes: Calling a Zabbix distributed monitoring tool to perform business monitoring on a target distributed application, wherein the target distributed application runs in a distributed cluster including a plurality of server nodes; In response to the abnormal alarm information sent by the Zabbix distributed monitoring tool, determine the abnormal application module in which the abnormality occurs in the target distributed application according to the abnormal alarm information; Determine a target server node running the abnormal application module from the distributed cluster; Identify the target alarm type corresponding to the abnormal alarm information; Determine a target script library corresponding to the abnormal application module, and query a target operation and maintenance script of the target alarm type from the target script library; If the target operation and maintenance script is found, calling the target server node to execute the target operation and maintenance script to repair the abnormal application module; The target alarm type corresponding to the abnormal alarm information is identified as follows: Obtaining a historical log file of the target distributed application, and extracting a plurality of abnormal alarm information contained in the historical log file to construct an alarm information set; Receiving annotation information input by a user, and annotating a corresponding alarm type for each abnormal alarm information in the alarm information set according to the annotation information; Based on the alarm information set, classification and recognition training is performed on a preset initial network model to obtain an alarm type recognition model; Calling a preset alarm type recognition model to identify the abnormal alarm information, and obtaining a target alarm type corresponding to the abnormal alarm information; The determining a target script library corresponding to the abnormal application module, and querying a target operation and maintenance script of the target alarm type from the target script library, includes: Determine a target script logic library corresponding to the abnormal application module, and query the script logic text corresponding to the target alarm type from the target script logic library; The script logic text is input into a preset script generation template to generate the target operation and maintenance script.

2. The distributed application operation and maintenance method according to claim 1, It is characterized in that If the target operation and maintenance script is found, then after calling the target server node to execute the target operation and maintenance script to repair the abnormal application module, the method further includes: If the target operation and maintenance script is not found, determining the target operation and maintenance engineer responsible for the target server; Generate a target operation and maintenance work order containing the abnormal alarm information in a preset work order platform; The target operation and maintenance work order is dispatched to the account of the target operation and maintenance engineer in the work order platform.

3. The distributed application operation and maintenance method according to claim 2, It is characterized in that After generating a target operation and maintenance work order including the abnormal alarm information in a preset work order platform, and before dispatching the target operation and maintenance work order to the account of the target operation and maintenance engineer in the work order platform, the method further includes: Obtain the exception resolution strategy corresponding to the exception alarm information, and add the exception resolution strategy to the target operation and maintenance work order.

4. The distributed application operation and maintenance method according to claim 3, It is characterized in that The obtaining of the exception resolution strategy corresponding to the exception alarm information and adding the exception resolution strategy to the target operation and maintenance work order includes: Determine a target policy library corresponding to the abnormal application module; The target alarm type exception resolution strategy is queried from the target strategy library, and the exception resolution strategy is added to the target operation and maintenance work order.

5. The operation and maintenance method of a distributed application according to any one of claims 1 to 4, It is characterized in that After responding to the abnormal alarm information sent by the Zabbix distributed monitoring tool and determining the abnormal application module having an abnormality in the target distributed application according to the abnormal alarm information, the method further includes: The abnormal alarm information sent by the Zabbix distributed monitoring tool is counted, and the counted abnormal alarm information is persistently stored based on a preset type of database.

6. An operation and maintenance device for distributed applications, It is characterized in that The distributed application operation and maintenance device includes: A business monitoring module, used to call the Zabbix distributed monitoring tool to perform business monitoring on a target distributed application, wherein the target distributed application runs in a distributed cluster including multiple server nodes; An abnormality identification module is used to respond to the abnormality alarm information sent by the Zabbix distributed monitoring tool, and determine the abnormal application module in which the abnormality occurs in the target distributed application according to the abnormality alarm information; A server determination module, used to determine a target server node running the abnormal application module from the distributed cluster; An alarm identification module is used to identify the target alarm type corresponding to the abnormal alarm information; A script query module, used to determine a target script library corresponding to the abnormal application module, and query a target operation and maintenance script of the target alarm type from the target script library; an abnormality repair module, for invoking the target server node to execute the target operation and maintenance script to repair the abnormal application module if the target operation and maintenance script is found; The alarm identification module specifically includes: a set construction unit, which is used to obtain the historical log file of the target distributed application, and extract multiple abnormal alarm information contained in the historical log file to construct an alarm information set; a data annotation unit, which is used to receive annotation information input by a user, and annotate the corresponding alarm type for each abnormal alarm information in the alarm information set according to the annotation information; a model training unit, which is used to perform classification and recognition training on a preset initial network model based on the alarm information set to obtain an alarm type recognition model; a type recognition unit, which is used to call a preset alarm type recognition model to identify the abnormal alarm information and obtain the target alarm type corresponding to the abnormal alarm information; the script query module is also used to: determine the target script logic library corresponding to the abnormal application module, and query the script logic text corresponding to the target alarm type from the target script logic library; input the script logic text into a preset script generation template to generate the target operation and maintenance script.

7. A distributed application operation and maintenance device, It is characterized in that The distributed application operation and maintenance device comprises: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instruction in the memory to enable the distributed application operation and maintenance device to perform each step of the distributed application operation and maintenance method as described in any one of claims 1-5.

8. A computer-readable storage medium having instructions stored thereon, It is characterized in that When the instructions are executed by the processor, the steps of the operation and maintenance method of the distributed application as described in any one of claims 1 to 5 are implemented.

Citation Information

Patent Citations

  • Abnormal operation environment restoration method and device, computer equipment and storage medium

    CA3129985A1

  • Processing method and processing system for system exceptions

    CN108737182A

  • A method and a device for monitoring distributed application services

    CN108959038A

  • Monitoring method, system, device and medium for distributed file storage cluster

    CN109522287A

  • Alarm information processing method and device, computer device and storage medium

    CN111522704A