Big data operation and maintenance management method, device and equipment and readable storage medium
By performing multi-dimensional analysis of big data cluster information and automated fault repair, the problem that the existing technology cannot meet the operation and maintenance needs of big data clusters is solved, and operation and maintenance efficiency and system stability are improved.
Patent Information
- Application Number
- CN202411042379.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-12-06
AI Technical Summary
The existing technology cannot effectively meet the operation and maintenance needs of big data clusters, and a single management tool cannot meet the complexity and scale requirements of cluster operations and maintenance, resulting in low operation and maintenance efficiency.
By obtaining cluster information for multi-dimensional analysis, analytical data, and sending warning information when the warning indicators are met, matching automatic recovery plans for fault repair, and improving operation and maintenance efficiency.
It has achieved improvement in operation and maintenance efficiency of the big data platform, ensured system stability and performance, and met the efficient operation and maintenance needs of the big data cluster through automated fault repair and component parameter optimization.
Smart Images

Figure CN119088636A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of Internet technology, and in particular to a big data operation and maintenance management method, device, equipment and readable storage medium. Background Art
[0002] Big data operation and maintenance refers to the work of operating and maintaining big data systems. However, most users are still in the era of tool-based operation and maintenance. Operation and maintenance personnel can understand the health status of Hadoop component services through management tools. Users who are familiar with the enterprise version of Hadoop can complete daily operation and maintenance operations manually and semi-automatically through the management interface or related operation and maintenance tools.
[0003] However, as the number and scale of clusters continue to expand, the manual semi-automatic operation and maintenance of big data through tools can no longer meet the operation and maintenance needs of big data clusters. In addition, the operation and maintenance threshold of big data clusters is relatively high, and the operation and maintenance personnel are required to be strict. A single management tool can no longer effectively solve the problem. Summary of the invention
[0004] The present application provides a big data operation and maintenance management method, device, equipment and readable storage medium, which can at least solve the problem in the related technology that semi-automation of operation and maintenance tools can no longer meet the current big data cluster operation and maintenance needs.
[0005] A first aspect of an embodiment of the present application provides a big data operation and maintenance management method, including: Get the cluster information of the cluster to be managed; Performing multi-dimensional analysis on the cluster information and obtaining analysis data corresponding to the cluster information; When the data index of the analysis data reaches the warning index, sending a warning message to the management terminal; If a fault corresponding to the data indicator is detected, the fault is repaired by matching the automatic recovery plan according to the early warning information.
[0006] By adopting the above solution, the cluster information in the operation and maintenance platform is analyzed in multiple dimensions, and the corresponding analysis data is obtained. When the data indicators of the analysis data reach the warning indicators, the warning information is sent to the management terminal. If the data indicator opinion fails, the fault is repaired according to the corresponding automatic repair solution matched with the warning information, thereby improving the operation and maintenance efficiency of the big data platform.
[0007] Optionally, the cluster information includes cluster resources, and the step of performing multi-dimensional analysis on the cluster information and obtaining analysis data corresponding to the cluster information includes: Determining allocation information of the cluster resources within the corresponding cluster; The resource usage information of the cluster is analyzed according to the allocation information, and resource analysis data is obtained.
[0008] By adopting the above solution, the resource usage information of the cluster is determined according to the allocation information of the cluster resources in the corresponding cluster, and the change trend of each resource in the cluster is analyzed, so as to determine the resource analysis data in the cluster and improve the accuracy of cluster operation and maintenance management.
[0009] Optionally, the cluster information further includes log information, and the step of performing multi-dimensional analysis on the cluster information and obtaining analysis data corresponding to the cluster information further includes: Determine a search keyword for the log information; Performing matching analysis in the operation and maintenance knowledge base according to the search keywords; Obtain abnormal information existing in the log information according to the analysis result.
[0010] By adopting the above solution, matching analysis is performed in the operation and maintenance knowledge base according to the search keywords, so that abnormal information that may exist in the log information can be accurately found, thereby improving the accuracy of the log information.
[0011] Optionally, the analysis data includes the resource analysis data and the abnormal information, and when the data indicator of the analysis data reaches the warning indicator, the step of sending the warning information to the management terminal includes: Obtaining indicator data of the cluster information; Determine the early warning indicator according to the change trend of the indicator data; Performing estimated analysis on the data indicators of the analysis data; When the data indicator reaches the warning indicator, a warning message is sent to the management terminal according to the processing solution for the abnormal information in the operation and maintenance knowledge base.
[0012] By adopting the above solution, when the data indicator reaches the warning indicator, the processing solution corresponding to the abnormal information that reaches the warning indicator in the operation and maintenance knowledge base is determined, and the abnormal information and the processing solution are sent to the management terminal in the form of warning information, so that the operation and maintenance personnel can prevent possible problems in the cluster in time.
[0013] Optionally, if the fault corresponding to the data indicator is detected to occur, the step of matching the automatic recovery plan according to the warning information to perform fault repair includes: If a fault corresponding to the data indicator is detected, the target cluster where the fault occurs is determined according to the warning information; Matching an automatic recovery plan according to the fault information of the target cluster; If there is a solution to the abnormal information in the automatic recovery plan, the automatic handling process is triggered to repair the fault.
[0014] By adopting the above solution, the automatic recovery plan is matched according to the corresponding fault information in the early warning information, and the handling process of the automatic recovery plan is triggered. When the operation and maintenance personnel have not handled the cluster failure, the faulty cluster is automatically repaired, thereby improving the operation efficiency of the cluster.
[0015] Optionally, after the step of triggering the automatic handling process to repair the fault, the method further includes: Detecting whether the fault is automatically repaired; If the fault is not automatically repaired, obtaining a repair log of the management terminal repairing the fault; Determine a repair plan for the fault according to the repair log; The fault and the repair solution are updated to the operation and maintenance knowledge base.
[0016] Optionally, the method further includes: Detecting component parameters of the cluster to be managed within a preset period; Determining whether the configuration of the component parameters is a warning configuration; If the configuration is lower than the warning configuration, the optimal component parameter configuration is determined by testing the component parameters.
[0017] By adopting the above solution, the component parameters are detected within the preset period. If the configuration of the component parameters is lower than the warning configuration, the optimal component parameter configuration is determined by testing the component parameters, thereby effectively improving the performance and stability of the cluster to be managed and ensuring the normal operation of the system.
[0018] A second aspect of the embodiment of the present application provides a big data operation and maintenance management device, including: An acquisition module is used to obtain cluster information of the cluster to be managed; An analysis module, used to perform multi-dimensional analysis on the cluster information and obtain analysis data corresponding to the cluster information; A sending module, used for sending warning information to a management terminal when the data index of the analysis data reaches the warning index; The repair module is used to repair the fault by matching the automatic recovery plan according to the early warning information if a fault corresponding to the data indicator is detected.
[0019] A third aspect of an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the processor is used to execute a computer program stored in the memory, and when the processor executes the computer program, it implements each step of the big data operation and maintenance management method provided in the first aspect of the embodiment of the present application.
[0020] The fourth aspect of the embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, each step of the big data operation and maintenance management method provided in the first aspect of the embodiment of the present application is implemented.
[0021] In summary, the beneficial effects of this application are: 1. Perform multi-dimensional analysis on the cluster information in the operation and maintenance platform and obtain the corresponding analysis data. When the data indicators of the analysis data reach the warning indicators, send warning information to the management terminal. If a data indicator fails, the fault is repaired according to the corresponding automatic repair plan matched with the warning information, thereby improving the operation and maintenance efficiency of the big data platform.
[0022] 2. Detect component parameters within a preset period. If the configuration of component parameters is lower than the warning configuration, determine the optimal component parameter configuration by testing the component parameters, effectively improve the performance and stability of the managed cluster, and ensure the normal operation of the system. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 A flowchart of a big data operation and maintenance management method provided in an embodiment of the present application; Figure 2 A schematic diagram of a program module of a big data operation and maintenance management device provided in an embodiment of the present application; Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0024] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.
[0025] In order to solve the problem that semi-automation of operation and maintenance tools in the related art can no longer meet the current big data cluster operation and maintenance needs, the embodiment of the present application provides a big data operation and maintenance management method, such as Figure 1The following is a flow chart of a big data operation and maintenance management method provided in this embodiment. The big data operation and maintenance management method includes the following steps: Step 110: Obtain cluster information of the cluster to be managed.
[0026] Specifically, in this embodiment, the operation and maintenance platform refers to a unified self-service operation and maintenance platform that supports multi-cluster access. The cluster contains multiple tenants, including but not limited to application software, operating systems, and data platforms. As tenants in the cluster, the operation and maintenance platform is required to monitor them, generate a cluster diagram and cluster overview for the cluster, uniformly manage the tenant information of the cluster, allocate cluster resources to each tenant, grant data access rights to the tenants, and when an abnormal fault occurs in the cluster, the operation and maintenance platform needs to automatically repair the abnormal fault. Therefore, the operation and maintenance platform needs to obtain the cluster information corresponding to the cluster to be managed that is connected to the operation and maintenance platform. The cluster information includes but is not limited to the resource usage information of each tenant in the cluster and the log information of the cluster.
[0027] Step 120: Perform multi-dimensional analysis on the cluster information and obtain analysis data corresponding to the cluster information.
[0028] Specifically, in this embodiment, the multi-dimensional analysis of cluster information includes but is not limited to the analysis of cluster resource usage information and the analysis of operation logs within the cluster, and the analysis of the changing trends of computing and storage resources within the cluster, so as to promptly discover excessive resource waste or insufficient resources and avoid increased business risks due to resource problems. The analysis of logs requires the full collection of big data cluster-related logs, centralized log management and quasi-real-time search and analysis, and intelligent analysis of abnormal logs in combination with the knowledge base. After the analysis is completed, the analysis results corresponding to the cluster information are obtained, and based on the analysis results, it is determined whether the cluster is at risk of failure.
[0029] Step 130: When the data index of the analyzed data reaches the warning index, a warning message is sent to the management terminal.
[0030] Specifically, in this embodiment, when monitoring the hosts, jobs, and service status within the cluster, different warning indicators are set according to the indicator trends of each service in the analysis data, the business impact associated with the indicators is marked, and predictions and alarms are made on performance, failures, etc. When the data indicators of the analysis data reach the warning indicators, the corresponding warning information is sent to the management terminal.
[0031] Step 140: If a fault corresponding to the data indicator is detected, the fault is repaired by matching the automatic recovery plan according to the warning information.
[0032] Specifically, in this embodiment, the early warning information is used to remind the operation and maintenance personnel of the management terminal to perform maintenance on the data in the early warning information that exceeds the early warning index. If the operation and maintenance personnel fail to perform maintenance on the cluster in time according to the early warning information, which leads to a failure in the cluster, the early warning information is combined with the accumulation of operation and maintenance problems in the big data cluster scenario to automatically handle common problems in a timely manner, thus avoiding business problems caused by common failures during the rest of the personnel.
[0033] In an optional implementation of this embodiment, the steps of performing multi-dimensional analysis on cluster information and obtaining analysis data corresponding to the cluster information include: determining allocation information of cluster resources within the corresponding cluster; analyzing the resource usage information of the cluster according to the allocation information, and obtaining resource analysis data.
[0034] Specifically, cluster information includes cluster resources. In this embodiment, the operation and maintenance platform may be connected to multiple clusters at the same time, so it is necessary to analyze the allocation of cluster resources of each cluster to determine the allocation information of cluster resources in the corresponding cluster. Cluster resources include but are not limited to HDFS quotas, YARN resources, Impala resources, HBase resources, etc., where HDFS (Hadoop Distributed File System) is a distributed file system that allows administrators to set quotas for the number of names used and the amount of space used for a single directory. YARN (Yet Another Resource Negotiator) is a new Hadoop resource manager, which is a general resource management system that can provide unified resource management and scheduling for upper-level applications. Impala resources are query resources of the cluster. When querying, query statements will only be executed if all resources are available. HBase resources belong to the storage resources of the cluster. The usage of each resource in the cluster is further determined based on the allocation information, and the change trend of each resource in the cluster is analyzed, so as to determine the resource analysis data in the cluster.
[0035] In an optional implementation of the present embodiment, the step of performing a multi-dimensional analysis on the cluster information and obtaining analysis data corresponding to the cluster information also includes: determining a search keyword for the log information; performing a matching analysis in the operation and maintenance knowledge base based on the search keyword; and obtaining abnormal information in the log information based on the analysis result.
[0036] Specifically, cluster information also includes log data. The operation and maintenance platform is equipped with a unified underlying log access layer, which is responsible for collecting the operation logs of all components of each cluster, cluster MapReduce, Spark, Hive, scheduling logs, and operating system logs, etc. The self-service operation and maintenance platform provides a unified log portal interface to retrieve and analyze the logs of services and jobs, set log monitoring rules based on the log content, and push alarms for important logs. In this embodiment, the operation and maintenance platform will regularly collect all logs related to the big data cluster, conduct centralized log management and quasi-real-time search and analysis, and determine the corresponding search keywords in different types of logs. It should be understood that the search keywords corresponding to logs of different types, different contents, and different formats are all different, so as to ensure that the selected keywords can accurately represent the important information in the logs. Secondly, the keywords are determined by analyzing common problems, key processes, and system characteristics in the logs. The search keywords are matched and analyzed through the operation and maintenance knowledge base. It should be understood that the operation and maintenance knowledge base contains rich information, such as system configuration, troubleshooting experience, performance optimization suggestions, etc. When it is detected through the search keyword that there are multiple pieces of information related to the search keyword in the operation and maintenance knowledge base, other keywords in the log information are obtained again, and further matching analysis is performed on the detected information based on the other keywords until a unique analysis result is determined, and based on the content of the operation and maintenance knowledge base matched in the analysis result, it is judged whether there is abnormal information in the log information, and if so, the corresponding abnormal information is obtained.
[0037] In an optional implementation of the present embodiment, when the data indicator of the analysis data reaches the warning indicator, the step of sending a warning information to the management terminal includes: obtaining the indicator data of the cluster information; determining the warning indicator according to the changing trend of the indicator data; performing an estimated analysis on the data indicator of the analysis data; when the data indicator reaches the warning indicator, sending a warning information to the management terminal according to the processing solution for abnormal information in the operation and maintenance knowledge base.
[0038] Specifically, in this embodiment, the indicator data includes various performance parameters, resource usage information, etc. By analyzing historical data, key indicators in the indicator data that may indicate problems or anomalies can be determined, and the changing trends of the indicator data can be used to estimate and analyze the data indicators in the analysis data to determine whether the data indicators will reach the warning indicators of the corresponding indicator data. It should be understood that the analysis data includes resource analysis data and abnormal information in the log information. When the data indicator reaches the warning indicator, the processing plan corresponding to the abnormal information that reaches the warning indicator is determined in the operation and maintenance knowledge base, and the abnormal information and the processing plan are sent to the management terminal in the form of warning information, so that the operation and maintenance personnel can prevent problems that may occur in the cluster in a timely manner.
[0039] In an optional implementation of the present embodiment, if a fault corresponding to a data indicator is detected, the step of performing fault repair by matching an automatic recovery plan with the early warning information includes: if a fault corresponding to the data indicator is detected, determining a target cluster where the fault occurs according to the early warning information; matching an automatic recovery plan with the fault information of the target cluster; if a solution to the abnormal information exists in the automatic recovery plan, triggering an automatic handling process to repair the fault.
[0040] Specifically, in this embodiment, after the warning information is sent, if the fault corresponding to the data indicator still occurs, it means that the operation and maintenance personnel have not promptly maintained the possible faults in the cluster, so the operation and maintenance platform needs to automatically maintain the cluster where the fault occurs. When the fault corresponding to the data indicator is detected, the operation and maintenance will determine the target cluster where the fault occurs based on the warning information, and match the automatic recovery plan in the operation and maintenance knowledge base based on the abnormal information of the target cluster failure. If the automatic recovery plan in the operation and maintenance knowledge base has a solution corresponding to the abnormal information, the automatic disposal process is triggered to repair the fault. For example, according to the warning information, the system determines that the target cluster where the fault occurs is cluster A. After analyzing the abnormal information of cluster A, it is found that the increase in disk usage is due to a large number of temporary files generated by a specific application. The solution for this abnormal situation is matched in the automatic recovery solution library, that is, automatically cleaning up temporary files and releasing disk space. After the automatic disposal process is triggered, the operation and maintenance platform will automatically perform the following operations: 1. Find and delete temporary files generated by applications that generate a large number of temporary files. 2. Release disk space to ensure that disk usage returns to normal levels. After the repair is complete, check the cluster status and corresponding indicator data. If the indicator data returns to normal, it is determined that the fault repair of the target cluster is complete, and the automatic repair information is recorded.
[0041] Optionally, in actual applications, since the operation and maintenance knowledge base is a knowledge base generated based on the experience accumulation and continuous updating of multiple fault operation and maintenance, when encountering a fault that is not recorded in the operation and maintenance knowledge base, the fault cannot be repaired. Therefore, in this embodiment, after the step of triggering the automatic handling process to repair the fault, it is detected whether the fault is automatically repaired. If the fault is not automatically repaired, it is necessary to wait for the operation and maintenance personnel of the management terminal to repair the fault in the target cluster, obtain the repair log of the management terminal repairing the fault, and determine the repair plan for the fault based on the repair log, and update the fault information and repair plan of the fault to the operation and maintenance knowledge base.
[0042] Optionally, in actual applications, in order to ensure the efficient and stable operation of the cluster to be managed, its component parameters need to be detected within a preset period. Therefore, in this embodiment, in the process of detecting the cluster to be managed, it is first necessary to determine the scope and component parameters of the cluster to be managed, including but not limited to server performance parameters, network connection parameters, storage device parameters, etc., and regularly collect component parameter data according to the set detection cycle, and compare the collected component parameters with the warning configuration, wherein the warning configuration is a threshold value set according to the normal operating range and historical data of the system, and if the configuration of the component parameter is lower than the warning configuration, it indicates that there may be performance problems in the cluster to be managed, so it is necessary to perform more detailed tests on the component parameters, including but not limited to simulation and experimentation of different component parameter configurations to evaluate their impact on system performance, analyze the test results, determine which parameter configurations can provide the best performance and stability, and consider multiple factors, such as resource utilization, response time, reliability, etc., to select the optimal component parameter configuration, effectively improve the performance and stability of the cluster to be managed, and ensure the normal operation of the system.
[0043] Based on the implementation scheme of the above application, the cluster information of the cluster to be managed is obtained; the cluster information is analyzed in multiple dimensions, and the analysis data corresponding to the cluster information is obtained; when the data index of the analysis data reaches the warning index, the warning information is sent to the management terminal; if a fault corresponding to the data index is detected, the fault is repaired according to the automatic recovery scheme matched with the warning information. Through the implementation of the scheme of this application, the cluster information in the operation and maintenance platform is analyzed in multiple dimensions, and the corresponding analysis data is obtained. When the data index of the analysis data reaches the warning index, the warning information is sent to the management terminal. If the data index opinion fails, the fault is repaired according to the automatic repair scheme matched with the warning information, thereby improving the operation and maintenance efficiency of the big data platform.
[0044] Figure 2 A big data operation and maintenance management device is provided in an embodiment of the present application, and the big data operation and maintenance management device can be used to implement the big data operation and maintenance management method in the above-mentioned embodiment. Figure 2 As shown, the big data operation and maintenance management device mainly includes: An acquisition module 10 is used to acquire cluster information of the cluster to be managed; The analysis module 20 is used to perform multi-dimensional analysis on the cluster information and obtain analysis data corresponding to the cluster information; A sending module 30, used to send warning information to the management terminal when the data index of the analyzed data reaches the warning index; The repair module 40 is used to perform fault repair according to the automatic recovery plan matched with the early warning information if a fault corresponding to the data indicator is detected.
[0045] In an optional implementation of this embodiment, the analysis module is specifically used to: determine allocation information of cluster resources in the corresponding cluster; analyze resource usage information of the cluster according to the allocation information, and obtain resource analysis data.
[0046] In an optional implementation of this embodiment, the analysis module is further specifically used to: determine the search keywords of the log information; perform matching analysis in the operation and maintenance knowledge base according to the search keywords; and obtain abnormal information existing in the log information according to the analysis results.
[0047] Furthermore, in an optional implementation of the present embodiment, the sending module is specifically used to: obtain indicator data of cluster information; determine early warning indicators based on the changing trend of the indicator data; perform predictive analysis on the data indicators of the analysis data; when the data indicators reach the early warning indicators, send early warning information to the management terminal according to the processing plan for abnormal information in the operation and maintenance knowledge base.
[0048] Furthermore, in an optional implementation of the present embodiment, the repair module is specifically used for: if a fault corresponding to a data indicator is detected, determining the target cluster where the fault occurs based on the early warning information; matching an automatic recovery plan based on the fault information of the target cluster; and if there is a solution to the abnormal information in the automatic recovery plan, triggering an automatic handling process to repair the fault.
[0049] Furthermore, in an optional implementation of this embodiment, the big data operation and maintenance management device also includes: a detection module, a determination module, and an update module. The detection module is used to detect whether the fault is automatically repaired. The acquisition module is also used to obtain a repair log of the management terminal repairing the fault if the fault is not automatically repaired. The determination module is used to determine a repair plan for the fault based on the repair log. The update module is used to update the fault and the repair plan to the operation and maintenance knowledge base.
[0050] In an optional implementation of this embodiment, the big data operation and maintenance management device further includes: a judgment module. The detection module is also used to: detect the component parameters of the cluster to be managed within a preset period. The judgment module is used to: determine whether the configuration of the component parameters is a warning configuration. The determination module is used to: if the configuration is lower than the warning configuration, determine the optimal component parameter configuration by testing the component parameters.
[0051] According to the big data operation and maintenance management device provided by the present application, the cluster information of the cluster to be managed is obtained; the cluster information is analyzed in multiple dimensions, and the analysis data corresponding to the cluster information is obtained; when the data index of the analysis data reaches the warning index, the warning information is sent to the management terminal; if a fault corresponding to the data index is detected, the fault is repaired according to the automatic recovery plan matched with the warning information. Through the implementation of the present application, the cluster information in the operation and maintenance platform is analyzed in multiple dimensions, and the corresponding analysis data is obtained. When the data index of the analysis data reaches the warning index, the warning information is sent to the management terminal. If a data index fails, the fault is repaired according to the automatic repair plan matched with the warning information, thereby improving the operation and maintenance efficiency of the big data platform.
[0052] Figure 3 An electronic device provided in an embodiment of the present application. The electronic device can be used to implement the big data operation and maintenance management method in the aforementioned embodiment, mainly including: The memory 301, the processor 302, and the computer program 303 stored in the memory 301 and executable on the processor 302, the memory 301 and the processor 302 are connected by communication. When the processor 302 executes the computer program 303, the big data operation and maintenance management method in the aforementioned embodiment is implemented. The number of processors may be one or more.
[0053] The memory 301 may be a high-speed random access memory (RAM) memory, or a non-volatile memory (non-volatile memory), such as a disk memory. The memory 301 is used to store executable program codes, and the processor 302 is coupled to the memory 301 .
[0054] Furthermore, the present application also provides a computer-readable storage medium, which may be provided in the electronic device in the above embodiments. Figure 3 Memory in the illustrated embodiment.
[0055] The computer readable storage medium stores a computer program, and when the program is executed by the processor, the big data operation and maintenance management method in the aforementioned embodiment is implemented. Furthermore, the computer storable medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a RAM, a magnetic disk or an optical disk, and other media that can store program codes.
[0056] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.
[0057] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0058] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.
[0059] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a readable storage medium, including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned readable storage medium includes: U disk, mobile hard disk, ROM, RAM, disk or CD-ROM and other media that can store program codes.
[0060] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.
[0061] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0062] The above is a description of the big data operation and maintenance management method, device, equipment and readable storage medium provided by this application. For technical personnel in this field, according to the ideas of the embodiments of this application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on this application.
Claims
1. A big data operation and maintenance management method, characterized in that: include: Get the cluster information of the cluster to be managed; Performing multi-dimensional analysis on the cluster information and obtaining analysis data corresponding to the cluster information; When the data index of the analysis data reaches the warning index, sending a warning message to the management terminal; If a fault corresponding to the data indicator is detected, the fault is repaired by matching the automatic recovery plan according to the early warning information.
2. The big data operation and maintenance management method according to claim 1, characterized in that: The cluster information includes cluster resources, and the step of performing multi-dimensional analysis on the cluster information and obtaining analysis data corresponding to the cluster information includes: Determining allocation information of the cluster resources within the corresponding cluster; The resource usage information of the cluster is analyzed according to the allocation information, and resource analysis data is obtained.
3. The big data operation and maintenance management method according to claim 1, characterized in that: The cluster information also includes log information, and the step of performing multi-dimensional analysis on the cluster information and obtaining analysis data corresponding to the cluster information further includes: Determine a search keyword for the log information; Performing matching analysis in the operation and maintenance knowledge base according to the search keywords; Obtain abnormal information existing in the log information according to the analysis result.
4. The big data operation and maintenance management method according to claim 3 is characterized in that: The analysis data includes the resource analysis data and the abnormal information. When the data index of the analysis data reaches the warning index, the step of sending the warning information to the management terminal includes: Obtaining indicator data of the cluster information; Determine the early warning indicator according to the change trend of the indicator data; Performing estimated analysis on the data indicators of the analysis data; When the data indicator reaches the warning indicator, a warning message is sent to the management terminal according to the processing solution for the abnormal information in the operation and maintenance knowledge base.
5. The big data operation and maintenance management method according to claim 4 is characterized in that: If the fault corresponding to the data indicator is detected, the step of performing fault repair according to the early warning information matching the automatic recovery plan includes: If a fault corresponding to the data indicator is detected, the target cluster where the fault occurs is determined according to the warning information; Matching an automatic recovery plan according to the fault information of the target cluster; If there is a solution to the abnormal information in the automatic recovery plan, the automatic handling process is triggered to repair the fault.
6. The big data operation and maintenance management method according to claim 5, characterized in that: After the step of triggering the automatic handling process to repair the fault, the method further includes: Detecting whether the fault is automatically repaired; If the fault is not automatically repaired, obtaining a repair log of the management terminal repairing the fault; Determine a repair plan for the fault according to the repair log; The fault and the repair solution are updated to the operation and maintenance knowledge base.
7. The big data operation and maintenance management method according to claim 1, characterized in that: The method further comprises: Detecting component parameters of the cluster to be managed within a preset period; Determining whether the configuration of the component parameters is a warning configuration; If the configuration is lower than the warning configuration, the optimal component parameter configuration is determined by testing the component parameters.
8. A big data operation and maintenance management device, characterized in that: include: An acquisition module is used to obtain cluster information of the cluster to be managed; An analysis module, used to perform multi-dimensional analysis on the cluster information and obtain analysis data corresponding to the cluster information; A sending module, used for sending warning information to a management terminal when the data index of the analysis data reaches the warning index; The repair module is used to repair the fault by matching the automatic recovery plan according to the early warning information if a fault corresponding to the data indicator is detected.
9. An electronic device, characterized in that: The device comprises a memory and a processor, wherein: The processor is used to execute the computer program stored in the memory; When the processor executes the computer program, the steps in the big data operation and maintenance management method described in any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps in the big data operation and maintenance management method described in any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Operation and maintenance automation system and method
CN105323111A
Detection and recovery system and method for RabbitMQ cluster fault
CN110290012A
Operation and maintenance system and method
CN110659173A
Database cluster fault self-healing method and device
CN111459698A
Method for automatically testing, positioning and repairing a fault and storage medium
CN113656323A
