An automated management system for managing an in-house elasticsearch cluster

Abstract

The application discloses an automatic management system for managing an enterprise internal elasticsearch cluster, which comprises ES cluster life cycle management, index management, account management, one-key analysis, alarm subscription and graphical monitoring and control. The ES cluster life cycle management can automatically manage the whole life cycle of the ES cluster without the intervention of operation and maintenance personnel, and can dynamically adjust the number of instances and resource configuration of the cluster according to actual resource requirements. The graphical monitoring and control can analyze and monitor various indexes of the ES through multidimensional indexes, helping operation and maintenance personnel and developers to quickly locate the cluster status, troubleshoot problems and optimize the cluster status. The application lowers the middleware use threshold of developers, enables the developers to enjoy the convenience and performance improvement brought by the management system, further liberates the operation and maintenance personnel, improves the system stability and reduces the daily workload of the operation and maintenance personnel.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of enterprise management, and in particular to an automatic management system for managing an enterprise internal elasticsearch cluster. BACKGROUND

[0002] Currently, the management system of Elasticsearch in the industry can help administrators monitor, manage and maintain Elasticsearch clusters. It provides a set of tools and functions that enable administrators to perform the following tasks: cluster monitoring and health management: administrators can monitor the status, performance and health of the entire Elasticsearch cluster through the management system, which provides real-time indicators and graphical interfaces to enable administrators to identify and solve any potential problems; configuration and management of indexes: administrators can use the management system to create, configure and manage Elasticsearch indexes. They can define the mapping of indexes, set the shards and replicas of indexes, perform optimization and maintenance operations on indexes; data backup and recovery: the management system usually provides the function of backing up and restoring data to ensure that data can be recovered in the event of failure or data loss. Administrators can back up data regularly and perform recovery operations when needed; security management: the management system provides security management functions for Elasticsearch clusters, including user authentication, access control and permission management. Administrators can configure users and roles and control who can access which resources and operations of the cluster.

[0003] However, the prior art does not have the ability to manage and control the entire ES cluster life cycle, and is biased towards the use of operation and maintenance personnel. Development personnel cannot enjoy the convenience and efficiency improvement brought by the management system, and cannot meet the needs of enterprise internal resource utilization rate statistics and budget billing. Therefore, in view of the above problems, an automatic management system for managing an enterprise internal elasticsearch cluster is proposed. SUMMARY

[0004] The purpose of the application is to provide an automatic management system for managing an enterprise internal elasticsearch cluster, which mainly solves the technical problem of how to further liberate operation and maintenance personnel and improve the work efficiency of operation and maintenance personnel.

[0005] An automatic management system for managing an enterprise internal elasticsearch cluster, including ES cluster life cycle management, index monitoring and graphical, alarm subscription, index management, account management, one-key analysis;

[0006] The ES cluster life cycle management: through this function, users can automatically manage the entire life cycle of the ES cluster without the need for operation and maintenance intervention; the number of instances and resource configuration of the cluster can also be dynamically adjusted according to actual resource needs;

[0007] The index monitoring and graphical display: through multi-dimensional analysis and monitoring of various indexes of ES, the cluster status can be quickly located, problems can be checked and the cluster state can be optimized for operation and maintenance personnel and developers;

[0008] The alarm subscription: users can configure alarms and subscribe according to needs or existing strategies, and obtain SMS or telephone reminders when the cluster has an abnormality;

[0009] The index management: through deep development of ES, near real-time index metadata capture is realized; index structure viewing, modification and closing can be performed in the function module;

[0010] The account management: management of ES system accounts is realized, in this function module, users can define ES accounts and authorize according to the index prefix, and the authorization type supports read-only, write-only and all three;

[0011] The one-key analysis: the one-key analysis function can diagnose cluster problems in real time in combination with multiple dimensions such as ES cluster state, network condition, index write rate, query rate, SLA, pending tasks, system and instance load, shard full load rate, index Pool, query Pool and ManagePool, and give humanized scientific suggestions.

[0012] Further, the specific implementation scheme of the ES cluster life cycle management is as follows:

[0013] With the help of the custom form function of the work order system, an ES automatic creation, destruction, expansion and contraction entrance is provided for users;

[0014] After the work order approval, the information submitted by the user will be submitted to the ES management system for further processing, and after the ES management system obtains the information of the approved work order, in combination with the detailed configuration of the package and the template file, a yaml file executable by the ElasticSearch Operator will be dynamically generated;

[0015] The ES management system submits the yaml file to the K8S service API exposed by the container management system, and the API interface submits the yaml file to the ElasticSearch Operator for running;

[0016] After the ElasticSearch Operator obtains the yaml file, it will perform cluster creation, destruction, and scaling operations according to the file description, thereby realizing the full life cycle management of the ES cluster.

[0017] Further, the index monitoring is generally divided into two parts: cluster monitoring and graphicalization, and instance monitoring and graphicalization.

[0018] Further, the cluster monitoring and graphicalization includes cluster state monitoring, JVM monitoring, CPU monitoring, query monitoring, write time-consuming monitoring, GC monitoring, pending tasks monitoring, and shard monitoring.

[0019] Further, the instance monitoring and graphicalization includes instance JVM monitoring, CPU monitoring, query time-consuming monitoring, query rate monitoring, index write rate monitoring, GC frequency monitoring, GC time-consuming monitoring, and rejection queue monitoring.

[0020] Further, the ES management system will report the resource situation and utilization of each cluster to the CMDB system at regular intervals. During the life cycle management of the ES cluster, the ES management system reports the cluster changes to the budget system every time the cluster changes and scales. When the budget is insufficient, a prompt and consumption bill are given. Users can also understand the resource usage or consumption details from the CMDB system or the budget system.

[0021] Further, ES cluster containerization is achieved using Elastic Operator and dynamic templates, realizing cluster creation, destruction, and scaling. Elastic Operator is a technology for deploying and managing Elasticsearch clusters in a Kubernetes environment. Elasticsearch is an open-source distributed search and analysis engine with powerful data processing and search capabilities.

[0022] Further, by integrating the custom ticket system, budget system, and CMDB system, developers can apply for ES clusters on demand through the ticket self-service according to the package configuration. The resource occupation, utilization, and machine room information are reported to the enterprise's internal budget system and CMDB resource management system.

[0023] Further, through system design and function development, ES cluster and node monitoring are completed, supporting the development and operation to view the ES node and index state and performance indicators. Custom alarms can be made according to demand, and the life cycle of the index and ES account permissions are managed.

[0024] Through the above-mentioned embodiments of the present application, by integrating the CMDB system and the budget system, the ES resource use and expense situation of each business line in the enterprise is clear at a glance, which provides scientific data support for guiding the enterprise to open up new sources and save resources, reasonably applying and using resources, and improving resource utilization. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, below will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0026] Figure 1 The system architecture diagram of an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the technical personnel in the art better understand the present application scheme, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0028] It should be noted that the terms "first", "second" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0029] In the present application, the terms "upper", "lower", "left", "right", "front", "back", "top", "bottom", "inner", "outer", "middle", "vertical", "horizontal", "lateral", "longitudinal" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings. These terms are mainly used to better describe the present application and its embodiments, and are not used to limit the indicated devices, elements or components to have a specific orientation, or to be constructed and operated in a specific orientation.

[0030] Furthermore, in addition to indicating direction or positional relationship, some of the aforementioned terms may also have other meanings. For example, the term "above" may also be used in certain situations to indicate a dependency or connection. Those skilled in the art can understand the specific meaning of these terms in this invention based on the specific circumstances.

[0031] Furthermore, the terms "installation," "setup," "equipped with," "connection," "linking," and "socketing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral structure; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium, or an internal connection between two devices, components, or parts. Those skilled in the art can understand the specific meaning of these terms in this invention based on the specific circumstances.

[0032] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0033] Please see Figure 1 As shown, an automated management system for managing an enterprise's internal Elasticsearch cluster includes ES cluster lifecycle management, metric monitoring and graphical representation, alarm subscription, index management, account management, and one-click analysis.

[0034] The ES cluster lifecycle management feature allows users to automatically manage the entire lifecycle of the ES cluster without operational intervention; it also allows for dynamic adjustment of the number of cluster instances and resource configuration based on actual resource requirements.

[0035] The aforementioned indicator monitoring and graphical representation: Through multi-dimensional analysis and monitoring of various Elasticsearch (ES) indicators, it helps operations and maintenance personnel and developers quickly locate cluster status, troubleshoot problems, and optimize cluster status.

[0036] The alarm subscription allows users to configure and subscribe to alarms according to their needs or existing policies, and receive SMS or telephone alerts when an anomaly occurs in the cluster.

[0037] The index management module enables near real-time crawling of index metadata through in-depth development of Elasticsearch; the index structure can be viewed, modified, and closed within this module.

[0038] The account management module manages Elasticsearch (ES) system accounts. In this module, users can define and authorize ES accounts according to the index prefix. The authorization types support read-only, write-only, and all.

[0039] The one-key analysis: the one-key analysis function can diagnose cluster problems in real time in combination with ES cluster state, network condition, index writing rate, query rate, SLA, pending tasks, system and instance load, shard full load rate, index Pool, query Pool, ManagePool multiple dimensions, and give humanized scientific suggestions;

[0040] Further, the specific implementation scheme of the ES cluster life cycle management is as follows:

[0041] With the help of the custom form function of the work order system, the user is provided with an ES automatic creation, destruction, expansion, and contraction entrance;

[0042] After the work order approval, the information submitted by the user is submitted to the ES management system for further processing. After the ES management system obtains the information of the approved work order, in combination with the detailed configuration of the package and the template file, a yaml file executable by the ElasticSearch Operator is dynamically generated;

[0043] The ES management system submits the yaml file to the K8S service API exposed by the container management system. The API interface submits the yaml file to the ElasticSearch Operator for running;

[0044] After the ElasticSearch Operator obtains the yaml file, it will execute cluster creation, destruction, expansion, and contraction operations according to the file description, thereby realizing the full life cycle management of the ES cluster.

[0045] Further, the index monitoring is divided into two parts, one is cluster monitoring and graphical, and the other is instance monitoring and graphical.

[0046] Further, the cluster monitoring and graphical include cluster state monitoring, JVM monitoring, CPU monitoring, query monitoring, write time-consuming monitoring, GC monitoring, pending tasks monitoring, and shard monitoring.

[0047] Further, the instance monitoring and graphical include instance JVM monitoring, CPU monitoring, query time-consuming monitoring, query rate monitoring, index writing rate monitoring, GC frequency monitoring, GC time-consuming monitoring, and rejection queue monitoring.

[0048] Further, the ES management system will report the resource condition and utilization of each cluster to the CMDB system at regular intervals. In the ES cluster life cycle management process, the ES management system reports the cluster changes to the budget system every time the cluster is expanded or scaled. When the budget is insufficient, a prompt and consumption bill are given. Users can also understand the resource usage or consumption details from the CMDB system or the budget system.

[0049] Further, the ES cluster containerization is realized by using Elastic Operator and dynamic templates to realize cluster creation, destruction, and capacity expansion. Elastic Operator is a technology for deploying and managing Elasticsearch clusters in a Kubernetes environment. Elasticsearch is an open-source distributed search and analysis engine with powerful data processing and search capabilities.

[0050] Further, by integrating the custom ticket system, budget system, and CMDB system, developers can apply for ES clusters through the ticket system according to the package configuration. The resource occupation, utilization, and machine room information are reported to the budget system and the CMDB resource management system.

[0051] Further, through system design and function development, ES cluster and node monitoring are completed, supporting the development and operation to view the ES node and index state and performance indicators. Custom alarms can be set according to demand, and the life cycle of the index and ES account permissions are managed.

[0052] The advantages of the present application are:

[0053] 1. The present application reduces the middleware use threshold of developers, allowing developers to enjoy the convenience and efficiency improvement brought by the management system.

[0054] 2. The present application further liberates the operation and maintenance personnel, improves the system stability, reduces the daily workload of the operation and maintenance personnel, and allows them to invest time and effort into more important matters.

[0055] 3. The present application provides intelligent fault prediction, capacity prediction, and multi-dimensional index analysis, making it easier for developers and operation and maintenance personnel to analyze and locate problems.

[0056] 4. The present application integrates the CMDB system and the budget system, allowing the enterprise internal business line ES resource usage and spending to be easily understood, providing scientific data support for guiding the enterprise internal open source and thrift, reasonable application and use of resources, and improving resource utilization.

[0057] The above merely describes the preferred embodiments of the present application and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An automated management system for managing an enterprise's internal Elasticsearch cluster, characterized in that: Includes ES cluster lifecycle management, metric monitoring and graphical representation, alarm subscription, index management, account management, and one-click analysis; The ES cluster lifecycle management feature allows users to automatically manage the entire lifecycle of the ES cluster without operational intervention; it also allows for dynamic adjustment of the number of cluster instances and resource configuration based on actual resource requirements. The aforementioned indicator monitoring and graphical representation: Through multi-dimensional analysis and monitoring of various Elasticsearch (ES) indicators, it helps operations and maintenance personnel and developers quickly locate cluster status, troubleshoot problems, and optimize cluster status. The alarm subscription allows users to configure and subscribe to alarms according to their needs or existing policies, and receive SMS or telephone alerts when an anomaly occurs in the cluster. The index management: Through in-depth development of Elasticsearch, near real-time crawling of index metadata is achieved; The index structure can be viewed, modified, or turned off in this functional module; The account management module manages Elasticsearch (ES) system accounts. In this module, users can define and authorize ES accounts according to the index prefix. The authorization types support read-only, write-only, and all. The one-click analysis function can diagnose cluster problems in real time by combining multiple dimensions such as ES cluster status, network conditions, index write speed, query speed, SLA, pending tasks, system and instance load, shard full load rate, index pool, query pool, and managed pool, and provide user-friendly and scientific suggestions. The specific implementation scheme for the ES cluster lifecycle management is as follows: By leveraging the custom form functionality of the work order system, users are provided with an entry point for automated creation, destruction, expansion, and shrinking of Elasticsearch. After the work order is approved, the information submitted by the user will be submitted to the ES management system for further processing. After the ES management system obtains the information that the work order has been approved, it will dynamically generate a YAML file that can be run by ElasticSearch Operator, based on the detailed configuration of the package and the template file. The ES management system submits the YAML file to the Kubernetes service API exposed by the container management system. This API then submits the YAML file to the ElasticSearch Operator for execution. After obtaining the YAML file, the ElasticSearch Operator will perform cluster creation, destruction, and scaling operations according to the file description, thereby realizing full lifecycle management of the ES cluster. By integrating the custom work order system, budget system, and CMDB system, developers can apply for an ES cluster on demand through work orders according to the package configuration; and report resource usage, utilization rate, and data center information to the enterprise's internal budget system and CMDB resource management system respectively.

2. The automated management system for managing an enterprise's internal Elasticsearch cluster according to claim 1, characterized in that: The overall indicator monitoring is divided into two parts: one is cluster monitoring and graphical representation, and the other is instance monitoring and graphical representation.

3. The automated management system for managing an enterprise's internal Elasticsearch cluster according to claim 2, characterized in that: The cluster monitoring and graphical interface includes cluster status monitoring, JVM monitoring, CPU monitoring, query monitoring, write time monitoring, GC monitoring, pending tasks monitoring, and sharding monitoring.

4. The automated management system for managing an enterprise's internal Elasticsearch cluster according to claim 2, characterized in that: The instance monitoring and graphical interface includes JVM monitoring, CPU monitoring, query time monitoring, query rate monitoring, index write rate monitoring, GC count monitoring, GC time monitoring, and rejection queue monitoring.

5. An automated management system for managing an enterprise's internal Elasticsearch cluster according to claim 1, characterized in that: The ES management system regularly reports the resource status and utilization of each cluster to the CMDB system. During the ES cluster lifecycle management process, the ES management system reports the cluster changes to the budget system for each cluster change, expansion, or contraction. When the budget is insufficient, it provides prompts and consumption bills. Users can also check the resource usage or consumption details in the CMDB system or budget system themselves.

6. The automated management system for managing an enterprise's internal Elasticsearch cluster according to claim 1, characterized in that: This section describes how to containerize an Elasticsearch cluster using Elastic Operator and dynamic templates, enabling cluster creation, destruction, scaling, and reduction. Elastic Operator is a technology for deploying and managing Elasticsearch clusters in a Kubernetes environment. Elasticsearch is an open-source distributed search and analytics engine with powerful data processing and search capabilities.

7. An automated management system for managing an enterprise's internal Elasticsearch cluster according to claim 1, characterized in that: Through system design and function development, ES cluster and node monitoring has been completed, supporting developers and operators to view the status and performance indicators of ES nodes and indexes. Alarms can be customized according to needs, and the lifecycle of indexes and ES account permissions can be managed.

Images