Kafka cluster mode clouding implementation system

Abstract

The invention discloses a Kafka cluster mode cloud implementation system, and the system comprises a Kafka Cluster resource module which is used for customizing a field to describe the configuration and state of a cluster; the KafkaTopic resource module is used for describing a topic object of the kafka cluster; the KafkaConnects resource module is used for describing the connection of the kafka; the KafkaPartitionreAssignments resource module is used for describing the re-balancing operation and maintenance operation of the kafka cluster partition, and the Kafka cluster partition is used for carrying out the re-balancing operation and maintenance operation of the kafka cluster partition; and the KafkaClusterOperator component module is used for describing a core component of the business logic of the Kafka cluster, and deploying and managing the Kafka cluster. The method can help enterprises to improve business flexibility, reduce operation and maintenance cost, and improve system reliability and safety.

Description

Technical Field

[0001] The present invention belongs to the technical field of cloud computing, and in particular is a system for implementing a Kafka cluster mode on the cloud. Background Art

[0002] As the amount of data and user demand grows, traditional locally deployed Kafka clusters may not be able to meet the requirements of elasticity and scalability. With the maturity and popularity of cloud computing technology, deploying Kafka clusters in the cloud can make it easier to dynamically expand the cluster size according to demand to cope with traffic peaks and data growth. Enjoy the elastic expansion, high availability and cost advantages brought by cloud computing. As a distributed messaging system, high availability and fault tolerance are crucial for Kafka. Deploying Kafka clusters in the cloud can take advantage of the high availability solutions provided by cloud technology, such as cross-availability zones or cross-regional deployment, to ensure data persistence and reliability. At the same time, cloud deployment platforms usually provide a wealth of management and monitoring tools that can help administrators more easily manage and monitor the operating status of Kafka clusters, including automated expansion, fault detection and repair, and other functions.

[0003] In cloud environments, Statefulset K8S native resource declarations are generally used to deploy stateful applications such as middleware. StatefulSet is a controller in Kubernetes that is used to manage stateful applications. Unlike Deployment, which is mainly used to manage stateless applications, StatefulSet is mainly used to manage stateful applications that require stable network identifiers and stable storage. StatefulSet ensures that each Pod instance has a unique and stable host name that follows specific naming rules and can be distinguished from other Pod instances. These Pod names remain unchanged during the Pod life cycle and can be used to communicate with other Pods, i.e., internal DNS access addresses. In addition, StatefulSet can also manage the stable and persistent storage volumes required by the application, determine the topology of the cluster foundation, and ensure that they can correctly return to the original state of the specific Pod instance after Pod migration or restart.

[0004] Implementing KafkaCluster on the cloud using the standard Statefulset method has the following problems:

[0005] Statefulset only provides a default logical entry. You can only control the startup order of Pods in StatefulSet through the podManagementPolicy field in the StatefulSet spec. You cannot customize more actions related to cluster services. For example, when the cluster is started, you need to initialize the Kafka nodes in steps and then assemble the current nodes into a Kafka cluster. Such operations cannot be well implemented in StatefulSet.

[0006] The internal maintenance logic of KafkaCluster is relatively complex. It is not possible to implement complex logic based on the native Statefulset through simple parameter interface configuration. For example, it is not possible to achieve the expected automatic operation and maintenance, high availability and other requirements in the container cloud environment, such as rolling upgrades and deletion logic. In addition, during the rolling upgrade process, we hope that the cluster is stable and reliable to ensure that the cluster can continue to provide services. Summary of the invention

[0007] In view of the above problems, the present invention is proposed to provide a Kafka cluster mode cloud implementation system that overcomes the above problems or at least partially solves the above problems.

[0008] To achieve the above object, the present invention adopts the following technical solutions:

[0009] A Kafka cluster mode cloud implementation system, the system includes:

[0010] KafkaCluster resource module, which is used to customize fields to describe the configuration and status of the cluster;

[0011] KafkaTopics resource module, used to describe the topic object of the Kafka cluster;

[0012] KafkaConnects resource module, used to describe Kafka connections;

[0013] KafkaPartitionreAssignments resource module, used to describe the Kafka cluster partition rebalancing operation and maintenance operations;

[0014] The KafkaClusterOperator component module is used to describe the core components of the Kafka cluster business logic and deploy and manage the Kafka cluster.

[0015] Optionally, the KafkaCluster resource module is combined with a custom Operator controller to realize automatic creation and operation and maintenance of Kafka.

[0016] Optionally, the implementation of automatic creation and operation of Kafka includes:

[0017] Use the KafkaCluster resource module to describe the node list configuration of the cluster, create the configuration files required by the Kafka cluster, and start Kafka nodes of different roles by specifying different configuration files;

[0018] After monitoring the creation of the KafkaCluster object, the Operator controller creates the dependent ConfigMap, Volume, and Service sub-object resources based on the object's description information, and then starts the cluster nodes.

[0019] Optionally, the KafkaTopics resource module combines with a custom Operator controller to provide automatic creation, update, deletion, and query of topic objects.

[0020] Optionally, the KafkaConnects resource module is divided into two types: sink and source. The sink type describes backing up the message flow within the Kafka cluster to an external data source, and the source type describes restoring the message flow of the Kafka external data source to an instance within the Kafka cluster.

[0021] Optionally, the KafkaConnects resource module includes quickly starting a stream processing program with the kafka cluster instance as the object through the description of KafkaConnects combined with the collaboration of the KafkaCluster resource object.

[0022] Optionally, the KafkaPartitionreAssignments resource module records the partition distribution of the Kafka partition before rebalancing and the new partition distribution and proposes it for the user to perform operation and maintenance operations.

[0023] Optionally, the KafkaPartitionReassignments resource module includes implementing a partition reallocation function with a kafka cluster instance as an object through the description of KafkaPartitionReassignments combined with the collaboration of a KafkaCluster resource object.

[0024] Optionally, the KafkaClusterOperator component module is built on the Kubernetes CRD framework.

[0025] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0026] 1. The present invention automates daily business processes, reduces manual intervention, and improves efficiency; it uses workflow automation tools to manage and optimize business processes, which can help enterprises improve business flexibility and reduce operation and maintenance costs.

[0027] 2. The present invention uses cloud computing services to allocate resources on demand and improve resource utilization. Through virtualization technology, dynamic allocation and optimization of resources are achieved. Disaster recovery plans are formulated and implemented to ensure business continuity. Data backup and recovery drills are performed regularly. The reliability and security of the system are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A schematic diagram of a system flow for implementing a Kafka cluster mode on the cloud provided in an embodiment of the present application;

[0029] Figure 2 A design architecture diagram of a Kafka cluster mode cloud implementation system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0031] See also Figure 1 and Figure 2 This embodiment provides a system for implementing a Kafka cluster mode on the cloud, the system comprising:

[0032] KafkaCluster resource module, used to customize fields to describe the configuration and status of the cluster.

[0033] The KafkaCluster resource module combines with a custom Operator controller to realize automatic creation and operation and maintenance of Kafka.

[0034] The implementation of automatic creation and operation of Kafka includes:

[0035] Use the KafkaCluster resource module to describe the node list configuration of the cluster, create the configuration files required by the Kafka cluster, and start Kafka nodes of different roles by specifying different configuration files.

[0036] After monitoring the creation of the KafkaCluster object, the Operator controller creates the dependent ConfigMap, Volume, and Service sub-object resources based on the object's description information, and then starts the cluster nodes.

[0037] The KafkaTopics resource module is used to describe the topic object of the Kafka cluster and provides complete topic parameter configuration options.

[0038] The KafkaTopics resource module combines with a custom Operator controller to provide automatic creation, update, deletion, and query of topic objects.

[0039] The KafkaConnects resource module is used to describe the connection of Kafka and the real-time backup of messages from the topic of the Kafka cluster to a new topic. It supports real-time backup and recovery of synchronized data within or across clusters.

[0040] The KafkaConnects resource module is divided into two types: sink and source. The sink type describes the backup of the message flow within the Kafka cluster to an external data source, and the source type describes the restoration of the message flow of the Kafka external data source to the internal instance of the Kafka cluster.

[0041] The KafkaConnects resource module includes a method of quickly starting a stream processing program with a kafka cluster instance as an object through the description of KafkaConnects combined with the collaboration of the KafkaCluster resource object, so as to facilitate business users to import or export data simply and conveniently.

[0042] The KafkaPartitionreAssignments resource module is used to describe the Kafka cluster partition rebalancing operation and maintenance operations. It provides partition allocation recommendation solutions and supports user-defined allocation relationships. By adding a few simple parameters and submitting them to the cluster, the Operator executes the Kafka cluster operation and maintenance commands according to the object description.

[0043] The KafkaPartitionreAssignments resource module records the partition distribution of Kafka partitions before rebalancing and the new partition distribution and proposes it for users to perform operation and maintenance operations.

[0044] The KafkaPartitionReassignments resource module includes implementing the partition reallocation function with the Kafka cluster instance as the object through the description of KafkaPartitionReassignments combined with the collaboration of the KafkaCluster resource object, so as to facilitate business users to manage data in the partitions.

[0045] The KafkaClusterOperator component module is used to describe the core components of the Kafka cluster business logic and deploy and manage the Kafka cluster.

[0046] The KafkaClusterOperator component module is built on the Kubernetes CRD framework, providing a simple and easy-to-use interface for Kafka to deploy a Kafka cluster on Kubernetes.

[0047] This embodiment automates daily business processes, reduces manual intervention, and improves efficiency; workflow automation tools are used to manage and optimize business processes, which can help enterprises improve business flexibility and reduce operation and maintenance costs.

[0048] This embodiment uses cloud computing services to allocate resources on demand and improve resource utilization. Through virtualization technology, dynamic allocation and optimization of resources are achieved. Disaster recovery plans are formulated and implemented to ensure business continuity. Data backup and recovery drills are performed regularly. The reliability and security of the system are improved.

[0049] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A Kafka cluster mode cloud implementation system, characterized by: The system comprises: KafkaCluster resource module, which is used to customize fields to describe the configuration and status of the cluster; KafkaTopics resource module, used to describe the topic object of the Kafka cluster; KafkaConnects resource module, used to describe Kafka connections; KafkaPartitionreAssignments resource module, used to describe the Kafka cluster partition rebalancing operation and maintenance operations; The KafkaClusterOperator component module is used to describe the core components of the Kafka cluster business logic and deploy and manage the Kafka cluster.

2. A Kafka cluster mode cloud implementation system as described in claim 1, characterized in that: The KafkaCluster resource module combines with a custom Operator controller to realize automatic creation and operation and maintenance of Kafka.

3. A Kafka cluster mode cloud implementation system as described in claim 2, characterized in that: The implementation of automatic creation and operation of Kafka includes: Use the KafkaCluster resource module to describe the node list configuration of the cluster, create the configuration files required by the Kafka cluster, and start Kafka nodes of different roles by specifying different configuration files; After monitoring the creation of the KafkaCluster object, the Operator controller creates the dependent ConfigMap, Volume, and Service sub-object resources based on the object's description information, and then starts the cluster nodes.

4. A Kafka cluster mode cloud implementation system as claimed in claim 1, characterized in that: The KafkaTopics resource module combines with a custom Operator controller to provide automatic creation, update, deletion, and query of topic objects.

5. A Kafka cluster mode cloud implementation system as claimed in claim 1, characterized in that: The KafkaConnects resource module is divided into two types: sink and source. The sink type describes the backup of the message flow within the Kafka cluster to an external data source, and the source type describes the restoration of the message flow of the Kafka external data source to the internal instance of the Kafka cluster.

6. A Kafka cluster mode cloud implementation system as claimed in claim 1, characterized in that: The KafkaConnects resource module includes a method for quickly starting a stream processing program with a Kafka cluster instance as an object by combining the description of KafkaConnects with the collaboration of a KafkaCluster resource object.

7. A Kafka cluster mode cloud implementation system as claimed in claim 1, characterized in that: The KafkaPartitionreAssignments resource module records the partition distribution of Kafka partitions before rebalancing and the new partition distribution and proposes it for users to perform operation and maintenance operations.

8. A Kafka cluster mode cloud implementation system as claimed in claim 1, characterized in that: The KafkaPartitionReassignments resource module includes implementing a partition reallocation function using a kafka cluster instance as an object through the description of KafkaPartitionReassignments combined with the collaboration of a KafkaCluster resource object.

9. A Kafka cluster mode cloud implementation system as claimed in claim 1, characterized in that: The KafkaClusterOperator component module is built on the Kubernetes CRD framework.

Images